使用 Docker 建置 QA 環境

這篇是延續上一篇《整合 Jenkins 和 Docker》，使用 Docker 來建置 QA 環境的想法。

在上一篇有提到，原本的設計是當 Github 收到 Pull Request 時，就讓 Jenkins 來跑測試，如果測試通過，就直接用原本建置出來的 docker image 建立起一個 QA 環境，這樣就可以直接透過連到 <ticket-number>.qa.domain.internal 的方式來驗收，確定無誤後再按下 merge 按鈕，然後 trigger 自動關閉此 QA 環境。

不過後來因為這個情境不適合我們的 workflow, 所以最後沒有這樣實做。而是寫成 rake task 來運用。

具體的方式是，把整個 docker 中的 app 目錄 mount 出來，然後 nginx 透過 subdomain 來決定 app root 和 proxy_pass backend. 由於可能會有許多 qa 環境同時存在，要弄轉 port 還挺麻煩的，所以這邊都使用 unix domain socket.

Nginx

先來看一下 nginx 這邊的設定，其實很單純，簡化過後大概就長這樣：

server { listen 80; server_name "~^(?<sub>.+)\.qa\.domain\.internal$" "~^(.+)\.(?<sub>.+)\.qa-cc\.domain\.internal$"; root /var/www/qa/$sub/appname/public; charset utf-8; try_files $uri/index.html $uri.html $uri @app; location @app { proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; proxy_redirect off; proxy_pass http://unix:/var/www/qa/$sub/appname/puma.sock; error_page 502 /qa_404.html; # Does not exist. } location ~* ^\/(images|javascripts)\/.*\.(ico|css|js|gif|jpe?g|png)(\?[0-9]+)?$ { expires max; break; } }

這樣透過 newfeature.qa.domain.internal 就可以連到從 docker 裡面 mount 出來到 /var/www/newfeature 這個目錄的靜態檔案和 unix domain socket.

Makefile

Makefile 是用來 build docker image, 這部分做的事情大致上是切到一個 workspace 目錄，此目錄是我 app 的 git repo, 然後切到我要建置的 QA branch 後開始 build docker image.

all: base gem test qa base: docker build --rm --no-cache -t myapp/base base gem: fetch_workspace rm -f gem/* cp workspace/myapp/Gemfile* gem/ test: cp -R gem test/ docker build --rm --no-cache -t myapp/test test rm -rf test/gem qa: cp -R gem qa/ docker build --rm --no-cache -t myapp/qa qa rm -rf qa/gem clean_workspace: cd workspace/myapp;\ git clean -f;\ git reset --hard HEAD fetch_workspace: clean_workspace cd workspace/myapp;\ git fetch;\ git checkout master;\ git pull qa_branch: ifdef QA_BRANCH $(info ******* Start building branch ${QA_BRANCH} for name ${QA_DOMAIN} ********) # Check out the branch cd workspace/myapp;\ git clean -f;\ git reset --hard HEAD;\ git fetch;\ git checkout ${QA_BRANCH};\ git pull # Copy required files cp -r qa/config/* workspace/myapp/config/ # use sed to replace some config in config/*.yml here. cp qa/*.sh workspace/myapp/ echo "FROM myapp/qa" > workspace/myapp/Dockerfile # Store branch and domain name into files echo "${QA_DOMAIN}" > workspace/myapp/qa_domain echo "${QA_BRANCH}" > workspace/myapp/qa_branch # Build docker build --rm -t myapp_qa/${QA_DOMAIN} workspace/myapp/ # Clean cd workspace/myapp;\ git clean -f;\ git reset --hard HEAD;\ git checkout master else $(info ******* Please specify QA_BRANCH ********) endif .PHONY: all base test qa qa_branch gem

gem 的部分再上一篇有提過，預先安裝 gem 是為了加速整個建置的過程，這邊會丟到 crontab 每天晚上自動執行更新。

執行 make qa_branch QA_BRANCH=master QA_DOMAIN=master 即可建置出特定 branch 的 docker image.

可以看到我直接把 branch name 和 domain name 寫到 app 資料夾中，這樣就不用資料庫來紀錄什麼 domain name 對應什麼 branch 了。

這邊需要注意由於 domain name 用在三個地方：

domain
路徑
docker image name

比較麻煩的是 domain name 只能有 - 不能有 _, 而 docker image name 則剛好相反，不能有 - 只能有 _ 所以在輸入名稱的時候要特別注意。由於我外面是包 messaging bot 來下指令，那邊有做檢查，所以這邊就沒有另外再做檢查。

Rakefile

Rakefile 是拿來啟動 / 關閉 QA 環境用的。

require 'yaml' require 'docker' require 'pty' require 'slack-notifier' Docker.url = "tcp://127.0.0.1:4243" QA_DIR = "/var/www/qa" QA_URL = "qa.domain.internal" QA_IMAGE_REGEX = /myapp_qa\/(\w+):latest/ ROOT_PATH = File.dirname(__FILE__) desc "List running containers" task :running do puts `docker ps` end desc "Clean stopped containers" task :remove_stopped_containers do puts "----- Cleaning stopped containers -----" puts `#{ROOT_PATH}/bin/remove_stopped_containers.sh` end namespace :qa do desc "List all running QA containers" task :list do running_qa_domains = qa_running_containers.map {|c| c.info["Image"].match(QA_IMAGE_REGEX)[1]} puts "----- Current QA containers -----" running_qa_domains.each do |domain| puts "http://#{domain}.#{QA_URL} (#{`cat #{QA_DIR}/#{domain}/myapp/qa_branch`.strip})" end end desc "Build and start a QA container" task :start, [:branch_name, :domain_name] do |t, args| branch_name = args[:branch_name] domain_name = args[:domain_name] hipchat_notify("Start to build QA container *myapp_qa/#{domain_name}* for :branch: *#{branch_name}*") puts "----- Building QA container myapp_qa/#{domain_name} for branch #{branch_name}" pipe_exec "cd #{ROOT_PATH} && make qa_branch QA_BRANCH=#{branch_name} QA_DOMAIN=#{domain_name}" puts "----- Starting QA container myapp_qa/#{domain_name} for branch #{branch_name}" puts `sudo rm -rf #{QA_DIR}/#{domain_name}` # Remove workspace if it existed pipe_exec "docker run -d -i -t -v #{QA_DIR}/#{domain_name}:/opt/run:rw --privileged myapp_qa/#{domain_name}" puts "----- Running and initializing fake data and indexes... -----" status_file = "#{QA_DIR}/#{domain_name}/myapp/status" status = "init" loop do # Loop until ready. if File.exists?(status_file) current_status = File.read(status_file).strip break if current_status == "ready" if status != current_status puts "#### Status changed from #{status} to #{current_status} ####" status = current_status end end sleep 5 end puts "=====> http://#{domain_name}.#{QA_URL} <=====" hipchat_notify("QA environment for :branch: *#{branch_name}* is ready at http://#{domain_name}.#{QA_URL}") end desc "Stop QA container" task :stop, [:domain_name] do |t, args| domain_name = args[:domain_name] container = nil Docker::Container.all.each do |c| container = c if c.info["Image"] == "myapp_qa/#{domain_name}:latest" end if container puts "----- Stopping QA container #{domain_name} -----" puts container.stop puts "----- Deleting QA container #{domain_name} -----" puts container.delete puts "----- Deleting QA workspace #{domain_name} -----" puts `sudo rm -rf #{QA_DIR}/#{domain_name}` puts "----- Deleting QA image myapp_qa/#{domain_name} -----" puts `docker rmi myapp_qa/#{domain_name}` puts "----- Done. -----" else puts "There's no running QA container of domain #{domain_name}" end end desc "Clean stopped container and remove unused workspaces." task :clean do Rake::Task["remove_stopped_containers"].invoke running_qa_domains = qa_running_containers.map {|c| c.info["Image"].match(QA_IMAGE_REGEX)[1]} puts "----- Deleting unused QA images -----" qa_images.map do |image| unless running_qa_domains.include?(image.info["RepoTags"].first.match(QA_IMAGE_REGEX)[1]) puts "Image #{image.info["RepoTags"].first} removed." image.remove end end puts "----- Cleaning stopped QA workspaces -----" Dir.glob("#{QA_DIR}/*").each do |f| unless running_qa_domains.include?(File.basename(f)) puts `sudo rm -rf #{f}` end end end end ######## Helpers ######## # def pipe_exec(command) begin PTY.spawn(command) do |stdin, stdout, pid| begin stdin.each { |line| print line } rescue Errno::EIO end end rescue PTY::ChildExited puts "The child process exited!" end end def qa_running_containers Docker::Container.all.select {|c| c.info["Image"] =~ QA_IMAGE_REGEX } end def qa_images Docker::Image.all.select { |c| c.info["RepoTags"].first =~ QA_IMAGE_REGEX } end def hipchat_notify(msg) config = YAML.load_file('config/config.yml') return if config['slack_token'] == nil || config['slack_token'] == "" notifier = Slack::Notifier.new "kkbox", config['slack_token'], channel: '#myapp', username: 'Docker' notifier.ping msg, icon_emoji: ":myapp:" end

這邊值得一提的是，由於 container build 好，到整個 service run 起來其實還有一段時間差，是用來啟動 service, initial db 等等動作…所以在 app 目錄下會寫一個 status file 來判斷現在 app initial 到什麼階段，等他變成 ready 後才判斷為建置完成。

start_rails.sh

這是 QA docker image run 起來後會執行的 script, 比較重要的有兩個部分，一個是上面提的把目前階段寫入 status 檔案，讓外面知道現在進行到什麼步驟；另一個則是使用 socat 把 port 轉為 unix domain socket 再 mount 出去，就可以讓外面的 nginx 跟 docker 內部的 web services 溝通，而不需要處理 docker 的 port 了。

把 8080 port 轉到 /opt/run/myapp/go.sock：

socat UNIX-LISTEN:/opt/run/myapp/go.sock,reuseaddr,fork TCP:localhost:8080

echo "dev:fake" > /opt/run/myapp/status cd /opt/run/myapp && rvm all do bundle exec rake dev:build cd /opt/run/myapp && rvm all do bundle exec rake dev:load cd /opt/run/myapp && rvm all do unicorn_rails -c config/unicorn.rb -D echo "building golang" > /opt/run/myapp/status if [ -d "/opt/run/myapp/go" ]; then #cd /opt/run/myapp/go && gom install # This takes some time. cd /opt/run/myapp/go && ./make build cd /opt/run/myapp/go && ./kktix_go start -d socat UNIX-LISTEN:/opt/run/myapp/go.sock,reuseaddr,fork TCP:localhost:8080 & fi echo "sidekiq" > /opt/run/myapp/status cd /opt/run/myapp && rvm all do bundle exec sidekiq -c 5 -d -L log/sidekiq.log & cd /opt/run/myapp && rvm all do ruby -rsidekiq/api -e 'loop do;count = Sidekiq::Stats.new.enqueued;puts "Remaining jobs: #{count}";if count == 0;`echo "ready" > /opt/run/kktix/status`;break;end;sleep 5;end' if [ -e "/opt/run/myapp/go.sock" ]; then chmod 777 /opt/run/myapp/go.sock fi cd /opt/run/myapp && /bin/bash

Conclusion

這篇的 scripts 有點繁雜，不過概念其實很簡單，只是細節上有不少需要注意的地方。

使用 docker 來建置 QA branch 可以方便快速的建出乾淨的環境，相較以往要處理非常多資料庫 / 轉 port / services isolation 等等問題，實在是輕鬆太多了。

hSATAC