使用 Docker 建置 QA 環境

這篇是延續上一篇《整合 Jenkins 和 Docker》,使用 Docker 來建置 QA 環境的想法。

在上一篇有提到,原本的設計是當 Github 收到 Pull Request 時,就讓 Jenkins 來跑測試,如果測試通過,就直接用原本建置出來的 docker image 建立起一個 QA 環境,這樣就可以直接透過連到 <ticket-number>.qa.domain.internal 的方式來驗收,確定無誤後再按下 merge 按鈕,然後 trigger 自動關閉此 QA 環境。

不過後來因為這個情境不適合我們的 workflow, 所以最後沒有這樣實做。而是寫成 rake task 來運用。

具體的方式是,把整個 docker 中的 app 目錄 mount 出來,然後 nginx 透過 subdomain 來決定 app root 和 proxy_pass backend. 由於可能會有許多 qa 環境同時存在,要弄轉 port 還挺麻煩的,所以這邊都使用 unix domain socket.

Nginx

先來看一下 nginx 這邊的設定,其實很單純,簡化過後大概就長這樣:

server {
listen 80;
server_name "~^(?<sub>.+)\.qa\.domain\.internal$" "~^(.+)\.(?<sub>.+)\.qa-cc\.domain\.internal$";
root /var/www/qa/$sub/appname/public;
charset utf-8;
try_files $uri/index.html $uri.html $uri @app;
location @app {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://unix:/var/www/qa/$sub/appname/puma.sock;
error_page 502 /qa_404.html; # Does not exist.
}
location ~* ^\/(images|javascripts)\/.*\.(ico|css|js|gif|jpe?g|png)(\?[0-9]+)?$ {
expires max;
break;
}
}
view raw nginx.conf hosted with ❤ by GitHub

這樣透過 newfeature.qa.domain.internal 就可以連到從 docker 裡面 mount 出來到 /var/www/newfeature 這個目錄的靜態檔案和 unix domain socket.

Makefile

Makefile 是用來 build docker image, 這部分做的事情大致上是切到一個 workspace 目錄,此目錄是我 app 的 git repo, 然後切到我要建置的 QA branch 後開始 build docker image.

all: base gem test qa
base:
docker build --rm --no-cache -t myapp/base base
gem: fetch_workspace
rm -f gem/*
cp workspace/myapp/Gemfile* gem/
test:
cp -R gem test/
docker build --rm --no-cache -t myapp/test test
rm -rf test/gem
qa:
cp -R gem qa/
docker build --rm --no-cache -t myapp/qa qa
rm -rf qa/gem
clean_workspace:
cd workspace/myapp;\
git clean -f;\
git reset --hard HEAD
fetch_workspace: clean_workspace
cd workspace/myapp;\
git fetch;\
git checkout master;\
git pull
qa_branch:
ifdef QA_BRANCH
$(info ******* Start building branch ${QA_BRANCH} for name ${QA_DOMAIN} ********)
# Check out the branch
cd workspace/myapp;\
git clean -f;\
git reset --hard HEAD;\
git fetch;\
git checkout ${QA_BRANCH};\
git pull
# Copy required files
cp -r qa/config/* workspace/myapp/config/
# use sed to replace some config in config/*.yml here.
cp qa/*.sh workspace/myapp/
echo "FROM myapp/qa" > workspace/myapp/Dockerfile
# Store branch and domain name into files
echo "${QA_DOMAIN}" > workspace/myapp/qa_domain
echo "${QA_BRANCH}" > workspace/myapp/qa_branch
# Build
docker build --rm -t myapp_qa/${QA_DOMAIN} workspace/myapp/
# Clean
cd workspace/myapp;\
git clean -f;\
git reset --hard HEAD;\
git checkout master
else
$(info ******* Please specify QA_BRANCH ********)
endif
.PHONY: all base test qa qa_branch gem
view raw Makefile hosted with ❤ by GitHub

gem 的部分再上一篇有提過,預先安裝 gem 是為了加速整個建置的過程,這邊會丟到 crontab 每天晚上自動執行更新。

執行 make qa_branch QA_BRANCH=master QA_DOMAIN=master 即可建置出特定 branch 的 docker image.

可以看到我直接把 branch name 和 domain name 寫到 app 資料夾中,這樣就不用資料庫來紀錄什麼 domain name 對應什麼 branch 了。

這邊需要注意由於 domain name 用在三個地方:

  1. domain
  2. 路徑
  3. docker image name

比較麻煩的是 domain name 只能有 - 不能有 _, 而 docker image name 則剛好相反,不能有 - 只能有 _ 所以在輸入名稱的時候要特別注意。由於我外面是包 messaging bot 來下指令,那邊有做檢查,所以這邊就沒有另外再做檢查。

Rakefile

Rakefile 是拿來啟動 / 關閉 QA 環境用的。

require 'yaml'
require 'docker'
require 'pty'
require 'slack-notifier'
Docker.url = "tcp://127.0.0.1:4243"
QA_DIR = "/var/www/qa"
QA_URL = "qa.domain.internal"
QA_IMAGE_REGEX = /myapp_qa\/(\w+):latest/
ROOT_PATH = File.dirname(__FILE__)
desc "List running containers"
task :running do
puts `docker ps`
end
desc "Clean stopped containers"
task :remove_stopped_containers do
puts "----- Cleaning stopped containers -----"
puts `#{ROOT_PATH}/bin/remove_stopped_containers.sh`
end
namespace :qa do
desc "List all running QA containers"
task :list do
running_qa_domains = qa_running_containers.map {|c| c.info["Image"].match(QA_IMAGE_REGEX)[1]}
puts "----- Current QA containers -----"
running_qa_domains.each do |domain|
puts "http://#{domain}.#{QA_URL} (#{`cat #{QA_DIR}/#{domain}/myapp/qa_branch`.strip})"
end
end
desc "Build and start a QA container"
task :start, [:branch_name, :domain_name] do |t, args|
branch_name = args[:branch_name]
domain_name = args[:domain_name]
hipchat_notify("Start to build QA container *myapp_qa/#{domain_name}* for :branch: *#{branch_name}*")
puts "----- Building QA container myapp_qa/#{domain_name} for branch #{branch_name}"
pipe_exec "cd #{ROOT_PATH} && make qa_branch QA_BRANCH=#{branch_name} QA_DOMAIN=#{domain_name}"
puts "----- Starting QA container myapp_qa/#{domain_name} for branch #{branch_name}"
puts `sudo rm -rf #{QA_DIR}/#{domain_name}` # Remove workspace if it existed
pipe_exec "docker run -d -i -t -v #{QA_DIR}/#{domain_name}:/opt/run:rw --privileged myapp_qa/#{domain_name}"
puts "----- Running and initializing fake data and indexes... -----"
status_file = "#{QA_DIR}/#{domain_name}/myapp/status"
status = "init"
loop do # Loop until ready.
if File.exists?(status_file)
current_status = File.read(status_file).strip
break if current_status == "ready"
if status != current_status
puts "#### Status changed from #{status} to #{current_status} ####"
status = current_status
end
end
sleep 5
end
puts "=====> http://#{domain_name}.#{QA_URL} <====="
hipchat_notify("QA environment for :branch: *#{branch_name}* is ready at http://#{domain_name}.#{QA_URL}")
end
desc "Stop QA container"
task :stop, [:domain_name] do |t, args|
domain_name = args[:domain_name]
container = nil
Docker::Container.all.each do |c|
container = c if c.info["Image"] == "myapp_qa/#{domain_name}:latest"
end
if container
puts "----- Stopping QA container #{domain_name} -----"
puts container.stop
puts "----- Deleting QA container #{domain_name} -----"
puts container.delete
puts "----- Deleting QA workspace #{domain_name} -----"
puts `sudo rm -rf #{QA_DIR}/#{domain_name}`
puts "----- Deleting QA image myapp_qa/#{domain_name} -----"
puts `docker rmi myapp_qa/#{domain_name}`
puts "----- Done. -----"
else
puts "There's no running QA container of domain #{domain_name}"
end
end
desc "Clean stopped container and remove unused workspaces."
task :clean do
Rake::Task["remove_stopped_containers"].invoke
running_qa_domains = qa_running_containers.map {|c| c.info["Image"].match(QA_IMAGE_REGEX)[1]}
puts "----- Deleting unused QA images -----"
qa_images.map do |image|
unless running_qa_domains.include?(image.info["RepoTags"].first.match(QA_IMAGE_REGEX)[1])
puts "Image #{image.info["RepoTags"].first} removed."
image.remove
end
end
puts "----- Cleaning stopped QA workspaces -----"
Dir.glob("#{QA_DIR}/*").each do |f|
unless running_qa_domains.include?(File.basename(f))
puts `sudo rm -rf #{f}`
end
end
end
end
######## Helpers ########
#
def pipe_exec(command)
begin
PTY.spawn(command) do |stdin, stdout, pid|
begin
stdin.each { |line| print line }
rescue Errno::EIO
end
end
rescue PTY::ChildExited
puts "The child process exited!"
end
end
def qa_running_containers
Docker::Container.all.select {|c| c.info["Image"] =~ QA_IMAGE_REGEX }
end
def qa_images
Docker::Image.all.select { |c| c.info["RepoTags"].first =~ QA_IMAGE_REGEX }
end
def hipchat_notify(msg)
config = YAML.load_file('config/config.yml')
return if config['slack_token'] == nil || config['slack_token'] == ""
notifier = Slack::Notifier.new "kkbox", config['slack_token'], channel: '#myapp', username: 'Docker'
notifier.ping msg, icon_emoji: ":myapp:"
end
view raw Rakefile hosted with ❤ by GitHub

這邊值得一提的是,由於 container build 好,到整個 service run 起來其實還有一段時間差,是用來啟動 service, initial db 等等動作…所以在 app 目錄下會寫一個 status file 來判斷現在 app initial 到什麼階段,等他變成 ready 後才判斷為建置完成。

start_rails.sh

這是 QA docker image run 起來後會執行的 script, 比較重要的有兩個部分,一個是上面提的把目前階段寫入 status 檔案,讓外面知道現在進行到什麼步驟;另一個則是使用 socat 把 port 轉為 unix domain socket 再 mount 出去,就可以讓外面的 nginx 跟 docker 內部的 web services 溝通,而不需要處理 docker 的 port 了。

把 8080 port 轉到 /opt/run/myapp/go.sock

socat UNIX-LISTEN:/opt/run/myapp/go.sock,reuseaddr,fork TCP:localhost:8080

echo "dev:fake" > /opt/run/myapp/status
cd /opt/run/myapp && rvm all do bundle exec rake dev:build
cd /opt/run/myapp && rvm all do bundle exec rake dev:load
cd /opt/run/myapp && rvm all do unicorn_rails -c config/unicorn.rb -D
echo "building golang" > /opt/run/myapp/status
if [ -d "/opt/run/myapp/go" ]; then
#cd /opt/run/myapp/go && gom install # This takes some time.
cd /opt/run/myapp/go && ./make build
cd /opt/run/myapp/go && ./kktix_go start -d
socat UNIX-LISTEN:/opt/run/myapp/go.sock,reuseaddr,fork TCP:localhost:8080 &
fi
echo "sidekiq" > /opt/run/myapp/status
cd /opt/run/myapp && rvm all do bundle exec sidekiq -c 5 -d -L log/sidekiq.log &
cd /opt/run/myapp && rvm all do ruby -rsidekiq/api -e 'loop do;count = Sidekiq::Stats.new.enqueued;puts "Remaining jobs: #{count}";if count == 0;`echo "ready" > /opt/run/kktix/status`;break;end;sleep 5;end'
if [ -e "/opt/run/myapp/go.sock" ]; then
chmod 777 /opt/run/myapp/go.sock
fi
cd /opt/run/myapp && /bin/bash
view raw start_rails.sh hosted with ❤ by GitHub

Conclusion

這篇的 scripts 有點繁雜,不過概念其實很簡單,只是細節上有不少需要注意的地方。

使用 docker 來建置 QA branch 可以方便快速的建出乾淨的環境,相較以往要處理非常多資料庫 / 轉 port / services isolation 等等問題,實在是輕鬆太多了。

Comments