使用 Docker 建置 QA 環境
這篇是延續上一篇《整合 Jenkins 和 Docker》,使用 Docker 來建置 QA 環境的想法。
在上一篇有提到,原本的設計是當 Github 收到 Pull Request 時,就讓 Jenkins 來跑測試,如果測試通過,就直接用原本建置出來的 docker image 建立起一個 QA 環境,這樣就可以直接透過連到 <ticket-number>.qa.domain.internal
的方式來驗收,確定無誤後再按下 merge 按鈕,然後 trigger 自動關閉此 QA 環境。
不過後來因為這個情境不適合我們的 workflow, 所以最後沒有這樣實做。而是寫成 rake task 來運用。
具體的方式是,把整個 docker 中的 app 目錄 mount 出來,然後 nginx 透過 subdomain 來決定 app root 和 proxy_pass backend. 由於可能會有許多 qa 環境同時存在,要弄轉 port 還挺麻煩的,所以這邊都使用 unix domain socket.
Nginx
先來看一下 nginx 這邊的設定,其實很單純,簡化過後大概就長這樣:
server { | |
listen 80; | |
server_name "~^(?<sub>.+)\.qa\.domain\.internal$" "~^(.+)\.(?<sub>.+)\.qa-cc\.domain\.internal$"; | |
root /var/www/qa/$sub/appname/public; | |
charset utf-8; | |
try_files $uri/index.html $uri.html $uri @app; | |
location @app { | |
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; | |
proxy_set_header Host $http_host; | |
proxy_redirect off; | |
proxy_pass http://unix:/var/www/qa/$sub/appname/puma.sock; | |
error_page 502 /qa_404.html; # Does not exist. | |
} | |
location ~* ^\/(images|javascripts)\/.*\.(ico|css|js|gif|jpe?g|png)(\?[0-9]+)?$ { | |
expires max; | |
break; | |
} | |
} |
這樣透過 newfeature.qa.domain.internal
就可以連到從 docker 裡面 mount 出來到 /var/www/newfeature
這個目錄的靜態檔案和 unix domain socket.
Makefile
Makefile 是用來 build docker image, 這部分做的事情大致上是切到一個 workspace 目錄,此目錄是我 app 的 git repo, 然後切到我要建置的 QA branch 後開始 build docker image.
all: base gem test qa | |
base: | |
docker build --rm --no-cache -t myapp/base base | |
gem: fetch_workspace | |
rm -f gem/* | |
cp workspace/myapp/Gemfile* gem/ | |
test: | |
cp -R gem test/ | |
docker build --rm --no-cache -t myapp/test test | |
rm -rf test/gem | |
qa: | |
cp -R gem qa/ | |
docker build --rm --no-cache -t myapp/qa qa | |
rm -rf qa/gem | |
clean_workspace: | |
cd workspace/myapp;\ | |
git clean -f;\ | |
git reset --hard HEAD | |
fetch_workspace: clean_workspace | |
cd workspace/myapp;\ | |
git fetch;\ | |
git checkout master;\ | |
git pull | |
qa_branch: | |
ifdef QA_BRANCH | |
$(info ******* Start building branch ${QA_BRANCH} for name ${QA_DOMAIN} ********) | |
# Check out the branch | |
cd workspace/myapp;\ | |
git clean -f;\ | |
git reset --hard HEAD;\ | |
git fetch;\ | |
git checkout ${QA_BRANCH};\ | |
git pull | |
# Copy required files | |
cp -r qa/config/* workspace/myapp/config/ | |
# use sed to replace some config in config/*.yml here. | |
cp qa/*.sh workspace/myapp/ | |
echo "FROM myapp/qa" > workspace/myapp/Dockerfile | |
# Store branch and domain name into files | |
echo "${QA_DOMAIN}" > workspace/myapp/qa_domain | |
echo "${QA_BRANCH}" > workspace/myapp/qa_branch | |
# Build | |
docker build --rm -t myapp_qa/${QA_DOMAIN} workspace/myapp/ | |
# Clean | |
cd workspace/myapp;\ | |
git clean -f;\ | |
git reset --hard HEAD;\ | |
git checkout master | |
else | |
$(info ******* Please specify QA_BRANCH ********) | |
endif | |
.PHONY: all base test qa qa_branch gem |
gem 的部分再上一篇有提過,預先安裝 gem 是為了加速整個建置的過程,這邊會丟到 crontab 每天晚上自動執行更新。
執行 make qa_branch QA_BRANCH=master QA_DOMAIN=master
即可建置出特定 branch 的 docker image.
可以看到我直接把 branch name 和 domain name 寫到 app 資料夾中,這樣就不用資料庫來紀錄什麼 domain name 對應什麼 branch 了。
這邊需要注意由於 domain name 用在三個地方:
- domain
- 路徑
- docker image name
比較麻煩的是 domain name 只能有 -
不能有 _
, 而 docker image name 則剛好相反,不能有 -
只能有 _
所以在輸入名稱的時候要特別注意。由於我外面是包 messaging bot 來下指令,那邊有做檢查,所以這邊就沒有另外再做檢查。
Rakefile
Rakefile 是拿來啟動 / 關閉 QA 環境用的。
require 'yaml' | |
require 'docker' | |
require 'pty' | |
require 'slack-notifier' | |
Docker.url = "tcp://127.0.0.1:4243" | |
QA_DIR = "/var/www/qa" | |
QA_URL = "qa.domain.internal" | |
QA_IMAGE_REGEX = /myapp_qa\/(\w+):latest/ | |
ROOT_PATH = File.dirname(__FILE__) | |
desc "List running containers" | |
task :running do | |
puts `docker ps` | |
end | |
desc "Clean stopped containers" | |
task :remove_stopped_containers do | |
puts "----- Cleaning stopped containers -----" | |
puts `#{ROOT_PATH}/bin/remove_stopped_containers.sh` | |
end | |
namespace :qa do | |
desc "List all running QA containers" | |
task :list do | |
running_qa_domains = qa_running_containers.map {|c| c.info["Image"].match(QA_IMAGE_REGEX)[1]} | |
puts "----- Current QA containers -----" | |
running_qa_domains.each do |domain| | |
puts "http://#{domain}.#{QA_URL} (#{`cat #{QA_DIR}/#{domain}/myapp/qa_branch`.strip})" | |
end | |
end | |
desc "Build and start a QA container" | |
task :start, [:branch_name, :domain_name] do |t, args| | |
branch_name = args[:branch_name] | |
domain_name = args[:domain_name] | |
hipchat_notify("Start to build QA container *myapp_qa/#{domain_name}* for :branch: *#{branch_name}*") | |
puts "----- Building QA container myapp_qa/#{domain_name} for branch #{branch_name}" | |
pipe_exec "cd #{ROOT_PATH} && make qa_branch QA_BRANCH=#{branch_name} QA_DOMAIN=#{domain_name}" | |
puts "----- Starting QA container myapp_qa/#{domain_name} for branch #{branch_name}" | |
puts `sudo rm -rf #{QA_DIR}/#{domain_name}` # Remove workspace if it existed | |
pipe_exec "docker run -d -i -t -v #{QA_DIR}/#{domain_name}:/opt/run:rw --privileged myapp_qa/#{domain_name}" | |
puts "----- Running and initializing fake data and indexes... -----" | |
status_file = "#{QA_DIR}/#{domain_name}/myapp/status" | |
status = "init" | |
loop do # Loop until ready. | |
if File.exists?(status_file) | |
current_status = File.read(status_file).strip | |
break if current_status == "ready" | |
if status != current_status | |
puts "#### Status changed from #{status} to #{current_status} ####" | |
status = current_status | |
end | |
end | |
sleep 5 | |
end | |
puts "=====> http://#{domain_name}.#{QA_URL} <=====" | |
hipchat_notify("QA environment for :branch: *#{branch_name}* is ready at http://#{domain_name}.#{QA_URL}") | |
end | |
desc "Stop QA container" | |
task :stop, [:domain_name] do |t, args| | |
domain_name = args[:domain_name] | |
container = nil | |
Docker::Container.all.each do |c| | |
container = c if c.info["Image"] == "myapp_qa/#{domain_name}:latest" | |
end | |
if container | |
puts "----- Stopping QA container #{domain_name} -----" | |
puts container.stop | |
puts "----- Deleting QA container #{domain_name} -----" | |
puts container.delete | |
puts "----- Deleting QA workspace #{domain_name} -----" | |
puts `sudo rm -rf #{QA_DIR}/#{domain_name}` | |
puts "----- Deleting QA image myapp_qa/#{domain_name} -----" | |
puts `docker rmi myapp_qa/#{domain_name}` | |
puts "----- Done. -----" | |
else | |
puts "There's no running QA container of domain #{domain_name}" | |
end | |
end | |
desc "Clean stopped container and remove unused workspaces." | |
task :clean do | |
Rake::Task["remove_stopped_containers"].invoke | |
running_qa_domains = qa_running_containers.map {|c| c.info["Image"].match(QA_IMAGE_REGEX)[1]} | |
puts "----- Deleting unused QA images -----" | |
qa_images.map do |image| | |
unless running_qa_domains.include?(image.info["RepoTags"].first.match(QA_IMAGE_REGEX)[1]) | |
puts "Image #{image.info["RepoTags"].first} removed." | |
image.remove | |
end | |
end | |
puts "----- Cleaning stopped QA workspaces -----" | |
Dir.glob("#{QA_DIR}/*").each do |f| | |
unless running_qa_domains.include?(File.basename(f)) | |
puts `sudo rm -rf #{f}` | |
end | |
end | |
end | |
end | |
######## Helpers ######## | |
# | |
def pipe_exec(command) | |
begin | |
PTY.spawn(command) do |stdin, stdout, pid| | |
begin | |
stdin.each { |line| print line } | |
rescue Errno::EIO | |
end | |
end | |
rescue PTY::ChildExited | |
puts "The child process exited!" | |
end | |
end | |
def qa_running_containers | |
Docker::Container.all.select {|c| c.info["Image"] =~ QA_IMAGE_REGEX } | |
end | |
def qa_images | |
Docker::Image.all.select { |c| c.info["RepoTags"].first =~ QA_IMAGE_REGEX } | |
end | |
def hipchat_notify(msg) | |
config = YAML.load_file('config/config.yml') | |
return if config['slack_token'] == nil || config['slack_token'] == "" | |
notifier = Slack::Notifier.new "kkbox", config['slack_token'], channel: '#myapp', username: 'Docker' | |
notifier.ping msg, icon_emoji: ":myapp:" | |
end |
這邊值得一提的是,由於 container build 好,到整個 service run 起來其實還有一段時間差,是用來啟動 service, initial db 等等動作…所以在 app 目錄下會寫一個 status
file 來判斷現在 app initial 到什麼階段,等他變成 ready
後才判斷為建置完成。
start_rails.sh
這是 QA docker image run 起來後會執行的 script, 比較重要的有兩個部分,一個是上面提的把目前階段寫入 status
檔案,讓外面知道現在進行到什麼步驟;另一個則是使用 socat 把 port 轉為 unix domain socket 再 mount 出去,就可以讓外面的 nginx 跟 docker 內部的 web services 溝通,而不需要處理 docker 的 port 了。
把 8080 port 轉到 /opt/run/myapp/go.sock
:
socat UNIX-LISTEN:/opt/run/myapp/go.sock,reuseaddr,fork TCP:localhost:8080
echo "dev:fake" > /opt/run/myapp/status | |
cd /opt/run/myapp && rvm all do bundle exec rake dev:build | |
cd /opt/run/myapp && rvm all do bundle exec rake dev:load | |
cd /opt/run/myapp && rvm all do unicorn_rails -c config/unicorn.rb -D | |
echo "building golang" > /opt/run/myapp/status | |
if [ -d "/opt/run/myapp/go" ]; then | |
#cd /opt/run/myapp/go && gom install # This takes some time. | |
cd /opt/run/myapp/go && ./make build | |
cd /opt/run/myapp/go && ./kktix_go start -d | |
socat UNIX-LISTEN:/opt/run/myapp/go.sock,reuseaddr,fork TCP:localhost:8080 & | |
fi | |
echo "sidekiq" > /opt/run/myapp/status | |
cd /opt/run/myapp && rvm all do bundle exec sidekiq -c 5 -d -L log/sidekiq.log & | |
cd /opt/run/myapp && rvm all do ruby -rsidekiq/api -e 'loop do;count = Sidekiq::Stats.new.enqueued;puts "Remaining jobs: #{count}";if count == 0;`echo "ready" > /opt/run/kktix/status`;break;end;sleep 5;end' | |
if [ -e "/opt/run/myapp/go.sock" ]; then | |
chmod 777 /opt/run/myapp/go.sock | |
fi | |
cd /opt/run/myapp && /bin/bash |
Conclusion
這篇的 scripts 有點繁雜,不過概念其實很簡單,只是細節上有不少需要注意的地方。
使用 docker 來建置 QA branch 可以方便快速的建出乾淨的環境,相較以往要處理非常多資料庫 / 轉 port / services isolation 等等問題,實在是輕鬆太多了。