PHP-Resque Auto Scale Workers
PHP-Resque is an amazing PHP port of Resque. After playing it for a while, an idea crossed my mind: It’s a total waste to create numbers of workers when there’s not many jobs to do. How about auto scale it? With the EventListener design of PHP-Resque, we could achieve it by writing some simple hooks.
Also, it solved issue #32 of PHP-Resque.
Here’s my code: PHP-Resque Auto Scale
This is a project trying to build an auto scale architecture of PHP-Resque.
afterEnqueueto check the total job number of this queue.
If the number larger than
15than check the total number of workers involved in this queue.
If the worker number is not enough, create one or more workers.
If there are more than one server, divided the number equally to each server.
In the mean time, try to create workers that deal the same queues on each server.
beforeForkto check the total job number and worker number, close the useless ones.
Number of Jobs and Workers
1~15 jobs => 1 worker
16~25 jobs => 2 workers
26~40 jobs => 3 workers
41~60 jobs => 4 workers
60+ jobs => 5 workers
- You need to add the queue type as a member static variable to your Job class. Like this:
1 2 3 4 5 6
plugin.phpboth in your resque init script and enqueue part.
Change the setting and code in
plugin.phpbased on your need.
Start only one worker via your resque init script.
Now it’s auto-scalable.
For now it’s all experimental design.
All numbers and codes are not from production enviroments nor runned benchmarks. It’s just a prototype for now, but it does what it says.