I've been helping out with the Great Git Migration effort lately, particularly the queue infrastructure. As a result of this work, I created the Waiting Queue module. A design goal of the module is to make it easy to run a queue with workers that wait indefinitely for jobs. A worker that waits indefinitely will pull jobs off the queue as soon as they are available, which keeps the time a job waits in the queue very small. The Drupal.org queue infrastructure will be based around Beanstalkd, a simple, highly efficient queue implementation. This queue system works well with workers that wait indefinitely for jobs. This sort of queue processing is not compatible with the design of the core Drupal Queue module, because a single queue with workers that wait indefinitely will block all other queues called via a site's cron run. This is why we felt the need to write a new module. The Drupal.org infrastructure will use Supervisord to manage our long running processes, making it easy to guarantee that there will always be a worker ready to process jobs as soon as they are added to the queue. The Waiting Queue module includes a sample Supervisord configuration file.
January 18, 2011