Change in Worker Tier to avoid http server and allow typical Ruby/Python task queues

[januszm]

Worker Tier has the potential to be better used in applications based on Python or Ruby and Frameworks such as Django or Rails, which are very popular. However, the specificity of the current Worker Tier makes it rarely used/hard to use. This is due to the use of a very unusual approach based on converting SQS messages into http POST requests to the application.

Instead, I think most teams use dedicated Task Queue applications, such as for Ruby on Rails - Sidekiq or Shoryuken. Similarly, Python uses the PyQS or Django-Q.

I'd like to propose the following changes to make the Worker Tier more usable with popular frameworks:

1. remove the requirement for the HTTP server to run on Worker Tier, it is completely unnecessary
2. instead allow additional task queues to be launched in Procfile or extra Procfile.worker file

Example Procfile with the workaround my team is currently using (I think many people use the same approach):

# Procfile
web: bin/web
worker: bin/worker

Where:

# bin/web
#!/usr/bin/env ruby
require "fileutils"

# path to your application root.
APP_ROOT = File.expand_path("..", __dir__)

FileUtils.chdir APP_ROOT do
 if ENV["WORKER"].nil?
   system "bundle exec puma -C /opt/elasticbeanstalk/config/private/pumaconf.rb --pidfile /var/app/current/tmp/pids/server.pid"
 else
   system "bundle exec ruby worker_http.rb"
 end
end

and the "worker_http.rb"

require 'sinatra'

get '/' do
 'OK'
end

get '/health' do
 'OK'
end
`

Lastly, the new `bin/worker` script

```ruby
#!/usr/bin/env ruby
require "fileutils"

# path to your application root.
APP_ROOT = File.expand_path("..", __dir__)

if ENV["WORKER"].nil?
 exit 0
end

FileUtils.chdir APP_ROOT do
 system "bundle exec shoryuken -R -P tmp/pids/shoryuken.pid -L log/shoryuken.log"
end

An ENV variable WORKER is used to differentiate between the Web and Worker EB Tier.

In the example above, I use a trick that enables a minimalist http server Sinatra to satisfy the need for an http server. However, this http application Sinatra does nothing, it just responds to health checks.

This system should be changed so that, on the Worker tier, the Web server is not required at all. The whole thing should be controlled only by Procfile, or in combination with a new Procfile.worker file, so as to distinguish easily which programs should be run on which Tier and avoid writing custom shell scripts, e.g.

# Procfile:
web: bundle exec puma -C /opt/elasticbeanstalk/config/private/pumaconf.rb --pidfile /var/app/current/tmp/pids/server.pid
worker: bundle exec shoryuken -R -P tmp/pids/shoryuken.pid -L log/shoryuken.log

here, "web" should be ignored, ie. not executed and monitored at all on the Worker Tier.

OR with an additional file respected only by the Worker tier (where the regular Procfile is not interpreted at all):

# Procfile.worker:
worker: bundle exec shoryuken -R -P tmp/pids/shoryuken.pid -L log/shoryuken.log