[Solved]-How to layout a queue/worker structure to support large tasks for multiple environments?

for 1,2 use multiple queues and launch workers with -Q to specify what queue to listen.
Also configure CELERYD_PREFETCH_MULTIPLIER = 1, for only one task at a time.

To get the queue lenght (tested with rabbitmq), you can use something like this:

from kombu.connection import BrokerConnection
connection = BrokerConnection(BROKER_HOST, BROKER_USER...)
channel = connection.channel()
q, j, c = channel.queue_declare('celery', passive=True)
print 'celery %d jobs in queue' % j

‘queue_delcare’ as a side effect, give you the queue’s length.
Hope this can help you.

I would take a look at zeromq it can do messaging and multi-threading in one super fast library. It also supports a large number of languages and has built in load balancing.