yotaq - Your Own Task Queue for Python
So you need a task queue for your Python project. Sure you could check celery, and after three months trying to understand the basic configuration options you'll be good to go. Or you could use a simpler task queue like huey or rq.
Why don't you try building your own task queue? Well, now you can!
First, we'll use redis as our message broker. There's no need to install redis, we'll use docker so we keep our environment clean. Open a terminal and run:
docker run -p 6379:6379 redis
There you go. Now let's create a Python virtual environment to handle our dependencies, which are the redis python library and dill:
virtualenv env
source env/bin/activate
pip install redis dill
Pretty good. Our python code will use dill to serialize the functions to be run and redis to store the tasks.
The client
The client will issue the tasks to be enqueued, so open up an editor, create a file called client.py. There, we'll define the task that will be sent to the workers, for example:
import random
import time
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)