Hao Nguyen
Caleb Smith recently gave a talk, “Intro to Celery,” at PyOhio (video below). Celery is a pretty popular topic for us here at Caktus. We use it often in our client work and find it very handy. So we were happy Caleb was out in the world, promoting its use. We sat down with him to hear more about PyOhio and Celery.
What did you enjoy about PyOhio?
PyOhio had good quality talks and a broad range of topics including system administration, web development, and scientific programming. This year, they had over 100 talk submissions and 38 spots, so there was a huge interest in speakers and a lot of variety as a result. They have four tracks and sprints every evening.
Also, PyOhio is free. The value of a free conference is that it lowers the barrier to attend to the costs of hotel, food, and travel. Things are pretty affordable in Columbus. So that’s good for students or people without an employer to help cover costs, like freelancers. People do come from a pretty big range of places across the Midwest and South.
They have a good team of volunteers that take care of everything.
Aside from a vegetable, what is Celery and why should developers use it?
Celery is for offloading background tasks so you can have work happening behind-the-scenes while running a web project. A typical web app does everything within requests and any periodic work with cronjobs. A lot of web projects will block a request on work that needs to be done before giving a response. For example, an image upload form might make the user wait while thumbnails are produced. Sometimes, there’s work that your web project needs to do that doesn’t fit within the upper limit of 30 seconds or so to fulfill a request before timing out the request. Celery allows for offloading this work outside of the web request. It also allows for the distribution of work as needed on multiple machines. You can trigger background tasks periodically for things like nightly backups, importing data, checking on updates to a feed or API, or whatever work that needs to run asynchronously in the background. We use this a ton with some of our client work.
What are Celery alternatives?
There are a few significant ones such as RQ, pyres, gearman and kuyruk. I think Celery is the most common choice among these. You can also just use system cron jobs for the periodic tasks, but cron jobs only work on one machine and are rarely well maintained. A task queue solution such as Celery coordinates with a broker to work on different machines.
What do you think are the challenges to getting started with Celery?
A lot of people think that it only works with Django. That was true when Celery was first released but is no longer true. There’s also somewhat of a barrier to entry because of the terminology involved, the work of setting up system resources such as the message broker, and understanding its role within a project.
You were a former public school music teacher and often teach Python in the community for organizations like Girl Develop It. Is there a relationship you see to giving talks?
Giving talks does feel like an extension of teaching. You learn a lot trying to prepare for it. My talk was about how to get everything set up, the basics of how Celery works, and developing a mental model for programming Celery tasks. A project like Celery can seem very difficult if you are approaching the documentation on your own. The high level overview is a little daunting so it’s nice to provide an on-ramp for people.
Our other blog posts contain more on Celery with Python.
