We've been using RapidSMS, a Django-powered SMS framework, more and more frequently here at Caktus. It's evolved a lot over the past year-- from being reworked to feel more like a Django app, to merging the rapidsms-core-dev and rapidsms-contrib-apps-dev repositories into a single codebase (no more submodules!), to finally becoming installable via pypi. The "new core" is in a great state now and is much easier to work with. However, one particular aspect of RapidSMS, the route process, has always been complicated and confusing to deal with. Tobias began the conversation on this issue after returning from a 6-week long UNICEF project in Zambia. He summarized the route process like so:
- The route process as it currently stands is complicated; it includes a number of threads and the ways in which they interact is not always intuitive
- If the route process dies unexpectedly, all backends (and hence message processing) are brought offline
- Automated testing is difficult and inefficient, because the router (and all its threads) needs to be started/stopped for each test
The RapidSMS router is a globally instantiated object that routes incoming messages through each RapidSMS app and sends outgoing messages via installed backends. The run_router management command starts the router process and creates individual threads for each backend defined in the settings module. I'm not entirely certain as to why the route process was originally threaded, but I assume it was designed to more easily integrate blocking backends (like gsm) into RapidSMS. However, with the standardization of Kannel and SMS-based web services, like Twilio, both of which offload the low level communication work, I believe the threading aspect is now less important. So recently, in what started as a proof of concept, we began work on a decoupled router implementation called rapidsms-threadless-router. rapidsms-threadless-router provides a threadless_router app, which removes the threading functionality from the legacy Router class. Rather, all inbound requests are handled via the main HTTP thread. threadless_router attempts to:
- Make RapidSMS backends more Django-like. Use Django’s URL routing and views to handle inbound HTTP requests
- Remove clutter and complexity of route process and threaded backends
- Ease testing – no more threading or Queue modules slowing down tests
In comparison to the legacy route process, threadless_router handles all inbound and outbound backend communication from within the main HTTP thread. Each request creates a new router instance and no separate process or thread is created. This simplifies the Router class significantly. Additionally, threadless_router allows inbound messages to be easily passed off to an asynchronous task queue, such as Celery. Task queues allow message processing to be handled outside of the HTTP request/response cycle, which is perfect for SMS-based applications, as out of band responses are more than acceptable.
threadless_router is not, however, a drop-in replacement for the legacy router. Legacy backends will not work and as all routing is handled from within the HTTP thread, non-HTTP backends, such as pygsm, are not currently compatible with threadless_router. A simple wrapper around pygsm could be written to talk both to the modem and spin up a simple HTTP server to communicate with RapidSMS. This would decouple pygsm from RapidSMS and exist as it's own separate process. Integrating with supervisord would work great here too. Several contrib applications, such as httptester and scheduler, are also not compatible. We've bundled a new httptester as a replacement and celerybeat can be used to mimic the scheduler functionality. A full list of caveats can be found in the docs.
The full documentation for rapidsms-threadless-router, including installation instructions and examples, can be found on readthedocs.org. If you're already familiar with the internals of RapidSMS and would like to see examples of threadless backend implementations, I suggest reviewing the bundled http and httptester backends and our updated twilio backend.
I would like to mention that Nicolas Pottier and Eric Newcomer created rapidsms-httprouter, which also handles all messages within the main HTTP thread. The main difference between rapidsms-httprouter and rapidsms-threadless-router is that, while httprouter handles inbound messages in a Django view, it still starts up threads (for handling outgoing messages) like the current router (also from within a Django view). Make sure to check it out as well and let us know what you think!
