Auto-scaling Online Survey

The University of Chicago’s Consortium on Chicago School Research has surveyed the city’s schools since 1990 to track the impact of landmark education reforms. As one of the only comprehensive longitudinal studies being conducted, CCSR survey results are highly influential nationally. Any changes to survey administration had to be carefully managed.

The sheer number of CCSR survey takers—thousands upon thousands at the same moment—would crash existing systems.

The paper-based Scantron bubble sheets CCSR used to administer surveys posed significant resource challenges. In addition to cost, there was staff time to manually distribute, collect, and aggregate survey results across 650 schools. Online surveys had become mainstream, but the sheer number of CCSR survey takers—thousands upon thousands at the same moment—would crash existing systems. The risk was unacceptable. In addition, after decades of survey administration, CCSR had developed specific workflows, function requirements, and security protocols that a survey application must have.

Through a process of agile development, Caktus worked closely with CCSR to address the main challenges to survey administration. The challenges, scaling reliability and survey workflow, were interdependent. A great survey creation application would be useless if the system crashed on the first launch and, vice versa, a system that could scale well would not succeed if the interface were cumbersome. Our team tackled both challenges.

To address the need to scale and prevent the system from being overloaded, our team needed to ensure processing speed was lightning fast and that there was automatic way to bring servers on-board as needed. Using JMeter for extensive, iterative load testing we were able to discover inefficiencies as they came up and verify solutions. We could then optimize our code quickly, minimizing database queries using a common cache. We also fine-tuned the PostgreSQL configuration. Ultimately, the system we built handles upwards of 1,250 HTTP requests per second and 9,500 PostgreSQL transactions per second.

For the server architecture, we created scripts to dynamically scale servers so that it was not necessary to have all available servers always running--the infrastructure could expand or contract as needed. We leveraged Amazon Auto Scaling and wrote additional scripts to simplify the creation of server images during new deployments, manage data security, and create encrypted partitions. We separated all the different server types (database master, database slave, background worker, web, and cache) onto separate instances.

For survey creators and administrators, the survey application we created simplified workflows. We developed a dashboard and survey administration tools that made it intuitive for staff to securely track changes, view live results, sort data, and create new surveys with complex features like skip logic. CCSR ultimately had simplified workflows while also gaining far greater flexibility.

For more technical details, watch Caktus CEO Tobias McNulty’s DjangoCon 2013 presentation: Scaling Your Write-Heavy Django App: A Case Study

Where survey administration and data collection could take weeks and hundreds of pounds of Scranton bubble sheets, researchers now can access scored surveys instantly.

The migration from paper to online surveys so exceeded expectations, CCSR rolled out the tool to the rest of Illinois and now uses it for multiple types of surveys. Their research partner at the UChicago’s Urban Education Institute now uses the survey tool for research in Michigan, Montana, Minnesota, Massachusetts, Maryland, Oregon, DC, Connecticut, Colorado, Louisiana, Iowa, Missouri, and Florida. There are plans to increase the use of the tool in other locations.

Due to the success of the initial survey project, CCSR continues to engage Caktus in the development of new features. We’re pleased to enhance their reputation as one of the most sophisticated survey research bodies in the country.