Dmitriy Chukhin
Subtests are the best
Testing our code is important. Because developers write bugs, it’s valuable to catch and correct them before the code gets to production so our apps work as they should. Specifically, we want tests that are DRY (Don’t Repeat Yourself), thorough, and readable. Though there are many ways to try to accomplish these goals, subtests make each of them easier. If you’re not using subtests in your test classes, you probably should be.
Subtests were introduced in Python 3.4, and a few uses are briefly covered in the Python documentation. However, there are many more uses that have made my testing better. Their value is probably best seen through an example.
DRYer code and using parameters for cleaner errors
Let’s say we have a function is_user_error() that takes a status code and returns True if it means a user error (note that any 400-level status code is considered user error) has occurred. We could test this function with one test that has many assertions:
import unittest
from ourapp.functions import is_user_error
class IsUserErrorTestCase(unittest.TestCase):
def test_yes(self):
"""User errors return True."""
self.assertTrue(is_user_error(400))
self.assertTrue(is_user_error(401))
self.assertTrue(is_user_error(402))
self.assertTrue(is_user_error(403))
self.assertTrue(is_user_error(404))
self.assertTrue(is_user_error(405))
…
But that’s many lines of code, so to be DRYer, we could test the same functionality by writing a for loop:
import unittest
from ourapp.functions import is_user_error
class IsUserErrorTestCase(unittest.TestCase):
def test_yes(self):
"""User errors return True."""
for status_code in range(400, 499):
self.assertTrue(is_user_error(status_code))
That’s a lot DRYer, and allows us to test many more status codes, but if one of those status codes failed, we would get something like:
======================================================================
FAIL: test_yes (IsUserErrorTestCase)
User errors return True.
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_example.py", line 10, in test_yes
self.assertTrue(is_user_error(status_code))
AssertionError: False is not true
----------------------------------------------------------------------
Ran 1 test in 0.000s
FAILED (failures=1)
So we’re left wondering, which status code failed? Instead, we can use subtests with parameters:
import unittest
from ourapp.functions import is_user_error
class IsUserErrorTestCase(unittest.TestCase):
def test_yes(self):
"""User errors return True."""
for status_code in range(400, 499):
with self.subTest(status_code=status_code):
self.assertTrue(is_user_error(status_code))
Our failure becomes:
======================================================================
FAIL: test_yes (IsUserErrorTestCase) (status_code=405)
User errors return True.
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_example.py", line 10, in test_yes
self.assertTrue(is_user_error(status_code))
AssertionError: False is not true
----------------------------------------------------------------------
Ran 1 test in 0.002s
FAILED (failures=1)
Which lets us know in the first line that the status_code 405 was the one that failed.
As you can see, subtests give us DRYer code than a single test, and clearer error messages than multiple tests.
DRYer code and using a message (msg) for cleaner errors
Another example: Let's say we're working on a Django project where we created a new API endpoint (/api/people/), and we want to test the fields that are required for it. For the sake of this example, let's say that the endpoint looks like this:
{
"id": "",
"first_name": "",
"last_name": "",
"address": "",
"birthdate": "",
"favorite_color": "",
"favorite_number": "",
"timezone_name": ""
}
and the required fields for POSTing are: first_name, last_name, and address. One way to test which fields are required would be to write a single test:
from django.test import TestCase
class PeopleEndpointTestCase(TestCase):
def setUp(self):
super().setUp()
# Do the rest of the setup for logging in the user and giving the user
# the required permissions
def test_people_endpoint_post_data(self):
"""Test POSting to the people endpoint with valid and invalid data."""
url = '/api/people/'
minimum_required_data = {
"first_name": "Joe",
"last_name": "Shmo",
"address": "123 Fake Street",
}
# POSTing with all of the required data
response = self.client.post(
url,
data=minimum_required_data,
content_type='application/json')
self.assertEqual(response.status_code, 201)
# POSTing with the first_name missing
data = minimum_required_data.copy()
data.pop('first_name')
response = self.client.post(
url,
data=data,
content_type='application/json')
self.assertEqual(response.status_code, 400)
# POSTing with the last_name missing
data = minimum_required_data.copy()
data.pop('last_name')
response = self.client.post(
url,
data=data,
content_type='application/json')
self.assertEqual(response.status_code, 400)
# POSTing with the address missing
data = minimum_required_data.copy()
data.pop('address')
response = self.client.post(
url,
data=data,
content_type='application/json')
self.assertEqual(response.status_code, 400)
But that's not very DRY, since each section does the same thing. If we split each section into its own test it might be easier to read, but it would be even less DRY. Instead, we could write a loop for each of the sections like this:
from django.test import TestCase
class PeopleEndpointTestCase(TestCase):
def setUp(self):
super().setUp()
# Do the rest of the setup for logging in the user and giving the user
# the required permissions
def test_people_endpoint_post_data(self):
"""Test POSTing to the /people/ endpoint with valid and invalid data."""
url = '/api/people/'
minimum_required_data = {
"first_name": "Joe",
"last_name": "Shmo",
"address": "123 Fake Street",
}
with self.subTest('POSTing with all of the required data'):
response = self.client.post(
url,
data=minimum_required_data,
content_type='application/json')
self.assertEqual(response.status_code, 201)
missing_subtests = (
# A tuple of (field_name, subtest_description)
('first_name', 'Missing the first_name field'),
('last_name', 'Missing the last_name field'),
('address', 'Missing the address field'),
)
for field_name, subtest_description in missing_subtests:
with self.subTest(subtest_description):
# Remove the missing field from the minimum_required_data
data = minimum_required_data.copy()
data.pop(field_name)
# POST with the missing field name
response = self.client.post(
url,
data=data,
content_type='application/json')
self.assertEqual(response.status_code, 400)
Now the test uses up fewer lines, and any failures are logged into the console based on the subtest_description.
Independent tests
Another thing to be aware of is that subtests run independently, so we get all of the subtest failures printed into the console, rather than just the first one.
For example, let’s say we’re working on a Django project that has a User model in the our_cool_app app who sometimes creates Blogposts, and we want to track the most recent time that this user has posted, so we create a method called get_latest_activity(). The Blogpost model has a created_datetime field that tracks when it was created, and a creator field that tracks the user that created it.
# models.py
from django.db import models
from django.utils import timezone
from our_cool_app.models import User
class Blogpost(models.Model):
# ...other fields here...
created_datetime = models.DateTimeField(default=timezone.now)
creator = ForeignKey(User, on_delete=models.CASCADE)
We should test that this method works correctly, so we can write the test a few different ways. As one test:
import datetime
from django.test import TestCase
from django.utils import timezone
class UserTestCase(TestCase):
def test_get_latest_activity(self):
user = UserFactory()
# A user with no blogposts has no latest activity
self.assertIsNone(user.get_latest_activity())
# A user with 1 blogpost has that blogpost's created_time as the latest activity
blogpost = BlogpostFactory(creator=user)
self.assertEqual(user.get_latest_activity(), blogpost.created)
# A user with multiple blogposts
a_day_ago = timezone.now() - datetime.timedelta(days=1)
a_week_ago = timezone.now() - datetime.timedelta(days=7)
BlogpostFactory(creator=user, created_datetime=a_day_ago)
BlogpostFactory(creator=user, created_datetime=a_week_ago)
self.assertEqual(user.get_latest_activity(), blogpost.created)
# Future blogposts don't count
tomorrow = timezone.now() + datetime.timedelta(days=1)
BlogpostFactory(creator=user, created_datetime=tomorrow)
self.assertEqual(user.get_latest_activity(), blogpost.created)
# Other people's blogposts don't count
another_user = UserFactory()
BlogpostFactory(creator=another_user)
self.assertEqual(user.get_latest_activity(), blogpost.created)
This looks ok as long as we are careful to make clear comments and use blank lines for better readability. However, if the first assertion (# A user with no blogposts has no latest activity) fails, then the rest of the test doesn’t run. Instead, we can use subtests to write:
import datetime
from django.test import TestCase
from django.utils import timezone
class UserTestCase(TestCase):
def test_get_latest_activity(self):
"""Test the get_latest_activity() method."""
user = UserFactory()
with self.subTest("A user with no blogposts has no latest activity"):
self.assertIsNone(user.get_latest_activity())
with self.subTest("A user with one blogpost"):
blogpost = BlogpostFactory(creator=user)
self.assertEqual(user.get_latest_activity(), blogpost.created)
with self.subTest("A user with multiple blogposts"):
a_day_ago = timezone.now() - datetime.timedelta(days=1)
a_week_ago = timezone.now() - datetime.timedelta(days=7)
BlogpostFactory(creator=user, created_datetime=a_day_ago)
BlogpostFactory(creator=user, created_datetime=a_week_ago)
self.assertEqual(user.get_latest_activity(), blogpost.created)
with self.subTest("Future blogposts don't count"):
tomorrow = timezone.now() + datetime.timedelta(days=1)
BlogpostFactory(creator=user, created_datetime=tomorrow)
self.assertEqual(user.get_latest_activity(), blogpost.created)
with self.subTest("Other people's blogposts don't count"):
another_user = UserFactory()
BlogpostFactory(creator=another_user)
self.assertEqual(user.get_latest_activity(), blogpost.created)
As a result, our code looks more readable, is DRY, and provides useful error messages for each section that fails, rather than just the first one.
A Caveat
While subtests do run independently of each other they don’t run in database transactions, so any changes to the database that are made within one subtest will persist in any subsequent subtests until the end of the test. From our previous example, the user’s first blogpost was created in the “A user with one blogpost” subtest, and it persists until the end of the test (which is why the user’s last activity in the “A user with multiple blogposts” subtest is this first blogpost’s created_datetime).
Conclusion
Subtests allow us to write short DRY sections of code and to have meaningful error messages for when our code fails. There are many ways to use subtests, and more things you can do with them than what is outlined in this blog post, so go and use them to write tests that are thorough, DRY, and readable.
For further reading about newer features of Python, check out New year, New Python: 3.6.
