Automating PyPI releases with Github Actions

At Caktus, we maintain some open source packages, which are usually pieces of code that we've found useful when building an application, and we figure they might be useful to others. From time to time, those open source projects need a little maintenance: Maybe a bug fix has been provided from a community member, or we need to update to support a new version of Python or Django. I always hesitate to do this because I know there are multiple steps involved to ensure that any updates are properly tested, documented, versioned, and released. One of my silly blockers is that I always seem to forget how to upload a release to PyPI, which is the Python Package Index, a repository of software for Python. I call it silly because it shouldn’t really be a blocker, seeing how simple it actually is, but I still forget it each time and have to walk through the documentation step by step to make sure I do it properly. I recently found a tool that helps with this process.

During our December ShipIt (learn more about our ShipIt events), I got acquainted with Github Actions, a service that Github provides to automatically perform standard CI/CD tasks on the code in your repo. Github Actions is free for public repositories, and they provide a limited amount of minutes for private repositories (starting at 2,000 minutes per month on the free plan). Their introductory documentation includes an example demonstrating how to push to PyPI anytime you create a Github Release.

Start by adding this code to .github/workflows/release.yml in your repository:

name: Upload Python Package

on:
  release:
    types: [created]

jobs:
  deploy:
    runs-on: ubuntu-20.04

    steps:
    - uses: actions/checkout@v2
    - uses: actions/setup-python@v2
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install setuptools wheel twine
    - name: Build and publish
      env:
        TWINE_USERNAME: __token__
        TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
      run: |
        python setup.py sdist bdist_wheel
        twine upload dist/*

Github will try to run all of the jobs each time any of the events in the on clause occur. You could have multiple jobs run, and even have strict dependencies between the jobs so that certain jobs only run if other jobs pass. This allows you to set up workflows, for example, where the release job won’t run unless an earlier test job passes. The recipe above is very simple, though, and states that anytime a Github Release is created, the steps in the release job are run. This includes checking out the code, setting up Python, installing Python dependencies, running setup.py to build our distribution, and then finally using the twine tool to upload our distribution to PyPI using a secret TWINE_PASSWORD, which we’ll create next.

Follow these instructions to create a PyPI API token for your repo. Make sure to limit the scope of the token to only be allowed to upload packages for your repo. Copy the password, because you won't be able to see it again, and then paste it into Github secrets for your repo with the name PYPI_PASSWORD. We don't need to keep this password in shared storage anywhere because if Github ever "lost" it or if we felt it was compromised, we'd just deactivate the old one and create a new one.

So, now the workflow for making an update to an open source project is to

1. Make the normal updates to the code, including updating the version number in setup.py. (If you don’t update the version number, then twine will return a 400 error because PyPI will properly not let you overwrite an existing package with the same version.)
2. Create a new Release in the Github UI.

Step 2 kicks off a Github Action that will build your package and upload it to PyPI, thus eliminating my silly blocker of having to remember how to do this manually, and I can update our open source packages more efficiently!

Additional notes:

• PyPI has also published excellent instructions on creating a PyPI release using their custom Github Action which I found after finding the instructions that I followed.

• You could automate this process even more by making the PyPI upload happen anytime a git tag is created, or a push to a certain branch happens. I currently prefer the single manual step of creating a Github Release to make me feel like I have more control over the process.