Strawberry’s test suite has a lot of tests, our test matrix has 45 jobs, and I’ve noticed that uploading coverage to Codecov would fail from time to time and it would also report wrong coverage numbers.

To fix the failures I’ve decided to finally update our CI to only send the coverage result once. It turned out to be much easier than I expected!

Our CI

I’m not going to too much into details about our CI, but we use GitHub Actions in combination with nox to run our tests.

We use nox to generate a list of tests to run using nox -l and then use GitHub Actions’ matrix feature to parellize the tests.

The first implementation of this worked quite well, the tests were fast and we’ve been able to test multiple versions of our dependencies, but occasionally the coverage upload would fail. And very often the coverage numbers would be wrong.

The first problem is due to us uploading the coverage for each job, instead of only uploading them once.

Artifacts

In order to upload the coverage when we finish all the tests, we need to create a new GitHub action that will run after all the tests are done. This is pretty straightforward, since we can tell a job to depend on another job, like this:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Run tests
        run: nox -s tests

  upload-coverage:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Upload coverage
        run: coverage upload

Unfortunately this is not enough, since the new job won’t have access to the generated coverages files. We need a way to pass the coverage files from the test job to the upload-coverage job. This can be done using the actions/upload-artifact action.

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Run tests
        run: nox -s tests

      - name: Upload coverage files
        uses: actions/upload-artifact@v4
        with:
          name: coverage
          path: .coverage

Then we need to download the coverage files in the upload-coverage job:

jobs:
  upload-coverage:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Download coverage files
        uses: actions/download-artifact@v4

Note that we haven’t specified a name in the “Download coverage files” step, without specifying a name, the action will download all the artifacts, which, in our case, is exactly what we want!

Finally we can run the upload coverage step:

jobs:
  upload-coverage:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Download coverage files
        uses: actions/download-artifact@v4

      - name: Upload coverage
        run: coverage upload

And that’s it! Coverage will find all the coverage files and upload them to Codecov.

I’m not yet sure if this also fixed the issue with the numbers, but so far it’s been working quite well 😊