Over the past couple of weeks I have been working on one of my projects betterBuster to create a replacement for buster to make blogging a little easier, but also to give myself an excuse to use some cool new web services I just recently found out about:

waffle.io was extremely easy to setup and get running as it only took a Github automatic sign-in. travis-ci was also a simple service to setup and configure as the getting started page was easy to walkthrough and configure. It also helps that the community for travis-ci is quite large so it is not hard to find some help getting your repo setup. Now with coveralls, for some reason this took me a few late nights to finally get up and running...

To be honest, I had no idea what coveralls did or even what the Python test coverage tool (the tool used to generate the files uploaded to coveralls) did before I signed up and configured it into my project. I simply wanted to try it for experimentation to see if I wanted to use it for any future projects. After I got coveralls working, I then began to research what Python test coverage is: Python test coverage is a measurement to find how much of your codebase is covered by testing with the ultimate goal of making your project 100% covered by tests. It does this by running all of your tests (unit tests, integration tests, and doctests) and keeping track of what lines of the project source code is executed and not executed while running your tests.

Python programmers use a well-known tool called coverage.py to generate reports about what percentage of the project is covered by tests (unit, integration, doctests) and what parts of the project still need some work. After coverage creates these reports in a .coverage file in the root directory of your project, you upload the report to coveralls.io where you are then greeted with a great interface to view the reports.

coveralls.io interface

If your project uses Travis-CI, this process is extremely simple but if it is not, it only requires a couple extra steps to complete.


betterBuster includes doctests along with unit tests. This was also part of my experimentation as I have had little experience previously to Python testing techniques. I figured the language included a xUnit type of testing framework but no idea it also included nose, py.test, as well as doctests. Luckily, I used this wonderful tutorial to get up and running with doctests, nose, pytest and Python's build in unittest framework.

My issues

The reason that coveralls.io was more of a pain to setup and configure was not because coveralls is a terrible web service, not because coveralls and Travis-CI do not work well together (it is quite the opposite) but because generating the coverage report was quite a challenge to get right!

How to add coveralls.io to your Python project.

  1. Check out the coveralls.io official doc on Python setup. It directs you to two projects: python-coveralls which is what betterBuster currently uses and coveralls-python...what? What else would you call your project is someone already took the name python-coveralls?

    It does not matter much what project you decide to use. Try one, if you find it hard to follow, try the other.

  2. Follow the README guide to whatever Python coveralls project you decide to use. Each include instructions on how to enable the project if you use Travis-CI (recommended for simplicity) or a different CI system.

    The README files will help in installing the library through pip and how to upload the documents to coveralls.

Now that was the easy part as it only requires adding a couple lines to your .travis.yml file (if using Travis-CI)...

language: python
python:
  - "2.7"
install:
  - pip install python-coveralls  # Add me to install python-coveralls
script:
  - python setup.py install
after_success:
  - coveralls                     # Add me to upload reports to coveralls.io

...where I was hit with issues was with generating the coverage documents to upload to coveralls.

After many nights trying many different methods to get coveralls to work, I was hit with issues generating the .coverage reports locally, blank reports generated uploaded to coveralls, nose and py.test plugins not working correctly and mostly all because my project contains all the test files in a separate test package with 2+ different test_*.py files all using differnt testing frameworks. That was until I finally stumbled across this one document that solved it all. It turns out, nose is even more useful then you would expect and my solution involved configurating coverage.py to my project and then being executed by nose.

Add coverage.py and nose to your project by adding more files to your .travis.yml file:

language: python
python:
  - "2.7"
install:
  - pip install python-coveralls
  - pip install coverage       # Add me to install coverage.py
  - pip install nose           # Add me to install nose
script:
  - python setup.py install
after_success:
  - coveralls

Now, create a .coveragerc in your root directory of your project and add the following:

[report]
omit =
    */python?.?/*
    */site-packages/nose/*
    *__init__*

This will prevent your coverage report from containing test reports on the Python language, the nose package, along with your blank __init__.py files.

Lastly, add one more line to your .travis.yml file to generate the reports with nose:

language: python
python:
  - "2.7"
install:
  - pip install python-coveralls
  - pip install coverage
  - pip install nose
script:
  - python setup.py install
  - nosetests --with-coverage # Add me to run nose with coverage support.
after_success:
  - coveralls

And...that is it. Yup. Nose takes care of the rest of the work for you. Nose will run all of your unit, integration and doctests automatically as long as your test files begin with test_ for example: test_buster.py. Also, make sure all your tests pass!


After finally getting coverage to work, I probably could have solved the issue much earlier if I knew more about nose but I am still a beginner when it comes to Python testing. There may be a easier implementation then this to get coveralls to work with my Python project, but this seems to work with the least amount of maintenance and setup time.

If you know of a better method for adding coveralls.io support to a Python project, leave a comment, contact me, or try your implementation out on betterBuster and make a pull request!