While sometimes tedious, testing is critical if you’re at all interested in developing a robust and reasonably bug-free codebase that endures.

In software testing, these are your options:

Unit testing
End-to-end testing(e2e)
Integration testing

Unit testing involves testing snippets of code, single functions, classes, utility functions, components, etc. Here, each function, component, class, etc. is tested independently and as a single unit. This is the most basic type of testing.

In Python, we can test that a function works as expected.

>>> assert sum([5, 3, 6]) == 14, "Should be 14"

The above is an example of a unit test. The sum function is used to sum all the numbers in the list and returns the result. So, we are testing that the function works and returns the expected result. In the example above, we are testing that the sum of the numbers in the list is 14. If the return is not 14 then the test fails and we know that something is wrong with the sum function.

Integration testing involves testing the software modules, classes, and components of the software as a group. This integration testing is conducted after unit testing.

End-to-end testing is the most complex type of testing and tests the software as a group. This testing makes sure that the software is working as expected and runs from start to finish.

Now, we can either run our tests manually or we can automate them. Manual testing is when we run our software to check the features and experiment using them. This is called exploratory testing and it’s a form of manual testing.

Eventually, manual testing will become too boring for words and you’ll want to automate things.

Fortunately, there is software that can run our tests for us and return a pass/fail. This is called automation testing.

In Python, several testing frameworks are available to help us run our tests during development. These frameworks are very sophisticated, complex, and powerful. They give us the best experience in testing and make our work easier.

We will talk about the different types of testing frameworks in Python below.

Robot

Robot Framework is a generic open source automation framework. It can be used for test automation and robotic process automation (RPA).

Robot is an open-source automation test framework that supports many languages apart from Python. It has an easy syntax, is easy to learn, and has a rich set of features that can be used to test any kind of software.

Robot has also a rich set of plugins supported by developers around the world. These libraries can be used to test any kind of software. For example, we have the SeleniumLibrary which can be used to test web applications, also there is RESTinstance that can be used to test HTTP JSON APIs.

To start using the Robot Framework, we need to install it.

>> pip install robotframework

We can verify that we have installed it by running the following command.

>> robot --version
Robot Framework 5.0 (Python 3.8.10 on linux)

PyTest

The pytest framework makes it easy to write small, readable tests and can scale to support complex functional testing for applications and libraries.

pytest is a mature Python testing framework that is also easy to learn and use. It is a good choice for writing small tests and for writing tests that are complex and require a lot of code. pytest supports unit testing, functional testing, and API testing.

Currently, pytest is the most popular testing framework in the Python world. It runs on Python 3.7+ or PyPy3.

The advantages of choosing pytest as your testing framework for your next Python project are that it is easy to use, offers simple test suites, has many plugins you can use for different work, and has large community support.

pytest installation is quite simple.

>> pip install pytest

pytest will now be available in our environment.

Now, let’s write a test.

import pytest
def test_1():
    assert 6 + 1 == 7, "test passed"
def test_2():
    assert 6 + 1 !== 8, "test passed"

See how easy and simple it is to write a test in pytest? We just need to import the pytest library and write a test. The tests are methods. The first method, test_1 tests that 6 + 1 is 7. If the test passes, it will print test passed. The second method, test_2 tests that 6 + 1 is not 8. If the test passes, it will print test passed.

# test.py
def test_sum_14():
    assert sum([5, 3, 6]) == 14, "Should be 14"

def test_sum_10():
    assert sum([5, 5]) == 10, "Should be 10"

def test_sum_200():
    assert sum([10, 10, 10, 170]) == 200, "Should be 200"

You see how pytest allows us to write neat and well-structured tests. The first test from the look of it we can assert that it is testing that the sum function returns 14. The second test is testing that the sum function returns 10. The third test is testing that the sum function returns 200.

To run the test.py file, we will simply run the following command.

>> pytest test.py

We can just run the pytest standalone at the root of our project and it will run all the tests.

Unittest

unittest just as the name implies is a unit testing framework for the Python programming language that’s a fork of JUnit in Java. It has a very simple syntax and is easy to learn.

According to unittest docs, unittest supports test automation, sharing of setup and shutdown code for tests, aggregation of tests into collections, and independence of the tests from the reporting framework.

unittest is by default the testing framework that is used in Python because it is baked into the Python distribution. So once you install Python, you will have unittest installed.

Let’s see how we can write and run a simple test case in unittest.

# test.py
import unittest

class TestSum(unittest.TestCase):
    def test_sum_14(self):
        self.assertEqual(sum([5, 3, 6]), 14, "Should be 14")
   def test_sum_200(self):
        self.assertEqual(sum([100, 100])), 100, "Should be 200")
if __name__ == ' __main__':
    unittest.main()

We start by importing unittest package. Next, we define a class TestSum. This class is used to define the test cases. The class name is TestSum and it should inherit from the unittest.TestCase class.

The methods inside the class are called test cases. The first test case is test_sum_14. This test case asserts that the sum of the numbers in the list is 14. The second test case is test_sum_10. This test case asserts that the sum of the numbers in the list is 10. The third test case is test_sum_200. This test case asserts that the sum of the numbers in the list is 200.

Finally, we run the tests by running the unittest standalone at the root of our project.

>> python test.py

By using the python executable, unlike other frameworks where we have to use the framework name. This is because unittest is baked inside python, so Python will understand that the test.py file is a test file and will run unittest on it.

DocTest

DocTest is another testing framework for Python. Just like unittest, it supports test automation, sharing of setup and shutdown code for tests, aggregation of tests into collections, and independence of the tests from the reporting framework. Also, it is integrated into the Python distribution. So we don’t need to install it.

DocTest uses the white-box testing approach. It is a testing framework that is used to test the Python documentation. It is a good choice for writing tests for the Python documentation. According to the doctests docs, it works by searching for pieces of text that look like interactive Python sessions and then execute those sessions to verify that they work exactly as shown.

Let’s see an example:

# test.py
"""
>>> test(5)
500
"""

def test(n):
    """Return the multiplication of 2 and 100
    >>> test(2)
    200
    """
    if n == 0:
        raise ValueError("n must be >= 0")
    if n < 0:
        raise ValueError("n must be >= 0")
    else:
        return n * 100

if __name__ == " __main__":
    import doctest
    doctest.testmod()

Now, in the above code, we have a method, test. This method returns the multiplication of the number n by 100. If the number is 0, it will raise a ValueError. If the number is negative, it will raise a ValueError.

See that we have the documentation in the test method and also the comments inside the test method. The first comment outside the method tests that the test method returns 500 when passed 5. Also, the comment inside the test method tests that the test method returns 200 when passed 2. Doctests will parse the comments and run the tests in the comments.

Now, let’s run the test.py file.

>> python test.py
>>

There is no output. Let’s run it again with a -v flag.

>> python test.py -v

Now, we will see the output.

Trying:
  test(5)
Expecting:
  500
ok
Trying:
  test(2)
Expecting:
  200
ok

Neat!

Doctest is magical

Nose2

Nose2 is a unit testing framework that can also run unittests and doctests. It is an extension of unittests.

Let’s test our previous unittest test cases with nose2.

# test.py
import unittest

class TestSum(unittest.TestCase):
    def test_sum_14(self):
        self.assertEqual(sum([5, 3, 6]), 14, "Should be 14")

Now, we run it:

>> node2 test.py -v
test_sum_14 (test.TestSum) ... ok
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK

Nose2 can actually run all tests in a Project. For more documentation, see nose2 docs.

Testify

Testify is an open-source testing framework for Python built by Yelp.com. It is intended to replace Python’s built-in unittests and Nose.

Testify has so many nice features ahead of unittests and Nodes. Testify outputs results in a colorful way, it is easy to read, it is easy to write, and it is easy to use.

Testify is extensible, meaning that we can write plugins to add new features to Testify. It can also parse through folders much deeper than Nose to find and run tests. It has this class-level test case setup similar to unittests.

Let’s see a test case in Testify:

from testify import *
class SumTestCase(TestCase):
    def test(self):
        assert_equal(sum([2, 0]), 2)
if __name__ == " __main__":
    run()

In the above code, first, we import the testify package. Then, we define a class SumTestCase. This class is used to define the test cases. The class name is SumTestCase and it should inherit from the TestCase class.

Just like unittests, Testify has fixtures that we can use to add more functionality to our tests. For example, we can use the setup fixture to set up the environment before each test.

teardown fixture is used to clean up the environment after each test.
setup_class fixture is used to set up the environment before all the tests in the class.
teardown_class fixture is used to clean up the environment after all the tests in the class.
setup_method fixture is used to set up the environment before each test in the method.
teardown_method fixture is used to clean up the environment after each test in the method.
setup_package fixture is used to set up the environment before all the tests in the package.
teardown_package fixture is used to clean up the environment after all the tests in the package.

TestProject

TestProject is an automation testing framework. It is quite powerful and has lots of rich features. TestProject is also extensible, developers can write their addons to TestProject. Also, there is a marketplace for TestProject addons https://addons.testproject.io/.

TestProject can export the results of the tests to a variety of formats. It can export the results to a JSON file, a CSV file, an HTML file, an XML file, and a YAML file. It also offers a free test report in HTML/PDF format. There is also access to the execution history of the tests, a built-in test runner, and huge developer and community support.

TestProject installation

We can install the TestProject package using the following command:

pip3 install testproject-python-sdk

Now, we can write our tests:

from src.testproject.sdk.drivers import webdriver
def simple_test():
    driver = webdriver.Chrome()
    driver.get("http://test")
    passed = driver.find_element_by_css_selector("#test").is_displayed()
    print("Test passed") if passed else print("Test failed")
    driver.quit()

if __name__ == " __main__":
    simple_test()

The above is a simple test in TestProject. It will open a browser and visit the http://test page. Then, it will check if the element with the id test is displayed. If it is, it will print Test passed. If it is not, it will print Test failed.

PyUnit

PyUnit is a unit testing framework for Python. It is a simple, easy-to-use, and well-supported framework. Just like unittests, PyUnit comes as standard with the Python package, so it doesn’t need to be installed.

Let’s see a simple use case of PyUnit:

from unittest import TestCase
class TestSum(unitest.TestCase):
    def test_sum_14(self):
        """test case sum 14"""
        assert sum([5, 3, 6]) == 14, "Should be 14"

if __name__ == ' __main__':
    unittest.main()

It is a simple test case. It will check if the sum of 5, 3, and 6 is 14. If it is, it will print Test passed. If it is not, it will print Test failed. See that the structure is very similar to unittests.

We have fixtures in PyUnit. We can use the setUp and tearDown methods to set up and clean up the environment before and after each test. We can use the setUpClass and tearDownClass methods to set up and clean up the environment before and after all the tests in the class. We can use the setUpMethod and tearDownMethod methods to set up and clean up the environment before and after each test in the method. We can use the setUpPackage and tearDownPackage methods to set up and clean up the environment before and after all the tests in the package.

Aviator: Automate your cumbersome merge processes

Aviator automates tedious developer workflows by managing git Pull Requests (PRs) and continuous integration test (CI) runs to help your team avoid broken builds, streamline cumbersome merge processes, manage cross-PR dependencies, handle flaky tests while maintaining their security compliance.

There are 4 key components to Aviator:

MergeQueue – an automated queue that manages the merging workflow for your GitHub repository to help protect important branches from broken builds. The Aviator bot uses GitHub Labels to identify Pull Requests (PRs) that are ready to be merged, validates CI checks, processes semantic conflicts, and merges the PRs automatically.
ChangeSets – workflows to synchronize validating and merging multiple PRs within the same repository or multiple repositories. Useful when your team often sees groups of related PRs that need to merged together, or otherwise treated as a single broader unit of change.
FlakyBot – a tool to automatically detect, take action on and process results from flaky tests in your CI infrastructure.
Stacked PRs CLI – a command line tool that helps developers manage cross-PR dependencies. This tool also automates syncing and merging of stacked PRs. Useful when your team wants to promote a culture of smaller, incremental PRs instead of large changes, or when your workflows involve keeping multiple, dependent PRs in sync.

Try it for free.