Software Testing

For this week in my Open Source class, we looked into managing project complexity through Automated testing.

I've been taught that testing helps individual developers and communities of developers keep software development on the right track. Without tests, it becomes easy to break existing code, add unexpected bugs, or miss important edge cases. Manually testing software is feasible at first, but it becomes less practical as software becomes more complex.

Lab Exercise - Add Unit Testing

This week, I did a lab exercise to help me setup and learn the following testing topics:

• software testing concepts and terminology
• unit testing and integration testing

A quick disclaimer

Before going further, I must make it clear that I am no expert in this. Software testing is a large subject. Learning how to write good test cases and learning to write testable code is a skill that takes a lot of practice, and I'm only getting started. This blog post will only document my first steps in learning software testing and my experience setting up software testing for a C++ project.

How to select a testing framework

For this lab exercise, I had to select and setup a testing framework to my Markdown-to-HTML converter, ctil.

Testing framework choices can sometimes be an easy choice depending on the language or libraries used, but in other cases there will be a few popular options to choose from. For this lab we were recommended to pick the most obvious/popular choice for our language/platform, as these frameworks would have the most documentation, examples, and StackOverflow responses to questions we might have.

One popular C++ Testing Framework is Google Test, and is what I ended up using.

Researching Google Test

The Google Test Documentation provides a primer for first-time users. The primer introduces some basic concepts and terminology, some of which I've been able to learn for this lab exercise.

Basic Concepts

Test Case vs Test Suite

Test Case was the original term for a group of related tests. This term has since been replaced with Test Suite. The term Test refers to testing a specific program path with specific inputs and verifying the results.

Assertions

• Google Test consists of writing assertions, statements that check whether a condition is true.
• Assertions can have one of three results:
  • success
  • nonfatal failure
  • fatal failure
• fatal failures abort the current function; the program continues normally in all other cases
• assertions verify tested code's behaviour. if a test crashes or has a failed assertion it fails.

Assertions themselves are macros that resemble function calls. You can test a class or function by making assertions of its behaviour.

Creating a Test

• Tests can be made using the TEST() macro to define a test function. Inside TEST(), Google Test assertions are used to check values. The test's results are determined by these assertions. If any assertion fails (regardless of fatality), or if the test crashes, the entire test fails.

Setting Up Google Test

The Google Test documentation provides a method for adding Google Tests using CMake. Unfortunately my project does not use CMake. When contributing to projects for Hacktoberfest, I noticed many C++ projects used CMake. I'm starting to figure out why.

No CMake? No Problem

Microsoft provides a method for adding Google Test to a Visual Studio project. The document recommends creating a Google Test C++ Project and linking it to the C++ Visual Studio Project you want to test.

I originally setup Google Tests using this method, which would allow me to write and run unit tests locally. However, I think this method is impractical for a community project. Using this method, the Google Test library would exist in a different folder from my project. In other words, the Google Test library would not exist in my repository!

I answered my own forum post!

I wanted a way to directly integrate Google Tests into my project and into my project repository. I initially tried posting on the googletest github discussions page but also searched for solutions in the meantime. My search led me to this blog post, and although it works, I found this solution to be tedious.

I ended up experimenting with copying the contents of a generated Google Test project to my main project, and was able to integrate Google Tests. I ended up answering my own discussion post with my solution.

Google Tests Limitations (My Experience)

I've noticed the following limitations with using Google Tests:

• Since I'm doing this all on Visual Studio, my development environment is becoming less platform-agnostic (I'm not sure if Visual Studio supports Linux).
• I couldn't find a way to have test runner watch for changes and rerun tests, similar to JEST
• There doesn't seem to be an out-of-box support for coverage reports, at least not without additional tools

Adding my tests

When writing my tests, I initially had a hard time figuring how to actually test my methods. The default tests worked, but failed when I tried testing methods in the ctil module. My 'a-ha!' moment was remembering that the ctil module was a class, meaning in each of my tests I had to instantiate a cdot::ctil object. Figuring this out took longer than I'd like to admit, although it would've been avoided had I put more time reading into the documentation. In fact, it turned out each of my tests instantiate a cdot::ctil object, which may benefit from using a Fixture.

My tests

I added a few simple tests that test the ctil module of my program, a couple that confirm a filepath extension function works and a third that tests whether horizontal break lines in Markdown are properly converted to HTML. My tests didn't cover any interesting bugs or edge cases, although I have a good feeling when it comes to converting Markdown elements to HTML, I'll have to test a lot of cases.

Conclusion

I have some experience using JEST as a testing framework on Node JS projects, and setting up Google Test is a complicated system in comparison. To get JEST in a Node project, all you need to do is run

npm install --save-dev jest

on the project command line. Adding Google Tests to my project was more complicated. In addition, JEST also has a watch feature and coverage reporting out-of-box, which are things I wish Google Tests would support one day. I think I'll continue using testing frameworks in the future. However, I wish there was an easier way to add Google Tests to my project without CMake.