C/C++ Debugging Tools

In this blog on debugging and profiling, I would like to give you an overview of the debugging tools that exist for C and C++ applications.

Automated Testing

The first thing you need to do is make sure your code is properly tested. This, in itself, is not debugging, but it enables you to make sure that you don't introduce three new bugs when you fix one. One of the ideas that you can use for this is called Test-Driven-Development, which means writing the tests before you write the code that it will test. This way, you can make sure the test is testing the right thing. If the test fails, you fix it, and then it passes. It's a very good idea to add full unit test coverage for classes you're going to rewrite or refactor, so that you don't introduce regressions compared to the old code. To do this, you can use one of the existing unit test frameworks. As a Qt developer, I know especially QTestLib. But you can also use Google Test or Catch and there are actually many others. The goals of those is to save you time because you don't have to set up everything so you can write a test, make sure that all the test methods are called, how to handle failures, and all of that.

One step further is to integrate these tests with your continuous integration so that, after every commit or every night, you get a full build from scratch and a full run of all of the unit tests. All of this improves the quality of your application and is very necessary for the actual debugging that we are going to talk about because you'll make sure you don't introduce regressions when actually fixing a bug.

Code Coverage Tools

Another thing that is quite related to unit testing is code coverage. You want to make sure that you have a sufficient number of unit tests, and that they cover an important percentage of your code. Note however that it's usually pointless to try to go to 100% coverage. What's important is to test the parts of the application that are critical to you. That could be simple utility classes that are used everywhere, like the string class or the vector class if you have one of those. Or it can be the part of the application where the actual business logic is and you don't want any bugs in it. One thing you can do to find out if your unit tests cover enough of the important code is set up code coverage.

There are tools that will tell you 90% of these files have been covered by unit tests and you can then figure out exactly which part of the code you're missing in your tests. There are many tools for doing that. The most well known of them is gcov on Linux. It actually goes together with the compilers GCC and Clang, since those generate the information for gcov, when passing the --coverage compiler flag. There are also tools on Windows as part of Visual Studio, or you can install OpenCppCoverage. Also, Squish Coco is another one. All of these tools allow you to measure how much of your code is actually tested by your unit tests. This is an area where there are many, many concurrent solutions because it's something that's very much used in some domains, for instance anything that will fly in a plane has to be very well unit tested and covered by those tests. These are domains where code coverage is really important.

Static Code Analysis Tools

Another thing you can do to improve the quality of your code is set up static code analysis. This is another area with many different tools. The goal is to have bug detection without even running the application, simply having a tool that looks at the code and tells you where there's a construct or pattern that might have a bug. It's then up to a developer to look into it. These days, the most well-known tools for this are actually the compilers themselves. If you enable sufficient amounts of warnings in GCC and Clang, or Visual Studio, you will get feedback from your compiler about what you can improve in your code. So my recommendation would be to enable as many of those warnings as you can in order to detect as many problems as possible. Also, use more compilers than just one. If you can use two or three, that's better because you will get different feedback from the compilers. Then, there are some additional tools that you can use.

Then, there are many more specialized tools, like Coverity, PVS-Studio, and so many others. It's quite interesting to look at, for instance, what PVS-Studio can find by reading their blogs. They have lots of interesting finds in open source applications, for instance. It's quite educational. This is also something that you can set up as part of your continuous integration so you get regular feedback on what can be improved in your code.

Logging (debug messages)

Let's talk about things that look a little bit more like what you would expect in terms of debugging. The most well-known debugging technique would be printf or equivalent: some logging of messages for figuring out into which function we're going into, what is the value of this variable, and so on. So, there are many variants to this: obviously, the ones that are part of C and C++, like printf and cout, and all those that come with your framework, like qDebug in Qt, or additional libraries, like log4cxx, boost::log, log4cplus, easylogging++... One thing most of those have in common is that they make install to turn on and off whole sets of debugging statements. So, if you are debugging a parser you would only enable the parser set of messages or if you are debugging printing you would enable all of the messages that are about printing, and so on. This is a lot more practical than having 10,000 messages per second and then you have to figure this out, or having everything off by default and you need to comment what you need. This is a lot easier if you can just toggle a switch somewhere and get what you need.

Assertions

Now let's talk about assertions. These are, at runtime, a way to say "this will never happen". If it does happen, stop the application and let me debug why it happened. That is something you would use for logic errors. You'd think your program is done in a way that this can never happen. This is especially useful for preconditions, like this pointer shouldn't be known before passed to this function; or postconditions, like this function will never return a unit pointer; or invariants, things that shouldn't change while the function is being executed. But, of course, it's useful for many other things as well. You shouldn't use it for runtime errors, like this file was not found. That is not really a logic error, it's more of a setup problem. You don't want to abort your application just because the user misplaced a file.

Another kind of assertion is the static assertion. Those are part of the compilation step. The compiler will tell you that a check failed, so you have to look into it. For instance, you would use this for an enum with the wrong amount of values, for inconsistent read only data, for inheritance that isn't as we expected, for unsupported CPU architecture, and so on.

Tracing

Let's talk about tracing. One of the things you can do is investigate which dynamic libraries are being used by your application. That's something you would do by using tools like ldd on Linux, Dependencies.exe on Windows, or otool on MacOS. This will simply list the shared libraries that are used by your application because possibly you are not using the one you thought and this could be the reason for the bug.

At runtime you can also figure out what your application is doing, as a black box. From the outside, you ask the application to show you all the files it's opening. That's something you can do with strace or the equivalent tools on other operating systems. You can ask it to show you any time it's going to (attempt to) open a file, use a socket or anything else that goes through a system call. It's a common debugging technique when you don't actually have the source code for your application, but it's also very useful if you do.

The way it works is you record the run of your application, hopefully triggering the bug you're after. Then, when you replay that recording (in gdb, launched by RR), you can go backwards and forwards and skip ahead, skip backwards, and so on. You can navigate through the recorded run of your application any way you like. That is extremely useful when you would usually hit that problem with gdb where you think the value is wrong and wonder how it's calculated and need to go back in time to find out. You can't do that because you don't have a time machine (at least I don't). But RR has the full recording and can go back because it has recorded all of the information to be able to do that. So, that's a very worthy tool to look into.

What it can do is introspect a Qt application, which means look into it and tell you all of the QObjects that are present, show you all the QWidgets graphically, same for all the QML elements in the scene, and all of the 3D elements in the Qt 3D scene, and so on, and so on. It has a large number of modules where you can get all of that information and know more about your application. It's not a debugger, per se. It's not going to go step-by-step in your code. This is much more like getting an overview of your Qt application and everything that it has created, and it's a way to debug problems like why a widget isn't big enough. Is the minimum size wrong? Is the size policy wrong? You can get information about all of these properties from GammaRay. It even supports remote debugging if you're doing embedded development and it supports attaching to a running application, which is also quite interesting.

It will run your application in a very slow way. It takes its time, but it will look at what your application is doing, once compiled right. It's running on the binary. It will tell you things like you are using memory after you deleted it -- that's very bad -- or you are using this bit of memory without initializing it, and all of these kinds of errors. So that's quite useful when you have behavior that doesn't seem to be reproducible. You run the application again and get a different result. That's the type of bug where Valgrind really shines because it will tell you exactly where this source of invalid memory usage comes from. It has other tools besides this one. It has helgrind for race conditions. It has massif for memory usage and it has callgrind for profiling. For these three things, there are alternatives listed in this blog that end up actually being better. But the default tool, memcheck, is very, very useful.

Sanitizers

Finally, let's talk about sanitizers. This is another way to detect problems like the use of invalid memory or debug deletion. Instead of doing that with an outside tool, what we can do with sanitizers is actually ask the compiler to inject code into your own code that does all of these verifications. Any time your code is going to allocate memory, it will remember that. Any time your code will use an area of memory, it will first check that it's able to do so. This is extremely powerful. It's a lot faster than Valgrind. It is part of your compilers, if you're using GCC 4.9 (or later) or Clang 3.1 (or later) on Linux or if you're using Clang 6 (or later) or Visual Studio 2019 16.4 (or later) on Windows.

There are four types of sanitizers:

• the address sanitizer will tell you about all of the memory usage issues
• the leak sanitizer (included in the address sanitizer, but available separately) will tell you about memory leaks
• the thread sanitizer is extremely good at telling you about race conditions in particular, which is something that's really hard to detect otherwise
• the undefined behavior sanitizer tells you when your code has some undefined behavior

There are some additional sanitizers being developed, such as the memory sanitizer to detect uninitialized memory, but this one requires recompiling all your libraries with it, so it's not really convenient.

This was a summary of all of the debugging tools that are available for C and C++ applications. If you'd like more details about any of these, we actually do full trainings on this so that you can learn more about using these tools, what they can do in practice, and then do some exercises to get used to them.