Clang Tidy, part 2: Integrate qmake and other build systems using Bear Automated refactoring of your source code using powerful open-source tooling

25.07.2017 Kevin Funk 12 comments

Introduction

This article is part of a blog series about Clang Tidy. In the previous article we learned about the general usage of Clang Tidy to automatically refactor source code for projects using the CMake build system. In this particular episode we’ll discuss using Clang Tooling on projects using different build systems with the help of Bear.

Motivation: So you want to use Clang Tooling on your project — but what if your particular project of interest is using another build system such as qmake, a build system predominantly used in the Qt world? There is a way to leverage Clang Tooling on any build system out there by just using a small helper tool.

Introducing Bear

Bear (Build EAR) is a tool that generates a compilation database for clang tooling. As we’ve learned in the previous article in this blog series, all that Clang Tooling needs to know about your project is a JSON Compilation Database, which is basically a replay of all the compilations (i.e. compiler invocations) happening during a project build. While the CMake build system can generate one out of the box, other build systems usually require additional work to get one.

Bear is a generic way to generate the compilation database during the build process, i.e. when running make for any build system. The concept behind Bear is that it intercepts the exec calls done by the build tool (i.e. make) and uses the command-line arguments passed to the compiler and stores them in the compilation database. Think of a strace-like tool filtering for exec system calls.

Unfortunately, Bear currently only works on Unix-based systems (FreeBSD, GNU/Linux and macOS), since it’s heavily relying on the LD_PRELOAD mechanism on *BSD/Linux and its macOS counterpart to intercept the system calls.

Getting Bear

On Ubuntu (since at least 14.04 LTS) or Debian (since at least jessie):

sudo apt-get install bear

On any other distributions not providing a package:

git clone https://github.com/rizsotto/Bear
cd Bear
cmake .; make; make install

Refer to the official documentation for more information.

Using Bear

This section will show how to use Bear with different build systems. Always remember that Bear is just a helper tool used to intercept the build tool used. The end result of Bear is a compilation database which can be consumed by Clang Tooling later.

Using Bear with qmake

The qmake build system is predominantly used in the Qt world. While this little tutorial is specific to qmake, the same workflow can be applied to any other build system.

Let’s use Bear on an example project called qtremoteobjects, which is a Qt module hosted on Qt Git.

<checkout qtremoteobjects>
cd qtremoteobjects
mkcd build

PS: Tip: mkcd just creates a directory and cd’s into it — source.

At this point we want to set up the build system. Note we cannot just call qmake .. alone, since on Linux this will use the linux-gcc spec by default. The compile flags used in the Makefiles generated using this spec won’t be compatible with clang, thus we need to resort to using the linux-clang mkspec.

qmake -spec linux-clang ..

This generated the root Makefile. Now we have to run make, but since we want to generate the compilation database on the fly as well, we have to “preload” bear.

bear make

After bear make finished, you’ll end up with a file called compile_commands.json in the current working directory. Similarly as if you would have let CMake generate this compilation database for you.

Let’s check what’s inside:

head compile_commands.json
[
{
"command": "c++ -c -pipe -g -std=c++1z -fvisibility=hidden -fvisibility-inlines-hidden -fno-exceptions -Wall -W -D_REENTRANT -fPIC -DQT_BUILD_REMOTEOBJECTS_LIB ... -I/home/kfunk/devel/src/qt5.8/qtremoteobjects/src/remoteobjects ... -o .obj/qconnection_local_backend.o /home/kfunk/devel/src/qt5.8/qtremoteobjects/src/remoteobjects/qconnection_local_backend.cpp",
"directory": "/home/kfunk/devel/src/qt5.8/qtremoteobjects/build/src/remoteobjects",
"file": "/home/kfunk/devel/src/qt5.8/qtremoteobjects/src/remoteobjects/qconnection_local_backend.cpp"
},
{
"command": "c++ ... /home/kfunk/devel/src/qt5.8/qtremoteobjects/src/remoteobjects/qconnection_tcpip_backend.cpp",
...

Looking good.

Note: If you interrupt bear make, a consecutive run of bear make will just override this compile_commands.json file. In other words, you’ll lose the information inside that file from the previous make run, of targest which have been compiled already. Thus in order to get a complete compile_commands.json, clear the build directory first, then invoke bear ... as before again.

Using Bear with any other build tool (for completeness sake)

This is what you need:

bear <build tool>

The build tool could be anything — it can even be a totally different build system which does not leverage traditional build tools such as make/ninja to invoke the compiler.

Running clang-tidy as usual

Next, as soon as we have a valid compile_commands.json file, we can run run clang-tidy as we learned in part 1 of this blog series, using the run-clang-tidy script:

run-clang-tidy-3.9.py -clang-tidy-binary clang-tidy-3.9 -clang-apply-replacements-binary clang-apply-replacements-3.9 -header-filter='.*' -checks='-*,modernize-use-auto' -fix
...
/home/kfunk/devel/src/qt5.8/qtremoteobjects/src/remoteobjects/qremoteobjectnode.cpp:419:5: warning: use auto when initializing with new to avoid duplicating the type name [modernize-use-auto]
QConnectedReplicaPrivate *rp = new QConnectedReplicaPrivate(name, meta, q);
^
auto
/home/kfunk/devel/src/qt5.8/qtremoteobjects/src/remoteobjects/qremoteobjectnode.cpp:440:9: warning: use auto when initializing with new to avoid duplicating the type name [modernize-use-auto]
QInProcessReplicaPrivate *rp = new QInProcessReplicaPrivate(name, meta, q);
^
auto
...

We’ll end up with a couple of changes in the source directory which we can commit afterwards.

Troubleshooting

Unfortunately there are a couple of issues when running Bear which one needs to take into account when using it.

Problem with PCHs from a different compiler

If we had used qmake -spec linux-gcc ... before, the clang-tidy invocation would have failed. You’d run into issues such as this one here:

error: no suitable precompiled header file found in directory '.pch/Qt5RemoteObjects.gch' [clang-diagnostic-error]

In other words: Clang tries to read a precompiled header (PCH) generated by GCC which it isn’t capable of => compilation fails.

Solution: Use qmake -spec linux-clang ... in order to use the correct compiler from the start.

Problem with PCH from a different compiler version

If we had used qmake ... without CONFIG-=precompile_header and clang++ is of a different version than the clang-tidy executable, we would have run into this issue:

error: PCH file built from a different branch ((tags/RELEASE_400/rc1)) than the compiler ((tags/RELEASE_391/rc2)) [clang-diagnostic-error]

Solution: Either make sure the clang-tidy version and the clang version match, or just disable precompiled headers in qmake by passing CONFIG-=precompile_header to it.

clang-apply-replacements executable segfaulting

Problem: When running clang-apply-replacements on a set of suggested fixes (stored as .yml files in a tmp directory) — which is done as part of the run-clang-tidy script — one may encounter invalid file references. When encountering the same invalid file references a second time, any currently released version of clang-apply-replacements will crash.

Example invocation:

% clang-apply-replacements /tmp/tmpIqtp7m
...
Described file '.moc/../../../../../../src/qt5.8/qtremoteobjects/src/remoteobjects/qremoteobjectabstractitemmodelreplica_p.h' doesn't exist. Ignoring...
Described file '.moc/../../../../../../src/qt5.8/qtremoteobjects/src/remoteobjects/qremoteobjectpendingcall.h' doesn't exist. Ignoring...
zsh: segmentation fault  clang-apply-replacements /tmp/tmpIqtp7m

The clang-apply-replacements executable crashes. Note I haven’t investigated why the invalid file references happen in the first place, but I’ve fixed the crash in clang-apply-replacements upstream for good.

Solution: Apply this patch when you have a source checkout of the Clang Tooling.

Unfortunately that means you’ll have to compile the clang-apply-replacements project yourself in order to take advantage of the patch for now or wait for a new Clang release.

Conclusion

With Bear one can use Clang Tooling on projects based on other build systems such as CMake. It’s a simple strace-like tool which intercepts the build tool and logs the way the compiler is invoked in a compilation database file, which can later be consumed by clang-tidy and other Clang Tooling helpers.

Any thoughts, questions?

Need help?

KDAB employs several engineers who are working with Clang Tooling on a daily-basis. We’re happy to assist you in case you have troubles using Clang Tooling in your project or want to get the most out of it: by letting us implement project-specific code checks or run automatic refactoring tools on your code base. We’ve helped clients modernizing their very large code bases successfully using tooling – something which would not have been feasible for them cost-wise doing it manually.

Contact us

Kevin Funk Senior Software Engineer