Find what you need - explore our website and developer resources
Stephen Kelly
25 June 2012
One of the useful outcomes the work Bertjan did on tooling for program understanding and refactoring is a list of considerations we can use to assess the suitability of new tools.
Section 1.3.5 of his thesis details the requirements for a similar porting system:
The qt4to5 porting tool is scalable. It is designed by Google engineers to be used on large codebases, and to operate on multiple translation units at a time. As the tool is based on a compiler, if you have the resources to compile it, you have the resources to port it.
The tool has a full understanding of the C++ language. This does not really mean that everything can easily be ported however. For example:
template<typename T> void doSomething(const T&) {
T.intersect();
}In cases where T is a QRect, the intersect() should be changed to intersected(), but not if T is a QSet. This is determined by the caller, and the porting tool can not port such cases automatically (see also FX 6).
However, the intent of the tool is not to port everything automatically, but to port the boring and easily automatable parts automatically.
The tool works, but the command line arguments are not very convenient, and
probably never can be. I've written a python script to make it easier, but a
graphical tool would be better:
renameMethod("QPaintDevice", "numColors", "colorCount")
renameMethod("QImage", "numColors", "colorCount")
renameMethod("QImage", "setNumColors", "setColorCount")The graphical tool would be able to tell the engineer using it the refactoring
steps that can be done before starting the switch to Qt 5 (eg porting away
from Qt 3 support), so that it can be recompiled in between.
A Qt based tool could also link statically to the tooling framework and
avoid calling external processes. It would be possible to integrate such tooling into Qt creator.
The tooling is integrated with CMake already. The tool needs access to the
actual command line that should be used to build each compilation unit.
CMake provides that when compiled with -DCMAKE_EXPORT_COMPILE_COMMANDS since
CMake 2.8.5.
Currently it only works with the CMake "Unix Makefiles" generator and Ninja generator. The feature of creating such a compilation database could similarly be added to qmake presumably, but that has compilications
because for some porting steps you need to be compiling against Qt 4 (see FX 1 below), so it would need to be added to the qmake in Qt4 and probably Qt 5 too.
Alternatively, I also created a compiler wrapper which creates such a compilation database when invoked. There may even be a simpler solution by using the porting tool as the actual compiler.
Although the porting tool can port enums, the upstream stuff in the clang
repo can not.
I had to extend it, which was quite trivial:
namespace clang {
namespace ast_matchers {
const internal::VariadicDynCastAllOfMatcher<
clang::Decl,
clang::EnumConstantDecl> EnumeratorConstant;
}
}I created the EnumeratorConstant functor which can then be used to match AST
nodes of type EnumConstantDecl, which
basically means use of an enum.
That EnumeratorConstant can then be used just like the rest of the syntax
for matching, can contain inner matchers, etc:
DeclarationReference(
To(EnumeratorConstant(
HasName("QSsl::SslProtocol::TlsV1"))
)
)will find all used of QSsl::SslProtocol::TlsV1.
So, the tooling and API is fully extensible.
The tool only makes changes on parts of the code that are specified in the
matching expressions (such as the enum matcher above).
Therefore, the predictability of the impact of running the tool depends on the exactness of the
matching expressions.
The tool might also try to edit your system headers for example if renaming
a virtual method or changing its arguments. To prevent that, the tool takes a source-dir as an
argument and verifies that edited files are below the provided source dir. That feature is not fully used yet
however, and there is scope for clang upstream to add such a feature to ensure correct support for symlinks and relative paths etc.
The python wrapper script provided in the repository creates git commits with commit messages
describing the change being made at each step (micro commits).
The user of the tool can review all commits in gitk, or build any step by
checking it out (or build all steps by running a git script).
Any other graphical user tool could do the same thing. The result of such an execution of the script can be seen in kdelibs.
Section 2.2 details requirements for a fact extraction system. This is a necessary
component of a porting tool, as it relies on having accurate information to complete a port.
However, not every requirement of a fact extraction system is a requirement of a porting tool.
The tool has some fault tolerance, which comes from clang. If you remember
Chandlers GoingNative talk, he mentioned that if clang encounters an error, it tries to ignore it and keep processing.
Indeed, the python porting tool currently only runs cmake, but not make, which means that cmake dependency scanning and file generation is not
invoked. That means that moc files and ui_ files are not generated. When
running the tool clang will give an error about missing includes for the moc files, but will continue to
process the file.
However, while there is some fault tolerance, the specific code to be ported
does have to be correct. I think mostly we'd be porting code which already
compiles with Qt 4, so I don't think fault tolerance is a huge problem for a
porting tool (as opposed to a fact extraction system).
All parseable code in the software targetted for porting is extracted correctly, because it is based on a compiler.
However, this is only true for one particular target platform at a time (see also FX 9).
When used on linux, anything in #ifdef Q_OS_WIN for example will not be
ported. It may be possible to use Wine headers or a cross compilation build to port such code on a linux host.
The parser is compiliant to at least most of C++03 and parts of C++11. The
C++11 parts might be relevant if we ever have to port code which
uses C++11 and Qt 4, which is not unheard of.
While clang can find C++11 features such as lambdas, the new tooling system does not yet have API for matching or processing it.
This item is not relevant to our porting tool. This is only relevant to fact extraction
systems to for example find the amount of uses of a method. The porting tool
compiles each translation unit in isolation.
In our case the fact extraction framework is clang, and it extracts all
available information.
My investigation of the new clang tooling APIs did not include playing with preprocessor constructs much,
but I know that clang stores all information extracted from the preprocessor.
The C++ grammar is partially context dependent, and that can lead to ambiguity as described before when using templates for example:
template<typename T> int numColors(const T&) {
return T.numColors();
}Is T a QImage or a QPaintDevice?
This item is not relevant to porting tools. This is only relevant to the fact extraction
framework. In our case the fact extraction framework is clang, and it
extracts all available information.
The python wrapper script provided in the repository assumes the existance of a git repository, and uses git grep to find uses of a method which needs to be ported. This way not all files need to be processed for each step of porting, but only the ones which contain
something to be ported or a false positive (eg QSet::intersect).
This works well for renaming, but perhaps other tricks and heuristics would
be needed for more complex porting steps. For example, grep can't find
QAtomicInt::operator int(), but if we grep for QAtomicInt, which might be
used in a header file, then grep for uses of that header file we might get
all relevant files that need to be ported to use QAtomicInt::loadAcquire().
Any user interface tool should keep this git dependency to create
the clean patches. Even where code bases which do not use git exist,
the patches can still be created and reviewed in a local git repo created with git init.
Clang can currently only generate code on Unix systems, so it can't be used to
generate Windows binaries.
There have been some windows specific patches on the
mailing list though, and it's possible that the parser works just fine on
windows (the only part we need).
This still needs to be fully investigated.
clang and the required tooling for the porting tool are covered by a weak copyleft licence. This is not reversible, but does not prevent the code from being part of proprietry forks. The license choice assumes that doing so is not worth it anyway as a social measure if not a legal one.
The actual code of the porting tooling is currently in a branch, not in clang trunk, which
is a minor temporary inconvenience.
Choose between learning advanced QML programming, Modern C++, integrating Rust and Qt or 3d rendering with Vulkan. The KDAB Training Day 2025 will take place in Munich on the 8th of May, right after the Qt World Summit on 6th to 7th of May.
Learn more
Stay on top of the latest news, publications, events and more.
Go to Sign-up