How to integrate OpenGL code with Qt Quick 2 applications (part 1)

The Qt World Summit was a rocking blast! More than 800 delegates, two days packed with sessions (and an additional training day presented by KDAB). Many old faces, lots of new ones, and a packed programme of sessions with terrific technical content.

Yours truly also happened to give a talk, "Integrating OpenGL with Qt Quick 2 applications". In my session I showed different ways of integrating custom OpenGL code within a Qt Quick 2 application. This is a use case for which there's more and more demand from the industry. Think for instance, of putting some fancy Qt Quick controls (buttons, sliders, etc.) overlaid on top of a 3D scene, rendered using an existing OpenGL-based solution.

My session was very well received by the audience, and for that reason, instead of simply publishing my slides, I have decided to turn my session into a series of blog posts. Here we go!

The Qt Quick 2 renderer

In order to understand how we can hook our OpenGL code into Qt Quick, let's take a step back and discuss how Qt Quick 2 renders a scene. The renderer subsystem of Qt Quick 2 is in fact quite a complex piece of machinery.

Qt Quick 2 Rendering

Qt Quick 2 leverages OpenGL and a dedicated render thread to render Qt Quick elements.

First and foremost, it leverages OpenGL to render Qt Quick elements. This was a profound design decision, driven by the fact that virtually every user device these days has an OpenGL-capable GPU. Furthermore, OpenGL-based rendering is necessary if one wants to achieve drawing at a stable 60 frames per second, add special effects (drop shadows, particles, blur, etc.), have smooth animations, and so on. Hence, all of your visible elements in a scene are rendered by means of points, lines, triangles and shaders. This decision makes it also possible to easily integrate your custom, existing OpenGL within the Qt Quick renderer.

Second, it was decided that the renderer may use a dedicated render thread, different from the main thread (that is, the thread that runs main()). There are two ideas behind using a separate thread for rendering:

1. It allows the rendering to proceed (and keep animations going at 60fps) and your UI not to look stuck even if your CPU is somehow busy doing calculations, or your main thread is in some blocking function call, etc.;
2. Symmetrically, it allows the main thread not to get stuck by the GPU, in case by accident the GPU is being too slow at rendering a scene.

This dedicated thread is used on basically all the platforms supported by Qt 5.6 (excluding only ANGLE on Windows). Even if your platform isn't supported today, chances are it will be in a future version of Qt, hence you should always take threading into account when integrating custom code with the Qt Quick renderer.

The scene graph and the synchronization round

But how does the renderer actually render a scene? The renderer uses a specialized data structure called a scene graph. The name "scene graph" is actually very common in the topic of 3D rendering; it refers to any data structure that can be used to analyze and optimize certain tasks (not just rendering, but also handling things such as physics simulations, path finding, etc.). I will not go in depth about how the to use the scene graph to build custom Qt Quick elements, as the purpose of this blog series is how to integrate your existing OpenGL code, so I will just show the bits relevant to this discussion.

In Qt Quick's world, the scene graph is used for rendering purposes only; and it contains all the visual information that is used to draw a scene. Keep this in mind -- all the non-visual information is lost in the process. For instance, in order to draw a rectangular button, the scene graph does not know that it reacts to clicks in some way: all that it knows is that in order to draw that button, one needs first to draw a quadrilateral (actually two triangles) with certain per-vertex colors to simulate a background gradient, and draw on top a smaller quadrilateral with a texture (i.e. an image) that represents button's text.

The Qt Quick scene graph is implemented as a tree of nodes. Each node type (subclass of QSGNode, where SG stands indeed for scene graph) implements a piece of functionality, such as altering the transformation for the children, or the opacity, or holding a piece of geometry (some vertex buffers), shaders and textures for something that needs to be drawn. You can see some of the Qt Quick scene graph classes in this diagram:

The scene graph is built by iterating over the QML elements. Each Qt Quick visual element type (like Rectangle, Image, Text...) builds a small tree of nodes in its override of the QQuickItem::updatePaintNode function, and returns them to the renderer. The renderer can then take these trees, analyze them and decide what's the best strategy to paint them.

"Wait a second", the careful reader should be saying right now. How does the render thread exactly build the scene graph data structure from the QML elements, which live in the main thread? Of course, with multiple threads comes the problem of synchronization. The render thread can't just walk over the item tree if the main thread is also modifying the elements at the same time.

While one might think of starting to add locks all around but this quickly gets expensive. The solution chosen by Qt Quick is more drastic and removes the problems of handling locking explicitly. The render thread and the main thread will synchronize by pausing the main thread and therefore allowing the render thread to safely call the QQuickItem::updatePaintNode function on all the visual elements in the scene. (So, yes, updatePaintNode gets actually called from another thread). Once the render thread has done this, the main thread gets unlocked and it is free to go again. The render thread has now gathered all the information it needs to draw the scene on screen, and it will proceed to analyze the scene graph and render it.

Sometimes pictures speak louder than words; this is the small dance of the main thread and the render thread synchronizing (picture taken from the Qt documentation):

This is what happens every time a new frame gets requested, for instance, because a visual item in the scene has been marked as dirty (because some property changed) or because the QQuickWindow::update function gets called.

This picture actually reveals something else: that during the synchronization round, the render thread will emit signals. Our first way of integrating OpenGL with Qt Quick 2 uses those signals. We can connect slots to these signals, and call OpenGL functions from those slots, therefore mixing custom OpenGL code within Qt Quick 2's rendering, which is exactly what we wanted to do!

What now?

Now that we know how the Qt Quick 2 renderer works, in the subsequent posts we are going to investigate three different ways of integrating OpenGL within a Qt Quick 2 application:

1. OpenGL underlays and overlays
2. Custom OpenGL-based elements
3. Manually controlling the Qt Quick rendering

Stay tuned for the next blog post!