Integrating OpenGL with Qt Quick 2 applications (part 2) Implementing overlays and underlays with the Qt Quick 2 renderer

23.02.2017 Giuseppe D'Angelo 2 comments

In the last blog post we gave a very high level introduction to the Qt Quick 2 renderer. We also showed how various signals are emitted by the renderer during the synchronization and rendering steps. In this blog post we’re going to discuss those signals and show how they can be used to implement overlays and underlays.

Signals emitted by the Qt Quick renderer

Let’s start again by looking at this diagram of the synchronization between the render thread of the renderer and the main thread when a new frame needs to be rendered:

What exactly are the signals emitted by the Qt Quick 2 renderer that we can use? Some of them are visible in the picture above, some of them are not, so I will list all of them here:

• QQuickWindow::sceneGraphInitialized: emitted when the renderer initializes the OpenGL context and the scene graph resources.
• QQuickWindow::sceneGraphInvalidated: emitted when the renderer has invalidated the scene graph, and the OpenGL resources should be freed.
• QQuickWindow::beforeSynchronizing: emitted before the scene graph synchronizes with the main thread. However, the main thread is already blocked at this point.
• QQuickWindow::afterSynchronizing: emitted after the scene graph synchronized with the main thread. The main thread is still blocked at this point.
• QQuickWindow::beforeRendering: emitted before any rendering by the Qt Quick renderer has taken place.
• QQuickWindow::afterRendering: emitted after the Qt Quick renderer has rendered the scene graph.
• QQuickWindow::frameSwapped: emitted after the Qt Quick renderer has swapped the back and the front buffers.

That is a lot of signals indeed! But it’s extremely good news, because it means we can intervene at any point of the synchronization/render process by simply connecting slots to these signals.

An important consideration is that the OpenGL context used by the Qt Quick renderer will be bound when the signals are emitted. This means we can simply have OpenGL calls in our slots, without the problem of finding the “right” OpenGL context to use and make it current. Also, note that the signals are emitted from the render thread, not the main thread. This typically means that we will need to force the connection type to Qt::DirectConnection, if the receiver object lives in the main thread. It also means we will need to carefully manage shared state between the main thread and the GUI thread, for instance by protecting the state with mutexes.

Remember that after the synchronization is complete, the main thread is unblocked and free to go while the rendering signals are being emitted; we can easily have races between the main thread and slots invoked directly from the render thread.

OpenGL underlays and overlays

So how can we implement, for instance, a simple underlay (that is, drawing custom OpenGL content below a scene)? Here’s a screenshot of an example application:

The trefoil knot is rendered by an external piece of OpenGL code that we are not going to discuss here. Let’s assume it is some existing code we don’t want to touch, and treat it as a black box. The small controls on the bottom are instead implemented using Qt Quick. As you can see, the controls are truly over the OpenGL content, and nicely blended with it. The three sliders control the position of the virtual camera in the world (in spherical coordinates). For simplicity, these coordinates are wrapped by a Camera class, an instance of which gets exposed to the QML engine.

How can we obtain this result using the signals emitted by Qt Quick? Here you have the relevant parts of a possible implementation, done using a subclass of QQuickView. Let’s start by looking at the constructor:

QuickView::QuickView(QWindow *parent)
    : QQuickView(parent)
    , m_camera(new Camera(this))
    , m_renderer(new MeshRenderer(this))
{
    connect(this, &QQuickWindow::sceneGraphInitialized,
            this, &QuickView::initializeUnderlay,
            Qt::DirectConnection);

    connect(this, &QQuickWindow::beforeSynchronizing,
            this, &QuickView::synchronizeUnderlay,
            Qt::DirectConnection);

    connect(this, &QQuickWindow::beforeRendering,
            this, &QuickView::renderUnderlay,
            Qt::DirectConnection);

    connect(this, &QQuickWindow::sceneGraphInvalidated,
            this, &QuickView::invalidateUnderlay,
            Qt::DirectConnection);

    connect(m_camera, &Camera::azimuthChanged, this, &QQuickWindow::update);
    connect(m_camera, &Camera::elevationChanged, this, &QQuickWindow::update);
    connect(m_camera, &Camera::distanceChanged, this, &QQuickWindow::update);

    setClearBeforeRendering(false);

    setResizeMode(SizeRootObjectToView);
    rootContext()->setContextProperty("_camera", m_camera);
    setSource(QUrl("qrc:///qml/main.qml"));
}

In the constructor a Camera object is created, representing the position of our virtual camera, and a MeshRenderer object, which is our black box performing OpenGL rendering (drawing the knot). We then connect some of the signals emitted by the Qt Quick renderer to some local slots. We’re not interested in all of them; for this example we just want to know:

• when the scene graph is getting created and destroyed, so we can initialize and release the OpenGL resources;
• when the synchronization round is happening, so we can safely access state in the main thread (which is blocked);
• when Qt Quick is about to start rendering the next frame, so we can paint before it, therefore realizing an underlay.

Notice the Qt::DirectConnection connection type. Since the receiver (this) is living in the main thread, and the signal is emitted by the render thread, we must force the connection type otherwise it would result in a queued invocation, with disastrous results. After those connections, we connect the update slot to the various changed signals from the Camera object, because every time the camera moves we want to schedule a redraw. We then ask Qt Quick not to issue a glClear on the color buffer before starting its own rendering. Since we want to draw something before Qt Quick, clearing would result in not seeing anything of what we had rendered. The setClearBeforeRendering call disables (or enables) the automatic clearing. Finally, we expose the Camera object to QML (so that it can be manipulated through the Qt Quick sliders) and load a QML file.

What about the slots themselves? Let’s have a look:

void QuickView::initializeUnderlay()
{
    m_renderer->initialize();
    resetOpenGLState();
}

void QuickView::invalidateUnderlay()
{
    m_renderer->invalidate();
    resetOpenGLState();
}

void QuickView::renderUnderlay()
{
    m_renderer->render();
    resetOpenGLState();
}

void QuickView::synchronizeUnderlay()
{
    m_renderer->setAzimuth(m_camera->azimuth());
    m_renderer->setElevation(m_camera->elevation());
    m_renderer->setDistance(m_camera->distance());
}

Yes, it’s that simple! Since our renderer is a black box, we simply call into it from the slots. When the scene graph is initialized or invalidated, we respectively initialize or invalidate our renderer. When Qt Quick is about to render itself, we call our own rendering code, resulting in drawing below Qt Quick.

A bit more interesting is the synchronization slot. Remember that all these slots are called directly from the render thread; as such, concurrent access from the main thread could easily result in a data race. While we could expose the Camera object to our custom OpenGL code, we would then need to implement a thread-safe Camera, which may be tricky or just impossible. For this reason, we’ve decided to follow what the Qt Quick renderer does: we extract from the objects living in the main thread all the information required for rendering the frame. In this case, the object living in the main thread is m_camera, and the information is the spherical coordinates, which get stored somewhere else (for instance inside our renderer). All of this happens during the synchronization stage, while the main thread is blocked, so there is no risk of concurrent access and therefore data races.

Finally, what is the deal with all the resetOpenGLState calls? The fact is that the Qt Quick renderer does not like modifications to the OpenGL state. In other words, if in our slots we modify some OpenGL state (bind textures, change the used shader program, and so on), we must reset the OpenGL state to whatever it was before we touched it. While this is possible in theory (OpenGL has all the necessary APIs for querying state), it’s also terribly complicated due to the countless settings that OpenGL has; moreover, if we simply call into some third party code, we have no control over which state gets touched by that. For this reason, there is a convenience call that resets the OpenGL state: QQuickWindow::resetOpenGLState. Be sure to call this function every time some OpenGL state gets modified in a slot!

Further steps

In the next blog post we are going to deeply integrate a custom OpenGL item inside a Qt Quick 2 scene. Stay tuned!