New in Qt 3D 5.11: Generalized Ray Casting

17.04.2018 Mike Krus 3 comments

The 5.11 release of Qt 3D is mostly about speed and stability but it also introduces a number of new features.

One of them is generalized ray casting which can be used to find objects intersecting a 3d ray.

Object Picking

Available since 5.6, QObjectPicker can be used to detect mouse interactions with an entity. As the user moves the mouse and presses the buttons, events will be emitted by the component. This can be used to change properties of a entity based on the events, highlighting it when the mouse enters, scaling it up when a button is pressed, moving it when the mouse is dragged, etc.

By default, only mouse buttons will trigger events (if the mouse overlaps the geometry). This is done for performance reasons since computations can be expensive (see below). However, this can be changed by setting properties hoverEnabled, to detect when the mouse enters or leaves the entity, and dragEnabled, to detect when the mouse moves across the surface of the geometry.

When the mouse is pressed on an entity with a QObjectPicker component, that component will “grab” the mouse and only that object will be considered for picking events, until the mouse button is released.

When a hit is found, the generated event will contain information about the point of the geometry under the mouse. In particular, it will contain the coordinates of the mouse (in screen space) where the hit was made, the state of the buttons. But it will also include the coordinates of the point on the surface of the geometry both in local and in world coordinates.

By now it should be obvious that QObjectPicker, like MouseArea, is most suited to give behaviours to object in the scene, should as changing the scale to make them bigger when the mouse hover over them, or changing the colour when the object user “selects” the object by clicking on it.

While is very useful in its own right, it has several limitations:

• It is dependent on mouse events: picking is implemented by casting a ray through the scene (details below) but there is no API to construct an arbitrary ray and test for objects intersecting it.
• If the component is attached to an internal node of the scene graph, child objects will also be tested for hits but the resulting event does not contain any information about which child was actually hit.
• There is no event generated when no hits occur.

Ray Casting

Picking as described above is implemented by casting a ray through the scene and looking for objects which intersect the ray. This is roughly how the process works:

• For each mouse event, compute a ray between the near and far planes and transform it to world space.
• Traverse the scene graph and look for every entity which has an object picker and where the bounding volume intersects with the ray.
• If primitive picking is enabled (see below), traverse every primitive (triangle, line or point) to find which ones intersect with the ray.
• Once all the hits are collected, sort them by distance to the origin of the ray.
• Generate events for the closest hit or all of them depending on the settings (see below).

As you can imagine, on complex scene graphs with lots of primitives, this can be expensive. Qt 3D attempts to optimise things by using a bounding volume hierarchy to quickly eliminate entire portions of the scene graph which don’t intersect with the ray. It also uses a thread pool to test multiple objects in parallel. Best results can be achieved by properly constructing scene graphs and avoiding single objects with large numbers of triangles.

The algorithm can be controlled globally by using the pickingSettings property of the QRenderSettings class, which is an instance of the QPickingSettings class.

The most important property is the pickMethod:

• By default it’s set to BoundingVolumePicking which is the fastest method: it will only test for bounding volume intersection. While this is very fast, it is also very imprecise. Internally, Qt 3D uses bounding spheres which can grossly exaggerate the actual volume of the object. Also, a small object positioned in front of a large one may be entirely encompassed by the large bounding sphere and thus be unpickable.
• This property can also be set to TrianglePicking (and since 5.10 to LinePicking and PointPicking) to ask for hit tests to be made with individual primitives. This is of course far more time consuming.

Another important setting is the pickResultMode property:

• When set to NearestPick (the default), only the closest hit will trigger an event
• When set to AllPicks, an event will be generated for every intersection. If you imagine picking a sphere, in most cases this will generate two events. Events will be generated from front to back but they will be delivered separately and there is no way of knowing when the last event is received.

There are also settings to control how the face orientation affects picking and a world space tolerance factor to control line and point picking.

Generalized Ray Casting

The ray casting operations described above are those used by the QObjectPicker backend. As explained before, they rely on mouse events in order to compute the ray that will be used to do the hit tests.

But there are many uses cases for more generalized ray casting. Still in screen space, an application may use a specific cursor that can be controlled independently from the mouse cursor (like a gun sight). Or in an VR environment, you may want to find the object at the centre of the screen to implement gaze selection.

Also in a VR setting, you may want to implement a 3d “wand” which is manipulated using the VR controls. A 3d ray with a position, direction and length matching the virtual wand could be used to find intersecting objects. Using action buttons, you could then implement grabbing and dragging the intersecting objects.

QAbstractRayCaster

This is an abstract class that cannot be instantiated directly. If defines common properties for all sub-classes. Note that ray caster components cannot be shared between multiple entities.

The runMode property controls how often ray casting tests are performed. All ray caster instances are disabled by default. When enabled is set to true, a ray casting test will be done at the next frame. If runMode is set to SingleShot (the default), the component will disable itself and no further tests will be performed until it is enabled again. If runMode is set to Continuous, tests will be performed at every frame until the node is disabled.

Controlling what gets picked

The filterMode property and the list of QLayer instances can be used to control which portions of the scene graph are considered when casting rays. You may want to disable picking on certain portions of the scene graph, or use different rays in different sub-graphs.

This can be achieved by using QLayer components and then adding them to the relevant ray caster node. The filterMode will control if the layers enable or disable ray casting and how to evaluate multiple layers.

Remember the QLayer has a recursive property to control if the layer applies to the specific entity only or if it should affect all of its children.

Getting picking results

When a ray caster test completes, the list of results will be assigned to the hits property of the component. This list will be empty if nothing was found. A change notification will always be emitted even if two successive tests produce the same result.

Each entry in the list contains the details of a hit, sorted front to back. It contains:

• the type of hit (Entity, Triangle, Line, Point) depending on the setting used in QPickingSettings
• the id and a pointer to the entity containing the geometry
• the coordinates of the intersection point in local and model coordinates
• the distance to the origin of the ray
• the index of the picked primitive
• the indexes of the points used to build the picked primitive.

While the C++ implementation used a properly typed class, the QML api uses a QJSValue containing an array of javascript objects with the relevant properties included.

QScreenRayCaster

QScreenRayCaster is used to do ray casting queries based on window coordinates without relying on the user using the mouse pointer. A test will actually be performed for every active render target.

The 3d ray is computed by constructing a ray at the specified coordinates between the front and back clipping planes and transforming that ray back in to model coordinates using the relevant camera projection and transformation.

The component has helper methods to trigger a test. All that basically does is enable the component so that a test happens at the next frame.

QRayCaster

QRayCaster defines a ray in the local coordinate system of the entity the component belongs to. This means the actual ray is affected by the world transformation applied to the entity.

The ray has an origin and a direction.

The ray also has a length, any hit further away from the origin will be ignored. If that length is less or equal to zero, the ray is considered to be infinite. Bear in mind that it has an origin and a direction, so it’s only half infinite 🙂

The component also has helper methods to trigger a test, optionally setting the properties of the ray.

Example

Qt 3d source code contains a basic QML example for using both ray caster classes.

    ScreenRayCaster {
        id: screenRayCaster

        onHitsChanged: printHits("Screen hits", hits)
    }

    MouseHandler {
        id: mouseHandler
        sourceDevice:  MouseDevice {}
        onReleased: { screenRayCaster.trigger(Qt.point(mouse.x, mouse.y)) }
    }

    function printHits(desc, hits) {
        console.log(desc, hits.length)
        for (var i=0; i<hits.length; i++) {
            console.log("  " + hits[i].entity.objectName, hits[i].distance,
                        hits[i].worldIntersection.x, hits[i].worldIntersection.y, hits[i].worldIntersection.z)
        }
    }

In the case of the ScreenRayCaster, a test is triggered whenever the mouse button is released.

    RayCaster {
        id: raycaster
        origin: Qt.vector3d(0, 0, 4)
        direction: Qt.vector3d(0., 0., -1.)
        length: 5

        onHitsChanged: printHits("Model hits", hits)
    }

    KeyboardDevice { id: kbDevice }
    KeyboardHandler {
        id: kbHandler
        focus: true
        sourceDevice: kbDevice
        onPressed: {
            if (event.text.toLowerCase() == "a") { raycaster.origin.x -= .1; raycaster.trigger() }
            if (event.text.toLowerCase() == "f") { raycaster.origin.x += .1; raycaster.trigger() }
            if (event.text.toLowerCase() == "s") { raycaster.origin.y += .1; raycaster.trigger() }
            if (event.text.toLowerCase() == "d") { raycaster.origin.y -= .1; raycaster.trigger() }
        }
    }

For the RayCaster, it defines a ray can be moved around the scene using key combinations.

Not the most earth shattering demo :).   But it does exercise all the features of the newly introduced nodes and how to do picking without using QObjectPicker.

Notes

• These nodes should not be used for rays which change constantly. In particular if the change rate is faster than the actual display rate, the results are undetermined and it will not be possible to correlate results to the actual value to the ray used to produce them.
• At this time, the screen based ray caster has no way of controlling which render surface to test, they are all tested, including offscreen surfaces. This will be remedied in 5.12.
• Picking settings are global and cannot be controlled per ray. So you can’t have one ray to pick lines and one to pick triangles for example. This will be changed at a later time.