Efficient barcode scanning with QZXing Profiling Qt Zebras Crossing

24.04.2017 Milian Wolff 14 comments

QZXing is a very useful library: It provides an easy to use Qt integration API around the barcode scanning library ZXing (zebras crossing).

Because it is so easy to setup QZXing in a Qt application, we and most of our customers end up using it when they need to scan images for barcodes. There is, or rather was, a big problem though: When you want to analyze a live camera video stream, e.g. from a phone camera, the latency can easily be so large that it affects usability. The user needs to hold the camera very still on the barcode and often wait for multiple seconds until the barcode is detected.

Towards the end of last year, I decided to do something about this. At KDAB I have the freedom to spent 10% of my time on education projects. Optimizing and improving a commonly used library like QZXing also falls into this category.

The following outlines what I learned about using QZXing. I also talk a bit about the many optimizations I upstreamed to the project, which drastically improve its performance. Combined, these steps will lead to a much smoother user experience when scanning barcodes. If you have seen our Kurth KE3600 xDSL Multitester demo at Embedded World Nuremberg, you already got in direct contact with some of the fruits of this work.

Tip 1: Use a video filter

The old way to scan frames from a live video feed for barcodes was usually done in the following way:

• setup the camera to capture still images
• capture an image
• scan the captured image for barcodes
• capture the next image

While this works, it is prohibitively slow. If you are stuck with Qt 5.4 or older – tough luck, you cannot do anything about it. If you are using a more modern version though, i.e. Qt 5.5 or higher, you can leverage Qt Multimedia video filters for scanning the frames of a live camera feed. QZXing already provides you a helpful implementation of such a filter in the form of the QZXingFilter. The QZXingLive example shows it in action and is able to analyze far more frames than the naive old approach.

Using this new approach is actually dead easy: Instantiate a QZXingFilter object, then add it to the filter list of the VideoOutput that receives the camera video stream. Then you are basically done already, as the filter contains a QZXing decoder which will notify you when a barcode was scanned:

Camera {
    id: camera
}
VideoOutput {
    source: camera
    filters: [ zxingFilter ]
}
QZXingFilter {
    id: zxingFilter
    decoder {
        onTagFound: {
            console.log(tag);
        }
    }
}

Note that this technique will not scan every frame. Rather, it will scan frames in a background thread and ignores any frames that come in during this scan procedure. Once it has finished scanning a frame, it will pick up the next frame and so forth. For the user experience, it is thus still important to scan a single frame as quickly as possible, such that we can find the barcode easily. After all, the more frames you are able to scan per second, the higher the chances that you scan a frame with a readable version of the barcode on it.

Tip 2: Explicitly set the list of expected codecs

The fastest code is code that does not run. A quick way to speed the scan process up is to tell ZXing what kind of barcodes you expect. In all cases I have worked on so far, you usually expect one or two 1D barcode types and 2D QR codes:

QZXingFilter {
    decoder {
        enabledDecoders: QZXing.DecoderFormat_EAN_13 | QZXing.DecoderFormat_CODE_39 | QZXing.DecoderFormat_QR_CODE
    }
}

If you explicitly only enable those types, scanning of frames where no barcode can be found is going to be significantly faster. Otherwise, ZXing will hopelessly iterate linearly over all enabled codecs, which can accumulate a significant overhead.

Tip 3: Reduce the amount of frame data

I have added a new feature to the QZXingFilter – the capture rect. It allows you to limit the barcode scan to parts of a frame. The QZXingLive example shows this in action:

Only the area shaded in red will be scanned for barcodes. In this case, it is only 1/4 of the actual frame size, which means that ZXing also only has to inspect 1/4 of the pixels in the frame. This obviously makes the process much faster. On my desktop machine, I can scan 1920×1080 frames at about 42fps with the capture rect. If I use the full frame instead, the frame rate drops to ca. 26fps.

Setting up the capture rect can be somewhat tricky, as you need to use the camera source coordinate system. Use the mapping functions of VideoOutput for this purpose, but don’t forget to manually setup the bindings to thesourceRect and contentRect, otherwise the captureRect may not get updated properly:

QZXingFilter {
    captureRect: {
        // setup bindings
        videoOutput.contentRect;
        videoOutput.sourceRect;
        // only scan the central quarter of the area for a barcode
        return videoOutput.mapRectToSource(videoOutput.mapNormalizedRectToItem(Qt.rect(
        0.25, 0.25, 0.5, 0.5
    )));
}

Tip 4: Use a good camera

Scanning barcodes, especially 1D barcodes, can become quite hard when your camera only produces blurry images of low quality. The better your camera, the easier it will be to find barcodes. Most cheap consumer webcams do not support autofocus for example, which negatively influences the detection speed. Also, do not forget to enable autofocus on your camera!

Camera {
    focus {
        focusMode: CameraFocus.FocusContinuous
        focusPointMode: CameraFocus.FocusPointAuto
    }
}

I have often observed that the overall time to scan a barcode is faster on my Android phone compared to my beefy desktop machine. When you look at the time to scan a single frame, the opposite is true – the desktop machine can scan roughly six times as many frames per second than my phone. But once you inspect the image quality of the two respective cameras, the reason is quite obvious – the desktop tries to find barcodes in blurry unfocused screens with bad contrast. This often means that the scan algorithms fail, especially when you deal with 1D barcodes. The phone images on the other hand are much sharper, and have good contrast between the individual bars.

Tip 5: Optimize and improve QZXing

This tip is more a collection of things that I improved upstream. Thankfully QZXing is an open source project, which means everyone can improve it. I think there are more areas that could be improved, so consider this also as a motiviation to start hacking on QZXing to improve it for everyone that is using it.

I started off by profiling QZXing with Linux perf. It directly pointed me at many hotspots in QZXingLive which resulted in low per-frame performance. The reason was mostly due to repeated copies of the frame data. Initially the following happened when we scanned a frame:

• frame data is copied to be able to process it in a background thread
• data is converted into a QImage compatible format, often incurring undesired copies to the original frame data
• the image is scaled to not exceed the maximum size of 999×999, which not only copies the data, it also reduces the quality of the image that will be scanned
• the scaled image data is copied into a ArrayRef<byte> matrix class, such that ZXing can analyze its contents
• this matrix is copied once more, for good measure
• when no barcode was found, the next frame mirrors the image data (which again copies) before handing it off to ZXing. This is done in order to scan cameras also when the input image is rotated, but very negatively influences latency by up to three frames, since the image is first mirrored horizontally, then vertically, and then in both direction

You can imagine that this is not optimal. Actually, this situation was so bad that I can only explain it in one way: This code was never run through a real profiler.

Profiling C++ code can be extremely hard. But when you know what tools to use and how to interpret their results, it is a very rewarding job. And when you do it before anyone else, chances are high that you will find many code paths that are easy to fix! If you want to learn more about profiling, consider attending our Debugging and Profiling for Qt Developers training.

I used a combination of Linux perf and heaptrack and optimized the QZXingLive code in a series of patches that got upstreamed in January. With a recent version of QZXingLive, performance is much better and we have copy the frame data less often:

• the data is still copied to be able to process it in the background thread
• the data is directly and efficiently converted into a grayscale QImage
• this image is then copied into the ArrayRef<byte> matrix and analyzed in its original resolution
• if no barcode was found, and only if the tryHarder flag is set to true on the decoder, we flip the image directly and try again without waiting for the next frame

With some more effort, we should be able to get rid of the QImage in the middle, removing another copy of the data. I did not go down that route, because QZXingLive is now at a level that it performs well enough for my needs even on slower mobile devices.

While writing this report, I realized that I back then concentrated only on the performance on an iOS device that supplied RGB frames. On my desktop machine, I get YUV420p frames. The conversion step into a grayscale image for these frames did not yet take the capture rect into account, leading to unnecessary copying. Fixing this part improves QZxingLive frame decoding time from ~25ms down to ~16ms on my machine.

Summary

I hope you learned how to use QZXing effectively in your project. Remember to use the QZXingFilter class for efficient decoding of a live video stream. And also remember to profile your code. Copying data around needlessly, esp. on the heap, can be an extremely costly operation and should always be prevented if at all possible.