Discover Bottlenecks on QNX How to Speed Up the Process

29.04.2021 Sven Erdem No comments

For many developers, QNX is the operating system of choice, when it comes to developing on embedded systems. However, the performance limits are exceeded quickly, especially if you’re working on low-end hardware. As a result, you’ll likely need to perform an investigation to find the bottlenecks that contribute to reaching these limits, in order to configure your application to fit the requirements of your system.

There are many reasons why an application might be slow. Any number of bottlenecks could be causing the delays. In our experience, file reads and writes have been relevant factors, due to bandwidth limitations.

To speed up the investigation process, KDAB created a script which lets us dig for specific system calls and provides us with visual information about their occurrences. According to our findings, on QNX, you can gather information about system calls with the command fsysinfo.

Record samples

Recording the output of fsysinfo to a file gives a database for our script. Specify the time period between adjacent samples, in milliseconds. Wait an appropriate amount of time, then exit fsysinfo by killing it (or hit CTRL+C when executed from a terminal).

#   Tip: you can clear all values before recording with following command
$   fsysinfo -Z DEVICE|FILESYSTEM

#   Record samples
$   fsysinfo -l PERIOD DEVICE|FILESYSTEM > OUTPUTFILE

For more information on the fsysinfo command, please see http://www.qnx.com/developers/docs/7.1/#com.qnx.doc.neutrino.utilities/topic/f/fsysinfo.html.

Plot samples

Then, a decision of what shall be plotted must be made. This is done with arguments to our script.

$   fsysinfo_plot.py [OPTION]... FILE -- KEY...

Here are a few examples:

The first example plots the amount of bytes read and written to the disk, using a logarithmic scale. It calculates the difference between two points in time (samples) and gives a visual representation of when and how many bytes have been read and written. Also the legend is moved to the top right corner. Note that the x-axis corresponds to the previously recorded samples and the specified preriod is the inverse of samples per second (1 / period).

$   fsysinfo_plot.py --diff --log --legendright fsysinfo_rwdata_log -- "DISK I/O.read.bytes" "DISK I/O.write.bytes"

The next example presents the occurrences of 4 system calls and relates the corresponding amount by stacking them on top of each other.

$   fsysinfo_plot.py --stackplot fsysinfo_rwdata_log -- SYSCALL.create SYSCALL.delete SYSCALL.open SYSCALL.stat

To find out more about how and what to plot, use the help argument:

$   fsysinfo_plot.py --help
this script plots a qnx fsysinfo log
usage
    fsysinfo_plot.py [OPTION]... FILE -- KEY...

    whereas the key determines what shall be printed
    the keys are formatted like so: "SYSCALL.rename"
example
    fsysinfo_plot.py --diff my_fsysinfo_log -- SYSCALL.open "DISK I/O.read" "DISK I/O.write"
options
    --stackplot     performs a stackplot instead of a usual one
    --diff          calculates the difference between 2 samples
    --avg N         averages between N samples (applied after diff)
    --log           logarithmic scale (be careful with negative values or when using diff)
    --legendright   display legend on the top right instead of top left corner
keys
    DISK I/O    read, r/a, write, io req, direct, bad, read.bytes, r/a.bytes, write.bytes, io req.bytes, direct.bytes
    CACHE       read, write, mfu, mru, ratio, read.bytes, write.bytes
    SYSCALL     open, create, delete, trunc, stat, namei, modes, owner, rename, devctl, sync, pause, change, timed, write, read, write.bytes, read.bytes
    NAMES       exist, enoent, misses, unsuit, stale, rate
    BMAP        hit, miss, rate
    VNODES      create, hit, rate, lock, recycl
    SLAB        map, unmap, active
    THREADS     create, destro, pool

recording a log is possible with `fsysinfo -l PERIOD DEVICE|FILESYSTEM > OUTPUTFILE`
whereas PERIOD is in milliseconds
please look into the fsysinfo manual for more information

Please make sure to have the following dependencies installed:

• python3
• matplotlib

You can find the script here: https://github.com/KDAB/qnxtools/blob/main/fsysinfo_plot.py