One of the focus areas for 4.5 is performance. To that end, we decided to add a couple of benchmarking extentions to our beloved QTestLib.
There is exactly one new API, the QBENCHMARK macro:
QString str1 = "foo";
QString str2 = "foo";
QBENCHMARK {
str1.localeAwareCompare(str2);
}
(Full example is available at $QTDIR/examples/qtestlib/tutorial5.)
Under the hood, the benchmark macro starts the timing measurements, and then repeats the code under test enough times to get a measurable result. Running this test produces the following output:
RESULT : TestBenchmark::simple():
0.00094 msec per iteration (total: 31, iterations: 32768)
The default measuring back-end is based on QTime, and in this case we needed 32768 iterations in order to get a result. Repeating the code this many times is not ideal, since it takes a lot of time and might give inaccurate results due to caching effects. On select platforms we therefore have CPU tick counters available, which can be invoked by specifying “-tickcounter” on the command line:
./mytest -tickcounter
RESULT : TestBenchmark::simple():
3,206 ticks per iteration (total: 3206, iterations: 1)
There is also callgrind-based measurer available on linux/x86 systems, which produces 100% reliable instruction counts, but does not take things like I/O waits into account.
Benchmarking a single API is fun, but to be useful we have to compare it against something. Let’s add some data:
void TestBenchmark::series_data()
{
QTest::addColumn(”useLocaleCompare”);
QTest::addColumn(”stringSize”);
for (int i = 1; i < 10000; i += 2000) {
QByteArray size = QByteArray::number(i);
QTest::newRow(("locale aware compare--" + size).constData()) << true << i;
QTest::newRow(("standard compare--" + size).constData()) << false << i;
}
}
We’re benchmarking QString::localeAwareCompare() vs QString::operator==(), for different string sizes. The test function now looks like this:
void TestBenchmark::series()
{
QFETCH(bool, useLocaleCompare);
QFETCH(int, stringSize);
QString str1 = QString().fill('A', stringSize);
QString str2 = QString().fill('A', stringSize);
int result; // assign to result to prevent the optimizer from optimizing the compare away.
if (useLocaleCompare) {
QBENCHMARK {
result = str1.localeAwareCompare(str2);
}
} else {
QBENCHMARK {
result = (str1 == str2);
}
}
}
This gives lots of results:
RESULT : TestBenchmark::series():"locale aware compare--1":
0.00067 msec per iteration (total: 22, iterations: 32768)
RESULT : TestBenchmark::series():"standard compare--1":
0.000019 msec per iteration (total: 41, iterations: 2097152)
RESULT : TestBenchmark::series():"locale aware compare--1001":
0.016 msec per iteration (total: 33, iterations: 2048)
RESULT : TestBenchmark::series():"standard compare--1001":
0.00056 msec per iteration (total: 37, iterations: 65536)
RESULT : TestBenchmark::series():"locale aware compare--2001":
0.031 msec per iteration (total: 32, iterations: 1024)
...
It’s hard to make heads or tails of all this, so we decided to create a few tools to help us out. Unfortunately these tools didn’t make the cut for 4.5 (”Shipping Software Means Prioritizing”), but instead we’re making them available here on labs, with bugs and all. (run “git clone http://labs.trolltech.com/gitweb?p=qtestlib-tools” to download.)
So, to generate the nice graph pointing upwards, run the test in xml mode and then feed that xml to the generatereport tool.
./mytest -xml > results.xml generatereport results.xml
This was an underperforming lame excuse for an automatic door. And so, he fell backwards, dropped his two bags, and whammed into the ground, and of course, he too, cursed out loud, and angry.
