I’d like to share with the community a project I’m working on, while it’s still in its development phase (isn’t that what labs is for? :))
The goal of the project is to get CSS3 animations to a reasonable FPS performance, mainly on embedded hardware where it’s a pain.

See http://gitorious.org/~noamr/webkit/noamrs-webkit/commits/accel

The idea is to implement webkit’s GraphicsLayer concept, which allows platform-specific implementations of CSS transform and opacity animations, using the graphics-view and the Qt animation framework as a backend. This would only work for QGraphicsWebView and not for QWebView, as rendering a separate QGraphicsScene inside QWebView would probably not give us much performance benefit.

Preliminary results are very promising - The leaves demo, for example, runs 4 times faster on Maemo Fremantle than it does without the acceleration, and it looks graphically accurate.

The reason this gives us a performance benefit is mainly because of graphics-item caching: when a CSS animation occurs inside webkit, the item that’s being animated has to go through a re-layout and re-draw every so often, while with the accelerated approach we draw it once into a QPixmap (QGraphicsView takes care of that) and then it’s just a series of fast and furious pixmap blts. The hardware acceleration becomes relevant when the images are big and the blt itself becomes a bottleneck.

This project is not ready to go upstream, as it supports many delicate use-cases that need to be tested. But if you’re interested in participating (or to just comment!), this has so far been a fun project to hack on.

Instructions:

1. Get the Git repo from git://gitorious.org/~noamr/webkit/noamrs-webkit.git, branch accel
2. Build or get a relatively new version of Qt, possibly without building QtWebkit
3. Build Webkit from the downloaded Git repo:
   export QTDIR=[my-qt-4.6-root]
export PATH=$QTDIR/bin:$PATH
./WebKitTools/Scripts/build-webkit --qt
4. Run ./WebKitBuild/Release/bin/QGVLauncher --accel: This will enable the necessary web-setting for composite-layer acceleration. You can also create a small QGraphicsWebView example yourself, as long as you enable the new settings flag: QWebSettings::AcceleratedCompositingEnabled
5. Load a website with CSS transform/opacity animations: like this one or this one.
6. Hack. Most of the code is in WebCore/platform/graphics/qt/GraphicsLayerQt.cpp, or you could just search for the term USE(ACCELERATED_COMPOSITING)
7. Send merge requests through Gitorious or comments on Bugzilla

No’am

Previous posts in this topic:

• Whats Hot and whats Not
• An Overview
• The Raster Engine
• The OpenVG Engine
• The OpenGL Engine

So, its time for my next post. Todays topic is how convenience relates to performance, specifically in the context of QGraphicsView. My goal is to illustrate that the way to achieve fast graphics is to pack your QPainter draw calls as tightly together as possible. The more stuff that happens in the middle, the slower it gets.

To illustrate this, I’ve implemented a virtual keyboard. Granted, its not a very common layout nor is it usable, but the rendering is the point here, not the functionality. The full source code is here and it looks like this:

I’ve implemented the keyboard using three different approaches. One using proxy widgets, one using graphics items and one where the entire view is one graphics item. In addition to that, I added a number of options to tweak various properties, such as whether or not the text is drawn. I measured this on an N900 rather than a desktop because the difference becomes more profound on a small device. On the desktop it is easy to be fooled because most things complete in a matter of micro seconds anyway. It is only when the entire application comes together one notices that things are not as smooth as in the prototype, but too much work has been invested into the current design that one loses out on the super-slick feeling application.

QGraphicsProxyWidget

Since we’re implementing a series of clickable buttons, a natural and convenient starting point is to use an existing button class, such as the QPushButton. It already implements the logic for mouse/keyboard interaction and has signals for clicking and all sorts of other useful functionality. To get widgets into QGraphicsView, we use a QGraphicsProxyWidget. To make the test “fair”, I actually use a plain QWidget which just paints a pixmap and a draws a text. Had I gone through the styling API, these numbers would have been even worse.

Milliseconds spent per frame including blit to screen when using QGraphicsProxyWidgets. Low is better!

If we look at the plain “-proxywidgets” run, the fastest engine was the raster engine, running at 26ms per frame. If I wanted to slide this keyboard onto screen, I have 16ms available if I want it running at 60 FPS and 33ms available if I want to do it at 30 FPS. When each frame takes 26ms, I can barely do 30, but with only a little bit of slack, so if another process is soaking up CPU time, that number is also a bit difficult to reach. So, not very good. (BTW, the exact numbers in the graphs are listed as a comment in the top of the .cpp file I linked above).

The first thing I noticed with this approach was that the each button now had a gray background. This is of course the widget background. A QWidget embedded in QGraphicsView will be treated as a top-level and will therefore draw its background. I added an option “-no-widget-background” which sets the Qt::WA_NoBackground on the widget. This brings the rendering speed with raster down to 22ms. 4ms saved per frame, just by setting a flag, not too bad, but still pretty far from being awsome.

I’ve mentioned before that text drawing is not as fast as we would like it, so just to compare how it looks without text, I added a “-no-text” option to the test. This brings the raster results down to 13ms. That is pretty nice and below the 16ms threshold required to achieve 60 FPS, but only with a small margin. And I’m not drawing any text! Before I give up with this approach, I’ll enable item caching. By setting ItemCoordinateCache on each button, I cache both the background pixmap and the text in one single pixmap. This brings the raster results down to 8.5ms, and its starting to look acceptable. But at a very high memory cost… In my original usecase I had one shared pixmap for all the button backgrounds, but now I have one per button.

You may notice that there was a vast difference between item caching and the proxy widget drawing the pixmap. One thing that adds to the proxy widget cost is that the QPainter is recreated and initialized for each button in the buttons paint event. Also, as I mentioned in my previous post, An Overview, you may remember that I said that each widget has a system clip and that there is an overhead involved with calling the paintEvent. For items in QGraphicsView, there is already a painter, and I don’t need a clip, nor do I need any of the other stuff that goes on behind the scenes there. When we enable item coordinate caching, we don’t leave graphics view world and we don’t enter the widget world. This crossing is expensive, so by not going into the widget world, we save a lot.

So, if there is a lesson to be learned it is that QGraphicsProxyWidget should be used with extreme caution. If you really need it, use very few of them.

QGraphicsWidget

If proxy widgets are too slow to be usable in this scenario, then the next best thing is to use a QGraphicsWidget. This is a subclass of both QObject and QGraphicsItem, which gives me signals, slots and properties, but its not a QWidget and therefore still fairly lightweight. The numbers are as follows:

Milliseconds spent per frame including blit to screen when using QGraphicsWidgets. Lower is better!

Compared to the proxy widgets approach we’re starting out quite a bit better, with raster at 13 ms per frame, OpenGL at 20ms and X11 at 22ms. Below this line is a new line: “-no-indexing -optimize-flags”. QGraphicsView will by default put all the items in a view into a BSP tree for fast lookup, this is beneficial when the scene contains many items and you often need to find items that intersect with a small portion of the scene. In the testcase we’re always doing a full update, so there is no benefit from the index, so it can be disabled by calling scene->setItemIndexMethod(QGraphicsScene::NoIndex). Having a BSP is the default behaviour because graphics view was initially intended to be a static scene for many items. The most common usecase today is a few (a few hundred at max) items which tend to move a lot. For this reason, it is always a good idea to try to disable the BSP and see if it makes a difference in performance. If it helps, then leave it off.

I also know that the items play nice, meaning that they don’t change the clip, translate the painter, change the composition mode or modify any other state that would propagate to other items. This means I can safely set the DontSavePainterState optimization flag. Actually, based on an old habit, I set all possible optimization flags. I only consider unsetting them if my drawing code starts to look weird, at which point I would rather fix the drawing code and keep the flags set. By disabling indexing and enabling optimization shaves off 2ms per frame in for all rendering backends, so that is definitely worth it.

If I don’t do text, the performance is about twice as fast. Again we see that text drawing is a huge cost. We’re working on an API to fix this and we’ll have more information for you when we do. You may notice that enabling item caching drops the performance a bit compared to the “-no-text” case. There isn’t much overhead inside QGraphcisView for this path. A likely reason for the decrease is that reading from multiple memory sources (multiple pixmaps) results in a lot of cache misses, compared to the straight approach which draws the same pixmap over and over.

ButtonView Item

In my previous post I briefly mentioned that there is a slight overhead involved with the use of a QGraphicsItem too. Prior to calling the paint function, the painter is transformed to the coordinate system of the item and the painter state is saved. If the item draws a big polygon, this setup cost can be ignored, but when drawing just a pixmap and a few pixels of text, then it may be worth considering. In the spirit of “The more direct the painting code is, the faster it gets”, I implemented the keyboard as a single item. The numbers are as follows:

Milliseconds per frame including blit to screen when using a single item. Lower is better!

Raster is now down to 10ms, which is 1ms better than the QGraphicsWidget approach when all optimizations were enabled, so even though graphics items are cheaper than widgets, they still cost a bit. The keyboard is now rendered in a tight loop, and the major difference in performance here is caused by the fact that items in the scene have a transform associated with them. Prior to calling paint() a transform is set to match the painter to the items local coordinate system. This causes a state change in the paint engine. For each button we’re drawing a 32×32 pixmap which means alpha blending 1024 pixels, followed by doing text layout and drawing a single character. Even then do we save about 10% time by not having a QPainter::translate() in the midst, so bear that in mind. By enabling the optimization flags and disabling the index, raster drops a bit more, so having those are still a good idea.

You may have noticed that there is one dataset that is named “cheat” for OpenGL. I was reluctant to include this, because its using a private API that is not, and I really mean NOT, subject to binary compatibility rules. You cannot call this from your application. We’re going to add a public API for this in the future, hopefully 4.7, so until its there, wait. In the interest of showing what we are thinking internally, I thought I would show it.

OpenGL is really great for accelerating graphics, but its way of working does not map optimally to how Qt works. GL is really good at taking a few large datasets of triangles and rendering them, but its not so good at drawing loads of small things. Small things like button backgrounds, icons, single text items, etc. However, all the buttons backgrounds are the same pixmaps, so what if I could tell QPainter to draw the same pixmap in multiple places at once? In GL this would correspond to setting up a texture and one vertex and texture coordinate array and drawing some 40 pixmaps in one go. This fits much better with how GL is made to work. The result is that drawing the buttons drop from 5.2ms to 3.9ms, so another piece of juice squeezed out. Naturally, the more times the pixmap is drawn and the smaller the pixmap gets, the more benefit you get from batching commands like this.

There is a second option to OpenGL for the button view case, which is the “-ordered”. This was done after Tom brought to my attention that the testcase would do a shader program update for each painter call. In the default buttonview implementation we do:

                    for (int i=0; i < m_rects.size(); ++i) {
                        p->drawPixmap(m_rects.at(i), *theButtonPixmap);
                        p->drawText(m_rects.at(i), Qt::AlignCenter, m_texts.at(i));
                    }

Because pixmaps use one shader pipeline and text drawing uses another, the pipeline needs to be switched and reset all the time, which renders at 16m per frame. To see if it makes a difference, I added a second alternative rendering, “-ordered”, where I do all the pixmaps first, then all the text:

                    for (int i=0; i < m_rects.size(); ++i)
                        p->drawPixmap(m_rects.at(i), *theButtonPixmap);
                    for (int i=0; i&lt;m_rects.size(); ++i)
                        p->drawText(m_rects.at(i), Qt::AlignCenter, m_texts.at(i));

This prevents the shader pipeline updates and bring the rendering time per frame down to 13ms, so definitely worth it.

Summing Up

Milliseconds per frame including blit to screen for proxy widgets, graphics widgets and a single widget. Lower is better!

OpenGL comes out rather bad in this testcase, which I was a bit disappointed to see, but it did send Tom into an optimization frenzy, so we’re hoping to remove some of the constant overhead. It should also be said that when using the OpenGL graphics system, we enable multisampling by default, which increases rendering time on the N900 by around 30%. A plain QGLWidget would thus perform slightly better. Another aspect to OpenGL is that it uses a dedicated low-power chip, so even though it for this particular usecase runs at half the speed, it also uses a lot less battery, so it may still be the right choice. OpenGL will also scale significantly better than raster and X11 as the pixmaps get bigger or if the content of the button is slightly more advanced, say like a horizontal gradient.

The best numbers are definitely in the button view case, where all the content is rendered as one item, which is what I wanted to highlight with this blog. The button view item also opens up for other optimizations such as batching. We don’t have that many batching functions in QPainter today, its only drawRects(), drawLines() and drawPoints(), but we’re considering to add more, we are just not sure on how the API’s would look yet.

The bottom line is still that how Qt is used defines how well it performs. On one hand there may be an easy and convenient way to get the job done which performs quite sub-optimally. On the other hand there may be a more involved implementation which performs very well. I’m not trying to suggest that you do one or the other, there are a lot of good reasons for picking either one. But I hope that I’ve illustrated that some features come at a cost and that this is kept in mind along with what the target is when designs evaluated and chosen.

I’ll round off with a question. If you were to implement a particle effect when you press a button, which approach would you choose, having seen the numbers above?

Some time ago I wrote about our ideas for a new Qt widget set for QGraphicsView.
Up till now we have all been busy working on finishing Qt 4.6, so not much has happened since then.
But, it is now time to quote Hubert J. Farnsworth: “Good news, everyone!”. The Next Generation Qt Widgets project is of the ground!

Open Is The New Black

With Qt 4.6 released, we can now dedicate our time to Next Generation Qt Widgets. But we are not the only ones that will be working on this project. In addition to Qt developers, we will also have developers from INdT. And you are hereby invited to join as well!

liftoff by popofatticus on flickr

Jump Into The Fray

First, you want to clone the repository. It is also recommended that you browse the Qt wiki to get familiar with coding conventions and the contribution model. You may also want to read up on the rules. You should now be ready to join the discussions on IRC or on our mailing-list.
We are looking forward to seeing you there! Happy hacking!

For a long time the standard layouts shipped with Qt’s have been built around the concepts of linear and grid layouts. In the QLayout regime they are represented as QBoxLayout and QGridLayout. In the QGraphicsLayout regime they are represented as QGraphicsLinearLayout and QGraphicsGridLayout. We therefore started early this year with research on a new layout. The efforts of that culiminated into the QGraphicsAnchorLayout.

Motivation

We have quite often seen that the existing layouts in Qt can not always do what you want with just one layout. Take for instance Qt Linguist’s “Find dialog”.
The dialog is made up of four box layouts and one grid layout for the checkboxes inside the groupbox.

(note that Qt Designer does not illustrate the layout inside the groupbox or the toplevel layout)

In addition, in order to achieve this simple arrangement it uses three levels of nested layouts.
Luckilly, if you design it in Qt Designer it is quite easy, but you might not get it right at first attempt. Of course, doing it programmatically makes it even worse. The result are probably far from how a UI designer would specify the arrangement. We think that an anchor layout will narrow that gap in a lot of cases.

Concept

The QGraphicsAnchorLayout is a very flexible layout, yet the concept is simple. By creating an anchor between two edges of two layout items you are giving the anchor layout a constraint to fulfill. In the simplest case you are just saying that the distance between the two layout edges should be a fixed distance. You can do this with the left, right, top, bottom and the “center edges” of any layout item. You can also have an edge that has several anchors connected to it. This is not possible with linear layout and it is partly possible with a grid layout. This allows you to build up the structure the layout should have. The layout will then try to fulfill the constraints you have given it by potentially growing or shrinking the layout items. You can also configure each anchor to grow or shrink if that is preferred.

The API for configuring the layout structure reflects this concept. The addAnchor() function is all that you need to configure the structure of the layout, but we added some convenience functions for anchoring corners (surprise, addCornerAnchors()) and for fitting the size of an item to another (addAncors()). Thus, the following code will ensure that the left edge of item findEdit is always connected to the right edge of the item findLabel. In this case it means that there will typically be a bit spacing between the items, and the spacing will always be the same.

   anchorLayout->addAnchor(findLabel, Qt::RightAnchor, findEdit, Qt::LeftAnchor);

You can also anchor an item to a layout edge. If the edge of an item is anchored directly or indirectly to a layout edge, it’s geometry will react to changes in the layout geometry:

   anchorLayout->addAnchor(anchorLayout, Qt::LeftAnchor, findLabel, Qt::LeftAnchor);
   anchorLayout->addAnchor(findLabel, Qt::RightAnchor, findEdit, Qt::LeftAnchor);

Of course, you can also anchor items the same way in the vertical dimension.

Now, with this brief introduction we can go back to Qt Linguist’s “Find Dialog” example as I gave above.
It turns out we can make the same arrangement with just one anchor layout.

This is how the code could look like:

    QGraphicsAnchorLayout *l = new QGraphicsAnchorLayout;
    QGraphicsWidget *w = new QGraphicsWidget(0, Qt::Window);
    w->setLayout(l);

    // label, line edit and "Find Next" button
    l->addCornerAnchors(l, Qt::TopLeftCorner, findLabel, Qt::TopLeftCorner);
    l->addCornerAnchors(findLabel, Qt::TopRightCorner, findEdit, Qt::TopLeftCorner);
    l->addCornerAnchors(findEdit, Qt::TopRightCorner, findButton, Qt::TopLeftCorner);
    l->addCornerAnchors(findButton, Qt::TopRightCorner, l, Qt::TopRightCorner);
    // group box
    l->addCornerAnchors(l, Qt::BottomLeftCorner, groupBox, Qt::BottomLeftCorner);
    l->addCornerAnchors(groupBox, Qt::TopRightCorner, findEdit, Qt::BottomRightCorner);
    // cancel button
    l->addCornerAnchors(findButton, Qt::BottomLeftCorner, cancelButton, Qt::TopLeftCorner);
    QGraphicsAnchor *anchor = l->addAnchor(cancelButton, Qt::AnchorBottom, l, Qt::AnchorBottom);
    anchor->setSizePolicy(QSizePolicy::Minimum);
    l->addAnchor(cancelButton, Qt::AnchorRight, l, Qt::AnchorRight);

    // Done!
    scene.addItem(w);

Note, for simplicity the code does not populate the group box with items, since that is easier done with a grid layout. Thus, the anchor layout reduced 4 layouts into one.

There’s a lot of interesting things I could mention here, but I would then be just repeating what’s already written in the documentation, so if you are interested I suggest you take a look at that.

Conclusion

The anchor layout has some attractive properties. For some setups it provides a more intuitive API that is closer to how the UI designer would specify it. It is also very flexible: You can construct free-form graphs of anchors where one edge might be anchored to several other edges. You can modify size policy of each anchor, which means there is little use for a spacer item. You can also modify the spacing to be both a positive and negative value.

Of course, the anchor layout is not the silver bullet and in a lot of cases you are better off with just using a linear or a grid layout. The anchor layout is a flexible layout, and there is a price you have to pay for that: For some arrangements it can be very verbose compared to a linear layout, since you don’t have to anchor each item together (the “anchoring” is implicit by the order of the items).

Read the rest of this entry »

Introduction

Texas Instruments has a wiki which documents what is required to bring Qt
up on the Beagle board with full OpenGL ES (1/2) support:

http://www.tiexpressdsp.com/index.php/Building_Qt

and I would like to thank one of their engineers, Varun, for his quick turn
around times in addressing any questions I raised.

This blog entry is intended to serve a similar purpose, but is more verbose regarding
Qt considerations and the initial beagle board bring up. It attempts to serve
as a comprehensive independent source of information on getting Qt built
for the Beagle board with full OpenGL ES 2 support.

These instructions are intended for use with Qt 4.6 (and beyond), so grab
the release candidate or check Qt 4.6 out from the public git repository prior
to proceeding.

You can choose to use either Qt/Embedded or Qt/X11, both can
be successfully integrated with the Beagle board’s SGX GPU and the only
point of divergence in these instructions will be at (Qt) configure time
and the client side system (run time) configuration. Both implementations
offer window management, via QWS and X11 respectively, and operate at
around 27fps and 22fps respectively when running our hellogl_es2 example.
(16bit color depth at 1280×720)

I personally deploy Ångström on my Beagle board, it handles a large amount
of the logistics surrounding cross compilation and is generally very
agreeable, and these instructions are therefore going to be bolted to
Ångström for completeness. Feel free to establish an environment capable of
showing the OpenGL ES examples TI provide, then following the Qt level
considerations (Configuring Qt) accordingly.

For those holding a dormant Beagle board who are open to the author’s
distribution preferences:

Building the Ångström rootfs

Open Embedded is manifested in a git repository: in this posting we are
working within origin/stable/2009. Please follow the instructions give
here, they are comprehensive and got me completely off the ground.

http://www.angstrom-distribution.org/building-angstrom

These instructions end in you running:

bitbake base-image ; bitbake console-image ; bitbake x11-image

which actually builds an X11 angstrom image for your Beagle board. Please
note, you will need to build the X11-image if you want to build and deploy
the SGX packages (we will do this in the next section) via Ångström as opkg considers
X11 to be a required dependency of libgles-omap3_3.00.00.09. This is due
to one of the encapsulated windowing system libraries being X11 centric:

libpvrPVR2D_X11WSEGL.so

Regardless of the indicated X11 dependency, this package will bestow the required
kernel module on you for general OpenGL ES usage (console or X11). We will be
building our own QWS centric (libpvrPVR2D_X11WSEGL.so equivalent) library
behind the scenes for QWS in the Qt/Embedded instructions given later.

Ångström SGX integration

You now need to integrate the SGX drivers on your Ångström system.

You need to get your paws on:

OMAP35x_Graphics_SDK_setuplinux_3_00_00_09.bin

with the following MD5 checksum:

e15147ad76ddbe7c5aec682f5455b774

Getting this involves following the above link and going through the required registration/request process.
Once you have this file, you drop it in:

$OETREE/openembedded/recipes/powervr-drivers/libgles-omap3

and then run:

bitbake libgles-omap3-3.00.00.09

which generates the following packages:

libgles-omap3_3.00.00.09-r1.1_armv7a.ipk
libgles-omap3-dbg_3.00.00.09-r1.1_armv7a.ipk
libgles-omap3-demos_3.00.00.09-r1.1_armv7a.ipk
libgles-omap3-dev_3.00.00.09-r1.1_armv7a.ipk
libgles-omap3-tests_3.00.00.09-r1.1_armv7a.ipk

Deploy the x11-image to an sd-card, and copy these packages to the sd-card
for deployment on the target. If your beagle board does not have internet
access you will probably also require:

*  devmem2
*  libx11-6 (Only if you insisted on using a console build!)

as opkg will not be able to automatically install the required dependencies
from its repositories and you would hit the following error at deployment:

———————————————-
root@beagleboard:/opt/deploy# opkg install ./libgles-omap3_3.00.00.09-r1.1_armv7 a.ipk
Installing libgles-omap3 (3.00.00.09-r1.1) to root…
libgles-omap3: unsatisfied recommendation for libgles-omap3-tests
Collected errors:
* ERROR: Cannot satisfy the following dependencies for libgles-omap3:
*  devmem2 *  libx11-6 (>= 1.1.5) *
———————————————-

Once you have installed all the above packages, please reboot the board.

Your bootargs in U-Boot should look something like:

console=ttyS0,115200n8=noinitrd ip=dhcp rw root=/dev/mmcblk0p2 omapfb.mode=dvi:1280×720MR-16@60

assuming you want to output via DVI and are running a similar kernel
version (2.6.29-omap1 on my beagle) which accepts the same kernel
arguments indicated in the bootargs variable above.

Please note that we are specifying a 16 bit color depth which is intentional
and discussed in the “color depth considerations” section in the appendix

Please run the powervr demos (under X11) to establish that your drivers are
successfully installed and usable.

Configuring Qt

In order to build Qt now, all that is required for each target is an
appropriate mkspec:

For Qt/X11

You would fork your mkspec off the linux-g++ mkspec, the resulting mkspec’s
qmake.conf would resemble:

==================================================================
………….
include(../common/linux.conf)

# modifications to g++.conf
# These release optimization flags are TI supplied
# and a little more aggressive than Qt standard (gentoo types rejoice!)
QMAKE_CFLAGS_RELEASE     = -O3 -march=armv7-a -mtune=cortex-a8 -mfpu=neon -mfloat-abi=softfp
QMAKE_CXXFLAGS_RELEASE     = $$QMAKE_CFLAGS_RELEASE

QMAKE_CC         = $FULLY_QUALIFIED_COMPILER_PREFIX-gcc
QMAKE_CXX         = $FULLY_QUALIFIED_COMPILER_PREFIX-g++
QMAKE_LINK         = $FULLY_QUALIFIED_COMPILER_PREFIX-g++
QMAKE_LINK_SHLIB     = $FULLY_QUALIFIED_COMPILER_PREFIX-g++

# modifications to linux.conf
QMAKE_LIBS_EGL         = -lEGL -lIMGegl -lsrv_um
QMAKE_LIBS_OPENGL_QT     = -lEGL -lGLESv2 -lGLES_CM -lIMGegl -lsrv_um
QMAKE_LIBS_OPENVG     = -lEGL -lGLESv2 -lGLES_CM -lIMGegl -lsrv_um -lOpenVG -lOpenVGU

QMAKE_INCDIR         = $TARGET_STAGING_PATH/usr/include
QMAKE_LIBDIR         = $TARGET_STAGING_PATH/usr/lib

QMAKE_AR         = $FULLY_QUALIFIED_COMPILER_PREFIX-ar cqs
QMAKE_OBJCOPY         = $FULLY_QUALIFIED_COMPILER_PREFIX-objcopy
QMAKE_STRIP         = $FULLY_QUALIFIED_COMPILER_PREFIX-strip

load(qt_config)
==================================================================

and you would configure Qt with:

configure -arch arm -xplatform linux-omap3-g++ -opengl es2 -openvg

all that remains is to adjust /etc/powervr.ini on the target to be:

[default]
WindowSystem=libpvrPVR2D_FLIPWSEGL.so

Now compile an example, eg:

./examples/opengl/hellogl_es2

deploy it and Qt to the target and enjoy.

For Qt/Embedded

Since we don’t have the X11 abstraction, we have to interface with the
underlying hardware/interfaces with Qt/Embedded’s gfx abstraction layer. We
are going to be making some heavy use of the powervr driver resident under:

$QTSRCTREE/src/plugins/gfxdrivers/powervr

there is a README file in the powervr directory that is definitely
recommend reading, and lends some serious insight into our powervr driver
and Qt/Embedded in general. The same driver is used for MBX/SGX targets and
hence sees a fair amount of usage on a variety of target devices.

You would fork your mkspec off the qws/linux-arm-g++ mkspec, the resulting mkspec’s
qmake.conf would resemble:

==================================================================
…………….
include(../../common/qws.conf)

# modifications to g++.conf
QMAKE_CFLAGS_RELEASE     = -O3 -march=armv7-a -mtune=cortex-a8 -mfpu=neon -mfloat-abi=softfp
QMAKE_CXXFLAGS_RELEASE     = $$QMAKE_CFLAGS_RELEASE

QMAKE_CC         = $FULLY_QUALIFIED_COMPILER_PREFIX-gcc
QMAKE_CXX         = $FULLY_QUALIFIED_COMPILER_PREFIX-g++
QMAKE_LINK         = $FULLY_QUALIFIED_COMPILER_PREFIX-g++
QMAKE_LINK_SHLIB     = $FULLY_QUALIFIED_COMPILER_PREFIX-g++

# modifications to linux.conf
QMAKE_INCDIR         = $TARGET_STAGING_PATH/usr/include
QMAKE_LIBDIR         = $TARGET_STAGING_PATH/usr/lib

QMAKE_LIBS_EGL         = -lEGL -lIMGegl -lsrv_um
QMAKE_LIBS_OPENGL_QT     = -lEGL -lGLESv2 -lGLES_CM -lIMGegl -lsrv_um
QMAKE_LIBS_OPENVG     = -lEGL -lGLESv2 -lGLES_CM -lIMGegl -lsrv_um -lOpenVG -lOpenVGU

QMAKE_AR         = $FULLY_QUALIFIED_COMPILER_PREFIX-ar cqs
QMAKE_OBJCOPY         = $FULLY_QUALIFIED_COMPILER_PREFIX-objcopy
QMAKE_STRIP         = $FULLY_QUALIFIED_COMPILER_PREFIX-strip

#These defines are documented in the powervr README, please read it
DEFINES += QT_QWS_CLIENTBLIT QT_NO_QWS_CURSOR

load(qt_config)
==================================================================

and you would configure Qt with:

/opt/dev/source/qt-beagle-4.6/configure -embedded arm -little-endian -xplatform qws/linux-omap3-g++ -opengl es2 -openvg -plugin-gfx-powervr

all that remains is to adjust /etc/powervr.ini on the target to be:

[default]
WindowSystem=libpvrQWSWSEGL.so

Now compile an example, eg:

./examples/opengl/hellogl_es2

deploy it and Qt to your board, and after shutting down X, run the example with
the following arguments:

./hellogl_es2 -qws -display powervr

-qws - starts the application as the QWS server with exclusive access to the
system hardware which manages all subsequent Qt “client” applications

-display powervr - indicates that Qt should use the powervr driver we
compiled earlier

Summary

I hope that this posting encourages people to go forward and experiment
with a fully accelerated Qt 4.6 on the beagle board. Offloading the
painting work onto the GPU drastically reduces the load on the CPU and
broadens the range of applications which can feasibly be run on this
broadly available (cheap!) embedded hardware. The Beagle board has
really nice hardware, and it would be infinitely useful for us to have external
people using our powervr driver and getting it as broadly used/refined as
possible.

Appendix

Additional Benefits to OpenGL ES acceleration

If you take any Qt Graphics View based example and set a QGLWidget as its
viewport, a large amount of work will be offloaded on the GPU leaving your
CPU free to frolic. To put this in perspective, a modified version of:

./examples/animation/animatedtiles

which continually transitions runs smoothly at 720p on the beagle board
when using software, but consumes 100% CPU time according to top (99.3% to
be fair). It is therefore CPU bound and you are not going to be doing
anything else in the background.

When backed by a QGLWidget, the CPU usage drops to 20% on the exact same
example in the exact same conditions (720p, at 16bit color depth). The
frame rate suffers slightly, but at least this is mandated by the GPU

Minor clipping issue evident in hellogl_es2

The bubbles are evidently clipped on the right hand side, I will hopefully
beat you to reporting this at: http://bugreports.qt.nokia.com/secure/Dashboard.jspa

I have not seen any other artifacts, please file any additional bugs you
may encounter at the above URL.

Are these instructions applicable to OMAP3 targets in general

Yes. There is no theoretical reason these instructions would not suffice
for any OMAP3 based target, although I have not personally verified them
outside of Beagle board usage. Caveat emptor.

No Scratchbox2 usage when cross compiling

The more astute of your would recognize that I bypassed Scratchbox2 when
configuring Qt/X11 this time around. I payed dearly for it, and this X11
build has no fontconfig, dbus or glib support even though the Ångström
subsystem I am building against has support for all of them. If you want a
full fledged X11 build with decent font support and OpenGL ES support,
please either:

1) Invest your time in physically adjusting your MKSPEC (and/or wrestling
pkg-config) to get all desired dependencies detected and built against

-Or-

2) Take the easy road, refer to my previous blog posting “Cross compiling
Qt/X11″ and merge the above mkspec changes into the:

./mkspecs/unsupported/linux-scratchbox2-g++

mkspec in your Qt 4.6 source tree.

The same goes for Qt/Embedded which is more self sufficient, but which will
be built without dbus, glib, etc and additional external dependency support
without additional mkspec/environment modification or the use of Scratchbox2
to abstract this away.

Color depth considerations

1) The powervr implementation we are relying on does not support
PVRSRV_PIXEL_FORMAT_RGB888 (24bit color depths), it does however support
PVRSRV_PIXEL_FORMAT_RGB565 and PVRSRV_PIXEL_FORMAT_ARGB8888

2) Ångström is busybox based, and the fbset command you will need to set 32
bit color depths on the console will not work with the default fbset
busybox symlink. You will therefore have to install and use fbset(.real)
in order to get 32bit color depths, which is a simple opkg install away for
the connected Beagle board and a bitbake away for the stranded.

Please note the color depth specified in the boot arguments

console=ttyS0,115200n8=noinitrd ip=dhcp rw root=/dev/mmcblk0p2 omapfb.mode=dvi:1280×720MR-16@60

if you want 32 bit color depth, use:

console=ttyS0,115200n8=noinitrd ip=dhcp rw root=/dev/mmcblk0p2 omapfb.mode=dvi:1280×720MR-24@60

followed by:

/usr/sbin/fbset.real -depth 32 -rgba 8/16,8/8,8/0,8/24

after your Linux kernel drops you in userspace with a kiss on the cheek. A
brave man once tried leaving the color depth at 16 in his boot args, and
jumping all the way to 32bit with fbset so he could change between the more
performant 16 bit color space and the hardware compositing ARGB offering.
Running the dedicated fbset command halved his vertical resolution
regardless of any other parameters he tried to pass fbset and he eventually
ran off to fight another day.

There is a clear performance hit of 7 fps when running hellogl_es2 in
32bit rather than 16bit, taking you down to 20 fps. This hit is even more
pronounced when setting a QGLWidget on the viewport of a QGraphicsView. I
am not sure who is responsible for this, and will be personally
investigating it in the future. Any conjecture/feedback/research performed
by the reader would be greatly appreciated.

*Edited: Introduce rudimentary formatting to make the blog look less Vim forged

Do you know the main advantage of a Hummer? It can go pretty much everywhere, that’s why many armies are using it. If I talk about a Hummer it is because QGraphicsView can go pretty much everywhere too. Recently I was lucky enough to get a N900 (generously given by Jesper, don’t know for how long though) so I decided to test Qt on it. I mainly work on QGraphicsView here at Qt Development Frameworks so it was a way to test the speed of my toy (:p). Since I’ve also been a KDE user and developer for quite a bit, I thought I could try KDE on it and more precisely Plasma.

So I downloaded the Maemo 5 SDK and started building KDE and Qt with it.

First thing you have to know is that scratchbox is not easy to set-up because you always face problems and compiling inside scratchbox is really slow. The link step is a bit broken and you have to specify manually where are the libs you link with (apparently it’s expected to be like that). It seems odd… Anyways, after a couple of hours I had Qt 4.6 (Fremantle branch) up and running. I was quite happy to see that it performs well (if you are using the right things). Animated tiles are running smoothly and painting demos as well (using opengl es2 graphicssystem).

After this initial success I tried KDE. This was more painful than I originally thought. Broken packages (mysql-server), out-dated packages (shared-mime-info for instance) and the worst : CMake crashed during the configure step. I solved that by downloading the 2.8 RC version that I built myself. But after a couple of hours of fighting and debugging I got this:

Link to youtube video

As you can see Plasma is running fine on this device. Of course a lot of work is needed to make it “finger” enabled (the actual applet handles are not really appropriate) but it’s a good start and Plasma is flexible enough to allow that. I have created a separate “shell” called plasma-mobile (Plasma already has the desktop and netbook shell) and I pushed that on the KDE playground. It also needs a lot of work to integrate well with the device (battery, network, profiles and so on) but the goal is to invite people to contribute to Plasma in order to offer a real alternative to the hildon-desktop. Plasma already has many many features/applets that we can just use on the N900. This also comes with a global effort to bring KDE technologies to the device as Kevin said in his blog. You can also participate in discussions on the KDE mailing-list for Maemo.

So yes Plasma is also a hummer, it runs everywhere but it’s the luxury version.

So, are we in the future yet? It is 2009. Where are all the flying cars and personal jet-packs ? Surely we would all be wearing metallic outfits (complete with Employee Visualization Appendage) by now.

The Future Is A Canvas

Back in 2006 a little bit of the future arrived when QGraphicsView was introduced, we were aiming to replace our old QCanvas API with something more flexible and powerful.
It turns out that it is great for creating dynamic, smooth and animated GUIs. Over time it has become the backbone for next-generation user-interface projects like KDE’s Plasma and Qt Declarative UI.

Yet, for all its dynamic, smooth goodness, there are some missing pieces in the puzzle. People are creating full user interfaces in graphicsview, but Qt offers only the most basic widget types to help them out. In many cases it is enough to offer basic primitive types. But if you want to integrate with the platform you are running on or want to use more complex widgets, it becomes difficult.

The Shiny New Stuff

This is why we have started a research project aimed at providing a basic widget-set based on QGraphicsItem. It is a natural extension of the Next Generation Itemviews project. We will take the opportunity to look at how to improve the widget API, internal structure and how we do widget styling and theming.


HotWheels by Leap Kye on flickr

One of the lessons we have learned is that graphics hardware is generally happier when moving pixmaps around, than when drawing lines and shapes again and again. This has implications for the styling API, since our current QStyle is based around drawing complex controls imperatively. It would be better to build our widgets as a graph of (cached) primitive elements that can be transformed and updated individually if necessary.
Similarly it also makes sense to make the input regions in the widget into separate elements in that scene graph.

But, for all our fascination with the new and shiny, we should not forget that our existing widgets contain years of accumulated experience in the form of code that handles all sorts of weird and wonderful corner cases. We don’t want to go though the pain of re-discovering all these cases. It makes sense to re-use existing logic and behavioral code by moving this code into classes that can be used by both the old and the new widgets, when possible. This will also allow us to directly test and verify the internal widget logics independent of the widget front-end.

Still Kinkin’ It Old School ?

We are not forgetting our QWidget based widgets. They will get some initial benefit of better tested back-ends and hopefully some code cleanup. I also have some ideas that will drag QWidget kicking and screaming into the future, but I’ll leave those for another blog-post. Always leave the audience wanting more.

Talk To Us

Even though we are currently working full time on getting Qt 4.6 out the door, the widgets-ng repository is already public and we are already hanging out on #qt-widgets-ng on irc.freenode.net .
I will be attending the Qt Developer Days in München next week, and there will be several developers from the Qt Widgets Team attending the San Francisco event too. Feel free to discuss anything widget related (or any random topic, really) with us. See you there!

In the spirit of writing more demos for Qt/S60, I thought about having something that I will actually use on a daily basis. A weather applet came to my mind. While my previous weather example was more a visual web-scraping attempt based on QtWebKit, this time I wanted something else and hence I decided to use the unofficial Google Weather API just like what is used in Weather Globe Google gadget. The query for this Google service will give back the result in an XML format, which is quite easy to parse. Unfortunately Qt/S60 Tower release does not bundle the XML Patterns module yet, so I have to use the classic stream reader to digest the XML data.

Here is the screenshot of this little weather tool running on the desktop. Use right-click to pop up the context menu, from which you can choose one of the predefined places. The dialog to ask the user for a custom location is left as an exercise for the readers. The beautiful icons are taken from the Tango Desktop Project with an additional set from darkobra at devianart.com. Because I want to take into account the different S60 phone resolutions, these vector-based icons are rendered at run-time thanks to our QtSvg module.

Just like in the case of both the digital clock and kinetic scrolling demos, Alessandro was (more than) willing to waste his precious time filming the following video, for our pleasure. It shows the weather info running on Nokia 5800.

The code is just one git pull away from the usual repository, see the weatherinfo subdirectory. As you might guess, it is using Graphics View, although the layout and animation are done semi-manually. This is because I want to make sure it looks good both on Portrait and Landscape Mode. If you are brave and want to experiment, convert the simple animation to use the brand-new Animation Framework slated for Qt 4.6. Still need some challenges? How about using XPath and XQuery to process the XML? Feel free to implement auto refresh and multiple cities features, too!

Last but not least: thank you, Google!

As part of Qt 4.5, we added QGraphicsItem::opacity. Which is great! But it doesn’t work as well as it could. We’re receiving a few comments about how this implementation could work better. Trouble is, in Qt 4.5 we only have two ways of rendering: direct, and indirect (”cached”, e.g., ItemCoordinateCache). And in order to apply opacity, you really need full composition support… :-/

Here’s a rundown of the trouble with today’s opacity support:

• The current behavior modifies the input opacity of the painter
  • …so you can override this opacity on a per-item basis (intentionally or not!)
  • …and each primitive (line, rect, pixmap) you draw with QPainter will be composed one at a time inside the item (which is bad!)
• The behavior is slightly different depending on whether you cache an item or not.
  • …if you use caching, the offscreen pixmap is rendered with an opacity, whereas the item itself doesn’t have any opacity set on its painter.
• Each item applies opacity locally, and even though opacity propagates to children, each item is still rendered by itself, cause each item to be transparent (as opposed to rendering one subtree as a whole with an opacity)

All these problems could have been solved if we treated opacity as an effect that you can apply to one layer as a whole, instead of on each item. I think this is how opacity should work in the first place… but that’s how it goes sometimes. But fear not, we can fix this in a later release!

To illustrate the current behavior, here’s what you get if you construct a scene with four items, each a child of the previous item, in colors red, then green, then blue, then yellow. Logical structure in hypothetical markup:

Rect {
    color: red;
    rect: QRectF(0, 0, 100, 100);

    child : Rect {
        color : green;
        pos: QPointF(25, 25);
        rect: QRectF(0, 0, 100, 100);
        opacity: 0.5;

        child : Rect {
            color : blue;
            pos: QPointF(25, 25);
            rect: QRectF(0, 0, 100, 100);
            rotation: 45;

            child : Rect {
                color : yellow;
                pos: QPointF(25, 25);
                rect: QRectF(0, 0, 100, 100);
                scale: 2×2;
            };
        };
        };
};

This will in Qt 4.5 render the following output:

What’s important to notice here is how all elements are transparent; so you can see the blue through the yellow item, the green through the blue item, and the red item through the green item. But the yellow item doesn’t actually have any opacity assigned. It inherits opacity from the green item (opacity: 0.5).

By rendering the “green sub-tree” into a separate layer, we can combine all items and apply one uniform opacity as part of composing these items together. In my last blog I wrote about off-screen rendering. This work has progressed and is in quite a usable state (although the code is really ugly). It works! The rendering output for the same application as the above looks like this:

The essential difference is how the “green sub-tree” is treated as if it had one _combined_ opacity. The yellow item, for example, isn’t transparent by itself at all. You can’t see the blue through the yellow. By “collapsing” the subtree into a single layer we can both avoid unnecessary rerendering of items (i.e., if you move the green item around, the children are not repainted, not even from cache). Which is pretty cool!

(We get this at the cost of allocating and spending an extra pixmap, and the first time we render there’s an extra level of indirection.)

But there’s even more applications to this technique… With code for handling explicit composition of layers, we can now use pixmap filters and shaders to compose one layer onto another. And this is very useful for any future effects API we might add to Qt in general. Imagine applying a gaussian blur effect to the layer represented by the green item. The result is below:

The result is very neat; the whole layer is blurred before it’s rendered. You don’t get funny artifacts that might have occurred in overlapping regions if each of the items was blurred individually.

Now none of this gives any mind-blowing screenshots (at least not yet), but it’s an important step that provides faster rendering of subtrees (as sliding and transforming a complex graph of items ends up just being matrix operations on a single pixmap/texture, and in software this prevents overdrawing), a more accurate processing of the opacity property of QGraphicsItem, and it makes it very easy to apply fancy composition effects to groups of items.

Questions: Should layers be explicitly defined, or implicitly handled by logics inside Graphics View? In the example above, I’ve used the opacity property to detect whether an item’s subtree should be rendered into a separate layer or not. This is easy to do and very clear/unsurprising. But what if you want to collapse a subtree for performance reasons? An alternative, or possibly an additional approach would be to add an explicit setting: QGraphicsItem::setLayer(true) or setCacheMode(QGraphicsItem::DeepItemCoordinateCache). I’m torn. Experimentation and feedback will show (yes I know I need to push this code out, once our SCM goes public this will be much easier!).

There’s also the problem with these darn flags QGraphicsItem::ItemIgnoresParentOpacity and QGraphicsItem::ItemDoesntPropagateOpacityToChildren. There’s no way to make these flags work with a layered rendering approach. But are these flags very important, or could we just (*cough* *cough*) disable/deprecate them?

Finally I still haven’t found out how to handle ItemIgnoresTransformations. The most probable solution is that these items are automatically rendered into a separate layer.

Happy hacking!

In recent years the graphics capabilities of desktops and handhelds have been growing more rapidly than their central processors. This is a revolution that is being accelerated by animating user interfaces. This in turn is based on the fact that you can make more usable and more attractive user interfaces using animations.
Qt has been steadily working on solutions that allow more dynamic user interfaces and animations for core features to actually use this available graphics power. Providing support via a framework like Qt sets a baseline and thus allows developers to create usable apps faster.
In Qt this all started with QGraphicsView and we are seeing a coherent vision progress steadily. And now its time for itemviews to be looked at
Itemviews is the collective name for the widgets that show lists, tables and trees. Qt4 has a solution for these widgets already, which works fine for a large set of usecases. There are also usecases where they lack somewhat. Dynamic and fluid user interfaces are a major part of that.
We have started a research project to address all these shortcomings and provide a solution that makes it easier and faster to make beautiful and usable lists, tables and trees. In this blog I want to give an overview of what we have done and what our plans are going forward. Additionally we have put our research project online for everyone to download and play with.

Who is it for?

At this time of development we focussed on great design, good APIs and making easy the common usecases and possible the complex ones. For this reason there will be missing concepts, broken features and in general things may just fall apart. You have been warned
Who we are looking for are developers that want to kick the tires or our new baby, especially useful are developers that have specific usecases in mind of lists and tables that are a challenge to do in the Qt Itemviews. Having running examples build on top of the ItemViewsNG will help us a lot going forward and it will shape the direction we are going.
The actual repository where we work will be public and so people can just follow our progress as we go but we also encourage developers to contribute their patches back to our main repo.

Design

For the next generation of item views, we choose to use QGraphicsView concepts. The obvious advantage is reuse of concepts and features. If you look at the simple list example the individual list items (each row of text, really) is an individual graphics item (QGraphicsWidget, specifically). GraphicsView has various features that are very useful for us. It is very cheap graphics wise to move around the individual items, for example. This allows animations of individual items to use hardware acceleration automatically. GraphicsView also gives us features like being able to click on an item and that item having code to handle it so you can just use the existing concepts and code for making your individual items be drawn and behave the way you want. For old-graphicsview coders; this mirrors the concept of delegates in many ways.

All this talking about graphicsview may raise the question on how to integrate this with traditional user interfaces with QWidgets. A good question, and we created specific widgets for each of the 3 types of item views with proper defaults and simple to use APIs. These are the QtListWidgetNG, the QtTableWidgetNG and the QtTreeWidgetNG. These widgets come with several replaceable components that make it easy to change the look or the interaction separately and so mix and match to your liking.

Let me focus on the QtListWidgetNG here; behind the widget are various classes worth mentioning;

• The QtGraphicsListView allows customizers to decide how to show the individual list items. The default class uses separate graphics view items which has the advantage that we can cheaply animate them.
• The view decides which items to show by asking the QtListModelInterface. As the name states that is just an interface and thus has no implementation. We provide a simple implementation as the QtListDefaultModel.
• All the communication goes via the QtListController class. This includes things like handling keyboard and mouse input received by the QtListWidgetNG but also things like when the active item in the list is changed for some reason then the scrollbar moves to the correct position.

In the ItemViewNG repository today you will find several implementations of at least the listview which allows for an easy way to customize your list.
Here are some usages;

// Simples case; show a complete list
QtListWidgetNG widget1;
widget1.defaultModel()->appendItem(new QtListDefaultItem("Simple!"));
widget1.show();

// Use an alternative view that shows the items as
// icons in a flow and we also use a custom model here.
QtListWidgetNG widget2;
QtListController *controller = widget2.controller();
// custom model comes from the iconFlow example.
controller->setModel(new IconsModel(controller));
controller->setView(new QtGraphicsFlowView());
widget2.resize(QSize(240, 200));
widget2.show();

I started this blog stating that we created the new itemviews based on the idea that we needed pretty user interfaces, and here I am pasting code and showing really boring and standard widgets So lets show a bit more action; I made a screencast of the photoAlbum example that we made on top of the itemviews.

The demo (source-file) really shows a list of images quite similar to the boring list screenshot above, only instead of using the default QGraphicsListView we use a tweaked one to show the items vertically and almost full screen.
Next to that we changed the controller to a QtKineticController which provides the scrolling and flicklist behavior.

Where is it again?

On gitweb/labs.
You’ll appreciate the API docs too. Both are very clearly unfinished as this is research-in-progress!
Compilation requires Qt4.5 (works fine) or the kinetic branch (for some extra functionality, like the photo album).

Lets see who makes the coolest usage example list based on this