Some time ago we added support for pixmap effects to Qt, aka blurring, sharpening, etc. You may have noticed the example in examples/painting/pixmapfilter. They have implementation based on the underlying graphics system so when you’re filtering to a GL based QPainter you’re likely to get GL acceleration. We only added a few effects, so I thought I would show you how to write your own…

So I’ll show you the bloom filter, or glow or gloom all depending on where you read about it. The concept is to make the light parts of an image and make them glow, like illustrated below:

Original

with Gloom effect

There are a number of ways to achieve this, so I just picked on that seemed to produce ok results while still being slightly “tutorial” like. The way its implemented is that I first mask away all the pixels that are darker than a given threshold. Then I scale this masked image down, blur it, scale it up again and draw it using QPainter::CompositionMode_Plus. The scaling down is just to get an extra blurring effect and the composition mode is to add to the brightness rather than doing normal alpha blending.

So in a bit more detail… I implemented the brightness thresholding as one separate pixmap filter, called QPixmapBrightnessFilter. To do the actual processing I override the pure virtual draw() function to do the actual processing onto the device.

    virtual void draw(QPainter *painter,
                      const QPointF &p,
                      const QPixmap &src,
                      const QRectF &srcRect = QRectF()) const
    {
        QImage image = src.toImage();

        if (image.format() != QImage::Format_ARGB32_Premultiplied)
            image = image.convertToFormat(QImage::Format_ARGB32_Premultiplied);

        QRect rect = srcRect.toRect();
        if (rect.isEmpty())
            rect = src.rect();

        int ymin = rect.y();
        int ymax = rect.y() + rect.height();
        int xmin = rect.x();
        int xmax = rect.x() + rect.width();
        for (int y=ymin; y<ymax ; ++y) {
            uint *pixels = (uint *) image.scanLine(y);
            for (int x=xmin; x<xmax; ++x) {
                uint pix = pixels[x];
                int brightness = qRed(pix) + qGreen(pix) + qBlue(pix);
                if (brightness < 400)
                    pixels[x] = 0;
            }
        }

        painter->drawImage(p, image, rect);
    }

As you can see, I simply go through the pixels and set the pixel to fully transparent if the sum of the color channels is less than 400. I played a bit around with different values, and for the images I tested on 400 seemed like an “ok” number… feel free to experiment or make it settable

The bloom filter itself is implemented via the brightness filter in combination with QPainter functions and the QPixmapConvolutionFilter.

    virtual void draw(QPainter *painter,
                      const QPointF &p,
                      const QPixmap &src,
                      const QRectF &srcRect = QRectF()) const
    {
        // Draw the original...
        painter->drawPixmap(p, src, srcRect);

        QPixmap source = src;
        if (!srcRect.isEmpty())
            source = source.copy(srcRect.toRect() & src.rect());

        QPixmapBrightnessThresholdFilter threshold;
        QPixmap masked(source.size());
        masked.fill(Qt::transparent);
        QPainter pt(&masked);
        threshold.draw(&pt, QPointF(0, 0), source);
        pt.end();
        source = masked;

        painter->save();
        painter->setCompositionMode(QPainter::CompositionMode_Plus);
        painter->setRenderHint(QPainter::SmoothPixmapTransform);
        painter->setOpacity(painter->opacity() * 0.5);

        QPixmapConvolutionFilter filter;
        qreal kernel[] = {
            1/16., 2/16.0, 1/16.0,
            2/16., 4/16.0, 2/16.0,
            1/16., 2/16.0, 1/16.0
        };
        filter.setConvolutionKernel(kernel, 3, 3);

        for (int i=0; i<4; ++i) {
            painter->scale(2, 2);
            source = source.scaled(source.width() / 2.0,
                                   source.height() / 2.0);
            filter.draw(painter, p, source);
        }

        painter->restore();
    }

At the top of the function I draw the original pixmap. Then I use the QPixmapBrightnessFilter to extract the light-intensive parts of the and store the results in the masked pixmap. Then I save the painter play nice. As I mentioned before, I want to add brightness to the image, so I switch the composition mode to QPainter::CompositionMode_Plus which basically adds more light to the image. Since I already drew the original pixmap, the result will now only be brighter. Because the effect was rather intense, I toned it a bit down using QPainter::setOpacity(0.5).

The image is then scaled down by to half size and drawn blurred at double size. I repeat this progressively four times, thus drawing the original image 1/2, 1/4, 1/8 and 1/16 size.

Since most of the implementation in the QPixmapBloomFilter::draw() function is based on QPainter calls and the exisint QPixmapConvolution, these things can be hardware accelerated, so if I run the example with -graphicssystem opengl or the painter is initialized on a QGLWidget, these things will be hardware accelerated. The only software parts is then the thresholding and downscaling of the pixmaps. Unfortunately, there was a bug in the GL based pixmap filters, so -graphicssystem opengl crashes, but I submit a patch today so it should work in snapshots starting from Monday

So this implementation is only partially H/W accelerated, but it illustrates how filters work. Maybe I get around to do this “properly” over Christmas.

The source code should end up here once the server is synced:
svn checkout svn://labs.trolltech.com/svn/graphics/dojo/bloomfilter

Have a nice holiday!

Please take some time to study this image. The question is, what on earth can this be?

Answer: The blue area is a projection of the view rectangle in item coordinates for one of the dialogs in the “Embedded Dialogs” demo in Qt. So you could say, it’s QGraphicsView’s viewport as seen from a transformed item (the item’s view-of-the-world is, of course, that it itself is perfectly normal, it’s everybody else who’s “transformed” and all). Since the item’s bounds as seen from the view is that of a projected polygon, the view rect seen from the item is also a projected polygon.

Inside in green is what might be the largest axis aligned rectangle that can fit inside this polygon. Finding this rectangle is a bit tricky. Maybe you can help us find a good algorithm for this. One that’s fast, and which finds the perfect rectangle (defined as, the rect with the largest surface area (w*h), and then if there are multiple choices, the one closest to the center of the polygon). I tried brute force. It works ;-).

Now how is this useful? Popups in Qt tend to restrict their geometry to fit inside the desktop. This desktop rectangle (QDesktopWidget::screenGeometry()) doesn’t map very well to Graphics View, or rather, it’s a rather uninteresting rectangle. In QGV you can place widgets at (-200, -200) or (100000, 100000), and it’s not the desktop geometry that should restrict the positioning of popups. It’s more natural that it’s the active viewport that should restrict placement.

Still, QMenu needs to know exactly which QRect it can use to constrain its geometry. That’s simple! It’s the largest axis aligned rectangle that can fit inside the viewport rect mapped to its local coordinate system. If we can find such a rectangle easily, this will fix up the popups in the Embedded Dialogs example so they don’t open outside the viewport where you can’t see them. I’ve tested, it even works fine for complex transforms (e.g., projective transformations).

So, can anyone propose an algorithm?

When I started working on our multimedia framework (aka Phonon) our goal was pretty simple and clear : we needed sound and video support in Qt. We had 3 systems to support and thus 3 completely different underlying API to implement our backends. It was tough and we worked hard on it. But then we got some cool demos and we saw that basic multimedia integration in a cross-platform way is actually possible. We struggled a lot but we got there. The mediaplayer demo is a nice little app that’s fully cross-platform and easy to understand.

Having a look at the bigger picture we realized that another great feature of Qt 4.4 is the ability to get widgets on the canvas (aka QGraphicsView). All our features need to work correctly together. That’s the basic principle of a framework. Displaying video is usually hardware-accelerated and all multimedia API/system use their own way to the graphics card. That means no composition with other widgets, no ability to paint over it and of course no perspective transformation.
My first thought at the time was that we would never be able to achieve the goal of full integration of our own features (ie. having videos on the canvas) and that felt bad.

We still tried and our target was Qt4.5. The biggest challenge was to write our own video renderer. Until Qt 4.4 we always used the native video renderers (you know, the ones without composition at all and no transform).
The 1st thing you have to do is manage the rendering of video frames and use the Qt paint engine. This way you get all the power of Qt.
So after a few weeks of development, tweaking and bug fixing I had something working.

So TADA! (If you were at dev days you’ve already seen that)

Now you can simply do something like that with your VideoWidget:
QGraphicsScene scene;
QGraphicsView view(&scene);
scene.addWidget(videoWidget);

I’m a bit lazy so I didn’t put the code that creates the Phonon objects. You can find out in the documentation and examples how to do that.
And now you can apply transformations to your video. Add things on top with alpha-blending… Possibilities unilimited.
And now you’re probably thinking that there’s a catch… and you’re right. If you look at your CPU usage it will be high. Because now everything is done in software including resizing/transforming/compositing/drawing. So we optimized, rewrote some components provided by the system (”OMG, how can the colorspace-conversion in MS DirectShow be so slow?!”).

Still, there is a way of greatly improve performance. Simply add that line to your code:

view.setViewport(new QGLWidget);

This magical line enables OpenGL through the whole canvas including your video. So scaling/transformation/composition is done by your graphics card. Plus on Windows I also use fragment shader to do the conversion from YUV to RGB (colorspace conversion) on the GPU of your graphics.

To make it even clearer I just made up an additional demo. I simply took the “embedded dialogs” demo from the graphicsview guys and slightly changed it. You can run this videowall demo. You have to pass at least one parameter which is either a video file or a directory (that contains video files) path.
So for example you do: videowall c:\videos
An additional parameter is the dimension of the wall. By default it is 1, meaning 1 video. If you pass for example 3, it will build a video wall with 3×3 videos. I ran it successfully with 9 videos (3×3 wall) with WMV HD content that I got from here. If you want to watch those videos you need to make sure you have the necessary codecs (Windows has them by default).

You can get the little demo here Videowall demo

And if you run it with something like videowall c:\videos 3, you can get the following result:

Now that you have speed and functionality, you can do whatever you want with your videos including using the new animation API!

For the last year or so, several trolls have worked on ramping up Qt’s GUI capabilities by adding features and flexibility to our widget model, extending Qt with an animation framework, and investigating internal changes to our graphics system to meet the demand for highly animated GUIs (including excessive use of OpenGL). Some of the graphics improvements are merged into Qt 4.5, and available in snapshots. Today we’re sharing the rest of our code with you all, so you can see what’s going on.

* Labs project page: http://labs.trolltech.com/page/Projects/Graphics/Kinetic
* Git clone URL: git://labs.trolltech.com/qt/kinetic

The name of this project is “Kinetic”. Our goal is to make it easier for you as a developer to create highly animated, high-performance GUIs, without the hassle of managing complex structures, timers, and easing curves, not to mention the large state graphs that all animated GUIs tend to be full of. We’re investigating building blocks that you need in order to make the simple things easy, and everything else possible. Some parts are C++, some parts are XML.

We’d like to bring out the best from every good animation framework out there, and look for a way that makes animations fit neatly into Qt. If we can make an animation API that looks like it has always been there, and which inspires you to add nifty animations to your applications, then we have been successful.

For example, if you’d like to define two states for a widget’s geometry and an animated transition, you might do this:

QStateGroup *group = new QStateGroup;
QState *a = new QState(group);
QState *b = new QState(group);

QTransition *transition = new QTransition(a, b);
transition->addAnimation(new QAnimation(w, “geometry”));

Performance is extremely important; especially when the elements that define your UI suddenly rely on it. A sluggish animated UI is hopeless and defeats the purpose of having animations in the first place. It’s amazing how a slow animation can turn a good experience into a bad one. Kinetic places a whole new set of requirements on our graphics system. Performance starts with graphics hardware capabilities at one end of the scale, and “do what I mean” on the other end of the scale. There’s work to be done in between. We need to find ways to ensure that new UI effects are as pleasing to the eye as they are to the developer.

As for graphical effects; you might have seen recent posts from the Graphics Dojo about how cool effects are possible to do using Qt today. As part of Kinetic, we’d like to investigate how these effects can integrate neatly into the UI framework. Can we add composition effects like blurring, dropshadows, blooming, colorizing (e.g., backdrop darkening, fading from white, etc.), while keeping the APIs easy to use and the performance at 100%?

Our widget model poses some restrictions on animations. We’d like to “break out of this box”. We’re investigating how Graphics View can loosen up some of the basic constraints by allowing widgets to be scaled, rotated, faded and blurred. For the end user, a standard form looks the same by itself or embedded in Graphics View. But once you can combine widgets with custom items, and add animations and effects to the puzzle, we think you can create magic. In C++ all the details are exposed, but with XML it doesn’t have to be. We’re investigating how to evolve our UI format, which has essentially been the same for many years. Let’s face it… it’s time for a workout. We’re making a different format now, that has the same expressive power as the .UI format, but also allows you to embed states and animated transitions. This final part of the project is called “Declarative UI”.

Here’s a sample XML file that describes a simple animated UI (as a png because I can’t get Wordpress’s verbatim and pre tags to do what they should!).

So, please take this as an invitation to join us. You can find the new project on Labs and a Qt Solution for the Animation Framework in usual pages. There’s not a lot of bling there yet, because at this point it is mostly code. But there will be bling. ;-).

How do you animate a widget in Qt 4? Let’s start with a bit of history….

- Qt 4.0 : QTimer provides a low level timer.
- Qt 4.2 : QTimeLine provides a duration, looping, easing and basic states
- Qt 4.2 : QGraphicsItemAnimation which is built on top of QTimeLine.

We realized that these classes were not convenient enough for doing highly animated user interfaces, so we decided to write yet another world dominating framework for animating widgets :-). After a lot of research, prototyping, API discussions and rewrites we have finally something that will dominate the world.

Initially it was our intention to release the animation framework for 4.5. Unfortunately we felt it was not ready for 4.5, so we decided to postpone it in order to give it some more love. We feel that we are getting closer now, but as a service to people that have been waiting for this we have decided to release it as a Qt Solution. We also have a secret, scrupulous plan B with this: we want you guys to give feedback (qt-solutions@trolltech.com) on it so we can improve it even more :-). It is available under the normal Qt Solution conditions, either commercial or GPL.

For you gaming enthusiasts we have made a full blown world dominating game: