Since there is currently no official Spotify client that can run on Embedded Linux (wine doesn’t run on arm architectures), and since I really wanted to have access to my Spotify account from my N900, I decided to give the open source Spotify client library called “despotify” a run. This is a library of C functions used to access different parts of the Spotify API for use with premium Spotify accounts.

By playing around with the console clients that are included with despotify, I was able to access my play lists and play songs perfectly. However, I was unable to use any of the GUI front ends to despotify that I could find. My guess is that they do not play well with Maemo 5 as they were originally written for the n800-series.

Inspired by the fact that all this stuff actually existed, I decided to write my own front end to Despotify, using Qt 4.6. The results can be found in Gitorious, at:

http://qt.gitorious.org/qt-labs/qtspotify

To build the application, first compile and install despotify as explained here.

Make sure you enable “pulseaudio” as the back end for despotify by editing the Makefile, as the default gstreamer back end has some threading issues and will cause crashes if you access the GUI while it’s playing. When you are done, do a “make install”.

To build the front end, you also need to have Qt 4.6 available. For best results on the N900, use the Maemo branch of Qt.

When you are done, copy the executable to your phone and start it up. Use the menu to log in, and the search field to search for music. If you want to access your play lists, select the “Retrieve play lists” menu option and they will pop up in the search field menu.

This is what it will look like:

Note that this is an early version, so there are still bugs and some missing features (e.g. you don’t get more than fifty results for a search) that I intend to implement when I get the time, but the client should be usable as it is.

Together with the other platforms (finally!) we’ve now released the Qt 4.6.0 beta1 for Symbian. Get the nice installer here:

So what has happened since the last release (I’m talking about “Tower” released around 3 months ago).

• We’ve moved into the Qt 4.6 branch. This means we can break the other platforms - yay!
• We’ve added an experimental backend for Phonon (using MMF). It plays audio and video!
• We’ve fixed a lot of bugs! Still lots left though.
• We’ve added the possibility to have a mouse curser on non-touch devices!
• QtWebKit has been updated to a newer version
• The QtScript engine has been completely replaced with a nicer and faster one from WebKit.
• We’ve added daily snapshots.
• Ariya has added some new nice demos. Check out the video below.
• We’ve also removed support for compiling qmake with the Nokia Win32 compiler, but added support for MinGW and Visual Studio 200x instead.

If you want to try out these demos on your own S60 phone, here is what you do. Open your browser on the phone and navigate to:

Then simply click on the link found there, answer yes around 20 times to all the dialogs and security warnings. Then once finally done navigate to Applications and launch fluidlauncher.

For help on getting started with your own application development go here. And then finally, the obligatory video of whats new. This time I used a Samsung i8910 and my old trusty Nokia 5800. Enjoy!

This post includes some additional background information to support and
clarify our recent QtMobility Multimedia Blog post.

General architecture

The above block diagram gives a high-level view of multimedia in Qt.

Services

The Multimedia services classes that are in Qt 4.6 in the QtMultimedia module
form Qt’s access to the lowest level of Multimedia functionality. The Audio
classes provide a cross platform means to access audio devices, to send data to
or receive data from them, the audio data would usually be PCM, but could be in
other formats as well. The Video classes have a similar purpose, those in 4.6
exist to provide an abstraction for video outputs.

The Audio classes can be used in a self contained way when you are manipulating
or creating your audio data directly in your application. Some examples of this
would be, a sound recorder, an effect generator, VOIP client, event sounds,
games or a situation requiring intermittent or lower-latency playback.

The service classes are also meant for building other media functionality; to
simplify and provide cross platform support for the building of playback and
recording functionality.

High Level APIs

Phonon

Phonon is a high level playback API. Phonon provides facilities for playback of
media content. Phonon is also currently used in Qt for media playback. We will
continue to maintain Phonon for the duration of the 4.x series, but new
functionality will be added in the new multimedia module.

So why didn’t we try and improve Phonon? There are a lot of interconnected
reasons that would prevent Phonon from moving forward. We felt that that there
would either need to be too much compromise or major source-compatibility
breaking changes (let alone BC breakage), in order to achieve our goal of a
full featured high level framework.

Briefly, some of the problems found with Phonon were as follows -
- The MediaSource class can not adequately represent the variety of data
sources available. For example; Media sources are often not single items of
interest, but a choice of available content (not substreams of content).
Additionally it is often required to bundle additional data with the playback
content, previews, posters etc, there needs to be a way of uniformly
representing these items and tagging them appropriately
- Some playlist like functionality is bound into the media object and limits
the ability of a media subsystem to participate in playlist management. This
is vitally important for certain classes of playback service.
- Metadata support is bound to the media object, this limits the scope for
collaboration with other media service providers that may have useful metadata.
- The attempt to express a high-level media graph imposes a model of operation
on the underlying frameworks that can not necessarily be supported, the
obvious means of routing around this failure can not be made to work
reliability.
– A Qt Multimedia API needs to exist in a cross platform (hardware and
software) environment, this means the attempt at graph manipulation imposes an
added customization burden on existing Phonon backends.
- The backend is responsible for creating objects, not allowing applications to
become involved in the media conversation, this places limits on presentation
options.
- Dynamic backend changes
- Services only being available by plugins loaded by Phonon, again limiting the
media conversation.
- Non-local, out of process, playback services are difficult to implement.

QtMobility Multimedia APIs

The Mobility Multimedia APIs are high-level APIs. There is a place for
high-level APIs in a media universe, for although there are a class of
applications that require the ability to manipulate media streams at varying
levels, there is also a class of application that can happily work without
detailed knowledge of the media subsystem or the elements used in the subsystem
in order to do useful work.

The Mobility Media APIs are made to support playback, recording, playlist
management, metadata, radio, camera, and as time permits transcoding, media
editing, TV and potentially high-level stream management.

There is already support for playback, recording, radio, metadata and playlist
management and some experimental camera support.

Like all of the Mobility projects, the QtMobility Multimedia API is targeted to
merge with Qt at some future date, which has not been decided yet.

The QtMobility Multimedia API competes with Phonon on playback services.
There is little sense in having two competing media frameworks in Qt.

Future Direction

There are lots of places we would like to go, Multimedia is an exciting area to
work in. As mentioned above at the higher level we would like to see
transcoding, media editing, and TV support, we would also like to shift down
slightly and open up access to streams, without confusing the “top-level” API.
We would also like to build upon the services layer, provide a more complete
graph orientated audio only framework, as well as low level video capture
service, but only time will tell what gets implemented and when.

Further information

I hope this brings some clarity to your view of multimedia in Qt. If you have
any further questions you can always comment here, catch us on IRC (#qt-labs;
just ping multimedia, someone should pick it up) or send an email to
qt-interest at trolltech.com.

As you might know we introduced Phonon, a new multimedia layer in Qt 4.4. From the beginning the problem was that we couldn’t support mingw, the only free compiler we support on Windows.
The reality is more the other way around: mingw doesn’t support us, or more precisely, it doesn’t support DirectShow.

So people have been trying and trying to make Phonon build with mingw. A few months ago, I sat down and decided to write the necessary headers to handle Phonon. I succeeded and decided to post this to sourceforge. You can find the item here. Unfortunately it’s been sitting there since then. I hope that it will be picked up in the next iteration of mingw’s win32 API package.

And then, a few weeks ago, a new mingw was *finally* released based on gcc 4.4. That is truly great news for windows users. So I tested it and it works great. I had to fix some stuff but now Qt trunk (future 4.6) builds fine with it.

Back to Phonon, if you follow the link above you’ll find a patch at the end. You can download it and apply it to you own mingw.
To do that it is very simple. First dezip the file and copy the .h in c:\mingw\include and then apply the patch like this (I use patch.exe provided with msys-git):
cd /D c:\mingw\include
patch < dshow.patch

Now if you upgrade to Qt trunk and re-run configure.exe it should even auto-detect that your mingw has Phonon support and it will build it.

So in short, you can now have a decent/recent compiler based on gcc 4.4 + Phonon on Windows.

When I started working on our multimedia framework (aka Phonon) our goal was pretty simple and clear : we needed sound and video support in Qt. We had 3 systems to support and thus 3 completely different underlying API to implement our backends. It was tough and we worked hard on it. But then we got some cool demos and we saw that basic multimedia integration in a cross-platform way is actually possible. We struggled a lot but we got there. The mediaplayer demo is a nice little app that’s fully cross-platform and easy to understand.

Having a look at the bigger picture we realized that another great feature of Qt 4.4 is the ability to get widgets on the canvas (aka QGraphicsView). All our features need to work correctly together. That’s the basic principle of a framework. Displaying video is usually hardware-accelerated and all multimedia API/system use their own way to the graphics card. That means no composition with other widgets, no ability to paint over it and of course no perspective transformation.
My first thought at the time was that we would never be able to achieve the goal of full integration of our own features (ie. having videos on the canvas) and that felt bad.

We still tried and our target was Qt4.5. The biggest challenge was to write our own video renderer. Until Qt 4.4 we always used the native video renderers (you know, the ones without composition at all and no transform).
The 1st thing you have to do is manage the rendering of video frames and use the Qt paint engine. This way you get all the power of Qt.
So after a few weeks of development, tweaking and bug fixing I had something working.

So TADA! (If you were at dev days you’ve already seen that)

Now you can simply do something like that with your VideoWidget:
QGraphicsScene scene;
QGraphicsView view(&scene);
scene.addWidget(videoWidget);

I’m a bit lazy so I didn’t put the code that creates the Phonon objects. You can find out in the documentation and examples how to do that.
And now you can apply transformations to your video. Add things on top with alpha-blending… Possibilities unilimited.
And now you’re probably thinking that there’s a catch… and you’re right. If you look at your CPU usage it will be high. Because now everything is done in software including resizing/transforming/compositing/drawing. So we optimized, rewrote some components provided by the system (”OMG, how can the colorspace-conversion in MS DirectShow be so slow?!”).

Still, there is a way of greatly improve performance. Simply add that line to your code:

view.setViewport(new QGLWidget);

This magical line enables OpenGL through the whole canvas including your video. So scaling/transformation/composition is done by your graphics card. Plus on Windows I also use fragment shader to do the conversion from YUV to RGB (colorspace conversion) on the GPU of your graphics.

To make it even clearer I just made up an additional demo. I simply took the “embedded dialogs” demo from the graphicsview guys and slightly changed it. You can run this videowall demo. You have to pass at least one parameter which is either a video file or a directory (that contains video files) path.
So for example you do: videowall c:\videos
An additional parameter is the dimension of the wall. By default it is 1, meaning 1 video. If you pass for example 3, it will build a video wall with 3×3 videos. I ran it successfully with 9 videos (3×3 wall) with WMV HD content that I got from here. If you want to watch those videos you need to make sure you have the necessary codecs (Windows has them by default).

You can get the little demo here Videowall demo

And if you run it with something like videowall c:\videos 3, you can get the following result:

Now that you have speed and functionality, you can do whatever you want with your videos including using the new animation API!