Thiago Macieira
Qt
KDE
WebKit
Internet
Posted by Thiago Macieira
 in Qt, KDE, WebKit, Internet
 on Saturday, November 24, 2007 @ 10:28

As you can probably see from recent blogs, lots of new features are being merged into the mainline Qt (what will become Qt 4.4.0). The latest big one is the Network Access API we’ve been developing for the past few months. It’s in today’s snapshot, so you can try it out already. I’d like to thank Marius Monsen, Prasanth, Lars, Andreas, Simon and everyone else who contributed (even if they just contributed opinions).

It’s also one of the big pieces necessary for WebKit: what’s the use of a browser engine if you can’t download anything off the Internet? And another big new feature: an entirely new HTTP stack, written from the scratch to be fully HTTP/1.1 compatible. At this point in time, we’re throwing random junk and real HTTP replies into it to see how it handles broken servers out there (and broken applications too, like trying to download HTTP off the wrong port).

The design is fairly simple. And if it gives you a sensation of déjà vu, it may be because we designed it to be similar to KIO. And now you’re going to ask me: is it supposed to replace KIO? Well, definitely not, at least not in Qt 4.4: KIO does a lot more than file transfers. It also does file management operations (such as copying, moving, deleting, etc.), which is not part of our design here.

But enough of talking, let’s see some code. The following is a full application that downloads the contents of the first argument:

int main(int argc, char **argv)
{
    QCoreApplication app(argc, argv);

    QNetworkAccessManager manager;
    QNetworkReply *reply = manager.get(QUrl::fromEncoded(argv[1]));
    app.connect(reply, SIGNAL(finished()), SLOT(quit()));
    app.exec();
    qDebug() < < reply->readAll();
}

QNetworkAccessManager is the central class in the design. It holds some configuration (proxy, defaults, etc.) and some internal state. It takes one QNetworkRequest object and an operation (based on HTTP operations: get, post, put, head) and returns a QNetworkReply object.

QNetworkRequest is a simple, value-based implicitly-shared class that holds a URL and some meta-data: HTTP headers and extra attributes. It also does parsing of some tricky HTTP headers for you, so you don’t have to worry about getting the HTTP dates right.

Finally, QNetworkReply is an abstract QIODevice-based class. It’s open for reading by the time you get it, which means it will emit readyRead() whenever data is available for reading. Unlike KIO::TransferJob, if you don’t react to the signal, you won’t lose the data.

The next phase now is to make WebKit use this new infrastructure. The API will probably have a simple setManager() function that takes a QNetworkAccessManager object. When it needs more data from the network, it’ll simply call get() there.

We’ve also designed it to be extensible. You’ll be able to write your own QNetworkAccessManager class, that returns your own QNetworkReply objects and/or implements your application’s own network and security policies.

We’re interested in the feedback you can give us. The team and I are available for questions in the qt4-preview-feedback mailing list.

13 Responses to “One more piece falling into place: Network Access”

» Posted by Giovanni Bajo
 on Saturday, November 24, 2007 @ 13:04

Why do you call a HTTP library “Network Access”? It seems confusing, given that Qt already has a “Qt Network” module with much more than HTTP.

» Posted by Thomas Zander
 on Saturday, November 24, 2007 @ 13:53
» Posted by Ankur
 on Saturday, November 24, 2007 @ 14:15

QCoreApplication app(argc, argv);
QNetworkAccessManager manager;
QNetworkReply *trollReply = manager.get(QUrl::fromEncoded(argv[1]));
QNetworkReply *techReply = manager.get(QUrl::fromEncoded(argv[1]));
…..

Is QNetworkAccessManager MultiThreaded ? Is it that once trollReply’s is completed then only techReply will commence. Request are queued are they ?.
Would be great if QNetworkAccessManager could allow me to set no of simultaneous connections I can have.

» Reply from Thiago Macieira
 on Saturday, November 24, 2007 @ 14:33
Thiago Macieira

Because it’s not just HTTP. So far, the API supports HTTP, HTTPS, FTP, local files and Qt resources. We may extend that in the future.

And we may also provide a way of extending it. What’s more, it’s possible to replace the default implementation with one based on KIO.

As for why I chose the name Network Access… who knows, maybe because I liked KIO::NetAccess? In any case, some API can still change (including class names) before the final Qt 4.4.0 release.

» Posted by Harald Fernengel
 on Saturday, November 24, 2007 @ 15:49

Yay, finally I found a nitpick on Thiago’s blog :)

QUrl::fromEncoded(argv[1])

the Qt way to write this:

QUrl::fromEncoded(app.arguments().value(1))

(won’t crash on embedded systems where argv might be 0, and will also work on Chinese Windows versions with weird encodings)

» Posted by Vlad
 on Saturday, November 24, 2007 @ 21:36

1) Can the network facilities be used outside of the main thread?
2) Are there permission checks (isWritable(), isReadable(), etc)?
3) Have you looked at GIO/GVFS by Alexander Larsson to see if there are any good ideas that can be incorporated?

» Posted by Craig Scott
 on Saturday, November 24, 2007 @ 23:33

Something which seems missing from QNetworkRequest is direct support for setting/getting cookies. These are a little more complex than the other known headers provided with QNetworkRequest currently, since they have a name, value, expiry date, path, domain, etc. It would seem to be a good opportunity to simply the process of cookie handling.

» Posted by Chani
 on Sunday, November 25, 2007 @ 06:50

would it be possible for the proxy settings to be sync’d with kde’s proxy settings somehow?
one of the most annoying things about proxies is having to change the settings in half a dozen different places whenever I take my laptop somewhere else.

» Reply from Thiago Macieira
 on Sunday, November 25, 2007 @ 22:43
Thiago Macieira

Ankur: no, the new framework is not multi-threaded nor thread-safe, but it’s reentrant. That is, you’re allowed to use a QNeworkAccessManager only from one thread, but you can have many objects, one in each thread.

However, all connections are serialised: it will only open one connection to each FTP server and will process each request in a queue. For HTTP, it does what the RFC recommends: two connections to each server, no more.

Harald: QUrl::fromEncoded takes a QByteArray. URLs are always encoded in UTF-8, even in weird chinese systems. (RFC 3987)

Vlad: 1) see above
2) Like I said in the blog itself, we only support content retrieval, header retrieval and content sending. We have no intention to match KIO’s capabilities at this moment.
3) I looked at several APIs before designing this (including, but not limited to, Java’s java.net and Apple’s API). I don’t remember right now if I looked closely at GVFS.
I did look closely at KIO (I did a lot of work there as a KDE developer), which explains why my design closely resembles KIO.

» Reply from Thiago Macieira
 on Sunday, November 25, 2007 @ 22:52
Thiago Macieira

Craig: actually, there’s another thing missing, which is the content caching. If you look closely at the internals, you’ll see the beginnings of such a hook. But we had to postpone the cache support until a future Qt version, because it’s very complicated.

It also applies to cookies. But, unlike caching, WebKit has cookie support already. We’re working on something there and my hopes is that we’ll add a cookie jar support in the next weeks. What it will resemble is, so far, anyone’s guess.

I might add that I want to do cookies right or not at all, because it is a security issue to issue wrong cookies (cross-domain cookie injection or stealing).

Chani: yes, I hope to do that in the future. Once we get this working, I will synchronise the proxy configuration between Qt and KDE. Then I’ll propose it as an XDG standard (freedesktop.org).

» Posted by Matthias Kretz
 on Monday, November 26, 2007 @ 09:12

The documentation says “QNetworkReply is a sequential-access QIODevice”. Would it be possible to have some QNetworkReply objects be random-access? You might already want to have that feature for resuming downloads over http/ftp where you don’t want to seek by downloading and discarding…

If they were random access enabled you could easily do:
QNetworkReply *reply = manager.get(url);
Phonon::MediaObject *media = new Phonon::MediaObject;
media->setCurrentSource(reply);
media->play();
// if (!reply->isSequential()) then the following is possible:
media->seek(2000);

» Reply from Thiago Macieira
 on Tuesday, November 27, 2007 @ 20:13
Thiago Macieira

We can study that feature for the future. Right now, they’re sequential-access only.

» Posted by Norbert F
 on Thursday, December 06, 2007 @ 00:46

I believe there should be a Qt API which wraps around GIO/GVFS. At least on Unix/Linux. Applications need a common sub-system for working with SMB, SFTP, FTP shares, and GIO/GVFS is probably the best candidate for this purpose (desktop/toolkit independent design). I think KIO should also move in this direction.