Please take some time to study this image. The question is, what on earth can this be?

Answer: The blue area is a projection of the view rectangle in item coordinates for one of the dialogs in the “Embedded Dialogs” demo in Qt. So you could say, it’s QGraphicsView’s viewport as seen from a transformed item (the item’s view-of-the-world is, of course, that it itself is perfectly normal, it’s everybody else who’s “transformed” and all). Since the item’s bounds as seen from the view is that of a projected polygon, the view rect seen from the item is also a projected polygon.

Inside in green is what might be the largest axis aligned rectangle that can fit inside this polygon. Finding this rectangle is a bit tricky. Maybe you can help us find a good algorithm for this. One that’s fast, and which finds the perfect rectangle (defined as, the rect with the largest surface area (w*h), and then if there are multiple choices, the one closest to the center of the polygon). I tried brute force. It works ;-).

Now how is this useful? Popups in Qt tend to restrict their geometry to fit inside the desktop. This desktop rectangle (QDesktopWidget::screenGeometry()) doesn’t map very well to Graphics View, or rather, it’s a rather uninteresting rectangle. In QGV you can place widgets at (-200, -200) or (100000, 100000), and it’s not the desktop geometry that should restrict the positioning of popups. It’s more natural that it’s the active viewport that should restrict placement.

Still, QMenu needs to know exactly which QRect it can use to constrain its geometry. That’s simple! It’s the largest axis aligned rectangle that can fit inside the viewport rect mapped to its local coordinate system. If we can find such a rectangle easily, this will fix up the popups in the Embedded Dialogs example so they don’t open outside the viewport where you can’t see them. I’ve tested, it even works fine for complex transforms (e.g., projective transformations).

So, can anyone propose an algorithm?