backends: Calculate output scale correctly on vertical transforms

The code calculating the output scale involves calculations around pixel
and mm sizes, however we do compare post-transformation pixel sizes to
untransformed mm sizes, which breaks the DPI calculations. Fix this by
transforming back pixel sizes back to untransformed.

While we're at it, actually compare the output height to HIDPI_MIN_HEIGHT
instead of its width, it seems right according to the #define name and
comment.

https://bugzilla.gnome.org/show_bug.cgi?id=777687