Fix endless loop with cairo renderer (maybe bug 660574)

Dia's geometry function fillet() was not initializing
it's return values under certain conditions. Now it signals
these conditions by returning false, to allow the API user
(draw_rounded_polyline) to not use these values.

Still cairo should not go into an endless loop with completely
bogus values, but hopefully Dia does not trigger that bug
anymore ...