resolver: Return error looking up services on invalid hostnames

Make g_resolver_lookup_service() and
g_resolver_lookup_service_async() error out when invalid hostnames are
passed.

https://bugzilla.gnome.org/show_bug.cgi?id=772989