resolver: Return error looking up invalid hostnames

Make g_resolver_lookup_by_name() and g_resolver_lookup_by_name_async()
error out when invalid hostnames are passed.

https://bugzilla.gnome.org/show_bug.cgi?id=772989