socket: actually remove fd from poll when socket is closed

In my previous patch, I failed to call g_source_remove_unix_fd() in
order to actually stop polling the fd of the closed socket.

The test did not catch this, because the test only checks that the right
source callback is dispatched properly. I don't know how to test this.

https://bugzilla.gnome.org/show_bug.cgi?id=723655