socket: Don't poll the socket fd after close

This prevents polling on file descriptors that are no longer in use
or have been reused for something else.

Based on a patch by Mikhail Zabaluev

https://bugzilla.gnome.org/show_bug.cgi?id=723655