Be more careful when calling destroy notifies

If we are, we can allow modification of the hash table
from destroy notifies.

https://bugzilla.gnome.org/show_bug.cgi?id=650459