Be more careful with weak pointers

As a follow-up to bug #747422, be careful to remove weak pointers to
objects when it appears that the lifetime of the object can outlive the
lifetime of the memory location of the pointer.

https://bugzilla.gnome.org/show_bug.cgi?id=751858