daemon: don't allow worker manager interface to be unexported more than once

if a conversation is stopped asynchronously and then synchronously, we
will end up trying to unexport the bus interface more than once, leading
to log spew.

This commit protects against that.

https://bugzilla.gnome.org/show_bug.cgi?id=737391