keyboard: destroy actors when a group is removed

We can't reuse key actors since signals connected to them are
associated to the original group object.
Fix of commit 824fbe09.

https://bugzilla.gnome.org/show_bug.cgi?id=681735