Fix build breakage due to void pointer arithmetic

With recent gnome-common the build fails due to the arithmetics
using void pointers. Use the correct pointers to fix that.

https://bugzilla.gnome.org/show_bug.cgi?id=687755