Cast enum to int before doing calculation

It seems that it is possible for compilers to use unsigned interger
types to store enum values, so we should cast them to signed interger
types before doing calculation to avoid getting unexpected results.

https://bugzilla.gnome.org/show_bug.cgi?id=783428