Fix bounds checking for RRE, hextile & copyrect encodings

While the client would bounds check the overall update
region, it failed to bounds check the payload data
parameters.

Add a test case to validate bounds checking.

https://bugzilla.gnome.org/show_bug.cgi?id=778048



CVE-2017-5884

Signed-off-by: Daniel P. Berrange <berrange@redhat.com>