CVE-2011-1782: Buffer Overflow

Originally Common Vulnerabilities and Exposures assigned an identifier of CVE-2010-4543 to the following vulnerability:

Heap-based buffer overflow in the readchanneldata function in file-psp.c in the Paint Shop Pro (PSP) plugin in GIMP 2.6.11 allows remote attackers to cause a denial of service (application crash) or possibly execute arbitrary code via a PSPCOMPRLE (aka RLE compression) image file that begins a long run count at the end of the image. NOTE: some of these details are obtained from third party information.

Upstream bug report: [1] https://bugzilla.gnome.org/showbug.cgi?id=639203

Original patch proposal from Vincent Untz: [2] https://bugzilla.gnome.org/showbug.cgi?id=639203#c12

And final patch applied by Gimp upstream was: [3] http://git.gnome.org/browse/gimp/commit/?id=48ec15890e1751dede061f6d1f469b6508c13439

Later it was recognized by Nils Philippsen this patch to be incomplete due the following reasoning:

I've looked at this a bit further and found that even Vincent Untz's patch didn't take into account that the code that did runlength decoding actually advances by bytespp through the buffer into which it writes, so the runcount needs to be clamped to "(endq - q) / bytespp"() in order that subsequent loops don't advance past the end of the buffer.

(): This division is actually safe as bytespp is set by the plugin to one of several known values, therefore can't be 0.

— Red Hat