CVE-2012-3438: Buffer Overflow

Tom Lane (tgl) found an issue in ImageMagick that is also present in GraphicsMagick. Basically CVE-2011-3026 deals with libpng memory allocation, limitations have been added so that a bad PNG can't cause the system to allocate a lot of memory causing a denial of service. However on further investigation of ImageMagick Tom Lane found that PNG malloc function (Magickpngmalloc) in turn calls AcquireMagickMemory with an improper size argument:

#ifdef PNGUSERMEMSUPPORTED static pngvoidp Magickpngmalloc(pngstructp pngptr,pnguint32 size) { (void) pngptr; return((pngvoidp) AcquireMagickMemory((sizet) size)); }

Similar code is present in GraphicsMagick:

#ifdef PNGUSERMEMSUPPORTED static pngvoidp pngIMmalloc(pngstructp pngptr,pnguint32 size) { (void) pngptr; return MagickAllocateMemory(pngvoidp,(sizet) size); }

This is incorrect, the size argument should be declared pngallocsizet according to 1.5, or pngsizet according to 1.2.

"As this function stands, it invisibly does the wrong thing for any request over 4GB. On big-endian architectures it very possibly will do the wrong thing even for requests less than that. So the reason why the hard-wired 4GB limit prevents a core dump is that it masks the ABI mismatch here."

So basically we have memory allocations problems that can probably lead to a denial of service.