CVE-2008-3282: Integer Overflow

It was discovered, that OpenOffice.org memory allocator is not 64bit clean. rtlallocateMemory() function in sal/rtl/source/allocglobal.c accepts one argument - salSize n. On 64bit platforms such as x8664, salSize is defined as unsigned long int. This requested memory chunk size is later memory aligned as size (type salSize). size is later used to calculate int index to galloctable[] array:

int index = (size - 1) >> RTLMEMALIGNSHIFT;

However, as sizeof(int) == 4 and sizeof(salSize) == 8 on 64bit platforms, the calculated value may not fit into index (this can happen when rtlallocateMemory() is called with large argument, e.g. when some other flaw causes OpenOffice to attempt to allocate chunk of memory with negative size, which wraps to large positive value during signed -> unsigned type conversion, such as [1]). Value stored in the index is wrapped / truncated, possibly resulting in the index being negative.

Before index is used, it is checked not to exceed fixed upper limit, but it's not checked whether its value is >= 0:

if (index < RTLMEMORYCACHEDLIMIT >> RTLMEMALIGNSHIFT)

Negative index used in galloctable[index] will cause OpenOffice to access memory outside of the galloctable[] array. This may result it crash, or if that points to an attacker controlled memory, attacker may possibly be able to use this flaw to run arbitrary code.

[1] http://www.openoffice.org/issues/showbug.cgi?id=91818 http://scary.beasts.org/security/CESA-2008-006.html

— Red Hat