CVE-2011-4097: Integer Overflow
First published: Tue Nov 01 2011(Updated: )
<a href="https://lkml.org/lkml/2011/10/31/138">https://lkml.org/lkml/2011/10/31/138</a> An integer overflow will happen on 64bit archs if task's sum of rss, swapents and nr_ptes exceeds (2^31)/1000 value. This was introduced by commit f755a04 oom: use pte pages in OOM score where the oom score computation was divided into several steps and it's no longer computed as one expression in unsigned long(rss, swapents, nr_pte are unsigned long), where the result value assigned to points(int) is in range(1..1000). So there could be an int overflow while computing 176 points *= 1000; and points may have negative value. Meaning the oom score for a mem hog task will be one. 196 if (points <= 0) 197 return 1; For example: [ 3366] 0 3366 35390480 24303939 5 0 0 oom01 Out of memory: Kill process 3366 (oom01) score 1 or sacrifice child Here the oom1 process consumes more than 24303939(rss)*4096~=92GB physical memory, but it's oom score is one. In this situation the mem hog task is skipped and oom killer kills another and most probably innocent task with oom score greater than one. The points variable should be of type long instead of int to prevent the int overflow. Signed-off-by: Frantisek Hrbata <fhrbata> --- mm/oom_kill.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/mm/oom_kill.c b/mm/oom_kill.c index 626303b..e9a1785 100644 --- a/mm/oom_kill.c +++ b/mm/oom_kill.c @@ -162,7 +162,7 @@ static bool oom_unkillable_task(struct task_struct *p, unsigned int oom_badness(struct task_struct *p, struct mem_cgroup *mem, const nodemask_t *nodemask, unsigned long totalpages) { - int points; + long points; if (oom_unkillable_task(p, mem, nodemask)) return 0; Introduced by: <a href="http://git.kernel.org/linus/f755a04">http://git.kernel.org/linus/f755a04</a> Upstream commit: TBD Acknowledgements: Red Hat would like to thank Shubham Goyal for reporting this issue.
Credit: secalert@redhat.com secalert@redhat.com
| Affected Software | Affected Version | How to fix |
|---|---|---|
| Linux Linux kernel | <3.1.8 | |
| Redhat Enterprise Linux | =6.0 | |
| debian/linux-2.6 |
Never miss a vulnerability like this again
Sign up to SecAlerts for real-time vulnerability data matched to your software, aggregated from hundreds of sources.
Reference Links
- https://github.com/torvalds/linux/commit/56c6a8a4aadca809e04276eabe5552935c51387f
- https://bugzilla.redhat.com/show_bug.cgi?id=750399
- http://www.kernel.org/pub/linux/kernel/v3.x/ChangeLog-3.1.8
- http://www.openwall.com/lists/oss-security/2011/11/01/2
- https://launchpad.net/bugs/cve/CVE-2011-4097
- https://lkml.org/lkml/2011/10/31/138
- http://git.kernel.org/linus/f755a04
- https://bugzilla.redhat.com/show_bug.cgi/show_bug.cgi?id=750402
- https://rhn.redhat.com/errata/RHSA-2012-0333.html
- https://security-tracker.debian.org/tracker/CVE-2011-4097
- https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2011-4097
- https://nvd.nist.gov/vuln/detail/CVE-2011-4097
- https://ubuntu.com/security/CVE-2011-4097
- collector/nvd-historical
- collector/nvd-index
- agent/type
- agent/first-publish-date
- collector/launchpad-cve
- source/Launchpad
- collector/redhat-bugzilla
- source/Red Hat
- alias/CVE-2011-4097
- agent/author
- agent/softwarecombine
- collector/security-tracker-debian
- source/Debian
- agent/software-canonical-lookup
- collector/nvd-cve
- source/NVD
- agent/description
- agent/remedy
- agent/severity
- agent/weakness
- agent/references
- collector/mitre-cve
- source/MITRE
- agent/last-modified-date
- agent/tags
- agent/event
- collector/usn-cve
- source/Ubuntu
- vendor/linux
- canonical/linux linux kernel
- vendor/redhat
- canonical/redhat enterprise linux
- version/redhat enterprise linux/6.0
- package-manager/debian