* [PATCH] mm: incorrect overflow check in shrink_slab()
@ 2011-12-01 9:20 Xi Wang
2011-12-01 9:32 ` KAMEZAWA Hiroyuki
0 siblings, 1 reply; 5+ messages in thread
From: Xi Wang @ 2011-12-01 9:20 UTC (permalink / raw)
To: Andrew Morton, Mel Gorman, Minchan Kim, KAMEZAWA Hiroyuki,
Johannes Weiner
Cc: linux-mm, linux-kernel
total_scan is unsigned long, so the overflow check (total_scan < 0)
didn't work.
Signed-off-by: Xi Wang <xi.wang@gmail.com>
---
mm/vmscan.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index a1893c0..46a04e7 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -270,7 +270,7 @@ unsigned long shrink_slab(struct shrink_control *shrink,
delta *= max_pass;
do_div(delta, lru_pages + 1);
total_scan += delta;
- if (total_scan < 0) {
+ if ((long)total_scan < 0) {
printk(KERN_ERR "shrink_slab: %pF negative objects to "
"delete nr=%ld\n",
shrinker->shrink, total_scan);
--
1.7.5.4
^ permalink raw reply related [flat|nested] 5+ messages in thread* Re: [PATCH] mm: incorrect overflow check in shrink_slab()
2011-12-01 9:20 [PATCH] mm: incorrect overflow check in shrink_slab() Xi Wang
@ 2011-12-01 9:32 ` KAMEZAWA Hiroyuki
2011-12-01 9:57 ` Xi Wang
2011-12-01 20:54 ` Andrew Morton
0 siblings, 2 replies; 5+ messages in thread
From: KAMEZAWA Hiroyuki @ 2011-12-01 9:32 UTC (permalink / raw)
To: Xi Wang
Cc: Andrew Morton, Mel Gorman, Minchan Kim, Johannes Weiner, linux-mm,
linux-kernel
On Thu, 1 Dec 2011 04:20:34 -0500
Xi Wang <xi.wang@gmail.com> wrote:
> total_scan is unsigned long, so the overflow check (total_scan < 0)
> didn't work.
>
> Signed-off-by: Xi Wang <xi.wang@gmail.com>
Nice catch but.... the 'total_scan" shouldn't be long ?
Rather than type casting ?
Thanks,
-Kame
> ---
> mm/vmscan.c | 2 +-
> 1 files changed, 1 insertions(+), 1 deletions(-)
>
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index a1893c0..46a04e7 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -270,7 +270,7 @@ unsigned long shrink_slab(struct shrink_control *shrink,
> delta *= max_pass;
> do_div(delta, lru_pages + 1);
> total_scan += delta;
> - if (total_scan < 0) {
> + if ((long)total_scan < 0) {
> printk(KERN_ERR "shrink_slab: %pF negative objects to "
> "delete nr=%ld\n",
> shrinker->shrink, total_scan);
> --
> 1.7.5.4
>
>
>
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [PATCH] mm: incorrect overflow check in shrink_slab()
2011-12-01 9:32 ` KAMEZAWA Hiroyuki
@ 2011-12-01 9:57 ` Xi Wang
2011-12-01 20:54 ` Andrew Morton
1 sibling, 0 replies; 5+ messages in thread
From: Xi Wang @ 2011-12-01 9:57 UTC (permalink / raw)
To: KAMEZAWA Hiroyuki
Cc: Andrew Morton, Mel Gorman, Minchan Kim, Johannes Weiner, linux-mm,
linux-kernel
On Dec 1, 2011, at 4:32 AM, KAMEZAWA Hiroyuki wrote:
> Nice catch but.... the 'total_scan" shouldn't be long ?
> Rather than type casting ?
Could be.. I am just trying to avoid signed integer overflow like
"total_scan += delta" in that case, which is undefined, even though
the kernel is compiled with -fno-strict-overflow.
- xi
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] mm: incorrect overflow check in shrink_slab()
2011-12-01 9:32 ` KAMEZAWA Hiroyuki
2011-12-01 9:57 ` Xi Wang
@ 2011-12-01 20:54 ` Andrew Morton
2011-12-02 2:00 ` Xi Wang
1 sibling, 1 reply; 5+ messages in thread
From: Andrew Morton @ 2011-12-01 20:54 UTC (permalink / raw)
To: KAMEZAWA Hiroyuki
Cc: Xi Wang, Mel Gorman, Minchan Kim, Johannes Weiner, linux-mm,
linux-kernel
On Thu, 1 Dec 2011 18:32:02 +0900
KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> wrote:
> > total_scan is unsigned long, so the overflow check (total_scan < 0)
> > didn't work.
> >
> > Signed-off-by: Xi Wang <xi.wang@gmail.com>
>
> Nice catch but.... the 'total_scan" shouldn't be long ?
> Rather than type casting ?
Konstantin Khlebnikov's "vmscan: fix initial shrinker size handling"
does change it to `long'. That patch is in -mm and linux-next and is
queued for 3.3. It was queued for 3.2 but didn't make it due to some
me/Dave Chinner confusion.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] mm: incorrect overflow check in shrink_slab()
2011-12-01 20:54 ` Andrew Morton
@ 2011-12-02 2:00 ` Xi Wang
0 siblings, 0 replies; 5+ messages in thread
From: Xi Wang @ 2011-12-02 2:00 UTC (permalink / raw)
To: Andrew Morton
Cc: KAMEZAWA Hiroyuki, Mel Gorman, Minchan Kim, Johannes Weiner,
linux-mm, linux-kernel
On Dec 1, 2011, at 3:54 PM, Andrew Morton wrote:
> Konstantin Khlebnikov's "vmscan: fix initial shrinker size handling"
> does change it to `long'. That patch is in -mm and linux-next and is
> queued for 3.3. It was queued for 3.2 but didn't make it due to some
> me/Dave Chinner confusion.
I see. Cool. Thanks for the pointer.
- xi
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2011-12-02 2:00 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-12-01 9:20 [PATCH] mm: incorrect overflow check in shrink_slab() Xi Wang
2011-12-01 9:32 ` KAMEZAWA Hiroyuki
2011-12-01 9:57 ` Xi Wang
2011-12-01 20:54 ` Andrew Morton
2011-12-02 2:00 ` Xi Wang
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox