2.6.22-rc1-mm1 huge pages VM freeze (maybe?)

2.6.22-rc1-mm1 huge pages VM freeze (maybe?)

[Zan Lynx]

[-- Attachment #1: Type: text/plain, Size: 1467 bytes --]

I was playing with huge pages and libhugetlbfs.  Small programs like
"ls" work fine.  I tried running Evolution through libhugetlbfs and the
system slowly stops running.  One interesting thing is the "ps" command,
it gets stuck like this:

ps            D ffff81001e57ed40     0 103558 103483
 ffff81001f061dc8 0000000000000096 ffff81003d8586e8 ffff81001cbadc00
 0000000000000006 ffffffff80537009 0000000000000030 ffffffff807ff700
 ffffffff807ff700 ffffffff807ff700 ffffffff807ff700 ffffffff807ff700
Call Trace:
 [<ffffffff80537009>] _spin_unlock+0x29/0x50
 [<ffffffff80536425>] __down_read+0x75/0xaf
 [<ffffffff80295f49>] access_process_vm+0x49/0x190
 [<ffffffff802f3003>] proc_pid_cmdline+0xa3/0x130
 [<ffffffff802f4cea>] proc_info_read+0xba/0x100
 [<ffffffff802b0085>] vfs_read+0xc5/0x180
 [<ffffffff802b0583>] sys_read+0x53/0x90
 [<ffffffff8020c1de>] system_call+0x7e/0x83

and nothing will touch it after that.

Here's my kernel command line:
root=/dev/sda2 rootfstype=reiser4 rootflags=no_write_barrier ro
i8042.nomux elevator=cfq resume=/dev/sda3 panic=5 nmi_watchdog=2,panic
debug hugepages=32

Here's the "huge" script I was using to run programs:
#!/bin/sh
export LD_PRELOAD=libhugetlbfs.so
export HUGETLB_MORECORE=yes
export HUGETLB_PATH=/mnt/huge
export HUGETLB_VERBOSE=1
exec "$@"

I don't have any more info than that at the moment but I could reproduce
it with whatever, on request.

-- 
Zan Lynx <zlynx@acm.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: 2.6.22-rc1-mm1 huge pages VM freeze (maybe?)

[Randy Dunlap]

On Tue, 31 Jul 2007 15:44:21 -0600 Zan Lynx wrote:

> I was playing with huge pages and libhugetlbfs.  Small programs like
> "ls" work fine.  I tried running Evolution through libhugetlbfs and the
> system slowly stops running.  One interesting thing is the "ps" command,
> it gets stuck like this:

Do you mean 2.6.22-rc1-mm1 or 2.6.23-rc1-mm1?

There was a hugepage problem fixed very recently, in 2.6.23-rc1 IIRC.


> ps            D ffff81001e57ed40     0 103558 103483
>  ffff81001f061dc8 0000000000000096 ffff81003d8586e8 ffff81001cbadc00
>  0000000000000006 ffffffff80537009 0000000000000030 ffffffff807ff700
>  ffffffff807ff700 ffffffff807ff700 ffffffff807ff700 ffffffff807ff700
> Call Trace:
>  [<ffffffff80537009>] _spin_unlock+0x29/0x50
>  [<ffffffff80536425>] __down_read+0x75/0xaf
>  [<ffffffff80295f49>] access_process_vm+0x49/0x190
>  [<ffffffff802f3003>] proc_pid_cmdline+0xa3/0x130
>  [<ffffffff802f4cea>] proc_info_read+0xba/0x100
>  [<ffffffff802b0085>] vfs_read+0xc5/0x180
>  [<ffffffff802b0583>] sys_read+0x53/0x90
>  [<ffffffff8020c1de>] system_call+0x7e/0x83
> 
> and nothing will touch it after that.
> 
> Here's my kernel command line:
> root=/dev/sda2 rootfstype=reiser4 rootflags=no_write_barrier ro
> i8042.nomux elevator=cfq resume=/dev/sda3 panic=5 nmi_watchdog=2,panic
> debug hugepages=32
> 
> Here's the "huge" script I was using to run programs:
> #!/bin/sh
> export LD_PRELOAD=libhugetlbfs.so
> export HUGETLB_MORECORE=yes
> export HUGETLB_PATH=/mnt/huge
> export HUGETLB_VERBOSE=1
> exec "$@"
> 
> I don't have any more info than that at the moment but I could reproduce
> it with whatever, on request.

---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

Re: 2.6.22-rc1-mm1 huge pages VM freeze (maybe?)

[Zan Lynx]

[-- Attachment #1: Type: text/plain, Size: 675 bytes --]

On Tue, 2007-07-31 at 15:02 -0700, Randy Dunlap wrote:
> On Tue, 31 Jul 2007 15:44:21 -0600 Zan Lynx wrote:
> 
> > I was playing with huge pages and libhugetlbfs.  Small programs like
> > "ls" work fine.  I tried running Evolution through libhugetlbfs and the
> > system slowly stops running.  One interesting thing is the "ps" command,
> > it gets stuck like this:
> 
> Do you mean 2.6.22-rc1-mm1 or 2.6.23-rc1-mm1?

D'oh!  I mean 2.6.23-rc1-mm1, the 22 was a typo.  Cut & paste to be
sure:
Linux zephyr 2.6.23-rc1-mm1 #1 SMP PREEMPT Wed Jul 25 17:33:04 MDT 2007
x86_64 AMD Athlon(tm) 64 Processor 3400+ AuthenticAMD GNU/Linux

-- 
Zan Lynx <zlynx@acm.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: 2.6.22-rc1-mm1 huge pages VM freeze (maybe?)

[Nish Aravamudan]

On 7/31/07, Zan Lynx <zlynx@acm.org> wrote:
> On Tue, 2007-07-31 at 15:02 -0700, Randy Dunlap wrote:
> > On Tue, 31 Jul 2007 15:44:21 -0600 Zan Lynx wrote:
> >
> > > I was playing with huge pages and libhugetlbfs.  Small programs like
> > > "ls" work fine.  I tried running Evolution through libhugetlbfs and the
> > > system slowly stops running.  One interesting thing is the "ps" command,
> > > it gets stuck like this:
> >
> > Do you mean 2.6.22-rc1-mm1 or 2.6.23-rc1-mm1?
>
> D'oh!  I mean 2.6.23-rc1-mm1, the 22 was a typo.  Cut & paste to be
> sure:
> Linux zephyr 2.6.23-rc1-mm1 #1 SMP PREEMPT Wed Jul 25 17:33:04 MDT 2007
> x86_64 AMD Athlon(tm) 64 Processor 3400+ AuthenticAMD GNU/Linux

Hrm -- if you kill Evolution does the system come back? Or is it
unkillable/unusable. I guess you were able to run ps at the same time.

What is Evolution doing (sysrq+t)? For that matter, what was the
output from libhuge?

Thanks,
Nish

Re: 2.6.22-rc1-mm1 huge pages VM freeze (maybe?)

[Nish Aravamudan]

On 7/31/07, Zan Lynx <zlynx@acm.org> wrote:
> On Tue, 2007-07-31 at 15:02 -0700, Randy Dunlap wrote:
> > On Tue, 31 Jul 2007 15:44:21 -0600 Zan Lynx wrote:
> >
> > > I was playing with huge pages and libhugetlbfs.  Small programs like
> > > "ls" work fine.  I tried running Evolution through libhugetlbfs and the
> > > system slowly stops running.  One interesting thing is the "ps" command,
> > > it gets stuck like this:
> >
> > Do you mean 2.6.22-rc1-mm1 or 2.6.23-rc1-mm1?
>
> D'oh!  I mean 2.6.23-rc1-mm1, the 22 was a typo.  Cut & paste to be
> sure:
> Linux zephyr 2.6.23-rc1-mm1 #1 SMP PREEMPT Wed Jul 25 17:33:04 MDT 2007
> x86_64 AMD Athlon(tm) 64 Processor 3400+ AuthenticAMD GNU/Linux

Just to confirm, still happens with -mm2?

Thanks,
Nish

Re: 2.6.22-rc1-mm1 huge pages VM freeze (maybe?)

[Zan Lynx]

[-- Attachment #1: Type: text/plain, Size: 1669 bytes --]

On Wed, 2007-08-01 at 08:52 -0700, Nish Aravamudan wrote:
> On 7/31/07, Zan Lynx <zlynx@acm.org> wrote:
> > On Tue, 2007-07-31 at 15:02 -0700, Randy Dunlap wrote:
> > > On Tue, 31 Jul 2007 15:44:21 -0600 Zan Lynx wrote:
> > >
> > > > I was playing with huge pages and libhugetlbfs.  Small programs like
> > > > "ls" work fine.  I tried running Evolution through libhugetlbfs and the
> > > > system slowly stops running.  One interesting thing is the "ps" command,
> > > > it gets stuck like this:
> > >
> > > Do you mean 2.6.22-rc1-mm1 or 2.6.23-rc1-mm1?
> >
> > D'oh!  I mean 2.6.23-rc1-mm1, the 22 was a typo.  Cut & paste to be
> > sure:
> > Linux zephyr 2.6.23-rc1-mm1 #1 SMP PREEMPT Wed Jul 25 17:33:04 MDT 2007
> > x86_64 AMD Athlon(tm) 64 Processor 3400+ AuthenticAMD GNU/Linux
> 
> Just to confirm, still happens with -mm2?

No, it does not seem to.  Evolution runs OK.  ps, top, pmap all work
fine.

However, a couple of other things happened.  Could be unrelated or only
loosely related.

Evolution launches spamd (spamassassin) to filter junk mail.  spamd died
and I have this in dmesg to show for it:

VM: killing process spamd

spamd would have inherited the libhugetlbfs.so environment variables.
There are no other clues as to why it died though.

Also, immediately after launching evolution with libhugetlbfs, I got
that USB bug where the mouse starts creating keyboard input.  I got some
of these in dmesg:
keyboard.c: can't emulate rawmode for keycode 240

That could be pure coincidence, although I had been using the system
almost all day before that, and it hadn't happened.  
-- 
Zan Lynx <zlynx@acm.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: 2.6.22-rc1-mm1 huge pages VM freeze (maybe?)

[Nish Aravamudan]

On 8/1/07, Zan Lynx <zlynx@acm.org> wrote:
> On Wed, 2007-08-01 at 08:52 -0700, Nish Aravamudan wrote:
> > On 7/31/07, Zan Lynx <zlynx@acm.org> wrote:
> > > On Tue, 2007-07-31 at 15:02 -0700, Randy Dunlap wrote:
> > > > On Tue, 31 Jul 2007 15:44:21 -0600 Zan Lynx wrote:
> > > >
> > > > > I was playing with huge pages and libhugetlbfs.  Small programs like
> > > > > "ls" work fine.  I tried running Evolution through libhugetlbfs and the
> > > > > system slowly stops running.  One interesting thing is the "ps" command,
> > > > > it gets stuck like this:
> > > >
> > > > Do you mean 2.6.22-rc1-mm1 or 2.6.23-rc1-mm1?
> > >
> > > D'oh!  I mean 2.6.23-rc1-mm1, the 22 was a typo.  Cut & paste to be
> > > sure:
> > > Linux zephyr 2.6.23-rc1-mm1 #1 SMP PREEMPT Wed Jul 25 17:33:04 MDT 2007
> > > x86_64 AMD Athlon(tm) 64 Processor 3400+ AuthenticAMD GNU/Linux
> >
> > Just to confirm, still happens with -mm2?
>
> No, it does not seem to.  Evolution runs OK.  ps, top, pmap all work
> fine.

Interesting.

> However, a couple of other things happened.  Could be unrelated or only
> loosely related.
>
> Evolution launches spamd (spamassassin) to filter junk mail.  spamd died
> and I have this in dmesg to show for it:
>
> VM: killing process spamd
>
> spamd would have inherited the libhugetlbfs.so environment variables.
> There are no other clues as to why it died though.

Interesting. Any chance spamd can be run manually with those env
variables, but with HUGETLB_VERBOSE=99 (and/or in gdb) to see what
happens to it?

> Also, immediately after launching evolution with libhugetlbfs, I got
> that USB bug where the mouse starts creating keyboard input.  I got some
> of these in dmesg:
> keyboard.c: can't emulate rawmode for keycode 240
>
> That could be pure coincidence, although I had been using the system
> almost all day before that, and it hadn't happened.

Had you started evolution w/o libhugetlbs at all before that?

It does seem like that would be coincidence.

Thanks,
Nish

Re: 2.6.22-rc1-mm1 huge pages VM freeze (maybe?)

[Nish Aravamudan]

On 7/31/07, Zan Lynx <zlynx@acm.org> wrote:
> On Tue, 2007-07-31 at 15:02 -0700, Randy Dunlap wrote:
> > On Tue, 31 Jul 2007 15:44:21 -0600 Zan Lynx wrote:
> >
> > > I was playing with huge pages and libhugetlbfs.  Small programs like
> > > "ls" work fine.  I tried running Evolution through libhugetlbfs and the
> > > system slowly stops running.  One interesting thing is the "ps" command,
> > > it gets stuck like this:
> >
> > Do you mean 2.6.22-rc1-mm1 or 2.6.23-rc1-mm1?
>
> D'oh!  I mean 2.6.23-rc1-mm1, the 22 was a typo.  Cut & paste to be
> sure:
> Linux zephyr 2.6.23-rc1-mm1 #1 SMP PREEMPT Wed Jul 25 17:33:04 MDT 2007
> x86_64 AMD Athlon(tm) 64 Processor 3400+ AuthenticAMD GNU/Linux

Also, are we at all sure this isn't a reiser4 issue? I assume you're
able to use Evolution w/o libhuge on rc1-mm1 ok? Any chance to remove
reiser4 from the picture? Have you been using libhuge this way
regularly? Any chance you know it worked ok with some recent kernel
(say 2.6.23-rc1?).

Thanks,
Nish

Re: 2.6.22-rc1-mm1 huge pages VM freeze (maybe?)

[Nish Aravamudan]

On 7/31/07, Randy Dunlap <randy.dunlap@oracle.com> wrote:
> On Tue, 31 Jul 2007 15:44:21 -0600 Zan Lynx wrote:
>
> > I was playing with huge pages and libhugetlbfs.  Small programs like
> > "ls" work fine.  I tried running Evolution through libhugetlbfs and the
> > system slowly stops running.  One interesting thing is the "ps" command,
> > it gets stuck like this:
>
> Do you mean 2.6.22-rc1-mm1 or 2.6.23-rc1-mm1?
>
> There was a hugepage problem fixed very recently, in 2.6.23-rc1 IIRC.

Actually fixed just after 2.6.23-rc1:

git describe 5ab3ee7b1cd5c91eb2272764f9d7d1fe4749681e
v2.6.23-rc1-14-g5ab3ee7

Thanks,
Nish

Re: 2.6.22-rc1-mm1 huge pages VM freeze (maybe?)

[Randy Dunlap]

Nish Aravamudan wrote:
> On 7/31/07, Randy Dunlap <randy.dunlap@oracle.com> wrote:
>> On Tue, 31 Jul 2007 15:44:21 -0600 Zan Lynx wrote:
>>
>>> I was playing with huge pages and libhugetlbfs.  Small programs like
>>> "ls" work fine.  I tried running Evolution through libhugetlbfs and the
>>> system slowly stops running.  One interesting thing is the "ps" command,
>>> it gets stuck like this:
>> Do you mean 2.6.22-rc1-mm1 or 2.6.23-rc1-mm1?
>>
>> There was a hugepage problem fixed very recently, in 2.6.23-rc1 IIRC.
> 
> Actually fixed just after 2.6.23-rc1:
> 
> git describe 5ab3ee7b1cd5c91eb2272764f9d7d1fe4749681e
> v2.6.23-rc1-14-g5ab3ee7

Looks to me like Andrew included Ken's patch in his rc1-mm1 anyway,
so that shouldn't be the issue.  Or did I not read mm/hugetlb.c correctly?

-- 
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

Re: 2.6.22-rc1-mm1 huge pages VM freeze (maybe?)

[Nish Aravamudan]

On 8/1/07, Randy Dunlap <randy.dunlap@oracle.com> wrote:
> Nish Aravamudan wrote:
> > On 7/31/07, Randy Dunlap <randy.dunlap@oracle.com> wrote:
> >> On Tue, 31 Jul 2007 15:44:21 -0600 Zan Lynx wrote:
> >>
> >>> I was playing with huge pages and libhugetlbfs.  Small programs like
> >>> "ls" work fine.  I tried running Evolution through libhugetlbfs and the
> >>> system slowly stops running.  One interesting thing is the "ps" command,
> >>> it gets stuck like this:
> >> Do you mean 2.6.22-rc1-mm1 or 2.6.23-rc1-mm1?
> >>
> >> There was a hugepage problem fixed very recently, in 2.6.23-rc1 IIRC.
> >
> > Actually fixed just after 2.6.23-rc1:
> >
> > git describe 5ab3ee7b1cd5c91eb2272764f9d7d1fe4749681e
> > v2.6.23-rc1-14-g5ab3ee7
>
> Looks to me like Andrew included Ken's patch in his rc1-mm1 anyway,
> so that shouldn't be the issue.  Or did I not read mm/hugetlb.c correctly?

Yeah you're right, the -mm tree has that bug fixed.

Thanks,
Nish