bad pmd ffff810000207238(9090909090909090).

bad pmd ffff810000207238(9090909090909090).

[Fede]

Hi,

Today I tried to start a firewalling script and failed due to an 
unrelated issue, but when I checked the log I saw this:

May 27 20:38:15 kaoz ip_tables: (C) 2000-2006 Netfilter Core Team
May 27 20:38:28 kaoz Netfilter messages via NETLINK v0.30.
May 27 20:38:28 kaoz nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
May 27 20:38:28 kaoz ctnetlink v0.93: registering with nfnetlink.
May 27 20:38:28 kaoz ClusterIP Version 0.8 loaded successfully
May 27 20:38:28 kaoz mm/memory.c:127: bad pmd 
ffff810000207238(9090909090909090).

I also found another post with a very similar issue. The other post had 
almost the same message (*mm*/*memory*.*c*:*127*: *bad* *pmd* 
ffff810000207808(9090909090909090).)

Does anyone know what is it?

Re: bad pmd ffff810000207238(9090909090909090).

[Arjan van de Ven]

On Tue, 27 May 2008 23:01:01 -0300
Fede <fedux@lugmen.org.ar> wrote:

> Hi,
> 
> Today I tried to start a firewalling script and failed due to an 
> unrelated issue, but when I checked the log I saw this:
> 
> May 27 20:38:15 kaoz ip_tables: (C) 2000-2006 Netfilter Core Team
> May 27 20:38:28 kaoz Netfilter messages via NETLINK v0.30.
> May 27 20:38:28 kaoz nf_conntrack version 0.5.0 (16384 buckets, 65536
> max) May 27 20:38:28 kaoz ctnetlink v0.93: registering with nfnetlink.
> May 27 20:38:28 kaoz ClusterIP Version 0.8 loaded successfully
> May 27 20:38:28 kaoz mm/memory.c:127: bad pmd 
> ffff810000207238(9090909090909090).


909090 is asm for "nop nop nop", a pattern often used in rootkits to
deal with slightly unknown memory locations



I would REALLY REALLY strongly suggest that you check for rootkits and
the like on your system...



-- 
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

[PATCH] Re: bad pmd ffff810000207238(9090909090909090).

[Hugh Dickins]

On Tue, 27 May 2008, Fede wrote:
> 
> Today I tried to start a firewalling script and failed due to an unrelated
> issue, but when I checked the log I saw this:
> 
> May 27 20:38:15 kaoz ip_tables: (C) 2000-2006 Netfilter Core Team
> May 27 20:38:28 kaoz Netfilter messages via NETLINK v0.30.
> May 27 20:38:28 kaoz nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
> May 27 20:38:28 kaoz ctnetlink v0.93: registering with nfnetlink.
> May 27 20:38:28 kaoz ClusterIP Version 0.8 loaded successfully
> May 27 20:38:28 kaoz mm/memory.c:127: bad pmd
> ffff810000207238(9090909090909090).
> 
> I also found another post with a very similar issue. The other post had almost
> the same message (*mm*/*memory*.*c*:*127*: *bad* *pmd*
> ffff810000207808(9090909090909090).)
> 
> Does anyone know what is it?

Thanks a lot for re-reporting this: it was fun to work it out.
It's not a rootkit, it's harmless, but we ought to fix the noise.
Simple patch below, but let me explain more verbosely first.

What was really interesting in your report was that the address
is so close to that in OGAWA-San's report.  I had a look at that
page on my x86_64 boxes, and they have lots of 0x90s there too.
It's just some page alignment filler that x86_64 kernel startup
has missed cleaning up - patch below fixes that.  There's no
security aspect to it: the entries were already not-present,
they just generate this noise by triggering the pmd_bad test.

But why do you occasionally see those messages (I never have)?
I was puzzled awhile because those tests are usually for user
address space, but this is up in kernel address space: in the
modules area.  Ah, vmalloc.c uses them: it's almost coincidence,
but those user space tests do work on the vmalloc and modules
areas, though they'd fail on many other parts of the kernel
address space (because of pse and global and nx bits).

Those messages come just occasionally from unloading a module.
There's a years-old anomaly in vmalloc.c, that the allocation
routines add an empty guard page slot to the size, then the
freeing routines include that empty slot.  (I've always thought
the guard page should be private to vmalloc.c, consistently left
out of the public size; but that would need wider changes.)  When
it would occupy the first slot of a new page table, allocation won't
have assigned one, and freeing will then hit this "bad" pmd entry.

Hugh

[PATCH] x86: fix bad pmd ffff810000207xxx(9090909090909090)

OGAWA Hirofumi and Fede have reported rare pmd_ERROR messages:
mm/memory.c:127: bad pmd ffff810000207xxx(9090909090909090).

Initialization's cleanup_highmap was leaving alignment filler
behind in the pmd for MODULES_VADDR: when vmalloc's guard page
would occupy a new page table, it's not allocated, and then
module unload's vfree hits the bad 9090 pmd entry left over.

Signed-off-by: Hugh Dickins <hugh@veritas.com>
---

 arch/x86/mm/init_64.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

--- 2.6.26-rc4/arch/x86/mm/init_64.c	2008-05-03 21:54:41.000000000 +0100
+++ linux/arch/x86/mm/init_64.c	2008-05-28 17:38:19.000000000 +0100
@@ -206,7 +206,7 @@ void __init cleanup_highmap(void)
 	pmd_t *last_pmd = pmd + PTRS_PER_PMD;
 
 	for (; pmd < last_pmd; pmd++, vaddr += PMD_SIZE) {
-		if (!pmd_present(*pmd))
+		if (pmd_none(*pmd))
 			continue;
 		if (vaddr < (unsigned long) _text || vaddr > end)
 			set_pmd(pmd, __pmd(0));

Re: [PATCH] Re: bad pmd ffff810000207238(9090909090909090).

[Ingo Molnar]

* Hugh Dickins <hugh@veritas.com> wrote:

> [PATCH] x86: fix bad pmd ffff810000207xxx(9090909090909090)
> 
> OGAWA Hirofumi and Fede have reported rare pmd_ERROR messages: 
> mm/memory.c:127: bad pmd ffff810000207xxx(9090909090909090).
> 
> Initialization's cleanup_highmap was leaving alignment filler behind 
> in the pmd for MODULES_VADDR: when vmalloc's guard page would occupy a 
> new page table, it's not allocated, and then module unload's vfree 
> hits the bad 9090 pmd entry left over.

applied, nice catch!

	Ingo

Re: [PATCH] Re: bad pmd ffff810000207238(9090909090909090).

[Willy Tarreau]

On Wed, May 28, 2008 at 07:36:07PM +0100, Hugh Dickins wrote:
> On Tue, 27 May 2008, Fede wrote:
> > 
> > Today I tried to start a firewalling script and failed due to an unrelated
> > issue, but when I checked the log I saw this:
> > 
> > May 27 20:38:15 kaoz ip_tables: (C) 2000-2006 Netfilter Core Team
> > May 27 20:38:28 kaoz Netfilter messages via NETLINK v0.30.
> > May 27 20:38:28 kaoz nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
> > May 27 20:38:28 kaoz ctnetlink v0.93: registering with nfnetlink.
> > May 27 20:38:28 kaoz ClusterIP Version 0.8 loaded successfully
> > May 27 20:38:28 kaoz mm/memory.c:127: bad pmd
> > ffff810000207238(9090909090909090).
> > 
> > I also found another post with a very similar issue. The other post had almost
> > the same message (*mm*/*memory*.*c*:*127*: *bad* *pmd*
> > ffff810000207808(9090909090909090).)
> > 
> > Does anyone know what is it?
> 
> Thanks a lot for re-reporting this: it was fun to work it out.
> It's not a rootkit, it's harmless, but we ought to fix the noise.
> Simple patch below, but let me explain more verbosely first.
> 
> What was really interesting in your report was that the address
> is so close to that in OGAWA-San's report.  I had a look at that
> page on my x86_64 boxes, and they have lots of 0x90s there too.
> It's just some page alignment filler that x86_64 kernel startup
> has missed cleaning up - patch below fixes that.  There's no
> security aspect to it: the entries were already not-present,
> they just generate this noise by triggering the pmd_bad test.

Is there a particular reason we use 0x90 as an alignment filler ?
If we can put anything else, at least next time it will not get
confused with NOPs. We could use 0xAF (Alignment Filler) for
instance.

Well done BTW ;-)

Reagrds,
Willy

Re: [PATCH] Re: bad pmd ffff810000207238(9090909090909090).

[Jan Engelhardt]

On Wednesday 2008-05-28 21:56, Willy Tarreau wrote:
>On Wed, May 28, 2008 at 07:36:07PM +0100, Hugh Dickins wrote:
>> On Tue, 27 May 2008, Fede wrote:
>> > 
>> > Today I tried to start a firewalling script and failed due to an unrelated
>> > issue, but when I checked the log I saw this:
>> > 
>> > May 27 20:38:15 kaoz ip_tables: (C) 2000-2006 Netfilter Core Team
>> > May 27 20:38:28 kaoz Netfilter messages via NETLINK v0.30.
>> > May 27 20:38:28 kaoz nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
>> > May 27 20:38:28 kaoz ctnetlink v0.93: registering with nfnetlink.
>> > May 27 20:38:28 kaoz ClusterIP Version 0.8 loaded successfully
>> > May 27 20:38:28 kaoz mm/memory.c:127: bad pmd
>> > ffff810000207238(9090909090909090).
>> > 
>> > I also found another post with a very similar issue. The other post had almost
>> > the same message (*mm*/*memory*.*c*:*127*: *bad* *pmd*
>> > ffff810000207808(9090909090909090).)
>> > 
>> > Does anyone know what is it?
>> 
>> Thanks a lot for re-reporting this: it was fun to work it out.
>> It's not a rootkit, it's harmless, but we ought to fix the noise.
>> Simple patch below, but let me explain more verbosely first.
>> 
>> What was really interesting in your report was that the address
>> is so close to that in OGAWA-San's report.  I had a look at that
>> page on my x86_64 boxes, and they have lots of 0x90s there too.
>> It's just some page alignment filler that x86_64 kernel startup
>> has missed cleaning up - patch below fixes that.  There's no
>> security aspect to it: the entries were already not-present,
>> they just generate this noise by triggering the pmd_bad test.
>
>Is there a particular reason we use 0x90 as an alignment filler ?

Alignment within functions. You could use a JMP to jump over
the alignment, but that would be costly. So in order to
"run through the wall", you need an opcode that does not
do anything, something like 0x90.
0xAF would map to scasd on x86, and I'd hardly call that a
no-op.

Re: [PATCH] Re: bad pmd ffff810000207238(9090909090909090).

[Willy Tarreau]

On Wed, May 28, 2008 at 10:14:31PM +0200, Jan Engelhardt wrote:
> 
> On Wednesday 2008-05-28 21:56, Willy Tarreau wrote:
> >On Wed, May 28, 2008 at 07:36:07PM +0100, Hugh Dickins wrote:
> >> On Tue, 27 May 2008, Fede wrote:
> >> > 
> >> > Today I tried to start a firewalling script and failed due to an unrelated
> >> > issue, but when I checked the log I saw this:
> >> > 
> >> > May 27 20:38:15 kaoz ip_tables: (C) 2000-2006 Netfilter Core Team
> >> > May 27 20:38:28 kaoz Netfilter messages via NETLINK v0.30.
> >> > May 27 20:38:28 kaoz nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
> >> > May 27 20:38:28 kaoz ctnetlink v0.93: registering with nfnetlink.
> >> > May 27 20:38:28 kaoz ClusterIP Version 0.8 loaded successfully
> >> > May 27 20:38:28 kaoz mm/memory.c:127: bad pmd
> >> > ffff810000207238(9090909090909090).
> >> > 
> >> > I also found another post with a very similar issue. The other post had almost
> >> > the same message (*mm*/*memory*.*c*:*127*: *bad* *pmd*
> >> > ffff810000207808(9090909090909090).)
> >> > 
> >> > Does anyone know what is it?
> >> 
> >> Thanks a lot for re-reporting this: it was fun to work it out.
> >> It's not a rootkit, it's harmless, but we ought to fix the noise.
> >> Simple patch below, but let me explain more verbosely first.
> >> 
> >> What was really interesting in your report was that the address
> >> is so close to that in OGAWA-San's report.  I had a look at that
> >> page on my x86_64 boxes, and they have lots of 0x90s there too.
> >> It's just some page alignment filler that x86_64 kernel startup
> >> has missed cleaning up - patch below fixes that.  There's no
> >> security aspect to it: the entries were already not-present,
> >> they just generate this noise by triggering the pmd_bad test.
> >
> >Is there a particular reason we use 0x90 as an alignment filler ?
> 
> Alignment within functions. You could use a JMP to jump over
> the alignment, but that would be costly. So in order to
> "run through the wall", you need an opcode that does not
> do anything, something like 0x90.
> 0xAF would map to scasd on x86, and I'd hardly call that a
> no-op.

OK, I did not understand from Hugh's explanation that it was
all about alignment within functions. Of course, 0x90 is fine
there (though there are multi-byte NOPs available).

Cheers,
Willy

Re: [PATCH] Re: bad pmd ffff810000207238(9090909090909090).

[Jan Engelhardt]

On Wednesday 2008-05-28 22:43, Willy Tarreau wrote:
>> >
>> >Is there a particular reason we use 0x90 as an alignment filler ?
>> 
>> Alignment within functions. You could use a JMP to jump over
>> the alignment, but that would be costly. So in order to
>> "run through the wall", you need an opcode that does not
>> do anything, something like 0x90.
>> 0xAF would map to scasd on x86, and I'd hardly call that a
>> no-op.
>
>OK, I did not understand from Hugh's explanation that it was
>all about alignment within functions. Of course, 0x90 is fine
>there (though there are multi-byte NOPs available).

"All about alignment within functions" -- I am not sure about that,
you just happened to ask about 0x90 :)
And if you have a 1-byte NOP (which fits perfectly everywhere),
which is also a real NOP (and not just a filler byte that could
possibly be an opcode doing something very different), you've
got an ideal candidate for padding, no?
There is probably nothing wrong with padding .data sections
with 0xAF or even 0xDB and ud2 to catch execute-readonly-data
cases. To that end, I think something like that should be
proposed to binutils.

Re: [PATCH] Re: bad pmd ffff810000207238(9090909090909090).

[Hugh Dickins]

On Wed, 28 May 2008, Willy Tarreau wrote:
> On Wed, May 28, 2008 at 10:14:31PM +0200, Jan Engelhardt wrote:
> > On Wednesday 2008-05-28 21:56, Willy Tarreau wrote:
> > >On Wed, May 28, 2008 at 07:36:07PM +0100, Hugh Dickins wrote:
> > >> 
> > >> page on my x86_64 boxes, and they have lots of 0x90s there too.
> > >> It's just some page alignment filler that x86_64 kernel startup
> > >> has missed cleaning up - patch below fixes that.  There's no
> > >
> > >Is there a particular reason we use 0x90 as an alignment filler ?
> > 
> > Alignment within functions. You could use a JMP to jump over
> > the alignment, but that would be costly. So in order to
> > "run through the wall", you need an opcode that does not
> > do anything, something like 0x90.
> 
> OK, I did not understand from Hugh's explanation that it was
> all about alignment within functions. Of course, 0x90 is fine
> there (though there are multi-byte NOPs available).

I'm hardly the right person to answer on this, but I believe that
because the 0x90 NOP is particularly appropriate in .text (and
prevents stalls even where it cannot be reached?), it gets used
as the default alignment filler all over.   (It's even specified
for the arch-independent default _ALIGN in linux/linkage.h.)

Hugh