Re: remote memory reading using arp?

Re: remote memory reading using arp?

[Andi Kleen]

Bryan Rittmeyer <bryan@ixiacom.com> writes:

> It's not the ARP layer that's causing the padding... Ethernet has a
> minimum transmit size of 64 bytes (everything below that is disgarded
> by hardware as a fragment), so the network device driver or
> the hardware itself will pad any Linux skb smaller than 60 bytes up to
> that size (so that it's 64 bytes after appending CRC32). Apparently, in
> some cases that's done by just transmitting whatever uninitialized
> memory follows skb->data, which, after the system has been running
> for a while, may be inside a page previously used by userspace.

The driver should be fixed in that case. I would consider it a driver
bug. The cost of clearing the tail should be minimal, it is at most
two cache lines.

Is it known which driver caused this?

> 
> This is NOT a "remote memory reading" exploit, since there is no way to

It really is. 


-Andi

Re: remote memory reading using arp?

[warchild]

> The driver should be fixed in that case. I would consider it a driver
> bug. The cost of clearing the tail should be minimal, it is at most
> two cache lines.
> 
> Is it known which driver caused this?

My testing yesterday was done with a Xircom RBEM58G-100 (10/100 + 56k) and an
Aironet PC4800 (with xircom_tulip_cb and airo/airo_cs, respectively).
Since this may be a driver problem, I should note that the airo drivers are
from pcmcia-cs-3.1.33.

 
> > This is NOT a "remote memory reading" exploit, since there is no way to
> 
> It really is. 

I'm finding it really difficult to replicate this problem today, but if I
find anything else that may be of use to help shed more light on this
issue, I'll be sure and share it.

thanks

Re: remote memory reading using arp?

[Calin A. Culianu]

On 28 Apr 2002, Andi Kleen wrote:

> Bryan Rittmeyer <bryan@ixiacom.com> writes:
>
> > It's not the ARP layer that's causing the padding... Ethernet has a
> > minimum transmit size of 64 bytes (everything below that is disgarded
> > by hardware as a fragment), so the network device driver or
> > the hardware itself will pad any Linux skb smaller than 60 bytes up to
> > that size (so that it's 64 bytes after appending CRC32). Apparently, in
> > some cases that's done by just transmitting whatever uninitialized
> > memory follows skb->data, which, after the system has been running
> > for a while, may be inside a page previously used by userspace.
>
> The driver should be fixed in that case. I would consider it a driver
> bug. The cost of clearing the tail should be minimal, it is at most

Yes, I wholeheartedly agree.  Also, the notion that it's userspace's
responsibility to clear memory before exiting is preposterous.  That would
involve just about every piece of software ever made to be rewritten (you
could change glibc to clear memory on free()s but what about the stack?).


> two cache lines.
>
> Is it known which driver caused this?
>
> >
> > This is NOT a "remote memory reading" exploit, since there is no way to
>
> It really is.

Yes.  If you snoop long enough, it becomes possible to read passords and
other sensitive data.  Granted, this is only on the local network, but
it is still a major hole.

>
>
> -Andi
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: remote memory reading using arp?

[Andi Kleen]

On Mon, Apr 29, 2002 at 11:24:04AM -0400, Calin A. Culianu wrote:
> On 28 Apr 2002, Andi Kleen wrote:
> 
> > Bryan Rittmeyer <bryan@ixiacom.com> writes:
> >
> > > It's not the ARP layer that's causing the padding... Ethernet has a
> > > minimum transmit size of 64 bytes (everything below that is disgarded
> > > by hardware as a fragment), so the network device driver or
> > > the hardware itself will pad any Linux skb smaller than 60 bytes up to
> > > that size (so that it's 64 bytes after appending CRC32). Apparently, in
> > > some cases that's done by just transmitting whatever uninitialized
> > > memory follows skb->data, which, after the system has been running
> > > for a while, may be inside a page previously used by userspace.
> >
> > The driver should be fixed in that case. I would consider it a driver
> > bug. The cost of clearing the tail should be minimal, it is at most
> 
> Yes, I wholeheartedly agree.  Also, the notion that it's userspace's
> responsibility to clear memory before exiting is preposterous.  That would
> involve just about every piece of software ever made to be rewritten (you
> could change glibc to clear memory on free()s but what about the stack?).

It actually requires more changes. The skbuff allocator needs to 
be fixed too to ensure a 64 bytes minimum length of the skb. 
(or alternatively if you don't want to penalize non ethernet protocols
read minlen from a dev-> field similar to hard_header_len and compute it 
in the caller, but that's likely overkill) 

But should be done. 

-Andi

Re: remote memory reading using arp?

[warchild]

Greetings again,

I took the time today to gather as much arp traffic as my eyes could handle and
enough to shed some valuable light on this issue.

While it doesn't show who/what is at fault, it may possibly prove that I'm
missing something entirely or the problem is more widespread that was
thought.

Anyway, here goes.   

For test 1, I gathered ~1.5M of arp traffic from a monitoring port that
sustains 4-5Mb/s traffic in a mixed solaris/linux/win2k environment of
approximately 400 machines in an educational setting.

For test 2, I gathered ~.5M of arp traffic from a single interface of a
RedHat machine located in a "Small Business" environment consisting of
approximately 25 machines (linux/winXP).  

That much arp data is pretty unruly, so I used grep to see if anything
stuck out.  

I grepped for our domain name (ccs.neu.edu) and found this string 15 times in
test 1, and grepped for 'http' in test 2 and found that string 62 times.

Upon digging into the traffic a bit further, I saw an interesting trends.
All of the arp packets that contained interesting data contained it in the
last 18 bytes of the 60 byte arp packet.  After googling and browsing the
rfcs, I've seen these last 18 bytes referred to as both 'trailers' and
'padding'.  It is not clear to me what purpose they serve, but seems clear
that they can contain some potentially sensitive data.  

I know this may be getting a bit off topic, but I figured I share my
findings with the lists.  If I am incorrect in any of my statements, please
correct me.

thanks,

-jon 

Dirty memory? (WAS Re: remote memory reading using arp?)

[Calin A. Culianu]

On Mon, 29 Apr 2002, Andi Kleen wrote:

> On Mon, Apr 29, 2002 at 11:24:04AM -0400, Calin A. Culianu wrote:
> > On 28 Apr 2002, Andi Kleen wrote:
> >
> > > Bryan Rittmeyer <bryan@ixiacom.com> writes:
> > >
> > > > It's not the ARP layer that's causing the padding... Ethernet has a
> > > > minimum transmit size of 64 bytes (everything below that is disgarded
> > > > by hardware as a fragment), so the network device driver or
> > > > the hardware itself will pad any Linux skb smaller than 60 bytes up to
> > > > that size (so that it's 64 bytes after appending CRC32). Apparently, in
> > > > some cases that's done by just transmitting whatever uninitialized
> > > > memory follows skb->data, which, after the system has been running
> > > > for a while, may be inside a page previously used by userspace.
> > >
> > > The driver should be fixed in that case. I would consider it a driver
> > > bug. The cost of clearing the tail should be minimal, it is at most
> >
> > Yes, I wholeheartedly agree.  Also, the notion that it's userspace's
> > responsibility to clear memory before exiting is preposterous.  That would
> > involve just about every piece of software ever made to be rewritten (you
> > could change glibc to clear memory on free()s but what about the stack?).
>
> It actually requires more changes. The skbuff allocator needs to
> be fixed too to ensure a 64 bytes minimum length of the skb.
> (or alternatively if you don't want to penalize non ethernet protocols
> read minlen from a dev-> field similar to hard_header_len and compute it
> in the caller, but that's likely overkill)

Well, that sounds pretty elegant.  After all, it makes sense for the
caller to ensure the data field in his skbuff is at least as big as the
(MTU? is this the field we want in struct net_device?) for his dev...

>
> But should be done.

I am glad you think so.  More generally, once can make the argument that
to be 100% secure, one needs to revisit a lot of the kernel memory
allocations (not just in the networking code) and see if the following two
criteria are met:  any of the memory allocated could potentially be/remain
'dirty' for a while && there is a possibility that the contents of that
memory could make their way into 'untrusted' hands.  Things like giving
the dirty memory to userspace or sending portions of it down the network
wire are examples of placing it in untrusted hands.

I am not sure if this is a can of worms worth opening.  While it would be
nice for the kernel to reliably provide guarantees about memory and its
'cleanliness' and privateness (after all this is one third the reason for
memory protection in the first place!), it may be a lot of trouble
performancewise....  however it still is an interesting problem to look
at...

I wonder if any other operating systems have addressed the issue of memory
allocators and the possibility that the memory they return may contain
sensitive data?

-Calin

>
> -Andi
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: Dirty memory? (WAS Re: remote memory reading using arp?)

[Calin A. Culianu]

Actually nevermind.  It looks like memory allocated for userspace is
always zeroed out... hehe don't I feel dumb..

-Calin

On Wed, 1 May 2002, Calin A. Culianu wrote:

> On Mon, 29 Apr 2002, Andi Kleen wrote:
>
> > On Mon, Apr 29, 2002 at 11:24:04AM -0400, Calin A. Culianu wrote:
> > > On 28 Apr 2002, Andi Kleen wrote:
> > >
> > > > Bryan Rittmeyer <bryan@ixiacom.com> writes:
> > > >
> > > > > It's not the ARP layer that's causing the padding... Ethernet has a
> > > > > minimum transmit size of 64 bytes (everything below that is disgarded
> > > > > by hardware as a fragment), so the network device driver or
> > > > > the hardware itself will pad any Linux skb smaller than 60 bytes up to
> > > > > that size (so that it's 64 bytes after appending CRC32). Apparently, in
> > > > > some cases that's done by just transmitting whatever uninitialized
> > > > > memory follows skb->data, which, after the system has been running
> > > > > for a while, may be inside a page previously used by userspace.
> > > >
> > > > The driver should be fixed in that case. I would consider it a driver
> > > > bug. The cost of clearing the tail should be minimal, it is at most
> > >
> > > Yes, I wholeheartedly agree.  Also, the notion that it's userspace's
> > > responsibility to clear memory before exiting is preposterous.  That would
> > > involve just about every piece of software ever made to be rewritten (you
> > > could change glibc to clear memory on free()s but what about the stack?).
> >
> > It actually requires more changes. The skbuff allocator needs to
> > be fixed too to ensure a 64 bytes minimum length of the skb.
> > (or alternatively if you don't want to penalize non ethernet protocols
> > read minlen from a dev-> field similar to hard_header_len and compute it
> > in the caller, but that's likely overkill)
>
> Well, that sounds pretty elegant.  After all, it makes sense for the
> caller to ensure the data field in his skbuff is at least as big as the
> (MTU? is this the field we want in struct net_device?) for his dev...
>
> >
> > But should be done.
>
> I am glad you think so.  More generally, once can make the argument that
> to be 100% secure, one needs to revisit a lot of the kernel memory
> allocations (not just in the networking code) and see if the following two
> criteria are met:  any of the memory allocated could potentially be/remain
> 'dirty' for a while && there is a possibility that the contents of that
> memory could make their way into 'untrusted' hands.  Things like giving
> the dirty memory to userspace or sending portions of it down the network
> wire are examples of placing it in untrusted hands.
>
> I am not sure if this is a can of worms worth opening.  While it would be
> nice for the kernel to reliably provide guarantees about memory and its
> 'cleanliness' and privateness (after all this is one third the reason for
> memory protection in the first place!), it may be a lot of trouble
> performancewise....  however it still is an interesting problem to look
> at...
>
> I wonder if any other operating systems have addressed the issue of memory
> allocators and the possibility that the memory they return may contain
> sensitive data?
>
> -Calin
>
> >
> > -Andi
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/
> >
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

remote memory reading using arp?

[Warchild]

greetings,

While doing some work earlier today, I happened to have a window open on my
(Slackware 8) 2.4.16 box with tcpdump running.  Since I really wasn't doing
anything on the laptop the only traffic that was flying by via tcpdump was
the occassional ntp sync and some arp traffic here and there.

Out of the corner of my eye, I happened to notice something that seemed
very out of place -- human readable text included as part of the arp
packet.  Ok, so maybe the chances of that aren't _that_ slim, but the text
was familiar to me.  To see if this was just a one time thing or not, I
fired up tcpdump and told it to capture all arp traffic (tcpdump -ni eth0
-w /tmp/arp arp).  I did this 3 times while browsing the web, scp'ing stuff
around and screwing around on my laptop.  

I then replayed the tcpdump sessions and used the -X option (dumps the
packet in emacs-hexl like format), piped them to less and looked for
anything suspicous.  Here's some examples of what I found:

(pardon the paste)

15:20:09.186852 0:40:96:15:16:4f ff:ff:ff:ff:ff:ff 0806 60: arp who-has
192.168.1.1 tell 192.168.1.2
  0000: 0001 0800 0604 0001 0040 9615 164f c0a8  .........@...Oˬ
  0010: 0102 0000 0000 0000 c0a8 0101 652f 656e  ........ˬ..e/en
  0020: 6c69 6768 7465 6e6d 656e 742f 7468       lightenment/th


15:20:11.187501 0:6:25:3:ce:65 0:40:96:15:16:4f 0806 60: arp reply
192.168.1.1 is-at 0:6:25:3:ce:65
  0000: 0001 0800 0604 0002 0006 2503 ce65 c0a8  ..........%.ÎeÀ¨
  0010: 0101 0040 9615 164f c0a8 0102 6874 656e  ...@...Oˬ..hten
  0020: 6d65 6e74 2f74 6865 6d65 732f 4761       ment/themes/Ga

15:20:12.186895 0:40:96:15:16:4f ff:ff:ff:ff:ff:ff 0806 60: arp who-has
192.168.1.1 tell 192.168.1.2
  0000: 0001 0800 0604 0001 0040 9615 164f c0a8  .........@...Oˬ
  0010: 0102 0000 0000 0000 c0a8 0101 705f 632e  ........ˬ..p_c.
  0020: 706e 670a 2f75 7372 2f58 3131 5236       png./usr/X11R6
 
15:00:55.970007 0:6:25:3:ce:65 0:40:96:15:16:4f 0806 60: arp reply
192.168.1.1 is-at 0:6:25:3:ce:65
  0000: 0001 0800 0604 0002 0006 2503 ce65 c0a8  ..........%.ÎeÀ¨
  0010: 0101 0040 9615 164f c0a8 0102 6574 2d31  ...@...Oˬ..et-1
  0020: 2e30 2e32 612f 6269 6e2f 0500 0000       .0.2a/bin/....


15:03:07.290878 0:6:25:3:ce:65 0:40:96:15:16:4f 0806 60: arp reply
192.168.1.1 is-at 0:6:25:3:ce:65
  0000: 0001 0800 0604 0002 0006 2503 ce65 c0a8  ..........%.ÎeÀ¨
  0010: 0101 0040 9615 164f c0a8 0102 0d00 1d00  ...@...Oˬ......
  0020: 696e 6572 4861 6e64 6c65 722e 6874       inerHandler.ht


192.168.1.2 is a Slackware 8 box with 2.4.16, and 192.168.1.1 is an OpenBSD
3.1-current box.  It struck me as extremely strange that the contents of
the arp packets contained information about my laptop.   To ensure that
this wasn't just tcpdump acting dumb, I left tcpdump running while I wasn't
doing anything on the laptop at all.  I got similar results.  

I couldn't find anything in the archives about this, and also didn't see
any changes in the arp implementation of the 2.4 kernel between 2.4.16 and
2.4.18.  I also browsed rfc826 to see if there was any mention of 'padding
data', but nothing caught my eye.

Any ideas what is causing this?

thanks in advance,

-warchild

Re: remote memory reading using arp?

[Bryan Rittmeyer]

Warchild wrote:
 > [oh my god, i see userspace text strings in ARP packets]
 >
> I couldn't find anything in the archives about this, and also didn't see
> any changes in the arp implementation of the 2.4 kernel between 2.4.16 and
> 2.4.18.  I also browsed rfc826 to see if there was any mention of 'padding
> data', but nothing caught my eye.
> 
> Any ideas what is causing this?

It's not the ARP layer that's causing the padding... Ethernet has a
minimum transmit size of 64 bytes (everything below that is disgarded
by hardware as a fragment), so the network device driver or
the hardware itself will pad any Linux skb smaller than 60 bytes up to
that size (so that it's 64 bytes after appending CRC32). Apparently, in
some cases that's done by just transmitting whatever uninitialized
memory follows skb->data, which, after the system has been running
for a while, may be inside a page previously used by userspace.

This is NOT a "remote memory reading" exploit, since there is no way to
remotely control what address in memory gets used as padding. I guess
you could packet blast a machine and hope to find something
interesting, but that'd be a denial of service attack long before you
got a complete view of system memory. In any case, it's arguably
userspace's responsibility to clear any sensitive memory contents
before exiting. I would be more concerned if you can find data
from currently in use, userspace-allocated pages flying out as packet
padding (i.e. if reading past skb->data pushes you into somebody else's
page, which seems unlikely since new skb's tend to get allocated near
the beginning of a page).

If you are really concerned you could probably patch the network driver
to zero out memory that will be used as padding, though I don't think
the security risk justifies that performance hit.

-Bryan

Re: remote memory reading using arp?

[Warchild]

On Sat, Apr 27, 2002 at 01:48:43PM -0700, Bryan Rittmeyer wrote:

> > [oh my god, i see userspace text strings in ARP packets]

*sigh*


> It's not the ARP layer that's causing the padding... Ethernet has a
> minimum transmit size of 64 bytes (everything below that is disgarded
> by hardware as a fragment), so the network device driver or
> the hardware itself will pad any Linux skb smaller than 60 bytes up to
> that size (so that it's 64 bytes after appending CRC32). Apparently, in
> some cases that's done by just transmitting whatever uninitialized
> memory follows skb->data, which, after the system has been running
> for a while, may be inside a page previously used by userspace.

That makes perfect sense.  Thanks for the explanation.

> This is NOT a "remote memory reading" exploit, since there is no way to
> remotely control what address in memory gets used as padding. I guess
> you could packet blast a machine and hope to find something
> interesting, but that'd be a denial of service attack long before you
> got a complete view of system memory. In any case, it's arguably
> userspace's responsibility to clear any sensitive memory contents
> before exiting. I would be more concerned if you can find data
> from currently in use, userspace-allocated pages flying out as packet
> padding (i.e. if reading past skb->data pushes you into somebody else's
> page, which seems unlikely since new skb's tend to get allocated near
> the beginning of a page).

Correct.  It'd take far too long and I'd go cross-eyed long before I got
anything other than useless garbage.  Isn't this similar to the bug within
the last year that dealt with userland memory disclosure via tcp/icmp?  What
was the "verdict" with that?

Please note that nowhere in my email did I use the word exploit.
 

> If you are really concerned you could probably patch the network driver
> to zero out memory that will be used as padding, though I don't think
> the security risk justifies that performance hit.

Agreed.  It doesn't bother me much at all.  I was just curious what was
going on.

thanks, and keep up the good work,

-warchild