ext3 hacked filesystem (by debian exim4 exploit) available for analysis and bugreporting

ext3 hacked filesystem (by debian exim4 exploit) available for analysis and bugreporting

[Luke Kenneth Casson Leighton]

folks, hi,

i appreciate this was some time ago, but i encountered a quite serious
issue with an ext3 filesystem that had been hacked, and a rootkit
installed.  this was with a 2.6.26 kernel.  the issue encountered was
that the little fuckers directly modified the ext3 filesystem so that
some files they had created could *not* be deleted.  when i say "could
not be deleted" i mean "absolutely could not be deleted".  also, fsck
did *not* report any "problems".

and yes, please do give me credit for knowing that you should use a
different system (offline) to analyse the [damaged] filesystem :)  as
you can imagine, i was very very surprised to encounter this as an
issue.

first thing: has anyone else encountered this?

second thing: if answer "no" to above, would anyone who can prove
their credentials (public ssh key, public web site, notability blah
blah) like to analyse the 5gb filesystem?  i still have a copy (it's
an LVM2 partition on a Xen hosted server).

apart from anything, this really really should go into rkhunter /
chkrootkit, but it requires someone with expertise to actually analyse
what the bloody hell happened.  apart from anything, files which
cannot be deleted (and cannot be detected as "corrupted" by fsck.ext3)
is pretty damn serious.

l.

Re: ext3 hacked filesystem (by debian exim4 exploit) available for analysis and bugreporting

[Matthias Schniedermeyer]

On 25.07.2011 13:08, Luke Kenneth Casson Leighton wrote:
> folks, hi,
> 
> apart from anything, files which cannot be deleted (and cannot be 
> detected as "corrupted" by fsck.ext3) is pretty damn serious.

You did try lsattr and checked that the files aren't 'immutable'?




Bis denn

-- 
Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.

Re: ext3 hacked filesystem (by debian exim4 exploit) available for analysis and bugreporting

[Luke Kenneth Casson Leighton]

On Mon, Jul 25, 2011 at 2:45 PM, Matthias Schniedermeyer <ms@citd.de> wrote:
> On 25.07.2011 13:08, Luke Kenneth Casson Leighton wrote:
>> folks, hi,
>>
>> apart from anything, files which cannot be deleted (and cannot be
>> detected as "corrupted" by fsck.ext3) is pretty damn serious.
>
> You did try lsattr and checked that the files aren't 'immutable'?

 i didn't! :)  didn't know about (but should have guessed) ext3
attributes.  they are indeed - thank you matthias.

root@quietbaby:/mnt/horsebox/tmp3# lsattr *
----ia------------- bin3/kill
----ia------------- bin3/ps
----ia------------- c.pl
----ia------------- e.conf
----ia------------- sbin3/sysctl
----ia------------- usrbin3/uptime
----ia------------- usrbin3/tload
----ia------------- usrbin3/free
----ia------------- usrbin3/top
----ia------------- usrbin3/vmstat
----ia------------- usrbin3/watch
----ia------------- usrbin3/skill
----ia------------- usrbin3/pmap
----ia------------- usrbin3/pgrep
----ia------------- usrbin3/slabtop
----ia------------- usrbin3/pwdx
----ia------------- usrbin3/snice
----ia------------- usrbin3/pkill
----ia------------- usrbin3/w

so - looks like it's not as bad as i thought.

apologies for taking up peoples' time with this.

 l.

Re: ext3 hacked filesystem (by debian exim4 exploit) available for analysis and bugreporting

[Pavel Machek]

On Mon 2011-07-25 22:08:24, Luke Kenneth Casson Leighton wrote:
> On Mon, Jul 25, 2011 at 2:45 PM, Matthias Schniedermeyer <ms@citd.de> wrote:
> > On 25.07.2011 13:08, Luke Kenneth Casson Leighton wrote:
> >> folks, hi,
> >>
> >> apart from anything, files which cannot be deleted (and cannot be
> >> detected as "corrupted" by fsck.ext3) is pretty damn serious.
> >
> > You did try lsattr and checked that the files aren't 'immutable'?
> 
>  i didn't! :)  didn't know about (but should have guessed) ext3
> attributes.  they are indeed - thank you matthias.
> 
> root@quietbaby:/mnt/horsebox/tmp3# lsattr *
> ----ia------------- bin3/kill
> ----ia------------- bin3/ps
> ----ia------------- c.pl
> ----ia------------- e.conf
> ----ia------------- sbin3/sysctl
> ----ia------------- usrbin3/uptime
> ----ia------------- usrbin3/tload
> ----ia------------- usrbin3/free
> ----ia------------- usrbin3/top
> ----ia------------- usrbin3/vmstat
> ----ia------------- usrbin3/watch
> ----ia------------- usrbin3/skill
> ----ia------------- usrbin3/pmap
> ----ia------------- usrbin3/pgrep
> ----ia------------- usrbin3/slabtop
> ----ia------------- usrbin3/pwdx
> ----ia------------- usrbin3/snice
> ----ia------------- usrbin3/pkill
> ----ia------------- usrbin3/w
> 
> so - looks like it's not as bad as i thought.

Should ls -l be moddified to show something when file has immutable
(and friends) set?
								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: ext3 hacked filesystem (by debian exim4 exploit) available for analysis and bugreporting

[Matthias Schniedermeyer]

On 29.07.2011 21:59, Pavel Machek wrote:
> On Mon 2011-07-25 22:08:24, Luke Kenneth Casson Leighton wrote:
> > On Mon, Jul 25, 2011 at 2:45 PM, Matthias Schniedermeyer <ms@citd.de> wrote:
> > > On 25.07.2011 13:08, Luke Kenneth Casson Leighton wrote:
> > >> folks, hi,
> > >>
> > >> apart from anything, files which cannot be deleted (and cannot be
> > >> detected as "corrupted" by fsck.ext3) is pretty damn serious.
> > >
> > > You did try lsattr and checked that the files aren't 'immutable'?
> > 
> >  i didn't! :)  didn't know about (but should have guessed) ext3
> > attributes.  they are indeed - thank you matthias.
> > 
> > root@quietbaby:/mnt/horsebox/tmp3# lsattr *
> > ----ia------------- bin3/kill
> > ----ia------------- bin3/ps
> > ----ia------------- c.pl
> > ----ia------------- e.conf
> > ----ia------------- sbin3/sysctl
> > ----ia------------- usrbin3/uptime
> > ----ia------------- usrbin3/tload
> > ----ia------------- usrbin3/free
> > ----ia------------- usrbin3/top
> > ----ia------------- usrbin3/vmstat
> > ----ia------------- usrbin3/watch
> > ----ia------------- usrbin3/skill
> > ----ia------------- usrbin3/pmap
> > ----ia------------- usrbin3/pgrep
> > ----ia------------- usrbin3/slabtop
> > ----ia------------- usrbin3/pwdx
> > ----ia------------- usrbin3/snice
> > ----ia------------- usrbin3/pkill
> > ----ia------------- usrbin3/w
> > 
> > so - looks like it's not as bad as i thought.
> 
> Should ls -l be moddified to show something when file has immutable
> (and friends) set?

AFAICT lsattr, in e2fsprogs, only does a 'stat'(lib/e2p/fgetflags.c) and 
checks st_flags, which i can't see in the "man 2 stat"-man-page i have 
installed, but nonetheless that is what it appears to do.

So assuming there are no incompatibilites between filesystems, the 
information appears to come "free" with the stat(s) that ls has to do 
anyway. (In the sense that there doesn't appear to any excessive 
overhead involved, especially no additional I/O).

So i would say: definitly.




Bis denn

-- 
Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.

Re: ext3 hacked filesystem (by debian exim4 exploit) available for analysis and bugreporting

[Pádraig Brady]

On 07/29/2011 10:31 PM, Matthias Schniedermeyer wrote:
> On 29.07.2011 21:59, Pavel Machek wrote:
>> On Mon 2011-07-25 22:08:24, Luke Kenneth Casson Leighton wrote:
>>> On Mon, Jul 25, 2011 at 2:45 PM, Matthias Schniedermeyer <ms@citd.de> wrote:
>>>> On 25.07.2011 13:08, Luke Kenneth Casson Leighton wrote:
>>>>> folks, hi,
>>>>>
>>>>> apart from anything, files which cannot be deleted (and cannot be
>>>>> detected as "corrupted" by fsck.ext3) is pretty damn serious.
>>>>
>>>> You did try lsattr and checked that the files aren't 'immutable'?
>>>
>>>  i didn't! :)  didn't know about (but should have guessed) ext3
>>> attributes.  they are indeed - thank you matthias.
>>>
>>> root@quietbaby:/mnt/horsebox/tmp3# lsattr *
>>> ----ia------------- bin3/kill
>>> ----ia------------- bin3/ps
>>> ----ia------------- c.pl
>>> ----ia------------- e.conf
>>> ----ia------------- sbin3/sysctl
>>> ----ia------------- usrbin3/uptime
>>> ----ia------------- usrbin3/tload
>>> ----ia------------- usrbin3/free
>>> ----ia------------- usrbin3/top
>>> ----ia------------- usrbin3/vmstat
>>> ----ia------------- usrbin3/watch
>>> ----ia------------- usrbin3/skill
>>> ----ia------------- usrbin3/pmap
>>> ----ia------------- usrbin3/pgrep
>>> ----ia------------- usrbin3/slabtop
>>> ----ia------------- usrbin3/pwdx
>>> ----ia------------- usrbin3/snice
>>> ----ia------------- usrbin3/pkill
>>> ----ia------------- usrbin3/w
>>>
>>> so - looks like it's not as bad as i thought.
>>
>> Should ls -l be moddified to show something when file has immutable
>> (and friends) set?
> 
> AFAICT lsattr, in e2fsprogs, only does a 'stat'(lib/e2p/fgetflags.c) and 
> checks st_flags, which i can't see in the "man 2 stat"-man-page i have 
> installed, but nonetheless that is what it appears to do.
> 
> So assuming there are no incompatibilites between filesystems, the 
> information appears to come "free" with the stat(s) that ls has to do 
> anyway. (In the sense that there doesn't appear to any excessive 
> overhead involved, especially no additional I/O).
> 
> So i would say: definitly.

`strace lsattr ...` shows it calls ioctl (...FS_IOC_GETFLAGS...)
So there would be overhead.
The output of ls is fairly constrained too for compat reasons.

However these flags are not specific to ext2 so it would fit
quite well from that perspective.
It might be something we could add to the stat command at least?

cheers,
Pádraig.

Re: ext3 hacked filesystem (by debian exim4 exploit) available for analysis and bugreporting

[Kyle Moffett]

2011/7/29 Pádraig Brady <P@draigbrady.com>:
>>> Should ls -l be moddified to show something when file has immutable
>>> (and friends) set?
>>
>> AFAICT lsattr, in e2fsprogs, only does a 'stat'(lib/e2p/fgetflags.c) and
>> checks st_flags, which i can't see in the "man 2 stat"-man-page i have
>> installed, but nonetheless that is what it appears to do.
>>
>> So assuming there are no incompatibilites between filesystems, the
>> information appears to come "free" with the stat(s) that ls has to do
>> anyway. (In the sense that there doesn't appear to any excessive
>> overhead involved, especially no additional I/O).
>>
>> So i would say: definitly.
>
> `strace lsattr ...` shows it calls ioctl (...FS_IOC_GETFLAGS...)
> So there would be overhead.
> The output of ls is fairly constrained too for compat reasons.
>
> However these flags are not specific to ext2 so it would fit
> quite well from that perspective.
> It might be something we could add to the stat command at least?

Well, "ls" already displays information about POSIX ACLs by putting
a "+" after the UNIX permission bits.  Some other UNIXen (such as
Mac OS X) seem to use an "@" character to indicate xattrs or other
forks of a file, so perhaps it would not be too inappropriate to throw
an "@" symbol on a file for ext3 immutable bits and such.

The question is what to do if something has both ACLs and
ext3 attributes; should it get both characters?  Just one?
Perhaps a unique character?

I dunno...  Just food for thought...

Cheers,
Kyle Moffett

Re: ext3 hacked filesystem (by debian exim4 exploit) available for analysis and bugreporting

[Jim Meyering]

Pádraig Brady wrote:
...
> `strace lsattr ...` shows it calls ioctl (...FS_IOC_GETFLAGS...)
> So there would be overhead.
> The output of ls is fairly constrained too for compat reasons.

One way by which ls -l could inform us that a file has
"attributes" like that would be via what POSIX calls the
"optional alternate access method flag".  Currently,
it can be a space, a "+", or a ".":

Since coreutils-7.1 (Feb 2009), ls -l has been doing this:

    ls -l now marks SELinux-only files with the less obtrusive '.',
    rather than '+'.  A file with any other combination of MAC and ACL
    is still marked with a '+'.

Using other printable characters to denote attributes (trumping
SELinux, MAC and ACL settings) is possible.  Does any other ls
implementation already do that?  There would be the cost of the
additional ioctl... we'd have to measure that before deciding
whether to impose this on every use of ls -l.

Another possibility is to further overload the file type decorations
("*", "|", "="), that you see with -F.  Already, "*" is obviously
not a type indicator, so adding more attributes is not shocking.

> However these flags are not specific to ext2 so it would fit
> quite well from that perspective.
> It might be something we could add to the stat command at least?

Yes, this definitely deserves a new stat format directive or two.
At least one for each single-byte "attribute" described in chattr(1),
and maybe another for a readable string so we don't have to look up
the likes of "T" ("top" dir hint, for orlov block allocator) and
"t" (no-tail-merge).

Re: ext3 hacked filesystem (by debian exim4 exploit) available for analysis and bugreporting

[Luke Kenneth Casson Leighton]

On Fri, Jul 29, 2011 at 8:59 PM, Pavel Machek <pavel@ucw.cz> wrote:
> On Mon 2011-07-25 22:08:24, Luke Kenneth Casson Leighton wrote:
> [...]
>> On Mon, Jul 25, 2011 at 2:45 PM, Matthias Schniedermeyer <ms@citd.de> wrote:
>>> You did try lsattr and checked that the files aren't 'immutable'?
>>
>>  i didn't! :)  didn't know about (but should have guessed) ext3
>> attributes.  they are indeed - thank you matthias.
>> [...]

> Should ls -l be moddified to show something when file has immutable
> (and friends) set?
>                                                                Pavel

 +1 on that.  i've been using unix systems for 20 years, and had
absolutely no visual clue that there were ext3 attributes, from doing
"ls -altr", my usual "fingers-pre-programmed" command.

 i would say that something like "+" on the attributes -rwxrwx---+
would intuitively say "there's more!" but it would be very helpful for
its background to e.g. be in red [on ls --color].  that would
definitely grab peoples' attention in a "wtf is that??" way.

 yeah. whatever symbol is chosen, i believe that some sort of
eye-burning background colour would be more important, forcing people
to go "ls --help" or "man ls".

 my only concern with adding yet more attribute-markers is all the
commands that parse "ls -a" output.  well... t'be'honest... i think if
selinux attribute markers have already been added, then that bullet
has already been bitten once, so it's not as big a hairy deal as all
that.

peace.

l.

Re: ext3 hacked filesystem (by debian exim4 exploit) available for analysis and bugreporting

[Luke Kenneth Casson Leighton]

On Sat, Jul 30, 2011 at 10:14 AM, Jim Meyering <jim@meyering.net> wrote:
> Pádraig Brady wrote:
> ...
>> `strace lsattr ...` shows it calls ioctl (...FS_IOC_GETFLAGS...)
>> So there would be overhead.
>> The output of ls is fairly constrained too for compat reasons.

 well if that rule has been broken once for selinux (adding extra
symbols) then it can be broken again.

> One way by which ls -l could inform us that a file has
> "attributes" like that would be via what POSIX calls the
> "optional alternate access method flag".  Currently,
> it can be a space, a "+", or a ".":

 i'd advocate, when the ls --color option is set, changing the
background to something attention-grabbing (e.g. red).  it's far too
easy to miss an extra symbol otherwise, and "+", although intuitively
saying "there's more than meets the eye, here" is too similar to a
line of "-"s to really make people sit up and take notice.  i'd
certainly go "oh look, there's a +, how pretty", but a + on a red
background i might just stop and think, "eh? i no unnerstan, let's
look up man page or call some friends".

 l.

Re: ext3 hacked filesystem (by debian exim4 exploit) available for analysis and bugreporting

[Luke Kenneth Casson Leighton]

2011/7/29 Pádraig Brady <P@draigbrady.com>:
> On 07/29/2011 10:31 PM, Matthias Schniedermeyer wrote:
>> On 29.07.2011 21:59, Pavel Machek wrote:
>>> On Mon 2011-07-25 22:08:24, Luke Kenneth Casson Leighton wrote:
>>>> On Mon, Jul 25, 2011 at 2:45 PM, Matthias Schniedermeyer <ms@citd.de> wrote:
>>>>> On 25.07.2011 13:08, Luke Kenneth Casson Leighton wrote:
>>>>>> folks, hi,
>>>>>>
>>>>>> apart from anything, files which cannot be deleted (and cannot be
>>>>>> detected as "corrupted" by fsck.ext3) is pretty damn serious.
>>>>>
>>>>> You did try lsattr and checked that the files aren't 'immutable'?
>>>>
>>>>  i didn't! :)  didn't know about (but should have guessed) ext3
>>>> attributes.  they are indeed - thank you matthias.
>>>>
>>>> root@quietbaby:/mnt/horsebox/tmp3# lsattr *
>>>> ----ia------------- bin3/kill
>>>> ----ia------------- bin3/ps
>>>> ----ia------------- c.pl
>>>> ----ia------------- e.conf
>>>> ----ia------------- sbin3/sysctl
>>>> ----ia------------- usrbin3/uptime
>>>> ----ia------------- usrbin3/tload
>>>> ----ia------------- usrbin3/free
>>>> ----ia------------- usrbin3/top
>>>> ----ia------------- usrbin3/vmstat
>>>> ----ia------------- usrbin3/watch
>>>> ----ia------------- usrbin3/skill
>>>> ----ia------------- usrbin3/pmap
>>>> ----ia------------- usrbin3/pgrep
>>>> ----ia------------- usrbin3/slabtop
>>>> ----ia------------- usrbin3/pwdx
>>>> ----ia------------- usrbin3/snice
>>>> ----ia------------- usrbin3/pkill
>>>> ----ia------------- usrbin3/w
>>>>
>>>> so - looks like it's not as bad as i thought.
>>>
>>> Should ls -l be moddified to show something when file has immutable
>>> (and friends) set?
>>
>> AFAICT lsattr, in e2fsprogs, only does a 'stat'(lib/e2p/fgetflags.c) and
>> checks st_flags, which i can't see in the "man 2 stat"-man-page i have
>> installed, but nonetheless that is what it appears to do.
>>
>> So assuming there are no incompatibilites between filesystems, the
>> information appears to come "free" with the stat(s) that ls has to do
>> anyway. (In the sense that there doesn't appear to any excessive
>> overhead involved, especially no additional I/O).
>>
>> So i would say: definitly.
>
> `strace lsattr ...` shows it calls ioctl (...FS_IOC_GETFLAGS...)
> So there would be overhead.

$ strace -o log.txt ls -altr

 ...
 ...
 lstat(".", {st_mode=S_IFDIR|0755, st_size=12288, ...}) = 0
lgetxattr(".", "security.selinux", 0xeb5bf0, 255) = -1 ENODATA (No
data available)
getxattr(".", "system.posix_acl_access", 0x0, 0) = -1 EOPNOTSUPP
(Operation not supported)
 ...

so it looks like there already is overhead.

on the basis that "ls" is already looking for selinux extended
attributes, as well as.. oh - posix ACLs as well! - i'd say that
adding an extra call to check even the existence of anything that
lsattr does (not _what_ they are, just "are there any at all?") would
not be, relatively speaking, that much extra.

l.