Re: kernel.org status: hints on how to check your machine for intrusion

[Willy Tarreau <w@1wt.eu>]

Hi Greg,

On Fri, Sep 30, 2011 at 04:59:24PM -0700, Greg KH wrote:
> The compromise of kernel.org and related machines has made it clear that
> some developers, at least, have had their systems penetrated.  As we
> seek to secure our infrastructure, it is imperative that nobody falls
> victim to the belief that it cannot happen to them.  We all need to
> check our systems for intrusions.  Here are some helpful hints as
> proposed by a number of developers on how to check to see if your Linux
> machine might be infected with something:

I would like to add here a few controls I ran on firewall and system logs,
that are easy to perform and which report few false positives :

  - check that communications between your local machines are expected ;
    for instance if you have an SSH bouncing machine, it probably receives
    tens of thousands of SSH connection attempts from outside every day,
    but it should never ever attempt to connect to another machine unless
    it's you who are doing it. So checking the firewall logs for SSH
    connections on port 22 from local machines should only report your
    activity (and nothing should happen when you sleep).

  - no SSH log should report failed connection attempts between your
    local machines (you do have your keys and remember your password).
    And if it happens from time to time (eg: user mismatch between
    machines), it should look normal to you. You should never observe
    a connection attempt for a user you're not familiar with (eg: admin).

     $ grep sshd /var/log/messages
     $ grep sshd /var/log/messages | grep 'Invalid user'

  - outgoing connections from your laptop, desktop or anything should
    never happen when you're not there, unless there is a well known
    reason (package updates, browser left open and refreshing ads). All
    unexpected activity should be analysed (eg: connections to port 80
    not coming from a browser should only match one distro mirror).
    This is particularly true for cheap appliances which become more
    and more common and are rarely secured. A NAS or media server, a
    switch, a WiFi router, etc... has no reason to ever connect anywhere
    without you being aware of it (eg: download a firmware update).

  - check for suspicious DNS requests from machines that are normally
    not accessed. A number of services perform DNS requests when
    connected to, in order to log a resolved address. If the machine
    was penetrated and the logs wiped, the DNS requests will probably
    still lie in the firewall logs. While there's nothing suspect from
    a machine that does tens of thousands DNS requests a day, one that
    does 10 might be suspect.

  - check for outgoing SMTP connections. Most machines probably never
    send any mail outside or route them through a specific relay. If
    one machine suddenly tries to send mails directly to the outside,
    it might be someone trying to steal some data (eg: mail ssh keys).

  - check for long holes in logs various service logs. The idea is that
    if a system was penetrated and the guy notices he left a number of
    traces, he will probably have wiped some logs. A simple way to check
    for this is to count the number of events per hour and observe huge
    variations. Eg:

       $ cut -c1-9 < /var/log/syslog |uniq -c
       8490 Oct  1 00
       7712 Oct  1 01
       8316 Oct  1 02
       6743 Oct  1 03
       7428 Oct  1 04
       7041 Oct  1 05
       7762 Oct  1 06
       6562 Oct  1 07
       7137 Oct  1 08
        160 Oct  1 09

    Activity looks normal here. Something like this however would be
    extremely suspect :

       8490 Oct  1 00
        712 Oct  1 01
       6743 Oct  1 03

  - check that you never observe in logs a local address that you
    don't know. For instance, if your reverse proxy is on a DMZ which
    is provided by the same physical switch as your LAN and your switch
    becomes ill and loses all its VLAN configuration, it them becomes
    easy to add an alias to the reverse-proxy to connect directly to
    LAN machines and bypass a firewall (and its logs).

  - it's always a good exercise to check for setuids on all your machines.
    You'll generally discover a number of things you did not even suspect
    existed and will likely want to remove them. For instance, my file
    server had dbus-daemon-launch-helper setuid root. I removed this crap
    as dbus has nothing to do on such a machine. Similarly I don't need
    fdmount to mount floppies. I might not use floppies often, and if I do,
    I know how to use sudo.

       $ find / -user root -perm -4000 -ls

  - last considerations to keep in mind is that machines which receive
    incoming connections from outside should never be able to go out, and
    should be isolated in their own LAN. It's not hard to do at all, and
    it massively limits the ability to bounce between systems and to steal
    information. It also makes firewall logs much more meaningful, provided
    they are stored on a support with limited access, of course :-)

Regards,
Willy