From mboxrd@z Thu Jan 1 00:00:00 1970 From: Paul Whitney Subject: Recovery when disk_full_action=HALT Date: Thu, 16 Apr 2015 15:03:55 +0000 (GMT) Message-ID: <7cc5cace-6e9c-4529-a1d5-bf9b53cd9784@me.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============8032636226918633143==" Return-path: Received: from mx1.redhat.com (ext-mx15.extmail.prod.ext.phx2.redhat.com [10.5.110.20]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id t3GF3FV0032765 for ; Thu, 16 Apr 2015 11:03:15 -0400 Received: from nk11p00mm-asmtp003.mac.com (nk11p00mm-asmtp003.mac.com [17.158.161.2]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id t3GF3BZd010841 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA256 bits=256 verify=NO) for ; Thu, 16 Apr 2015 11:03:12 -0400 Received: from nk11p00mm-spool001.mac.com ([17.158.161.66]) by nk11p00mm-asmtp003.mac.com (Oracle Communications Messaging Server 7.0.5.35.0 64bit (built Dec 4 2014)) with ESMTP id <0NMW009ZDN518V00@nk11p00mm-asmtp003.mac.com> for linux-audit@redhat.com; Thu, 16 Apr 2015 15:03:11 +0000 (GMT) Received: from localhost ([17.158.237.79]) by nk11p00mm-spool001.mac.com (Oracle Communications Messaging Server 7.0.5.33.0 64bit (built Aug 27 2014)) with ESMTP id <0NMW004CDN56AF60@nk11p00mm-spool001.mac.com> for linux-audit@redhat.com; Thu, 16 Apr 2015 15:03:06 +0000 (GMT) List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: linux-audit@redhat.com List-Id: linux-audit@redhat.com --===============8032636226918633143== Content-type: multipart/alternative; boundary="Boundary_(ID_Z1ZtJ4SgL5SuATCD3CHz3g)" --Boundary_(ID_Z1ZtJ4SgL5SuATCD3CHz3g) Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: quoted-printable Hi Andrew,=0A=0ATo add to Steve Grubb's response, part of the configuratio= n should also include an option to notify the administrator via Email when= the partition is at 90% capacity. =A0Of course, this can be adjusted to b= etter suit your requirements (i.e. 90% on a Friday night might be too late= ).=0A=0AAnother solution is to use Steve's logrotate script. =A0We made so= me minor additions to it so that the script not only rotates the audit log= every 24-hours, but then it gets compressed using BZIP2 and moved to a la= rger partition under an archive folder (i.e. /opt/AUDIT_ARCHIVE) and time = stamped. This way your partition should rarely if ever fill up.=0A=0ACheer= s,=0APaul M. Whitney=0AE-mail: paul.whitney@mac.com=0ASent from my browser= .=0A=0A=0A=0AOn Apr 16, 2015, at 10:52 AM, Steve Grubb = wrote:=0A=0AOn Thursday, April 16, 2015 08:29:23 AM Andrew Ruch wrote:=0A= Hello,=0A=0AWe have a RHEL6 system with the disk_full_action set to HALT. = I'm=0Aworking on procedures for what to do if this case occurs. When the l= og=0Apartition fills up, the system shuts down. However, the system will=0A= not boot after this because as soon as auditd tries to start, the=0Asystem= immediately shuts down again. What are the options for=0Arecovering after= this happens? I've come up with two:=0A=0ANormally, I would think that sy= stem maintenance for a situation like this is =0Ato boot the computer into= Single User Mode. You should have switched the =0Asystem over to using su= login as the shell for single user mode. This way its =0Apassword protecte= d. Then once in, do what you need to archive and make room =0Aagain.=0A=0A= =0A1) Stop the boot process at grub and disable audit by adding a kernel=0A= parameter 'audit=3D0'.=0A=0AIf you don't use single user mode, then there = is the risk of someone doing =0Asomething while the audit system can't rec= ord anything. You probably don't =0Awant that possibility either.=0A=0A=0A= 2) If grub timeout is 0, use a live CD to access the audit partition.=0A=0A= This would work also, but Single User Mode is so much easier. :-)=0A=0A=0A= I'm sure there are some variations on option 1 using an interactive=0Aboot= . Are there any other options I missed, especially if grub timeout=0Ahas b= een set to 0?=0A=0AI wouldn't set it to 0. You can make it short like 2 or= 3. But you need to be =0Aable to get into the editor to tell it 'S' for s= ingle user mode.=0A=0A-Steve=0A=0A--=0ALinux-audit mailing list=0ALinux-au= dit@redhat.com=0Ahttps://www.redhat.com/mailman/listinfo/linux-audit=0A= --Boundary_(ID_Z1ZtJ4SgL5SuATCD3CHz3g) Content-type: multipart/related; boundary="Boundary_(ID_C7SPXFgsukgM5Awq9KvWng)"; type="text/html" --Boundary_(ID_C7SPXFgsukgM5Awq9KvWng) Content-type: text/html; CHARSET=US-ASCII Content-transfer-encoding: quoted-printable
Hi Andrew,

To add to Steve Grubb's response,= part of the configuration should also include an option to notify the adm= inistrator via Email when the partition is at 90% capacity.  Of cours= e, this can be adjusted to better suit your requirements (i.e. 90% on a Fr= iday night might be too late).

Another solution i= s to use Steve's logrotate script.  We made some minor additions to i= t so that the script not only rotates the audit log every 24-hours, but th= en it gets compressed using BZIP2 and moved to a larger partition under an= archive folder (i.e. /opt/AUDIT_ARCHIVE) and time stamped. This way your = partition should rarely if ever fill up.

Cheers,<= /div>
Paul M. Whitney=0AE-mail: paul.whitney@mac.com=0ASen=
t from my browser.=0A=0A=0A

On Apr 16, 2015, at 10:52 = AM, Steve Grubb <sgrubb@redhat.com> wrote:

On Thursday,= April 16, 2015 08:29:23 AM Andrew Ruch wrote:
Hello,
We have a RHEL6 system with the di= sk_full_action set to HALT. I'm
working on procedures for what to do if this case = occurs. When the log
partition fills up, the system shuts down. However, the syste= m will
n= ot boot after this because as soon as auditd tries to start, the
system immediatel= y shuts down again. What are the options for
recovering after this happens? I've c= ome up with two:

Normally, I would think that system maintenance= for a situation like this is
to boot the computer into Single User Mo= de. You should have switched the
system over to using sulogin as the s= hell for single user mode. This way its
password protected. Then once = in, do what you need to archive and make room
again.


1) Stop the = boot process at grub and disable audit by adding a kernel
parameter 'audit=3D0'.
If you don't use single user mode, then there is the risk of some= one doing
something while the audit system can't record anything. You = probably don't
want that possibility either.


=
2) If grub timeout i= s 0, use a live CD to access the audit partition.

This would work = also, but Single User Mode is so much easier. :-)


I'm sure there ar= e some variations on option 1 using an interactive
boot. Are there any other optio= ns I missed, especially if grub timeout
has been set to 0?

I wouldn't set it to 0. You can make it short l= ike 2 or 3. But you need to be
able to get into the editor to tell it = 'S' for single user mode.

-Steve

--
Linux-audit mailing l= ist
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit
= --Boundary_(ID_C7SPXFgsukgM5Awq9KvWng)-- --Boundary_(ID_Z1ZtJ4SgL5SuATCD3CHz3g)-- --===============8032636226918633143== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --===============8032636226918633143==--