Recovery when disk_full_action=HALT

Recovery when disk_full_action=HALT

[Andrew Ruch]

Hello,

We have a RHEL6 system with the disk_full_action set to HALT. I'm
working on procedures for what to do if this case occurs. When the log
partition fills up, the system shuts down. However, the system will
not boot after this because as soon as auditd tries to start, the
system immediately shuts down again. What are the options for
recovering after this happens? I've come up with two:

1) Stop the boot process at grub and disable audit by adding a kernel
parameter 'audit=0'.

2) If grub timeout is 0, use a live CD to access the audit partition.


I'm sure there are some variations on option 1 using an interactive
boot. Are there any other options I missed, especially if grub timeout
has been set to 0?

Thanks,
Andrew Ruch

Re: Recovery when disk_full_action=HALT

[leam hall]

[-- Attachment #1.1: Type: text/plain, Size: 1188 bytes --]

My plan would be to implement logrotate and clean out things well. Another
path is to ensure the log volume is larger than " max number of logs" *
"max log size".

On Thu, Apr 16, 2015 at 10:29 AM, Andrew Ruch <adruch2002@gmail.com> wrote:

> Hello,
>
> We have a RHEL6 system with the disk_full_action set to HALT. I'm
> working on procedures for what to do if this case occurs. When the log
> partition fills up, the system shuts down. However, the system will
> not boot after this because as soon as auditd tries to start, the
> system immediately shuts down again. What are the options for
> recovering after this happens? I've come up with two:
>
> 1) Stop the boot process at grub and disable audit by adding a kernel
> parameter 'audit=0'.
>
> 2) If grub timeout is 0, use a live CD to access the audit partition.
>
>
> I'm sure there are some variations on option 1 using an interactive
> boot. Are there any other options I missed, especially if grub timeout
> has been set to 0?
>
> Thanks,
> Andrew Ruch
>
> --
> Linux-audit mailing list
> Linux-audit@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-audit
>



-- 
Mind on a Mission <http://leamhall.blogspot.com/>

[-- Attachment #1.2: Type: text/html, Size: 1831 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Recovery when disk_full_action=HALT

[Steve Grubb]

On Thursday, April 16, 2015 08:29:23 AM Andrew Ruch wrote:
> Hello,
> 
> We have a RHEL6 system with the disk_full_action set to HALT. I'm
> working on procedures for what to do if this case occurs. When the log
> partition fills up, the system shuts down. However, the system will
> not boot after this because as soon as auditd tries to start, the
> system immediately shuts down again. What are the options for
> recovering after this happens? I've come up with two:

Normally, I would think that system maintenance for a situation like this is 
to boot the computer into Single User Mode. You should have switched the 
system over to using sulogin as the shell for single user mode. This way its 
password protected. Then once in, do what you need to archive and make room 
again.


> 1) Stop the boot process at grub and disable audit by adding a kernel
> parameter 'audit=0'.

If you don't use single user mode, then there is the risk of someone doing 
something while the audit system can't record anything. You probably don't 
want that possibility either.


> 2) If grub timeout is 0, use a live CD to access the audit partition.

This would work also, but Single User Mode is so much easier. :-)
 
 
> I'm sure there are some variations on option 1 using an interactive
> boot. Are there any other options I missed, especially if grub timeout
> has been set to 0?

I wouldn't set it to 0. You can make it short like 2 or 3. But you need to be 
able to get into the editor to tell it 'S' for single user mode.

-Steve

Recovery when disk_full_action=HALT

[Paul Whitney]

[-- Attachment #1.1: Type: text/plain, Size: 2452 bytes --]

Hi Andrew,

To add to Steve Grubb's response, part of the configuration should also include an option to notify the administrator via Email when the partition is at 90% capacity.  Of course, this can be adjusted to better suit your requirements (i.e. 90% on a Friday night might be too late).

Another solution is to use Steve's logrotate script.  We made some minor additions to it so that the script not only rotates the audit log every 24-hours, but then it gets compressed using BZIP2 and moved to a larger partition under an archive folder (i.e. /opt/AUDIT_ARCHIVE) and time stamped. This way your partition should rarely if ever fill up.

Cheers,
Paul M. Whitney
E-mail: paul.whitney@mac.com
Sent from my browser.



On Apr 16, 2015, at 10:52 AM, Steve Grubb <sgrubb@redhat.com> wrote:

On Thursday, April 16, 2015 08:29:23 AM Andrew Ruch wrote:
Hello,

We have a RHEL6 system with the disk_full_action set to HALT. I'm
working on procedures for what to do if this case occurs. When the log
partition fills up, the system shuts down. However, the system will
not boot after this because as soon as auditd tries to start, the
system immediately shuts down again. What are the options for
recovering after this happens? I've come up with two:

Normally, I would think that system maintenance for a situation like this is 
to boot the computer into Single User Mode. You should have switched the 
system over to using sulogin as the shell for single user mode. This way its 
password protected. Then once in, do what you need to archive and make room 
again.


1) Stop the boot process at grub and disable audit by adding a kernel
parameter 'audit=0'.

If you don't use single user mode, then there is the risk of someone doing 
something while the audit system can't record anything. You probably don't 
want that possibility either.


2) If grub timeout is 0, use a live CD to access the audit partition.

This would work also, but Single User Mode is so much easier. :-)


I'm sure there are some variations on option 1 using an interactive
boot. Are there any other options I missed, especially if grub timeout
has been set to 0?

I wouldn't set it to 0. You can make it short like 2 or 3. But you need to be 
able to get into the editor to tell it 'S' for single user mode.

-Steve

--
Linux-audit mailing list
Linux-audit@redhat.com
https://www.redhat.com/mailman/listinfo/linux-audit

[-- Attachment #1.2.1: Type: text/html, Size: 4267 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]