Cutting power without breaking RAID

Cutting power without breaking RAID

[Niccolo Rigacci]

Hello,

I'm facing this problem:

when my Linux box detects a POWER FAIL event from the UPS, it 
starts a normal shutdown. Just before the normal kernel poweroff, 
it sends to the UPS a signal on the serial line which says 
"cut-off the power to the server and switch-off the UPS".

This is required to reboot the server as soon as the power is 
restored.

The problem is that the root partition is on top of a RAID-1 
filesystem which is still mounted when the program that kills the 
power is run, so the system goes down with a non clean RAID 
volume.

What can be the proper action to do before killing the power to 
ensure that RAID will remain clean? It seems that remounting 
the partition read-only is not sufficient.

-- 
Niccolo Rigacci
Firenze - Italy

Iraq, missione di pace: 38475 morti - www.iraqbodycount.net

Re: Cutting power without breaking RAID

[Tim]

Your UPS won't accept a timer value to wait before actually cutting
power? That would probably be ideal, issue the power off command with
something like a 30 second timeout, which would give the system time to
power off cleanly first.

-Tim

Niccolo Rigacci wrote:
> Hello,
>
> I'm facing this problem:
>
> when my Linux box detects a POWER FAIL event from the UPS, it 
> starts a normal shutdown. Just before the normal kernel poweroff, 
> it sends to the UPS a signal on the serial line which says 
> "cut-off the power to the server and switch-off the UPS".
>
> This is required to reboot the server as soon as the power is 
> restored.
>
> The problem is that the root partition is on top of a RAID-1 
> filesystem which is still mounted when the program that kills the 
> power is run, so the system goes down with a non clean RAID 
> volume.
>
> What can be the proper action to do before killing the power to 
> ensure that RAID will remain clean? It seems that remounting 
> the partition read-only is not sufficient.
>
>   

Re: Cutting power without breaking RAID

[Neil Brown]

On Wednesday June 28, niccolo@rigacci.org wrote:
> Hello,
> 
> I'm facing this problem:
> 
> when my Linux box detects a POWER FAIL event from the UPS, it 
> starts a normal shutdown. Just before the normal kernel poweroff, 
> it sends to the UPS a signal on the serial line which says 
> "cut-off the power to the server and switch-off the UPS".
> 
> This is required to reboot the server as soon as the power is 
> restored.
> 
> The problem is that the root partition is on top of a RAID-1 
> filesystem which is still mounted when the program that kills the 
> power is run, so the system goes down with a non clean RAID 
> volume.
> 
> What can be the proper action to do before killing the power to 
> ensure that RAID will remain clean? It seems that remounting 
> the partition read-only is not sufficient.

Are you running a 2.4 kernel or a 2.6 kernel?

With 2.4, you cannot do what you want to do.

With 2.6, 
   killall -9 md0_raid1

should do the trick (assuming root is on /dev/md0.  If it is elsewhere,
choose a different process name).
After you "kill -9" the raid thread, the array will be marked clean
immediately all writes complete, and marked dirty again before
allowing another write.

NeilBrown

Re: Cutting power without breaking RAID

[Niccolo Rigacci]

On Wed, Jun 28, 2006 at 06:03:01PM -0700, Tim wrote:
> Your UPS won't accept a timer value to wait before actually cutting
> power?

Unfortunately my UPS des not!

-- 
Niccolo Rigacci
Firenze - Italy

Iraq, missione di pace: 38725 morti - www.iraqbodycount.net

Re: Cutting power without breaking RAID

[Niccolo Rigacci]

On Thu, Jun 29, 2006 at 02:00:09PM +1000, Neil Brown wrote:
> With 2.6, 
>    killall -9 md0_raid1
> 
> should do the trick (assuming root is on /dev/md0.  If it is elsewhere,
> choose a different process name).

Thanks, this is what I was looking for!

I will try remounting read-only and killing the md0_raid1.
I will keep you informed.

-- 
Niccolo Rigacci
Firenze - Italy

Iraq, missione di pace: 38725 morti - www.iraqbodycount.net

Re: Cutting power without breaking RAID

[Bill Davidsen]

Niccolo Rigacci wrote:

>On Thu, Jun 29, 2006 at 02:00:09PM +1000, Neil Brown wrote:
>  
>
>>With 2.6, 
>>   killall -9 md0_raid1
>>
>>should do the trick (assuming root is on /dev/md0.  If it is elsewhere,
>>choose a different process name).
>>    
>>
>
>Thanks, this is what I was looking for!
>
>I will try remounting read-only and killing the md0_raid1.
>I will keep you informed.
>
>  
>
Why should this trickery be needed? When an array is mounted r/o it 
should be clean. How can it be dirty. I assume readonly implies noatime, 
I mount physically readonly devices without explicitly saying noatime 
and nothing whines.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: Cutting power without breaking RAID

[alexwang.com (sent by Nabble.com)]

Looks impossible UPS design to me. May I ask what UPS do you use?

My APC smart UPS does has a timer, so PC can send command "shut the UPS down
after 30 seconds" and then shutdown itself.
-- 
View this message in context: http://www.nabble.com/Cutting-power-without-breaking-RAID-tf1864273.html#a5113118
Sent from the linux-raid forum at Nabble.com.

Re: Cutting power without breaking RAID

[Neil Brown]

On Thursday June 29, davidsen@tmr.com wrote:
> Why should this trickery be needed? When an array is mounted r/o it 
> should be clean. How can it be dirty. I assume readonly implies noatime, 
> I mount physically readonly devices without explicitly saying noatime 
> and nothing whines.

The 'filesystem' is mounted r/o.  The 'array' is not read-only, and
you cannot set an array to read-only while a filesystem is mounted
(because the array cannot tell that the mount is read-only).

A little while after the last write, an array will mark itself as
clean.  The effect of the 'kill -9', is to reduce this 'little while'
to 0.

So 
  remount readonly
  wait a little while
  kill machine 

would work too.

NeilBrown

Re: Cutting power without breaking RAID

[Molle Bestefich]

Tim wrote:
> That would probably be ideal, issue the power off command with
> something like a 30 second timeout, which would give the system time to
> power off cleanly first.

I don't think that's ideal.
Many systems restore power to the last known state, thus powering off
cleanly would result in the machine not coming back up after the power
cycle.

Some machines can be changed to always power on in the BIOS.
If the server administrator remembers to do so, that is.