Power cut in management

Power cut in management

[marco.stornelli]

Hi all,

I'm working with a board where the power is turn on/off through a key as
in a car. Is there any design pattern to afford that? It's the first time
I have to manage a situation where the power can suddenly cut in anytime.
Hardware guys are working to get "time" to do a clean shutdown (I turn the
key but I still have the power) but any suggestions are welcome. However,
I can't use a simply read-only filesystem (at least for root fs) because
the board has got an sdcard that can be inserted (and then mounted in rw
mode) anytime. Thanks.

Regards,

Marco

Re: Power cut in management

[Jamie Lokier]

marco.stornelli@coritel.it wrote:
> I'm working with a board where the power is turn on/off through a key as
> in a car. Is there any design pattern to afford that? It's the first time
> I have to manage a situation where the power can suddenly cut in anytime.
> Hardware guys are working to get "time" to do a clean shutdown (I turn the
> key but I still have the power) but any suggestions are welcome. However,
> I can't use a simply read-only filesystem (at least for root fs) because
> the board has got an sdcard that can be inserted (and then mounted in rw
> mode) anytime. Thanks.

This is off the top of my head, and may be rubbish.

From http://www.linux-mtd.infradead.org/doc/ubifs.html:

    We have heard reports that MMC and SD cards corrupt and loose data
    if power is cut during writing. Even the data which was there long
    time before may corrupt or disappear. This means that they have
    bad FTL which does not do things properly. But again, this does
    not have to be true for all MMCs and SDs - there are many
    different vendors. But again, you should be careful.

As long as you have sufficient emergency power to finish writing a
consistent filesystem state and no incomplete block to the SD card,
and to gracefully shut down the SD card so that it stops reorganising
data on the flash internally, it should be ok.  That means routing
emergency power to the CPU and SD card, so that you don't get
corruption in the middle of writing a block.

Can you use a journalling filesystem like ext3, reiserfs, xfs, or even
UBIFS on the card, or does it have to be FAT?  With a journalling
filesystem, they vary on the details but basically if you can finish
the current write request in progress, there's no fs-level corruption.

With FAT, completing just the write in progress is not enough.  In
principle a small number of writes could be used to get the fs into a
consistent state in an emergency, and the required number could be
kept bounded by changes to the FAT fs code.

Emergency power isn't necessarily required for the peripherals such as
panel LEDs, displays, CD drives, radios, serial ports, speakers
etc. that you might have, so by routing it only to a fewer essential
components, and shutting down the rest, you'll have a stable power for
longer off a smaller capacitor.

Make sure the hardware is able to notify the CPU by interrupt
immediately when the main power is cut, so the CPU can switch into the
emergency mode as quickly as possible, cut power to unnecessary
peripherals, only finish the I/O request already in progress, or the
minimum required for a consistent state, and gracefully shut down the
SD card.  This means the board's PSU should not simply keep going for
a couple of seconds after input power is lost: it must provide you
with a logic signal too.  Alos, make sure during this time that the
board's PSU voltage output remains stable and doesn't drop drastically
during the emergency shutdown, as that could lead to SD card
corruption too.

-- Jamie

Re: Power cut in management

[David Woodhouse]

On Sat, 2008-10-18 at 12:49 +0100, Jamie Lokier wrote:
> Can you use a journalling filesystem like ext3, reiserfs, xfs, or even
> UBIFS on the card, or does it have to be FAT?  With a journalling
> filesystem, they vary on the details but basically if you can finish
> the current write request in progress, there's no fs-level corruption.

Careful. When you use a solid state "disk", you're actually using your
file system on top of an underlying "pseudo-filesystem" which makes the
flash pretend to be a disk. Your data are at the mercy of that
underlying code, which in our past experience always seem to have been
implemented by the same crack-smoking hobos they drag in off the street
to write PC BIOSes. However good your own journalling file system on
_top_ of that 'disk' is, you can still lose everything with a badly
timed power-off.

If you want to store data on solid state media, my current advice is
either to do it on _real_ flash directly with a flash-based file system
like JFFS2 or UBIFS, or to pray regularly to the deity of your choice.

-- 
dwmw2

Re: Power cut in management

[Jamie Lokier]

David Woodhouse wrote:
> On Sat, 2008-10-18 at 12:49 +0100, Jamie Lokier wrote:
> > Can you use a journalling filesystem like ext3, reiserfs, xfs, or even
> > UBIFS on the card, or does it have to be FAT?  With a journalling
> > filesystem, they vary on the details but basically if you can finish
> > the current write request in progress, there's no fs-level corruption.
> 
> Careful. When you use a solid state "disk", you're actually using your
> file system on top of an underlying "pseudo-filesystem" which makes the
> flash pretend to be a disk. Your data are at the mercy of that
> underlying code, which in our past experience always seem to have been
> implemented by the same crack-smoking hobos they drag in off the street
> to write PC BIOSes. However good your own journalling file system on
> _top_ of that 'disk' is, you can still lose everything with a badly
> timed power-off.

I agree, hence the idea to complete the current write request _and
also_ gracefully shutdown the SD card to an inactive state, before
power is dropped.

To continue the filesystem on another pseudo-filesystem analogy,
that's like calling fsync on the pseudo-filesystem.

No guarantees - even Linux has fsync worthy of crack-smoking hobos at
times ;-) - but better than not calling fsync at all!

But is it even possible to shutdown an SD card gracefully, e.g. with a
power-down command?

> If you want to store data on solid state media, my current advice is
> either to do it on _real_ flash directly with a flash-based file system
> like JFFS2 or UBIFS, or to pray regularly to the deity of your choice.

Trouble is, that's not suitable for a dashboard unit where users plug
in their own media card.

Marco didn't say if the SD card is for users to plug in their own
media, or if it's internal storage for the device.

Marco, if it's just for the device itself to use, follow David's
suggestion if you can: use real flash with UBIFS or something like it,
not an SD card.  That will be much easier.

-- Jamie

Re: Power cut in management

[David Woodhouse]

On Sat, 2008-10-18 at 13:56 +0100, Jamie Lokier wrote:
> Trouble is, that's not suitable for a dashboard unit where users plug
> in their own media card.
> 
> Marco didn't say if the SD card is for users to plug in their own
> media, or if it's internal storage for the device.

True, but the situation is different for a removable card. Firstly, it's
unlikely to be mission-critical; the device will still operate without
it. Secondly, even the most naïve of users knows that these things are
disposable.

It's different if you're building a black box around them, with one of
these things inside.

-- 
David Woodhouse                            Open Source Technology Centre
David.Woodhouse@intel.com                              Intel Corporation

Re: Power cut in management

[Jamie Lokier]

David Woodhouse wrote:
> On Sat, 2008-10-18 at 13:56 +0100, Jamie Lokier wrote:
> > Trouble is, that's not suitable for a dashboard unit where users plug
> > in their own media card.
> > 
> > Marco didn't say if the SD card is for users to plug in their own
> > media, or if it's internal storage for the device.
> 
> True, but the situation is different for a removable card. Firstly, it's
> unlikely to be mission-critical; the device will still operate without
> it. Secondly, even the most naïve of users knows that these things are
> disposable.
> 
> It's different if you're building a black box around them, with one of
> these things inside.

I agree, this is very good advice.

-- Jamie

Re: Power cut in management

[Marco]

Jamie Lokier ha scritto:
> David Woodhouse wrote:
>> On Sat, 2008-10-18 at 13:56 +0100, Jamie Lokier wrote:
>>> Trouble is, that's not suitable for a dashboard unit where users plug
>>> in their own media card.
>>>
>>> Marco didn't say if the SD card is for users to plug in their own
>>> media, or if it's internal storage for the device.
>> True, but the situation is different for a removable card. Firstly, it's
>> unlikely to be mission-critical; the device will still operate without
>> it. Secondly, even the most naïve of users knows that these things are
>> disposable.
>>
>> It's different if you're building a black box around them, with one of
>> these things inside.
> 
> I agree, this is very good advice.
> 
> -- Jamie
> 
Thank you very much for your response. The sdcard, however, it's for the
user so I haven't control on it, I think probably it will be FAT. Jamie
you said to manage the situation with an interrupt, I agree. My first
plan was (more or less) to have an application in user space that it
checks the interrupt (UIO driver ?) and simulates a console command to
do a clean shutdown. Then in the kernel routine I do something to cut
the power definitely. For my environment (about the time) it's possible
even in relation with the "emergency power". The only problem is: what
happens if before I install the irq handler the user turn off the
system? I mean I can't "loose" the interrupt. In this case it'd be like
a spurious interrupt from system point view. Let me know if I'm wrong.
Thanks.

Re: Power cut in management

[Ben Nizette]

On Sat, 2008-10-18 at 18:01 +0200, Marco wrote:
> The only problem is: what
> happens if before I install the irq handler the user turn off the
> system? I mean I can't "loose" the interrupt. In this case it'd be like
> a spurious interrupt from system point view. Let me know if I'm wrong.
> Thanks.
> 

Make sure that
1- The grace period the hardware guys give you is long enough,
theoretically, to install the handler _and_ handle it
1a- The grace period is guaranteed no matter how long the system has
been on for
2- make the interrupt level-triggered so you don't miss it (note that
that interrupt should be disabled before a handler is registered so you
won't get a spurious interrupt)
3- ensure you don't do anything which would be affected by incorrect
power, such as fs writes, before the handler has been installed both in
the kernel and in userspace.

You should be fine :-)

	--Ben.