graceful handling of removing a plugable storage device that is being written to

graceful handling of removing a plugable storage device that is being written to

[Martin Steigerwald]

Cc to BTRFS mailinglist as it triggered the idea of mine again.


Hi!

Today I did it again and removed a BTRFS partition that is written too. 
That BTRFS as of Kernel 3.0.3 (debian package) does not like very much. I 
think thats a known issue and I wrote a mail to BTRFS mailing list about 
it.

In there I wrote:

> Expected results:
> 
> BTRFS fails gracefully except the loss of data from writes in flight, the 
> machine remains usable and BTRFS can be mounted again.

And then cause the expected results IMHO are by no way the ideal results:


> Ideal results (IMHO):
> 
> Linux behaved like AmigaOS and told me that I *must* insert the device 
> again and *continues* writing after I did this. 

But I never saw any other OS that did that.

And I see the problems with high bandwidth writes piling up in memory 
causing severe memory pressure.

But then could Linux just freeze processes that continue writing to the 
drive until it is replugged again? Of course that shouldn´t happen to the 
drive / resides on.

And there is a userspace part in it - the possibly udev and dbus  driven 
notification to the user.

Yet despite all of this NetBSD has a gsoc 2011 project at least suggested 
for exactly this behavior:

Graceful USB disk detach/reattach
http://wiki.netbsd.org/projects/gsoc_2011/disk-removal/

They even mention the Amiga in there.

Okay, its only for USB, not for eSATA, but I think it should be made 
generic for removable devices.

Would that be possible? I gladly file an enhancement request about it or 
help testing it.

I think thats the only approach that makes sense here. USB sticks and 
harddisks have no means to disallow device removal at any time. Thus the 
OS should offer the user a way to rethink the decision and plug the device 
in to prevent data loss. Actually I am surprised that no other operating 
system except AmigaOS seemed to offer this behavior. Well I am not quite 
sure about MS-DOS writing to disk. Maybe it even did that. But I did not 
use MS-DOS often. 

All current mainstream operating systems I know of default to loose data 
in that case. I think there is a better choice. What do you think? Might 
not be much of a server feature, but important for the desktop.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: graceful handling of removing a plugable storage device that is being written to

[Hin-Tak Leung]

--- On Sun, 11/9/11, Martin Steigerwald <Martin@lichtvoll.de> wrote:

> Cc to BTRFS mailinglist as it
> triggered the idea of mine again.
> 
> 
> Hi!
> 
> Today I did it again and removed a BTRFS partition that is
> written too. 
> That BTRFS as of Kernel 3.0.3 (debian package) does not
> like very much. I 
> think thats a known issue and I wrote a mail to BTRFS
> mailing list about 
> it.
> 
> In there I wrote:
> 
> > Expected results:
> > 
> > BTRFS fails gracefully except the loss of data from
> writes in flight, the 
> > machine remains usable and BTRFS can be mounted
> again.
> 
> And then cause the expected results IMHO are by no way the
> ideal results:
> 
> 
> > Ideal results (IMHO):
> > 
> > Linux behaved like AmigaOS and told me that I *must*
> insert the device 
> > again and *continues* writing after I did this. 
> 
> But I never saw any other OS that did that.
> 
> And I see the problems with high bandwidth writes piling up
> in memory 
> causing severe memory pressure.
> 
> But then could Linux just freeze processes that continue
> writing to the 
> drive until it is replugged again? Of course that
> shouldn´t happen to the 
> drive / resides on.
> 
> And there is a userspace part in it - the possibly udev and
> dbus  driven 
> notification to the user.

How do you cope with 
(1) headless systems (one where there is no udev/dbus notification or display).
(2) the user walking off in a hurry and never seeing the notification?
Should the kernel/user processes freeze indefinitely?

There is also a 3rd scenario - how how one malicious person or process doing a repeat insert/remove/write and get resource to pile up and crash the machine?

It is probably possible/recommended with Amiga because Amiga is seldomly run headless?

> 
> Yet despite all of this NetBSD has a gsoc 2011 project at
> least suggested 
> for exactly this behavior:
> 
> Graceful USB disk detach/reattach
> http://wiki.netbsd.org/projects/gsoc_2011/disk-removal/
> 
> They even mention the Amiga in there.
> 
> Okay, its only for USB, not for eSATA, but I think it
> should be made 
> generic for removable devices.
> 
> Would that be possible? I gladly file an enhancement
> request about it or 
> help testing it.
> 
> I think thats the only approach that makes sense here. USB
> sticks and 
> harddisks have no means to disallow device removal at any
> time. Thus the 
> OS should offer the user a way to rethink the decision and
> plug the device 
> in to prevent data loss. Actually I am surprised that no
> other operating 
> system except AmigaOS seemed to offer this behavior. Well I
> am not quite 
> sure about MS-DOS writing to disk. Maybe it even did that.
> But I did not 
> use MS-DOS often. 
> 
> All current mainstream operating systems I know of default
> to loose data 
> in that case. I think there is a better choice. What do you
> think? Might 
> not be much of a server feature, but important for the
> desktop.
> 
> Ciao,
> -- 
> Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
> GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599
> 84C7
> --
> To unsubscribe from this list: send the line "unsubscribe
> linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: graceful handling of removing a plugable storage device that is being written to

[Martin Steigerwald]

Am Sonntag, 11. September 2011 schrieb Hin-Tak Leung:
> --- On Sun, 11/9/11, Martin Steigerwald <Martin@lichtvoll.de> wrote:
> > Cc to BTRFS mailinglist as it
> > triggered the idea of mine again.
> > 
> > 
> > Hi!
> > 
> > Today I did it again and removed a BTRFS partition that is
> > written too.
> > That BTRFS as of Kernel 3.0.3 (debian package) does not
> > like very much. I
> > think thats a known issue and I wrote a mail to BTRFS
> > mailing list about
> > it.
> > 
> > In there I wrote:
> > > Expected results:
> > > 
> > > BTRFS fails gracefully except the loss of data from
> > 
> > writes in flight, the
> > 
> > > machine remains usable and BTRFS can be mounted
> > 
> > again.
> > 
> > And then cause the expected results IMHO are by no way the
> > 
> > ideal results:
> > > Ideal results (IMHO):
> > > 
> > > Linux behaved like AmigaOS and told me that I *must*
> > 
> > insert the device
> > 
> > > again and *continues* writing after I did this.
> > 
> > But I never saw any other OS that did that.
> > 
> > And I see the problems with high bandwidth writes piling up
> > in memory
> > causing severe memory pressure.
> > 
> > But then could Linux just freeze processes that continue
> > writing to the
> > drive until it is replugged again? Of course that
> > shouldn´t happen to the
> > drive / resides on.
> > 
> > And there is a userspace part in it - the possibly udev and
> > dbus  driven
> > notification to the user.
> 
> How do you cope with
> (1) headless systems (one where there is no udev/dbus notification or
> display). (2) the user walking off in a hurry and never seeing the
> notification? Should the kernel/user processes freeze indefinitely?
> 
> There is also a 3rd scenario - how how one malicious person or process
> doing a repeat insert/remove/write and get resource to pile up and
> crash the machine?
> 
> It is probably possible/recommended with Amiga because Amiga is
> seldomly run headless?

This all are important and valid questions, IMHO. Still I think the 
approach taken by AmigaOS has some merit here.

(1) headless systems:

a) servers usually do not have much to do with removable media. But still, 
what about FC or iSCSI LUNs? What should the kernel do here? Frankly, I 
don´t know. Maybe its best to default to current behavior which imposes a 
risk for data loss. But then NFS is used in enterprise environments, too, 
and it does block by default. Indefinetely. I have seen loads of 300 and 
more cause of that behavior which is there to *prevent* data loss on NFS 
clients.

b) headless media systems: maybe its best to have to default to current 
behavior, when its known that a notification can´t be done. But how to 
tell? Maybe best would be a timeout. Then the user even would have a 
chance to reinsert the media.

(2) I thought about how long to wait / possibly freeze processes as well: 
Maybe it would be good to let go after a while. But I think that also 
depends on whether more writes are done. If its an USB stick and the user 
just copied some files to it and removed it prematurely without noticing 
the notification, then I think the kernel could wait indefinetely.

*But* and this brings up a serious issue, I did not think about before: 
When the user mounts the USB stick somewhere else and finds out about the 
missing files only by then, there is a real risk for data loss, if the 
kernel of the machine that stalled the I/O insists on completing the 
writes if the user inserts the USB stick again.

Thus it seems to me that the kernel would have to check the last mount 
time and the filesystem state. If the last mount time is newer and/or the 
filesystem is cleanly unmounted, I think the kernel must refuse any further 
attempts to complete outstanding writes in order to protect filesystem 
integrity.

Frankly, I never tried this on AmigaOS. I know that AmigaOS expects the 
exact same floppy disk to be inserted again. Only the same name isn´t 
enough. But I have no idea, what AmigaOS would have done, when I inserted 
the disk into another Amiga, did something there and then insert it into 
the Amiga with the notification and pressed "okay". Probably it would have 
eaten the disk then.

This is a serious issue which makes implementing my suggestion more 
difficult. The kernel has to make sure not to eat a filesystem in order to 
complete outstanding writes!

(3) I wouldn´t worry too much about malicious persons. Why? Cause with 
current standard ulimit values there are way easier methods to stall a 
machine to a halt. I have seen more than once during holding Linux 
performance tuning courses, that running the command "stress" with 
aggressive parameters effectively offlines a Linux machine. I often do a 
check list on how often course participants make one of the Linux servers 
we work so unresponsive that a reboot is in order. So I think graceful 
handling of media removal doesn´t add much to the existing issues 
regarding that topic.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: graceful handling of removing a plugable storage device that is being written to

[Geert Uytterhoeven]

On Sun, Sep 11, 2011 at 18:53, Martin Steigerwald <Martin@lichtvoll.de> wrote:
> Frankly, I never tried this on AmigaOS. I know that AmigaOS expects the
> exact same floppy disk to be inserted again. Only the same name isn´t
> enough. But I have no idea, what AmigaOS would have done, when I inserted

Are you sure the volume name wasn't enough?
For most things, it relied on the volume name, that's how you could map
floppy names to hard drive directories using assigns.

> the disk into another Amiga, did something there and then insert it into
> the Amiga with the notification and pressed "okay". Probably it would have
> eaten the disk then.

Most probably.

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: graceful handling of removing a plugable storage device that is being written to

[Martin Steigerwald]

Am Sonntag, 11. September 2011 schrieb Geert Uytterhoeven:
> On Sun, Sep 11, 2011 at 18:53, Martin Steigerwald <Martin@lichtvoll.de> 
wrote:
> > Frankly, I never tried this on AmigaOS. I know that AmigaOS expects
> > the exact same floppy disk to be inserted again. Only the same name
> > isn´t enough. But I have no idea, what AmigaOS would have done, when
> > I inserted
> 
> Are you sure the volume name wasn't enough?
> For most things, it relied on the volume name, that's how you could map
> floppy names to hard drive directories using assigns.

Yes, I am pretty sure I tried that back then. Difficult to repeat at the 
moment, since I think at least the high density disk drive in my A4000 is 
toast since quite a while.

Anyway, if Linux implements something like that it should not only depend 
on a volume name ;).

-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7