graceful handling of removing a plugable storage device that is being written to

graceful handling of removing a plugable storage device that is being written to

[Martin Steigerwald]

Cc to BTRFS mailinglist as it triggered the idea of mine again.


Hi!

Today I did it again and removed a BTRFS partition that is written too.=
=20
That BTRFS as of Kernel 3.0.3 (debian package) does not like very much.=
 I=20
think thats a known issue and I wrote a mail to BTRFS mailing list abou=
t=20
it.

In there I wrote:

> Expected results:
>=20
> BTRFS fails gracefully except the loss of data from writes in flight,=
 the=20
> machine remains usable and BTRFS can be mounted again.

And then cause the expected results IMHO are by no way the ideal result=
s:


> Ideal results (IMHO):
>=20
> Linux behaved like AmigaOS and told me that I *must* insert the devic=
e=20
> again and *continues* writing after I did this.=20

But I never saw any other OS that did that.

And I see the problems with high bandwidth writes piling up in memory=20
causing severe memory pressure.

But then could Linux just freeze processes that continue writing to the=
=20
drive until it is replugged again? Of course that shouldn=B4t happen to=
 the=20
drive / resides on.

And there is a userspace part in it - the possibly udev and dbus  drive=
n=20
notification to the user.

Yet despite all of this NetBSD has a gsoc 2011 project at least suggest=
ed=20
for exactly this behavior:

Graceful USB disk detach/reattach
http://wiki.netbsd.org/projects/gsoc_2011/disk-removal/

They even mention the Amiga in there.

Okay, its only for USB, not for eSATA, but I think it should be made=20
generic for removable devices.

Would that be possible? I gladly file an enhancement request about it o=
r=20
help testing it.

I think thats the only approach that makes sense here. USB sticks and=20
harddisks have no means to disallow device removal at any time. Thus th=
e=20
OS should offer the user a way to rethink the decision and plug the dev=
ice=20
in to prevent data loss. Actually I am surprised that no other operatin=
g=20
system except AmigaOS seemed to offer this behavior. Well I am not quit=
e=20
sure about MS-DOS writing to disk. Maybe it even did that. But I did no=
t=20
use MS-DOS often.=20

All current mainstream operating systems I know of default to loose dat=
a=20
in that case. I think there is a better choice. What do you think? Migh=
t=20
not be much of a server feature, but important for the desktop.

Ciao,
--=20
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: graceful handling of removing a plugable storage device that is being written to

[Hin-Tak Leung]

--- On Sun, 11/9/11, Martin Steigerwald <Martin@lichtvoll.de> wrote:

> Cc to BTRFS mailinglist as it
> triggered the idea of mine again.
>=20
>=20
> Hi!
>=20
> Today I did it again and removed a BTRFS partition that is
> written too.=20
> That BTRFS as of Kernel 3.0.3 (debian package) does not
> like very much. I=20
> think thats a known issue and I wrote a mail to BTRFS
> mailing list about=20
> it.
>=20
> In there I wrote:
>=20
> > Expected results:
> >=20
> > BTRFS fails gracefully except the loss of data from
> writes in flight, the=20
> > machine remains usable and BTRFS can be mounted
> again.
>=20
> And then cause the expected results IMHO are by no way the
> ideal results:
>=20
>=20
> > Ideal results (IMHO):
> >=20
> > Linux behaved like AmigaOS and told me that I *must*
> insert the device=20
> > again and *continues* writing after I did this.=20
>=20
> But I never saw any other OS that did that.
>=20
> And I see the problems with high bandwidth writes piling up
> in memory=20
> causing severe memory pressure.
>=20
> But then could Linux just freeze processes that continue
> writing to the=20
> drive until it is replugged again? Of course that
> shouldn=B4t happen to the=20
> drive / resides on.
>=20
> And there is a userspace part in it - the possibly udev and
> dbus=A0 driven=20
> notification to the user.

How do you cope with=20
(1) headless systems (one where there is no udev/dbus notification or d=
isplay).
(2) the user walking off in a hurry and never seeing the notification?
Should the kernel/user processes freeze indefinitely?

There is also a 3rd scenario - how how one malicious person or process =
doing a repeat insert/remove/write and get resource to pile up and cras=
h the machine?

It is probably possible/recommended with Amiga because Amiga is seldoml=
y run headless?

>=20
> Yet despite all of this NetBSD has a gsoc 2011 project at
> least suggested=20
> for exactly this behavior:
>=20
> Graceful USB disk detach/reattach
> http://wiki.netbsd.org/projects/gsoc_2011/disk-removal/
>=20
> They even mention the Amiga in there.
>=20
> Okay, its only for USB, not for eSATA, but I think it
> should be made=20
> generic for removable devices.
>=20
> Would that be possible? I gladly file an enhancement
> request about it or=20
> help testing it.
>=20
> I think thats the only approach that makes sense here. USB
> sticks and=20
> harddisks have no means to disallow device removal at any
> time. Thus the=20
> OS should offer the user a way to rethink the decision and
> plug the device=20
> in to prevent data loss. Actually I am surprised that no
> other operating=20
> system except AmigaOS seemed to offer this behavior. Well I
> am not quite=20
> sure about MS-DOS writing to disk. Maybe it even did that.
> But I did not=20
> use MS-DOS often.=20
>=20
> All current mainstream operating systems I know of default
> to loose data=20
> in that case. I think there is a better choice. What do you
> think? Might=20
> not be much of a server feature, but important for the
> desktop.
>=20
> Ciao,
> --=20
> Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
> GPG: 03B0 0D6C 0040 0710 4AFA=A0 B82F 991B EAAC A599
> 84C7
> --
> To unsubscribe from this list: send the line "unsubscribe
> linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at=A0 http://vger.kernel.org/majordomo-info.html
>=20
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel=
" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: graceful handling of removing a plugable storage device that is being written to

[Martin Steigerwald]

Am Sonntag, 11. September 2011 schrieb Hin-Tak Leung:
> --- On Sun, 11/9/11, Martin Steigerwald <Martin@lichtvoll.de> wrote:
> > Cc to BTRFS mailinglist as it
> > triggered the idea of mine again.
> >=20
> >=20
> > Hi!
> >=20
> > Today I did it again and removed a BTRFS partition that is
> > written too.
> > That BTRFS as of Kernel 3.0.3 (debian package) does not
> > like very much. I
> > think thats a known issue and I wrote a mail to BTRFS
> > mailing list about
> > it.
> >=20
> > In there I wrote:
> > > Expected results:
> > >=20
> > > BTRFS fails gracefully except the loss of data from
> >=20
> > writes in flight, the
> >=20
> > > machine remains usable and BTRFS can be mounted
> >=20
> > again.
> >=20
> > And then cause the expected results IMHO are by no way the
> >=20
> > ideal results:
> > > Ideal results (IMHO):
> > >=20
> > > Linux behaved like AmigaOS and told me that I *must*
> >=20
> > insert the device
> >=20
> > > again and *continues* writing after I did this.
> >=20
> > But I never saw any other OS that did that.
> >=20
> > And I see the problems with high bandwidth writes piling up
> > in memory
> > causing severe memory pressure.
> >=20
> > But then could Linux just freeze processes that continue
> > writing to the
> > drive until it is replugged again? Of course that
> > shouldn=B4t happen to the
> > drive / resides on.
> >=20
> > And there is a userspace part in it - the possibly udev and
> > dbus  driven
> > notification to the user.
>=20
> How do you cope with
> (1) headless systems (one where there is no udev/dbus notification or
> display). (2) the user walking off in a hurry and never seeing the
> notification? Should the kernel/user processes freeze indefinitely?
>=20
> There is also a 3rd scenario - how how one malicious person or proces=
s
> doing a repeat insert/remove/write and get resource to pile up and
> crash the machine?
>=20
> It is probably possible/recommended with Amiga because Amiga is
> seldomly run headless?

This all are important and valid questions, IMHO. Still I think the=20
approach taken by AmigaOS has some merit here.

(1) headless systems:

a) servers usually do not have much to do with removable media. But sti=
ll,=20
what about FC or iSCSI LUNs? What should the kernel do here? Frankly, I=
=20
don=B4t know. Maybe its best to default to current behavior which impos=
es a=20
risk for data loss. But then NFS is used in enterprise environments, to=
o,=20
and it does block by default. Indefinetely. I have seen loads of 300 an=
d=20
more cause of that behavior which is there to *prevent* data loss on NF=
S=20
clients.

b) headless media systems: maybe its best to have to default to current=
=20
behavior, when its known that a notification can=B4t be done. But how t=
o=20
tell? Maybe best would be a timeout. Then the user even would have a=20
chance to reinsert the media.

(2) I thought about how long to wait / possibly freeze processes as wel=
l:=20
Maybe it would be good to let go after a while. But I think that also=20
depends on whether more writes are done. If its an USB stick and the us=
er=20
just copied some files to it and removed it prematurely without noticin=
g=20
the notification, then I think the kernel could wait indefinetely.

*But* and this brings up a serious issue, I did not think about before:=
=20
When the user mounts the USB stick somewhere else and finds out about t=
he=20
missing files only by then, there is a real risk for data loss, if the=20
kernel of the machine that stalled the I/O insists on completing the=20
writes if the user inserts the USB stick again.

Thus it seems to me that the kernel would have to check the last mount=20
time and the filesystem state. If the last mount time is newer and/or t=
he=20
filesystem is cleanly unmounted, I think the kernel must refuse any fur=
ther=20
attempts to complete outstanding writes in order to protect filesystem=20
integrity.

=46rankly, I never tried this on AmigaOS. I know that AmigaOS expects t=
he=20
exact same floppy disk to be inserted again. Only the same name isn=B4t=
=20
enough. But I have no idea, what AmigaOS would have done, when I insert=
ed=20
the disk into another Amiga, did something there and then insert it int=
o=20
the Amiga with the notification and pressed "okay". Probably it would h=
ave=20
eaten the disk then.

This is a serious issue which makes implementing my suggestion more=20
difficult. The kernel has to make sure not to eat a filesystem in order=
 to=20
complete outstanding writes!

(3) I wouldn=B4t worry too much about malicious persons. Why? Cause wit=
h=20
current standard ulimit values there are way easier methods to stall a=20
machine to a halt. I have seen more than once during holding Linux=20
performance tuning courses, that running the command "stress" with=20
aggressive parameters effectively offlines a Linux machine. I often do =
a=20
check list on how often course participants make one of the Linux serve=
rs=20
we work so unresponsive that a reboot is in order. So I think graceful=20
handling of media removal doesn=B4t add much to the existing issues=20
regarding that topic.

Ciao,
--=20
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: graceful handling of removing a plugable storage device that is being written to

[Geert Uytterhoeven]

On Sun, Sep 11, 2011 at 18:53, Martin Steigerwald <Martin@lichtvoll.de>=
 wrote:
> Frankly, I never tried this on AmigaOS. I know that AmigaOS expects t=
he
> exact same floppy disk to be inserted again. Only the same name isn=C2=
=B4t
> enough. But I have no idea, what AmigaOS would have done, when I inse=
rted

Are you sure the volume name wasn't enough?
=46or most things, it relied on the volume name, that's how you could m=
ap
floppy names to hard drive directories using assigns.

> the disk into another Amiga, did something there and then insert it i=
nto
> the Amiga with the notification and pressed "okay". Probably it would=
 have
> eaten the disk then.

Most probably.

Gr{oetje,eeting}s,

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-=
m68k.org

In personal conversations with technical people, I call myself a hacker=
=2E But
when I'm talking to journalists I just say "programmer" or something li=
ke that.
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0=C2=A0 =C2=A0=C2=A0 -- Linus Torvalds
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: graceful handling of removing a plugable storage device that is being written to

[Martin Steigerwald]

Am Sonntag, 11. September 2011 schrieb Geert Uytterhoeven:
> On Sun, Sep 11, 2011 at 18:53, Martin Steigerwald <Martin@lichtvoll.d=
e>=20
wrote:
> > Frankly, I never tried this on AmigaOS. I know that AmigaOS expects
> > the exact same floppy disk to be inserted again. Only the same name
> > isn=C2=B4t enough. But I have no idea, what AmigaOS would have done=
, when
> > I inserted
>=20
> Are you sure the volume name wasn't enough?
> For most things, it relied on the volume name, that's how you could m=
ap
> floppy names to hard drive directories using assigns.

Yes, I am pretty sure I tried that back then. Difficult to repeat at th=
e=20
moment, since I think at least the high density disk drive in my A4000 =
is=20
toast since quite a while.

Anyway, if Linux implements something like that it should not only depe=
nd=20
on a volume name ;).

--=20
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7