[PATCH] [RFC] New fsck option to ignore device-mapper crypto devices

[PATCH] [RFC] New fsck option to ignore device-mapper crypto devices

[Matthias Koenig]

[-- Attachment #1: Type: text/plain, Size: 1165 bytes --]

Hi,

Current practice in defining crypto devices in common distributions
has:
1. A definition of the device-mapper name with the corresponding device
   in /etc/crypttab
2. A definition in /etc/fstab for the mountpoint of the dm device.

Steps involved into setting up the crypto devices are
a. fsck local filesystems
b. mount local filesystems
c. device-mapper set up of crypto devices
d. fsck crypto filesystems
e. mount crypto filesystems

Steps a.+b. have to be done before the crypto device setup, because
the crypto device could be in a file container on a local filesystem.

Now, the problem appears if /etc/fstab contains a mount point of a
crypto device which is supposed to be fsck'd in step d.  fsck will
fail in step a., since this device does not exist at this point in
the boot process (it will be set up in step c.)

In order to address this, I propose a new option for fsck, lets say '-X'.
Enabling this will skip a device-mapper device which is currently
nonexistent, but is defined in /etc/crypttab.
In this way crypto devices could be skipped without fsck failure when
running fsck -A.
Proposed patch to implement this below.

Regards,
Matthias


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: e2fsprogs-1.40.7-ignore_crypto.patch --]
[-- Type: text/x-patch, Size: 3640 bytes --]

Index: e2fsprogs-1.40.7/misc/fsck.c
===================================================================
--- e2fsprogs-1.40.7.orig/misc/fsck.c
+++ e2fsprogs-1.40.7/misc/fsck.c
@@ -102,6 +102,7 @@ int noexecute = 0;
 int serialize = 0;
 int skip_root = 0;
 int ignore_mounted = 0;
+int ignore_crypto = 0;
 int notitle = 0;
 int parallel_root = 0;
 int progress = 0;
@@ -937,6 +938,67 @@ static int device_already_active(char *d
 	return 0;
 }
 
+/*
+ * Check if a device exists
+ */
+static int device_exists(const char *device)
+{
+	struct stat st;
+
+	if (stat(device, &st) == -1)
+		return 0;
+
+	if (!S_ISBLK(st.st_mode))
+		return 0;
+
+	return 1;
+}
+
+static void skipline(FILE *file)
+{
+	int c;
+
+	do {
+		c = fgetc(file);
+	} while (c != '\n' && c != EOF);
+}
+
+/*
+ * Check if a device is defined in /etc/crypttab
+ */
+static int device_in_crypttab(const char *device)
+{
+	const char crypttab[]="/etc/crypttab";
+	const char dmpath[]="/dev/mapper/";
+	char name[1024];
+	FILE *file;
+
+	/* check if device is a dm device
+	 * we can skip parsing if not
+	 */
+	if (strncmp(dmpath, device, strlen(dmpath)) != 0)
+		return 0;
+
+	file = fopen(crypttab, "r");
+	if (!file)
+		return 0;
+
+	while (fscanf(file, "%1023s", name) == 1) {
+		if (name[0] == '#') {
+			skipline(file);
+			continue;
+		}
+
+		if (strcmp(device+strlen(dmpath), name) == 0) {
+			fclose(file);
+			return 1;
+		}
+		skipline(file);
+	}
+	fclose(file);
+	return 0;
+}
+
 /* Check all file systems, using the /etc/fstab table. */
 static int check_all(NOARGS)
 {
@@ -1005,6 +1067,17 @@ static int check_all(NOARGS)
 				not_done_yet++;
 				continue;
 			}
+			/* ignore devices which do not exist yet, but are defined
+			 * in /etc/crypttab
+			 */
+			if (ignore_crypto && !device_exists(fs->device) &&
+					device_in_crypttab(fs->device)) {
+				if (verbose)
+					printf("Skipping nonexistent crypto device %s\n",
+							fs->device);
+				fs->flags |= FLAG_DONE;
+				continue;
+			}
 			if (ignore_mounted && is_mounted(fs->device)) {
 				fs->flags |= FLAG_DONE;
 				continue;
@@ -1058,7 +1131,7 @@ static int check_all(NOARGS)
 
 static void usage(NOARGS)
 {
-	fputs(_("Usage: fsck [-AMNPRTV] [ -C [ fd ] ] [-t fstype] [fs-options] [filesys ...]\n"), stderr);
+	fputs(_("Usage: fsck [-AMNPRTVX] [ -C [ fd ] ] [-t fstype] [fs-options] [filesys ...]\n"), stderr);
 	exit(EXIT_USAGE);
 }
 
@@ -1202,6 +1275,9 @@ static void PRS(int argc, char *argv[])
 				fstype = string_copy(tmp);
 				compile_fs_type(fstype, &fs_type_compiled);
 				goto next_arg;
+			case 'X':
+				ignore_crypto = 1;
+				break;
 			case '-':
 				opts_for_fsck++;
 				break;
@@ -1303,6 +1379,17 @@ int main(int argc, char *argv[])
 			if (!fs)
 				continue;
 		}
+		/*
+		 * Ignore devices which do not exist,
+		 * but are defined in /etc/crypttab
+		 */
+		if (ignore_crypto && !device_exists(fs->device) &&
+			   device_in_crypttab(fs->device)) {
+			if (verbose)
+				printf("Skipping nonexistent crypto device %s\n",
+						fs->device);
+			continue;
+		}
 		if (ignore_mounted && is_mounted(fs->device))
 			continue;
 		fsck_device(fs, interactive);
Index: e2fsprogs-1.40.7/misc/fsck.8.in
===================================================================
--- e2fsprogs-1.40.7.orig/misc/fsck.8.in
+++ e2fsprogs-1.40.7/misc/fsck.8.in
@@ -263,6 +263,10 @@ Don't show the title on startup.
 Produce verbose output, including all file system-specific commands
 that are executed.
 .TP
+.B \-X
+Skip device-mapper devices which currently do not exist, but are defined in 
+.BR crypttab (5).
+.TP
 .B fs-specific-options
 Options which are not understood by 
 .B fsck 

Re: [PATCH] [RFC] New fsck option to ignore device-mapper crypto devices

[Dave Kleikamp]

On Thu, 2008-03-06 at 14:41 +0100, Matthias Koenig wrote:
> Hi,
> 
> Current practice in defining crypto devices in common distributions
> has:
> 1. A definition of the device-mapper name with the corresponding device
>    in /etc/crypttab
> 2. A definition in /etc/fstab for the mountpoint of the dm device.
> 
> Steps involved into setting up the crypto devices are
> a. fsck local filesystems
> b. mount local filesystems
> c. device-mapper set up of crypto devices
> d. fsck crypto filesystems

How is fsck invoked here?  Does it use the -A flag?

> e. mount crypto filesystems
> 
> Steps a.+b. have to be done before the crypto device setup, because
> the crypto device could be in a file container on a local filesystem.
> 
> Now, the problem appears if /etc/fstab contains a mount point of a
> crypto device which is supposed to be fsck'd in step d.  fsck will
> fail in step a., since this device does not exist at this point in
> the boot process (it will be set up in step c.)

Should field 8 of /etc/fstab (fs_passno) be zero for these mount points?
Is there any reason for it to be anything different?

Alternately, would it make sense to define a special value for this
field that tells fsck to silently ignore it if the device does not
exist?

> In order to address this, I propose a new option for fsck, lets say '-X'.
> Enabling this will skip a device-mapper device which is currently
> nonexistent, but is defined in /etc/crypttab.

Could it be simplified to simply skip non-existent devices?  Should it
really be crypttab-specific?

> In this way crypto devices could be skipped without fsck failure when
> running fsck -A.
> Proposed patch to implement this below.
> 
> Regards,
> Matthias
-- 
David Kleikamp
IBM Linux Technology Center

Re: [PATCH] [RFC] New fsck option to ignore device-mapper crypto devices

[Matthias Koenig]

Dave Kleikamp <shaggy@linux.vnet.ibm.com> writes:

> On Thu, 2008-03-06 at 14:41 +0100, Matthias Koenig wrote:
>> Hi,
>> 
>> Current practice in defining crypto devices in common distributions
>> has:
>> 1. A definition of the device-mapper name with the corresponding device
>>    in /etc/crypttab
>> 2. A definition in /etc/fstab for the mountpoint of the dm device.
>> 
>> Steps involved into setting up the crypto devices are
>> a. fsck local filesystems
>> b. mount local filesystems
>> c. device-mapper set up of crypto devices
>> d. fsck crypto filesystems
>
> How is fsck invoked here?  Does it use the -A flag?

fsck is invoked with -A option only in step a.
Later on in step d. fsck is just called for the crypto filesystems in 
question.

>> e. mount crypto filesystems
>> 
>> Steps a.+b. have to be done before the crypto device setup, because
>> the crypto device could be in a file container on a local filesystem.
>> 
>> Now, the problem appears if /etc/fstab contains a mount point of a
>> crypto device which is supposed to be fsck'd in step d.  fsck will
>> fail in step a., since this device does not exist at this point in
>> the boot process (it will be set up in step c.)
>
> Should field 8 of /etc/fstab (fs_passno) be zero for these mount points?
> Is there any reason for it to be anything different?

Why? zero would mean that they should never get checked.
I think it is reasonable to have the choice to get your crypto
filesystems checked. Current practise for SuSE has been to allow
only 0, but checked this filesystem anyway, which has lead to complaints.
So we want to do this more consistent. 

> Alternately, would it make sense to define a special value for this
> field that tells fsck to silently ignore it if the device does not
> exist?

Yes, this would be an alternative. The mount option nofail which has
recently been added in util-linux-ng would be suitable for this. 
However, we prefer a solution to have fsck automatically do the right
thing.

>> In order to address this, I propose a new option for fsck, lets say '-X'.
>> Enabling this will skip a device-mapper device which is currently
>> nonexistent, but is defined in /etc/crypttab.
>
> Could it be simplified to simply skip non-existent devices?  Should it
> really be crypttab-specific?

Well, I would not disagree.
Currently fsck does not check and calls the default filesystem specific
checker fsck.ext2 if a device does not exist, which then fails with
exit code 8.
An option to simply skip non-existent devices is Ok for me.
I just haven't had the heart to change this to skip any device, since the
problem only relates to the crypto devices and I haven't been sure
if the current behaviour is intentional.

Matthias

Re: [PATCH] [RFC] New fsck option to ignore device-mapper crypto devices

[Dave Kleikamp]

On Thu, 2008-03-06 at 18:04 +0100, Matthias Koenig wrote:
> Dave Kleikamp <shaggy@linux.vnet.ibm.com> writes:

> > Should field 8 of /etc/fstab (fs_passno) be zero for these mount points?
> > Is there any reason for it to be anything different?
> 
> Why? zero would mean that they should never get checked.
> I think it is reasonable to have the choice to get your crypto
> filesystems checked. Current practise for SuSE has been to allow
> only 0, but checked this filesystem anyway, which has lead to complaints.
> So we want to do this more consistent.

Zero tells fsck not to check the filesystem during reboot.  It's what
tells fsck -A which filesystems to check.  If we don't expect the
filesystem to be check-able during that phase, a non-zero value won't
have any real meaning.
-- 
David Kleikamp
IBM Linux Technology Center

Re: [PATCH] [RFC] New fsck option to ignore device-mapper crypto devices

[Theodore Tso]

On Thu, Mar 06, 2008 at 11:23:47AM -0600, Dave Kleikamp wrote:
> Zero tells fsck not to check the filesystem during reboot.  It's what
> tells fsck -A which filesystems to check.  If we don't expect the
> filesystem to be check-able during that phase, a non-zero value won't
> have any real meaning.

I'm now beginning to understand why SuSE wanted fsck -M/-m (ignore
mounted filesystems).  Looks like SuSE has a very strange and
non-standard usage scenario with fsck -A which is *not* just at
boot-time.

So before we try to figure out whether Yet Another FSCK option makes
sense, maybe it would be good to get an explanation exactly *how* SuSE
is using fsck -A, besides just at boot-time?

						- Ted

Re: [PATCH] [RFC] New fsck option to ignore device-mapper crypto devices

[Matthias Koenig]

Dave Kleikamp <shaggy@linux.vnet.ibm.com> writes:

> On Thu, 2008-03-06 at 18:04 +0100, Matthias Koenig wrote:
>> Dave Kleikamp <shaggy@linux.vnet.ibm.com> writes:
>
>> > Should field 8 of /etc/fstab (fs_passno) be zero for these mount points?
>> > Is there any reason for it to be anything different?
>> 
>> Why? zero would mean that they should never get checked.
>> I think it is reasonable to have the choice to get your crypto
>> filesystems checked. Current practise for SuSE has been to allow
>> only 0, but checked this filesystem anyway, which has lead to complaints.
>> So we want to do this more consistent.
>
> Zero tells fsck not to check the filesystem during reboot.  It's what
> tells fsck -A which filesystems to check.  If we don't expect the
> filesystem to be check-able during that phase, a non-zero value won't
> have any real meaning.

I see, but what are we doing with crypto filesystems for which the devices
simply do not exist at this phase in the boot process?
How should we specify that we want these filesystems to be checked or not 
at a later time in the boot process after the crypto devices have been
set up? 

Matthias

Re: [PATCH] [RFC] New fsck option to ignore device-mapper crypto devices

[Dave Kleikamp]

On Fri, 2008-03-07 at 15:20 +0100, Matthias Koenig wrote:
> Dave Kleikamp <shaggy@linux.vnet.ibm.com> writes:
> 
> > On Thu, 2008-03-06 at 18:04 +0100, Matthias Koenig wrote:
> >> Dave Kleikamp <shaggy@linux.vnet.ibm.com> writes:
> >
> >> > Should field 8 of /etc/fstab (fs_passno) be zero for these mount points?
> >> > Is there any reason for it to be anything different?
> >> 
> >> Why? zero would mean that they should never get checked.
> >> I think it is reasonable to have the choice to get your crypto
> >> filesystems checked. Current practise for SuSE has been to allow
> >> only 0, but checked this filesystem anyway, which has lead to complaints.
> >> So we want to do this more consistent.
> >
> > Zero tells fsck not to check the filesystem during reboot.  It's what
> > tells fsck -A which filesystems to check.  If we don't expect the
> > filesystem to be check-able during that phase, a non-zero value won't
> > have any real meaning.
> 
> I see, but what are we doing with crypto filesystems for which the devices
> simply do not exist at this phase in the boot process?

I don't understand the question.  If the fs_passno field is zero, fsck
isn't even going to try to check the filesystem, so having no device is
no problem.

> How should we specify that we want these filesystems to be checked or not 
> at a later time in the boot process after the crypto devices have been
> set up?

This is why I asked if fsck was being run with the -A flag in step d.
If it's not, then I'm not clear on why fs_passno has anything to do with
it.  Is there some script that looks at this field in /etc/fstab for
step d?

If that's the case, then I suggest a more general solution.  Either some
special value for fs_passno that defers the fsck for a later pass, or a
simplified version of your proposed patch without the crypto-specific
part.
-- 
David Kleikamp
IBM Linux Technology Center

Re: [PATCH] [RFC] New fsck option to ignore device-mapper crypto devices

[Matthias Koenig]

Dave Kleikamp <shaggy@linux.vnet.ibm.com> writes:

> I don't understand the question.  If the fs_passno field is zero, fsck
> isn't even going to try to check the filesystem, so having no device is
> no problem.

Agreed, in case of zero there is no problem.

>> How should we specify that we want these filesystems to be checked or not 
>> at a later time in the boot process after the crypto devices have been
>> set up?
>
> This is why I asked if fsck was being run with the -A flag in step d.
> If it's not, then I'm not clear on why fs_passno has anything to do with
> it.  Is there some script that looks at this field in /etc/fstab for
> step d?

Right, fsck -A is *not* called in step d and the fs_passno is in
principle unrelated to this.
There is a boot script for setting up, fscking and mounting the 
crypto devices. As the crypto fileystems are supposed to be defined in
/etc/fstab looking at fs_passno by the script to find out if the
filesystem should be checked or not seems reasonable to me.

> If that's the case, then I suggest a more general solution.  Either some
> special value for fs_passno that defers the fsck for a later pass, or a
> simplified version of your proposed patch without the crypto-specific
> part.

1. Do we agree that fstab fs_passno could be used besides fsck -A by a 
   separate script to decide if a filesystem (that could not be checked 
   in the fsck -A phase) should be checked?

2. If yes, what would be a proper way?

   a) special value for fs_passno?
      I don't know if this is reasonable, we already have a "nofail"
      mount option. We can already call fsck with something like
      (assuming crypto devices being tagged with the nofail option)
      fsck -A -t noopts=nofail 
      So giving either a special fs_passno value or an additional option 
      seems to be the same: one has to put an additional value into fstab.

   b) a general "ignore if device does not exist" option?
      Still I am not sure if we really *generally* want to ignore
      devices which are not available at fsck -A phase. The behaviour
      of fsck to fail in this case is probably on purpose?

Note, the whole discussion applies to crypto filesystems living in a
container file. Encrypted block devices can be set up earlier so they
could be checked in the normal fsck -A stage (Suse currently does not
do this, but I am planning to change this).
So, since the issue will remain only for file containers, it might be not
worth adding a new option for fsck. But it would be nice have the
fsck behaviour on non-existent device failure clarified.

Thanks,
Matthias

Re: [PATCH] [RFC] New fsck option to ignore device-mapper crypto devices

[Theodore Tso]

On Wed, Mar 12, 2008 at 04:59:03PM +0100, Matthias Koenig wrote:
> 1. Do we agree that fstab fs_passno could be used besides fsck -A by a 
>    separate script to decide if a filesystem (that could not be checked 
>    in the fsck -A phase) should be checked?

Well, historically there have been /etc/fstab entries which look like
this:

/dev/zipdrive       /media/zipdisk   ext3 default,noauto 0 0

This is because there might not be a zip disk in the drive at the time
of the boot.  However, a root user could insert a zip disk, and then
run the command "fsck /dev/zipdrive" and expect it to work, and the
follow that up with "mount /dev/zipdrive".

So passno == 0 can't mean "don't mount the filesystem, ever".  All it
means is "don't mount it at boot".  So I don't think trying to kludge
new fields into fs_passno is the way to go.

------------

So here's another way you could do things that doesn't require any
patches to fsck at all.  Just add "comment=crypto" into the option
field of /etc/fstab, like this.  

/dev/mapper/crypto-frobnitz /crypto1           ext3    comment=crypto  1 1

Then you can skip filesystems that have the comment=crypto in the
options field via:

     fsck -AR -t opts=!comment=crypto

To check all of the filesystems that have the comment=crypto field
via:

     fsck -AR -t opts=comment=crypto

Why did I use "comment=crypto"?  Because if you use "crypto", then
mount will try to pass crypto to the kernel, and if the kernel doesn't
understand an option, it will reject the mount.  Mount doesn't send
"comment" mount options to the kernel, so this works.

Of course, you'll need to modify mount to be able to only mount all
crypto filesystems, or not to mount any crypto filesystems, so you
could change it to understand that "crypto" is not to be passed down
to the kernel, at which point you could just have this in /etc/fstab:

/dev/mapper/crypto-frobnitz /crypto1           ext3    crypto  1 1

And then the fsck invocations would become:

     fsck -AR -t opts=!crypto

and

     fsck -AR -t opts=crypto

Which of course looks much nicer.

I put in the "-t opts=" matching a long time ago (as in 10+ years
ago), because someone had wanted to do something strange like this.  I
had completely forgotten about it until now....

							- Ted

Re: [PATCH] [RFC] New fsck option to ignore device-mapper crypto devices

[Theodore Tso]

On Wed, Mar 12, 2008 at 04:02:36PM -0400, Theodore Tso wrote:
> So here's another way you could do things that doesn't require any
> patches to fsck at all.  Just add "comment=crypto" into the option
> field of /etc/fstab, like this.  
> 
> /dev/mapper/crypto-frobnitz /crypto1           ext3    comment=crypto  1 1
> 
> Then you can skip filesystems that have the comment=crypto in the
> options field via:
> 
>      fsck -AR -t opts=!comment=crypto

Oh by the way, in case you haven't tried this, the best way to test
exactly what fsck will do with a set of arguments is by using the -NV
options, i.e.:

      fsck -ARNV -t opts=!comment=crypto

						- Ted

Re: [PATCH] [RFC] New fsck option to ignore device-mapper crypto devices

[Dave Kleikamp]

This is just a nitpick that doesn't change your argument at all.

On Wed, 2008-03-12 at 16:02 -0400, Theodore Tso wrote:
> On Wed, Mar 12, 2008 at 04:59:03PM +0100, Matthias Koenig wrote:
> > 1. Do we agree that fstab fs_passno could be used besides fsck -A by a 
> >    separate script to decide if a filesystem (that could not be checked 
> >    in the fsck -A phase) should be checked?
> 
> Well, historically there have been /etc/fstab entries which look like
> this:
> 
> /dev/zipdrive       /media/zipdisk   ext3 default,noauto 0 0
> 
> This is because there might not be a zip disk in the drive at the time
> of the boot.  However, a root user could insert a zip disk, and then
> run the command "fsck /dev/zipdrive" and expect it to work, and the
> follow that up with "mount /dev/zipdrive".
> 
> So passno == 0 can't mean "don't mount the filesystem, ever".  All it
> means is "don't mount it at boot".  So I don't think trying to kludge
> new fields into fs_passno is the way to go.

s/mount/check/ in the above paragraph.  passno only tells fsck what to
do.  "noauto" tells mount what to mount at boot time.

-- 
David Kleikamp
IBM Linux Technology Center