[PATCH] zfs module update

[PATCH] zfs module update

[Toomas Soome]

[-- Attachment #1: Type: text/plain, Size: 2262 bytes --]

hi!

this is the major update to grub2 zfs module; the work is based on Oracle code drops from Solaris 11, and Illumos.
I am including here two separate patch sets, first is zfs itself, second one is update to allow embedding.

supports:
- all current OpenZFS features
- Solaris 11 zpool versions (including reading encrypted datasets)
- can recognise and boot all pool configurations with exception that log device is not inspected (thats something to be investigated in future, while this code does not mind log device, its still probably not good idea to actually have one).
- pool is readable as long as there is enough parity to reconstruct data from
- supports multiple vdevs
- supports both mirror and raidz
- supports bootloader embedding as long as it will fit to 3.5MB space reserved for bootblock.
- using negative cache for non-zfs devices

limitations: 
- no writes. at all. never will be;)
- actual pool configuration is limited by disks visibility for grub, and that depends on actual system. if grub can see 4 disks, that will set the limit.
- browsing encryped datasets by tree levels is not supported, full path needs to be used; as writing the encrypred datasets is possible only with solaris 11 and it has its own grub implementation, I just left encryption support as is, at least for now.
- mount cache code is there but not enabled; enabling it did trigger artefacts with grub graphical menu, reasons yet unknown - so far all checks with libumem for memory usage and integrity have been all OK.

tests done: booting with different zpool configurations, single disk, mirror, raidz, multiple vdev, 512B and 4096B sector sizes, missing disks (zpool offline and physically removed), tests are performed on illumos and unknown amount of linux systems - this code is already used by debian grub2 packages.

the embedding support adds zfs GPT partition tag to allow grub to be embedded directly to zfs partition, without the need for BIOS boot partition. If BIOS boot partition exists, it will be used. This functionality is already implemented in Illumos (with legacy grub), so, there is no reason not to support his with grub 2 as well. 

as zfs update is pretty large, i include it compressed…

rgds,
toomas



[-- Attachment #2: 0001-ZFS-update-to-latest-features.patch.gz --]
[-- Type: application/x-gzip, Size: 51885 bytes --]

[-- Attachment #3: 0002-added-support-for-zfs-embedding.patch --]
[-- Type: application/octet-stream, Size: 2141 bytes --]

From 0551a22219c3beae4992796c808f08c6f1c06494 Mon Sep 17 00:00:00 2001
From: Toomas Soome <tsoome@me.com>
Date: Sun, 12 Apr 2015 14:56:02 +0300
Subject: [PATCH 2/2] added support for zfs embedding

---
 grub-core/partmap/gpt.c      |    9 +++++++++
 include/grub/gpt_partition.h |    7 +++++++
 2 files changed, 16 insertions(+)

diff --git a/grub-core/partmap/gpt.c b/grub-core/partmap/gpt.c
index 83bcba7..cacc8e8 100644
--- a/grub-core/partmap/gpt.c
+++ b/grub-core/partmap/gpt.c
@@ -24,6 +24,8 @@
 #include <grub/dl.h>
 #include <grub/msdos_partition.h>
 #include <grub/gpt_partition.h>
+#include <grub/zfs/zio.h>
+#include <grub/zfs/vdev_impl.h>
 #include <grub/i18n.h>
 
 GRUB_MOD_LICENSE ("GPLv3+");
@@ -37,6 +39,7 @@ static const grub_gpt_part_type_t grub_gpt_partition_type_empty = GRUB_GPT_PARTI
 
 #ifdef GRUB_UTIL
 static const grub_gpt_part_type_t grub_gpt_partition_type_bios_boot = GRUB_GPT_PARTITION_TYPE_BIOS_BOOT;
+static const grub_gpt_part_type_t grub_gpt_partition_type_zfs = GRUB_GPT_PARTITION_TYPE_ZFS;
 #endif
 
 /* 512 << 7 = 65536 byte sectors.  */
@@ -162,6 +165,12 @@ find_usable_region (grub_disk_t disk __attribute__ ((unused)),
       return 1;
     }
 
+  if (! grub_memcmp (&gptdata.type, &grub_gpt_partition_type_zfs, 16))
+    {
+      ctx->start = p->start + (VDEV_BOOT_OFFSET >> GRUB_DISK_SECTOR_BITS);
+      ctx->len = (VDEV_BOOT_SIZE >> GRUB_DISK_SECTOR_BITS);
+      return 1;
+    }
   return 0;
 }
 
diff --git a/include/grub/gpt_partition.h b/include/grub/gpt_partition.h
index 1b32f67..04c9f97 100644
--- a/include/grub/gpt_partition.h
+++ b/include/grub/gpt_partition.h
@@ -50,6 +50,13 @@ typedef struct grub_gpt_part_type grub_gpt_part_type_t;
 	{ 0x85, 0xD2, 0xE1, 0xE9, 0x04, 0x34, 0xCF, 0xB3 }	\
   }
 
+#define GRUB_GPT_PARTITION_TYPE_ZFS \
+  { grub_cpu_to_le32_compile_time (0x6A898CC3U),\
+      grub_cpu_to_le16_compile_time (0x1DD2), \
+      grub_cpu_to_le16_compile_time (0x11B2),	       \
+	{ 0x99, 0xA6, 0x08, 0x00, 0x20, 0x73, 0x66, 0x31 }	\
+  }
+
 struct grub_gpt_header
 {
   grub_uint8_t magic[8];
-- 
1.7.9.2


[-- Attachment #4: Type: text/plain, Size: 2 bytes --]

Re: [PATCH] zfs module update

[Andrei Borzenkov]

В Tue, 14 Apr 2015 02:17:43 +0300
Toomas Soome <tsoome@me.com> пишет:

> hi!
> 
> this is the major update to grub2 zfs module; the work is based on Oracle code drops from Solaris 11, and Illumos.

IANAL but I suspect such changes require at least submission from
someone with Oracle E-Mail

+ *  Copyright (c) 2010, 2012, Oracle and/or its affiliates. All rights reserved

> I am including here two separate patch sets, first is zfs itself, second one is update to allow embedding.
> 
> supports:
> - all current OpenZFS features
> - Solaris 11 zpool versions (including reading encrypted datasets)
> - can recognise and boot all pool configurations with exception that log device is not inspected (thats something to be investigated in future, while this code does not mind log device, its still probably not good idea to actually have one).
> - pool is readable as long as there is enough parity to reconstruct data from
> - supports multiple vdevs
> - supports both mirror and raidz
> - supports bootloader embedding as long as it will fit to 3.5MB space reserved for bootblock.
> - using negative cache for non-zfs devices
> 
> limitations: 
> - no writes. at all. never will be;)
> - actual pool configuration is limited by disks visibility for grub, and that depends on actual system. if grub can see 4 disks, that will set the limit.
> - browsing encryped datasets by tree levels is not supported, full path needs to be used; as writing the encrypred datasets is possible only with solaris 11 and it has its own grub implementation, I just left encryption support as is, at least for now.
> - mount cache code is there but not enabled; enabling it did trigger artefacts with grub graphical menu, reasons yet unknown - so far all checks with libumem for memory usage and integrity have been all OK.
> 
> tests done: booting with different zpool configurations, single disk, mirror, raidz, multiple vdev, 512B and 4096B sector sizes, missing disks (zpool offline and physically removed), tests are performed on illumos and unknown amount of linux systems - this code is already used by debian grub2 packages.
> 
> the embedding support adds zfs GPT partition tag to allow grub to be embedded directly to zfs partition, without the need for BIOS boot partition. If BIOS boot partition exists, it will be used. This functionality is already implemented in Illumos (with legacy grub), so, there is no reason not to support his with grub 2 as well. 

GRUB already has embedding support for ZFS; why second patch is needed?

> 
> as zfs update is pretty large, i include it compressed…
> 

Any chance to split it in series of incremental self-contained patches?

Re: [PATCH] zfs module update

[Toomas Soome]

> On 14.04.2015, at 7:34, Andrei Borzenkov <arvidjaar@gmail.com> wrote:
> 
> В Tue, 14 Apr 2015 02:17:43 +0300
> Toomas Soome <tsoome@me.com> пишет:
> 
>> hi!
>> 
>> this is the major update to grub2 zfs module; the work is based on Oracle code drops from Solaris 11, and Illumos.
> 
> IANAL but I suspect such changes require at least submission from
> someone with Oracle E-Mail
> 
> + *  Copyright (c) 2010, 2012, Oracle and/or its affiliates. All rights reserved

the oracle related bits are from:
http://www.oracle.com/technetwork/opensource/systems-solaris-1562786.html

which states clearly: 
"The source code for open source software and firmware components, as licensed under the applicable open source licenses, may be found by following the links below.”

And grub code has open source license and is included in links provided by this page. Therefore I see no problem using this published source just as well as grub is using for example, the libgcrypt from gnupg  or using bits from illumos - which is just as open source as grub. The copyright lines added by Oracle are preserved, just as well as for other entities who have contributed updates to openzfs.


> 
>> I am including here two separate patch sets, first is zfs itself, second one is update to allow embedding.
>> 
>> supports:
>> - all current OpenZFS features
>> - Solaris 11 zpool versions (including reading encrypted datasets)
>> - can recognise and boot all pool configurations with exception that log device is not inspected (thats something to be investigated in future, while this code does not mind log device, its still probably not good idea to actually have one).
>> - pool is readable as long as there is enough parity to reconstruct data from
>> - supports multiple vdevs
>> - supports both mirror and raidz
>> - supports bootloader embedding as long as it will fit to 3.5MB space reserved for bootblock.
>> - using negative cache for non-zfs devices
>> 
>> limitations: 
>> - no writes. at all. never will be;)
>> - actual pool configuration is limited by disks visibility for grub, and that depends on actual system. if grub can see 4 disks, that will set the limit.
>> - browsing encryped datasets by tree levels is not supported, full path needs to be used; as writing the encrypred datasets is possible only with solaris 11 and it has its own grub implementation, I just left encryption support as is, at least for now.
>> - mount cache code is there but not enabled; enabling it did trigger artefacts with grub graphical menu, reasons yet unknown - so far all checks with libumem for memory usage and integrity have been all OK.
>> 
>> tests done: booting with different zpool configurations, single disk, mirror, raidz, multiple vdev, 512B and 4096B sector sizes, missing disks (zpool offline and physically removed), tests are performed on illumos and unknown amount of linux systems - this code is already used by debian grub2 packages.
>> 
>> the embedding support adds zfs GPT partition tag to allow grub to be embedded directly to zfs partition, without the need for BIOS boot partition. If BIOS boot partition exists, it will be used. This functionality is already implemented in Illumos (with legacy grub), so, there is no reason not to support his with grub 2 as well. 
> 
> GRUB already has embedding support for ZFS; why second patch is needed?

to support install using GPT labeled whole disk devices such as /dev/sda in linux. without it, the bios boot partition would be needed - which is in turn, unnecessary in case of zfs.

> 
>> 
>> as zfs update is pretty large, i include it compressed…
>> 
> 
> Any chance to split it in series of incremental self-contained patches?

technically I guess it may be, but it would require so much time that it would make whole process unreasonably long that I see no practical purpose - all the testing required in between steps etc…  the background of this submission is really simple one - I have been working to build grub2 support for illumos based systems for about year now, and one part of this work was to get up to date zfs support in grub itself. since zfs on linux project has reached to the state they are able to use the recent zfs features, i started to receive more and more requests for zfs update, and to reduce burden, I did guess the most reasonable thing to do was to set up this submission;)

rgds,
toomas

Re: [PATCH] zfs module update

[Toomas Soome]

[-- Attachment #1: Type: text/plain, Size: 1595 bytes --]


> On 14.04.2015, at 8:53, Toomas Soome <tsoome@me.com> wrote:
> 
> 
>> On 14.04.2015, at 7:34, Andrei Borzenkov <arvidjaar@gmail.com> wrote:
>> 
>> В Tue, 14 Apr 2015 02:17:43 +0300
>> Toomas Soome <tsoome@me.com> пишет:
>> 
>>> hi!
>>> 
>>> this is the major update to grub2 zfs module; the work is based on Oracle code drops from Solaris 11, and Illumos.
>> 
>> IANAL but I suspect such changes require at least submission from
>> someone with Oracle E-Mail
>> 
>> + *  Copyright (c) 2010, 2012, Oracle and/or its affiliates. All rights reserved
> 
> the oracle related bits are from:
> http://www.oracle.com/technetwork/opensource/systems-solaris-1562786.html
> 
> which states clearly: 
> "The source code for open source software and firmware components, as licensed under the applicable open source licenses, may be found by following the links below.”
> 
> And grub code has open source license and is included in links provided by this page. Therefore I see no problem using this published source just as well as grub is using for example, the libgcrypt from gnupg  or using bits from illumos - which is just as open source as grub. The copyright lines added by Oracle are preserved, just as well as for other entities who have contributed updates to openzfs.
> 

adding up a bit more; the parent page of this link (  Go back to Source Code for Open Source Software Components) is stating the intent and purpose more clearly. If we still need more clarification, it is possible to get it, but so far I have felt quite ok with it:)

rgds,
toomas


[-- Attachment #2.1: Type: text/html, Size: 2868 bytes --]

[-- Attachment #2.2: red-arrow-box-111282.gif --]
[-- Type: image/gif, Size: 88 bytes --]

Re: [PATCH] zfs module update

[Vladimir 'φ-coder/phcoder' Serbinenko]

On 14.04.2015 06:34, Andrei Borzenkov wrote:
> В Tue, 14 Apr 2015 02:17:43 +0300
> Toomas Soome <tsoome@me.com> пишет:
> 
>> hi!
>>
>> this is the major update to grub2 zfs module; the work is based on Oracle code drops from Solaris 11, and Illumos.
> 
> IANAL but I suspect such changes require at least submission from
> someone with Oracle E-Mail
> 
> + *  Copyright (c) 2010, 2012, Oracle and/or its affiliates. All rights reserved
> 
Normally true but for ZFS, it's a little bit special: the ZFS code is
based on the code that sun/Oracle publishes as GPLv2+ or GPLv3+. As long
as the code is officially published on Oracle or Sun website, is under
GPLv2+ or GPLv3+ and touches only ZFS files, it's fine from legal
standpoint to incorporate it.