Re: Why special linux code in grub2/util/grub-setup.c

Re: Why special linux code in grub2/util/grub-setup.c

[Vladimir 'φ-coder/phcoder' Serbinenko]

On 12.07.2013 14:43, Dr. Tilmann Bubeck wrote:
> Hi Vladimir,
>
> Am 12.07.2013 09:05, schrieb Vladimir 'φ-coder/phcoder' Serbinenko:
>> On 12.07.2013 08:52, Dr. Tilmann Bubeck wrote:
>>> Hi Vladimir,
>>>
>>> I am curently preparing an improvement for GRUB2 to let it embed into
>>> ext4
>>> based upon the new ioctl EXT4_IOC_SWAP_BOOT which I wrote for linux
>>> 3.10.
>>> This would allow to install GRUB SAFELY into a file system and omit the
>>> warning about block list being unreliable (for ext4).
>>>
>> What does this ioctl do?
>
> It was discussed with Ted T'so and modelled after his suggestion and
> included in 3.10. It takes the file associated with the file descriptor
> passed to the ioctl and stores the data blocks of that file in a special
> (reserved) inode called "BOOT_IMAGE_INO". This inode is reserved
> specifically for storing boot loaders inside the filesystem. Technically
> it copies the inodes data block pointers and some more data like
> timestamps from the given file's inode to the BOOT_IMAGE_INO.
>
> This means, that we have a reserved area for storing the boot file and
> that area is never touched again and therefore "sealed". No user tool is
> able to get access to that data (except debug2fs and similar). It is not
> visible in any directory and not destroyed or reported by e2fsck. It is
> similar to the reserved area for booting in btrfs.
>
> It is called "SWAP", because the previous content of the BOOT_IMAGE_INO
> is linked back to the given fd. Therefore one is able to get the
> previous boot code in exchange. GRUB would delete that right away.
>
> Using this ioctl we could make ext4 able to embed reliably which would
> allow installing GRUB into a partition without requiring "--force".
>
>>> To do so I need code to get the blocks of a file. I found that this is
>>> already present in grub2/util/grub-setup.c. I have a few questons based
>>> upon your changes in revision 4033.
>>>
>>> 1. Why did you add additional (linux only) methods to get block list
>>> based
>>> upon FS_IOC_FIEMAP and FIBMAP instead of sticking with the existing
>>> (generic) code based upon  file->read_hook = save_blocklists?
>>>
>> Because Linux buffering and caching is piece of shit and even issuing
>> flush, sync and flushing ioctl doesn't make metadata go to disk.
>  >
>>> 2. I would like to refactor the above code, so that there is a new
>>> method
>>> called "grub_getblocklist(dir, core_file, &blocklist);". This method
>>> move
>>> contain your code together with the generic code to find the blocklist.
>>> This method would then be called from the existing place (and my new
>>> place
>>> in fs/ext2.c if GRUB_UTIL).
>>>
>>> What do you think?
>>>
>> Relying on blocklist retrieving is likely wrong  and doesn't solve any
>> of blocklist problems. But I lack details to know for sure and right now
>> I'm under a lot of stress with my thesis on neural learning.
>
> I would use the blocklist of the above BOOT_IMAGE_INO which will never
> change and therefore will never break. However, I need a way to know
> where the blocks are stored, even if they are "save".
>
> Would you accept a patch from me, using the above IOCTL to make
> embedding to ext4 work reliable? Sure, we have to discuss the details
> but before I start I would like to get a feeling, if this would have a
> chance for inclusion.
>
What's with the allocation issues? The blocks have to be as near to the 
beginning as possible. Is it possible to modify the embedding zone 
without doing the "swapping"? There is a parallel issues in LVM which 
would prompt for the ability of overwrite only part of embedded zone.
> A long discussion about GRUBs inability to install into a partition
> without using "--force" which is 1 reason for Fedora to switch to
> extlinux for this use case can be found in
> https://bugzilla.redhat.com/show_bug.cgi?id=872826.
>
There is a multiboot wrapper for truecrypt and in future it should be 
integrated in GRUB. Just Truecrypt is considered as low priority not in 
the least because Fedora included it in the list of forbidden software
> Kind regards,
>   Till
>

Re: Why special linux code in grub2/util/grub-setup.c

[Dr. Tilmann Bubeck]

Am 25.07.2013 17:29, schrieb Vladimir 'φ-coder/phcoder' Serbinenko:
> On 12.07.2013 14:43, Dr. Tilmann Bubeck wrote:
>> Hi Vladimir,
>>
>> Am 12.07.2013 09:05, schrieb Vladimir 'φ-coder/phcoder' Serbinenko:
>>> On 12.07.2013 08:52, Dr. Tilmann Bubeck wrote:
>>>> Hi Vladimir,
>>>>
>>>> I am curently preparing an improvement for GRUB2 to let it embed into
>>>> ext4
>>>> based upon the new ioctl EXT4_IOC_SWAP_BOOT which I wrote for linux
>>>> 3.10.
>>>> This would allow to install GRUB SAFELY into a file system and omit the
>>>> warning about block list being unreliable (for ext4).
>>>>
>>> What does this ioctl do?
>>
>> It was discussed with Ted T'so and modelled after his suggestion and
>> included in 3.10. It takes the file associated with the file descriptor
>> passed to the ioctl and stores the data blocks of that file in a special
>> (reserved) inode called "BOOT_IMAGE_INO". This inode is reserved
>> specifically for storing boot loaders inside the filesystem. Technically
>> it copies the inodes data block pointers and some more data like
>> timestamps from the given file's inode to the BOOT_IMAGE_INO.
>>
>> This means, that we have a reserved area for storing the boot file and
>> that area is never touched again and therefore "sealed". No user tool is
>> able to get access to that data (except debug2fs and similar). It is not
>> visible in any directory and not destroyed or reported by e2fsck. It is
>> similar to the reserved area for booting in btrfs.
>>
>> It is called "SWAP", because the previous content of the BOOT_IMAGE_INO
>> is linked back to the given fd. Therefore one is able to get the
>> previous boot code in exchange. GRUB would delete that right away.
>>
>> Using this ioctl we could make ext4 able to embed reliably which would
>> allow installing GRUB into a partition without requiring "--force".
>>
>>>> To do so I need code to get the blocks of a file. I found that this is
>>>> already present in grub2/util/grub-setup.c. I have a few questons based
>>>> upon your changes in revision 4033.
>>>>
>>>> 1. Why did you add additional (linux only) methods to get block list
>>>> based
>>>> upon FS_IOC_FIEMAP and FIBMAP instead of sticking with the existing
>>>> (generic) code based upon  file->read_hook = save_blocklists?
>>>>
>>> Because Linux buffering and caching is piece of shit and even issuing
>>> flush, sync and flushing ioctl doesn't make metadata go to disk.
>>  >
>>>> 2. I would like to refactor the above code, so that there is a new
>>>> method
>>>> called "grub_getblocklist(dir, core_file, &blocklist);". This method
>>>> move
>>>> contain your code together with the generic code to find the blocklist.
>>>> This method would then be called from the existing place (and my new
>>>> place
>>>> in fs/ext2.c if GRUB_UTIL).
>>>>
>>>> What do you think?
>>>>
>>> Relying on blocklist retrieving is likely wrong  and doesn't solve any
>>> of blocklist problems. But I lack details to know for sure and right now
>>> I'm under a lot of stress with my thesis on neural learning.
>>
>> I would use the blocklist of the above BOOT_IMAGE_INO which will never
>> change and therefore will never break. However, I need a way to know
>> where the blocks are stored, even if they are "save".
>>
>> Would you accept a patch from me, using the above IOCTL to make
>> embedding to ext4 work reliable? Sure, we have to discuss the details
>> but before I start I would like to get a feeling, if this would have a
>> chance for inclusion.
>>

> What's with the allocation issues? The blocks have to be as near to the
> beginning as possible. Is it possible to modify the embedding zone
> without doing the "swapping"? There is a parallel issues in LVM which
> would prompt for the ability of overwrite only part of embedded zone.

The embedding zone is not accessible and could not be changed. This 
ensure, that block lists keep stable.

If you want to change the content, then you have to install a new file 
using IOCTL (and you will get back the previous embedded file in exchange).

-- 
Mit freundlichen Gruessen,
   Tilmann Bubeck


//// dr. tilmann bubeck, it professional & geek
////
//// till@bubecks.de       /  http://www.bubecks.de
//// mobile: 0172-8842972  /  fon: 0711-7227719     /  fax: 0711-7227734
//// widmaierstr. 58       /  70567 stuttgart       /  germany

Re: Why special linux code in grub2/util/grub-setup.c

[Vladimir 'φ-coder/phcoder' Serbinenko]

On 25.07.2013 21:40, Dr. Tilmann Bubeck wrote:
>
>
> Am 25.07.2013 17:29, schrieb Vladimir 'φ-coder/phcoder' Serbinenko:
>> On 12.07.2013 14:43, Dr. Tilmann Bubeck wrote:
>>> Hi Vladimir,
>>>
>>> Am 12.07.2013 09:05, schrieb Vladimir 'φ-coder/phcoder' Serbinenko:
>>>> On 12.07.2013 08:52, Dr. Tilmann Bubeck wrote:
>>>>> Hi Vladimir,
>>>>>
>>>>> I am curently preparing an improvement for GRUB2 to let it embed into
>>>>> ext4
>>>>> based upon the new ioctl EXT4_IOC_SWAP_BOOT which I wrote for linux
>>>>> 3.10.
>>>>> This would allow to install GRUB SAFELY into a file system and omit
>>>>> the
>>>>> warning about block list being unreliable (for ext4).
>>>>>
>>>> What does this ioctl do?
>>>
>>> It was discussed with Ted T'so and modelled after his suggestion and
>>> included in 3.10. It takes the file associated with the file descriptor
>>> passed to the ioctl and stores the data blocks of that file in a special
>>> (reserved) inode called "BOOT_IMAGE_INO". This inode is reserved
>>> specifically for storing boot loaders inside the filesystem. Technically
>>> it copies the inodes data block pointers and some more data like
>>> timestamps from the given file's inode to the BOOT_IMAGE_INO.
>>>
>>> This means, that we have a reserved area for storing the boot file and
>>> that area is never touched again and therefore "sealed". No user tool is
>>> able to get access to that data (except debug2fs and similar). It is not
>>> visible in any directory and not destroyed or reported by e2fsck. It is
>>> similar to the reserved area for booting in btrfs.
>>>
>>> It is called "SWAP", because the previous content of the BOOT_IMAGE_INO
>>> is linked back to the given fd. Therefore one is able to get the
>>> previous boot code in exchange. GRUB would delete that right away.
>>>
>>> Using this ioctl we could make ext4 able to embed reliably which would
>>> allow installing GRUB into a partition without requiring "--force".
>>>
>>>>> To do so I need code to get the blocks of a file. I found that this is
>>>>> already present in grub2/util/grub-setup.c. I have a few questons
>>>>> based
>>>>> upon your changes in revision 4033.
>>>>>
>>>>> 1. Why did you add additional (linux only) methods to get block list
>>>>> based
>>>>> upon FS_IOC_FIEMAP and FIBMAP instead of sticking with the existing
>>>>> (generic) code based upon  file->read_hook = save_blocklists?
>>>>>
>>>> Because Linux buffering and caching is piece of shit and even issuing
>>>> flush, sync and flushing ioctl doesn't make metadata go to disk.
>>>  >
>>>>> 2. I would like to refactor the above code, so that there is a new
>>>>> method
>>>>> called "grub_getblocklist(dir, core_file, &blocklist);". This method
>>>>> move
>>>>> contain your code together with the generic code to find the
>>>>> blocklist.
>>>>> This method would then be called from the existing place (and my new
>>>>> place
>>>>> in fs/ext2.c if GRUB_UTIL).
>>>>>
>>>>> What do you think?
>>>>>
>>>> Relying on blocklist retrieving is likely wrong  and doesn't solve any
>>>> of blocklist problems. But I lack details to know for sure and right
>>>> now
>>>> I'm under a lot of stress with my thesis on neural learning.
>>>
>>> I would use the blocklist of the above BOOT_IMAGE_INO which will never
>>> change and therefore will never break. However, I need a way to know
>>> where the blocks are stored, even if they are "save".
>>>
>>> Would you accept a patch from me, using the above IOCTL to make
>>> embedding to ext4 work reliable? Sure, we have to discuss the details
>>> but before I start I would like to get a feeling, if this would have a
>>> chance for inclusion.
>>>
>
>> What's with the allocation issues? The blocks have to be as near to the
>> beginning as possible. Is it possible to modify the embedding zone
>> without doing the "swapping"? There is a parallel issues in LVM which
>> would prompt for the ability of overwrite only part of embedded zone.
>
> The embedding zone is not accessible and could not be changed. This
> ensure, that block lists keep stable.
>
> If you want to change the content, then you have to install a new file
> using IOCTL (and you will get back the previous embedded file in exchange).
>
That's a problem since the old zone will stay here and it's one of big 
vectors of problems with blocklists. This approach actually *ensures* 
that blocklist will *change* every time so it makes some simultaneous 
installations impossible and will create an illusion that any lingering 
bootsector is still active. Better way would be:
1) have a way to reserve some space, near the beginning. E.g. IOCTL 
CREATE_EMBEDDING_ZONE with argument size=5M (number coming from my draft 
on LVM zones). This has to have synchronous semantics. Blocklist is 
fixed after this operation.
2) A way to retrieve its blocklist
3) A way to overwrite parts of its contents in a way that ensures 
bypassing journal, COW, compression, encryption, hyperspace or any other 
advanced features.

Re: Why special linux code in grub2/util/grub-setup.c

[Chris Murphy]

Summary: a way for grub-install to support installation to a partition when formatted ext[234], without --force and blocklists.


On Jul 25, 2013, at 3:17 PM, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> wrote:
>> 
>> The embedding zone is not accessible and could not be changed. This
>> ensure, that block lists keep stable.
>> 
>> If you want to change the content, then you have to install a new file
>> using IOCTL (and you will get back the previous embedded file in exchange).
>> 
> That's a problem since the old zone will stay here and it's one of big vectors of problems with blocklists. This approach actually *ensures* that blocklist will *change* every time so it makes some simultaneous installations impossible and will create an illusion that any lingering bootsector is still active. Better way would be:
> 1) have a way to reserve some space, near the beginning. E.g. IOCTL CREATE_EMBEDDING_ZONE with argument size=5M (number coming from my draft on LVM zones). This has to have synchronous semantics. Blocklist is fixed after this operation.
> 2) A way to retrieve its blocklist
> 3) A way to overwrite parts of its contents in a way that ensures bypassing journal, COW, compression, encryption, hyperspace or any other advanced features.

5MB? The current BIOS Boot is 1MB. And the Btrfs pad is 64KB. Are these still sufficient?

I'm uncertain of the scope, but the following bug indicates potentially widespread use of LBA 36 as a start LBA by Dell, which leaves a totally insufficiently sized MBR gap to install GRUB:

https://bugzilla.redhat.com/show_bug.cgi?id=986431

It seems there should be a way to accommodate this automagically by any installer, presently they'll fail when installing to the disk device, i.e. /dev/sda, where --force isn't applicable. It seems there'd need to be a test for a sufficiently large MBR gap, and if not fallback to a way to embed into a boot partition.


Chris Murphy