Support for the new OpenBSD disklabel format

Support for the new OpenBSD disklabel format

[Nikos Skalkotos]

Hello all,

Lately, I've been evaluating all available solutions for manipulationg
*BSD disks from Linux and found out that none recognizes the new OpenBSD
disklabel format correctly. I ended up writing my own tool in python for
manipulating OpenBSD partitions.

parted, that is more script-friendly than fdisk, does not support
disklabels at all. Older fdisk versions (the ones that are shipped with
the major linux distos like Debian and Ubuntu) only check for FreeBSD
and NetBSD partition IDs (but not OpenBSD). Latest versions of fdisk
recognize disklabels on OpenBSD partitions but OpenBSD has changed the
disklabel format.

In order to support disks and partitions that are larger that 2TB, the
OpenBSD folks have converted some 32bit disklabel fields to 48bit (e.g.
the "sectors per unit" field). In order to implement this, they have
replaced some obsolete "hardware characteristics" fields with the high
part (16 most significant bits) of the aforementioned. Even the
partition table has changed. The "filesystem basic fragment size"
(=32bit) has been replaced with the "starting sector high part" (=16bit)
and the "number of sectors high part" (=16bit) fields.

Besides the python tool I've written, for a more permanent solution, I'm
considering patching libfdisk to support the OpenBSD disklabel. I wanted
to know if you are interested for those patches. I think the best way to
do it is to split the disklabel parts that have changed into different
structs and use unions to create a disklabel struct that will work both
for NetBSD and OpenBSD. What do you think?

If this works out, I'd be happy to create python bindings for libfdisk
(like the ones you have for libmount) as a later step. I know that the
API is not currently stable, but most parts seem OK.


Best Regards,
Nikos Skalkotos

Re: Support for the new OpenBSD disklabel format

[Karel Zak]

On Mon, Dec 02, 2013 at 07:21:50PM +0200, Nikos Skalkotos wrote:
> Besides the python tool I've written, for a more permanent solution, I'm
> considering patching libfdisk to support the OpenBSD disklabel. I wanted
> to know if you are interested for those patches. I think the best way to
> do it is to split the disklabel parts that have changed into different
> structs and use unions to create a disklabel struct that will work both
> for NetBSD and OpenBSD. What do you think?

I'm just doing some invasive changes to the library -- it would be
probably better to wait to January to avoid some collisions and extra
work. 

We also have some very simple partitions table parser in libblkid
(see libblkid/src/partitions/, mostly dos.c for nested partition tables). It
would be nice to extend this partition table parser too.

Don't forget to send some test image with the new BSD disklabel! :-)

> If this works out, I'd be happy to create python bindings for libfdisk
> (like the ones you have for libmount) as a later step. I know that the
> API is not currently stable, but most parts seem OK.

It's probably too early, the API is not stable at all, maybe later in 
next year. 

Anyway, python binding is wanted -- we already have binding for
libmount, Ondrej promised binding for libblkid, so binding for
libfdisk would be excellent thing!

Now the library is good enough for dialogs driven interfaces (like
fdisk), but it's necessary to improve it for programs like cfdisk and
sfdisk. Now I'm working on new cfdisk.

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: Support for the new OpenBSD disklabel format

[Nikos Skalkotos]

Hello Karel,

On Tue 10 Dec 2013 12:32:08 PM EET, Karel Zak wrote:
> On Mon, Dec 02, 2013 at 07:21:50PM +0200, Nikos Skalkotos wrote:
>> Besides the python tool I've written, for a more permanent solution, I'm
>> considering patching libfdisk to support the OpenBSD disklabel. I wanted
>> to know if you are interested for those patches. I think the best way to
>> do it is to split the disklabel parts that have changed into different
>> structs and use unions to create a disklabel struct that will work both
>> for NetBSD and OpenBSD. What do you think?
>
> I'm just doing some invasive changes to the library -- it would be
> probably better to wait to January to avoid some collisions and extra
> work.
>
> We also have some very simple partitions table parser in libblkid
> (see libblkid/src/partitions/, mostly dos.c for nested partition tables). It
> would be nice to extend this partition table parser too.
>
> Don't forget to send some test image with the new BSD disklabel! :-)

No problem, I can wait till January. I'll keep an eye on the changes 
made in
libfdisk. I'll also examine the code under libblkid/src/partitions.

>
>> If this works out, I'd be happy to create python bindings for libfdisk
>> (like the ones you have for libmount) as a later step. I know that the
>> API is not currently stable, but most parts seem OK.
>
> It's probably too early, the API is not stable at all, maybe later in
> next year.
>
> Anyway, python binding is wanted -- we already have binding for
> libmount, Ondrej promised binding for libblkid, so binding for
> libfdisk would be excellent thing!
>
> Now the library is good enough for dialogs driven interfaces (like
> fdisk), but it's necessary to improve it for programs like cfdisk and
> sfdisk. Now I'm working on new cfdisk.

Actually I wanted to ask you about this. I've seen the fdisk_ask struct
and how the client can registers an ask_callback function, but it's not
clear to me how you are planning to allow programs to use the
library in an unattended way. Are you going to provide new methods
for this or are you going to document the data each method will ask
for to allow the user to provide the answers in advance?

If you rewrite sfdisk to use the library, I'll get all the answers I 
need. :-)

>
>     Karel
>

Re: Support for the new OpenBSD disklabel format

[Karel Zak]

On Fri, Dec 13, 2013 at 12:48:31PM +0200, Nikos Skalkotos wrote:
> > Now the library is good enough for dialogs driven interfaces (like
> > fdisk), but it's necessary to improve it for programs like cfdisk and
> > sfdisk. Now I'm working on new cfdisk.
> 
> Actually I wanted to ask you about this. I've seen the fdisk_ask struct
> and how the client can registers an ask_callback function, but it's not
> clear to me how you are planning to allow programs to use the
> library in an unattended way. Are you going to provide new methods

See libfdisk branch on github. There is a new 'struct fdisk_partition'
and some functions to modify this struct (well, it's not complete
yet). The idea is to use the struct as template for operations like
add partition. So the primary source for information will be the
template otherwise label drivers will ask by fdisk_ask API.

> If you rewrite sfdisk to use the library, I'll get all the answers I 
> need. :-)

Probably something like:

  struct fdisk_partition *pa = fdisk_new_partition();

  fdisk_partition_enable_default_start(pa);
  fdisk_partition_set_size(pa, 1024 * 1024 * 1024);

  fdisk_add_partition(cxt, pa);

to create 1GiB partition at the begin of the free space.

I can also imagine something like --interactive for sfdisk, so it will
possible to use incomplete input and then it will ask for the missing
things.

I guess it will be usable in python binding, because "*pa" could be an
object, so you can do things like:

 cxt = FdiskContext("/dev/sda")
 pa = FdiskPartition()
 pa[SIZE] = 1024 * 1024 * 1024
 pa[START] = FDISK_DEFAULT

 cxt.add(pa);

or so....

> or are you going to document the data each method will ask
> for to allow the user to provide the answers in advance?

I thought about it, for example add names for the ask API questions to
make it possible for the ask_callback function to detect what exactly is
wanted. But it seems to complex and callbacks driven libraries suck :-)  
(except UI events libs, etc.)

It would be probably better to keep the current ask API restricted to
things like err/warn messages, tables outputs and simple dialogs for
programs like fdisk.


Anyway, I believe that dialog driven fdisk is the best possible way
for advanced users because you can *talk* directly with label specific
code about label specific stuff. It's better than use extra
abstraction layers to describe labels and devices.  Unfortunately, for
programs like sfdisk we need something else (but I still believe that
we can make things in more lightweight way compare to libparted ;-)

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com