udf_count_free() and UDF discs with Metadata partition

udf_count_free() and UDF discs with Metadata partition

[Pali Rohár]

[-- Attachment #1: Type: text/plain, Size: 1263 bytes --]

Hello!

During testing of udfinfo tool (from udftools project) I found that
udfinfo's implementation for calculating free space does not work when
UDF filesystem has Metadata partition (according to OSTA UDF 2.50).

Year ago in udfinfo for calculating free space I used same algorithm as
is implemented in kernel UDF driver, function udf_count_free(). So I
suspect kernel driver could have it incorrectly implemented too, but I'm
not sure. So I'm sending this email to let you know about it.

What is the problem? UDF Metadata partition is stored directly on UDF
Physical partition and therefore free space calculation needs to be done
from Physical one (same applies for Virtual partition). But Metadata
partition contains mapping table for logical <--> physical blocks, so
reading data needs to be done always from Metadata partition. Also in
UDF terminology are two different things: Partition and Partition Map.
And "partition number" is a bit misleading as sometimes it refers to
"Partition" and sometimes to "Partition Map" what are two different
things.

Calculation problem in udfinfo I fixed in this commit:
https://github.com/pali/udftools/commit/1763c9f899bdbdb68b1a44a8cb5edd5141107043

-- 
Pali Rohár
pali.rohar@gmail.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: udf_count_free() and UDF discs with Metadata partition

[Jan Kara]

Hello!

On Thu 26-12-19 12:37:50, Pali Rohár wrote:
> During testing of udfinfo tool (from udftools project) I found that
> udfinfo's implementation for calculating free space does not work when
> UDF filesystem has Metadata partition (according to OSTA UDF 2.50).
> 
> Year ago in udfinfo for calculating free space I used same algorithm as
> is implemented in kernel UDF driver, function udf_count_free(). So I
> suspect kernel driver could have it incorrectly implemented too, but I'm
> not sure. So I'm sending this email to let you know about it.
> 
> What is the problem? UDF Metadata partition is stored directly on UDF
> Physical partition and therefore free space calculation needs to be done
> from Physical one (same applies for Virtual partition). But Metadata
> partition contains mapping table for logical <--> physical blocks, so
> reading data needs to be done always from Metadata partition. Also in
> UDF terminology are two different things: Partition and Partition Map.
> And "partition number" is a bit misleading as sometimes it refers to
> "Partition" and sometimes to "Partition Map" what are two different
> things.

Thanks for the note! You're right that we probably misreport amount of free
space in the UDF filesystems with Metadata partition. Luckily the kernel
driver supports filesystems with Metadata partition or Virtual partition
only in read-only mode so the bug does not cause any real harm.

> Calculation problem in udfinfo I fixed in this commit:
> https://github.com/pali/udftools/commit/1763c9f899bdbdb68b1a44a8cb5edd5141107043

Thanks for the link, I'll fixup the kernel code. BTW, how did you test
this? Do you have any UDF image with Metadata partition?

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: udf_count_free() and UDF discs with Metadata partition

[Pali Rohár]

On Tuesday 07 January 2020 15:45:18 Jan Kara wrote:
> Hello!
> 
> On Thu 26-12-19 12:37:50, Pali Rohár wrote:
> > During testing of udfinfo tool (from udftools project) I found that
> > udfinfo's implementation for calculating free space does not work when
> > UDF filesystem has Metadata partition (according to OSTA UDF 2.50).
> > 
> > Year ago in udfinfo for calculating free space I used same algorithm as
> > is implemented in kernel UDF driver, function udf_count_free(). So I
> > suspect kernel driver could have it incorrectly implemented too, but I'm
> > not sure. So I'm sending this email to let you know about it.
> > 
> > What is the problem? UDF Metadata partition is stored directly on UDF
> > Physical partition and therefore free space calculation needs to be done
> > from Physical one (same applies for Virtual partition). But Metadata
> > partition contains mapping table for logical <--> physical blocks, so
> > reading data needs to be done always from Metadata partition. Also in
> > UDF terminology are two different things: Partition and Partition Map.
> > And "partition number" is a bit misleading as sometimes it refers to
> > "Partition" and sometimes to "Partition Map" what are two different
> > things.
> 
> Thanks for the note! You're right that we probably misreport amount of free
> space in the UDF filesystems with Metadata partition. Luckily the kernel
> driver supports filesystems with Metadata partition or Virtual partition
> only in read-only mode so the bug does not cause any real harm.

For discs with Virtual partition I used in udfinfo different algorithm
for calculating free space: Free are only those blocks which are after
VAT block (therefore unrecorded blocks).

> > Calculation problem in udfinfo I fixed in this commit:
> > https://github.com/pali/udftools/commit/1763c9f899bdbdb68b1a44a8cb5edd5141107043
> 
> Thanks for the link, I'll fixup the kernel code. BTW, how did you test
> this? Do you have any UDF image with Metadata partition?

I have CD, DVD, HD-DVD and BD images with just one file (so they have
lot of empty space) in all variants (plain, Sparing, VAT) and all
possible UDF revisions (1.02 - 2.60) created by some very very ancient
Windows Nero software. Some of them discovered bugs in libblkid UDF
implementation and are therefore included as part of util-linux project
for running util-linux tests. Plain and Sparing UDF 2.50 and 2.60 images
have Metadata partition as required by specification.

If you want I can send you a whole pack of all those images.

-- 
Pali Rohár
pali.rohar@gmail.com

Re: udf_count_free() and UDF discs with Metadata partition

[Jan Kara]

On Tue 07-01-20 16:01:42, Pali Rohár wrote:
> On Tuesday 07 January 2020 15:45:18 Jan Kara wrote:
> > > Calculation problem in udfinfo I fixed in this commit:
> > > https://github.com/pali/udftools/commit/1763c9f899bdbdb68b1a44a8cb5edd5141107043
> > 
> > Thanks for the link, I'll fixup the kernel code. BTW, how did you test
> > this? Do you have any UDF image with Metadata partition?
> 
> I have CD, DVD, HD-DVD and BD images with just one file (so they have
> lot of empty space) in all variants (plain, Sparing, VAT) and all
> possible UDF revisions (1.02 - 2.60) created by some very very ancient
> Windows Nero software. Some of them discovered bugs in libblkid UDF
> implementation and are therefore included as part of util-linux project
> for running util-linux tests. Plain and Sparing UDF 2.50 and 2.60 images
> have Metadata partition as required by specification.
> 
> If you want I can send you a whole pack of all those images.

That would be nice. Thanks!

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR