Unable to read UDF fs on a DVD

Unable to read UDF fs on a DVD

[Kronos]

Hi,
I'm trying to read a DVD+RW disk with a UDF fs on it. The disk was
created under Windows, using Easy CD Creator (don't know the details).

I can mount (kernel 2.6.5) w/o problems:
root@dreamland:~# mount -t udf -oro /dev/hdc /cdrom

dmesg:
udf: registering filesystem
UDF-fs DEBUG fs/udf/lowlevel.c:57:udf_get_last_session: XA disk: no, vol_desc_start=0
UDF-fs DEBUG fs/udf/super.c:1546:udf_fill_super: Multi-session=0
UDF-fs DEBUG fs/udf/super.c:534:udf_vrs: Starting at sector 16 (2048 byte sectors)
UDF-fs DEBUG fs/udf/super.c:561:udf_vrs: ISO9660 Primary Volume Descriptor found
UDF-fs DEBUG fs/udf/super.c:570:udf_vrs: ISO9660 Volume Descriptor Set Terminator found
UDF-fs DEBUG fs/udf/super.c:877:udf_load_pvoldesc: recording time 1082448386/0, 2004/04/20 09:06 (103c)
UDF-fs DEBUG fs/udf/super.c:888:udf_load_pvoldesc: volIdent[] = 'CDROM'
UDF-fs DEBUG fs/udf/super.c:895:udf_load_pvoldesc: volSetIdent[] = '040420_0906'
UDF-fs DEBUG fs/udf/super.c:1087:udf_load_logicalvol: Partition (0:0) type 1 on volume 1
UDF-fs DEBUG fs/udf/super.c:1097:udf_load_logicalvol: FileSet found in LogicalVolDesc at block=0, partition=0
UDF-fs DEBUG fs/udf/super.c:925:udf_load_partdesc: Searching map: (0 == 0)
UDF-fs DEBUG fs/udf/super.c:1007:udf_load_partdesc: Partition (0:0 type 1511) starts at physical 257, block length 2172880
UDF-fs DEBUG fs/udf/super.c:1340:udf_load_partition: Using anchor in block 256
UDF-fs DEBUG fs/udf/super.c:1573:udf_fill_super: Lastblock=0
UDF-fs DEBUG fs/udf/super.c:849:udf_find_fileset: Fileset at block=0, partition=0
UDF-fs DEBUG fs/udf/super.c:911:udf_load_fileset: Rootdir at block=2, partition=0
UDF-fs INFO UDF 0.9.8.1 (2004/29/09) Mounting volume 'CDROM', timestamp 2004/04/20 10:06 (1078)

But I'm unable to stat/read/whatever the files:

root@dreamland:~# ls /cdrom
/bin/ls: /cdrom/Bakuretsu Tenshi - 01.avi: No such file or directory
/bin/ls: /cdrom/Bakuretsu Tenshi - 02.avi: No such file or directory
[etc]

I can mount the disk and read it using ISO9660 instead of UDF (filenames
are 8+3, no Joliet it seems), and I can read it under WinXP. It
shouldn't be damaged.

Any clue?

Luca
-- 
Home: http://kronoz.cjb.net
"Su cio` di cui non si puo` parlare e` bene tacere".
 Ludwig Wittgenstein

Re: Unable to read UDF fs on a DVD

[Pat LaVarre]

> DVD+RW ... UDF ... can mount (kernel 2.6.5) w/o problems:
> ...# mount -t udf -oro /dev/hdc /cdrom
> dmesg:
> udf: registering filesystem
> ...
> UDF-fs INFO UDF 0.9.8.1 (2004/29/09) Mounting volume 'CDROM', timestamp 2004/04/20 10:06 (1078)
> ...

So far so good.

> But ... unable to stat/read/whatever the files:
> 
> ....# ls /cdrom
> /bin/ls: /cdrom/Bakuretsu Tenshi - 01.avi: No such file or directory
> /bin/ls: /cdrom/Bakuretsu Tenshi - 02.avi: No such file or directory
> [etc]
> 
> I can mount the disk and read it using ISO9660 instead of UDF (filenames
> are 8+3, no Joliet it seems), and I can read it under WinXP. It
> shouldn't be damaged.

Q1) Any Linux dmesg as you try to read or umount?

Q2) What text does the DIR /S command of Windows produce?

> created under Windows, using Easy CD Creator
> (don't know the details).

Q3) What does Linux fsck tell you, before you mount -o ro (or after you
umount)?

Pat LaVarre

P.S. The subscriber-only archives of linux_udf@h... currently show
Linux-2.6.5 issues now under discussion, including an issue people have
reproduced by downloading a huge trial .exe into Windows and then
copying a file of more than 2 GiB to the disc.

-----Forwarded Message-----
From: Pat LaVarre
Cc: linux_udf@h...
Subject: Re: Bug with UDF file system
Date: 20 Apr 2004 10:34:47 -0600
...

My own most recent blog re the mystery of install & run of the phg fsck
in Linux from virus-free source is:

phg fsck of UDF for Linux
http://udfko.blog-city.com/read/577369.htm
...

Re: Unable to read UDF fs on a DVD

[Kronos]

Il Fri, Apr 23, 2004 at 11:56:42AM -0600, Pat LaVarre ha scritto: 
> > ....# ls /cdrom
> > /bin/ls: /cdrom/Bakuretsu Tenshi - 01.avi: No such file or directory
> > /bin/ls: /cdrom/Bakuretsu Tenshi - 02.avi: No such file or directory
> > [etc]
> > 
> > I can mount the disk and read it using ISO9660 instead of UDF (filenames
> > are 8+3, no Joliet it seems), and I can read it under WinXP. It
> > shouldn't be damaged.
> 
> Q1) Any Linux dmesg as you try to read or umount?

No, nothing at all.


> Q2) What text does the DIR /S command of Windows produce?

 Il volume nell'unita` D e` CDROM
 Numero di serie del volume: A381-DC88

 Directory di D:\

10/04/2004  20.59       240.695.296 Bakuretsu Tenshi - 01.avi
16/04/2004  15.13       240.715.776 Bakuretsu Tenshi - 02.avi
13/04/2004  18.57       182.452.224 Full Metal Alchemist - 11.avi
13/04/2004  18.57       182.452.224 Full Metal Alchemist - 12.avi
13/04/2004  18.57       182.452.224 Full Metal Alchemist - 13.avi
13/04/2004  18.57       178.257.920 Full Metal Alchemist - 14.avi
13/04/2004  18.57       178.257.920 Full Metal Alchemist - 15.avi
14/04/2004  15.55       178.257.920 Full Metal Alchemist - 16.avi
14/04/2004  15.55       178.257.920 Full Metal Alchemist - 17.avi
14/04/2004  15.55       178.257.920 Full Metal Alchemist - 18.avi
14/04/2004  15.35       178.257.920 Full Metal Alchemist - 19.avi
14/04/2004  15.35       178.257.920 Full Metal Alchemist - 20.avi
17/04/2004  15.15       178.257.920 Full Metal Alchemist - 21.avi
17/04/2004  17.32       178.257.920 Full Metal Alchemist - 22.avi
17/04/2004  20.41       178.257.920 Full Metal Alchemist - 23.avi
18/04/2004  01.26       178.257.920 Full Metal Alchemist - 24.avi
18/04/2004  11.07       178.257.920 Full Metal Alchemist - 25.avi
18/04/2004  16.48       178.257.920 Full Metal Alchemist - 26.avi
13/04/2004  10.26       178.257.920 Full Metal Alchemist - 27.avi
19/04/2004  11.30       184.088.576 Full Metal Alchemist - 28.avi
14/02/2004  12.41       224.397.312 Ghost in the Shell - Stand Alone Complex - 00.avi
14/02/2004  12.13       147.292.160 Ghost in the Shell - Stand Alone Complex - 01.avi
14/02/2004  21.28       187.267.072 Ghost in the Shell - Stand Alone Complex - 02.avi
13/04/2004  04.12       182.564.864 Planetes - phase 13.avi
              24 File  4.449.988.608 byte

     Totale file elencati:
              24 File  4.449.988.608 byte
               0 Directory               0 byte disponibili

> > created under Windows, using Easy CD Creator
> > (don't know the details).
> 
> Q3) What does Linux fsck tell you, before you mount -o ro (or after you
> umount)?

I used udfct utility (from Philips). The output is quite long, I put it
here:

http://web.tiscali.it/kronoz/ucf_test.log

I don't see anything strange.

Btw, don't know if it's related but I was unable to run ucf_test without
scsi emulation: it complained about unknown image chunk size. I can't
read files even with ide-scsi, though.

> P.S. The subscriber-only archives of linux_udf@h... currently show
> Linux-2.6.5 issues now under discussion, including an issue people have
> reproduced by downloading a huge trial .exe into Windows and then
> copying a file of more than 2 GiB to the disc.

I think that this is a different issue, files on my disk are smaller.

thanks,
Luca
-- 
Home: http://kronoz.cjb.net
Porc i' mond che cio' sott i piedi!
V. Catozzo

Re: Unable to read UDF fs on a DVD

[Pat LaVarre]

> http://web.tiscali.it/kronoz/ucf_test.log
> I don't see anything strange.

I now agree, that disc passed fsck and mount well enough to make the ls
failure interesting.

What to try next after fsck passes, I do not know, ouch, sorry.

Offline I'm working to comment the fs/udf/ source.  Even me finishing
that might not do you much good, unless we can figure out how to
reproduce your trouble at my desk.

Pat LaVarre

P.S. Five postscripts:

1)

> even with ide-scsi, though.

Whoa.  You weren't engaging in the taboo act of running ide-scsi in 2.6
back when ls failed, were you?  (If you are, then please remove
ide-scsi, substitute ide-cd, and confirm or deny that exercise actually
made no difference.)

2)

The disc didn't actually pass the phgfsck without complaint:  The
standard phgfsck egrep is:

$ egrep -i '(info|warning|error):' http://web.tiscali.it/kronoz/ucf_test.log
        PVD  72  Warning: Volume Set Identifier: "040420_0906",
        PVD  72  Warning: Volume Set Identifier: "040420_0906",
        Error: Number of AVDPs less than 2: 1, AVDP at 256
$

Non-compliance!

All the same, I'm guessing these complaints do Not explain the ls
failure, since to my newbie ear these sound like mount issues and we
know you can mount.

3)

I can't now rapidly reproduce the collection of file lengths you report,
because sparse files in UDF, even when the underlying volume is not
sparse, as yet crash my Linux-2.6.5.

3)

> Btw, don't know if it's related but I was unable to run ucf_test without
> scsi emulation: it complained about unknown image chunk size. I can't
> read files even with ide-scsi, though.

Yes phgfsck trouble like that is normal, thanks for asking.

4)

> > P.S. The subscriber-only archives of linux_udf@h... currently show
> > Linux-2.6.5 issues now under discussion, including an issue people have
> > reproduced by downloading a huge trial .exe into Windows and then
> > copying a file of more than 2 GiB to the disc.
> 
> I think that this is a different issue, files on my disk are smaller.

I agree your conclusion is reasonable, I do not myself yet know the
udf.ko code well enough to firmly confirm or deny your conclusion.

5)

> http://web.tiscali.it/kronoz/ucf_test.log

Any chance this link will still work, a year from now?  (I ask because
I'm hoping to see a collection of observed UDF non-compliance come into
being.)

Re: Unable to read UDF fs on a DVD

[Kronos]

Il Fri, Apr 23, 2004 at 02:21:15PM -0600, Pat LaVarre ha scritto: 
> P.S. Five postscripts:
> 
> 1)
> 
> > even with ide-scsi, though.
> 
> Whoa.  You weren't engaging in the taboo act of running ide-scsi in 2.6
> back when ls failed, were you?

No, I wasn't. I'm aware that ide-scsi is not needed with 2.6. I had to
recompile the kernel with ide-scsi to make Philips fsck happy.

> 3)
> 
> I can't now rapidly reproduce the collection of file lengths you report,
> because sparse files in UDF, even when the underlying volume is not
> sparse, as yet crash my Linux-2.6.5.

Well, Ben Fennema said that he has identified the issue. If it can be
usefull I can have my friend burn another DVD (with less data) with Easy
CD and upload the image somewhere.

> 5)
> 
> > http://web.tiscali.it/kronoz/ucf_test.log
> 
> Any chance this link will still work, a year from now?

Yup, no problem.

Luca
-- 
Home: http://kronoz.cjb.net
Alcuni pensano che io sia una persona orribile, ma non e` vero. Ho il
cuore di un ragazzino - in un vaso sulla scrivania.

Re: Unable to read UDF fs on a DVD

[Bill Davidsen]

Kronos wrote:
> Il Fri, Apr 23, 2004 at 02:21:15PM -0600, Pat LaVarre ha scritto: 
> 
>>P.S. Five postscripts:
>>
>>1)
>>
>>
>>>even with ide-scsi, though.
>>
>>Whoa.  You weren't engaging in the taboo act of running ide-scsi in 2.6
>>back when ls failed, were you?
> 
> 
> No, I wasn't. I'm aware that ide-scsi is not needed with 2.6. I had to
> recompile the kernel with ide-scsi to make Philips fsck happy.

I believe that it would be more correct to say that ide-scsi is not 
required to burn CDs and DVDs, providing you use cdrecord which has been 
modified to work with ide-cd. Since that's the major use it equates to 
"not needed" if that's all you do.

As you seem to note, some DVD burners don't work without ide-scsi unless 
you have some tricks and/or patches, but do work with ide-scsi. Yes, 
even in 2.6.

I totally agree you should try to run without it, but I'd compile it as 
a module in case you need some additional data points.


-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

Re: Unable to read UDF fs on a DVD

[Pat LaVarre]

Kronos:

> Well, Ben Fennema said that he has identified the issue. If it can be
> usefull I can have my friend burn another DVD (with less data) with 
> Easy
> CD and upload the image somewhere.

1) Useful to Ben, maybe not, Ben already understands much.

2) Useful to you, maybe not, maybe nobody on Earth but Ben can help you 
rapidly.

3) Useful to me, yes.  You put the image on the web, and I'll try to 
download it, try to reproduce your trouble, and then try to find time 
to learn ever more about how unamerican chars work inside udf.ko.

>> 5)
>>
>>> http://web.tiscali.it/kronoz/ucf_test.log
>>
>> Any chance this link will still work, a year from now?
>
> Yup, no problem.

Thank you, now blogged as the first of:

udf fsck failures observed
http://udfko.blog-city.com/read/585805.htm

Pat LaVarre

Re: Unable to read UDF fs on a DVD

[Pat LaVarre]

> So concluding, I see no reason why this disc is not correctly
> read by Linux UDF.

Thanks for explaining.

I remember separately I saw the famous Ben F guess that the unamerican
chars were at issue: maybe the truthful or slanderous rumours of UDF in
2004 tripping over unamerican chars as often as other software does have
substance.

I imagine to experiment we may need to bring both Windows and Linux
together: either alone might convert between bytes and chars in a
lossless consistent way that only happened not to be standard.

> > $ egrep -i '(info|warning|error):'
> http://web.tiscali.it/kronoz/ucf_test.log
> >         PVD  72  Warning: Volume Set Identifier: "040420_0906",
> >         PVD  72  Warning: Volume Set Identifier: "040420_0906",
> >         Error: Number of AVDPs less than 2: 1, AVDP at 256
> > $
> > 
> > Non-compliance! ... I'm guessing ... Not explain the ls failure, 
> ...
> This error is no problem at all, nor are the warnings.
> The 2nd AVDP is required by UDF but redundant. It is only required to
> cope with the situation that the 1st AVDP at 256 is not readable.
> 
> No UDF implementation should refuse to mount because of this.

Lost me, help?

"No problem at all"?!?

We're discussing NON COMPLIANCE ... are we not?

That is, we're discussing a plain failure to meet the requirement,
tolerated because, by design, less than all actual UDF implementations
actually require all the published requirements.

Indeed, we know some programmers prefer to write code to comply only
with the de jure theory of the merely public specifications ... rather
than writing code that consciously quietly tolerates and thereby
encourages continuing violations of those specifications.

How then can we flatly say no UDF implementation should refuse to mount
because of a missing redundant AVDP?

Why can't someone reasonably choose to make their UDF more trustworthy
and simultaneously less friendly by requiring the redundant AVDP to be
present and in agreement?

Why must everyone trust the failure model that says the missing AVDP
doesn't matter?

Indeed, someone could reasonably require the Philips UDF Verifier to
bless a disc before automounting that disc.  No?

> ...

All that said, then yes for this specific case of Linux UDF, I agree
with you, Linux by design interoperates well with Windows, both as
delivered from Microsoft and as commonly patched, so here the de facto
standard of:
 
UDF IUVD LVInfo1: "!roxioUDF!n>2173136!s>0!d>286!"

wins over the de jure theory of a requirement for two redundant AVDP.

> 	PVD  72  Warning: Volume Set Identifier: "040420_0906",
> -		 expected: CS0 representation of unique hex number in
> -			   first 8 character positions, UDF 2.2.2.5.

Again the de jure theory of conformant VSID loses to the de facto
interoperability of tolerating such abuse?

> > >  Il volume nell'unita` D e` CDROM
> > >  Numero di serie del volume: A381-DC88
> > >  Directory di D:\
> > > 10/04/2004  20.59       240.695.296 Bakuretsu Tenshi - 01.avi
> > > 16/04/2004  15.13       240.715.776 Bakuretsu Tenshi - 02.avi
> > > 13/04/2004  18.57       182.452.224 Full Metal Alchemist - 11.avi
> > > ...
> 
> The Windows dir listing as reported by Luca is exactly what is
> reported by
> the UDF verifier.

"Thanks for explaining."

> The timestamps look different, but that is because the UDF
> verifier reports in UCT by default. With the '-localtime' option, also
> the times would be identical.

"Thanks for explaining."

> cc: linux_udf@h..., linux_udf-owner@h...

I hope I did right by deleting the linux_udf-owner from the cc.

Pat LaVarre

Re: Unable to read UDF fs on a DVD

[Pat LaVarre]

Three points:

1)

> > I remember separately I saw the ... guess that the
> > unamerican chars were at issue: maybe the truthful or slanderous
> > rumours of UDF in 2004 tripping over unamerican chars as often as
> > other software does have substance.
> 
> I do not know what 'unamerican chars' problem there could be in:
>  
> > http://web.tiscali.it/kronoz/ucf_test.log

Same as you, I do not yet understand, but:

The rumour of unamerican char troubles in UDF implementations reached
this thread without coming from me.  From others we have, in the
subscriber-only linux_udf@h... archives:

-----
Cc: linux_udf@h...
Subject: Re: Unable to read UDF fs on a DVD
Date: 23 Apr 2004 14:15:10 -0600

On Fri, Apr 23, 2004 at 09:50:04PM +0200, ... wrote:
> 
> I used udfct utility (from Philips). The output is quite long, I put it
> here:
> 
> http://web.tiscali.it/kronoz/ucf_test.log

Ok, that confirmed my guess as to what the problem is (it's a problem
with 16 bit characters vs 8 bit characters)

I'll work up a patch this weekend.
...
-----

2)

My local records tell me no patch has reached `cvs co udf` or `cvs co
udftools` at sourceforge.net/linux-udf/ since 2004-03-22.

3)

I say "unamerican chars" to mean the printable chars that do not appear
on a US keyboard, even when lowercased.  Do you prefer some other term?

I can't be sure how literally the authority quoted above meant 16 vs. 8
bit.  For example, I fear together all of:

$'\x23' # "octothorpe"
$'\xA3' £ "pound sterling, ... Italian lira, ... etc."
$'\xA5' ¥ "YEN SIGN"
$'\xE2\x82\xAC' x20AC € "EURO SIGN"

Yet only the x20AC of these four chars is unequivocally not "8 bit".

The char names I quote I took from http://www.unicode.org/charts/

The arcane $'\xXX\xXX\xXX' UTF-8 notation is from `man bash` re \xHH.

Pat LaVarre

Re: Unable to read UDF fs on a DVD

[Pat LaVarre]

> compression id 16 (search for "cid:"),
> means that the characters are coded 16 bits per character.
> 
> UDF 2.1.1:
> UDF supports standard Unicode 2.0 except the 'byte-order mark'
> chars #FEFF and #FFFE.
> These characters are coded in OSTA Compressed Unicode format,
> which means 8 bits per char or 16 bits per char.
> If a file identifier contains only unicode chars with al value
> less than #0100, compression id 8 can be used.

Link! Thank you. I clicked thru to:

--- http://www.osta.org/specs/pdf/udf250.pdf
--- (page 17 of 165)

2.1.1 Character Sets

The character set used by UDF for the structures defined in this
document is the ... OSTA CS0 character set ... defined as follows:
...

---

Between your English and their English, I conclude,

I should expect to see 8 or 16 bits per char.  Specifically, when I'm
looking at hex bytes, if I see x08 then thereafter I should see 8 bits
per char thereafter, but if I see x10 then thereafter I should see x10
bits per char.

That sure sounds easier than UTF-8 is, to decode visually from a
hexdump.  For example, I now think, with "OSTA Compressed Unicode", also
known as the "OSTA CS0 character set", the $'\xE2\x82\xAC' x20AC € "EURO
SIGN" will always appear as the plain hex byte pair x 20 AC.

With this much context in place, now the 2004-04-23 guess of "a problem
with 16 bit characters vs 8 bit characters" makes sense.  That guess
says cid 8 maybe works better than cid 16, maybe especially when we need
cid 16 to express a char outside of the x00..FF range.

Pat LaVarre