Re: Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1

Re: Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1

[Martin Steigerwald]

Sorry, forgot to keep Cc.

I think I will keep the disk as is for a while. I just won´t plug it to
the Amiga, since I have my Amiga backup on another dedicated
disk and do not want to overwrite my Linux backups. So its available
for testing for a while. It should be save as long as I use the disk
in Linux only.

For safety I will recreate all backups. I can checksum the BTRFS based
backup partition for checksum errors, but fscking the other non
BTRFS ones will only give a vague hint.


Am Sonntag, 17. Juni 2012 schrieb Martin Steigerwald:
> Hi Jens, hi Linux m68k developers,
> 
> I reported that as
> 
> https://bugzilla.kernel.org/show_bug.cgi?id=43511
> 
> I will attach there some more debug data like binary copy of RDB and
> such.
> 
> But maybe its easier to discuss here.

I think this one is pretty serious:

merkaba:~> pvdisplay /dev/sdb1
  --- Physical volume ---
  PV Name               /dev/sdb1
  VG Name               steigerwald
  PV Size               1,82 TiB / not usable 4,08 MiB
  Allocatable           yes 
  PE Size               4,00 MiB
  Total PE              476931
  Free PE               105731
  Allocated PE          371200
  PV UUID               ZXMECC-JAir-lX8Q-rLhS-W1cS-quwz-b3FXWp
   
merkaba:~> vgdisplay steigerwald
  --- Volume group ---
  VG Name               steigerwald
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  7
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                5
  Open LV               1
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               1,82 TiB
  PE Size               4,00 MiB
  Total PE              476931
  Alloc PE / Size       371200 / 1,42 TiB
  Free  PE / Size       105731 / 413,01 GiB
  VG UUID               uhjjE1-0yrD-Ch1A-d9qL-P5jY-UDXE-io4bpi


with

merkaba:~#1> amiga-fdisk -l /dev/sdc
Disk /dev/sdc: 3 heads, 16 sectors, 81396441 cylinders, RDB: 0
Logical Cylinders from 43 to 81396440, 24576  bytes/Cylinder 

Device     Boot Mount   Begin      End     Size   Pri  BBlks    System
/dev/sdc1       *        43   65536043   1572864024     0      0  Linux 
native
/dev/sdc2       *    65536044   78643244   314572824     0      0  
[unknown]
/dev/sdc3       *    78643245   81396440   66076704     0      0  Amiga 
FFS Int.


Due to truncating oversized partition instead of refusing to use
them the PV and consequently VG span the whole disk instead
of leaving space for the >350GB in the two other partitions:

merkaba:~> cat /proc/partitions 
major minor  #blocks  name

 […]
   8       16 1953514584 sdb
   8       17 1953513552 sdb1

(now sdb instead of sdc back when I run amiga-fdisk)


This is just asking for trouble. Thus:

Bug 43521 - Amiga RDB partitions: truncates miscaluted partitions size 
instead of refusing to use it
https://bugzilla.kernel.org/show_bug.cgi?id=43521


My suggestion is to bail out if a partition size looks insane and 
impossible.
Having to manually fix it up to access the data on disk still looks better
to me than potentially overwriting lots of data.



Until today I never actually did an Amiga backup to the disk, I just
created the two Amiga partitions and copied some screenshots of them
but that might be enough to explain the checksum errors I found in a
BTRFS backup partition.

I am now checking my Linux backups. They might be save, cause not all
space on the volume group is used.

See:

kernel got struck while scrubbing BTRFS with node- and leafsize 32768
http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg17108.html

s/struck/stuck

Thanks,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1

[jdow]

By the way, what did you use to setup the Amiga RDBs?

{^_^}   Joanne Dow

On 2012/06/17 01:33, Martin Steigerwald wrote:
> Sorry, forgot to keep Cc.
>
> I think I will keep the disk as is for a while. I just won´t plug it to
> the Amiga, since I have my Amiga backup on another dedicated
> disk and do not want to overwrite my Linux backups. So its available
> for testing for a while. It should be save as long as I use the disk
> in Linux only.
>
> For safety I will recreate all backups. I can checksum the BTRFS based
> backup partition for checksum errors, but fscking the other non
> BTRFS ones will only give a vague hint.
>
>
> Am Sonntag, 17. Juni 2012 schrieb Martin Steigerwald:
>> Hi Jens, hi Linux m68k developers,
>>
>> I reported that as
>>
>> https://bugzilla.kernel.org/show_bug.cgi?id=43511
>>
>> I will attach there some more debug data like binary copy of RDB and
>> such.
>>
>> But maybe its easier to discuss here.
>
> I think this one is pretty serious:
>
> merkaba:~> pvdisplay /dev/sdb1
>    --- Physical volume ---
>    PV Name               /dev/sdb1
>    VG Name               steigerwald
>    PV Size               1,82 TiB / not usable 4,08 MiB
>    Allocatable           yes
>    PE Size               4,00 MiB
>    Total PE              476931
>    Free PE               105731
>    Allocated PE          371200
>    PV UUID               ZXMECC-JAir-lX8Q-rLhS-W1cS-quwz-b3FXWp
>
> merkaba:~> vgdisplay steigerwald
>    --- Volume group ---
>    VG Name               steigerwald
>    System ID
>    Format                lvm2
>    Metadata Areas        2
>    Metadata Sequence No  7
>    VG Access             read/write
>    VG Status             resizable
>    MAX LV                0
>    Cur LV                5
>    Open LV               1
>    Max PV                0
>    Cur PV                1
>    Act PV                1
>    VG Size               1,82 TiB
>    PE Size               4,00 MiB
>    Total PE              476931
>    Alloc PE / Size       371200 / 1,42 TiB
>    Free  PE / Size       105731 / 413,01 GiB
>    VG UUID               uhjjE1-0yrD-Ch1A-d9qL-P5jY-UDXE-io4bpi
>
>
> with
>
> merkaba:~#1> amiga-fdisk -l /dev/sdc
> Disk /dev/sdc: 3 heads, 16 sectors, 81396441 cylinders, RDB: 0
> Logical Cylinders from 43 to 81396440, 24576  bytes/Cylinder
>
> Device     Boot Mount   Begin      End     Size   Pri  BBlks    System
> /dev/sdc1       *        43   65536043   1572864024     0      0  Linux
> native
> /dev/sdc2       *    65536044   78643244   314572824     0      0
> [unknown]
> /dev/sdc3       *    78643245   81396440   66076704     0      0  Amiga
> FFS Int.
>
>
> Due to truncating oversized partition instead of refusing to use
> them the PV and consequently VG span the whole disk instead
> of leaving space for the >350GB in the two other partitions:
>
> merkaba:~> cat /proc/partitions
> major minor  #blocks  name
>
>   […]
>     8       16 1953514584 sdb
>     8       17 1953513552 sdb1
>
> (now sdb instead of sdc back when I run amiga-fdisk)
>
>
> This is just asking for trouble. Thus:
>
> Bug 43521 - Amiga RDB partitions: truncates miscaluted partitions size
> instead of refusing to use it
> https://bugzilla.kernel.org/show_bug.cgi?id=43521
>
>
> My suggestion is to bail out if a partition size looks insane and
> impossible.
> Having to manually fix it up to access the data on disk still looks better
> to me than potentially overwriting lots of data.
>
>
>
> Until today I never actually did an Amiga backup to the disk, I just
> created the two Amiga partitions and copied some screenshots of them
> but that might be enough to explain the checksum errors I found in a
> BTRFS backup partition.
>
> I am now checking my Linux backups. They might be save, cause not all
> space on the volume group is used.
>
> See:
>
> kernel got struck while scrubbing BTRFS with node- and leafsize 32768
> http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg17108.html
>
> s/struck/stuck
>
> Thanks,
>

Re: Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1

[Martin Steigerwald]

Hi Joanne,

Great to see you around. Was quite a surprise for me! ;)

Am Sonntag, 17. Juni 2012 schrieb jdow:
> By the way, what did you use to setup the Amiga RDBs?

Media Toolbox, the AmigaOS successor of your fine HDToolBox on 
hdwrench.library.

Thanks
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1

[Martin Steigerwald]

Hi Jens, hi Linux m68k developers,

I reported that as

https://bugzilla.kernel.org/show_bug.cgi?id=43511

I will attach there some more debug data like binary copy of RDB and such.

But maybe its easier to discuss here.

I am not sure whether its an issue with Linux or an issue with the RDB 
format and disks with big sizes. But AFAIK RDB format is capable of 
handling 2 TB disks.




With my 2 TB mixed Amiga/Linux backup disk, which I partioned under 
AmigaOS 4.0/1 with Media Toolbox I get the following in Linux:


Jun 17 07:28:14 merkaba kernel: [30852.968978] sata_sil24 0000:05:00.0: 
enabling device (0000 -> 0003)
Jun 17 07:28:14 merkaba kernel: [30852.969401] scsi9 : sata_sil24
Jun 17 07:28:14 merkaba kernel: [30852.969533] ata10: SATA max UDMA/100 
host m128@0xf1c02000 port 0xf1c00000 irq 19
Jun 17 07:28:16 merkaba kernel: [30855.163712] ata10: SATA link up 3.0 
Gbps (SStatus 123 SControl 0)
Jun 17 07:28:16 merkaba kernel: [30855.165014] ata10.00: ATA-8: Hitachi 
HDS5C3020ALA632, ML6OA580, max UDMA/133
Jun 17 07:28:16 merkaba kernel: [30855.165017] ata10.00: 3907029168 
sectors, multi 16: LBA48 NCQ (depth 31/32)
Jun 17 07:28:16 merkaba kernel: [30855.166378] ata10.00: configured for 
UDMA/100
Jun 17 07:28:16 merkaba kernel: [30855.166477] scsi 9:0:0:0: Direct-Access     
ATA      Hitachi HDS5C302 ML6O PQ: 0 ANSI: 5
Jun 17 07:28:16 merkaba kernel: [30855.166653] sd 9:0:0:0: [sdb] 
3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
Jun 17 07:28:16 merkaba kernel: [30855.166699] sd 9:0:0:0: [sdb] Write 
Protect is off
Jun 17 07:28:16 merkaba kernel: [30855.166702] sd 9:0:0:0: [sdb] Mode 
Sense: 00 3a 00 00
Jun 17 07:28:16 merkaba kernel: [30855.166726] sd 9:0:0:0: [sdb] Write 
cache: enabled, read cache: enabled, doesn't support DPO or FUA
Jun 17 07:28:16 merkaba kernel: [30855.200936]  sdb: RDSK (512) sdb1 
(LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb 4)
Jun 17 07:28:16 merkaba kernel: [30855.200943] sdb: p1 size 
18446744072560312368 extends beyond EOD, enabling native capacity
Jun 17 07:28:16 merkaba kernel: [30855.201344]  sdb: RDSK (512) sdb1 
(LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb 4)
Jun 17 07:28:16 merkaba kernel: [30855.201347] sdb: p1 size 
18446744072560312368 extends beyond EOD, truncated
Jun 17 07:28:16 merkaba kernel: [30855.201398] sdb: p2 start 
18446744072560314432 is beyond EOD, truncated
Jun 17 07:28:16 merkaba kernel: [30855.201400] sdb: p3 start 
18446744073189460080 is beyond EOD, truncated
Jun 17 07:28:16 merkaba kernel: [30855.201570] sd 9:0:0:0: [sdb] Attached 
SCSI disk


The first partition seems to be way to big:

merkaba:~#1> amiga-fdisk -l /dev/sdc
Disk /dev/sdc: 3 heads, 16 sectors, 81396441 cylinders, RDB: 0
Logical Cylinders from 43 to 81396440, 24576  bytes/Cylinder 

Device     Boot Mount   Begin      End     Size   Pri  BBlks    System
/dev/sdc1       *        43   65536043   1572864024     0      0  Linux 
native
/dev/sdc2       *    65536044   78643244   314572824     0      0  
[unknown]
/dev/sdc3       *    78643245   81396440   66076704     0      0  Amiga 
FFS Int.

(sdc2 is JXFS, a new filesystem in AmigaOS 4 that is not known to Linux 
yet)


In cat /proc/partitions I get:

merkaba:~> cat /proc/partitions 
major minor  #blocks  name

 […]
   8       16 1953514584 sdb
   8       17 1953513552 sdb1
merkaba:~> 


Thus the Debian Linux Kernel 3.4.1 I am using here, truncates the first 
oversized partition and has no space for the other, too, which therefore 
are inaccessible under Linux.

I didn´t notice this initially, but it happened from the beginning, I have 
an old amiga-fdisk listing that is exactly the same.

Amiga Mediatoolbox has a different oppinion on the partition layout:

LVM aka sdc1: 1500 GByte, 43 to 65536043 cyl, size 65536001 Zylinder
BAK aka sdc2: 300 GByte, 65536044 to 78643244 cyl, size 13107201 Zylinder
TAUSCH2 aka sdc3: 63,016 GByte, 78643245 to  81396440 cyl, didn´t note the 
size here, but as far as I remember was okay as well

But it seems to be confused about the whole size of the disk as well:

Logical sizes:

Blocks per cylinder: 48
Cylinders: 81396441
Sectors: -397938128
Blocksize: 512

The sectors seem overflowed.

So it might be a problem with RDB and 2TB disks and nothing to do with 
Linux. But still on AmigaOS 4.1 I can access the two Amiga partitions 
after the Linux partition.



I have another 500GB disk for backup up my Sam440ep AmigaOS 4.1 "Amiga", I 
plan to repartition the 2 TB disk as GPT anyway, but since MediaToolBox in 
AmigaOS 4.1 has a different meaning about the partioning and this can cause 
serious data loss, I think its good to look at it.

I had a BTRFS filesystem that had some checksum errors. Maybe thats somehow 
related to this issue and AmigaOS and/or Linux overwrote something it 
shouldn´t have touched.

I will report a bug with AmigaOS 4.1 developers as well to get more 
details.


merkaba:~> cat /proc/version
Linux version 3.4-trunk-amd64 (Debian 3.4.1-1~experimental.1)
(debian-kernel@lists.debian.org) (gcc version 4.6.3 (Debian 4.6.3-1) ) #1 
SMP
Wed Jun 6 10:34:53 CEST 2012

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1

[jdow]

I'll have to make time to look it over, considering I am more or less
the mommy of the RDBs. Now, 2TB may be a tad beyond the abilities of
the RDBs to express. Memory insists that the RDBs worked on either
BYTE or block counts. Your seeing a problem at 2TB suggests it really
stored bit counts. The numbers are supposed to be unsigned. And whether
the RDBs store values in blocks or BYTEs is immaterial when you get
down to it. (The RDBs do store the disk's actual and virtual block
sizes. The latter is what the filesystem uses.)

However, since C programs seek plus and minus in BYTEs you are stuck
with 31 bits worth of disk as your limit. That matches nicely with
your 31 bits. The RDBs should still be useful if you make sure to
plant them on block 1 or higher leaving block zero for other OSs and
you only use the first 2 TB for the Amiga OS files. (RDPrep and the
HDWrench library both exhibited a strong preference for starting the
RDB blocks on block 3 of the disk.)

{^_^}   Joanne Dow, yeah, that antique broad.

On 2012/06/16 23:41, Martin Steigerwald wrote:
> Hi Jens, hi Linux m68k developers,
>
> I reported that as
>
> https://bugzilla.kernel.org/show_bug.cgi?id=43511
>
> I will attach there some more debug data like binary copy of RDB and such.
>
> But maybe its easier to discuss here.
>
> I am not sure whether its an issue with Linux or an issue with the RDB
> format and disks with big sizes. But AFAIK RDB format is capable of
> handling 2 TB disks.
>
>
>
>
> With my 2 TB mixed Amiga/Linux backup disk, which I partioned under
> AmigaOS 4.0/1 with Media Toolbox I get the following in Linux:
>
>
> Jun 17 07:28:14 merkaba kernel: [30852.968978] sata_sil24 0000:05:00.0:
> enabling device (0000 -> 0003)
> Jun 17 07:28:14 merkaba kernel: [30852.969401] scsi9 : sata_sil24
> Jun 17 07:28:14 merkaba kernel: [30852.969533] ata10: SATA max UDMA/100
> host m128@0xf1c02000 port 0xf1c00000 irq 19
> Jun 17 07:28:16 merkaba kernel: [30855.163712] ata10: SATA link up 3.0
> Gbps (SStatus 123 SControl 0)
> Jun 17 07:28:16 merkaba kernel: [30855.165014] ata10.00: ATA-8: Hitachi
> HDS5C3020ALA632, ML6OA580, max UDMA/133
> Jun 17 07:28:16 merkaba kernel: [30855.165017] ata10.00: 3907029168
> sectors, multi 16: LBA48 NCQ (depth 31/32)
> Jun 17 07:28:16 merkaba kernel: [30855.166378] ata10.00: configured for
> UDMA/100
> Jun 17 07:28:16 merkaba kernel: [30855.166477] scsi 9:0:0:0: Direct-Access
> ATA      Hitachi HDS5C302 ML6O PQ: 0 ANSI: 5
> Jun 17 07:28:16 merkaba kernel: [30855.166653] sd 9:0:0:0: [sdb]
> 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
> Jun 17 07:28:16 merkaba kernel: [30855.166699] sd 9:0:0:0: [sdb] Write
> Protect is off
> Jun 17 07:28:16 merkaba kernel: [30855.166702] sd 9:0:0:0: [sdb] Mode
> Sense: 00 3a 00 00
> Jun 17 07:28:16 merkaba kernel: [30855.166726] sd 9:0:0:0: [sdb] Write
> cache: enabled, read cache: enabled, doesn't support DPO or FUA
> Jun 17 07:28:16 merkaba kernel: [30855.200936]  sdb: RDSK (512) sdb1
> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb 4)
> Jun 17 07:28:16 merkaba kernel: [30855.200943] sdb: p1 size
> 18446744072560312368 extends beyond EOD, enabling native capacity
> Jun 17 07:28:16 merkaba kernel: [30855.201344]  sdb: RDSK (512) sdb1
> (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb 4)
> Jun 17 07:28:16 merkaba kernel: [30855.201347] sdb: p1 size
> 18446744072560312368 extends beyond EOD, truncated
> Jun 17 07:28:16 merkaba kernel: [30855.201398] sdb: p2 start
> 18446744072560314432 is beyond EOD, truncated
> Jun 17 07:28:16 merkaba kernel: [30855.201400] sdb: p3 start
> 18446744073189460080 is beyond EOD, truncated
> Jun 17 07:28:16 merkaba kernel: [30855.201570] sd 9:0:0:0: [sdb] Attached
> SCSI disk
>
>
> The first partition seems to be way to big:
>
> merkaba:~#1> amiga-fdisk -l /dev/sdc
> Disk /dev/sdc: 3 heads, 16 sectors, 81396441 cylinders, RDB: 0
> Logical Cylinders from 43 to 81396440, 24576  bytes/Cylinder
>
> Device     Boot Mount   Begin      End     Size   Pri  BBlks    System
> /dev/sdc1       *        43   65536043   1572864024     0      0  Linux
> native
> /dev/sdc2       *    65536044   78643244   314572824     0      0
> [unknown]
> /dev/sdc3       *    78643245   81396440   66076704     0      0  Amiga
> FFS Int.
>
> (sdc2 is JXFS, a new filesystem in AmigaOS 4 that is not known to Linux
> yet)
>
>
> In cat /proc/partitions I get:
>
> merkaba:~> cat /proc/partitions
> major minor  #blocks  name
>
>   […]
>     8       16 1953514584 sdb
>     8       17 1953513552 sdb1
> merkaba:~>
>
>
> Thus the Debian Linux Kernel 3.4.1 I am using here, truncates the first
> oversized partition and has no space for the other, too, which therefore
> are inaccessible under Linux.
>
> I didn´t notice this initially, but it happened from the beginning, I have
> an old amiga-fdisk listing that is exactly the same.
>
> Amiga Mediatoolbox has a different oppinion on the partition layout:
>
> LVM aka sdc1: 1500 GByte, 43 to 65536043 cyl, size 65536001 Zylinder
> BAK aka sdc2: 300 GByte, 65536044 to 78643244 cyl, size 13107201 Zylinder
> TAUSCH2 aka sdc3: 63,016 GByte, 78643245 to  81396440 cyl, didn´t note the
> size here, but as far as I remember was okay as well
>
> But it seems to be confused about the whole size of the disk as well:
>
> Logical sizes:
>
> Blocks per cylinder: 48
> Cylinders: 81396441
> Sectors: -397938128
> Blocksize: 512
>
> The sectors seem overflowed.
>
> So it might be a problem with RDB and 2TB disks and nothing to do with
> Linux. But still on AmigaOS 4.1 I can access the two Amiga partitions
> after the Linux partition.
>
>
>
> I have another 500GB disk for backup up my Sam440ep AmigaOS 4.1 "Amiga", I
> plan to repartition the 2 TB disk as GPT anyway, but since MediaToolBox in
> AmigaOS 4.1 has a different meaning about the partioning and this can cause
> serious data loss, I think its good to look at it.
>
> I had a BTRFS filesystem that had some checksum errors. Maybe thats somehow
> related to this issue and AmigaOS and/or Linux overwrote something it
> shouldn´t have touched.
>
> I will report a bug with AmigaOS 4.1 developers as well to get more
> details.
>
>
> merkaba:~> cat /proc/version
> Linux version 3.4-trunk-amd64 (Debian 3.4.1-1~experimental.1)
> (debian-kernel@lists.debian.org) (gcc version 4.6.3 (Debian 4.6.3-1) ) #1
> SMP
> Wed Jun 6 10:34:53 CEST 2012
>
> Ciao,
>

Re: Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1

[Martin Steigerwald]

Am Sonntag, 17. Juni 2012 schrieb jdow:
> I'll have to make time to look it over, considering I am more or less
> the mommy of the RDBs. Now, 2TB may be a tad beyond the abilities of
> the RDBs to express. Memory insists that the RDBs worked on either
> BYTE or block counts. Your seeing a problem at 2TB suggests it really
> stored bit counts. The numbers are supposed to be unsigned. And whether
> the RDBs store values in blocks or BYTEs is immaterial when you get
> down to it. (The RDBs do store the disk's actual and virtual block
> sizes. The latter is what the filesystem uses.)

Hmmm, I thought any 2 TB limit was gone meanwhile, but then your 
argumentation has some merit.

Some hints that it might be gone nonetheless:

| JXFS 64 bit file system
|
| With AmigaOS 4.x a new file system has been introduced called JXFS. It is 
| a totally new 64 bit file system that supports partitions up to 16 TB in 
| size. It is a modern journalling file system, which means that it reduces 
| data loss if data writes to the disk are interrupted. It is the fastest 
| and most reliable file system ever created for AmigaOS.

http://www.amigaos.net/content/1/features

Well I asked AmigaOS 4 developers about this issue as well. Lets see what 
they say about 2 TB limits.

But nonetheless, I think, Linux should refuse to use a partition that is 
clearly out of bound.

Scrubbing the one backup BTRFS that has 360 GiB of data was okay. Also the 
fsck´s went well. Well the volume group is not completely full so some 
space on the end of the disk is not yet used by it.

I am recreating all my backups on BTRFS volumes now - except maybe the one 
thats on the BTRFS volume already and scrubs okay.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1

[Geert Uytterhoeven]

On Sun, Jun 17, 2012 at 2:58 PM, Martin Steigerwald <Martin@lichtvoll.de> wrote:
> Am Sonntag, 17. Juni 2012 schrieb jdow:
> | JXFS 64 bit file system
> |
> | With AmigaOS 4.x a new file system has been introduced called JXFS. It is
> | a totally new 64 bit file system that supports partitions up to 16 TB in
> | size. It is a modern journalling file system, which means that it reduces
> | data loss if data writes to the disk are interrupted. It is the fastest
> | and most reliable file system ever created for AmigaOS.
>
> http://www.amigaos.net/content/1/features
>
> Well I asked AmigaOS 4 developers about this issue as well. Lets see what
> they say about 2 TB limits.

16 TB = 2 TB * 8. Perhaps they increased the block size from 512 to 4096?

block/partitions/amiga.c reads the block size from
RigidDiskBlock.rdb_BlockBytes,
but after conversion to 512-byte blocks, all further calculations are done on
"int", so it will overflow for disks larger than 2 TiB.

Note that in your profile-binary.img, the field is 0x200, i.e. 512
bytes per block,
so I'll have to get a deeper look into your RDB first...

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1

[Martin Steigerwald]

Am Sonntag, 17. Juni 2012 schrieb Geert Uytterhoeven:
> On Sun, Jun 17, 2012 at 2:58 PM, Martin Steigerwald <Martin@lichtvoll.de> wrote:
> > Am Sonntag, 17. Juni 2012 schrieb jdow:
> > | JXFS 64 bit file system
> > | 
> > | With AmigaOS 4.x a new file system has been introduced called JXFS.
> > | It is a totally new 64 bit file system that supports partitions up
> > | to 16 TB in size. It is a modern journalling file system, which
> > | means that it reduces data loss if data writes to the disk are
> > | interrupted. It is the fastest and most reliable file system ever
> > | created for AmigaOS.
> > 
> > http://www.amigaos.net/content/1/features
> > 
> > Well I asked AmigaOS 4 developers about this issue as well. Lets see
> > what they say about 2 TB limits.
> 
> 16 TB = 2 TB * 8. Perhaps they increased the block size from 512 to
> 4096?

I hope to get anything back from the AmigaOS 4 developers.

> block/partitions/amiga.c reads the block size from
> RigidDiskBlock.rdb_BlockBytes,
> but after conversion to 512-byte blocks, all further calculations are
> done on "int", so it will overflow for disks larger than 2 TiB.
> 
> Note that in your profile-binary.img, the field is 0x200, i.e. 512
> bytes per block,
> so I'll have to get a deeper look into your RDB first...

Okay, thanks!

I did not get any information regarding the current size limit yet.

Strangely on AmigaOS 4.1 all values seem to be fine except for the total
sectors value. 

And on Linux begin and end cylinders are correct, only size is off:

merkaba:~> amiga-fdisk -l /dev/sdb                         
Disk /dev/sdb: 3 heads, 16 sectors, 81396441 cylinders, RDB: 0
Logical Cylinders from 43 to 81396440, 24576  bytes/Cylinder 

Device     Boot Mount   Begin      End     Size   Pri  BBlks    System
/dev/sdb1       *        43   65536043   1572864024     0      0  Linux native
/dev/sdb2       *    65536044   78643244   314572824     0      0  [unknown]
/dev/sdb3       *    78643245   81396440   66076704     0      0  Amiga FFS Int.

But not only from the first, also of the second and third one it seems.

65536043 - 43 = 65536000

78643244 - 65536044 = 13107200

81396440 - 78643245 = 2753195

I would think that if there is a overflow it would already be in the cylinder
numbers.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1

[jdow]

On 2012/06/17 14:06, Martin Steigerwald wrote:
> Am Sonntag, 17. Juni 2012 schrieb Geert Uytterhoeven:
>> On Sun, Jun 17, 2012 at 2:58 PM, Martin Steigerwald <Martin@lichtvoll.de> wrote:
>>> Am Sonntag, 17. Juni 2012 schrieb jdow:
>>> | JXFS 64 bit file system
>>> |
>>> | With AmigaOS 4.x a new file system has been introduced called JXFS.
>>> | It is a totally new 64 bit file system that supports partitions up
>>> | to 16 TB in size. It is a modern journalling file system, which
>>> | means that it reduces data loss if data writes to the disk are
>>> | interrupted. It is the fastest and most reliable file system ever
>>> | created for AmigaOS.
>>>
>>> http://www.amigaos.net/content/1/features
>>>
>>> Well I asked AmigaOS 4 developers about this issue as well. Lets see
>>> what they say about 2 TB limits.
>>
>> 16 TB = 2 TB * 8. Perhaps they increased the block size from 512 to
>> 4096?
>
> I hope to get anything back from the AmigaOS 4 developers.
>
>> block/partitions/amiga.c reads the block size from
>> RigidDiskBlock.rdb_BlockBytes,
>> but after conversion to 512-byte blocks, all further calculations are
>> done on "int", so it will overflow for disks larger than 2 TiB.
>>
>> Note that in your profile-binary.img, the field is 0x200, i.e. 512
>> bytes per block,
>> so I'll have to get a deeper look into your RDB first...
>
> Okay, thanks!
>
> I did not get any information regarding the current size limit yet.
>
> Strangely on AmigaOS 4.1 all values seem to be fine except for the total
> sectors value.
>
> And on Linux begin and end cylinders are correct, only size is off:

Ah, you DO know that "cylinders, surfaces, and tracks" are polite
fictions in AmigaOS, don't you? Start and End blocks are all that
matter on a real Amiga. The fictions arose because at first it was
thought they could be used to optimize disk accesses. Once drives
were notched these values became meaningless. So they're created on
the fly picking values out of the nose or something. (RDPrep tries
to find reasonable size factors for the total block counts.)

> merkaba:~> amiga-fdisk -l /dev/sdb

Are you sure "amiga-fdisk" is not broken?

> Disk /dev/sdb: 3 heads, 16 sectors, 81396441 cylinders, RDB: 0
> Logical Cylinders from 43 to 81396440, 24576  bytes/Cylinder
>
> Device     Boot Mount   Begin      End     Size   Pri  BBlks    System
> /dev/sdb1       *        43   65536043   1572864024     0      0  Linux native
> /dev/sdb2       *    65536044   78643244   314572824     0      0  [unknown]
> /dev/sdb3       *    78643245   81396440   66076704     0      0  Amiga FFS Int.
>
> But not only from the first, also of the second and third one it seems.
>
> 65536043 - 43 = 65536000

So the size in bytes is 24 times the byte offset of the start of the
next partition. Fascinating. Let's see. You are working in 512 byte blocks
it looks like. With RDBs in blocks that means you can get up to
1099511627776 bytes, 2147483648 blocks, or 44739242. So you are already
WAY over what can be expressed in the 32 bit values in the RDBs. So the
software that prepared your partitioning needs some repair work of some
sort or something other than traditional Amiga FFS format disks.

The first thing on the agenda is "fixing" the partitioning software. Is
the version of Amiga FFS you are using cognizant of 64 bit values? If not
you will have to go to block sizes larger than 512 bytes. It looks like
1k is suitable for this instance. Given the way Amiga FFS stores data on
the disk I'd go for 4k or 8k block sizes unless you have lots of very
small files.

> 78643244 - 65536044 = 13107200
>
> 81396440 - 78643245 = 2753195
>
> I would think that if there is a overflow it would already be in the cylinder
> numbers.
>
> Ciao,

The RDB reading software may not be at fault. I'll have to check. Your
partitioning tool let you down. It looks like you've overflowed the 31
(useful) bit values in your RDBs. So not much can recover from this.
Play with signed arithmetic and treat the final value as unsigned and
see if you get the size noted.

{^_^}

Re: Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1

[Martin Steigerwald]

Am Sonntag, 17. Juni 2012 schrieb jdow:
> On 2012/06/17 14:06, Martin Steigerwald wrote:
> > Am Sonntag, 17. Juni 2012 schrieb Geert Uytterhoeven:
> >> On Sun, Jun 17, 2012 at 2:58 PM, Martin Steigerwald 
<Martin@lichtvoll.de> wrote:
> >>> Am Sonntag, 17. Juni 2012 schrieb jdow:
> >>> | JXFS 64 bit file system
> >>> | 
> >>> | With AmigaOS 4.x a new file system has been introduced called
> >>> | JXFS. It is a totally new 64 bit file system that supports
> >>> | partitions up to 16 TB in size. It is a modern journalling file
> >>> | system, which means that it reduces data loss if data writes to
> >>> | the disk are interrupted. It is the fastest and most reliable
> >>> | file system ever created for AmigaOS.
> >>> 
> >>> http://www.amigaos.net/content/1/features
> >>> 
> >>> Well I asked AmigaOS 4 developers about this issue as well. Lets
> >>> see what they say about 2 TB limits.
> >> 
> >> 16 TB = 2 TB * 8. Perhaps they increased the block size from 512 to
> >> 4096?
> > 
> > I hope to get anything back from the AmigaOS 4 developers.
> > 
> >> block/partitions/amiga.c reads the block size from
> >> RigidDiskBlock.rdb_BlockBytes,
> >> but after conversion to 512-byte blocks, all further calculations
> >> are done on "int", so it will overflow for disks larger than 2 TiB.
> >> 
> >> Note that in your profile-binary.img, the field is 0x200, i.e. 512
> >> bytes per block,
> >> so I'll have to get a deeper look into your RDB first...
> > 
> > Okay, thanks!
> > 
> > I did not get any information regarding the current size limit yet.
> > 
> > Strangely on AmigaOS 4.1 all values seem to be fine except for the
> > total sectors value.
> 
> > And on Linux begin and end cylinders are correct, only size is off:
> Ah, you DO know that "cylinders, surfaces, and tracks" are polite
> fictions in AmigaOS, don't you? Start and End blocks are all that
> matter on a real Amiga. The fictions arose because at first it was
> thought they could be used to optimize disk accesses. Once drives
> were notched these values became meaningless. So they're created on
> the fly picking values out of the nose or something. (RDPrep tries
> to find reasonable size factors for the total block counts.)

I know there are pure fiction on Linux as well. As in any other modern 
operating system.

Actually Media Toolbox shows both. Physical and logical sizes. See the 
screenshots I attached to the bug report.

> > merkaba:~> amiga-fdisk -l /dev/sdb
> 
> Are you sure "amiga-fdisk" is not broken?

Not at all, but there is also the syslog.

> > Disk /dev/sdb: 3 heads, 16 sectors, 81396441 cylinders, RDB: 0
> > Logical Cylinders from 43 to 81396440, 24576  bytes/Cylinder
> > 
> > Device     Boot Mount   Begin      End     Size   Pri  BBlks   
> > System /dev/sdb1       *        43   65536043   1572864024     0    
> >  0  Linux native /dev/sdb2       *    65536044   78643244  
> > 314572824     0      0  [unknown] /dev/sdb3       *    78643245  
> > 81396440   66076704     0      0  Amiga FFS Int.
> > 
> > But not only from the first, also of the second and third one it
> > seems.
> > 
> > 65536043 - 43 = 65536000
> 
> So the size in bytes is 24 times the byte offset of the start of the
> next partition. Fascinating. Let's see. You are working in 512 byte
> blocks it looks like. With RDBs in blocks that means you can get up to
> 1099511627776 bytes, 2147483648 blocks, or 44739242. So you are
> already WAY over what can be expressed in the 32 bit values in the
> RDBs. So the software that prepared your partitioning needs some
> repair work of some sort or something other than traditional Amiga FFS
> format disks.
> 
> The first thing on the agenda is "fixing" the partitioning software. Is
> the version of Amiga FFS you are using cognizant of 64 bit values? If
> not you will have to go to block sizes larger than 512 bytes. It looks
> like 1k is suitable for this instance. Given the way Amiga FFS stores
> data on the disk I'd go for 4k or 8k block sizes unless you have lots
> of very small files.

Is there still something to fix in there? Just still catching up the mail 
exchange. From what I could see on AmigaOS 4.1 all seemed well. I already 
reported the negative sector count value.

Anyway, if there is an issue left we can discuss this privately. This 
would have nothing to do with Linux and I can make sure that current 
AmigaOS developers hear about your oppinion.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1

[jdow]

On 2012/06/17 14:06, Martin Steigerwald wrote:
...

I have a usage note for you and the person who did that partitioning.
If you start the partition at block 1 instead of block zero it probably
can coexist nicely with normal fdisk partitioning as well. (It used to
be able to since x86 partition tables didn't bother with blocks on the
first "cylinder" other than block zero.) I designed RDPrepX and
HDWrench to take advantage of that. (And I used it myself. Erm I used a
LOT of abominations such as safely nested partitions.)

{^_^}

Re: Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1

[jdow]

On 2012/06/17 09:36, Geert Uytterhoeven wrote:
> On Sun, Jun 17, 2012 at 2:58 PM, Martin Steigerwald <Martin@lichtvoll.de> wrote:
>> Am Sonntag, 17. Juni 2012 schrieb jdow:
>> | JXFS 64 bit file system
>> |
>> | With AmigaOS 4.x a new file system has been introduced called JXFS. It is
>> | a totally new 64 bit file system that supports partitions up to 16 TB in
>> | size. It is a modern journalling file system, which means that it reduces
>> | data loss if data writes to the disk are interrupted. It is the fastest
>> | and most reliable file system ever created for AmigaOS.
>>
>> http://www.amigaos.net/content/1/features
>>
>> Well I asked AmigaOS 4 developers about this issue as well. Lets see what
>> they say about 2 TB limits.
>
> 16 TB = 2 TB * 8. Perhaps they increased the block size from 512 to 4096?
>
> block/partitions/amiga.c reads the block size from
> RigidDiskBlock.rdb_BlockBytes,
> but after conversion to 512-byte blocks, all further calculations are done on
> "int", so it will overflow for disks larger than 2 TiB.
>
> Note that in your profile-binary.img, the field is 0x200, i.e. 512
> bytes per block,
> so I'll have to get a deeper look into your RDB first...
>
> Gr{oetje,eeting}s,
>
>                          Geert

Note that the work I did on the Linux RDB code eons ago took full
cognizance of the physical and virtual block sizes.

I still have, I believe, a working Fuji Magneto Optical drive with 2k
sectors. I used that for ages for moving data from systems at two
different houses as I moved back and forth. I worked on the Linux RDB
code because I wanted to be able to read those disks. I've been vaguely
wondering what happened to the code in the latest builds. Now I guess I
will find out.

If RDBs are going to remain backwards compatible and AmigaOS disk usage
is to remain sensible larger logical blocks, at the very least, are needed.
Since both values (should) exist within the RDBs the partitions that are
described in the RDBs can be managed by reading the logical block size and
multiplying it by the ending block number storing as 64 bits. It sounds
like this is not being done correctly. I may need some guidance to see
about where to put the 64 bit byte position of the starting and ending
blocks.

I've asked Martin for a digital copy of his RDBs and what he thinks the
partition(s) should look like. I should also be told whether the disk
is supposed to be solely Amiga OSs or not. I gather it's not.

(And for God's sake do NOT implement the two virus storage tools within
the RDBs, the RDB encapsulated filesystems and the RDB encapsulated
driver code. I worried about that potential since RDBs were first
introduced. Oddly I never heard of them being used. So I kept quiet
about it. These days those facilities could be extended to use signing
and validation certificates, I suppose.)

{^_^}

Re: Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1

[Geert Uytterhoeven]

Hi Joanne,

On Sun, Jun 17, 2012 at 11:06 PM, jdow <jdow@earthlink.net> wrote:
> I've asked Martin for a digital copy of his RDBs and what he thinks the
> partition(s) should look like. I should also be told whether the disk
> is supposed to be solely Amiga OSs or not. I gather it's not.

His RDB is attached to https://bugzilla.kernel.org/show_bug.cgi?id=43521

> (And for God's sake do NOT implement the two virus storage tools within
> the RDBs, the RDB encapsulated filesystems and the RDB encapsulated
> driver code. I worried about that potential since RDBs were first
> introduced. Oddly I never heard of them being used. So I kept quiet
> about it. These days those facilities could be extended to use signing
> and validation certificates, I suppose.)

JFYI, I used the former to store my MultiUser File System ;-)

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1

[jdow]

On 2012/06/17 14:15, Geert Uytterhoeven wrote:
> Hi Joanne,
>
> On Sun, Jun 17, 2012 at 11:06 PM, jdow <jdow@earthlink.net> wrote:
>> I've asked Martin for a digital copy of his RDBs and what he thinks the
>> partition(s) should look like. I should also be told whether the disk
>> is supposed to be solely Amiga OSs or not. I gather it's not.
>
> His RDB is attached to https://bugzilla.kernel.org/show_bug.cgi?id=43521


I want the raw binary on the blocks 0 through the end of the RDBs or
128 blocks, which ever comes first. I don't see that there. I just
see the amiga-fdisk printout of it. With the actual binary I can
parse it myself and see what comes out.

(As a side note - RDBs work with large sector sizes. They pretty much
always have.)


...

>
> Gr{oetje,eeting}s,
>
>                          Geert

{^_^}

Re: Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1

[Martin Steigerwald]

[-- Attachment #1: Type: Text/Plain, Size: 2112 bytes --]

Am Sonntag, 17. Juni 2012 schrieb jdow:
> On 2012/06/17 09:36, Geert Uytterhoeven wrote:
> > On Sun, Jun 17, 2012 at 2:58 PM, Martin Steigerwald 
<Martin@lichtvoll.de> wrote:
> >> Am Sonntag, 17. Juni 2012 schrieb jdow:
> >> | JXFS 64 bit file system
> >> | 
> >> | With AmigaOS 4.x a new file system has been introduced called
> >> | JXFS. It is a totally new 64 bit file system that supports
> >> | partitions up to 16 TB in size. It is a modern journalling file
> >> | system, which means that it reduces data loss if data writes to
> >> | the disk are interrupted. It is the fastest and most reliable
> >> | file system ever created for AmigaOS.
> >> 
> >> http://www.amigaos.net/content/1/features
> >> 
> >> Well I asked AmigaOS 4 developers about this issue as well. Lets see
> >> what they say about 2 TB limits.
> > 
> > 16 TB = 2 TB * 8. Perhaps they increased the block size from 512 to
> > 4096?
> > 
> > block/partitions/amiga.c reads the block size from
> > RigidDiskBlock.rdb_BlockBytes,
> > but after conversion to 512-byte blocks, all further calculations are
> > done on "int", so it will overflow for disks larger than 2 TiB.
> > 
> > Note that in your profile-binary.img, the field is 0x200, i.e. 512
> > bytes per block,
> > so I'll have to get a deeper look into your RDB first...
[…]
> Note that the work I did on the Linux RDB code eons ago took full
> cognizance of the physical and virtual block sizes.
[…]
> I've asked Martin for a digital copy of his RDBs and what he thinks the
> partition(s) should look like. I should also be told whether the disk
> is supposed to be solely Amiga OSs or not. I gather it's not.

Its all in the bug report. profile-binary.img should be it.

Its small so I attach it.

Meanwhile I try to get the currently supported maximum disk size out from 
the OS 4 developers. Maybe JXFS is playing tricks that other filesystems do 
not play or simply has a different implementation.

Thanks,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

[-- Attachment #2: profile-binary.img --]
[-- Type: application/octet-stream, Size: 2048 bytes --]

Re: Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1

[jdow]

On 2012/06/17 14:20, Martin Steigerwald wrote:
> Am Sonntag, 17. Juni 2012 schrieb jdow:
>> On 2012/06/17 09:36, Geert Uytterhoeven wrote:
>>> On Sun, Jun 17, 2012 at 2:58 PM, Martin Steigerwald
> <Martin@lichtvoll.de> wrote:
>>>> Am Sonntag, 17. Juni 2012 schrieb jdow:
>>>> | JXFS 64 bit file system
>>>> |
>>>> | With AmigaOS 4.x a new file system has been introduced called
>>>> | JXFS. It is a totally new 64 bit file system that supports
>>>> | partitions up to 16 TB in size. It is a modern journalling file
>>>> | system, which means that it reduces data loss if data writes to
>>>> | the disk are interrupted. It is the fastest and most reliable
>>>> | file system ever created for AmigaOS.
>>>>
>>>> http://www.amigaos.net/content/1/features
>>>>
>>>> Well I asked AmigaOS 4 developers about this issue as well. Lets see
>>>> what they say about 2 TB limits.
>>>
>>> 16 TB = 2 TB * 8. Perhaps they increased the block size from 512 to
>>> 4096?
>>>
>>> block/partitions/amiga.c reads the block size from
>>> RigidDiskBlock.rdb_BlockBytes,
>>> but after conversion to 512-byte blocks, all further calculations are
>>> done on "int", so it will overflow for disks larger than 2 TiB.
>>>
>>> Note that in your profile-binary.img, the field is 0x200, i.e. 512
>>> bytes per block,
>>> so I'll have to get a deeper look into your RDB first...
> […]
>> Note that the work I did on the Linux RDB code eons ago took full
>> cognizance of the physical and virtual block sizes.
> […]
>> I've asked Martin for a digital copy of his RDBs and what he thinks the
>> partition(s) should look like. I should also be told whether the disk
>> is supposed to be solely Amiga OSs or not. I gather it's not.
>
> Its all in the bug report. profile-binary.img should be it.
>
> Its small so I attach it.
>
> Meanwhile I try to get the currently supported maximum disk size out from
> the OS 4 developers. Maybe JXFS is playing tricks that other filesystems do
> not play or simply has a different implementation.
>
> Thanks,

Ah it was on the report to which your bugzilla entry I had referred to.
OK, it looks like it's not been overwritten - one of the usual overflow
symptoms. Now I have to hook it up with my handy dandy translator and see
what that says. I couldn't find it on 43521.

{^_^}

Re: Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1

[jdow]

On 2012/06/17 14:20, Martin Steigerwald wrote:
> Am Sonntag, 17. Juni 2012 schrieb jdow:
>> On 2012/06/17 09:36, Geert Uytterhoeven wrote:
>>> On Sun, Jun 17, 2012 at 2:58 PM, Martin Steigerwald
> <Martin@lichtvoll.de> wrote:
>>>> Am Sonntag, 17. Juni 2012 schrieb jdow:
>>>> | JXFS 64 bit file system
>>>> |
>>>> | With AmigaOS 4.x a new file system has been introduced called
>>>> | JXFS. It is a totally new 64 bit file system that supports
>>>> | partitions up to 16 TB in size. It is a modern journalling file
>>>> | system, which means that it reduces data loss if data writes to
>>>> | the disk are interrupted. It is the fastest and most reliable
>>>> | file system ever created for AmigaOS.
>>>>
>>>> http://www.amigaos.net/content/1/features
>>>>
>>>> Well I asked AmigaOS 4 developers about this issue as well. Lets see
>>>> what they say about 2 TB limits.
>>>
>>> 16 TB = 2 TB * 8. Perhaps they increased the block size from 512 to
>>> 4096?
>>>
>>> block/partitions/amiga.c reads the block size from
>>> RigidDiskBlock.rdb_BlockBytes,
>>> but after conversion to 512-byte blocks, all further calculations are
>>> done on "int", so it will overflow for disks larger than 2 TiB.
>>>
>>> Note that in your profile-binary.img, the field is 0x200, i.e. 512
>>> bytes per block,
>>> so I'll have to get a deeper look into your RDB first...
> […]
>> Note that the work I did on the Linux RDB code eons ago took full
>> cognizance of the physical and virtual block sizes.
> […]
>> I've asked Martin for a digital copy of his RDBs and what he thinks the
>> partition(s) should look like. I should also be told whether the disk
>> is supposed to be solely Amiga OSs or not. I gather it's not.
>
> Its all in the bug report. profile-binary.img should be it.
>
> Its small so I attach it.
>
> Meanwhile I try to get the currently supported maximum disk size out from
> the OS 4 developers. Maybe JXFS is playing tricks that other filesystems do
> not play or simply has a different implementation.
>
> Thanks,

I've sent Jens a proposed fix changing these lines in amiga.c.
===8<--- was
         struct PartitionBlock *pb;
         int start_sect, nr_sects, blk, part, res = 0;
         int blksize = 1;        /* Multiplier for disk block size */
===8<--- becomes changing line 338 into lines 338-339
         struct PartitionBlock *pb;
         sector_t start_sect, nr_sects;
         int blk, part, res = 0;
         int blksize = 1;        /* Multiplier for disk block size */
===8<---

(I'm working without proper tools at the moment or I'd make a real diff.
Sorry.)

This fix may not be complete. The RDBs contain provisions for describing
the physical disk block size and how many physical blocks are used in a
file system's logical blocks. This MAY not work quite right large physical
block sizes.

{^_^}   Joanne Dow

Re: Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1

[Martin Steigerwald]

Am Montag, 18. Juni 2012 schrieb jdow:
> On 2012/06/17 14:20, Martin Steigerwald wrote:
> > Am Sonntag, 17. Juni 2012 schrieb jdow:
> >> On 2012/06/17 09:36, Geert Uytterhoeven wrote:
> >>> On Sun, Jun 17, 2012 at 2:58 PM, Martin Steigerwald
> > 
> > <Martin@lichtvoll.de> wrote:
> >>>> Am Sonntag, 17. Juni 2012 schrieb jdow:
> >>>> | JXFS 64 bit file system
> >>>> | 
> >>>> | With AmigaOS 4.x a new file system has been introduced called
> >>>> | JXFS. It is a totally new 64 bit file system that supports
> >>>> | partitions up to 16 TB in size. It is a modern journalling file
> >>>> | system, which means that it reduces data loss if data writes to
> >>>> | the disk are interrupted. It is the fastest and most reliable
> >>>> | file system ever created for AmigaOS.
> >>>> 
> >>>> http://www.amigaos.net/content/1/features
> >>>> 
> >>>> Well I asked AmigaOS 4 developers about this issue as well. Lets
> >>>> see what they say about 2 TB limits.
> >>> 
> >>> 16 TB = 2 TB * 8. Perhaps they increased the block size from 512 to
> >>> 4096?
> >>> 
> >>> block/partitions/amiga.c reads the block size from
> >>> RigidDiskBlock.rdb_BlockBytes,
> >>> but after conversion to 512-byte blocks, all further calculations
> >>> are done on "int", so it will overflow for disks larger than 2
> >>> TiB.
> >>> 
> >>> Note that in your profile-binary.img, the field is 0x200, i.e. 512
> >>> bytes per block,
> >>> so I'll have to get a deeper look into your RDB first...
> > 
> > […]
> > 
> >> Note that the work I did on the Linux RDB code eons ago took full
> >> cognizance of the physical and virtual block sizes.
> > 
> > […]
> > 
> >> I've asked Martin for a digital copy of his RDBs and what he thinks
> >> the partition(s) should look like. I should also be told whether
> >> the disk is supposed to be solely Amiga OSs or not. I gather it's
> >> not.
> > 
> > Its all in the bug report. profile-binary.img should be it.
> > 
> > Its small so I attach it.
> > 
> > Meanwhile I try to get the currently supported maximum disk size out
> > from the OS 4 developers. Maybe JXFS is playing tricks that other
> > filesystems do not play or simply has a different implementation.
> > 
> > Thanks,
> 
> I've sent Jens a proposed fix changing these lines in amiga.c.
> ===8<--- was
>          struct PartitionBlock *pb;
>          int start_sect, nr_sects, blk, part, res = 0;
>          int blksize = 1;        /* Multiplier for disk block size */
> ===8<--- becomes changing line 338 into lines 338-339
>          struct PartitionBlock *pb;
>          sector_t start_sect, nr_sects;
>          int blk, part, res = 0;
>          int blksize = 1;        /* Multiplier for disk block size */
> ===8<---
> 
> (I'm working without proper tools at the moment or I'd make a real
> diff. Sorry.)
> 
> This fix may not be complete. The RDBs contain provisions for
> describing the physical disk block size and how many physical blocks
> are used in a file system's logical blocks. This MAY not work quite
> right large physical block sizes.

Way better:

dmesg:

Jun 19 21:19:09 merkaba kernel: [ 7891.819552] ata8: SATA link up 3.0 Gbps (SStatus 123 SControl 0)
Jun 19 21:19:09 merkaba kernel: [ 7891.821306] ata8.00: ATA-8: Hitachi HDS5C3020ALA632, ML6OA580, max UDMA/133
Jun 19 21:19:09 merkaba kernel: [ 7891.821315] ata8.00: 3907029168 sectors, multi 0: LBA48 NCQ (depth 31/32)
Jun 19 21:19:09 merkaba kernel: [ 7891.823252] ata8.00: configured for UDMA/100
Jun 19 21:19:09 merkaba kernel: [ 7891.823589] scsi 7:0:0:0: Direct-Access     ATA      Hitachi HDS5C302 ML6O PQ: 0 ANSI: 5
Jun 19 21:19:09 merkaba kernel: [ 7891.824126] sd 7:0:0:0: [sdb] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
Jun 19 21:19:09 merkaba kernel: [ 7891.824440] sd 7:0:0:0: [sdb] Write Protect is off
Jun 19 21:19:09 merkaba kernel: [ 7891.824452] sd 7:0:0:0: [sdb] Mode Sense: 00 3a 00 00
Jun 19 21:19:09 merkaba kernel: [ 7891.824531] sd 7:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Jun 19 21:19:09 merkaba kernel: [ 7891.843284]  sdb: RDSK (512) sdb1 (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)
(res 2 spb 4)
Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb] Attached SCSI disk


cat /proc/partitions:
major minor  #blocks  name
[…]
   8       16 1953514584 sdb
   8       17 1572864024 sdb1
   8       18  314572824 sdb2
   8       19   66076704 sdb3

1572864024 + 314572824 + 66076704 = 1953513552 which is below the complete
size of the disk.


Partition start analysis:

merkaba:~> file -sk /dev/sdb1
/dev/sdb1: sticky LVM2 PV (Linux Logical Volume Manager), UUID: ZXMECC-JAir-lX8Q-rLhS-W1cS-quwz-b3FXWp, size: 2000397877248
merkaba:~> file -sk /dev/sdb2
/dev/sdb2: sticky data
merkaba:~> file -sk /dev/sdb3
/dev/sdb3: sticky Amiga Inter FFS disk

merkaba:~> hd /dev/sdb2 | head
00000000  4a 58 46 04 11 1a 0f 0c  00 00 00 00 00 00 00 00  |JXF.............|
00000010  00 00 00 00 00 11 00 00  3e db 3d 54 40 00 00 00  |........>.=T@...|
00000020  00 00 00 00 00 00 00 00  00 00 01 77 00 10 80 00  |...........w....|
00000030  00 00 01 c2 00 10 e0 00  25 80 00 30 00 00 02 00  |........%..0....|
00000040  00 00 00 30 00 00 00 00  00 00 00 00 00 00 00 00  |...0............|
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000060  00 00 00 00 00 00 00 00  00 00 08 00 00 00 00 00  |................|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200  54 52 4f 4b ab ad b0 b1  00 00 00 01 00 00 00 00  |TROK............|

I don´t know the on-disk format for JXFS, but this could be it. JFX/04
pretty much looks like the DOS type for JXFS. Yes, thats it. JXF/04 is
the DOS type in use for JXFS as I verified by looking at a JFXS partition
with Media Toolbox.


amiga-fdisk looks the same as before:

merkaba:~> amiga-fdisk -l /dev/sdb
Disk /dev/sdb: 3 heads, 16 sectors, 81396441 cylinders, RDB: 0
Logical Cylinders from 43 to 81396440, 24576  bytes/Cylinder 

Device     Boot Mount   Begin      End     Size   Pri  BBlks    System
/dev/sdb1       *        43   65536043   1572864024     0      0  Linux native
/dev/sdb2       *    65536044   78643244   314572824     0      0  [unknown]
/dev/sdb3       *    78643245   81396440   66076704     0      0  Amiga FFS Int.

But obviously its right anyway: It seems to display the size in blocks,
not in cylinders.


Media Toolbox says:

LVM aka sdc1: 1500 GByte, 43 to 65536043 cyl, size 65536001 Zylinder
BAK aka sdc2: 300 GByte, 65536044 to 78643244 cyl, size 13107201 Zylinder
TAUSCH2 aka sdc3: 63,016 GByte, 78643245 to  81396440 cyl, didn´t note the size
here, but as far as I remember was okay as well

with

Blocks per cylinder: 48
Cylinders: 81396441

So thats:

65536001 * 48 / 2 = 1572864024

13107201 * 48 / 2 = 314572824

( 81396440 - 78643245 ) * 48 / 2 = 66076680

(I verified from a windowshot I made that 81396440 - 78643245 = 2753195
is indeed displayed by Media Toolbox as size of the partition)



So this is looking pretty good.

Many thanks, Joanne for your detective work and the fix.

Tested with:

martin@merkaba:~> cat /proc/version
Linux version 3.5.0-rc3-fix-bug-43511+ (martin@merkaba) (gcc version 4.7.0 (Debian 4.7.1-1) ) #1 SMP Tue Jun 19 14:31:56 CEST 2012


Tested-By: Martin Steigerwald <martin@lichtvoll.de>
Reviewed-By: Martin Steigerwald <martin@lichtvoll.de>


Patch from Joanne in diff format:

martin@merkaba:~/Computer/Merkaba/Kernel/linux-2.6> git diff HEAD~ | cat
diff --git a/block/partitions/amiga.c b/block/partitions/amiga.c
index 70cbf44..42d36f9 100644
--- a/block/partitions/amiga.c
+++ b/block/partitions/amiga.c
@@ -29,7 +29,8 @@ int amiga_partition(struct parsed_partitions *state)
        unsigned char *data;
        struct RigidDiskBlock *rdb;
        struct PartitionBlock *pb;
-       int start_sect, nr_sects, blk, part, res = 0;
+       sector_t start_sect, nr_sects;
+       int blk, part, res = 0;
        int blksize = 1;        /* Multiplier for disk block size */
        int slot = 1;
        char b[BDEVNAME_SIZE];


Jens, please take that patch. If you want I can prepare it further with above
testing report and some explaination as changelog and a From Joanne Dow
line ;). But it might be easier if you just take it as is and attribute it
to her. Feel free to use as much of my test report as you want for in
commit message. But I will copy this into the bug report anyway.

If you taken it, please tell me the commit id. I think this should go to
stable kernel since its a potential data loss issue and an isolated
overflow fix.


Only LVM is unhappy now, but thats to be expected since /dev/sdb1
that it used is now smaller than the whole disk as it was before with
the overflowing and truncating Amiga RDB code without Joanne´s fix.

merkaba:~> vgscan
  Reading all physical volumes.  This may take a while...
  /dev/sdb1: lseek 2000396746752 failed: Das Argument ist ungültig
  /dev/sdb1: lseek 2000396746752 failed: Das Argument ist ungültig
  /dev/sdb1: lseek 2000396746752 failed: Das Argument ist ungültig
  WARNING: Inconsistent metadata found for VG steigerwald - updating to use version 7
  /dev/sdb1: lseek 2000396746752 failed: Das Argument ist ungültig
  Automatic metadata correction failed
  Recovery of volume group "steigerwald" failed.
  Found volume group "merkaba" using metadata type lvm2


So I will now format the disk as GPT and use it only for Linux for now,
as I now have a different backup disk for the Amiga anyway.

So I have to recreate backups once again. Oh well, I possibly could
still boot an older kernel to access the LVM again.

Thanks,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1

[Martin Steigerwald]

Am Sonntag, 17. Juni 2012 schrieb Geert Uytterhoeven:
> On Sun, Jun 17, 2012 at 2:58 PM, Martin Steigerwald 
> <Martin@lichtvoll.de> wrote:
> > Am Sonntag, 17. Juni 2012 schrieb jdow:
> > | JXFS 64 bit file system
> > | 
> > | With AmigaOS 4.x a new file system has been introduced called JXFS.
> > | It is a totally new 64 bit file system that supports partitions up
> > | to 16 TB in size. It is a modern journalling file system, which
> > | means that it reduces data loss if data writes to the disk are
> > | interrupted. It is the fastest and most reliable file system ever
> > | created for AmigaOS.
> > 
> > http://www.amigaos.net/content/1/features
> > 
> > Well I asked AmigaOS 4 developers about this issue as well. Lets see
> > what they say about 2 TB limits.
> 
> 16 TB = 2 TB * 8. Perhaps they increased the block size from 512 to
> 4096?
> 
> block/partitions/amiga.c reads the block size from
> RigidDiskBlock.rdb_BlockBytes,
> but after conversion to 512-byte blocks, all further calculations are
> done on "int", so it will overflow for disks larger than 2 TiB.

Meanwhile I got a response from a AmigaOS 4 developer.

I documented the stuff as I understood it in the AmigaOS wiki:

| Disk size
|
| The RDB has a quite high limit on the maximum device size, but note that 
| currently each filesystem interprets the partition layout by itself.

| The raw, theoretical limit on the maximum device capacity is about 2^105 
| bytes:
|
| 32 bit rdb_Cylinders * 32 bit rdb_Heads * 32 bit rdb_Sectors * 512 
| bytes/sector for the HD size in struct RigidDiskBlock
|
| It's even much more if the sector size (rdb_BlockBytes and de_SizeBlock) 
| is larger than 512 bytes, but AmigaOS 4.1 doesn't support anything but 
| 512 bytes/sector HDs yet.
|
| Partition size
|
| For the partitions the maximum size is:
| 32 bit (de_HighCyl + 1 - de_LowCyl) (to get the partition size) * 32 bit 
| de_Surfaces * 32 bit de_SectorsPerTrack * 512 bytes/sector in struct 
| DosEnvec (=pb_Environment[]) in struct PartitionBlock.
|
| That's from the physical drive part, the actual disk size limit for the 
| partitions may be much smaller depending on the partitioning software, 
| if it's only using the logical sizes instead, which is likely the case, 
| it's only 8 ZiB with 512 bytes/sector: 32 bit rdb_HiCylinder * 32 bit 
| rdb_CylBlocks * 512 bytes/sector = 2^73 bytes. For using the logical 
| sizes using simple uint64 calculations (with some overflow checks) should 
| be enough, for more a math library with support for larger integers 
| needs to be used which probably no partitioning software does.
|
| But note: Nothing in struct RigiDiskBlock is used by the file systems for 
| mounting the partitions, they only get the information from the struct 
| PartitionBlock blocks, it's only a problem for the partitioning software 
| creating the partitions correctly - as soon as there are HDs larger than 
| 8 ZB while still using 512 bytes/sector if that ever happens.

http://wiki.amigaos.net/index.php/RDB

Please note that the documentation there might be updated or corrected in 
the future. But thats the current state.

> Note that in your profile-binary.img, the field is 0x200, i.e. 512
> bytes per block,
> so I'll have to get a deeper look into your RDB first...

I am a bit overwhelmed by Joanne´s analysis. I didn´t yet take time to 
completely read and try to understand it. I first wanted to get this 
information out.

Thanks,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1

[jdow]

On 2012/06/18 13:39, Martin Steigerwald wrote:
> Am Sonntag, 17. Juni 2012 schrieb Geert Uytterhoeven:
>> On Sun, Jun 17, 2012 at 2:58 PM, Martin Steigerwald
>> <Martin@lichtvoll.de> wrote:
>>> Am Sonntag, 17. Juni 2012 schrieb jdow:
>>> | JXFS 64 bit file system
>>> |
>>> | With AmigaOS 4.x a new file system has been introduced called JXFS.
>>> | It is a totally new 64 bit file system that supports partitions up
>>> | to 16 TB in size. It is a modern journalling file system, which
>>> | means that it reduces data loss if data writes to the disk are
>>> | interrupted. It is the fastest and most reliable file system ever
>>> | created for AmigaOS.
>>>
>>> http://www.amigaos.net/content/1/features
>>>
>>> Well I asked AmigaOS 4 developers about this issue as well. Lets see
>>> what they say about 2 TB limits.
>>
>> 16 TB = 2 TB * 8. Perhaps they increased the block size from 512 to
>> 4096?
>>
>> block/partitions/amiga.c reads the block size from
>> RigidDiskBlock.rdb_BlockBytes,
>> but after conversion to 512-byte blocks, all further calculations are
>> done on "int", so it will overflow for disks larger than 2 TiB.
>
> Meanwhile I got a response from a AmigaOS 4 developer.
>
> I documented the stuff as I understood it in the AmigaOS wiki:
>
> | Disk size
> |
> | The RDB has a quite high limit on the maximum device size, but note that
> | currently each filesystem interprets the partition layout by itself.
>
> | The raw, theoretical limit on the maximum device capacity is about 2^105
> | bytes:
> |
> | 32 bit rdb_Cylinders * 32 bit rdb_Heads * 32 bit rdb_Sectors * 512
> | bytes/sector for the HD size in struct RigidDiskBlock
> |
> | It's even much more if the sector size (rdb_BlockBytes and de_SizeBlock)
> | is larger than 512 bytes, but AmigaOS 4.1 doesn't support anything but
> | 512 bytes/sector HDs yet.
> |
> | Partition size
> |
> | For the partitions the maximum size is:
> | 32 bit (de_HighCyl + 1 - de_LowCyl) (to get the partition size) * 32 bit
> | de_Surfaces * 32 bit de_SectorsPerTrack * 512 bytes/sector in struct
> | DosEnvec (=pb_Environment[]) in struct PartitionBlock.
> |
> | That's from the physical drive part, the actual disk size limit for the
> | partitions may be much smaller depending on the partitioning software,
> | if it's only using the logical sizes instead, which is likely the case,
> | it's only 8 ZiB with 512 bytes/sector: 32 bit rdb_HiCylinder * 32 bit
> | rdb_CylBlocks * 512 bytes/sector = 2^73 bytes. For using the logical
> | sizes using simple uint64 calculations (with some overflow checks) should
> | be enough, for more a math library with support for larger integers
> | needs to be used which probably no partitioning software does.
> |
> | But note: Nothing in struct RigiDiskBlock is used by the file systems for
> | mounting the partitions, they only get the information from the struct
> | PartitionBlock blocks, it's only a problem for the partitioning software
> | creating the partitions correctly - as soon as there are HDs larger than
> | 8 ZB while still using 512 bytes/sector if that ever happens.
>
> http://wiki.amigaos.net/index.php/RDB

He's in the right ballpark but he missed something in the RDISK block:
     __u32   rdb_LoCylinder;	// 88	0x2b		Ayup - checks
     __u32   rdb_HiCylinder;	// 8C   0x04da02d8	Whole disk used.
     __u32   rdb_CylBlocks;	// 90	0x30		checks

So he has two 32 bit unsigned integers not three to multiply together.
If he ignores that detail he can, indeed, go out to three ULONGS for the
total disk logical block count. Then he has two more ULONGs for physical
block size and number of physical blocks per filesystem blocks. So the
size could, in theory, go past 2^100 bytes.

The filesystem is still likely limited to 2 TB or so unless there is a
64 bit version now.

> Please note that the documentation there might be updated or corrected in
> the future. But thats the current state.
>
>> Note that in your profile-binary.img, the field is 0x200, i.e. 512
>> bytes per block,
>> so I'll have to get a deeper look into your RDB first...
>
> I am a bit overwhelmed by Joanne´s analysis. I didn´t yet take time to
> completely read and try to understand it. I first wanted to get this
> information out.

When you can please recompile a kernel with that one change to it. It
MIGHT turn the trick. It changes an int to a sector_t (unsigned long long)
so the math does not go funkity going into the "put_partition" function,
which has disk blocks as sector_t values. (I think, but can't be sure,
that the put_partition call used int when I "committed" that sin way back
when. I've not yet compared that code with the code I submitted many years
ago to see if all the right stuff is still there.)

{^_^}