Re: Custom driver FS brokenness at 4GB?

[Jan Kara <jack@suse.cz>]

On Wed 27-05-15 09:56:29, Rob Harris wrote:
> Greetings. I have an odd issue and need some ideas of where to go
> next -- I'm out of hair to rip out.
> 
> I'm writing a custom block device driver talking to some custom RAID
> hardware (>32TB) using DMA scatter-gather, with no partitions and am
> using make_request() to service all the BIO requests to simplify
> debugging. I have the driver working to the point where using DD
> against the block device seems to work fine (I'm setting
> iflag|oflag=direct to ensure it's writing to the disk). I also have
> the blk_queue set to only request a single 4k I/O per BIO (again to
> simplify debugging for now.) Also, again to debug, I have a mutex
> wrapping the entire make_request call to ensure that only a single
> request is being serviced at a time. So, this should be as "simple"
> as I can make the environment to debug this problem.
> 
> Once the driver is loaded, when I try to create a file system (ext4
> but the same thing happens with xfs) it seems like there is some
> corruption occurring, but only when I set the sector size of the
> block device over 4GB. For instance, when I set the size to 4G, I
> can mkfs.ext4, but after 2 or 3 mount/umounts the FS refuses to
> mount anymore and the kernel log complains that the journal is
> missing. This was discovered running this loop...
  Hard to tell exactly but with 4GB being 32-bit limit, I would first look
for some int / unsigned int number overflow. You could possibly better
debug this when writing some pattern via DD that is different for each
block to verify that each block indeed lands in the expected location...

								Honza
> 
> #!/bin/sh
> COUNT=4032
> 
> while [ 1 ] ; do
> 
> figlet ${COUNT}
> 
> ( umount /mnt ; rmmod smc ) || true
> modprobe smc capacity_in_mb=${COUNT} debug=1
> mkfs.ext4 -m 0 /dev/smcd
> 
> mount /dev/smcd /mnt
> cp count_512m.dat /mnt/test
> umount /mnt
> mount /dev/smcd /mnt
> umount /mnt
> mount /dev/smcd /mnt
> cmp count_512m.dat /mnt/test
> umount /mnt
> mount /dev/smcd /mnt # ***
> sync
> umount /mnt
> mount /dev/smcd /mnt
> sleep 1
> umount /mnt
> 
> COUNT=$(( COUNT + 64 ))
> sleep 1
> 
> done
> 
> Sometimes I'll get in the kernel log:
> May 27 09:39:01 febtober kernel: [64547.304695] EXT4-fs (smcd):
> ext4_check_descriptors: Checksum for group 0 failed (7009!=0)
> May 27 09:39:01 febtober kernel: [64547.305744] EXT4-fs (smcd):
> group descriptors corrupted!
> 
> Others I'll get:
> May 27 09:46:49 ryftone-smcdrv kernel: [65014.342850] EXT4-fs
> (smcd): no journal found
> 
> 
> I've seen this loop fail as early as COUNT=4096, but as late as
> COUNT=4220; removing the sync changes the behavior.
> When it fails, it usually does so on the 3rd mount (***).
> FYI, I effectively call: set_capacity( disk, capacity_in_mb * 2048
> ); ( 2048 * 512b (kernel sector) = 1M )
> 
> Another example: if I set the sector count of the disk to 16G, I can
> run mkfs.ext4 but the first mount fails and I see May 27 09:07:27
> febtober kernel: [62653.269387] EXT4-fs (smcd):
> ext4_check_descriptors: Block bitmap for group 0 not in group (block
> 4294967295)!
> 
> But, again, if I set the sector size < 4G, everything seems fine. I
> can currently DD read and write across that 4G boundary without
> issue -- it's ONLY the filesystem accesses. My gut is screaming
> there's 32/64 bit overflow condition somewhere but for the life of
> me I can't find it. Is there something I need to set to tell the
> block layer I have a 64-bit addressible device? set_capacity is
> always the number of LINUX KERNEL sectors (not what I set
> blk_queue_logical|physical_block_size to) correct?
> 
> I'm currently on 3.16.0 (Ubuntu 14.04.2 LTS) if it matters.
> 
> Any help/pointers would be greatly appreciated.
> 
> --Rob Harris
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR