'holey files' not holey enough.

'holey files' not holey enough.

[Adam]

Is this feature or bug?

First a test on 'stable' 2.2.x kernel:

	eax /tmp % uname -a
	Linux eax 2.2.17pre15 #1 Sat Aug 5 14:31:19 EDT 2000 i586 unknown

	eax /tmp % dd if=/dev/zero of=holed.file bs=1000 seek=5000 count=1000
	1000+0 records in
	1000+0 records out

	eax /tmp % ls -l holed.file 
	-rw-rw-r--    1 adam     adam      6000000 Nov 29 08:57 holed.file

	eax /tmp % du -sh holed.file 
	983k	holed.file

Above holey file is as expected aproximately 1mb. 
However, when I try on 'development' 2.4.x kernel

	[adam@pepsi /tmp]$ uname -a
	Linux pepsi 2.4.0-test7-packet #24 SMP Fri Sep 8 20:26:35 EDT 2000 i686

	[adam@pepsi /tmp]$  dd if=/dev/zero of=holed.file bs=1000 seek=5000 count=1000
	1000+0 records in
	1000+0 records out

	[adam@pepsi /tmp]$ ls -l holed.file 
	-rw-rw-r--    1 adam     adam      6000000 Nov 29 08:52 holed.file

	[adam@pepsi /tmp]$ du -sh holed.file 
	1.9M    holed.file

The holey file is twice as big, 

at almost 2mb instead of expected 1mb.

-- 
Adam
http://www.eax.com	The Supreme Headquarters of the 32 bit registers

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: 'holey files' not holey enough.

[Tigran Aivazian]

On Wed, 29 Nov 2000, Adam wrote:
> However, when I try on 'development' 2.4.x kernel
> 
> 	[adam@pepsi /tmp]$ uname -a
> 	Linux pepsi 2.4.0-test7-packet #24 SMP Fri Sep 8 20:26:35 EDT 2000 i686
> 
> 	[adam@pepsi /tmp]$  dd if=/dev/zero of=holed.file bs=1000 seek=5000 count=1000
> 	1000+0 records in
> 	1000+0 records out
> 
> 	[adam@pepsi /tmp]$ ls -l holed.file 
> 	-rw-rw-r--    1 adam     adam      6000000 Nov 29 08:52 holed.file
> 
> 	[adam@pepsi /tmp]$ du -sh holed.file 
> 	1.9M    holed.file
> 

what filesystem type? on ext2 filesystem on 2.4.0-test12-pre3 I get
expected result:

# dd if=/dev/zero of=hole bs=1000 seek=5000 count=1000
1000+0 records in
1000+0 records out
# l hole
-rw-r--r--    1 root     root      6000000 Nov 29 13:06 hole
# du -sh hole
988k    hole
# u
Linux penguin 2.4.0-test12 #1 Wed Nov 29 09:08:13 GMT 2000 i686 unknown

Regards,
Tigran

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: 'holey files' not holey enough.

[Adam]

> > 	[adam@pepsi /tmp]$ uname -a
> > 	Linux pepsi 2.4.0-test7-packet #24 SMP Fri Sep 8 20:26:35 EDT 2000 i686
> > 
> > 	[adam@pepsi /tmp]$  dd if=/dev/zero of=holed.file bs=1000 seek=5000 count=1000
> > 	[adam@pepsi /tmp]$ ls -l holed.file 
> > 	-rw-rw-r--    1 adam     adam      6000000 Nov 29 08:52 holed.file
> > 	[adam@pepsi /tmp]$ du -sh holed.file 
> > 	1.9M    holed.file

> what filesystem type? on ext2 filesystem on 2.4.0-test12-pre3 I get
> expected result:
> 
> # dd if=/dev/zero of=hole bs=1000 seek=5000 count=1000
> 1000+0 records in
> 1000+0 records out
> # l hole
> -rw-r--r--    1 root     root      6000000 Nov 29 13:06 hole
> # du -sh hole
> 988k    hole
> # u
> Linux penguin 2.4.0-test12 #1 Wed Nov 29 09:08:13 GMT 2000 i686 unknown

It is ext2fs with 4kb blocks.

[root@pepsi /tmp]# rpm -q -f /usr/bin/du
fileutils-4.0-21

[root@pepsi /tmp]# ldd /usr/bin/du
        libc.so.6 => /lib/libc.so.6 (0x4001f000)
        /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)

[root@pepsi /tmp]# rpm -q -f /lib/libc.so.6 
glibc-2.1.3-21

-- 
Adam
http://www.eax.com	The Supreme Headquarters of the 32 bit registers

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: 'holey files' not holey enough.

[Martin Josefsson]

On Wed, 29 Nov 2000, Adam wrote:

> On Wed, 29 Nov 2000, Tigran Aivazian wrote:
> 
> > > 	[adam@pepsi /tmp]$  dd if=/dev/zero of=holed.file bs=1000 seek=5000 count=1000
> > > 	[adam@pepsi /tmp]$ ls -l holed.file 
> > > 	-rw-rw-r--    1 adam     adam      6000000 Nov 29 08:52 holed.file
> > > 	[adam@pepsi /tmp]$ du -sh holed.file 
> > > 	1.9M    holed.file
> > 
> > what filesystem type? on ext2 filesystem on 2.4.0-test12-pre3 I get
> > expected result:
> 
> More datapoints. I have asked around, and I have two users of
> 2.4.0-test10. One is getting expected 1mb, other is getting (just like me)
> 1.9mb.
> 
> So far I don't see pattern, here. It does not seems to depend on block
> size, nor packet writing patches.

I don't use ext2, I use reiserfs and it works fine here (test10)

tux:~$dd if=/dev/zero of=holed.file bs=1000 seek=5000 count=1000
1000+0 records in
1000+0 records out
tux:~$ls -l holed.file 
-rw-r--r--    1 gandalf  gandalf   6000000 Nov 29 20:55 holed.file
tux:~$du -sh holed.file 
980k    holed.file

tested on a router (with test10 and ext2 and reiserfs) and it works fine
on both filsystems

tested on another router with reiserfs and ext2 running test11
works fine there too on both filesystems

works fine on test8 with both ext2 and reiserfs

(1k blocksize on ext2 and 4k blocksize on reiserfs)

/Martin

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: 'holey files' not holey enough.

[Adam]

On Wed, 29 Nov 2000, Tigran Aivazian wrote:

> > 	[adam@pepsi /tmp]$  dd if=/dev/zero of=holed.file bs=1000 seek=5000 count=1000
> > 	[adam@pepsi /tmp]$ ls -l holed.file 
> > 	-rw-rw-r--    1 adam     adam      6000000 Nov 29 08:52 holed.file
> > 	[adam@pepsi /tmp]$ du -sh holed.file 
> > 	1.9M    holed.file
> 
> what filesystem type? on ext2 filesystem on 2.4.0-test12-pre3 I get
> expected result:

More datapoints. I have asked around, and I have two users of
2.4.0-test10. One is getting expected 1mb, other is getting (just like me)
1.9mb.

So far I don't see pattern, here. It does not seems to depend on block
size, nor packet writing patches.

-- 
Adam
http://www.eax.com      The Supreme Headquarters of the 32 bit registers


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: 'holey files' not holey enough.

[Marc Mutz]

kernel 2.2.17, '/' being a 1k blocksize ext2fs:

root@adam:/ > dd if=/dev/zero of=holed.file bs=1000 seek=5000 count=1000
1000+0 records in
1000+0 records out
root@adam:/ > ls -l holed.file
-rw-r--r--   1 root     root      6000000 Nov 29 23:33 holed.file
root@adam:/ > du -sh holed.file
5.7M    holed.file

Now that seems funny.

Marc

-- 
Marc Mutz <Marc@Mutz.com>     http://EncryptionHOWTO.sourceforge.net/
University of Bielefeld, Dep. of Mathematics / Dep. of Physics

PGP-keyID's:   0xd46ce9ab (RSA), 0x7ae55b9e (DSS/DH)

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: 'holey files' not holey enough.

[safemode]

On Wed, 29 Nov 2000 17:36:12 Marc Mutz wrote:
> kernel 2.2.17, '/' being a 1k blocksize ext2fs:
> 
> root@adam:/ > dd if=/dev/zero of=holed.file bs=1000 seek=5000 count=1000
> 1000+0 records in
> 1000+0 records out
> root@adam:/ > ls -l holed.file
> -rw-r--r--   1 root     root      6000000 Nov 29 23:33 holed.file
> root@adam:/ > du -sh holed.file
> 5.7M    holed.file
> 
> Now that seems funny.

why is 5.7MB funny with 6000000 bytes ?   or are you talking about
something else?  Don't forget that BYTES and thus MEGABYTES are found by
powers of 2.  They do not function like the decimal "bits".  quick way to
do the conversion is divide bytes by 1024 to get Kilobytes and then again
by 1024 to get Megabytes.  This is how it should be displayed.   Or am i
missing the point of your comment?


> Marc
> 
> -- 
> Marc Mutz <Marc@Mutz.com>     http://EncryptionHOWTO.sourceforge.net/
> University of Bielefeld, Dep. of Mathematics / Dep. of Physics
> 
> PGP-keyID's:   0xd46ce9ab (RSA), 0x7ae55b9e (DSS/DH)
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel"
> in
> the body of a message to majordomo@vger.kernel.org
> Please read the FAQ at http://www.tux.org/lkml/
> 

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: 'holey files' not holey enough.

[Andreas Dilger]

Marc Mutz writes:
> kernel 2.2.17, '/' being a 1k blocksize ext2fs:
> 
> root@adam:/ > dd if=/dev/zero of=holed.file bs=1000 seek=5000 count=1000
> 1000+0 records in
> 1000+0 records out
> root@adam:/ > ls -l holed.file
> -rw-r--r--   1 root     root      6000000 Nov 29 23:33 holed.file
> root@adam:/ > du -sh holed.file
> 5.7M    holed.file

Strangely, I have 2.2.17 (TurboLinux patched), on a 1k filesystem and
I have no problems.  I have 1k, 2k, and 4k ext2 fs, all OK.

What people who have the problem should be doing is:
> ls -li holed.file        # find inode number
10732 -rw-r--r--    1 root     root      6000000 Nov 29 16:17 holed.file
> du -sk holed.file        # see what "stat" thinks
983k    holed.file
> debugfs /dev/XXX
debugfs> stats           # find out ext2 block size
...
Block size = 1024, fragment size = 1024
...
debugfs> stat <10732>  # (with < and >)
Inode: 10732   Type: regular    Mode:  0644   Flags: 0x0   Generation:
4048594821
User:     0   Group:     0   Size: 6000000
File ACL: 0    Directory ACL: 0
Links: 1   Blockcount: 1966
                       ^^^^ these are 512-byte blocks, so / 2 for ~kB
		            they include indirect blocks and such
Fragment:  Address: 0    Number: 0    Size: 0
ctime: 0x3a258e82 -- Wed Nov 29 16:17:22 2000
atime: 0x3a258e82 -- Wed Nov 29 16:17:22 2000
mtime: 0x3a258e82 -- Wed Nov 29 16:17:22 2000
BLOCKS:
47512 47513 47514 47515 47516 47517 47518 47519 47520 ... 48723 48724
TOTAL: 983
       ^^^ these are ext2fs sized blocks, not necessarily kB

If what debugfs says doesn't match du, then it is du/libc/stat that is
broken.  If debugfs says the file actually has 6000000 bytes of data,
then it is the filesystem that is broken.

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: 'holey files' not holey enough.

[Adam]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1915 bytes --]

On Wed, 29 Nov 2000, Andreas Dilger wrote:

> What people who have the problem should be doing is:
> [desc snipped]
> > ls -li holed.file        # find inode number
> 10732 -rw-r--r--    1 root     root      6000000 Nov 29 16:17 holed.file
> > du -sk holed.file        # see what "stat" thinks
> 983k    holed.file
> > debugfs /dev/XXX
> debugfs> stats           # find out ext2 block size
> ...
> Block size = 1024, fragment size = 1024
> ...
> debugfs> stat <10732>  # (with < and >)
> Inode: 10732   Type: regular    Mode:  0644   Flags: 0x0   Generation:
> 4048594821
> User:     0   Group:     0   Size: 6000000
> File ACL: 0    Directory ACL: 0
> Links: 1   Blockcount: 1966
>                        ^^^^ these are 512-byte blocks, so / 2 for ~kB
> 		            they include indirect blocks and such
> Fragment:  Address: 0    Number: 0    Size: 0
> ctime: 0x3a258e82 -- Wed Nov 29 16:17:22 2000
> atime: 0x3a258e82 -- Wed Nov 29 16:17:22 2000
> mtime: 0x3a258e82 -- Wed Nov 29 16:17:22 2000
> BLOCKS:
> 47512 47513 47514 47515 47516 47517 47518 47519 47520 ... 48723 48724
> TOTAL: 983
>        ^^^ these are ext2fs sized blocks, not necessarily kB
> 
> If what debugfs says doesn't match du, then it is du/libc/stat that is
> broken.  If debugfs says the file actually has 6000000 bytes of data,
> then it is the filesystem that is broken.

I just did what suggested, and it seems that DU reports correct values,
I have attached 'sript' log of the above example on my filesystem.
Here are some highlights:

[adam@pepsi /tmp]$ ls -lis holed.file
3085069 5872 -rw-rw-r--    1 adam     adam      6000000 Nov 30 09:11 holed.file

Block size = 4096, fragment size = 4096
Links: 1   Blockcount: 11744
TOTAL: 1468

so it seems DU reports correct values as :
	1966/2 =983
	11744/2=5872
and
	4096*1468=6012928
	1024*983 =1006592



-- 
Adam
http://www.eax.com      The Supreme Headquarters of the 32 bit registers


[-- Attachment #2: Type: TEXT/PLAIN, Size: 2143 bytes --]

Script started on Thu Nov 30 09:11:40 2000
[adam@pepsi /tmp]$ dd if=/dev/zero of=holed.file bs=1000 seek=5000 count=1000
1000+0 records in
1000+0 records out
[adam@pepsi /tmp]$ ls -lis holed.file
3085069 5872 -rw-rw-r--    1 adam     adam      6000000 Nov 30 09:11 holed.file
[adam@pepsi /tmp]$ du -sh holed.file
5.7M	holed.file
[adam@pepsi /tmp]$ df .
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/hde3             25474728  16986488   7173180  70% /
[adam@pepsi /tmp]$ su
Password: 
root@pepsi /tmp]# /usr/sbin/debugf /dev/hde3
debugfs 1.19, 13-Jul-2000 for EXT2 FS 0.5b, 95/08/09
debugfs:  stats
Filesystem is read-only
Volume name = (none)
Last mounted directory = (none)
Filesystem UUID = dacab8b6-cbae-11d3-867d-00200084fdb4
Filesystem features:(none)
Last mount time = Thu Nov 23 23:18:05 2000
Last write time = Thu Nov 30 09:12:56 2000
Mount counts = 1 (maximal = 20)
Filesystem OS type = Linux
Superblock size = 1024
Block size = 4096, fragment size = 4096
Inode size = 128
6579936 inodes, 6014036 free
6575310 blocks, 2122060 free, 328765 reserved, first block = 0
32768 blocks per group
32768 fragments per group
32736 inodes per group
201 groups (2 descriptors blocks)
 Group  0: block bitmap at 3, inode bitmap at 4, inode table at 5
           0 free blocks, 30118 free inodes, 106 used directories
 [....]
 Group 200: block bitmap at 6553603, inode bitmap at 6553604, inode table at 6553605
           13151 free blocks, 29887 free inodes, 334 used directories
debugfs:  stat <3085069>
Inode: 3085069   Type: regular    Mode:  0664   Flags: 0x0   Generation: 6660658
User:   500   Group:   500   Size: 6000000
File ACL: 0    Directory ACL: 0
Links: 1   Blockcount: 11744
Fragment:  Address: 0    Number: 0    Size: 0
ctime: 0x3a266023 -- Thu Nov 30 09:11:47 2000
atime: 0x3a243c51 -- Tue Nov 28 18:14:25 2000
mtime: 0x3a266023 -- Thu Nov 30 09:11:47 2000
BLOCKS:
213226 [....] 167493 
TOTAL: 1468
debugfs:   quit
adam@pepsi: /tmp\a[root@pepsi /tmp]# exit
exit
adam@pepsi: /tmp\a[adam@pepsi /tmp]$ exit
exit
Script done on Thu Nov 30 09:16:27 2000

Re: 'holey files' not holey enough.

[Andreas Dilger]

Adam writes:
> I just did what suggested, and it seems that DU reports correct values,
> I have attached 'sript' log of the above example on my filesystem.
> Here are some highlights:
> 
> [adam@pepsi /tmp]$ ls -lis holed.file
> 3085069 5872 -rw-rw-r--    1 adam     adam      6000000 Nov 30 09:11 holed.file
> 
> Block size = 4096, fragment size = 4096
> Links: 1   Blockcount: 11744
> TOTAL: 1468
> 
> so it seems DU reports correct values as :
> 	11744/2=5872
> and
> 	4096*1468=6012928

I guess the next thing to check is if your "dd" is actually seeking, or
just pretending to...  Maybe an strace of the "dd" call will tell us if
it is screwing with our minds.  Also, if you could make a scratch ext2
filesystem with 1k blocks, and see if it does the same thing.  Even
better would be to try a different kernel to see if it affects this.

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: 'holey files' not holey enough.

[Marc Mutz]

Andreas Dilger wrote:
> 
> Marc Mutz writes:
> > kernel 2.2.17, '/' being a 1k blocksize ext2fs:
> >
> > root@adam:/ > dd if=/dev/zero of=holed.file bs=1000 seek=5000 count=1000
> > 1000+0 records in
> > 1000+0 records out
> > root@adam:/ > ls -l holed.file
> > -rw-r--r--   1 root     root      6000000 Nov 29 23:33 holed.file
> > root@adam:/ > du -sh holed.file
> > 5.7M    holed.file
> 
> Strangely, I have 2.2.17 (TurboLinux patched), on a 1k filesystem and
> I have no problems.  I have 1k, 2k, and 4k ext2 fs, all OK.
> 
> What people who have the problem should be doing is:
> > ls -li holed.file        # find inode number
> 10732 -rw-r--r--    1 root     root      6000000 Nov 29 16:17 holed.file
> > du -sk holed.file        # see what "stat" thinks
> 983k    holed.file

dito

> > debugfs /dev/XXX
> debugfs> stats           # find out ext2 block size
> ...
> Block size = 1024, fragment size = 1024
> ...

dito

> Links: 1   Blockcount: 1966

dito

> TOTAL: 983

dito

> If what debugfs says doesn't match du, then it is du/libc/stat that is
> broken.  If debugfs says the file actually has 6000000 bytes of data,
> then it is the filesystem that is broken.
> 
$ du --version
du (GNU fileutils) 3.16
$ ls -l /lib/libc*
lrwxrwxrwx   1 root     root           13 May  8  1999 /lib/libc.so.4 \
-> libc.so.4.7.6
-rwxr-xr-x   1 root     root       634880 Apr 29  1996
/lib/libc.so.4.7.6
-rwxr-xr-x   1 root     root      2478585 Dec 14  1998 /lib/libc.so.6
-rwxr-xr-x   1 root     root        85699 Dec 14  1998
/lib/libcrypt.so.1
$ ldd /usr/bin/du
        libc.so.6 => /lib/libc.so.6 (0x40009000)
        /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x80000000)

all from SuSE 6.0 (I know, it's old...)

Marc

-- 
Marc Mutz <Marc@Mutz.com>     http://EncryptionHOWTO.sourceforge.net/
University of Bielefeld, Dep. of Mathematics / Dep. of Physics

PGP-keyID's:   0xd46ce9ab (RSA), 0x7ae55b9e (DSS/DH)


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: 'holey files' not holey enough.

[Jaakko Hyvätti]

  Hi!

On Sat, 2 Dec 2000, Adam wrote:
> It seems you are right. If I remove the file first, then it will show
> correct amount.

  (For the list: In private mail it was discovered that this behaviour was
caused by the file 'holed.file' not being empty before the dd command.)

  dd behaves here correctly.  It does not append to file, but it just
opens the file for writing.  It does not remove or truncate it first.

  If you have executed this command:

dd if=/dev/zero of=holed.file bs=1000 count=1000

You have a simple file with zeroes, like this:

'000000000'

Then with this command dd opens the file for write but does not destroy
its contents - it is not supposed to do that.  (You can alter files with
dd, you can overwrite single bytes if you like.)

 dd if=/dev/zero of=holed.file bs=1000 seek=5000 count=1000

After opening the file dd seeks to what you specify, and at the same time
extends the file by seeking and Linux converts this to holes:

'000000000-----------------------------------------'

  And after seeking it writes the new block of zeroes:

'000000000-----------------------------------------000000000'

..and you have 2MB of stuff there instead of 1M!

Yours,
Jaakko

-- 
Weather Service Finland Ltd                          Jaakko.Hyvatti@weather.fi
Pursimiehenkatu 29-31 B, FIN-00150 Helsinki, Finland     http://www.weather.fi

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/