Re: write 'O_DIRECT' file w/odd amount of data: desirable result?

From: "Linda A. Walsh" <lkml@tlinx.org>
To: Padraig Brady <PadraigBrady.com>
Cc: LKML <linux-kernel@vger.kernel.org>, xfs-oss <xfs@oss.sgi.com>
Subject: Re: write 'O_DIRECT' file w/odd amount of data: desirable result?
Date: Wed, 23 Feb 2011 10:04:30 -0800	[thread overview]
Message-ID: <4D654C2E.2000703@tlinx.org> (raw)
In-Reply-To: <4D64E2BB.7010000@draigBrady.com>

FWIW -- xfs-oss, included as 'last line' was of minor interest; known bug on
this kernel?:
Linux Ishtar 2.6.35.7-T610-Vanilla-1 #2 SMP PREEMPT Mon Oct 11 17:19:41 
PDT 2010 x86_64 x86_64 x86_64 GNU/Linux

Pádraig Brady wrote:
> On 23/02/11 04:30, Linda Walsh wrote:
>   
>> I understand, somewhat, what is happening. I have two different utils, 
>> 'dd' and mbuffer both of which have a 'direct' option to write to disk. 
>> mbuffer was from my distro with a direct added, which is
>>
>> I'm not sure if it's truncating the write to the lower bound of the 
>> sector size or the file-allocation-unit size but from a {dump|cat},
>> piped into {cat, dd mbuffer}, the output sizes are:
>> file              size       delta
>> -------------   ----------   ----
>> dumptest.cat    5776419696
>> dumptest.dd     5776343040   76656   
>> dumptest.mbuff  5368709120   407710576
>> - params:
>> dd of=dumptest.dd bs=512M oflag=direct
>> mbuffer -b 5 -s 512m -direct -f -o dumptest.mbuff
>> ----
>> I'm not aware of what either did, but no doubt neither expected an
>> error in the final write and didn't handle the results properly.
>> Vanilla kernel 2.6.35-7 x86_64 (SMP PREMPT)
>>     
> Note dd will turn off O_DIRECT for the last write
> if it's less than the block size.
> http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=commitdiff;h=5929322c
>   
------

FWIW, 'dd' is from 'coreutils-7.1-3.2.x86_64' (from the suse 11.2 release):

While I used dump (xfsdump to be precise) to produce my initial output
to mbuffer, it was the error message at the end which caught my attention.
Prior I had a tried a series of filters after the initial mem-to-mem
buffer performed by 'dd', then later 'mbuffer'.  The filters were
successively lower-io compress options over the years as disk and
network speeds rose and cpu-compression became the choke-point.

(xfsdump -b 512m )|(initially 'dd', later, 'mbuffer' )| \
         (su -f -m backup -c "$umask $um;${Compress:-} ${Compress_ops:-} \
                >${Dmpfile}${Compress_ext}" )
---
Eventually I wanted to get rid of the final filter step altogether and
have that 'buffer' statement after the 'dump' go direct to disk, then
later "--direct" to disk...

    It was adding the 'DIRECT' flag then that I noticed mbuffer's error.

    My first debug step was to go for a shorter dump file (the one that
failed on was over 3TB and took over 3h to reproduce).  Then I substituted
'cat' as that final filter and ended up with my 'testfile' I used for later
tests for 'mbuffer' and 'dd'.

NOTE:

I tried using the 'iflag=fullblock' as you recommend and it made the problem
'consistent' with the output of 'mbuffer', i.e. it transfered less data
and the truncation was consistent with a 512M divisor, indicating it was
'cat' default record output size that was causing the difference. 
If I use 'dd' to read the base file (no direct i/o) I get consistent results
with 'mbuffer' and 'dd':

Input: DumpTest.out: 5776419696

Output file sizes are as reported by 'dd', with 'test1' giving the closest
answer (short record line concatenated with ' & '):

test1> cat DumpTest.out |dd of=dumptest.dd-fb oflag=direct 
bs=512M          
dd: writing `dumptest.dd-fb': Invalid argument
0+7346 records in & 0+7345 records out
5776343040 bytes (5.8 GB) copied, 12.4361 s, 464 MB/s

test2> cat DumpTest.out |dd of=dumptest.dd+fb oflag=direct bs=512M 
iflag=fullblock
dd: writing `dumptest.dd+fb': Invalid argument
10+1 records in & 10+0 records out
5368709120 bytes (5.4 GB) copied, 12.581 s, 427 MB/s

test3> dd if=DumpTest.out bs=512M |dd of=dumptest2.dd+fb oflag=direct 
bs=512M iflag=fullblock               
10+1 records in & 10+1 records out
5776419696 bytes (5.8 GB) copieddd: writing `dumptest2.dd+fb', 11.6493 
s, 496 MB/s
: Invalid argument
10+1 records in & 10+0 records out
5368709120 bytes (5.4 GB) copied, 11.6513 s, 461 MB/s

test4> dd if=DumpTest.out bs=512M |dd of=dumptest2.dd-fb oflag=direct 
bs=512M
10+1 records in & 10+1 records out
dd: writing `dumptest2.dd-fb'5776419696 bytes (5.8 GB) copied, 11.4503 
s, 504 MB/s
: Invalid argument
10+1 records in & 10+0 records out
5368709120 bytes (5.4 GB) copied, 11.4503 s, 469 MB/s

---
I've tried significantly shorter files and NOT had this problem
(record size=64k, and 2 files one @ 57k and one at 64+57k). Both copied 
fine.
Something to do with large file buffers.

Of *SIGNIFICANT* note.  In trying to create an empty file of the size
used, from scratch, using 'xfs_mkfile', I got an error:

>  xfs_mkfile 5776419696 testfile
pwrite64: Invalid argument

---
I'm having problems generating new kernels (will ask in separate
message) so will have to fix those before moving ahead...

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs