Re: 2.6.0-test6-mm4 - oops in __aio_run_iocbs()

[Suparna Bhattacharya <suparna@in.ibm.com>]

On Thu, Oct 09, 2003 at 10:38:41AM -0700, Daniel McNeil wrote:
> On Thu, 2003-10-09 at 04:16, Suparna Bhattacharya wrote:
> > On Wed, Oct 08, 2003 at 04:18:15PM -0700, Daniel McNeil wrote:
> > > I'm been testing AIO on test6-mm4 using a ext3 file system and
> > > copying a 88MB file to an already existing preallocated file of 88MB.
> > > I been using my aiocp program to copy the file using i/o sizes of
> > > 1k to 512k and outstanding aio requests of between 1 and 64 using
> > > O_DIRECT, O_SYNC and O_DIRECT & O_SYNC.  Everything works as long
> > > as the file is pre-allocated.  When copying the file to a new file
> > > (O_CREAT|O_DIRECT), I get the following oops:
> > 
> > What are the i/o sizes and block sizes for which you get the oops ?
> > Is this only for large i/o sizes ?
> 
> 
> I've done more testing and it is a little confusing.
> I originally got the oops running a shell script which copied 4
> 88MB files one at a time to a sub-directory:
> 
> for i in fff ff1 ff2 ff3
> do
>         aiocp -b 128k -n 8 -f CREAT -f DIRECT $i junkdir/$i
> done
> sync
> 
> This script would always cause the oops and the machine would lock up.
> 
> I ran aiocp manually using different block sizes (4k-128k) to copy
> 1 file to a subdirectory.  I removed the file in the subdirectory
> afterward.  These tests completed without any problems or oopses.
> 
> > __aio_run_iocbs should have been called only for buffered i/o, 
> > so this sounds like an O_DIRECT fallback to buffered i/o.
> > Possibly after already submitting some blocks direct to BIO,
> > the i/o completion path for which ends up calling aio_complete
> > releasing the iocb. That could explain the use-after-free situation
> > you see.
> 
> mm4 has my extra iocb ref count for retries patch.  So the iocb should
> not be being freed by aio_complete.  The stack trace looks like the
> fault is on the ctx or ctx->runlist.

The race I was suspecting is a different one - a case where the dio code
calls aio_complete before a fallback to buffered i/o, and the latter
queues up a retry. By the time the retry gets to run the reference to
the iocb would have gone. (your extra iocb ref count patch wouldn't
be able to guard against this - the correct solution would be to
avoid doing aio_complete if we run into -ENOTBLK i.e. when we intend 
to fallback to buffered i/o).

But, if you are sure that its not the iocb but the ctx thats got freed,
I don't yet see why that would happen (since the workqueue should have 
been flushed before terminating the ioctx at exit).

Regards
Suparna

> 
> > 
> > But, O_DIRECT write should fallback to buffered i/o only if it 
> > encounters holes in the middle of the file, not for simple appends 
> > as in your case. Need to figure out how this could have happened ...
> > 
> > Could you try placing a few printks to find out if this is 
> > the case or if we need to look elsewhere ?
> 
> I'll do more debugging and let you know what I find.
> 
> Daniel
> 

-- 
Suparna Bhattacharya (suparna@in.ibm.com)
Linux Technology Center
IBM Software Labs, India