getblk spins endlessly in 2.4.19 SMP

getblk spins endlessly in 2.4.19 SMP

[Peter T. Breuer]

get_blk() loops forever internally for in a sort piece of driver code of
mine.  Why?  Somebody send me a clue, please!  get_hash_table returns
NULL inside get_blk and get_blk loops calling grow_buffers and
free_more_memory until death occurs hours later.

The same code works fine on the same kernel (binary!) on my laptop
with a 8M test device, and yet fails as described on my real SMP
machine on a 2GB test device.

What I'm trying to do is read one device and write to another,
in a loop. The code does:

    bh = bread (dev, blocknr, blksize);  // never gets past here! first time.
    if (!bh) return -EINVAL;
    req = my_make_request(dev2, bh);
    my_wait_until_request_done_timeout(req, timeout);
    bforget(bh);

and loops.

On the laptop, it loops through the whole (8M) device, block by block.
On the server, it hangs on bread() forever on the first pass through the
loop, with blocknr=0, because bread spins forever inside getblk, because
get_hash_table(dev, block, blksize) fails, and it calls free_more_memory
...  if I use get_hash_table directly, it runs through each block of
the device, failing on each block.

The laptop has 256M, and the server has 128M. The blksize is 1K on
both.

Both are newly booted.

The server is a REAL smp machine. The laptop is only running the SMP
kernel - it's not SMP.

I need a clue.  Please somebody send a clue.  Can it be that bread()
needs the buffer in cache?  I don't believe it - surely it should cause
a read request to drop through to the device when it gets to do its
ll_rw_block(), which it will as soon as getblk() returns.  But getblk
does not return.

Yet  memory is available:


               total       used       free     shared    buffers cached
   Mem:        127016     115532      11484          0 2164      68608
   -/+ buffers/cache:      44760      82256
   Swap:       144504          0     144504


I'm currently stumped. I thought it was a locking issue, and that
bread() needed to be run inside some lock, but I don't see one.
What else could distinguish a true SMP issue?  

The code runs in a kernel_thread, started from an ioctl, and reparented
to init.

There must be somebody out there who knows immediately what it is
I'm missing! If you do, I'd be very happy to hear from you!


Peter

Re: getblk spins endlessly in 2.4.19 SMP

[Andrew Morton]

"Peter T. Breuer" wrote:
> 
> get_blk() loops forever internally for in a sort piece of driver code of
> mine.

probably it found some buffers with the wrong ->b_size, tried to
get rid of them via try_to_free_buffers(), failed, and then fell
into the "oh, we're out of memory" loop.

You need to work out why grow_dev_page() is seeing buffers with
the wrong size against the page.  Be looking for incorrect or
missing calls to set_blocksize().

Re: getblk spins endlessly in 2.4.19 SMP

[Peter T. Breuer]

Hi - thanks for replying!

In article <3E14906E.7F6D3226@digeo.com> you wrote:
> "Peter T. Breuer" wrote:
>> 
>> get_blk() loops forever internally for in a sort piece of driver code of
>> mine.

> probably it found some buffers with the wrong ->b_size, tried to
> get rid of them via try_to_free_buffers(), failed, and then fell
> into the "oh, we're out of memory" loop.

Sounds just like what I see.

> You need to work out why grow_dev_page() is seeing buffers with
> the wrong size against the page.  Be looking for incorrect or
> missing calls to set_blocksize().

Set_blocksize was indeed not called before. I had set the size on the
kernel blksize_size array by hand. I now see that in addition to that
set_blocksize sets the bdev->bd_inode->i_blkbits value.

But I'm now calling set_blocksize when I set the device block size,
and no change.  The portable (running SMP with one processor) still
works fine.  The SMP server goes into a loop on the first call to
getblk, because get_hash_table returns NULL when asked for a block of
size 1024. Always. Same binaries. Both machines are running the same
kernel and modules.

Can you see anything that would explain the difference? Should
I call for a 4K block instead? I'm mystified. Tomorrow I'll boot
the server nosmp and see what happens.

I'll look inside grow_dev_page. It this called from free_more_memory
or from grow_buffers? Ahh .. the latter. Indeed, if it returns 0,
grow_buffers will return 0, and then we try free_more_memory,
whoch presumably does nothing, and loop.

grow_dev_page fails if find_or_create_page fails. Or if it returns a
page of the wrong size. Or if create_buffers fails. Sigh .. I guess
I get a page of the wrong size. I'll try asking for 4K instead of 1K.

Is there some way I  can "seed" things so that a call to getblk
will succeed? I mean, by making a buffer of teg right size be
available?

Peter