freeze_bdev hangs

freeze_bdev hangs

[Prashant Shah]

Hi,

I am writing a filter driver for ext3 in which it replaces the block
device request queue 'make_request_fn' with the driver custom
function. Calling freeze_bdev() or freeze_super() before replacing the
'make_request_fn' hangs the kernel. What am I doing wrong ? Will this
method work ?

full code : https://gist.github.com/3839380

Snippet :

        sb = freeze_bdev(bd);
        if (sb)
                printk(KERN_INFO "filter: freez block device\n");

        if (bd->bd_disk->queue->make_request_fn == filter_mrqfn) {
                printk(KERN_INFO "filter: modules request function is
already active\n");
        } else {
                orig_mrqfn = bd->bd_disk->queue->make_request_fn;
                bd->bd_disk->queue->make_request_fn = filter_mrqfn;
        }
        thaw_bdev(bd, sb);

Please forgive my ignorance, I am a fresher in file systems :)

Regards.

Re: freeze_bdev hangs

[Eric Sandeen]

On 10/5/12 6:58 AM, Prashant Shah wrote:
> Hi,
> 
> I am writing a filter driver for ext3 in which it replaces the block
> device request queue 'make_request_fn' with the driver custom
> function. Calling freeze_bdev() or freeze_super() before replacing the
> 'make_request_fn' hangs the kernel. What am I doing wrong ? Will this
> method work ?

I haven't looked at your custom code, but freeze has changed a bit lately,
and "hangs" is a very vague description

For starters I'd try a sysrq-w when the system "hangs" and see where things
are at, that would offer you some clues.

-Eric

Re: freeze_bdev hangs

[Prashant Shah]

Hi,

On Fri, Oct 5, 2012 at 11:03 PM, Eric Sandeen <sandeen@redhat.com> wrote:
> I haven't looked at your custom code, but freeze has changed a bit lately,
> and "hangs" is a very vague description
>
> -Eric

I am monitoring the file system (ext3) for block writes and on
linux-2.6.18 (centos - 5.8 64bit) there are lots of writes request per
second with inode number set to 0 even when the file system is idle
and without any work loads. Didnt see this happen on linux-3.x.

filter: sector=242983 inode=0
filter: sector=242991 inode=0
filter: sector=242999 inode=0
filter: sector=243007 inode=0
filter: sector=243015 inode=0
filter: sector=243023 inode=0
filter: sector=243031 inode=0
filter: sector=243039 inode=0

What are these request with inode number as 0 ? All the special inode
numbers start from 1.

http://lxr.linux.no/#linux+v2.6.32/include/linux/ext3_fs.h#L56

Regards.

Re: freeze_bdev hangs

[Jan Kara]

On Mon 08-10-12 13:13:34, Prashant Shah wrote:
> Hi,
> 
> On Fri, Oct 5, 2012 at 11:03 PM, Eric Sandeen <sandeen@redhat.com> wrote:
> > I haven't looked at your custom code, but freeze has changed a bit lately,
> > and "hangs" is a very vague description
> >
> > -Eric
> 
> I am monitoring the file system (ext3) for block writes and on
> linux-2.6.18 (centos - 5.8 64bit) there are lots of writes request per
> second with inode number set to 0 even when the file system is idle
> and without any work loads. Didnt see this happen on linux-3.x.
> 
> filter: sector=242983 inode=0
> filter: sector=242991 inode=0
> filter: sector=242999 inode=0
> filter: sector=243007 inode=0
> filter: sector=243015 inode=0
> filter: sector=243023 inode=0
> filter: sector=243031 inode=0
> filter: sector=243039 inode=0
  And what tool are you using for monitoring?

								Honza

-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: freeze_bdev hangs

[Prashant Shah]

Hi,

On Mon, Oct 8, 2012 at 9:49 PM, Jan Kara <jack@suse.cz> wrote:
>   And what tool are you using for monitoring?

I am using printk(). A custom module that changes the make_request_fn
of the block device to a custom callback. In the callback I am
printing the request bio details and passing the bio back to the
original make_request_fn.

Regards.

Re: freeze_bdev hangs

[Jan Kara]

On Mon 08-10-12 22:05:38, Prashant Shah wrote:
> Hi,
> 
> On Mon, Oct 8, 2012 at 9:49 PM, Jan Kara <jack@suse.cz> wrote:
> >   And what tool are you using for monitoring?
> 
> I am using printk(). A custom module that changes the make_request_fn
> of the block device to a custom callback. In the callback I am
> printing the request bio details and passing the bio back to the
> original make_request_fn.
  OK. There are several ways how I could imagine you hit ino == 0. Either
someone is using direct IO or inode is a block device inode...

								Honza

Re: freeze_bdev hangs

[Prashant Shah]

Hi,

On Wed, Oct 10, 2012 at 3:19 AM, Jan Kara <jack@suse.cz> wrote:
>   OK. There are several ways how I could imagine you hit ino == 0. Either
> someone is using direct IO or inode is a block device inode...
>

Fixed one of the issues. It turn out that printk() in the request
function cause the /var/log/messages file to be updated that resulted
in more block copies in request function which in turn triggered more
printk's.

I will check the direct IO operations and report back.

Regards.

Re: freeze_bdev hangs

[Prashant Shah]

Hi,

On Wed, Oct 10, 2012 at 3:19 AM, Jan Kara <jack@suse.cz> wrote:
>>   OK. There are several ways how I could imagine you hit ino == 0. Either
>> someone is using direct IO or inode is a block device inode...
>>

Most of the inode 0 writes belonged to journal (verified with the
sector number). I think that such journal writes should set the inode
to 8 or whatever is reserved for the journal rather than 0.

Regards.

Re: freeze_bdev hangs

[Eric Sandeen]

On 10/8/12 2:43 AM, Prashant Shah wrote:
> Hi,
> 
> On Fri, Oct 5, 2012 at 11:03 PM, Eric Sandeen <sandeen@redhat.com> wrote:
>> I haven't looked at your custom code, but freeze has changed a bit lately,
>> and "hangs" is a very vague description
>>
>> -Eric
> 
> I am monitoring the file system (ext3) for block writes and on
> linux-2.6.18 (centos - 5.8 64bit) there are lots of writes request per
> second with inode number set to 0 even when the file system is idle
> and without any work loads. Didnt see this happen on linux-3.x.
> 
> filter: sector=242983 inode=0

if you dumpe2fs your fs you may find those sectors contain metadata not data.

Anyway, did you try the suggestions I offered to track down your original
question about the freeze deadlock?

-Eric

> filter: sector=242991 inode=0
> filter: sector=242999 inode=0
> filter: sector=243007 inode=0
> filter: sector=243015 inode=0
> filter: sector=243023 inode=0
> filter: sector=243031 inode=0
> filter: sector=243039 inode=0
> 
> What are these request with inode number as 0 ? All the special inode
> numbers start from 1.
> 
> http://lxr.linux.no/#linux+v2.6.32/include/linux/ext3_fs.h#L56
> 
> Regards.
> 

Re: freeze_bdev hangs

[Prashant Shah]

Hi,

On Mon, Oct 8, 2012 at 11:51 PM, Eric Sandeen <sandeen@redhat.com> wrote:
>
> if you dumpe2fs your fs you may find those sectors contain metadata not data.

Why is inode number 0 even if its journal or meta data ? Also I am
getting *lots* of write per second on a bare metal centos with nothing
running. Its strange. Didn't see it on 3.x.

> Anyway, did you try the suggestions I offered to track down your original
> question about the freeze deadlock?

I am not facing any deadlocks on 2.6.18 but was facing it on 3.x.
freeze_bdev() and thaw_bdev() work correctly on 2.6.18. I will try it
again on 3.x and confirm.

Regards.

Re: freeze_bdev hangs

[Prashant Shah]

Hi,

>> Anyway, did you try the suggestions I offered to track down your original
>> question about the freeze deadlock?
>
> I am not facing any deadlocks on 2.6.18 but was facing it on 3.x.
> freeze_bdev() and thaw_bdev() work correctly on 2.6.18. I will try it
> again on 3.x and confirm.
>

I tested it on 3.x and freeze_bdev() / thaw_bdev() work correctly. It
turns out I was doing something wrong related to sleeping/locks in the
custom make_request_fn() that caused it to freeze.

> Regards.