[Drbd-dev] [CASE-30] dec_ap_bio wakeup condition difference between V8.4.7 and V9.0.1

[Drbd-dev] [CASE-30] dec_ap_bio wakeup condition difference between V8.4.7 and V9.0.1

[Jaeheon Kim]

[-- Attachment #1: Type: text/plain, Size: 3111 bytes --]

Hi,


I have one more question to the case-29.

According to the Windows and Linux DRBD kernel stack, there are a similar
pattern.

1. Stacks

1) Windows Stack dump when the file copy command was pending.

   - Please see "inc_ap_bio"

    nt!KiSwapThread+0x266
    nt!KiCommitThreadWait+0x1df
    nt!KeWaitForMultipleObjects+0x535
    drbd!schedule+0x114
    drbd!inc_ap_bio+0x49
    drbd!drbd_make_request+0x29
    drbd!DoSplitIo+0xfb
    drbd!mvolReadWriteDevice+0x2b0
    drbd!mvolWorkThread+0x114


2) Linux Stack dump when the file copy command was pending.

  - Pleae check previous post.

    [Drbd-dev] [CASE-29] After re-connect, WFBitMapS-WFBitMapT status has
sustained
                                    continuously and copy command hangs
     http://lists.linbit.com/pipermail/drbd-dev/2016-March/003328.html


  - Please see the "7) drbd pending point #1"

    [CASE-29] primary log
     http://pastebin.com/3hAWNkHn

    Mar  3 14:52:04 drbd9-01 kernel: Call Trace:
    Mar  3 14:52:04 drbd9-01 kernel: [<ffffffff816096a9>] schedule+0x29/0x70
    Mar  3 14:52:04 drbd9-01 kernel: [<ffffffffa04a03da>]
drbd_make_request+0x14a/0x360 [drbd]
    Mar  3 14:52:04 drbd9-01 kernel: [<ffffffff812aa56f>] ?
generic_make_request_checks+0x24f/0x380
    Mar  3 14:52:04 drbd9-01 kernel: [<ffffffff81098230>] ?
wake_up_bit+0x30/0x30
    Mar  3 14:52:04 drbd9-01 kernel: [<ffffffff812aa782>]
generic_make_request+0xe2/0x130
    Mar  3 14:52:04 drbd9-01 kernel: [<ffffffff812aa841>]
submit_bio+0x71/0x150
    Mar  3 14:52:04 drbd9-01 kernel: [<ffffffffa0531c55>]
ext4_io_submit+0x25/0x50 [ext4]



2. Sources

1) Ver 9.0.1

    static inline void inc_ap_bio(struct drbd_device *device, int rw)
    {
        wait_event(device->misc_wait, inc_ap_bio_cond(device, rw));
    }

    static inline void dec_ap_bio(struct drbd_device *device, int rw)
    {
         int ap_bio = atomic_dec_return(&device->ap_bio_cnt[rw]);
       .........

         if (ap_bio == 0)
                wake_up(&device->misc_wait);
    }


2) Ver 8.4.7

    static inline void inc_ap_bio(struct drbd_device *device)
    {
         wait_event(device->misc_wait, inc_ap_bio_cond(device));
    }

    static inline void dec_ap_bio(struct drbd_device *device)
    {
         int mxb = drbd_get_max_buffers(device);
         int ap_bio = atomic_dec_return(&device->ap_bio_cnt);

        .............


         /* this currently does wake_up for every dec_ap_bio!
          * maybe rather introduce some type of hysteresis?
          * e.g. (ap_bio == mxb/2 || ap_bio == 0) ? */

          if (ap_bio < mxb)
                 wake_up(&device->misc_wait);
    }



3. Questions

1) See "1. Stacks".
    I think, The Linux "drbd_make_request+0x14a/0x360 [drbd]" will probably
    mean the same thing as the Windows "drbd!inc_ap_bio+0x49".
    What do you think about it?

2) In version 8.4.7, dec_ap_bio wakes up misc_wait when condition is
"ap_bio < max".
    I mean, this version has greater tolerance than version 9.0.1.
    My question is Why do you change this condition from "ap_bio < max" to
"ap_bio == 0"  in version 9.0.1?


Thanks.

[-- Attachment #2: Type: text/html, Size: 4020 bytes --]

Re: [Drbd-dev] [CASE-30] dec_ap_bio wakeup condition difference between V8.4.7 and V9.0.1

[Lars Ellenberg]

On Thu, Mar 10, 2016 at 09:39:46PM +0900, Jaeheon Kim wrote:
> 2. Sources
> 
> 1) Ver 9.0.1
> 
>     static inline void inc_ap_bio(struct drbd_device *device, int rw)
>     {
>         wait_event(device->misc_wait, inc_ap_bio_cond(device, rw));
>     }
> 
>     static inline void dec_ap_bio(struct drbd_device *device, int rw)
>     {
>          int ap_bio = atomic_dec_return(&device->ap_bio_cnt[rw]);
>        .........
> 
>          if (ap_bio == 0)
>                 wake_up(&device->misc_wait);
>     }
> 
> 
> 2) Ver 8.4.7
> 
>     static inline void inc_ap_bio(struct drbd_device *device)
>     {
>          wait_event(device->misc_wait, inc_ap_bio_cond(device));
>     }
> 
>     static inline void dec_ap_bio(struct drbd_device *device)
>     {
>          int mxb = drbd_get_max_buffers(device);
>          int ap_bio = atomic_dec_return(&device->ap_bio_cnt);
> 
>         .............
> 
> 
>          /* this currently does wake_up for every dec_ap_bio!
>           * maybe rather introduce some type of hysteresis?
>           * e.g. (ap_bio == mxb/2 || ap_bio == 0) ? */
> 
>           if (ap_bio < mxb)
>                  wake_up(&device->misc_wait);
>     }
> 
> 
> 
> 3. Questions
> 
> 1) See "1. Stacks".
>     I think, The Linux "drbd_make_request+0x14a/0x360 [drbd]" will probably
>     mean the same thing as the Windows "drbd!inc_ap_bio+0x49".
>     What do you think about it?
> 
> 2) In version 8.4.7, dec_ap_bio wakes up misc_wait when condition is
> "ap_bio < max".
>     I mean, this version has greater tolerance than version 9.0.1.
>     My question is Why do you change this condition from "ap_bio < max" to
> "ap_bio == 0"  in version 9.0.1?


Because there is no more "max" in 9, no-one can possibly wait for
count < max. There are however things that still wait for count == 0.

Though we probably will re-introduce some (resource wide) limit
on the maximum amount of "in-flight bios" we accept to be queued.

"misc" wait queue is used for a lot of things,
and a lot of things wake it.


-- 
: Lars Ellenberg
: LINBIT | Keeping the Digital World Running
: DRBD -- Heartbeat -- Corosync -- Pacemaker
: R&D, Integration, Ops, Consulting, Support

DRBD® and LINBIT® are registered trademarks of LINBIT

Re: [Drbd-dev] [CASE-30] dec_ap_bio wakeup condition difference between V8.4.7 and V9.0.1

[Jaeheon Kim]

[-- Attachment #1.1: Type: text/plain, Size: 3302 bytes --]

2016-03-10 23:04 GMT+09:00 Lars Ellenberg <lars.ellenberg@linbit.com>:

> On Thu, Mar 10, 2016 at 09:39:46PM +0900, Jaeheon Kim wrote:
> > 2. Sources
> >
> > 1) Ver 9.0.1
> >
> >     static inline void inc_ap_bio(struct drbd_device *device, int rw)
> >     {
> >         wait_event(device->misc_wait, inc_ap_bio_cond(device, rw));
> >     }
> >
> >     static inline void dec_ap_bio(struct drbd_device *device, int rw)
> >     {
> >          int ap_bio = atomic_dec_return(&device->ap_bio_cnt[rw]);
> >        .........
> >
> >          if (ap_bio == 0)
> >                 wake_up(&device->misc_wait);
> >     }
> >
> >
> > 2) Ver 8.4.7
> >
> >     static inline void inc_ap_bio(struct drbd_device *device)
> >     {
> >          wait_event(device->misc_wait, inc_ap_bio_cond(device));
> >     }
> >
> >     static inline void dec_ap_bio(struct drbd_device *device)
> >     {
> >          int mxb = drbd_get_max_buffers(device);
> >          int ap_bio = atomic_dec_return(&device->ap_bio_cnt);
> >
> >         .............
> >
> >
> >          /* this currently does wake_up for every dec_ap_bio!
> >           * maybe rather introduce some type of hysteresis?
> >           * e.g. (ap_bio == mxb/2 || ap_bio == 0) ? */
> >
> >           if (ap_bio < mxb)
> >                  wake_up(&device->misc_wait);
> >     }
> >
> >
> >
> > 3. Questions
> >
> > 1) See "1. Stacks".
> >     I think, The Linux "drbd_make_request+0x14a/0x360 [drbd]" will
> probably
> >     mean the same thing as the Windows "drbd!inc_ap_bio+0x49".
> >     What do you think about it?
> >
> > 2) In version 8.4.7, dec_ap_bio wakes up misc_wait when condition is
> > "ap_bio < max".
> >     I mean, this version has greater tolerance than version 9.0.1.
> >     My question is Why do you change this condition from "ap_bio < max"
> to
> > "ap_bio == 0"  in version 9.0.1?
>
>
> Because there is no more "max" in 9, no-one can possibly wait for
> count < max. There are however things that still wait for count == 0.
>
>
I see.


> Though we probably will re-introduce some (resource wide) limit
> on the maximum amount of "in-flight bios" we accept to be queued.
>
>
Could you explain more detail?


> "misc" wait queue is used for a lot of things,
> and a lot of things wake it.
>
>
Yes, I know that.
However, conversely, if anyone do not wake up to this "misc",  this pending
case could be occurred.
This case had generated during the big file copy. And  you can see
"drbd_make_request+0x14a/0x360
[drbd]" at Linux stack which is similar Windows.
So I doubt whether inc_ap_bio is pending forever because the wake-up
condition failed at this dec_ap_bio function.

Anyway, Did you reproduce it in Linux?

Thanks.



> --
> : Lars Ellenberg
> : LINBIT | Keeping the Digital World Running
> : DRBD -- Heartbeat -- Corosync -- Pacemaker
> : R&D, Integration, Ops, Consulting, Support
>
> DRBD® and LINBIT® are registered trademarks of LINBIT
> _______________________________________________
> drbd-dev mailing list
> drbd-dev@lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-dev
>



-- 

[image: 설명: logo]

*JaeHeon Kim.  Director        jhkim*@mantech.co.kr

[image: 본문 이미지 1]

[-- Attachment #1.2: Type: text/html, Size: 6493 bytes --]

[-- Attachment #2: image001.png --]
[-- Type: image/png, Size: 4544 bytes --]

[-- Attachment #3: image.png --]
[-- Type: image/png, Size: 16229 bytes --]