Missed Stable backport? maybe [PATCH v2 0/4] xen-blk: bug fixes

Missed Stable backport? maybe [PATCH v2 0/4] xen-blk: bug fixes

[Jason Andryuk]

Hi Roger,

I'm seeing a crash on shutdown in xen_blkbk_unmap on 3.12.22.  kdb backtrace below.  Could this be the shutdown race fixed by "[PATCH v2 3/4] xen-blkback: fix shutdown race"?  That email does not provide the observed backtrace, so I cannot compare.

The cover letter for "[PATCH v2 0/4] xen-blk: bug fixes" <Message-Id: 1391509575-3949-1-git-send-email-roger.pau@citrix.com> mentions backport to stable.  However, stable was not CC-ed and the patches seem to be missing from linux-stable.  Was this just an oversight?

Regards,

Jason

[0]kdb> bt
Stack traceback for pid 6523
0xffff880010033b10     6523     6522  1    0   R  0xffff880010033fd0 *block
 ffff88007c603ae8 000000000000e02b ffffffff8167bec7 ffff88007c603b50
 ffff8800103fc030 ffffffff81009f40 ffff88007c603b10 ffffffff8101ba19
 ffff88007c603b38 ffffffff810ea205 0000000000000001 ffff880010033b10
Call Trace:
 <IRQ>  [<ffffffff8167bec7>] ? xen_blkbk_unmap+0x247/0x2f0
 [<ffffffff81009f40>] ? xen_clocksource_read+0x20/0x30
 [<ffffffff8101ba19>] ? sched_clock+0x9/0x10
 [<ffffffff810ea205>] ? sched_clock_cpu+0xb5/0x100
 [<ffffffff81147cab>] ? rcu_irq_exit+0x4b/0x90
 [<ffffffff810bbcee>] ? irq_exit+0x4e/0x90
 [<ffffffff8154d295>] ? xen_evtchn_do_upcall+0x35/0x50
 [<ffffffff81b6ad7e>] ? xen_do_hypervisor_callback+0x1e/0x30
 [<ffffffff81b68eca>] ? error_exit+0x2a/0x60
 [<ffffffff81b689fb>] ? retint_restore_args+0x5/0x6
 [<ffffffff81470cd2>] ? strlcpy+0x42/0x60
 [<ffffffff8112803c>] ? cgroup_path+0xbc/0xd0
 [<ffffffff8167bfff>] ? __end_block_io_op+0x8f/0x190
 [<ffffffff8167c116>] ? end_block_io_op+0x16/0x30
 [<ffffffff8120e47d>] ? bio_endio+0x1d/0x30
 [<ffffffff818d4ea5>] ? dec_pending+0x195/0x300
 [<ffffffff818d531f>] ? clone_endio+0x6f/0xa0
 [<ffffffff8120e47d>] ? bio_endio+0x1d/0x30
 [<ffffffff8144a608>] ? blk_update_request+0xc8/0x410
 [<ffffffff8144a96c>] ? blk_update_bidi_request+0x1c/0x80
 [<ffffffff8144a9ef>] ? blk_end_bidi_request+0x1f/0x60
 [<ffffffff8144aab0>] ? blk_end_request+0x10/0x20
 [<ffffffff8168a5f1>] ? scsi_io_completion+0xf1/0x640
 [<ffffffff81683250>] ? scsi_finish_command+0xb0/0x110
 [<ffffffff8168a3fe>] ? scsi_softirq_done+0x12e/0x160
 [<ffffffff81450543>] ? blk_done_softirq+0x83/0xa0
 [<ffffffff810bba70>] ? __do_softirq+0x100/0x250
 [<ffffffff81b6ad1c>] ? call_softirq+0x1c/0x30
 [<ffffffff81015cb5>] ? do_softirq+0x55/0x90
 [<ffffffff810bbd1d>] ? irq_exit+0x7d/0x90
 [<ffffffff8154d295>] ? xen_evtchn_do_upcall+0x35/0x50
 [<ffffffff81b6ad7e>] ? xen_do_hypervisor_callback+0x1e/0x30
 <EOI>  [<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20
 [<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20
 [<ffffffff81009f0d>] ? xen_force_evtchn_callback+0xd/0x10
 [<ffffffff8100a792>] ? check_events+0x12/0x20
 [<ffffffff8100a77f>] ? xen_restore_fl_direct_reloc+0x4/0x4
 [<ffffffff810085c4>] ? __xen_pgd_pin+0x124/0x2b0
 [<ffffffff81008772>] ? xen_dup_mmap+0x22/0x30
 [<ffffffff810b46c5>] ? dup_mm+0x385/0x580
 [<ffffffff810b552d>] ? copy_process.part.33+0xc3d/0x1450
 [<ffffffff810b5ef5>] ? do_fork+0xd5/0x300
 [<ffffffff810c5a82>] ? __set_task_blocked+0x32/0x70
 [<ffffffff810b61a6>] ? SyS_clone+0x16/0x20
 [<ffffffff81b697c9>] ? stub_clone+0x69/0x90
 [<ffffffff81b69469>] ? system_call_fastpath+0x16/0x1b

Re: Missed Stable backport? maybe [PATCH v2 0/4] xen-blk: bug fixes

[Roger Pau Monné]

On 20/06/14 16:32, Jason Andryuk wrote:
> Hi Roger,
> 
> I'm seeing a crash on shutdown in xen_blkbk_unmap on 3.12.22.  kdb backtrace below.  Could this be the shutdown race fixed by "[PATCH v2 3/4] xen-blkback: fix shutdown race"?  That email does not provide the observed backtrace, so I cannot compare.

Could be related, but you didn't provide the reason for the crash, just
the backtrace, have you tried applying those patches on top of your kernel?

> The cover letter for "[PATCH v2 0/4] xen-blk: bug fixes" <Message-Id: 1391509575-3949-1-git-send-email-roger.pau@citrix.com> mentions backport to stable.  However, stable was not CC-ed and the patches seem to be missing from linux-stable.  Was this just an oversight?

Yes, they are committed to upstream but not stable, I think they should
all be backported to stable (3.10 and 3.12 AFAICT).

Roger.

Re: Missed Stable backport? maybe [PATCH v2 0/4] xen-blk: bug fixes

[Jason Andryuk]

On 6/20/2014 11:12 AM, Roger Pau Monné wrote:
> On 20/06/14 16:32, Jason Andryuk wrote:
>> Hi Roger,
>>
>> I'm seeing a crash on shutdown in xen_blkbk_unmap on 3.12.22.  kdb backtrace below.  Could this be the shutdown race fixed by "[PATCH v2 3/4] xen-blkback: fix shutdown race"?  That email does not provide the observed backtrace, so I cannot compare.
> 
> Could be related, but you didn't provide the reason for the crash, just
> the backtrace, have you tried applying those patches on top of your kernel?

The backtrace was seen rebooting my box.  kdb is configured in the hardware domain, so it broke into that.  Not all guests were shutdown prior to the restart.

It's intermittent.  I just rebooted ready to capture the serial log, but there was no crash.

I haven't tried the patches yet.  They came up when googling for related error messages.  I hope to try them soon.

Regards,

Jason

>> The cover letter for "[PATCH v2 0/4] xen-blk: bug fixes" <Message-Id: 1391509575-3949-1-git-send-email-roger.pau@citrix.com> mentions backport to stable.  However, stable was not CC-ed and the patches seem to be missing from linux-stable.  Was this just an oversight?
> 
> Yes, they are committed to upstream but not stable, I think they should
> all be backported to stable (3.10 and 3.12 AFAICT).
> 
> Roger.
> 

Re: Missed Stable backport? maybe [PATCH v2 0/4] xen-blk: bug fixes

[Jason Andryuk]

On 6/20/2014 12:34 PM, Jason Andryuk wrote:
> On 6/20/2014 11:12 AM, Roger Pau Monné wrote:
>> On 20/06/14 16:32, Jason Andryuk wrote:
>>> Hi Roger,
>>> 
>>> I'm seeing a crash on shutdown in xen_blkbk_unmap on 3.12.22.
>>> kdb backtrace below.  Could this be the shutdown race fixed by
>>> "[PATCH v2 3/4] xen-blkback: fix shutdown race"?  That email does
>>> not provide the observed backtrace, so I cannot compare.
>> 
>> Could be related, but you didn't provide the reason for the crash,
>> just the backtrace, have you tried applying those patches on top of
>> your kernel?
> 
> The backtrace was seen rebooting my box.  kdb is configured in the
> hardware domain, so it broke into that.  Not all guests were shutdown
> prior to the restart.
> 
> It's intermittent.  I just rebooted ready to capture the serial log,
> but there was no crash.
> 
> I haven't tried the patches yet.  They came up when googling for
> related error messages.  I hope to try them soon.

Friday afternoon, I cherry picked the first three patches.  I've rebooted a handful of times without error, but it was not reproducible without the patches.  I'll report if I see the error again.

Regards,

Jason