Re: LTP test related to virtio releasing and reassigning resource leads to guest hung

Re: LTP test related to virtio releasing and reassigning resource leads to guest hung

[Michael Tokarev]

10.08.2023 11:57, longguang.yue wrote:
> Hi, all:
> A ltp test leads to guest hung(io hung), the test releases virtio device resource and then reassign.
> I find device’s mem prefetchable resource 64-bit is changed.
> 
> ltp 
> test: https://github.com/linux-test-project/ltp/blob/522d7fba4afc84e07b252aa4cd91b241e81d6613/testcases/kernel/device-drivers/pci/tpci_kernel/ltp_tpci.c#L428
> 
> Do you know what cause the problem?
> 
> Thanks very much.
> 
> --------------------------
> ENV: kernel 5.10.0, qemu 6.2

Current qemu is 8.1 (well, almost, to be released this month;
previous release is 8.0 anyway).

This might be interesting to test in a current version before
going any further.

Thanks,

/mjt

Re: LTP test related to virtio releasing and reassigning resource leads to guest hung

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2073 bytes --]

On Thu, Aug 10, 2023 at 06:35:32PM +0800, longguang.yue wrote:
> could you please give me some tips to diagnose?  I could do tests on qemu 8.0, but product environment could not update.
> I test on different kernel version 5.10.0-X, one is better and results show problem is more about host kernel  rather than qemu.
> 
> 
> test cases are different combination of i440fx/q35 and virtio/scsi and kernel.

Can you post the guest kernel messages (dmesg)? If the guest is hanging
then it may be easiest to configure a serial console so the kernel
messages are sent to the host where you can see them.

Does the hang occur during the LTP code you linked or afterwards when
the PCI device is bound to a virtio driver?

Which virtio device causes the problem?

Can you describe the hang in more detail: is the guest still responsive
(e.g. console or network)? Is the QEMU HMP/QMP monitor still responsive?

Thanks,
Stefan

> 
> 
> 
> 
> thanks
> 
> 
> 
> 
> ---- Replied Message ----
> | From | Michael Tokarev<mjt@tls.msk.ru> |
> | Date | 08/10/2023 17:08 |
> | To | longguang.yue<kvmluck@163.com> ,
> qemu-devel<qemu-devel@nongnu.org> ,
> linux-kernel<linux-kernel@vger.kernel.org> |
> | Subject | Re: LTP test related to virtio releasing and reassigning resource leads to guest hung |
> 10.08.2023 11:57, longguang.yue wrote:
> Hi, all:
> A ltp test leads to guest hung(io hung), the test releases virtio device resource and then reassign.
> I find device’s mem prefetchable resource 64-bit is changed.
> 
> ltp
> test: https://github.com/linux-test-project/ltp/blob/522d7fba4afc84e07b252aa4cd91b241e81d6613/testcases/kernel/device-drivers/pci/tpci_kernel/ltp_tpci.c#L428
> 
> Do you know what cause the problem?
> 
> Thanks very much.
> 
> --------------------------
> ENV: kernel 5.10.0, qemu 6.2
> 
> Current qemu is 8.1 (well, almost, to be released this month;
> previous release is 8.0 anyway).
> 
> This might be interesting to test in a current version before
> going any further.
> 
> Thanks,
> 
> /mjt

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: LTP test related to virtio releasing and reassigning resource leads to guest hung

[Stefan Hajnoczi]

On Thu, 10 Aug 2023 at 10:14, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> On Thu, Aug 10, 2023 at 06:35:32PM +0800, longguang.yue wrote:
> > could you please give me some tips to diagnose?  I could do tests on qemu 8.0, but product environment could not update.
> > I test on different kernel version 5.10.0-X, one is better and results show problem is more about host kernel  rather than qemu.
> >
> >
> > test cases are different combination of i440fx/q35 and virtio/scsi and kernel.
>
> Can you post the guest kernel messages (dmesg)? If the guest is hanging
> then it may be easiest to configure a serial console so the kernel
> messages are sent to the host where you can see them.
>
> Does the hang occur during the LTP code you linked or afterwards when
> the PCI device is bound to a virtio driver?

I didn't see your original email so I missed the panic. I'd still like
to see the earlier kernel messages before the panic in order to
understand how the PCI device is bound.

Is the vda device with hung I/O the same device that was accessed by
the LTP test earlier? I guess the LTP test runs against the device and
then the virtio driver binds to the device again afterwards?

>
> Which virtio device causes the problem?
>
> Can you describe the hang in more detail: is the guest still responsive
> (e.g. console or network)? Is the QEMU HMP/QMP monitor still responsive?
>
> Thanks,
> Stefan
>
> >
> >
> >
> >
> > thanks
> >
> >
> >
> >
> > ---- Replied Message ----
> > | From | Michael Tokarev<mjt@tls.msk.ru> |
> > | Date | 08/10/2023 17:08 |
> > | To | longguang.yue<kvmluck@163.com> ,
> > qemu-devel<qemu-devel@nongnu.org> ,
> > linux-kernel<linux-kernel@vger.kernel.org> |
> > | Subject | Re: LTP test related to virtio releasing and reassigning resource leads to guest hung |
> > 10.08.2023 11:57, longguang.yue wrote:
> > Hi, all:
> > A ltp test leads to guest hung(io hung), the test releases virtio device resource and then reassign.
> > I find device’s mem prefetchable resource 64-bit is changed.
> >
> > ltp
> > test: https://github.com/linux-test-project/ltp/blob/522d7fba4afc84e07b252aa4cd91b241e81d6613/testcases/kernel/device-drivers/pci/tpci_kernel/ltp_tpci.c#L428
> >
> > Do you know what cause the problem?
> >
> > Thanks very much.
> >
> > --------------------------
> > ENV: kernel 5.10.0, qemu 6.2
> >
> > Current qemu is 8.1 (well, almost, to be released this month;
> > previous release is 8.0 anyway).
> >
> > This might be interesting to test in a current version before
> > going any further.
> >
> > Thanks,
> >
> > /mjt

Re: LTP test related to virtio releasing and reassigning resource leads to guest hung

[Stefan Hajnoczi]

On Thu, 10 Aug 2023 at 22:27, longguang.yue <kvmluck@163.com> wrote:
>
>
> 一）
> Can you post the guest kernel messages (dmesg)? If the guest is hanging
> then it may be easiest to configure a serial console so the kernel
> messages are sent to the host where you can see them.
>
> Does the hang occur during the LTP code you linked or afterwards when
> the PCI device is bound to a virtio driver?
>
>
> >   I used conosle, the hang occurred afterwards.   dmesg shows that tpci test is finished without error.
> LTP test case: https://github.com/linux-test-project/ltp/blob/522d7fba4afc84e07b252aa4cd91b241e81d6613/testcases/kernel/device-drivers/pci/tpci_kernel/ltp_tpci.c#L428
> kernel 5.10, qemu 6.2
>
> different guest-configuration tests show different results.  guest did not crash if hung-task-panic=0, in my case  i enable hung-task-panic in order to trace.
>
> test case 1:
> xml machine pc,virtio disk, virtio net ——  guest's io hung, network broke down, though console is avilable but io operation hung.
>
> #ps -aux| grep D
> root           7  0.0  0.0      0     0 ?        D    14:37   0:00 [kworker/u16:0+flush-253:0]
> root         483  0.0  0.0      0     0 ?      D    14:37   0:00 [jbd2/vda3-8]
>
> test case 2:
> xml machine q35,virtio/q35,scsi ——disk did not hung but network broke down. ping errors though everything looks ok and no crash and no kernel error
>
>
>
> 二）
> I didn't see your original email so I missed the panic. I'd still like
> to see the earlier kernel messages before the panic in order to
> understand how the PCI device is bound.
>
> Is the vda device with hung I/O the same device that was accessed by
> the LTP test earlier? I guess the LTP test runs against the device and
> then the virtio driver binds to the device again afterwards?
>
> > the test is
> ```
> // iterate all devices
> ……
> for (i = 0; i < 7; ++i) {  // iterate current device's resources
>   if (r->flags & IORESOURCE_MEM &&
>   r->flags & IORESOURCE_PREFETCH) {
>   pci_release_resource(dev, i);
>   ret = pci_assign_resource(dev, i);
>   prk_info("assign resource to '%d', ret '%d'", i, ret);
>   rc |= (ret < 0 && ret != -EBUSY) ? TFAIL : TPASS;
>   }
> }
> ```
> test does not do virtio device unbind and  bind.
> I only notice mem resource changed. see 'test-case 12'
>
> ———————————
> [   88.905705] ltp_tpci: test-case 12
> [   88.905706] ltp_tpci: assign resources
> [   88.905706] ltp_tpci: assign resource #0
> [   88.905707] ltp_tpci: name = 0000:00:07.0, flags = 262401, start 0xc080, end 0xc0ff
> [   88.905707] ltp_tpci: assign resource #1
> [   88.905708] ltp_tpci: name = 0000:00:07.0, flags = 262656, start 0xfebd4000, end 0xfebd4fff
> [   88.905709] ltp_tpci: assign resource #2
> [   88.905709] ltp_tpci: name = 0000:00:07.0, flags = 0, start 0x0, end 0x0
> [   88.905710] ltp_tpci: assign resource #3
> [   88.905710] ltp_tpci: name = 0000:00:07.0, flags = 0, start 0x0, end 0x0
> [   88.905711] ltp_tpci: assign resource #4
> [   88.905711] ltp_tpci: name = 0000:00:07.0, flags = 1319436, start 0xfe00c000, end 0xfe00ffff
> [   88.905713] virtio-pci 0000:00:07.0: BAR 4: releasing [mem 0xfe00c000-0xfe00ffff 64bit pref]
> [   88.905715] virtio-pci 0000:00:07.0: BAR 4: assigned [mem 0x24000c000-0x24000ffff 64bit pref]
> [   88.906693] ltp_tpci: assign resource to '4', ret '0'
> [   88.906694] ltp_tpci: assign resource #5
> [   88.906694] ltp_tpci: name = (null), flags = 0, start 0x0, end 0x0
> [   88.906695] ltp_tpci: assign resource #6
> [   88.906695] ltp_tpci: name = 0000:00:07.0, flags = 0, start 0x0, end 0x0
>
> [   88.906800] ltp_tpci: test-case 13

I don't know. Maybe the test case is leaving the device is a state
that conflicts with the virtio drivers that are bound after testing
finishes.

One approach is to trace the PCI BAR accesses after the test runs and
compare against a trace when the tpci driver hasn't been loaded. That
way you might be able to find out what is different.

Stefan