Re: About Xen bridged pci devices and suspend/resume for the X10SAE motherboard (SuperMicro)

["M. Ivanov" <mihail.ivanov93@gmail.com>]

[-- Attachment #1.1: Type: text/plain, Size: 14130 bytes --]

On Fri, 2015-08-14 at 12:15 -0400, Konrad Rzeszutek Wilk wrote:
> On Fri, Aug 14, 2015 at 05:08:32PM +0300, M. Ivanov wrote:
> > On Thu, 2015-08-13 at 15:49 -0400, Konrad Rzeszutek Wilk wrote:
> > > On Mon, Aug 10, 2015 at 09:21:35PM +0300, M. Ivanov wrote:
> > > > On Mon, 2015-08-10 at 10:47 -0400, Konrad Rzeszutek Wilk wrote:
> > > > > On Mon, Aug 10, 2015 at 05:14:28PM +0300, M. Ivanov wrote:
> > > > > > On Mon, 2015-08-10 at 09:58 -0400, Konrad Rzeszutek Wilk wrote:
> > > > > > > On Mon, Aug 10, 2015 at 02:11:38AM +0300, M. Ivanov wrote:
> > > > > > > > Hello,
> > > > > > > > 
> > > > > > > > excuse me for bothering you, but I've read an old thread on a mailing
> > > > > > > > list about X10SAE compatibility. 
> > > > > > > > http://lists.xen.org/archives/html/xen-devel/2014-02/msg02111.html
> > > > > > > 
> > > > > > > CC-ing Xen devel.
> > > > > > > > 
> > > > > > > > Currently I own this board and am trying to use it with Xen and be able
> > > > > > > > to suspend and resume.
> > > > > > > > 
> > > > > > > > But I am getting errors from the USB 3 Renesas controller about parity
> > > > > > > > in my bios event log, and my system hangs on resume,
> > > > > > > > so I was wondering if that is connected to the bridge(tundra) you've
> > > > > > > > mentioned.
> > > > > > > 
> > > > > > > Did you update the BIOS to the latest version?
> > > > > > Will updating to version 3 solve my issue?
> > > > > > Can you do a suspend/resume on your X10SAE?
> > > > > 
> > > > > It did work at some point. I will find out when I am at home later today.
> > > > > 
> > > > Looking forward to your reply and am really thankful for your time,
> > > > so far I've tried changing many of the settings in the bios,
> > > > fiddling with Xen's kernel params,
> > > > blacklisting the xhci driver, doing a xl detach.
> > > > 
> > > > The only thing I haven't done yet is updating the bios,
> > > > but Supermicro's support couldn't give me a changelog:
> > > > 
> > > > "The primary objective for ver3.0 BIOS release is to support Intel
> > > > Broadwell CPUs"
> > > > "We do not know if BIOS update will fix the issue you are seeing as we
> > > > never tested it with Xen."
> > > 
> > > I did test it remotely and it did something very odd. It suspended and then
> > > immediately resumed with tons of VT-d errors!?
> > > 
> > > I will try again but be actually right by it.
> > Thanks for your effort,
> > Can you suggest a way for me to log what is happening?
> 
> I usually have an serial cable attached to it and log that.
> Cranking up the debug on everything gives me some idea.
Reasonable enough, will do.

> > 
> > Since my machine just hangs up and I don't get a picture on the screen.
> > Or when I do(sometimes) - it's just a cursor,(on a black screen with
> > nothing else,no errors shown) and the machine doesn't react to any key
> > combinations.
> 
> Yeah. That is frustrating.
> > 
> > On a side note:
> > 
> > I did try updating the bios,
> > but got some really strange result.
> > The first time I got checksums about everything OK,
> > (erase,flash,verify),
> > but then it said - FDT is locked!
> > And I am at least 90% sure I've enabled reflash in 
> > the BIOS setup prior to flashing.
> > So I've restarted but didn't clear the CMOS(through the jumper on the
> > board). And it said Bios v 3.0 in the setup, then I also did the
> > Reset to optmized defaults.
> > 
> > Tried suspending and couldn't resume like always.
> > But perhaps currently my BIOS is in a broken/corrupted state.
> > 
> > After that I've tried reflashing the bios again. But this time - 
> > after the messages about Erasing,Flashing,Verifying:
> > it just froze. I didn't get any message about FDT, restarting or
> > whatsoever.
> > 
> > So I rebooted and it seems to work like before,
> > says version 3.0 in the BIOS setup.
> > I wonder if I should try clearing the CMOS and
> > flashing again.
> 
> One test I hadn't done is to try to suspend/resume
> under baremental Linux and see how that works. Does it work for you?
> 
Yep, without Xen I can suspend fine.

> The DMI tells me:
> [    7.276963] Hardware name: Supermicro X10SAE/X10SAE, BIOS 2.00 04/21/2014
> 
> So a bit older BIOS.
So we've had the same bios versions. But now I have a broken BIOS 3.0..
(probably).
Maybe I will try reflashing to 2.0 if you have hints about getting
suspend/resume on it.

> The last thing I see:
> 
> 
> # dmesg | grep -i Super
> [    0.000000] DMI: Supermicro X10SAE/X10SAE, BIOS 2.00 04/21/2014
> [    0.000000] ACPI: RSDP 0x00000000000F0490 000024 (v02-MB 01072009 AMI  00010013)
> [    0.000000] ACPI: DSDT 0x000000009B9DA1E8 00CDDA (v02 SUPERM SMCI--MB 00000000 INTL 20120711)
> [    0.000000] ACPI: APIC 0x000000009B9E70D8 000092 (v03 SUPERM SMCI--MB 01072009 AMI  00010013)
> [    0.000000] ACPI: FPDT 0x000000009B9E7170 000044 (v01 SUPERM SMCI--MB 01072009 AMI  00010013)
> [    0.000000] ACPI: MCFG 0x000000009B9E9488 00003C (v01 SUPERM SMCI--MB 01072009 MSFT 00000097)
> [    0.000000] ACPI: HPET 0x000000009B9E94C8 000038 (v01 SUPERM SMCI--MB 01072009 AMI. 00000005)
> [    7.623166] Hardware name: Supermicro X10SAE/X10SAE, BIOS 2.00 04/21/2014
> # \a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\apm-sus\apoe\a\b \b\b \b# ^[[Jcd /sys/power
> # ls
> disk	    pm_freeze_timeout  pm_trace		   resume
> image_size  pm_print_times     pm_trace_dev_match  state
> pm_async    pm_test	       reserved_size	   wakeup_count
> # c# ^[[Jcat state
> freeze standby disk
> # echo standby > state
Since I am using CentOS in my experiments(version 6.7 has no systemd
support and 7 has no Xen support...) 
I do a echo mem > /sys/power/state.
> [  204.755164] PM: Syncing filesystems ... done.
> [  204.755223] PM: Preparing system for standby sleep
> [  204.755606] Freezing user space processes ... (elapsed 0.000 seconds) done.
> [  204.756506] Freezing reý+Ë+Ë•ÿezable tasks ... (elapsed 0.001 seconds) done.
> [  204.757607] PM: Entering standby sleep
> [  204.757762] Suspending console(s affinity for irq 40
> (XEN) Broke affinity for irq 41
> (XEN) Broke affinity for irq 23
> (XEN) Broke affinity for irq 16
> (XEN) Broke affinity fouppress EOI broadcast on CPU#3
> (XEN) masked ExtINT on CPU#3
> (XEN) Suppress EOI broadcast on CPU#4
> (XEN) masked ExtINT on CPU#4
> (XEN) Suppress EOI broadcast on CPU#5
> (XEN) masked ExtINT on CPU#5
> (XEN) Suppress EOI broadcast on CPU#6
> (XEN) masked ExtINT on CPU#6
> (XEN) Suppress EOI broadcast on CPU#7
> (XEN) masked ExtINT on CPU#7
> (XEN) traps.c:2653:d0v1 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
> (XEN) traps.c:2653:d0v1 Domain attempted WRMSR 00000000c0000082 from 0xffff83082727b000 to 0xffffffff8172e910.
> (XEN) traps.c:2653:d0v1 Domain attempted WRMSR 00000000c0000083 from 0xffff83082727b080 to 0xffffffff81730c40.
> (XEN) traps.c:2653:d0v1 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
> (XEN) traps.c:2653:d0v1 Domain attempted WRMSR 0000000000000175 from 0xffff83082727ffc0 to 0x0000000000000000.
> (XEN) traps.c:2653:d0v1 Domain attempted WRMSR 0000000000000176 from 0xffff82d08022c7d0 to 0xffffffff81730c70.
> (XEN) traps.c:2653:d0v1 Domain attempted WRMSR 00000000c0000083 from 0xffff83082727b080 to 0xffffffff81730eb0.
> (XEN) traps.c:2653:d0v1 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
> (XEN) d0 attempted to change d0v1's CR4 flags 00002660 -> 00042760
> (XEN) traps.c:2653:d0v2 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
> (XEN) traps.c:2653:d0v2 Domain attempted WRMSR 00000000c0000082 from 0xffff82d0803db000 to 0xffffffff8172e910.
> (XEN) traps.c:2653:d0v2 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0803db080 to 0xffffffff81730c40.
> (XEN) traps.c:2653:d0v2 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
> (XEN) traps.c:2653:d0v2 Domain attempted WRMSR 0000000000000175 from 0xffff82d0803dffc0 to 0x0000000000000000.
> (XEN) traps.c:2653:d0v2 Domain attempted WRMSR 0000000000000176 from 0xffff82d08022c7d0 to 0xffffffff81730c70.
> (XEN) traps.c:2653:d0v2 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0803db080 to 0xffffffff81730eb0.
> (XEN) traps.c:2653:d0v2 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
> (XEN) d0 attempted to change d0v2's CR4 flags 00002660 -> 00042760
> (XEN) traps.c:2653:d0v3 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
> (XEN) traps.c:2653:d0v3 Domain attempted WRMSR 00000000c0000082 from 0xffff82d0803db000 to 0xffffffff8172e910.
> (XEN) traps.c:2653:d0v3 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0803db080 to 0xffffffff81730c40.
> (XEN) traps.c:2653:d0v3 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
> (XEN) traps.c:2653:d0v3 Domain attempted WRMSR 0000000000000175 from 0xffff82d0803dffc0 to 0x0000000000000000.
> (XEN) traps.c:2653:d0v3 Domain attempted WRMSR 0000000000000176 from 0xffff82d08022c7d0 to 0xffffffff81730c70.
> (XEN) traps.c:2653:d0v3 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0803db080 to 0xffffffff81730eb0.
> (XEN) traps.c:2653:d0v3 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
> (XEN) d0 attempted to change d0v3's CR4 flags 00002660 -> 00042760
> (XEN) traps.c:2653:d0v4 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
> (XEN) traps.c:2653:d0v4 Domain attempted WRMSR 00000000c0000082 from 0xffff82d0803db000 to 0xffffffff8172e910.
> (XEN) traps.c:2653:d0v4 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0803db080 to 0xffffffff81730c40.
> (XEN) traps.c:2653:d0v4 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
> (XEN) traps.c:2653:d0v4 Domain attempted WRMSR 0000000000000175 from 0xffff82d0803dffc0 to 0x0000000000000000.
> (XEN) traps.c:2653:d0v4 Domain attempted WRMSR 0000000000000176 from 0xffff82d08022c7d0 to 0xffffffff81730c70.
> (XEN) traps.c:2653:d0v4 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0803db080 to 0xffffffff81730eb0.
> (XEN) traps.c:2653:d0v4 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
> (XEN) d0 attempted to change d0v4's CR4 flags 00002660 -> 00042760
> (XEN) traps.c:2653:d0v5 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
> (XEN) traps.c:2653:d0v5 Domain attempted WRMSR 00000000c0000082 from 0xffff8308272f3000 to 0xffffffff8172e910.
> (XEN) traps.c:2653:d0v5 Domain attempted WRMSR 00000000c0000083 from 0xffff8308272f3080 to 0xffffffff81730c40.
> (XEN) traps.c:2653:d0v5 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
> (XEN) traps.c:2653:d0v5 Domain attempted WRMSR 0000000000000175 from 0xffff8308272f7fc0 to 0x0000000000000000.
> (XEN) traps.c:2653:d0v5 Domain attempted WRMSR 0000000000000176 from 0xffff82d08022c7d0 to 0xffffffff81730c70.
> (XEN) traps.c:2653:d0v5 Domain attempted WRMSR 00000000c0000083 from 0xffff8308272f3080 to 0xffffffff81730eb0.
> (XEN) traps.c:2653:d0v5 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
> (XEN) d0 attempted to change d0v5's CR4 flags 00002660 -> 00042760
> (XEN) traps.c:2653:d0v6 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
> (XEN) traps.c:2653:d0v6 Domain attempted WRMSR 00000000c0000082 from 0xffff82d0803db000 to 0xffffffff8172e910.
> (XEN) traps.c:2653:d0v6 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0803db080 to 0xffffffff81730c40.
> (XEN) traps.c:2653:d0v6 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
> (XEN) traps.c:2653:d0v6 Domain attempted WRMSR 0000000000000175 from 0xffff82d0803dffc0 to 0x0000000000000000.
> (XEN) traps.c:2653:d0v6 Domain attempted WRMSR 0000000000000176 from 0xffff82d08022c7d0 to 0xffffffff81730c70.
> (XEN) traps.c:2653:d0v6 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0803db080 to 0xffffffff81730eb0.
> (XEN) traps.c:2653:d0v6 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
> (XEN) d0 attempted to change d0v6's CR4 flags 00002660 -> 00042760
> (XEN) event_fifo.c:50:d0v6 domain 0, port 50 already on a queue
> (XEN) traps.c:2653:d0v7 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
> (XEN) traps.c:2653:d0v7 Domain attempted WRMSR 00000000c0000082 from 0xffff83082727b000 to 0xffffffff8172e910.
> (XEN) traps.c:2653:d0v7 Domain attempted WRMSR 00000000c0000083 from 0xffff83082727b080 to 0xffffffff81730c40.
> (XEN) traps.c:2653:d0v7 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
> (XEN) traps.c:2653:d0v7 Domain attempR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
> (XEN) d0 attempted to change d0v7's CR4 flags 00002660 -> 00042760
> ^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
> 
> telnet> Connection closed.
> [Connecting to system 50 ]
> 
> Great, so 'Domain 0 crashed'!
> 
> Pff - and oddly enough the same kernel suspend/resumes fine on my desktop and laptop.
And yes, it's not about the kernel, it's about Xen.
I've tried like at least 7 Linux kernels from 3.11 to 3.19.

Also did you get the parity error in your bios event log?
About the Bus06 or Bus05(if you have nothing in your PCI-E slots).
Which I think is the Renesas USB 3 controller.
> > 
> > > > > > > > 
> > > > > > > > I will be very glad if you could share any information regarding this
> > > > > > > > matter. 
> > > > > > > > 
> > > > > > > > Best regards,
> > > > > > > > M. Ivanov
> > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > > 
> > > > 
> > > 
> > > 
> > 
> 
> 


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel