Re: Xen on Seattle

[Suravee Suthikulanit <suravee.suthikulpanit@amd.com>]

[-- Attachment #1: Type: text/plain, Size: 2999 bytes --]

On 9/2/2014 1:31 PM, Suravee Suthikulanit wrote:
> On 9/2/2014 1:15 PM, Stefano Stabellini wrote:
>> On Tue, 2 Sep 2014, Suravee Suthikulanit wrote:
>>> On 9/2/2014 1:06 PM, Stefano Stabellini wrote:
>>>> On Tue, 2 Sep 2014, Suravee Suthikulanit wrote:
>>>>> Hi all,
>>>>>
>>>>> I am running into an issue booting Xen on Seattle platform where dom0
>>>>> kernel
>>>>> failed to make the following hypercall in arch/arm/xen/enlighten.c:
>>>>> xen_percpu_init().
>>>>>
>>>>>    HYPERVISOR_vcpu_op(VCPUOP_register_vcpu_info, cpu, &info)
>>>>>
>>>>> After investigation, the hypercall failed in the xen/common/domain.c:
>>>>> map_vcpu_info(), where it check
>>>>>
>>>>>       if ( offset > (PAGE_SIZE - sizeof(vcpu_info_t)) )
>>>>>           return -EINVAL;
>>>>>
>>>>> Here:
>>>>>
>>>>> (XEN) DEBUG0: map_vcpu_info: offset = 0x6790
>>>>> (XEN) DEBUG0: map_vcpu_info: PAGE_SIZE = 0x1000
>>>>> (XEN) DEBUG0: map_vcpu_info: sizeof(vcpu_info_t) = 0x30
>>>>>
>>>>> If I understand correctly, it seems that the hypervisor is making
>>>>> assumption
>>>>> that the Dom0 guest is using 4K page size.  The PAGE_SHIFT in
>>>>> include/asm-arm/config.h is currently set to 12.  So, I have tried
>>>>> changing
>>>>> this to 16 (for 64K page). However, there is a restriction on the
>>>>> 16-bit
>>>>> value
>>>>> of the struct active_grant_entry { length:16; }. So, I cannot boot
>>>>> dom0
>>>>> with
>>>>> 64K page size. Is this accurate?
>>>>
>>>> Hello Suravee,
>>>> yes, that is correct: all pages used in hypercalls and shared with the
>>>> hypervisor need to be 4K pages.
>>>>
>>>
>>> Ok, the next question would be, would it make sense/possible to
>>> extend the
>>> support to allow 64K pages?
>>
>> It might make sense, but it is not trivial: all Xen hypercall interfaces
>> are built for 4K pages and would need a significant revamp.  Certainly
>> too late for the next release. It might be something to look at for Xen
>> 4.6.
>>
>> There is nothing preventing a guest kernel from using 64K pages anywhere
>> else though.
>>
> The alternative is to try booting with dom0 kernel configured with 4K
> pages. Currently, with 3.16 kernel, this is not working yet with Seattle
> since we would need level 4 page table support to boot with 4K page size.
>
> There is a patch from Catalin which should add support for level 4 page.
>   Let us experiment with that and get back with you guys.
>

Quick update:

After configure kernel w/ 4K page and Using 48-bit (w/ 4 level page 
table), Dom0 kernel seems to be coming along further at this point up 
until the following point:

.....
(XEN) DOM0: NET: Registered protocol family 10
(XEN) DOM0: sit: IPv6 over IPv4 tunneling driver
(XEN) DOM0: NET: Registered protocol family 17
(XEN) DOM0: Bridge firewalling registered
(XEN) DOM0: 9pnet: Installing 9P2000 support
(XEN) DOM0: NET: Registered protocol family 37
(XEN) DOM0: bootconsole [uart0] disabled

Please see attachment for full boot log. I am still working on trying to 
get to Dom0 console.

Suravee


[-- Attachment #2: xen-seattle-4k.txt --]
[-- Type: text/plain, Size: 14954 bytes --]

Loading Xen image..
- UART enabled -
- CPU 00000000 booting -
- Current EL 00000008 -
- Xen starting at EL2 -
- Zero BSS -
- Setting up control registers -
- Turning on paging -
- Ready -
(XEN) Checking for initrd in /chosen
(XEN) RAM: 0000008000000000 - 00000083ffffffff
(XEN)
(XEN) MODULE[0]: 000000801fffe000 - 000000801ffffc79 Device Tree
(XEN) MODULE[1]: 0000008008800000 - 0000008009142a50 Kernel
(XEN)
(XEN) Command line: console=dtuart dtuart=serial0 noreboot sync_console
(XEN) Placing Xen at 0x00000083ffe00000-0x0000008400000000
(XEN) Update BOOTMOD_XEN from 0000008008700000-00000080087fad81 => 00000083ffe00000-00000083ffefad81
(XEN) Domain heap initialised
(XEN) Looking for UART console serial0
 Xen 4.5-unstable
(XEN) Xen version 4.5-unstable (ssuthiku@amd.com) (aarch64-linux-gnu-gcc (crosstool-NG linaro-1.13.1-4.8-2014.01 - Linaro GCC 2013.11) 4.8.3 20140106 (prerelease)) debug=y Wed Sep 10 17:12:52 CDT 2014
(XEN) Latest ChangeSet: Mon Aug 25 22:15:10 2014 -0500 git:8a26eb8           A0
(XEN) Console output is synchronous.
(XEN) Processor: 410fd070: "ARM Limited", variant: 0x0, part 0xd07, rev 0x0
(XEN) 64-bit Execution:
(XEN)   Processor Features: 0000000000002222 0000000000000000
(XEN)     Exception Levels: EL3:64+32 EL2:64+32 EL1:64+32 EL0:64+32
(XEN)     Extensions: FloatingPoint AdvancedSIMD
(XEN)   Debug Features: 0000000010305106 0000000000000000
(XEN)   Auxiliary Features: 0000000000000000 0000000000000000
(XEN)   Memory Model Features: 0000000000001124 0000000000000000
(XEN)   ISA Features:  0000000000011120 0000000000000000
(XEN) 32-bit Execution:
(XEN)   Processor Features: 00000131:00011011
(XEN)     Instruction Sets: AArch32 Thumb Thumb-2 Jazelle
(XEN)     Extensions: GenericTimer Security
(XEN)   Debug Features: 03010066
(XEN)   Auxiliary Features: 00000000
(XEN)   Memory Model Features: 10101105 40000000 01260000 02102211
(XEN)  ISA Features: 02101110 13112111 21232042 01112131 00011142 00011121
(XEN) Platform: SEATTLE
(XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27
(XEN) Using generic timer at 187500 KHz
(XEN) GICv2 initialization:
(XEN)         gic_dist_addr=00000000e1110000
(XEN)         gic_cpu_addr=00000000e112f000
(XEN)         gic_hyp_addr=00000000e1140000
(XEN)         gic_vcpu_addr=00000000e1160000
(XEN)         gic_maintenance_irq=24
(XEN) GICv2: 448 lines, 8 cpus, secure (IID 0200143b).
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) I/O virtualisation disabled
(XEN) Allocated console ring of 64 KiB.
(XEN) Bringing up CPU1
- CPU 00000201 booting -
- Current EL 00000008 -
- Xen starting at EL2 -
- Setting up control registers -
- Turning on paging -
- Ready -
(XEN) CPU 1 booted.
(XEN) Bringing up CPU2
- CPU 00000200 booting -
- Current EL 00000008 -
- Xen starting at EL2 -
- Setting up control registers -
- Turning on paging -
- Ready -
(XEN) CPU 2 booted.
(XEN) Bringing up CPU3
- CPU 00000101 booting -
- Current EL 00000008 -
- Xen starting at EL2 -
- Setting up control registers -
- Turning on paging -
- Ready -
(XEN) CPU 3 booted.
(XEN) Bringing up CPU4
- CPU 00000100 booting -
- Current EL 00000008 -
- Xen starting at EL2 -
- Setting up control registers -
- Turning on paging -
- Ready -
(XEN) CPU 4 booted.
(XEN) Bringing up CPU5
- CPU 00000001 booting -
- Current EL 00000008 -
- Xen starting at EL2 -
- Setting up control registers -
- Turning on paging -
- Ready -
(XEN) Brought up 6 CPUs
(XEN) CPU 5 booted.
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Loading kernel from boot module @ 0000008008800000
(XEN) Allocating 1:1 mappings totalling 128MB for dom0:
(XEN) BANK[0] 0x00008010000000-0x00008018000000 (128MB)
(XEN) Loading zImage from 0000008008800000 to 0000008010080000-00000080109c2a50
(XEN) Loading dom0 DTB to 0x0000008017e00000-0x0000008017e019eb
(XEN) Scrubbing Free RAM on 1 nodes using 6 CPUs
(XEN) ......................done.
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) **********************************************
(XEN) ******* WARNING: CONSOLE OUTPUT IS SYNCHRONOUS
(XEN) ******* This option is intended to aid debugging of Xen by ensuring
(XEN) ******* that all output is synchronously delivered on the serial line.
(XEN) ******* However it can introduce SIGNIFICANT latencies and affect
(XEN) ******* timekeeping. It is NOT recommended for production use!
(XEN) **********************************************
(XEN) 3... 2... 1...
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) Freed 252kB init memory.
(XEN) DOM0: Linux version 3.17.0-rc4-styx-xen+ (ssuthiku@sos-arm-dev) (gcc version 4.8.2 20140110 (prerelease) [ibm/gcc-4_8-branch merged
(XEN) DOM0: from gcc-4_8-branch, revision 205847] (Ubuntu/Linaro 4.8.2-13ubuntu1) ) #15 SMP Wed Sep 10 16:54:52 CDT 2014
(XEN) DOM0: CPU: AArch64 Processor [410fd070] revision 0
(XEN) DOM0: Detected PIPT I-cache on CPU0
(XEN) DOM0: Early serial console at I/O port 0x0 (options '')
(XEN) DOM0: bootconsole [uart0] enabled
(XEN) DOM0: efi: Getting EFI parameters from FDT:
(XEN) DOM0: efi: UEFI not found.
(XEN) DOM0: psci: probing for conduit method from DT.
(XEN) DOM0: psci: PSCIv0.2 detected in firmware.
(XEN) DOM0: psci: Using standard PSCI v0.2 function IDs
(XEN) DOM0: PERCPU: Embedded 12 pages/cpu @ffff800007fa7000 s18048 r8192 d22912 u49152
(XEN) DOM0: Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 32320
(XEN) DOM0: Kernel command line: console=ttyAMA0,115200 earlycon=pl011,0xe1010000 show_styx_info root=/dev/sda2 rootwait maxcpus=1
(XEN) DOM0: PID hash table entries: 512 (order: 0, 4096 bytes)
(XEN) DOM0: Dentry cache hash table entries: 16384 (order: 5, 131072 bytes)
(XEN) DOM0: Inode-cache hash table entries: 8192 (order: 4, 65536 bytes)
(XEN) DOM0: Memory: 106164K/131072K available (6470K kernel code, 502K rwdata, 2196K rodata, 305K init, 507K bss, 24908K reserved)
(XEN) DOM0: Virtual kernel memory layout:
(XEN) DOM0:     vmalloc : 0xffff000000000000 - 0xffff7c7fbfff0000   (127486 GB)
(XEN) DOM0:     vmemmap : 0xffff7c7fc0000000 - 0xffff7fffc0000000   (  3584 GB maximum)
(XEN) DOM0:               0xffff7c8180380000 - 0xffff7c8180540000   (     1 MB actual)
(XEN) DOM0:     PCI I/O : 0xffff7ffffa000000 - 0xffff7ffffb000000   (    16 MB)
(XEN) DOM0:     fixed   : 0xffff7ffffbdfe000 - 0xffff7ffffbdff000   (     4 KB)
(XEN) DOM0:     modules : 0xffff7ffffc000000 - 0xffff800000000000   (    64 MB)
(XEN) DOM0:     memory  : 0xffff800000000000 - 0xffff800008000000   (   128 MB)
(XEN) DOM0:       .init : 0xffff8000008f8000 - 0xffff800000944680   (   306 KB)
(XEN) DOM0:       .text : 0xffff800000080000 - 0xffff8000008f7184   (  8669 KB)
(XEN) DOM0:       .data : 0xffff800000945000 - 0xffff8000009c2a50   (   503 KB)
(XEN) DOM0: SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=6, Nodes=1
(XEN) DOM0: Hierarchical RCU implementation.
(XEN) DOM0:     RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=6.
(XEN) DOM0: RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=6
(XEN) DOM0: NR_IRQS:64 nr_irqs:64 0
(XEN) DOM0: Architected cp15 timer(s) running at 187.50MHz (virt).
(XEN) DOM0: sched_clock: 56 bits at 187MHz, resolution 5ns, wraps every 2932031094784ns
(XEN) DOM0: Calibrating delay loop (skipped), value calculated using timer frequency.. 375.00 BogoMIPS (lpj=1875000)
(XEN) DOM0: pid_max: default: 32768 minimum: 301
(XEN) DOM0: Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
(XEN) DOM0: Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes)
(XEN) DOM0: Initializing cgroup subsys perf_event
(XEN) DOM0: hw perfevents: enabled with arm/armv8-pmuv3 PMU driver, 1 counters available
(XEN) DOM0: EFI services will not be available.
(XEN) DOM0: Xen 4.5 support found, events_irq=31 gnttab_frame=0x00000000b0000000
(XEN) DOM0: xen:grant_table: Grant tables using version 1 layout
(XEN) DOM0: Grant table initialized
(XEN) DOM0: xen:events: Using FIFO-based ABI
(XEN) DOM0: Xen: initializing cpu0
(XEN) DOM0: Brought up 1 CPUs
(XEN) DOM0: SMP: Total of 1 processors activated.
(XEN) DOM0: devtmpfs: initialized
(XEN) DOM0: xor: measuring software checksum speed
(XEN) DOM0:    8regs     :  4656.400 MB/sec
(XEN) DOM0:    8regs_prefetch:  4202.800 MB/sec
(XEN) DOM0:    32regs    :  4238.400 MB/sec
(XEN) DOM0:    32regs_prefetch:  3563.600 MB/sec
(XEN) DOM0: xor: using function: 8regs (4656.400 MB/sec)
(XEN) DOM0: NET: Registered protocol family 16
(XEN) DOM0: vdso: 2 pages (1 code @ ffff80000094d000, 1 data @ ffff80000094c000)
(XEN) DOM0: hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
(XEN) DOM0: software IO TLB [mem 0x8016000000-0x8016400000] (4MB) mapped at [ffff800006000000-ffff8000063fffff]
(XEN) DOM0: software IO TLB [mem 0x8015c00000-0x8016000000] (4MB) mapped at [ffff800005c00000-ffff800005ffffff]
(XEN) DOM0: Serial: AMBA PL011 UART driver
(XEN) DOM0: raid6: int64x1    907 MB/s
(XEN) DOM0: raid6: int64x2   1326 MB/s
(XEN) DOM0: raid6: int64x4   1896 MB/s
(XEN) DOM0: raid6: int64x8   1783 MB/s
(XEN) DOM0: raid6: neonx1    1828 MB/s
(XEN) DOM0: raid6: neonx2    2532 MB/s
(XEN) DOM0: raid6: neonx4    2881 MB/s
(XEN) DOM0: raid6: neonx8    2951 MB/s
(XEN) DOM0: raid6: using algorithm neonx8 (2951 MB/s)
(XEN) DOM0: raid6: using intx1 recovery algorithm
(XEN) DOM0: xen:balloon: Initialising balloon driver
(XEN) DOM0: xen_balloon: Initialising balloon driver
(XEN) DOM0: vgaarb: loaded
(XEN) DOM0: SCSI subsystem initialized
(XEN) DOM0: ssp-pl022 e1020000.ssp: ARM PL022 driver, device ID: 0x00241022
(XEN) DOM0: ssp-pl022 e1020000.ssp: mapped registers from 0x00000000e1020000 to ffff000000010000
(XEN) DOM0: of_dma_request_slave_channel: dma-names property of node '/smb/motherboard/ssp@1,1020000' missing or empty
(XEN) DOM0: ssp-pl022 e1020000.ssp: Failed to work in dma mode, work without dma!
(XEN) DOM0: ssp-pl022 e1030000.ssp: ARM PL022 driver, device ID: 0x00241022
(XEN) DOM0: ssp-pl022 e1030000.ssp: mapped registers from 0x00000000e1030000 to ffff000000012000
(XEN) DOM0: of_dma_request_slave_channel: dma-names property of node '/smb/motherboard/ssp@1,1030000' missing or empty
(XEN) DOM0: ssp-pl022 e1030000.ssp: Failed to work in dma mode, work without dma!
(XEN) DOM0: pps_core: LinuxPPS API ver. 1 registered
(XEN) DOM0: pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
(XEN) DOM0: PTP clock support registered
(XEN) DOM0: Switched to clocksource arch_sys_counter
(XEN) DOM0: NET: Registered protocol family 2
(XEN) DOM0: TCP established hash table entries: 1024 (order: 1, 8192 bytes)
(XEN) DOM0: TCP bind hash table entries: 1024 (order: 2, 16384 bytes)
(XEN) DOM0: TCP: Hash tables configured (established 1024 bind 1024)
(XEN) DOM0: TCP: reno registered
(XEN) DOM0: UDP hash table entries: 256 (order: 1, 8192 bytes)
(XEN) DOM0: UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
(XEN) DOM0: NET: Registered protocol family 1
(XEN) DOM0: RPC: Registered named UNIX socket transport module.
(XEN) DOM0: RPC: Registered udp transport module.
(XEN) DOM0: RPC: Registered tcp transport module.
(XEN) DOM0: RPC: Registered tcp NFSv4.1 backchannel transport module.
(XEN) DOM0: futex hash table entries: 2048 (order: 5, 131072 bytes)
(XEN) DOM0: HugeTLB registered 2 MB page size, pre-allocated 0 pages
(XEN) DOM0: Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
(XEN) DOM0: JFS: nTxBlock = 829, nTxLock = 6635
(XEN) DOM0: SGI XFS with security attributes, no debug enabled
(XEN) DOM0: 9p: Installing v9fs 9p2000 file system support
(XEN) DOM0: msgmni has been set to 207
(XEN) DOM0: bounce: pool size: 64 pages
(XEN) DOM0: Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
(XEN) DOM0: io scheduler noop registered
(XEN) DOM0: io scheduler deadline registered
(XEN) DOM0: io scheduler cfq registered (default)
(XEN) DOM0: pl061_gpio e1040000.gpio: PL061 GPIO chip @0x00000000e1040000 registered
(XEN) DOM0: pl061_gpio e1050000.gpio: PL061 GPIO chip @0x00000000e1050000 registered
(XEN) DOM0: AMD-PCI DEBUG: amd_pcie_probe
(XEN) DOM0: AMD-PCI: amd_pcie_probe: matched STYX
(XEN) DOM0: AMD-PCI DEBUG: amd_pcie_parse_dt
(XEN) DOM0: amd-pcie pcie-controller: missing "reg" property
(XEN) DOM0: amd-pcie: probe of pcie-controller failed with error -22
(XEN) DOM0: dma-pl330: probe of e0500000.dma failed with error -5
(XEN) DOM0: xen:xen_evtchn: Event-channel device installed
(XEN) DOM0: Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
(XEN) DOM0: loop: module loaded
(XEN) DOM0: ahci e0300000.sata: SSS flag set, parallel bus scan disabled
(XEN) DOM0: ahci e0300000.sata: AHCI 0001.0300 32 slots 8 ports 6 Gbps 0xff impl platform mode
(XEN) DOM0: ahci e0300000.sata: flags: 64bit ncq sntf stag pm led clo only pmp pio slum part ccc apst
(XEN) DOM0: scsi host0: ahci_platform
(XEN) DOM0: scsi host1: ahci_platform
(XEN) DOM0: scsi host2: ahci_platform
(XEN) DOM0: scsi host3: ahci_platform
(XEN) DOM0: scsi host4: ahci_platform
(XEN) DOM0: scsi host5: ahci_platform
(XEN) DOM0: scsi host6: ahci_platform
(XEN) DOM0: scsi host7: ahci_platform
(XEN) DOM0: ata1: SATA max UDMA/133 mmio [mem 0xe0300000-0xe03007ff] port 0x100 irq 387
(XEN) DOM0: ata2: SATA max UDMA/133 mmio [mem 0xe0300000-0xe03007ff] port 0x180 irq 387
(XEN) DOM0: ata3: SATA max UDMA/133 mmio [mem 0xe0300000-0xe03007ff] port 0x200 irq 387
(XEN) DOM0: ata4: SATA max UDMA/133 mmio [mem 0xe0300000-0xe03007ff] port 0x280 irq 387
(XEN) DOM0: ata5: SATA max UDMA/133 mmio [mem 0xe0300000-0xe03007ff] port 0x300 irq 387
(XEN) DOM0: ata6: SATA max UDMA/133 mmio [mem 0xe0300000-0xe03007ff] port 0x380 irq 387
(XEN) DOM0: ata7: SATA max UDMA/133 mmio [mem 0xe0300000-0xe03007ff] port 0x400 irq 387
(XEN) DOM0: ata8: SATA max UDMA/133 mmio [mem 0xe0300000-0xe03007ff] port 0x480 irq 387
(XEN) DOM0: tun: Universal TUN/TAP device driver, 1.6
(XEN) DOM0: tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
(XEN) DOM0: libphy: amd_xgbe_phy: probed
(XEN) DOM0: amd-xgbe e0700000.xgmac eth0: net device enabled
(XEN) DOM0: libphy: amd_xgbe_phy: probed
(XEN) DOM0: amd-xgbe e0900000.xgmac eth1: net device enabled
(XEN) DOM0: xen_netfront: Initialising Xen virtual ethernet driver
(XEN) DOM0: VFIO - User Level meta-driver version: 0.3
(XEN) DOM0: device-mapper: ioctl: 4.27.0-ioctl (2013-10-30) initialised: dm-devel@redhat.com
(XEN) DOM0: device-mapper: multipath: version 1.7.0 loaded
(XEN) DOM0: device-mapper: multipath round-robin: version 1.0.0 loaded
(XEN) DOM0: mmc_spi spi32765.0: OF: voltage-ranges unspecified
(XEN) DOM0: mmc_spi spi32765.0: ASSUMING 3.2-3.4 V slot power
(XEN) DOM0: mmc_spi spi32765.0: SD/MMC host mmc0, no WP, no poweroff
(XEN) DOM0: TCP: cubic registered
(XEN) DOM0: NET: Registered protocol family 10
(XEN) DOM0: sit: IPv6 over IPv4 tunneling driver
(XEN) DOM0: NET: Registered protocol family 17
(XEN) DOM0: Bridge firewalling registered
(XEN) DOM0: 9pnet: Installing 9P2000 support
(XEN) DOM0: NET: Registered protocol family 37
(XEN) DOM0: bootconsole [uart0] disabled


[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel