Re: ARM64:Porting xen to new hardware

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Fri, Sep 08, 2017 at 10:19:55PM +0300, Oleksandr Tyshchenko wrote:
> Hi Bharat
> 
> On Thu, Sep 7, 2017 at 4:30 PM, bharat gohil <ghl.bhrt@gmail.com> wrote:
> > Hello Olensandr,
> >
> > I able to boot xen and trying to boot dom0 but there are no console log for
> > dom0.
> >
> > following log for xen and it stuck booting dom0.
> >
> > (XEN) I/O virtualisation disabled
> > (XEN) build-id: 7c2a3c70fb94754801d18c4cb9e3db3ffa01d8c4
> > (XEN) alternatives: Patching with alt table 00000000400d2e08 ->
> > 00000000400d32dc
> > (XEN) *** LOADING DOMAIN 0 ***
> > (XEN) Loading kernel from boot module @ 0000000040148158
> > (XEN) Allocating 1:1 mappings totalling 128MB for dom0:
> > (XEN) BANK[0] 0x00000048000000-0x00000050000000 (128MB)
> > (XEN) Grant table range: 0x000000bfe00000-0x000000bfe65000
> > (XEN) Loading zImage from 0000000040148158 to
> > 0000000048080000-0000000049480000
> > (XEN) Allocating PPI 16 for event channel interrupt
> > (XEN) Loading dom0 DTB to 0x000000004fe00000-0x000000004fe0f31e
> > (XEN) Scrubbing Free RAM on 1 nodes using 3 CPUs
> > (XEN) ......done.
> > (XEN) Initial low memory virq threshold set at 0x4000 pages.
> > (XEN) Std. Loglevel: All
> > (XEN) Guest Loglevel: All
> > (XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to
> > Xen)
> > (XEN) Freed 272kB init memory.
> >
> > I have done all the xen configuration in linux kernel 4.9. This kernel
> > booting fine without xen.
> >
> > following are the DTB changes,
> >
> >     chosen {
> >         #address-cells = <1>;
> >         #size-cells = <1>;
> >         bootargs = "console=dtuart dtuart=serial0 dom0_mem=128M";
> >         stdout-path = "serial0";
> >         module: module@0 {
> >             compatible = "xen,linux-zimage", "xen,multiboot-module";
> >             reg = <0x40148158 0x1400000>;
> >             bootargs = "console=hvc0,921600n8 earlyprintk=xen debug

It should be just 'console=hvc0', not 'console=hvc0,921600n8'

> > ignore_loglevel rw root=/dev/mmcblk0p7";
> >         };
> >
> >     };
> >
> > Can you tell me how to debug dom0 booting or anything which i can check?
> 
> Don't now much about "debug dom0 booting", I leave it for competent people.
> 
> Looks weird, even with earlyprintk no logs.
> Do you have DEBUG_LL and all related options enabled in your dom0 kernel config?
> 
> 1. Check that following options are enabled in your kernel config file:
> 
> CONFIG_HVC_XEN=y
> CONFIG_HVC_XEN_FRONTEND=y
> 
> 2. Check that dom0 kernel doesn't disable clock for console.
> 
> BTW, could you post full Xen log, kernel config and device-tree you are using?
> If you have some changes on top of Xen, post them too.
> These may help people to identify what is wrong.
> 
> >
> >
> > Thanks,
> > Bharat
> >
> > On Wed, Sep 6, 2017 at 3:49 PM, Oleksandr Tyshchenko <olekstysh@gmail.com>
> > wrote:
> >>
> >> Hi Bharat
> >>
> >> On Wed, Sep 6, 2017 at 10:01 AM, bharat gohil <ghl.bhrt@gmail.com> wrote:
> >> > Hello Oleksandr,
> >> >
> >> > Thank you very much.It resolved my issue.
> >> Sounds great!
> >>
> >> >
> >> > Thanks,
> >> > Bharat
> >> >
> >> > On Mon, Sep 4, 2017 at 6:24 PM, Oleksandr Tyshchenko
> >> > <olekstysh@gmail.com>
> >> > wrote:
> >> >>
> >> >> Hi Bharat
> >> >>
> >> >> On Mon, Sep 4, 2017 at 7:13 AM, bharat gohil <ghl.bhrt@gmail.com>
> >> >> wrote:
> >> >> > Hello Oleksandr,
> >> >> >
> >> >> > I have corrected  GIC settings but no success.Following line
> >> >> > disappear
> >> >> > from
> >> >> > log.
> >> >> >>>XEN) GICv2: WARNING: The GICC size is too small: 0x1000 expected
> >> >> >>> 0x2000
> >> >> >
> >> >> > Is anything else which can I try.
> >> >> >
> >> >> > I don’t know much about xen internal for ARM architecture. As you
> >> >> > mentioned,
> >> >> >>>Wrong GIC settings might lead to that IPIs won't work as expected.
> >> >> >>> And
> >> >> >>>boot CPU will get stuck waiting for another CPU.
> >> >> >
> >> >> > Can you explain it with some boot sequence and relation with IPI?
> >> >>
> >> >> Well, we faced similar issue with R-Car Gen3 H3 SoC. Xen hung at
> >> >> smp_call_function (one CPU didn't receive interrupt from another one).
> >> >> Next patch helped us to fix this issue:
> >> >> https://patchwork.kernel.org/patch/9163065/
> >> >>
> >> >> I assume the SoC you are working with has "arm,gic-400" compatible GIC.
> >> >> Can you take a look at the patch, maybe it is your case too.
> >> >>
> >> >> >
> >> >> > Thanks,
> >> >> > Bharat
> >> >> >
> >> >> >
> >> >> > On Thu, Aug 31, 2017 at 5:28 PM, Oleksandr Tyshchenko
> >> >> > <olekstysh@gmail.com>
> >> >> > wrote:
> >> >> >>
> >> >> >> On Thu, Aug 31, 2017 at 2:13 PM, bharat gohil <ghl.bhrt@gmail.com>
> >> >> >> wrote:
> >> >> >> > Hello Oleksandr,
> >> >> >> Hi Bharat
> >> >> >>
> >> >> >> >
> >> >> >> > I had removed A72 cluster and tried to boot only two A35 but I got
> >> >> >> > same
> >> >> >> > error.
> >> >> >> >
> >> >> >> > Is anything added or missing in A35 compare to A53?
> >> >> >> Unfortunately, I don't know.
> >> >> >>
> >> >> >> BTW, did you check your GIC settings in the device-tree?
> >> >> >>
> >> >> >> >
> >> >> >> > Regards,
> >> >> >> > Bharat
> >> >> >> >
> >> >> >> > On Wed, Aug 30, 2017 at 8:00 PM, bharat gohil <ghl.bhrt@gmail.com>
> >> >> >> > wrote:
> >> >> >> >>
> >> >> >> >> Hello Oleksandr,
> >> >> >> >> Thank you very much for your input.
> >> >> >> >>
> >> >> >> >> Yes. agree. I will check by removing A72 core from DT.
> >> >> >> >>
> >> >> >> >> Thanks,
> >> >> >> >> Bharat
> >> >> >> >>
> >> >> >> >> On Wed, Aug 30, 2017 at 7:44 PM, Oleksandr Tyshchenko
> >> >> >> >> <olekstysh@gmail.com> wrote:
> >> >> >> >>>
> >> >> >> >>> Hi,
> >> >> >> >>>
> >> >> >> >>> Not sure that I am a competent person, just my assumptions.
> >> >> >> >>>
> >> >> >> >>> CCed ARM guys.
> >> >> >> >>>
> >> >> >> >>> On Tue, Aug 29, 2017 at 5:21 PM, bharat gohil
> >> >> >> >>> <ghl.bhrt@gmail.com>
> >> >> >> >>> wrote:
> >> >> >> >>> > Hello All
> >> >> >> >>> >
> >> >> >> >>> > I am trying to run Xen on new hardware which has two A35 and
> >> >> >> >>> > one
> >> >> >> >>> > A72
> >> >> >> >>> > core.
> >> >> >> >>> > Xen booted intially but it hangs at
> >> >> >> >>> > smp_call_function(setup_virt_paging_one,
> >> >> >> >>> > (void *)val, 1) function call.
> >> >> >> >>>
> >> >> >> >>> It might be a consequence of that CPU cores are different. And
> >> >> >> >>> they
> >> >> >> >>> might have different set of features, or even settings.
> >> >> >> >>> And these features/settings the boot CPU has don't compatible
> >> >> >> >>> with
> >> >> >> >>> other (non-boot) CPUs.
> >> >> >> >>> Can you try not to bringup A72 core (remove it from DT or
> >> >> >> >>> another
> >> >> >> >>> way), leave only two A35 and see what will happen.
> >> >> >> >>>
> >> >> >> >>> > Find following log of Xen booting,same set of features.
> >> >> >> >>> >
> >> >> >> >>> > - UART enabled -
> >> >> >> >>> > - CPU 00000000 booting -
> >> >> >> >>> > - Current EL 00000008 -
> >> >> >> >>> > - Xen starting at EL2 -
> >> >> >> >>> > - Zero BSS -
> >> >> >> >>> > - Setting up control registers -
> >> >> >> >>> > - Turning on paging -
> >> >> >> >>> > - Ready -
> >> >> >> >>> > (XEN) Checking for initrd in /chosen
> >> >> >> >>> > (XEN) RAM: 0000000040000000 - 00000000bfffffff
> >> >> >> >>> > (XEN)
> >> >> >> >>> > (XEN) MODULE[0]: 0000000044000000 - 000000004400fd5a Device
> >> >> >> >>> > Tree
> >> >> >> >>> > (XEN)
> >> >> >> >>> > (XEN) Command line: <NULL>
> >> >> >> >>> Why? Does your device-tree have bootargs?
> >> >> >> >>>
> >> >> >> >>> > (XEN) Placing Xen at 0x00000000bfe00000-0x00000000c0000000
> >> >> >> >>> > (XEN) Update BOOTMOD_XEN from
> >> >> >> >>> > 0000000040080000-0000000040194e01
> >> >> >> >>> > =>
> >> >> >> >>> > 00000000bfe01
> >> >> >> >>> > (XEN) Domain heap initialised
> >> >> >> >>> > (XEN) Booting using Device Tree
> >> >> >> >>> > (XEN) Platform: Generic System
> >> >> >> >>> > (XEN) Taking dtuart configuration from /chosen/stdout-path
> >> >> >> >>> > (XEN) Looking for dtuart at "serial0", options ""
> >> >> >> >>> >  __  __            _  _    _  ___                     _
> >> >> >> >>> > _
> >> >> >> >>> > _
> >> >> >> >>> >  \ \/ /___ _ __   | || |  / |/ _ \    _   _ _ __  ___| |_ __
> >> >> >> >>> > _|
> >> >> >> >>> > |__
> >> >> >> >>> > | |
> >> >> >> >>> > ___
> >> >> >> >>> >   \  // _ \ '_ \  | || |_ | | | | |__| | | | '_ \/ __| __/ _`
> >> >> >> >>> > |
> >> >> >> >>> > '_
> >> >> >> >>> > \|
> >> >> >> >>> > |/ _ \
> >> >> >> >>> >   /  \  __/ | | | |__   _|| | |_| |__| |_| | | | \__ \ || (_|
> >> >> >> >>> > |
> >> >> >> >>> > |_)
> >> >> >> >>> > | |
> >> >> >> >>> > __/
> >> >> >> >>> >  /_/\_\___|_| |_|    |_|(_)_|\___/    \__,_|_|
> >> >> >> >>> > |_|___/\__\__,_|_.__/|_|\___|
> >> >> >> >>> >
> >> >> >> >>> > (XEN) Xen version 4.10-unstable (bgohil@)
> >> >> >> >>> > (aarch64-linux-gnu-gcc
> >> >> >> >>> > (Ubuntu/Linaro7
> >> >> >> >>> > (XEN) Latest ChangeSet: Fri Aug 11 19:02:51 2017 +0100
> >> >> >> >>> > git:9053a74-dirty
> >> >> >> >>> > (XEN) Processor: 410fd041: "ARM Limited", variant: 0x0, part
> >> >> >> >>> > 0xd04,
> >> >> >> >>> > rev
> >> >> >> >>> > 0x1
> >> >> >> >>> > (XEN) 64-bit Execution:
> >> >> >> >>> > (XEN)   Processor Features: 0000000000002222 0000000000000000
> >> >> >> >>> > (XEN)     Exception Levels: EL3:64+32 EL2:64+32 EL1:64+32
> >> >> >> >>> > EL0:64+32
> >> >> >> >>> > (XEN)     Extensions: FloatingPoint AdvancedSIMD
> >> >> >> >>> > (XEN)   Debug Features: 0000000010305106 0000000000000000
> >> >> >> >>> > (XEN)   Auxiliary Features: 0000000000000000 0000000000000000
> >> >> >> >>> > (XEN)   Memory Model Features: 0000000000101122
> >> >> >> >>> > 0000000000000000
> >> >> >> >>> > (XEN)   ISA Features:  0000000000011120 0000000000000000
> >> >> >> >>> > (XEN) 32-bit Execution:
> >> >> >> >>> > (XEN)   Processor Features: 00000131:00011011
> >> >> >> >>> > (XEN)     Instruction Sets: AArch32 A32 Thumb Thumb-2 Jazelle
> >> >> >> >>> > (XEN)     Extensions: GenericTimer Security
> >> >> >> >>> > (XEN)   Debug Features: 03010066
> >> >> >> >>> > (XEN)   Auxiliary Features: 00000000
> >> >> >> >>> > (XEN)   Memory Model Features: 10201105 40000000 01260000
> >> >> >> >>> > 02102211
> >> >> >> >>> > (XEN)  ISA Features: 02101110 13112111 21232042 01112131
> >> >> >> >>> > 00011142
> >> >> >> >>> > 00011121
> >> >> >> >>> > (XEN) Using PSCI-1.0 for SMP bringup
> >> >> >> >>> > (XEN) SMP: Allowing 3 CPUs
> >> >> >> >>> > (XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 13000
> >> >> >> >>> > KHz
> >> >> >> >>> > (XEN) GICv2: WARNING: The GICC size is too small: 0x1000
> >> >> >> >>> > expected
> >> >> >> >>> > 0x2000
> >> >> >> >>> Sounds like GIC settings are not completely correct.
> >> >> >> >>> Wrong GIC settings might lead to that IPIs won't work as
> >> >> >> >>> expected.
> >> >> >> >>> And
> >> >> >> >>> boot CPU will
> >> >> >> >>> get stuck waiting for another CPU.
> >> >> >> >>> Just double check.
> >> >> >> >>>
> >> >> >> >>> > (XEN) GICv2 initialization:
> >> >> >> >>> > (XEN)         gic_dist_addr=0000000010510000
> >> >> >> >>> > (XEN)         gic_cpu_addr=0000000010520000
> >> >> >> >>> > (XEN)         gic_hyp_addr=0000000010540000
> >> >> >> >>> > (XEN)         gic_vcpu_addr=0000000010560000
> >> >> >> >>> > (XEN)         gic_maintenance_irq=25
> >> >> >> >>> > (XEN) GICv2: 384 lines, 6 cpus, secure (IID 0200143b).
> >> >> >> >>> > (XEN) Using scheduler: SMP Credit Scheduler (credit)
> >> >> >> >>> > (XEN) Allocated console ring of 32 KiB.
> >> >> >> >>> > (XEN) Bringing up CPU1
> >> >> >> >>> > - CPU 00000001 booting -
> >> >> >> >>> > - Current EL 00000008 -
> >> >> >> >>> > - Xen starting at EL2 -
> >> >> >> >>> > - Setting up control registers -
> >> >> >> >>> > - Turning on paging -
> >> >> >> >>> > - Ready -
> >> >> >> >>> > (XEN) CPU 1 booted.
> >> >> >> >>> > (XEN) Bringing up CPU2
> >> >> >> >>> > - CPU 00000200 booting -
> >> >> >> >>> > - Current EL 00000008 -
> >> >> >> >>> > - Xen starting at EL2 -
> >> >> >> >>> > - Setting up control registers -
> >> >> >> >>> > - Turning on paging -
> >> >> >> >>> > - Ready -
> >> >> >> >>> > (XEN) CPU 2 booted.
> >> >> >> >>> > (XEN) Brought up 3 CPUs
> >> >> >> >>> > (XEN) P2M: 40-bit IPA with 40-bit PA and 8-bit VMID
> >> >> >> >>> > (XEN) P2M: 3 levels with order-1 root, VTCR 0x80023558
> >> >> >> >>> >
> >> >> >> >>> > Can anyone guide me how to debug this problem or what could be
> >> >> >> >>> > wrong
> >> >> >> >>> > here?
> >> >> >> >>> >
> >> >> >> >>> > It looks, writing into VTCR_EL2 hang the system.
> >> >> >> >>> >
> >> >> >> >>> > --
> >> >> >> >>> > Regards,
> >> >> >> >>> > Bharat Gohil
> >> >> >> >>> >
> >> >> >> >>> >
> >> >> >> >>> > _______________________________________________
> >> >> >> >>> > Xen-devel mailing list
> >> >> >> >>> > Xen-devel@lists.xen.org
> >> >> >> >>> > https://lists.xen.org/xen-devel
> >> >> >> >>> >
> >> >> >> >>>
> >> >> >> >>> --
> >> >> >> >>> Regards,
> >> >> >> >>>
> >> >> >> >>> Oleksandr Tyshchenko
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> --
> >> >> >> >> Regards,
> >> >> >> >> Bharat Gohil
> >> >> >> >> Sr.Software Engineer
> >> >> >> >> bharat.gohil@harman.com
> >> >> >> >> +919427054633
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > --
> >> >> >> > Regards,
> >> >> >> > Bharat Gohil
> >> >> >> > Sr.Software Engineer
> >> >> >> > bharat.gohil@harman.com
> >> >> >> > +919427054633
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> Regards,
> >> >> >>
> >> >> >> Oleksandr Tyshchenko
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Regards,
> >> >> > Bharat Gohil
> >> >> > Sr.Software Engineer
> >> >> > bharat.gohil@harman.com
> >> >> > +919427054633
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Regards,
> >> >>
> >> >> Oleksandr Tyshchenko
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Regards,
> >> > Bharat Gohil
> >> > Sr.Software Engineer
> >> > bharat.gohil@harman.com
> >> > +919427054633
> >>
> >>
> >>
> >> --
> >> Regards,
> >>
> >> Oleksandr Tyshchenko
> >
> >
> >
> >
> > --
> > Regards,
> > Bharat Gohil
> > Sr.Software Engineer
> > bharat.gohil@harman.com
> > +919427054633
> 
> 
> 
> -- 
> Regards,
> 
> Oleksandr Tyshchenko
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel