From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:58732) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YVIom-0006NQ-39 for qemu-devel@nongnu.org; Tue, 10 Mar 2015 07:59:29 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YVIok-000793-GH for qemu-devel@nongnu.org; Tue, 10 Mar 2015 07:59:24 -0400 Message-ID: <54FECF87.6060406@redhat.com> Date: Tue, 10 Mar 2015 13:03:35 +0200 From: Marcel Apfelbaum MIME-Version: 1.0 References: <1425813387-31231-1-git-send-email-marcel@redhat.com> <1425887036.17517.17.camel@nilsson.home.kraxel.org> <54FD65F0.1040007@redhat.com> <1425896323.20588.15.camel@nilsson.home.kraxel.org> <54FD7442.5090508@redhat.com> <1425910792.20588.19.camel@nilsson.home.kraxel.org> <54FDC9B8.2000706@redhat.com> <1425920102.20588.48.camel@nilsson.home.kraxel.org> <54FDF2AE.4000109@redhat.com> <54FE8DF6.9040500@ozlabs.ru> In-Reply-To: <54FE8DF6.9040500@ozlabs.ru> Content-Type: text/plain; charset=koi8-r; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [Qemu-ppc] [PATCH v4 for-2.3 00/25] hw/pc: implement multiple primary busses for pc machines List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Alexey Kardashevskiy , Gerd Hoffmann Cc: pbonzini@redhat.com, quintela@redhat.com, seabios@seabios.org, qemu-devel@nongnu.org, alex.williamson@redhat.com, kevin@koconnor.net, mst@redhat.com, hare@suse.de, imammedo@redhat.com, amit.shah@redhat.com, qemu-ppc@nongnu.org, leon.alrae@imgtec.com, rth@twiddle.net On 03/10/2015 08:23 AM, Alexey Kardashevskiy wrote: > On 03/10/2015 06:21 AM, Marcel Apfelbaum wrote: >> On 03/09/2015 06:55 PM, Gerd Hoffmann wrote: >>> On Mo, 2015-03-09 at 18:26 +0200, Marcel Apfelbaum wrote: >>>> On 03/09/2015 04:19 PM, Gerd Hoffmann wrote: >>>>> Hi, >>>>> >>>>>> My series is based on commit 09d219a. Try please on top of this commit. >>>>> >>>>> Ok, that works. Going to play with that now ;) >>>> Good luck! ... and tell me what you think :) >>>> If you need any help with the command line of the pxb device, let me know,. >>> >>> First thing I've noticed: You need to define a numa node so you can >>> pass a valid numa node to the pxb-device. Guess that is ok as the whole >>> point of this is to assign pci devices to numa nodes. More complete >>> test instructions would be nice though. >> Exactly, this is by design. But you can also use it without specifying the >> NUMA node... >> >> A detailed command line would be: >> >> [qemu-bin + storage options] >> -bios [seabios-dir]/out/bios.bin -L [seabios-dir]/out/ >> -m 2G >> -object memory-backend-ram,size=1024M,policy=bind,host-nodes=0,id=ram-node0 >> -numa node,nodeid=0,cpus=0,memdev=ram-node0 >> -object >> memory-backend-ram,size=1024M,policy=interleave,host-nodes=0,id=ram-node1 >> -numa node,nodeid=1,cpus=1,memdev=ram-node1 >> -device pxb-device,id=bridge1,bus=pci.0,numa_node=1,bus_nr=4 -netdev >> user,id=nd-device e1000,bus=bridge1,addr=0x4,netdev=nd >> -device pxb-device,id=bridge2,bus=pci.0,numa_node=0,bus_nr=8 -device >> e1000,bus=bridge2,addr=0x3 >> -device pxb-device,id=bridge3,bus=pci.0,bus_nr=40 -drive >> if=none,id=drive0,file=[img] -device >> virtio-blk-pci,drive=drive0,scsi=off,bus=bridge3,addr=1 > > > I replayed this patchset on top of 09d219a "acpi: update generated files" and got this: > > > qemu-system-x86_64: -object memory-backend-ram,size=1024M,policy=bind,host-nodes=0,id=ram-node0: NUMA node binding are not supported by this QEMU > qemu-system-x86_64: -object memory-backend-ram,size=1024M,policy=interleave,host-nodes=0,id=ram-node1: NUMA node binding are not supported by this QEMU Hi, Please check your configuration (after you run ./configure script). See if you have a line like this: - NUMA host support yes > > > This is my exact command line: > > /scratch/alexey/p/qemu-build/x86_x86_64/x86_64-softmmu/qemu-system-x86_64 \ > -L /home/alexey/p/qemu/pc-bios/ \ > -hda x86/fc19_24GB_x86.qcow2 \ > -enable-kvm \ > -kernel x86/vmlinuz-3.12.11-201.fc19.x86_64 \ > -initrd x86/initramfs-3.12.11-201.fc19.x86_64.img \ > -append "root=/dev/sda3 console=ttyS0" \ > -nographic \ > -nodefaults \ > -chardev stdio,id=id2,signal=off,mux=on \ > -device isa-serial,id=id3,chardev=id2 \ > -mon id=id4,chardev=id2,mode=readline \ > -m 2G \ > -object memory-backend-ram,size=1024M,policy=bind,host-nodes=0,id=ram-node0 \ > -numa node,nodeid=0,cpus=0,memdev=ram-node0 \ > -object memory-backend-ram,size=1024M,policy=interleave,host-nodes=0,id=ram-node1 \ > -numa node,nodeid=1,cpus=1,memdev=ram-node1 \ > -device pxb-device,id=bridge1,bus=pci.0,numa_node=1,bus_nr=4 \ > -netdev user,id=nd-device e1000,bus=bridge1,addr=0x4,netdev=nd \ > -device pxb-device,id=bridge2,bus=pci.0,numa_node=0,bus_nr=8 \ > -device e1000,bus=bridge2,addr=0x3 \ > -device pxb-device,id=bridge3,bus=pci.0,bus_nr=40 \ > -drive if=none,id=drive0,file=debian_lenny_powerpc_desktop.qcow2 \ > -device virtio-blk-pci,drive=drive0,scsi=off,bus=bridge3,addr=1 \ > > > What am I missing here? See above, check for NUMA host support > > > What I actually wanted to find out (instead of asking what I am doing now) is is this PXB device a PCI device sitting on the same PCI host bus adapter (1) or it is a separate PHB (2) with its own PCI > domain (new XXXX in XXXX:00:00.0 PCI address)? I would think it is (1) but then what exactly do you call "A primary PCI bus" here (that's my ignorance speaking, yes :) )? Thanks. You are right, the PXB is a device on the piix host-bridge bus and its bus uses the same PCI domain. However, the bus behind is exposed through ACPI as Primary PCI bus and starts a new PCI hierarchy. You have a similar approach on Intel 450x chipset: http://www.intel.com/design/chipsets/datashts/243771.htm Look for 82454NX PCI Expander Bridge (PXB) Thanks, Marcel > > > >> Here you have: >> - 2 NUMA nodes for the guest, 0 and 1. (both mapped to the same NUMA node >> in host, but you can and should put it in different host NUMA nodes) >> - a pxb host bridge attached to NUMA 1 with an e1000 behind it >> - a pxb host bridge attached to NUMA 0 with an e1000 behind it >> - a pxb host bridge not attached to any NUMA with a hard drive behind it. >> >> As you can see, since you already "decide" NUMA mapping at command line, it >> is "natural" also to attach the pxbs to the NUMA nodes. >> >>> >>> Second thing: Booting with an unpatched seabios has bad effects: >>> >>> [root@localhost ~]# cat /proc/iomem >>> 00000000-000fffff : PCI Bus 0000:10 >>> 00000000-00000fff : reserved >>> 00001000-0009fbff : System RAM >>> 0009fc00-0009ffff : reserved >>> 000c0000-000c91ff : Video ROM >>> 000c9800-000ca1ff : Adapter ROM >>> 000ca800-000ccbff : Adapter ROM >>> 000f0000-000fffff : reserved >>> 000f0000-000fffff : System ROM >>> 00100000-3ffdffff : System RAM >>> 01000000-0174bde4 : Kernel code >>> 0174bde5-01d30cff : Kernel data >>> 01eaa000-0202afff : Kernel bss >>> 3ffe0000-3fffffff : reserved >>> fd000000-fdffffff : 0000:00:02.0 >>> fd000000-fdffffff : bochs-drm >>> febc0000-febdffff : 0000:00:03.0 >>> febc0000-febdffff : e1000 >>> febf0000-febf0fff : 0000:00:02.0 >>> febf0000-febf0fff : bochs-drm >>> fec00000-fec003ff : IOAPIC 0 >>> fed00000-fed003ff : HPET 0 >>> fed00000-fed003ff : PNP0103:00 >>> fee00000-fee00fff : Local APIC >>> feffc000-feffffff : reserved >>> fffc0000-ffffffff : reserved >>> >>> "PCI Bus 0000:10" is bogus and "PCI Bus 0000:00" isn't there at all. >> Yes, you shouldn't use pxb if you are not using the corresponding SeaBIOS. >> However, as I understand we always attach a SeaBIOS binary with a QEMU >> release, >> so we should be OK with this. >> >> And this is the reason I wanted bios support *before* the PXB device >> implementation, >> but anyway, even if we have them in the same time, as long as the release >> has both pxb and BIOS with pxb support, is OK. (I think...) >> >> I appreciate you looking into this and if you need further assistance >> don't hesitate to mail me! :) >> >> Thanks, >> Marcel >> >>> >>> cheers, >>> Gerd >>> >>> >> >> > >