* lpar issue for ZONE_DEVICE p2pmem in 4.14-rc @ 2017-10-17 20:38 Stephen Bates 2017-10-17 21:29 ` Balbir Singh 0 siblings, 1 reply; 8+ messages in thread From: Stephen Bates @ 2017-10-17 20:38 UTC (permalink / raw) To: linuxppc-dev@lists.ozlabs.org; +Cc: Logan Gunthorpe SGkgQWxsDQoNCkkgYW0gaG9waW5nIHNvbWVvbmUgY2FuIGhlbHAgc2hlZCBzb21lIGxpZ2h0IG9u IGFuIGlzc3VlIEkgYW0gc2VlaW5nIHdpdGggbXkgYXR0ZW1wdCB0byBhZGQgcDJwbWVtIFsxXSB0 byB0aGUgcHBjNjRlbCBrZXJuZWwuIHAycG1lbSBpcyBhIChjdXJyZW50bHkpIG91dC1vZi10cmVl IHBhdGNoc2V0IHRoYXQgYWxsb3dzIG9uZSB0byBhZGQgZGV2aWNlIG1lbW9yeSB3aXRoIHN0cnVj dCBwYWdlIGJhY2tpbmcgaW50byB0aGUgTGludXgga2VybmVsLiBJdCBsZXZlcmFnZXMgTUVNT1JZ X0hPVFBMVUcgYW5kIFpPTkVfREVWSUNFIHdoaWNoIHdlcmUgYWRkZWQgdG8gcHBjNjQgaW4gNC4x NCBzbyBJIHRob3VnaHQgaXQgd291bGQgYmUgZnVuIHRvIHRyeSBpdCBvdXQuDQoNCldlIGNvbnN0 cnVjdGVkIGEgcGF0Y2hzZXQgYmFzZWQgb2ZmIDQuMTQtcmMxIFsxXSBhbmQgYnVpbGQgYSBrZXJu ZWwgYmFzZWQgb2ZmIHRoZSBwc2VyaWVzIGRlZmNvbmZpZyBhbmQgcmFuIHRoaXMgb24gdXBzdHJl YW0gcWVtdS1zeXN0ZW0tcHBjNjQuIFRoZSBleGFjdCBjb21tYW5kIHRvIHJ1biBRRU1VIHdhczoN Cg0KcWVtdS1zeXN0ZW0tcHBjNjQgXA0KICAgIC1tYWNoaW5lIHBzZXJpZXMgXA0KICAgIC1jcHUg cG93ZXI4IFwNCiAgICAtc21wIDEgLW0gMjA0OCBcDQogICAgLWtlcm5lbCB+L2tlcm5lbC9saW51 eC1wcGM2NGVsL3ZtbGludXggXA0KICAgIC1hcHBlbmQgIm52bWUudXNlX2NtYj0yNCBjb25zb2xl PWh2YyByb290PS9kZXYvc2RhIHJvb3R3YWl0IHJ3IiBcDQogICAgLXNlcmlhbCBtb246c3RkaW8g LWRyaXZlIGZpbGU9aW1hZ2UtcHBjNjRlbC5pbWcsaWY9c2NzaSxmb3JtYXQ9cmF3LGluZGV4PTAg XA0KICAgIC1ub2dyYXBoaWMgXA0KICAgIC1kcml2ZSBmaWxlPS4uL2ltYWdlcy9udm1lLnFjb3cy LGlmPW5vbmUsaWQ9bnZtZTEsc25hcHNob3Q9b24gXA0KICAgIC1kZXZpY2UgbnZtZSxkcml2ZT1u dm1lMSxzZXJpYWw9bnZtZTEgXA0KICAgIC1kcml2ZSBmaWxlPS4uL2ltYWdlcy9udm1lMi5xY293 MixpZj1ub25lLGlkPW52bWUyLHNuYXBzaG90PW9uIFwNCiAgICAtZGV2aWNlIG52bWUsZHJpdmU9 bnZtZTIsc2VyaWFsPW52bWUyLGNtYl9zaXplX21iPTY0DQoNClRoaXMgcmVzdWx0ZWQgaW4gdGhl IGZvbGxvd2luZyBleHRyYWN0IGZyb20gZG1lc2cgd2hlbiByZWdpc3RlcmluZyB0aGUgcDJwbWVt IGFzc29jaWF0ZWQgd2l0aCBvbmUgb2YgdGhlIE5WTWUgU1NEcy4NCg0KWyAgICAzLjUwODQ5N10g bnZtZSAwMDAwOjAwOjAzLjA6IGVuYWJsaW5nIGRldmljZSAoMDEwMCAtPiAwMTAyKQ0KWyAgICAz LjUxMDc0M10gbnZtZSAwMDAwOjAwOjAzLjA6IFVzaW5nIDY0LWJpdCBkaXJlY3QgRE1BIGF0IG9m ZnNldCA4MDAwMDAwMDAwMDAwMDANClsgICAgMy41MzU3MDZdIHAycG1lbSBwMnBtZW0wOiByZWdp c3RlcmVkDQpbICAgIDMuNTM3NzgwXSBscGFyOiBBdHRlbXB0aW5nIHRvIHJlc2l6ZSBIUFQgdG8g c2hpZnQgMjENClsgICAgMy41MzkyNTFdIFVuYWJsZSB0byByZXNpemUgaGFzaCBwYWdlIHRhYmxl IHRvIHRhcmdldCBvcmRlciAyMTogLTENClsgICAgMy41NDEwNzldIFVuYWJsZSB0byBjcmVhdGUg bWFwcGluZyBmb3IgaG90IGFkZGVkIG1lbW9yeSAweGMwMDAyMTAwMDAwMDAwMDAuLjB4YzAwMDIx MDAwNDAwMDAwMDogLTINClsgICAgMy41NTA0MDddIHAycG1lbSBwMnBtZW0wOiB1bnJlZ2lzdGVy ZWQNCg0KSSBhbSBub3QgdGhhdCBmYW1pbGlhciB3aXRoIHRoZSBwc2VyaWVzIGFyY2hpdGVjdHVy ZSBzbyBJIHdhcyBob3BpbmcgZm9yIHNvbWUgZ3VpZGFuY2UgY29uY2VybmluZyB0aGUgbHBhciBl cnJvciBtZXNzYWdlLiBJZiBhbnkgcHBjNjQgY29kZXJzIGZhbmN5IGhhdmluZyBhIGdvIGF0IHRo aXMgdGhlIGtlcm5lbCBjb2RlIGlzIGF0IFsxXSBhbmQgSeKAmWQgYmUgaGFwcHkgdG8gcHJvdmlk ZSB0aGUgLmNvbmZpZyB1c2VkIGlmIG5lZWRlZC4gSnVzdCBsZXQgbWUga25vdy4NCg0KQ2hlZXJz DQogDQpTdGVwaGVuDQoNClsxXSBodHRwczovL2dpdGh1Yi5jb20vc2JhdGVzMTMwMjcyL2xpbnV4 LXAycG1lbQ0KDQoNCg== ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: lpar issue for ZONE_DEVICE p2pmem in 4.14-rc 2017-10-17 20:38 lpar issue for ZONE_DEVICE p2pmem in 4.14-rc Stephen Bates @ 2017-10-17 21:29 ` Balbir Singh 2017-10-21 15:03 ` Stephen Bates 0 siblings, 1 reply; 8+ messages in thread From: Balbir Singh @ 2017-10-17 21:29 UTC (permalink / raw) To: Stephen Bates; +Cc: linuxppc-dev@lists.ozlabs.org, Logan Gunthorpe On Wed, Oct 18, 2017 at 7:38 AM, Stephen Bates <sbates@raithlin.com> wrote= : > Hi All > > I am hoping someone can help shed some light on an issue I am seeing with= my attempt to add p2pmem [1] to the ppc64el kernel. p2pmem is a (currently= ) out-of-tree patchset that allows one to add device memory with struct pag= e backing into the Linux kernel. It leverages MEMORY_HOTPLUG and ZONE_DEVIC= E which were added to ppc64 in 4.14 so I thought it would be fun to try it = out. > > We constructed a patchset based off 4.14-rc1 [1] and build a kernel based= off the pseries defconfig and ran this on upstream qemu-system-ppc64. The = exact command to run QEMU was: > > qemu-system-ppc64 \ > -machine pseries \ > -cpu power8 \ > -smp 1 -m 2048 \ > -kernel ~/kernel/linux-ppc64el/vmlinux \ > -append "nvme.use_cmb=3D24 console=3Dhvc root=3D/dev/sda rootwait rw"= \ > -serial mon:stdio -drive file=3Dimage-ppc64el.img,if=3Dscsi,format=3D= raw,index=3D0 \ > -nographic \ > -drive file=3D../images/nvme.qcow2,if=3Dnone,id=3Dnvme1,snapshot=3Don= \ > -device nvme,drive=3Dnvme1,serial=3Dnvme1 \ > -drive file=3D../images/nvme2.qcow2,if=3Dnone,id=3Dnvme2,snapshot=3Do= n \ > -device nvme,drive=3Dnvme2,serial=3Dnvme2,cmb_size_mb=3D64 > > This resulted in the following extract from dmesg when registering the p2= pmem associated with one of the NVMe SSDs. > > [ 3.508497] nvme 0000:00:03.0: enabling device (0100 -> 0102) > [ 3.510743] nvme 0000:00:03.0: Using 64-bit direct DMA at offset 80000= 0000000000 > [ 3.535706] p2pmem p2pmem0: registered > [ 3.537780] lpar: Attempting to resize HPT to shift 21 > [ 3.539251] Unable to resize hash page table to target order 21: -1 I am guessing that the hotplug of ZONE_DEVICE memory was done incorrectly as it lead to HPT resizing (the system thinking this is normal memory). Ideally one would expect that the driver would online ZONE_DEVICE memory and not go through the HOTPLUG path. Are you using devm_memremap_pages() path to add these pages? Balbir Singh. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: lpar issue for ZONE_DEVICE p2pmem in 4.14-rc 2017-10-17 21:29 ` Balbir Singh @ 2017-10-21 15:03 ` Stephen Bates 2017-10-23 2:53 ` Balbir Singh 0 siblings, 1 reply; 8+ messages in thread From: Stephen Bates @ 2017-10-21 15:03 UTC (permalink / raw) To: Balbir Singh; +Cc: linuxppc-dev@lists.ozlabs.org, Logan Gunthorpe DQo+IEkgYW0gZ3Vlc3NpbmcgdGhhdCB0aGUgaG90cGx1ZyBvZiBaT05FX0RFVklDRSBtZW1vcnkg d2FzIGRvbmUNCj4gaW5jb3JyZWN0bHkgYXMgaXQgbGVhZCB0byBIUFQgcmVzaXppbmcgKHRoZSBz eXN0ZW0gdGhpbmtpbmcgdGhpcyBpcw0KPiBub3JtYWwgbWVtb3J5KS4gSWRlYWxseSBvbmUgd291 bGQgZXhwZWN0IHRoYXQgdGhlIGRyaXZlciB3b3VsZCBvbmxpbmUNCj4gWk9ORV9ERVZJQ0UgbWVt b3J5IGFuZCBub3QgZ28gdGhyb3VnaCB0aGUgSE9UUExVRyBwYXRoLiBBcmUgeW91IHVzaW5nDQo+ IGRldm1fbWVtcmVtYXBfcGFnZXMoKSBwYXRoIHRvIGFkZCB0aGVzZSBwYWdlcz8NCj4NCg0KVGhh bmtzIGZvciB0aGUgcmVzcG9uc2UgQmFsYmlyLiBZZXMgd2UgdXNlIGRldm1fbWVtcmVtYXBfcGFn ZXMoKSB0byBhZGQgdGhlc2UgcGFnZXMgYW5kIGl0IGRvZXMgY2FsbCBhcmNoX2FkZF9tZW1vcnko KS4gV2UgZG8gaGF2ZSBhbiBhbHRlcm5hdGUgc2V0IG9mIHBhdGNoZXMgd2hpY2ggc3RpbGwgY2Fs bHMgZGV2bV9tZW1yZW1hcF9wYWdlcygpIGJ1dCBjYW4gdGFrZSBhIGZsYWcgdG8gaW5kaWNhdGUg dGhlIG1lbW9yeSBiZWluZyBhZGRlZCBpcyBpbyBtZW1vcnkgYW5kIHVzZXMgaW9fcmVtYXAoKSBy YXRoZXIgdGhhbiBhcmNoX2FkZF9tZW1vcnkoKSBmb3IgdGhhdCB0eXBlIG9mIG1lbW9yeSBbMV0u IFdvdWxkIHRoYXQgYmUgYSBiZXR0ZXIgYXBwcm9hY2ggZm9yIHRoaXMgYXJjaD8gSSBjYW4gdHJ5 IGFuZCBhcHBseSB0aGlzIHBhdGNoIGJ1dCBfX2FkZF9wYWdlcygpIGhhcyBnb25lIHRocm91Z2gg c29tZSBjaGFuZ2VzIHJlY2VudGx5IHNvIGl0IHdpbGwgdGFrZSBtZSBhIGZldyBkYXlzIHRvIGdl dCB0byB0aGF0LiANCg0KU3RlcGhlbg0KDQpbMV0gaHR0cHM6Ly9naXRodWIuY29tL3NiYXRlczEz MDI3Mi9saW51eC1wMnBtZW0vY29tbWl0L2FjNzM1ODcxZmNkMmM2M2JkMzNjODE0YWEzOTQxY2Ez ZWY1M2I2MzYNCg0KDQo= ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: lpar issue for ZONE_DEVICE p2pmem in 4.14-rc 2017-10-21 15:03 ` Stephen Bates @ 2017-10-23 2:53 ` Balbir Singh 2017-10-23 20:17 ` Stephen Bates 0 siblings, 1 reply; 8+ messages in thread From: Balbir Singh @ 2017-10-23 2:53 UTC (permalink / raw) To: Stephen Bates; +Cc: linuxppc-dev@lists.ozlabs.org, Logan Gunthorpe On Sat, 21 Oct 2017 15:03:29 +0000 "Stephen Bates" <sbates@raithlin.com> wrote: > > I am guessing that the hotplug of ZONE_DEVICE memory was done > > incorrectly as it lead to HPT resizing (the system thinking this is > > normal memory). Ideally one would expect that the driver would online > > ZONE_DEVICE memory and not go through the HOTPLUG path. Are you using > > devm_memremap_pages() path to add these pages? > > > > Thanks for the response Balbir. Yes we use devm_memremap_pages() to add these pages and it does call arch_add_memory(). We do have an alternate set of patches which still calls devm_memremap_pages() but can take a flag to indicate the memory being added is io memory and uses io_remap() rather than arch_add_memory() for that type of memory [1]. Would that be a better approach for this arch? I can try and apply this patch but __add_pages() has gone through some changes recently so it will take me a few days to get to that. I just double checked, for pmem you do need to come in via arch_add_memory(). I was confused by what we do for HMM, which is call __add_pages(), but we do need a section mapping so the interface is correct. The following [ 3.537780] lpar: Attempting to resize HPT to shift 21 [ 3.539251] Unable to resize hash page table to target order 21: -1 [ 3.541079] Unable to create mapping for hot added memory 0xc000210000000000..0xc000210004000000: -2 Needs to be debugged further. For #1 above please check if your qemu supports H_RESIZE_HPT_* hcalls? For create mapping failures, the rc is -ENOENT. Can you help debug this further? We could do hcall tracing or enable debugging. Balbir Singh. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: lpar issue for ZONE_DEVICE p2pmem in 4.14-rc 2017-10-23 2:53 ` Balbir Singh @ 2017-10-23 20:17 ` Stephen Bates 2017-10-25 14:34 ` Oliver 0 siblings, 1 reply; 8+ messages in thread From: Stephen Bates @ 2017-10-23 20:17 UTC (permalink / raw) To: Balbir Singh; +Cc: linuxppc-dev@lists.ozlabs.org, Logan Gunthorpe DQo+IFsgICAgMy41Mzc3ODBdIGxwYXI6IEF0dGVtcHRpbmcgdG8gcmVzaXplIEhQVCB0byBzaGlm dCAyMQ0KPiBbICAgIDMuNTM5MjUxXSBVbmFibGUgdG8gcmVzaXplIGhhc2ggcGFnZSB0YWJsZSB0 byB0YXJnZXQgb3JkZXIgMjE6IC0xDQo+IFsgICAgMy41NDEwNzldIFVuYWJsZSB0byBjcmVhdGUg bWFwcGluZyBmb3IgaG90IGFkZGVkIG1lbW9yeSAweGMwMDAyMTAwMDAwMDAwMDAuLjB4YzAwMDIx MDAwNDAwMDAwMDogLTINCg0KPiBGb3IgIzEgYWJvdmUgcGxlYXNlIGNoZWNrIGlmIHlvdXIgcWVt dSBzdXBwb3J0cyBIX1JFU0laRV9IUFRfKiBoY2FsbHM/IA0KDQpCYWxiaXIgZG8geW91IGhhdmUg YW55IHN1Z2dlc3Rpb25zIGFzIHRvIGhvdyB0byB0ZXN0IGZvciB0aGlzIHN1cHBvcnQ/IE5vdGUg SSBhbSBydW5uaW5nIHRoaXMgb24gbXkgeDg2XzY0IGhvc3Qgc28gdGhlcmUgaXMgbm8gdmlydHVh bGl6YXRpb24gaGFyZHdhcmUgaW4gbXkgUUVNVS4gTXkgcWVtdSBpcyB2ZXJ5IHJlY2VudCAoUUVN VSBlbXVsYXRvciB2ZXJzaW9uIDIuMTAuNTAgKHYyLjEwLjAtMTAyNi1nZDhmOTMyYy1kaXJ0eSkp Lg0KDQo+IEZvciBjcmVhdGUgbWFwcGluZyBmYWlsdXJlcywgdGhlIHJjIGlzIC1FTk9FTlQuIENh biB5b3UgaGVscCBkZWJ1ZyB0aGlzIGZ1cnRoZXI/IFdlIGNvdWxkIGRvIGhjYWxsIHRyYWNpbmcg b3IgZW5hYmxlIGRlYnVnZ2luZy4NCg0KU3VyZSBJIGNhbiBoZWxwIGRlYnVnLiBNeSBvcmlnaW5h bCBlbWFpbCBhbHNvIGhhZCBhbGwgeW91IG5lZWRlZCB0byByZWNyZWF0ZSB0aGlzIGlzc3VlIHNv IHRoYXTigJlzIGFuIG9wdGlvbiB0b28/DQoNClN0ZXBoZW4NCg0KDQo= ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: lpar issue for ZONE_DEVICE p2pmem in 4.14-rc 2017-10-23 20:17 ` Stephen Bates @ 2017-10-25 14:34 ` Oliver 2017-10-27 8:07 ` Oliver 0 siblings, 1 reply; 8+ messages in thread From: Oliver @ 2017-10-25 14:34 UTC (permalink / raw) To: Stephen Bates Cc: Balbir Singh, Logan Gunthorpe, linuxppc-dev@lists.ozlabs.org On Tue, Oct 24, 2017 at 7:17 AM, Stephen Bates <sbates@raithlin.com> wrote= : > >> [ 3.537780] lpar: Attempting to resize HPT to shift 21 >> [ 3.539251] Unable to resize hash page table to target order 21: -1 >> [ 3.541079] Unable to create mapping for hot added memory 0xc00021000= 0000000..0xc000210004000000: -2 > >> For #1 above please check if your qemu supports H_RESIZE_HPT_* hcalls? > > Balbir do you have any suggestions as to how to test for this support? No= te I am running this on my x86_64 host so there is no virtualization hardwa= re in my QEMU. My qemu is very recent (QEMU emulator version 2.10.50 (v2.10= .0-1026-gd8f932c-dirty)). Honestly I'd just ignore the resize error. The hash table stores PTE entries so it should be sized based on the amount of memory in the system. If it's drastically under sized there'll be a performance hit, but everything should still work. >> For create mapping failures, the rc is -ENOENT. Can you help debug this = further? We could do hcall tracing or enable debugging. > > Sure I can help debug. My original email also had all you needed to recre= ate this issue so that=E2=80=99s an option too? I'm not too sure what's happening there. My hunch is that the hypervisor (qemu in this case) is rejecting the attempt to map the PCI device MMIO space as cachable memory. On bare metal systems this can result in cache paradoxes which will kill the system so the hypervisor has an incentive to prevent that situation. > > Stephen > > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: lpar issue for ZONE_DEVICE p2pmem in 4.14-rc 2017-10-25 14:34 ` Oliver @ 2017-10-27 8:07 ` Oliver 2017-10-28 20:54 ` Stephen Bates 0 siblings, 1 reply; 8+ messages in thread From: Oliver @ 2017-10-27 8:07 UTC (permalink / raw) To: Stephen Bates Cc: Balbir Singh, Logan Gunthorpe, linuxppc-dev@lists.ozlabs.org, aik On Thu, Oct 26, 2017 at 1:34 AM, Oliver <oohall@gmail.com> wrote: > On Tue, Oct 24, 2017 at 7:17 AM, Stephen Bates <sbates@raithlin.com> wro= te: >> >>> [ 3.537780] lpar: Attempting to resize HPT to shift 21 >>> [ 3.539251] Unable to resize hash page table to target order 21: -1 >>> [ 3.541079] Unable to create mapping for hot added memory 0xc0002100= 00000000..0xc000210004000000: -2 >> >>> For #1 above please check if your qemu supports H_RESIZE_HPT_* hcalls? >> >> Balbir do you have any suggestions as to how to test for this support? N= ote I am running this on my x86_64 host so there is no virtualization hardw= are in my QEMU. My qemu is very recent (QEMU emulator version 2.10.50 (v2.1= 0.0-1026-gd8f932c-dirty)). > > Honestly I'd just ignore the resize error. The hash table stores PTE > entries so it should be sized based on the amount of memory in the > system. If it's drastically under sized there'll be a performance hit, > but everything should still work. > >>> For create mapping failures, the rc is -ENOENT. Can you help debug this= further? We could do hcall tracing or enable debugging. >> >> Sure I can help debug. My original email also had all you needed to recr= eate this issue so that=E2=80=99s an option too? > > I'm not too sure what's happening there. My hunch is that the > hypervisor (qemu in this case) is rejecting the attempt to map the PCI > device MMIO space as cachable memory. On bare metal systems this can > result in cache paradoxes which will kill the system so the hypervisor > has an incentive to prevent that situation. So I had a deeper look and found the hypervisor interface spec (PAPR) says the hypervisor should reject attempts to map memory with inappropriate attributes for the type of memory being mapped. The pseries model in qemu interprets this by only allowing cacheable mappings on memory ranges that it considers as RAM. While KVM will allow any mappings provided they have the same cachable attribute as the hypervisor's mapping. Either way trying to use devm_memremap_pages() like this on pseries is fundementally broken. The alternative approach you mentioned that uses ioremap() should work fine though. Also, Alexy (+cc) said he was interested in trying this on some real hardware. Is there a test suite for p2pmem floating around that he can use? Thanks, Oliver ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: lpar issue for ZONE_DEVICE p2pmem in 4.14-rc 2017-10-27 8:07 ` Oliver @ 2017-10-28 20:54 ` Stephen Bates 0 siblings, 0 replies; 8+ messages in thread From: Stephen Bates @ 2017-10-28 20:54 UTC (permalink / raw) To: Oliver Cc: Balbir Singh, Logan Gunthorpe, linuxppc-dev@lists.ozlabs.org, aik@ozlabs.ru DQpPbGl2ZXINCg0KPlRoZSBhbHRlcm5hdGl2ZSBhcHByb2FjaCB5b3UgbWVudGlvbmVkIHRoYXQg dXNlcyBpb3JlbWFwKCkgc2hvdWxkIHdvcmsNCj5maW5lIHRob3VnaC4NCg0KR3JlYXQsIHRoYW5r cyBmb3IgdGhpcyB1c2VmdWwgaW5mb3JtYXRpb24gT2xpdmVyLiBJIGFtIHdvcmtpbmcgb24gcmVi YXNpbmcgdGhlIGlvcmVtYXAoKSANCnBhdGNoIG9uIDQuMTQgYW5kIHdpbGwgbGV0IHlvdSBrbm93 IGhvdyBpdCBnb2VzLg0KDQo+IEFsc28sIEFsZXh5ICgrY2MpIHNhaWQgaGUgd2FzIGludGVyZXN0 ZWQgaW4gdHJ5aW5nIHRoaXMgb24gc29tZSByZWFsDQo+IGhhcmR3YXJlLiBJcyB0aGVyZSBhIHRl c3Qgc3VpdGUgZm9yIHAycG1lbSBmbG9hdGluZyBhcm91bmQgdGhhdCBoZSBjYW4NCj4gdXNlPw0K DQpFeGNlbGxlbnQhIEkgdHlwaWNhbGx5IHVzZSBwMnBtZW0tdGVzdCBbMV0uIFlvdSB3aWxsIG5l ZWQgYXQgbGVhc3Qgb25lIE5WTWUgU1NEIGFuZCBvbmUgcDJwbWVtIGNhcGFibGUgZGV2aWNlIGFu ZCB5b3UgKm1pZ2h0KiBuZWVkIHRvIHBsYWNlIHRob3NlIHR3byBkZXZpY2VzIGJlaGluZCBhIFBD SWUgc3dpdGNoIGFuZCBjb25maWd1cmUgQUNTIFsyLTNdIGRlcGVuZGluZyBvbiBob3cgdGhlIENQ VSBSQyB0cmVhdHMgcGVlciB0byBwZWVyIFBDSSBUTFBzLiBJIGFtIGhhcHB5IHRvIGhlbHAgQWxl eCBzZXQgdGhpcyBhbGwgdXAgc28gcmVhY2ggb3V0IGVpdGhlciBwdWJsaWNhbGx5IG9yIHByaXZh dGVseSBpZiB5b3UgbGlrZS4gSSBqdXN0IGFkZGVkIGEgKGhvcGVmdWxseSkgZGVjZW50IFJFQURN RSB0byBoZWxwIGdldCBwZW9wbGUgc3RhcnRlZC4NCg0KU3RlcGhlbg0KDQpbMV0gaHR0cHM6Ly9n aXRodWIuY29tL3NiYXRlczEzMDI3Mi9wMnBtZW0tdGVzdA0KWzJdIGh0dHBzOi8vd3d3LnN1cGVy bWljcm8uY29tL3N1cHBvcnQvZmFxcy9mYXEuY2ZtP2ZhcT0yMDczMg0KWzNdIGh0dHBzOi8vbGtt bC5vcmcvbGttbC8yMDE3LzEwLzI2LzY3OCANCg0K ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2017-10-28 20:54 UTC | newest] Thread overview: 8+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2017-10-17 20:38 lpar issue for ZONE_DEVICE p2pmem in 4.14-rc Stephen Bates 2017-10-17 21:29 ` Balbir Singh 2017-10-21 15:03 ` Stephen Bates 2017-10-23 2:53 ` Balbir Singh 2017-10-23 20:17 ` Stephen Bates 2017-10-25 14:34 ` Oliver 2017-10-27 8:07 ` Oliver 2017-10-28 20:54 ` Stephen Bates
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).