* Re: [PATCH] powerpc: Use common error handling code in setup_new_fdt()
From: Dan Carpenter @ 2018-03-15 11:57 UTC (permalink / raw)
To: Thiago Jung Bauermann
Cc: SF Markus Elfring, linuxppc-dev, Benjamin Herrenschmidt,
Borislav Petkov, Brijesh Singh, Josh Sklar, Kees Cook,
Michael Ellerman, Paul Mackerras, Thomas Gleixner, Tom Lendacky,
LKML, kernel-janitors
In-Reply-To: <87fu52jqtc.fsf@morokweng.localdomain>
On Wed, Mar 14, 2018 at 06:22:07PM -0300, Thiago Jung Bauermann wrote:
>
> SF Markus Elfring <elfring@users.sourceforge.net> writes:
>
> > From: Markus Elfring <elfring@users.sourceforge.net>
> > Date: Sun, 11 Mar 2018 09:03:42 +0100
> >
> > Add a jump target so that a bit of exception handling can be better reused
> > at the end of this function.
> >
> > This issue was detected by using the Coccinelle software.
> >
> > Signed-off-by: Markus Elfring <elfring@users.sourceforge.net>
> > ---
> > arch/powerpc/kernel/machine_kexec_file_64.c | 28 ++++++++++++----------------
> > 1 file changed, 12 insertions(+), 16 deletions(-)
>
> I liked it. Thanks!
>
> Reviewed-by: Thiago Jung Bauermann <bauerman@linux.vnet.ibm.com>
>
You know that compilers already re-use string constants so this doesn't
actually save memory? Also we should be preserving the error codes
instead of always returning -EINVAL.
regards,
dan carpenter
^ permalink raw reply
* Re: [PATCH 0/5] DPAA Ethernet fixes
From: David Miller @ 2018-03-15 15:15 UTC (permalink / raw)
To: madalin.bucur
Cc: joakim.tjernlund, linuxppc-dev, netdev, linux-kernel, leoyang.li,
camelia.groza, linux-arm-kernel
In-Reply-To: <AM5PR04MB3267BC402A80D9E772BC8903ECD00@AM5PR04MB3267.eurprd04.prod.outlook.com>
From: Madalin-cristian Bucur <madalin.bucur@nxp.com>
Date: Thu, 15 Mar 2018 09:32:35 +0000
>> -----Original Message-----
>> From: Joakim Tjernlund [mailto:Joakim.Tjernlund@infinera.com]
>> Sent: Wednesday, March 14, 2018 8:43 PM
>> To: davem@davemloft.net; Madalin-cristian Bucur
>> <madalin.bucur@nxp.com>
>>
>> On Wed, 2018-03-14 at 08:37 -0500, Madalin Bucur wrote:
>> > Hi,
>> >
>> > This patch set is addressing several issues in the DPAA Ethernet
>> > driver suite:
>> >
>> > - module unload crash caused by wrong reference to device being left
>> > in the cleanup code after the DSA related changes
>> > - scheduling wile atomic bug in QMan code revealed during dpaa_eth
>> > module unload
>> > - a couple of error counter fixes, a duplicated init in dpaa_eth.
>>
>> hmm, some of these(all?) bugs are in stable too, CC: stable perhaps?
>
> Hi Jocke,
>
> I did not check that, they should be added.
Networking patches are not queued to stable by CC:'ing stable.
Instead you just simply need to ask me explicitly for -stable submission
which I will do at the appropriate time after the fixes have been able
to cook in Linus's tree for a little while.
I've queued this series up for that purpose, thanks.
^ permalink raw reply
* Re: [PATCH] powerpc: Use common error handling code in setup_new_fdt()
From: Joe Perches @ 2018-03-15 15:03 UTC (permalink / raw)
To: Dan Carpenter, Thiago Jung Bauermann
Cc: SF Markus Elfring, linuxppc-dev, Benjamin Herrenschmidt,
Borislav Petkov, Brijesh Singh, Josh Sklar, Kees Cook,
Michael Ellerman, Paul Mackerras, Thomas Gleixner, Tom Lendacky,
LKML, kernel-janitors
In-Reply-To: <20180315115753.rjejlkrp3fhdxbyk@mwanda>
On Thu, 2018-03-15 at 14:57 +0300, Dan Carpenter wrote:
> On Wed, Mar 14, 2018 at 06:22:07PM -0300, Thiago Jung Bauermann wrote:
> >
> > SF Markus Elfring <elfring@users.sourceforge.net> writes:
> >
> > > From: Markus Elfring <elfring@users.sourceforge.net>
> > > Date: Sun, 11 Mar 2018 09:03:42 +0100
> > >
> > > Add a jump target so that a bit of exception handling can be better reused
> > > at the end of this function.
> > >
> > > This issue was detected by using the Coccinelle software.
> > >
> > > Signed-off-by: Markus Elfring <elfring@users.sourceforge.net>
> > > ---
> > > arch/powerpc/kernel/machine_kexec_file_64.c | 28 ++++++++++++----------------
> > > 1 file changed, 12 insertions(+), 16 deletions(-)
> >
> > I liked it. Thanks!
> >
> > Reviewed-by: Thiago Jung Bauermann <bauerman@linux.vnet.ibm.com>
> >
>
> You know that compilers already re-use string constants so this doesn't
> actually save memory?
And modern compilers create their own jump labels
so this doesn't change object code either?
^ permalink raw reply
* Re: [PATCH v3] x86: treat pkey-0 special
From: Thomas Gleixner @ 2018-03-15 9:46 UTC (permalink / raw)
To: Ram Pai
Cc: mingo, mpe, linuxppc-dev, linux-mm, x86, linux-arch, linux-kernel,
akpm, dave.hansen, benh, paulus, khandual, aneesh.kumar,
bsingharora, hbabu, mhocko, bauerman, ebiederm, corbet, arnd,
fweimer, msuchanek, Ulrich.Weigand
In-Reply-To: <1521061214-22385-1-git-send-email-linuxram@us.ibm.com>
On Wed, 14 Mar 2018, Ram Pai wrote:
> Applications need the ability to associate an address-range with some
> key and latter revert to its initial default key. Pkey-0 comes close to
> providing this function but falls short, because the current
> implementation disallows applications to explicitly associate pkey-0 to
> the address range.
>
> This patch clarifies the semantics of pkey-0 and provides the
grep 'This patch' Documentation/process
> corresponding implementation on powerpc.
>
> Pkey-0 is special with the following semantics.
> (a) it is implicitly allocated and can never be freed. It always exists.
> (b) it is the default key assigned to any address-range.
> (c) it can be explicitly associated with any address-range.
>
> Tested on x86_64.
I'm curious how the corresponding implementation on powerpc can be tested
on x86_64. Copy and paste is not enough ...
>
> History:
> v3 : added clarification of the semantics of pkey0.
> -- suggested by Dave Hansen
> v2 : split the patch into two, one for x86 and one for powerpc
> -- suggested by Michael Ellermen
Please put the history below the --- seperator. It's not part of the
changelog. That way the tools can discard it when picking up the patch.
>
> cc: Dave Hansen <dave.hansen@intel.com>
> cc: Michael Ellermen <mpe@ellerman.id.au>
> cc: Ingo Molnar <mingo@kernel.org>
> Signed-off-by: Ram Pai <linuxram@us.ibm.com>
> ---
> arch/x86/include/asm/pkeys.h | 5 +++--
> 1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/arch/x86/include/asm/pkeys.h b/arch/x86/include/asm/pkeys.h
> index a0ba1ff..6ea7486 100644
> --- a/arch/x86/include/asm/pkeys.h
> +++ b/arch/x86/include/asm/pkeys.h
> @@ -52,7 +52,7 @@ bool mm_pkey_is_allocated(struct mm_struct *mm, int pkey)
> * from pkey_alloc(). pkey 0 is special, and never
> * returned from pkey_alloc().
> */
> - if (pkey <= 0)
> + if (pkey < 0)
> return false;
> if (pkey >= arch_max_pkey())
> return false;
> @@ -92,7 +92,8 @@ int mm_pkey_alloc(struct mm_struct *mm)
> static inline
> int mm_pkey_free(struct mm_struct *mm, int pkey)
> {
> - if (!mm_pkey_is_allocated(mm, pkey))
> + /* pkey 0 is special and can never be freed */
This comment is pretty useless. How should anyone figure out whats special
about pkey 0?
> + if (!pkey || !mm_pkey_is_allocated(mm, pkey))
Why this extra check? mm_pkey_is_allocated(mm, 0) should not return true
ever. If it does, then this wants to be fixed.
Thanks,
tglx
^ permalink raw reply
* RE: [PATCH 0/5] DPAA Ethernet fixes
From: Madalin-cristian Bucur @ 2018-03-15 9:32 UTC (permalink / raw)
To: jocke@infinera.com, davem@davemloft.net
Cc: linuxppc-dev@lists.ozlabs.org, netdev@vger.kernel.org,
linux-kernel@vger.kernel.org, Leo Li, Camelia Alexandra Groza,
linux-arm-kernel@lists.infradead.org
In-Reply-To: <1521052994.4790.148.camel@infinera.com>
PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBKb2FraW0gVGplcm5sdW5kIFtt
YWlsdG86Sm9ha2ltLlRqZXJubHVuZEBpbmZpbmVyYS5jb21dDQo+IFNlbnQ6IFdlZG5lc2RheSwg
TWFyY2ggMTQsIDIwMTggODo0MyBQTQ0KPiBUbzogZGF2ZW1AZGF2ZW1sb2Z0Lm5ldDsgTWFkYWxp
bi1jcmlzdGlhbiBCdWN1cg0KPiA8bWFkYWxpbi5idWN1ckBueHAuY29tPg0KPiANCj4gT24gV2Vk
LCAyMDE4LTAzLTE0IGF0IDA4OjM3IC0wNTAwLCBNYWRhbGluIEJ1Y3VyIHdyb3RlOg0KPiA+IEhp
LA0KPiA+DQo+ID4gVGhpcyBwYXRjaCBzZXQgaXMgYWRkcmVzc2luZyBzZXZlcmFsIGlzc3VlcyBp
biB0aGUgRFBBQSBFdGhlcm5ldA0KPiA+IGRyaXZlciBzdWl0ZToNCj4gPg0KPiA+ICAtIG1vZHVs
ZSB1bmxvYWQgY3Jhc2ggY2F1c2VkIGJ5IHdyb25nIHJlZmVyZW5jZSB0byBkZXZpY2UgYmVpbmcg
bGVmdA0KPiA+ICAgIGluIHRoZSBjbGVhbnVwIGNvZGUgYWZ0ZXIgdGhlIERTQSByZWxhdGVkIGNo
YW5nZXMNCj4gPiAgLSBzY2hlZHVsaW5nIHdpbGUgYXRvbWljIGJ1ZyBpbiBRTWFuIGNvZGUgcmV2
ZWFsZWQgZHVyaW5nIGRwYWFfZXRoDQo+ID4gICAgbW9kdWxlIHVubG9hZA0KPiA+ICAtIGEgY291
cGxlIG9mIGVycm9yIGNvdW50ZXIgZml4ZXMsIGEgZHVwbGljYXRlZCBpbml0IGluIGRwYWFfZXRo
Lg0KPiANCj4gaG1tLCBzb21lIG9mIHRoZXNlKGFsbD8pIGJ1Z3MgYXJlIGluIHN0YWJsZSB0b28s
IENDOiBzdGFibGUgcGVyaGFwcz8NCg0KSGkgSm9ja2UsDQoNCkkgZGlkIG5vdCBjaGVjayB0aGF0
LCB0aGV5IHNob3VsZCBiZSBhZGRlZC4NCg0KVGhhbmtzLA0KTWFkYWxpbg0KDQo+ID4NCj4gPiBN
YWRhbGluDQo+ID4NCj4gPiBDYW1lbGlhIEdyb3phICgzKToNCj4gPiAgIGRwYWFfZXRoOiByZW1v
dmUgZHVwbGljYXRlIGluaXRpYWxpemF0aW9uDQo+ID4gICBkcGFhX2V0aDogaW5jcmVtZW50IHRo
ZSBSWCBkcm9wcGVkIGNvdW50ZXIgd2hlbiBuZWVkZWQNCj4gPiAgIGRwYWFfZXRoOiByZW1vdmUg
ZHVwbGljYXRlIGluY3JlbWVudCBvZiB0aGUgdHhfZXJyb3JzIGNvdW50ZXINCj4gPg0KPiA+IE1h
ZGFsaW4gQnVjdXIgKDIpOg0KPiA+ICAgc29jL2ZzbC9xYm1hbjogZml4IGlzc3VlIGluIHFtYW5f
ZGVsZXRlX2Nncl9zYWZlKCkNCj4gPiAgIGRwYWFfZXRoOiBmaXggZXJyb3IgaW4gZHBhYV9yZW1v
dmUoKQ0KPiA+DQo+ID4gIGRyaXZlcnMvbmV0L2V0aGVybmV0L2ZyZWVzY2FsZS9kcGFhL2RwYWFf
ZXRoLmMgfCAgOCArKysrLS0tLQ0KPiA+ICBkcml2ZXJzL3NvYy9mc2wvcWJtYW4vcW1hbi5jICAg
ICAgICAgICAgICAgICAgIHwgMjggKysrKystLS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4gPiAgMiBm
aWxlcyBjaGFuZ2VkLCA5IGluc2VydGlvbnMoKyksIDI3IGRlbGV0aW9ucygtKQ0KPiA+DQo+ID4g
LS0NCj4gPiAyLjEuMA0KPiA+DQo=
^ permalink raw reply
* Re: more warnings
From: Stephen Rothwell @ 2018-03-15 9:27 UTC (permalink / raw)
To: Balbir Singh; +Cc: ppc-dev
In-Reply-To: <CAKTCnz=4Fun3C5RMp1x3N+rmaU8-Yy3WOVZFyrc_5MNRKaYskg@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1291 bytes --]
Hi Balbir,
On Thu, 15 Mar 2018 19:42:33 +1100 Balbir Singh <bsingharora@gmail.com> wrote:
>
> On Wed, Mar 14, 2018 at 4:23 PM, Stephen Rothwell <sfr@canb.auug.org.au> wrote:
> >
> > I also get these from a powerpc allyesconfig build of linux-next:
> >
> > WARNING: vmlinux.o(.text+0x7c81c): Section mismatch in reference from the function .stop_machine_change_mapping() to the function .meminit.text:.create_physical_mapping()
> > The function .stop_machine_change_mapping() references
> > the function __meminit .create_physical_mapping().
> > This is often because .stop_machine_change_mapping lacks a __meminit
> > annotation or the annotation of .create_physical_mapping is wrong.
> >
> > WARNING: vmlinux.o(.text+0x7c828): Section mismatch in reference from the function .stop_machine_change_mapping() to the function .meminit.text:.create_physical_mapping()
> > The function .stop_machine_change_mapping() references
> > the function __meminit .create_physical_mapping().
> > This is often because .stop_machine_change_mapping lacks a __meminit
> > annotation or the annotation of .create_physical_mapping is wrong.
> >
>
> Patches have been sent to fix this - http://patchwork.ozlabs.org/patch/883991/
Excellent, thanks.
--
Cheers,
Stephen Rothwell
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply
* Re: more warnings
From: Balbir Singh @ 2018-03-15 8:42 UTC (permalink / raw)
To: Stephen Rothwell; +Cc: ppc-dev
In-Reply-To: <20180314162325.1b2dea8d@canb.auug.org.au>
On Wed, Mar 14, 2018 at 4:23 PM, Stephen Rothwell <sfr@canb.auug.org.au> wrote:
> Hi all,
>
> I also get these from a powerpc allyesconfig build of linux-next:
>
> WARNING: vmlinux.o(.text+0x7c81c): Section mismatch in reference from the function .stop_machine_change_mapping() to the function .meminit.text:.create_physical_mapping()
> The function .stop_machine_change_mapping() references
> the function __meminit .create_physical_mapping().
> This is often because .stop_machine_change_mapping lacks a __meminit
> annotation or the annotation of .create_physical_mapping is wrong.
>
> WARNING: vmlinux.o(.text+0x7c828): Section mismatch in reference from the function .stop_machine_change_mapping() to the function .meminit.text:.create_physical_mapping()
> The function .stop_machine_change_mapping() references
> the function __meminit .create_physical_mapping().
> This is often because .stop_machine_change_mapping lacks a __meminit
> annotation or the annotation of .create_physical_mapping is wrong.
>
Hi, Stephen
Patches have been sent to fix this - http://patchwork.ozlabs.org/patch/883991/
Balbir Singh
^ permalink raw reply
* Re: [Y2038] [PATCH v4 02/10] include: Move compat_timespec/ timeval to compat_time.h
From: Arnd Bergmann @ 2018-03-15 8:04 UTC (permalink / raw)
To: Deepa Dinamani
Cc: kbuild test robot, Mark Rutland,
open list:RALINK MIPS ARCHITECTURE, Peter Zijlstra,
Benjamin Herrenschmidt, Heiko Carstens, Paul Mackerras,
H. Peter Anvin, sparclinux, devel, linux-s390, y2038 Mailman List,
Michael Ellerman, Helge Deller, the arch/x86 maintainers, sebott,
James E.J. Bottomley, Christian Borntraeger, Ingo Molnar,
oprofile-list, Catalin Marinas, Peter Oberparleiter,
Robert Richter, Chris Metcalf, Will Deacon,
Arnaldo Carvalho de Melo, Julian Wiedmann, John Stultz,
Steven Rostedt, Thomas Gleixner, gerald.schaefer, Parisc List,
Greg KH, cohuck, Linux Kernel Mailing List, Ralf Baechle,
David S. Miller, Jan Hoeppner, kbuild-all, Stefan Haberland,
Martin Schwidefsky, linuxppc-dev, Ursula Braun
In-Reply-To: <CABeXuvpFfD+a6tSSOvni=v23DuJ-bWeZwmnzg4SU+TR=WHxs7Q@mail.gmail.com>
On Thu, Mar 15, 2018 at 3:51 AM, Deepa Dinamani <deepa.kernel@gmail.com> wrote:
> On Wed, Mar 14, 2018 at 1:52 PM, Arnd Bergmann <arnd@arndb.de> wrote:
>> On Wed, Mar 14, 2018 at 4:50 AM, Deepa Dinamani <deepa.kernel@gmail.com> wrote:
>>> The file arch/arm64/kernel/process.c needs asm/compat.h also to be
>>> included directly since this is included conditionally from
>>> include/compat.h. This does seem to be typical of arm64 as I was not
>>> completely able to get rid of asm/compat.h includes for arm64 in this
>>> series. My plan is to have separate patches to get rid of asm/compat.h
>>> includes for the architectures that are not straight forward to keep
>>> this series simple.
>>> I will fix this and update the series.
>>>
>>
>> I ran across the same thing in two more files during randconfig testing on
>> arm64 now, adding this fixup on top for the moment, but maybe there
>> is a better way:
>
> I was looking at how Al tested his uaccess patches:
> https://www.spinics.net/lists/linux-fsdevel/msg108752.html
>
> He seems to be running the kbuild bot tests on his own git.
> Is it possible to verify it this way on the 2038 tree? Or, I could
> host a tree also.
The kbuild bot should generally pick up any branch on git.kernel.org,
and the patches sent to the mailing list. It tests a lot of things
configurations, but I tend to find some things that it doesn't find
by doing lots of randconfig builds on fewer target architectures
(I only build arm, arm64 and x86 regularly).
I remember that there was some discussion about a method
to get the bot to test other branches (besides asking Fengguang
to add it manually), but I don't remember what came out of that.
Arnd
^ permalink raw reply
* Re: [PATCH kernel] powerpc/npu: Do not try invalidating 32bit table when 64bit table is enabled
From: Alexey Kardashevskiy @ 2018-03-15 7:39 UTC (permalink / raw)
To: Alistair Popple; +Cc: linuxppc-dev, David Gibson, Russell Currey
In-Reply-To: <1946633.GlQphcvEDT@new-mexico>
On 15/3/18 4:18 pm, Alistair Popple wrote:
> I must admit I haven't really looked at the iommu_table_group code in depth so
> don't fully understand it, but I was wondering why we don't hit this for NVLink1
> as well?
>
> The error that Skiboot is printing is because the requested page size doesn't
> match expected so I guess there is a possibility the address might not meet the
> required alignment (I don't know we don't just check that directly).
>
> The reason you don't see this on NVLink1 is that we rely on the kernel to do the
> TCE invalidate directly rather than via an OPAL call, and the kernel seems to do
> less checking.
Correct.
> So I wonder if there is a bug there as well (invalidate for a
> mismatched page size/address alignment)?
> And wether this would also fix that?
Correct. Although afair we simply reset the entire cache (due to some
hardware bug about which I am not now sure that it actually exists) so we
did not see it. May be with this fix applied we won't have to do that and
can invalidate individual tces (but I doubt because of the bug).
>
> - Alistair
>
> On Tuesday, 13 February 2018 4:51:35 PM AEDT Alexey Kardashevskiy wrote:
>> GPUs and the corresponding NVLink bridges get different PEs as they have
>> separate translation validation entries (TVEs). We put these PEs to
>> the same IOMMU group so they cannot be passed through separately.
>> So the iommu_table_group_ops::set_window/unset_window for GPUs do set
>> tables to the NPU PEs as well which means that iommu_table's list of
>> attached PEs (iommu_table_group_link) has both GPU and NPU PEs linked.
>> This list is used for TCE cache invalidation.
>>
>> The problem is that NPU PE has just a single TVE and can be programmed
>> to point to 32bit or 64bit windows while GPU PE has two (as any other PCI
>> device). So we end up having an 32bit iommu_table struct linked to both
>> PEs even though only the 64bit TCE table cache can be invalidated on NPU.
>> And a relatively recent skiboot detects this and prints errors.
>>
>> This changes GPU's iommu_table_group_ops::set_window/unset_window to make
>> sure that NPU PE is only linked to the table actually used by the hardware.
>> If there are two tables used by an IOMMU group, the NPU PE will use
>> the last programmed one which with the current use scenarios is expected
>> to be a 64bit one.
>>
>> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>> ---
>>
>> Do we need BUG_ON(IOMMU_TABLE_GROUP_MAX_TABLES != 2)?
>>
>>
>> This is an example for:
>>
>> 0004:04:00.0 3D: NVIDIA Corporation Device 1db1 (rev a1)
>> 0006:00:00.0 Bridge: IBM Device 04ea (rev 01)
>> 0006:00:00.1 Bridge: IBM Device 04ea (rev 01)
>>
>> Before the patch (npu2_tce_kill messages are from skiboot):
>>
>> pci 0004:04 : [PE# 00] Setting up window#0 0..3fffffff pg=1000
>> pci 0006:00:00.0: [PE# 0d] Setting up window 0..3fffffff pg=1000
>> pci 0004:04 : [PE# 00] Setting up window#1 800000000000000..8000000ffffffff pg=10000
>> pci 0006:00:00.0: [PE# 0d] Setting up window 800000000000000..8000000ffffffff pg=10000
>> NPU6: npu2_tce_kill: Unexpected TCE size (got 0x1000 expected 0x10000)
>> NPU6: npu2_tce_kill: Unexpected TCE size (got 0x1000 expected 0x10000)
>> NPU6: npu2_tce_kill: Unexpected TCE size (got 0x1000 expected 0x10000)
>> NPU6: npu2_tce_kill: Unexpected TCE size (got 0x1000 expected 0x10000)
>> NPU6: npu2_tce_kill: Unexpected TCE size (got 0x1000 expected 0x10000)
>> ...
>> pci 0004:04 : [PE# 00] Removing DMA window #0
>> pci 0006:00:00.0: [PE# 0d] Removing DMA window
>> pci 0004:04 : [PE# 00] Removing DMA window #1
>> pci 0006:00:00.0: [PE# 0d] Removing DMA window
>> pci 0004:04 : [PE# 00] Setting up window#0 0..3fffffff pg=1000
>> pci 0006:00:00.0: [PE# 0d] Setting up window 0..3fffffff pg=1000
>> pci 0004:04 : [PE# 00] Setting up window#1 800000000000000..8000000ffffffff pg=10000
>> pci 0006:00:00.0: [PE# 0d] Setting up window 800000000000000..8000000ffffffff pg=10000
>>
>> After the patch (no errors here):
>>
>> pci 0004:04 : [PE# 00] Setting up window#0 0..3fffffff pg=1000
>> pci 0006:00:00.0: [PE# 0d] Setting up window 0..3fffffff pg=1000
>> pci 0004:04 : [PE# 00] Setting up window#1 800000000000000..8000000ffffffff pg=10000
>> pci 0006:00:00.0: [PE# 0d] Removing DMA window
>> pci 0006:00:00.0: [PE# 0d] Setting up window 800000000000000..8000000ffffffff pg=10000
>> pci 0004:04 : [PE# 00] Removing DMA window #0
>> pci 0004:04 : [PE# 00] Removing DMA window #1
>> pci 0006:00:00.0: [PE# 0d] Removing DMA window
>> pci 0004:04 : [PE# 00] Setting up window#0 0..3fffffff pg=1000
>> pci 0006:00:00.0: [PE# 0d] Setting up window 0..3fffffff pg=1000
>> pci 0004:04 : [PE# 00] Setting up window#1 800000000000000..8000000ffffffff pg=10000
>> pci 0006:00:00.0: [PE# 0d] Removing DMA window
>> pci 0006:00:00.0: [PE# 0d] Setting up window 800000000000000..8000000ffffffff pg=10000
>> ---
>> arch/powerpc/platforms/powernv/pci-ioda.c | 27 ++++++++++++++++++++++++---
>> 1 file changed, 24 insertions(+), 3 deletions(-)
>>
>> diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
>> index 496e476..2f91815 100644
>> --- a/arch/powerpc/platforms/powernv/pci-ioda.c
>> +++ b/arch/powerpc/platforms/powernv/pci-ioda.c
>> @@ -2681,14 +2681,23 @@ static struct pnv_ioda_pe *gpe_table_group_to_npe(
>> static long pnv_pci_ioda2_npu_set_window(struct iommu_table_group *table_group,
>> int num, struct iommu_table *tbl)
>> {
>> + struct pnv_ioda_pe *npe = gpe_table_group_to_npe(table_group);
>> + int num2 = (num == 0) ? 1 : 0;
>> long ret = pnv_pci_ioda2_set_window(table_group, num, tbl);
>>
>> if (ret)
>> return ret;
>>
>> - ret = pnv_npu_set_window(gpe_table_group_to_npe(table_group), num, tbl);
>> - if (ret)
>> + if (table_group->tables[num2])
>> + pnv_npu_unset_window(npe, num2);
>> +
>> + ret = pnv_npu_set_window(npe, num, tbl);
>> + if (ret) {
>> pnv_pci_ioda2_unset_window(table_group, num);
>> + if (table_group->tables[num2])
>> + pnv_npu_set_window(npe, num2,
>> + table_group->tables[num2]);
>> + }
>>
>> return ret;
>> }
>> @@ -2697,12 +2706,24 @@ static long pnv_pci_ioda2_npu_unset_window(
>> struct iommu_table_group *table_group,
>> int num)
>> {
>> + struct pnv_ioda_pe *npe = gpe_table_group_to_npe(table_group);
>> + int num2 = (num == 0) ? 1 : 0;
>> long ret = pnv_pci_ioda2_unset_window(table_group, num);
>>
>> if (ret)
>> return ret;
>>
>> - return pnv_npu_unset_window(gpe_table_group_to_npe(table_group), num);
>> + if (!npe->table_group.tables[num])
>> + return 0;
>> +
>> + ret = pnv_npu_unset_window(npe, num);
>> + if (ret)
>> + return ret;
>> +
>> + if (table_group->tables[num2])
>> + ret = pnv_npu_set_window(npe, num2, table_group->tables[num2]);
>> +
>> + return ret;
>> }
>>
>> static void pnv_ioda2_npu_take_ownership(struct iommu_table_group *table_group)
>>
>
>
--
Alexey
^ permalink raw reply
* [RFC PATCH] powerpc/fadump: Reservationless firmware assisted dump
From: Mahesh J Salgaonkar @ 2018-03-15 6:57 UTC (permalink / raw)
To: linuxppc-dev
Cc: Hari Bathini, Ananth N Mavinakayanahalli, Aneesh Kumar K.V,
kernelfans, Anshuman Khandual
From: Mahesh Salgaonkar <mahesh@linux.vnet.ibm.com>
One of the primary issues with Firmware Assisted Dump (fadump) on Power
is that it needs a large amount of memory to be reserved. On large
systems with TeraBytes of memory, this reservation can be quite
significant.
In some cases, fadump fails if the memory reserved is insufficient, or
if the reserved memory was DLPAR hot-removed.
In the normal case, post reboot, the preserved memory is filtered to
extract only relevant areas of interest using the makedumpfile tool.
While the tool provides flexibility to determine what needs to be part
of the dump and what memory to filter out, all supported distributions
default this to "Capture only kernel data and nothing else".
We take advantage of this default and the Linux kernel's Contiguous
Memory Allocator (CMA) to fundamentally change the memory reservation
model for fadump. Fadump can now only capture kernel memory.
Instead of setting aside a significant chunk of memory nobody can use,
this patch uses CMA instead, to reserve a significant chunk of memory
that the kernel is prevented from using (due to MIGRATE_CMA), but
applications are free to use it.
Essentially, on a P9 LPAR with 2 cores, 8GB RAM and current upstream:
[root@zzxx-yy10 ~]# free -m
total used free shared buff/cache available
Mem: 7557 193 6822 12 541 6725
Swap: 4095 0 4095
With this patch:
[root@zzxx-yy10 ~]# free -m
total used free shared buff/cache available
Mem: 8133 194 7464 12 475 7338
Swap: 4095 0 4095
Changes made here are completely transparent to how fadump has
traditionally worked.
An additional advantage is that this mechanism is immune to DLPAR memory
remove (since the CMA memory cannot be removed).
Thanks to Aneesh Kumar and Anshuman Khandual for helping us understand
CMA and its usage.
TODO:
- Handle case where CMA reservation spans nodes.
Signed-off-by: Ananth N Mavinakayanahalli <ananth@linux.vnet.ibm.com>
Signed-off-by: Mahesh Salgaonkar <mahesh@linux.vnet.ibm.com>
---
arch/powerpc/include/asm/fadump.h | 3 +
arch/powerpc/kernel/fadump.c | 179 ++++++++++++++++++++++++++++++++-----
2 files changed, 156 insertions(+), 26 deletions(-)
diff --git a/arch/powerpc/include/asm/fadump.h b/arch/powerpc/include/asm/fadump.h
index 5a23010af600..776cba0baec4 100644
--- a/arch/powerpc/include/asm/fadump.h
+++ b/arch/powerpc/include/asm/fadump.h
@@ -119,6 +119,7 @@ struct fadump_mem_struct {
struct fadump_section cpu_state_data;
struct fadump_section hpte_region;
struct fadump_section rmr_region;
+ struct fadump_section metadata_region;
};
/* Firmware-assisted dump configuration details. */
@@ -141,6 +142,8 @@ struct fw_dump {
unsigned long fadump_supported:1;
unsigned long dump_active:1;
unsigned long dump_registered:1;
+ /* flag to indicate fadump metadata area is cma allocated */
+ unsigned long cma_alloc:1;
};
/*
diff --git a/arch/powerpc/kernel/fadump.c b/arch/powerpc/kernel/fadump.c
index 3c2c2688918f..5ca0cb1b6028 100644
--- a/arch/powerpc/kernel/fadump.c
+++ b/arch/powerpc/kernel/fadump.c
@@ -34,6 +34,7 @@
#include <linux/crash_dump.h>
#include <linux/kobject.h>
#include <linux/sysfs.h>
+#include <linux/cma.h>
#include <asm/debugfs.h>
#include <asm/page.h>
@@ -45,11 +46,59 @@
static struct fw_dump fw_dump;
static struct fadump_mem_struct fdm;
static const struct fadump_mem_struct *fdm_active;
+static struct cma *fadump_cma;
static DEFINE_MUTEX(fadump_mutex);
struct fad_crash_memory_ranges crash_memory_ranges[INIT_CRASHMEM_RANGES];
int crash_mem_ranges;
+/**
+ * fadump_cma_reserve() - reserve area for fadump memory reservation
+ *
+ * This function reserves memory from early allocator. It should be
+ * called by arch specific code once the memblock allocator
+ * has been activated.
+ */
+int __init fadump_cma_reserve(void)
+{
+ unsigned long long base, size;
+ int rc;
+
+ if (!fw_dump.fadump_enabled)
+ return 0;
+
+ base = fw_dump.reserve_dump_area_start;
+ size = fw_dump.reserve_dump_area_size;
+ pr_debug("Original reserve area base %ld, size %ld\n",
+ (unsigned long)base >> 20,
+ (unsigned long)size >> 20);
+ if (!size)
+ return 0;
+
+ printk("%s: reserving %ld MiB for firmware-assisted dump area\n"
+ , __func__, (unsigned long)size >> 20);
+ rc = cma_declare_contiguous(base, size, 0, 0, 0, false,
+ "fadump_cma", &fadump_cma);
+ if (rc) {
+ printk(KERN_ERR "fadump: Failed to reserve cma area for "
+ "firmware-assisted dump, %d\n", rc);
+ fw_dump.reserve_dump_area_size = 0;
+ return 0;
+ }
+ /*
+ * So we now have cma area reserved for fadump. base may be different
+ * from what we requested.
+ */
+ fw_dump.reserve_dump_area_start = cma_get_base(fadump_cma);
+ fw_dump.reserve_dump_area_size = cma_get_size(fadump_cma);
+ printk("Reserved %ldMB cma area at %ldMB for firmware-assisted dump "
+ "(System RAM: %ldMB)\n",
+ (unsigned long)cma_get_size(fadump_cma) >> 20,
+ (unsigned long)cma_get_base(fadump_cma) >> 20,
+ (unsigned long)(memblock_phys_mem_size() >> 20));
+ return 1;
+}
+
/* Scan the Firmware Assisted dump configuration details. */
int __init early_init_dt_scan_fw_dump(unsigned long node,
const char *uname, int depth, void *data)
@@ -188,17 +237,42 @@ static void fadump_show_config(void)
pr_debug("Boot memory size : %lx\n", fw_dump.boot_memory_size);
}
+static unsigned long get_fadump_metadata_size(void)
+{
+ unsigned long size = 0;
+
+ size += sizeof(struct fadump_crash_info_header);
+ size += sizeof(struct elfhdr); /* ELF core header.*/
+ size += sizeof(struct elf_phdr); /* place holder for cpu notes */
+ /* Program headers for crash memory regions. */
+ size += sizeof(struct elf_phdr) * (memblock_num_regions(memory) + 2);
+
+ size = PAGE_ALIGN(size);
+ pr_debug("fadump Metadata size is %ld\n", size);
+ return size;
+}
+
static unsigned long init_fadump_mem_struct(struct fadump_mem_struct *fdm,
unsigned long addr)
{
+ uint16_t num_sections = 0;
+ unsigned long metadata_base = 0;
+ unsigned long metadata_size = 0;
+
if (!fdm)
return 0;
+ if (fw_dump.cma_alloc) {
+ /* Skip the fadump metadata area. */
+ metadata_base = addr;
+ metadata_size = get_fadump_metadata_size();
+ addr += metadata_size;
+ }
+
memset(fdm, 0, sizeof(struct fadump_mem_struct));
addr = addr & PAGE_MASK;
fdm->header.dump_format_version = cpu_to_be32(0x00000001);
- fdm->header.dump_num_sections = cpu_to_be16(3);
fdm->header.dump_status_flag = 0;
fdm->header.offset_first_dump_section =
cpu_to_be32((u32)offsetof(struct fadump_mem_struct, cpu_state_data));
@@ -222,6 +296,7 @@ static unsigned long init_fadump_mem_struct(struct fadump_mem_struct *fdm,
fdm->cpu_state_data.source_address = 0;
fdm->cpu_state_data.source_len = cpu_to_be64(fw_dump.cpu_state_data_size);
fdm->cpu_state_data.destination_address = cpu_to_be64(addr);
+ num_sections++;
addr += fw_dump.cpu_state_data_size;
/* hpte region section */
@@ -230,6 +305,7 @@ static unsigned long init_fadump_mem_struct(struct fadump_mem_struct *fdm,
fdm->hpte_region.source_address = 0;
fdm->hpte_region.source_len = cpu_to_be64(fw_dump.hpte_region_size);
fdm->hpte_region.destination_address = cpu_to_be64(addr);
+ num_sections++;
addr += fw_dump.hpte_region_size;
/* RMA region section */
@@ -238,8 +314,29 @@ static unsigned long init_fadump_mem_struct(struct fadump_mem_struct *fdm,
fdm->rmr_region.source_address = cpu_to_be64(RMA_START);
fdm->rmr_region.source_len = cpu_to_be64(fw_dump.boot_memory_size);
fdm->rmr_region.destination_address = cpu_to_be64(addr);
+ num_sections++;
addr += fw_dump.boot_memory_size;
+ if (!fw_dump.cma_alloc)
+ goto out;
+
+ /*
+ * fadump metadata section.
+ * Add entry with source len a zero. We are only intereseted in
+ * source address which will help us to detect the location of
+ * metadata area where faump header and elf core header is placed.
+ */
+ fdm->metadata_region.request_flag = cpu_to_be32(FADUMP_REQUEST_FLAG);
+ fdm->metadata_region.source_data_type =
+ cpu_to_be16(FADUMP_REAL_MODE_REGION);
+ fdm->metadata_region.source_address = cpu_to_be64(metadata_base);
+ //fdm->metadata_region.source_len = cpu_to_be64(metadata_size);
+ fdm->metadata_region.source_len = 0;
+ fdm->metadata_region.destination_address = cpu_to_be64(addr);
+ num_sections++;
+
+out:
+ fdm->header.dump_num_sections = cpu_to_be16(num_sections);
return addr;
}
@@ -325,16 +422,21 @@ static unsigned long get_fadump_area_size(void)
size += fw_dump.cpu_state_data_size;
size += fw_dump.hpte_region_size;
size += fw_dump.boot_memory_size;
- size += sizeof(struct fadump_crash_info_header);
- size += sizeof(struct elfhdr); /* ELF core header.*/
- size += sizeof(struct elf_phdr); /* place holder for cpu notes */
- /* Program headers for crash memory regions. */
- size += sizeof(struct elf_phdr) * (memblock_num_regions(memory) + 2);
-
+ size += get_fadump_metadata_size();
size = PAGE_ALIGN(size);
return size;
}
+static inline unsigned long get_fadump_metadata_base(
+ const struct fadump_mem_struct *fdm_active)
+{
+ if (be16_to_cpu(fdm_active->header.dump_num_sections) == 4)
+ return be64_to_cpu(fdm_active->metadata_region.source_address);
+
+ return (be64_to_cpu(fdm_active->rmr_region.destination_address) +
+ be64_to_cpu(fdm_active->rmr_region.source_len));
+}
+
int __init fadump_reserve_mem(void)
{
unsigned long base, size, memory_boundary;
@@ -395,11 +497,13 @@ int __init fadump_reserve_mem(void)
(unsigned long)(size >> 20),
(unsigned long)(base >> 20));
- fw_dump.fadumphdr_addr =
- be64_to_cpu(fdm_active->rmr_region.destination_address) +
- be64_to_cpu(fdm_active->rmr_region.source_len);
+ pr_info("Number of kernel Dump sections: %d\n",
+ be16_to_cpu(fdm_active->header.dump_num_sections));
+ fw_dump.fadumphdr_addr = get_fadump_metadata_base(fdm_active);
pr_debug("fadumphdr_addr = %p\n",
(void *) fw_dump.fadumphdr_addr);
+ fw_dump.reserve_dump_area_start = base;
+ fw_dump.reserve_dump_area_size = size;
} else {
size = get_fadump_area_size();
@@ -416,21 +520,10 @@ int __init fadump_reserve_mem(void)
!memblock_is_region_reserved(base, size))
break;
}
- if ((base > (memory_boundary - size)) ||
- memblock_reserve(base, size)) {
- pr_err("Failed to reserve memory\n");
- return 0;
- }
-
- pr_info("Reserved %ldMB of memory at %ldMB for firmware-"
- "assisted dump (System RAM: %ldMB)\n",
- (unsigned long)(size >> 20),
- (unsigned long)(base >> 20),
- (unsigned long)(memblock_phys_mem_size() >> 20));
+ fw_dump.reserve_dump_area_start = base;
+ fw_dump.reserve_dump_area_size = size;
+ return fadump_cma_reserve();
}
-
- fw_dump.reserve_dump_area_start = base;
- fw_dump.reserve_dump_area_size = size;
return 1;
}
@@ -1068,6 +1161,30 @@ static unsigned long init_fadump_header(unsigned long addr)
return addr;
}
+static unsigned long allocate_metadata_area(void)
+{
+ int nr_pages;
+ unsigned long size;
+ struct page *page = NULL;
+
+ /* If fadump_cma->count == 0 means cma activation has failed. */
+ if (!cma_get_size(fadump_cma))
+ return 0;
+
+ size = get_fadump_metadata_size();
+ nr_pages = ALIGN(size, PAGE_SIZE) >> PAGE_SHIFT;
+ printk("Fadump metadata size = %ld (nr_pages = %d)\n", size, nr_pages);
+
+ page = cma_alloc(fadump_cma, nr_pages, 0, GFP_KERNEL);
+ if (page) {
+ pr_debug("Allocated fadump metadata area at %ldMB (cma)\n",
+ (unsigned long)page_to_phys(page) >> 20);
+ fw_dump.cma_alloc = 1;
+ return page_to_phys(page);
+ }
+ return 0;
+}
+
static int register_fadump(void)
{
unsigned long addr;
@@ -1082,7 +1199,12 @@ static int register_fadump(void)
fadump_setup_crash_memory_ranges();
- addr = be64_to_cpu(fdm.rmr_region.destination_address) + be64_to_cpu(fdm.rmr_region.source_len);
+ if (fw_dump.cma_alloc)
+ addr = fw_dump.reserve_dump_area_start;
+ else
+ addr = be64_to_cpu(fdm.rmr_region.destination_address)
+ + be64_to_cpu(fdm.rmr_region.source_len);
+
/* Initialize fadump crash info header. */
addr = init_fadump_header(addr);
vaddr = __va(addr);
@@ -1490,8 +1612,13 @@ int __init setup_fadump(void)
fadump_invalidate_release_mem();
}
/* Initialize the kernel dump memory structure for FAD registration. */
- else if (fw_dump.reserve_dump_area_size)
+ else if (fw_dump.reserve_dump_area_size) {
+ /* By this time cma area has been activated. Allocate memory
+ * for metadata from cma region.
+ */
+ allocate_metadata_area();
init_fadump_mem_struct(&fdm, fw_dump.reserve_dump_area_start);
+ }
fadump_init_files();
return 1;
^ permalink raw reply related
* Re: [PATCH] cxl: Perform NULL check for 'cxl_afu *' at various places in cxl
From: Vaibhav Jain @ 2018-03-15 6:14 UTC (permalink / raw)
To: Frederic Barrat, linuxppc-dev
In-Reply-To: <c97946a0-2488-5892-66ae-5ad8e5a0007f@linux.vnet.ibm.com>
Thanks for reviewing this patch Fred,
Frederic Barrat <fbarrat@linux.vnet.ibm.com> writes:
> So we are calling functions with an invalid afu argument. We can verify
> in the callees the value of the afu pointer, like you're doing here, but
> why not tackle it at source and avoid calling the function in the first
> place?
Couple of reasons:
* Having checks for NULL afu at the call sites rather than the functions
being called will lead to these checks scattered at more than a few
places in code. Many of these functions are called via indirect branch
from cxl_backend_ops and have their call sites scattered in the
code. Having this check in the called functions is a much smaller change
in comparison.
* Some of these functions are called are called from outside CXL e.g
cxl_dev_context_init(). So putting the NULL check at every caller site
might not be possible.
* Some funtions need to handle AFU init failures in special way e.g like
cxl_vphb_error_detected() need to force a card hotplug so that AFU
reinit can be tried.
* None of these functions are in any hotpath in the code so having an
extra check at the beginning of the code wont have a much performance
impact.
--
Vaibhav Jain <vaibhav@linux.vnet.ibm.com>
Linux Technology Center, IBM India Pvt. Ltd.
^ permalink raw reply
* Re: [PATCH kernel] powerpc/npu: Do not try invalidating 32bit table when 64bit table is enabled
From: Alistair Popple @ 2018-03-15 5:18 UTC (permalink / raw)
To: Alexey Kardashevskiy; +Cc: linuxppc-dev, David Gibson, Russell Currey
In-Reply-To: <20180213055135.25639-1-aik@ozlabs.ru>
I must admit I haven't really looked at the iommu_table_group code in depth so
don't fully understand it, but I was wondering why we don't hit this for NVLink1
as well?
The error that Skiboot is printing is because the requested page size doesn't
match expected so I guess there is a possibility the address might not meet the
required alignment (I don't know we don't just check that directly).
The reason you don't see this on NVLink1 is that we rely on the kernel to do the
TCE invalidate directly rather than via an OPAL call, and the kernel seems to do
less checking. So I wonder if there is a bug there as well (invalidate for a
mismatched page size/address alignment)? And wether this would also fix that?
- Alistair
On Tuesday, 13 February 2018 4:51:35 PM AEDT Alexey Kardashevskiy wrote:
> GPUs and the corresponding NVLink bridges get different PEs as they have
> separate translation validation entries (TVEs). We put these PEs to
> the same IOMMU group so they cannot be passed through separately.
> So the iommu_table_group_ops::set_window/unset_window for GPUs do set
> tables to the NPU PEs as well which means that iommu_table's list of
> attached PEs (iommu_table_group_link) has both GPU and NPU PEs linked.
> This list is used for TCE cache invalidation.
>
> The problem is that NPU PE has just a single TVE and can be programmed
> to point to 32bit or 64bit windows while GPU PE has two (as any other PCI
> device). So we end up having an 32bit iommu_table struct linked to both
> PEs even though only the 64bit TCE table cache can be invalidated on NPU.
> And a relatively recent skiboot detects this and prints errors.
>
> This changes GPU's iommu_table_group_ops::set_window/unset_window to make
> sure that NPU PE is only linked to the table actually used by the hardware.
> If there are two tables used by an IOMMU group, the NPU PE will use
> the last programmed one which with the current use scenarios is expected
> to be a 64bit one.
>
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> ---
>
> Do we need BUG_ON(IOMMU_TABLE_GROUP_MAX_TABLES != 2)?
>
>
> This is an example for:
>
> 0004:04:00.0 3D: NVIDIA Corporation Device 1db1 (rev a1)
> 0006:00:00.0 Bridge: IBM Device 04ea (rev 01)
> 0006:00:00.1 Bridge: IBM Device 04ea (rev 01)
>
> Before the patch (npu2_tce_kill messages are from skiboot):
>
> pci 0004:04 : [PE# 00] Setting up window#0 0..3fffffff pg=1000
> pci 0006:00:00.0: [PE# 0d] Setting up window 0..3fffffff pg=1000
> pci 0004:04 : [PE# 00] Setting up window#1 800000000000000..8000000ffffffff pg=10000
> pci 0006:00:00.0: [PE# 0d] Setting up window 800000000000000..8000000ffffffff pg=10000
> NPU6: npu2_tce_kill: Unexpected TCE size (got 0x1000 expected 0x10000)
> NPU6: npu2_tce_kill: Unexpected TCE size (got 0x1000 expected 0x10000)
> NPU6: npu2_tce_kill: Unexpected TCE size (got 0x1000 expected 0x10000)
> NPU6: npu2_tce_kill: Unexpected TCE size (got 0x1000 expected 0x10000)
> NPU6: npu2_tce_kill: Unexpected TCE size (got 0x1000 expected 0x10000)
> ...
> pci 0004:04 : [PE# 00] Removing DMA window #0
> pci 0006:00:00.0: [PE# 0d] Removing DMA window
> pci 0004:04 : [PE# 00] Removing DMA window #1
> pci 0006:00:00.0: [PE# 0d] Removing DMA window
> pci 0004:04 : [PE# 00] Setting up window#0 0..3fffffff pg=1000
> pci 0006:00:00.0: [PE# 0d] Setting up window 0..3fffffff pg=1000
> pci 0004:04 : [PE# 00] Setting up window#1 800000000000000..8000000ffffffff pg=10000
> pci 0006:00:00.0: [PE# 0d] Setting up window 800000000000000..8000000ffffffff pg=10000
>
> After the patch (no errors here):
>
> pci 0004:04 : [PE# 00] Setting up window#0 0..3fffffff pg=1000
> pci 0006:00:00.0: [PE# 0d] Setting up window 0..3fffffff pg=1000
> pci 0004:04 : [PE# 00] Setting up window#1 800000000000000..8000000ffffffff pg=10000
> pci 0006:00:00.0: [PE# 0d] Removing DMA window
> pci 0006:00:00.0: [PE# 0d] Setting up window 800000000000000..8000000ffffffff pg=10000
> pci 0004:04 : [PE# 00] Removing DMA window #0
> pci 0004:04 : [PE# 00] Removing DMA window #1
> pci 0006:00:00.0: [PE# 0d] Removing DMA window
> pci 0004:04 : [PE# 00] Setting up window#0 0..3fffffff pg=1000
> pci 0006:00:00.0: [PE# 0d] Setting up window 0..3fffffff pg=1000
> pci 0004:04 : [PE# 00] Setting up window#1 800000000000000..8000000ffffffff pg=10000
> pci 0006:00:00.0: [PE# 0d] Removing DMA window
> pci 0006:00:00.0: [PE# 0d] Setting up window 800000000000000..8000000ffffffff pg=10000
> ---
> arch/powerpc/platforms/powernv/pci-ioda.c | 27 ++++++++++++++++++++++++---
> 1 file changed, 24 insertions(+), 3 deletions(-)
>
> diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
> index 496e476..2f91815 100644
> --- a/arch/powerpc/platforms/powernv/pci-ioda.c
> +++ b/arch/powerpc/platforms/powernv/pci-ioda.c
> @@ -2681,14 +2681,23 @@ static struct pnv_ioda_pe *gpe_table_group_to_npe(
> static long pnv_pci_ioda2_npu_set_window(struct iommu_table_group *table_group,
> int num, struct iommu_table *tbl)
> {
> + struct pnv_ioda_pe *npe = gpe_table_group_to_npe(table_group);
> + int num2 = (num == 0) ? 1 : 0;
> long ret = pnv_pci_ioda2_set_window(table_group, num, tbl);
>
> if (ret)
> return ret;
>
> - ret = pnv_npu_set_window(gpe_table_group_to_npe(table_group), num, tbl);
> - if (ret)
> + if (table_group->tables[num2])
> + pnv_npu_unset_window(npe, num2);
> +
> + ret = pnv_npu_set_window(npe, num, tbl);
> + if (ret) {
> pnv_pci_ioda2_unset_window(table_group, num);
> + if (table_group->tables[num2])
> + pnv_npu_set_window(npe, num2,
> + table_group->tables[num2]);
> + }
>
> return ret;
> }
> @@ -2697,12 +2706,24 @@ static long pnv_pci_ioda2_npu_unset_window(
> struct iommu_table_group *table_group,
> int num)
> {
> + struct pnv_ioda_pe *npe = gpe_table_group_to_npe(table_group);
> + int num2 = (num == 0) ? 1 : 0;
> long ret = pnv_pci_ioda2_unset_window(table_group, num);
>
> if (ret)
> return ret;
>
> - return pnv_npu_unset_window(gpe_table_group_to_npe(table_group), num);
> + if (!npe->table_group.tables[num])
> + return 0;
> +
> + ret = pnv_npu_unset_window(npe, num);
> + if (ret)
> + return ret;
> +
> + if (table_group->tables[num2])
> + ret = pnv_npu_set_window(npe, num2, table_group->tables[num2]);
> +
> + return ret;
> }
>
> static void pnv_ioda2_npu_take_ownership(struct iommu_table_group *table_group)
>
^ permalink raw reply
* [PATCHv5 3/3] ppc64 boot: Wait for boot cpu to show up if nr_cpus limit is about to hit.
From: Pingfan Liu @ 2018-03-15 4:41 UTC (permalink / raw)
To: linuxppc-dev; +Cc: cascardo, gpiccoli, kexec, paulus, mpe, benh
In-Reply-To: <1521088912-31742-1-git-send-email-kernelfans@gmail.com>
From: Mahesh Salgaonkar <mahesh@linux.vnet.ibm.com>
The kernel boot parameter 'nr_cpus=' allows one to specify number of
possible cpus in the system. In the normal scenario the first cpu (cpu0)
that shows up is the boot cpu and hence it gets covered under nr_cpus
limit.
But this assumption will be broken in kdump scenario where kdump kenrel
after a crash can boot up on an non-zero boot cpu. The paca structure
allocation depends on value of nr_cpus and is indexed using logical cpu
ids. This definetly will be an issue if boot cpu id > nr_cpus
This patch modifies allocate_pacas() and smp_setup_cpu_maps() to
accommodate boot cpu for the case where boot_cpuid > nr_cpu_ids.
This change would help to reduce the memory reservation requirement for
kdump on ppc64.
Signed-off-by: Mahesh Salgaonkar <mahesh@linux.vnet.ibm.com>
Signed-off-by: Thadeu Lima de Souza Cascardo <cascardo@canonical.com>
Tested-by: Guilherme G. Piccoli <gpiccoli@linux.vnet.ibm.com>
Signed-off-by: Pingfan Liu <kernelfans@gmail.com> (separate the logical for
cpu id mapping and stick to the semantics of nr_cpus)
Signed-off-by: Pingfan Liu <kernelfans@gmail.com>
---
arch/powerpc/include/asm/paca.h | 3 +++
arch/powerpc/kernel/paca.c | 19 ++++++++++++++-----
2 files changed, 17 insertions(+), 5 deletions(-)
diff --git a/arch/powerpc/include/asm/paca.h b/arch/powerpc/include/asm/paca.h
index b62c310..49ab29d 100644
--- a/arch/powerpc/include/asm/paca.h
+++ b/arch/powerpc/include/asm/paca.h
@@ -49,6 +49,9 @@ extern unsigned int debug_smp_processor_id(void); /* from linux/smp.h */
#define get_lppaca() (get_paca()->lppaca_ptr)
#define get_slb_shadow() (get_paca()->slb_shadow_ptr)
+/* Maximum number of threads per core. */
+#define MAX_SMT 8
+
struct task_struct;
/*
diff --git a/arch/powerpc/kernel/paca.c b/arch/powerpc/kernel/paca.c
index 95ffedf..9c6f135 100644
--- a/arch/powerpc/kernel/paca.c
+++ b/arch/powerpc/kernel/paca.c
@@ -209,6 +209,7 @@ void __init allocate_pacas(void)
{
u64 limit;
int cpu;
+ unsigned int nr_cpus_aligned;
#ifdef CONFIG_PPC_BOOK3S_64
/*
@@ -220,20 +221,28 @@ void __init allocate_pacas(void)
limit = ppc64_rma_size;
#endif
- paca_size = PAGE_ALIGN(sizeof(struct paca_struct) * nr_cpu_ids);
+ /*
+ * Alloc the paca[] align up to SMT threads.
+ * This will help us to prepare for a situation where
+ * boot cpu id > nr_cpus.
+ * We keep the semantics of nr_cpus in kernel cmdline, but
+ * waste a bit memory
+ */
+ nr_cpus_aligned = _ALIGN_UP(nr_cpu_ids, MAX_SMT);
+ paca_size = PAGE_ALIGN(sizeof(struct paca_struct) * nr_cpus_aligned);
paca = __va(memblock_alloc_base(paca_size, PAGE_SIZE, limit));
memset(paca, 0, paca_size);
printk(KERN_DEBUG "Allocated %u bytes for %u pacas at %p\n",
- paca_size, nr_cpu_ids, paca);
+ paca_size, nr_cpus_aligned, paca);
- allocate_lppacas(nr_cpu_ids, limit);
+ allocate_lppacas(nr_cpus_aligned, limit);
- allocate_slb_shadows(nr_cpu_ids, limit);
+ allocate_slb_shadows(nr_cpus_aligned, limit);
/* Can't use for_each_*_cpu, as they aren't functional yet */
- for (cpu = 0; cpu < nr_cpu_ids; cpu++)
+ for (cpu = 0; cpu < nr_cpus_aligned; cpu++)
initialise_paca(&paca[cpu], cpu);
}
--
2.7.4
^ permalink raw reply related
* [PATCHv5 2/3] powerpc, cpu: handling the special case when boot_cpuid greater than nr_cpus
From: Pingfan Liu @ 2018-03-15 4:41 UTC (permalink / raw)
To: linuxppc-dev; +Cc: cascardo, gpiccoli, kexec, paulus, mpe, benh
In-Reply-To: <1521088912-31742-1-git-send-email-kernelfans@gmail.com>
For kexec -p, after boot_cpuid is mapping into the range of [0,
threads_per_core), then if nr_cpus is small, we will have the bitmap
[0,..., nr_cpus, ..., boot_cpuid, ...). This patch chooses cpus inside
the range of [boot_cpuid - nr_cpus +1, ..., boot_cpuid] to be online.
With this patch and the next, on a P9 machine with thread_per_core=4,
and set nr_cpus=2 for the crash kernel.
After
taskset -c 11 sh -c "echo c > /proc/sysrq-trigger"
Then
kdump:/sys/devices/system/cpu# cat possible
2-3
kdump:/sys/devices/system/cpu# cat present
2-3
kdump:/sys/devices/system/cpu# cat online
2-3
Signed-off-by: Pingfan Liu <kernelfans@gmail.com>
---
arch/powerpc/kernel/setup-common.c | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/arch/powerpc/kernel/setup-common.c b/arch/powerpc/kernel/setup-common.c
index a683ed1..f3aaf9f 100644
--- a/arch/powerpc/kernel/setup-common.c
+++ b/arch/powerpc/kernel/setup-common.c
@@ -468,6 +468,7 @@ void __init smp_setup_cpu_maps(void)
int online_num = 0;
int next_cpu = boot_dn_nthreads;
bool btdn_handled = false;
+ int first_threadid = 0;
DBG("smp_setup_cpu_maps()\n");
@@ -497,7 +498,12 @@ void __init smp_setup_cpu_maps(void)
if (boot_cpuid < nthreads &&
be32_to_cpu(intserv[boot_cpuid]) == boot_cpuhwid) {
- cpu = 0;
+ /* choose a bunch of continous threads */
+ if (boot_cpuid > nr_cpu_ids - 1) {
+ first_threadid = boot_cpuid - nr_cpu_ids + 1;
+ }
+ /* keep the mapping of logical and thread */
+ cpu = first_threadid;
btdn_handled = true;
} else if ( !btdn_handled &&
nr_cpu_ids - online_num <= boot_dn_nthreads)
@@ -509,7 +515,8 @@ void __init smp_setup_cpu_maps(void)
}
- for (j = 0; j < nthreads && online_num < nr_cpu_ids; j++) {
+ for (j = first_threadid;
+ j < nthreads && online_num < nr_cpu_ids; j++) {
bool avail;
DBG(" thread %d -> cpu %d (hard id %d)\n",
--
2.7.4
^ permalink raw reply related
* [PATCHv5 1/3] powerpc, cpu: partially unbind the mapping between cpu logical id and its seq in dt
From: Pingfan Liu @ 2018-03-15 4:41 UTC (permalink / raw)
To: linuxppc-dev; +Cc: cascardo, gpiccoli, kexec, paulus, mpe, benh
In-Reply-To: <1521088912-31742-1-git-send-email-kernelfans@gmail.com>
For kexec -p, the boot cpu can be not the cpu0, this causes the problem
to alloc paca[]. In theory, there is no requirement to assign cpu's logical
id as its present seq by device tree. But we have something like
cpu_first_thread_sibling(), which makes assumption on the mapping inside
a core. Hence partially changing the mapping, i.e. unbind the mapping of
core while keep the mapping inside a core. After this patch, boot-cpu
will always be mapped into the range [0,threads_per_core).
Signed-off-by: Pingfan Liu <kernelfans@gmail.com>
---
arch/powerpc/include/asm/smp.h | 2 ++
arch/powerpc/kernel/prom.c | 27 ++++++++++++++++-----------
arch/powerpc/kernel/setup-common.c | 28 +++++++++++++++++++++++++---
3 files changed, 43 insertions(+), 14 deletions(-)
diff --git a/arch/powerpc/include/asm/smp.h b/arch/powerpc/include/asm/smp.h
index fac963e..920a927 100644
--- a/arch/powerpc/include/asm/smp.h
+++ b/arch/powerpc/include/asm/smp.h
@@ -30,6 +30,8 @@
#include <asm/percpu.h>
extern int boot_cpuid;
+extern int boot_cpuhwid;
+extern int boot_dn_nthreads;
extern int spinning_secondaries;
extern void cpu_die(void);
diff --git a/arch/powerpc/kernel/prom.c b/arch/powerpc/kernel/prom.c
index da67606..aca15c1 100644
--- a/arch/powerpc/kernel/prom.c
+++ b/arch/powerpc/kernel/prom.c
@@ -315,8 +315,7 @@ static int __init early_init_dt_scan_cpus(unsigned long node,
const __be32 *intserv;
int i, nthreads;
int len;
- int found = -1;
- int found_thread = 0;
+ bool found = false;
/* We are scanning "cpu" nodes only */
if (type == NULL || strcmp(type, "cpu") != 0)
@@ -341,8 +340,12 @@ static int __init early_init_dt_scan_cpus(unsigned long node,
if (fdt_version(initial_boot_params) >= 2) {
if (be32_to_cpu(intserv[i]) ==
fdt_boot_cpuid_phys(initial_boot_params)) {
- found = boot_cpu_count;
- found_thread = i;
+ /* always map the boot-cpu logical id into the
+ * the range of [0, thread_per_core)
+ */
+ boot_cpuid = i;
+ boot_dn_nthreads = nthreads;
+ found = true;
}
} else {
/*
@@ -351,8 +354,11 @@ static int __init early_init_dt_scan_cpus(unsigned long node,
* off secondary threads.
*/
if (of_get_flat_dt_prop(node,
- "linux,boot-cpu", NULL) != NULL)
- found = boot_cpu_count;
+ "linux,boot-cpu", NULL) != NULL) {
+ boot_cpuid = i;
+ boot_dn_nthreads = nthreads;
+ found = true;
+ }
}
#ifdef CONFIG_SMP
/* logical cpu id is always 0 on UP kernels */
@@ -361,13 +367,12 @@ static int __init early_init_dt_scan_cpus(unsigned long node,
}
/* Not the boot CPU */
- if (found < 0)
+ if (!found)
return 0;
- DBG("boot cpu: logical %d physical %d\n", found,
- be32_to_cpu(intserv[found_thread]));
- boot_cpuid = found;
- set_hard_smp_processor_id(found, be32_to_cpu(intserv[found_thread]));
+ boot_cpuhwid = be32_to_cpu(intserv[boot_cpuid]);
+ DBG("boot cpu: logical %d physical %d\n", boot_cpuid, boot_cpuhwid);
+ set_hard_smp_processor_id(boot_cpuid, boot_cpuhwid);
/*
* PAPR defines "logical" PVR values for cpus that
diff --git a/arch/powerpc/kernel/setup-common.c b/arch/powerpc/kernel/setup-common.c
index 66f7cc6..a683ed1 100644
--- a/arch/powerpc/kernel/setup-common.c
+++ b/arch/powerpc/kernel/setup-common.c
@@ -86,7 +86,11 @@ struct machdep_calls *machine_id;
EXPORT_SYMBOL(machine_id);
int boot_cpuid = -1;
+int boot_cpuhwid = -1;
+int boot_dn_nthreads = -1;
EXPORT_SYMBOL_GPL(boot_cpuid);
+EXPORT_SYMBOL_GPL(boot_cpuhwid);
+EXPORT_SYMBOL_GPL(boot_dn_nthreads);
/*
* These are used in binfmt_elf.c to put aux entries on the stack
@@ -461,6 +465,9 @@ void __init smp_setup_cpu_maps(void)
struct device_node *dn;
int cpu = 0;
int nthreads = 1;
+ int online_num = 0;
+ int next_cpu = boot_dn_nthreads;
+ bool btdn_handled = false;
DBG("smp_setup_cpu_maps()\n");
@@ -488,7 +495,21 @@ void __init smp_setup_cpu_maps(void)
nthreads = len / sizeof(int);
- for (j = 0; j < nthreads && cpu < nr_cpu_ids; j++) {
+ if (boot_cpuid < nthreads &&
+ be32_to_cpu(intserv[boot_cpuid]) == boot_cpuhwid) {
+ cpu = 0;
+ btdn_handled = true;
+ } else if ( !btdn_handled &&
+ nr_cpu_ids - online_num <= boot_dn_nthreads)
+ /* online threads in boot dn before using up quota */
+ continue;
+ else {
+ cpu = next_cpu;
+ next_cpu += nthreads;
+ }
+
+
+ for (j = 0; j < nthreads && online_num < nr_cpu_ids; j++) {
bool avail;
DBG(" thread %d -> cpu %d (hard id %d)\n",
@@ -503,12 +524,13 @@ void __init smp_setup_cpu_maps(void)
set_hard_smp_processor_id(cpu, be32_to_cpu(intserv[j]));
set_cpu_possible(cpu, true);
cpu++;
+ online_num++;
}
-
- if (cpu >= nr_cpu_ids) {
+ if (online_num >= nr_cpu_ids) {
of_node_put(dn);
break;
}
+
}
/* If no SMT supported, nthreads is forced to 1 */
--
2.7.4
^ permalink raw reply related
* [PATCHv5 0/3] enable nr_cpus for powerpc
From: Pingfan Liu @ 2018-03-15 4:41 UTC (permalink / raw)
To: linuxppc-dev; +Cc: cascardo, gpiccoli, kexec, paulus, mpe, benh
This topic has a very long history. It comes from Mahesh Salgaonkar <mahesh@linux.vnet.ibm.com>
For v3: https://patchwork.ozlabs.org/patch/834860/
I hope we can acquire it for "kexec -p" soon.
V4->V5:
improve the [1/3] implementation based on Benjamin's suggestion.
Mahesh Salgaonkar (1):
ppc64 boot: Wait for boot cpu to show up if nr_cpus limit is about to
hit.
Pingfan Liu (2):
powerpc, cpu: partially unbind the mapping between cpu logical id and
its seq in dt
powerpc, cpu: handling the special case when boot_cpuid greater than
nr_cpus
arch/powerpc/include/asm/paca.h | 3 +++
arch/powerpc/include/asm/smp.h | 2 ++
arch/powerpc/kernel/paca.c | 19 ++++++++++++++-----
arch/powerpc/kernel/prom.c | 27 ++++++++++++++++-----------
arch/powerpc/kernel/setup-common.c | 35 ++++++++++++++++++++++++++++++++---
5 files changed, 67 insertions(+), 19 deletions(-)
--
2.7.4
^ permalink raw reply
* Re: [Y2038] [PATCH v4 02/10] include: Move compat_timespec/ timeval to compat_time.h
From: Deepa Dinamani @ 2018-03-15 2:51 UTC (permalink / raw)
To: Arnd Bergmann
Cc: kbuild test robot, Mark Rutland,
open list:RALINK MIPS ARCHITECTURE, Peter Zijlstra,
Benjamin Herrenschmidt, Heiko Carstens, Paul Mackerras,
H. Peter Anvin, sparclinux, devel, linux-s390, y2038 Mailman List,
Michael Ellerman, Helge Deller, the arch/x86 maintainers, sebott,
James E.J. Bottomley, Christian Borntraeger, Ingo Molnar,
oprofile-list, Catalin Marinas, Peter Oberparleiter,
Robert Richter, Chris Metcalf, Will Deacon,
Arnaldo Carvalho de Melo, Julian Wiedmann, John Stultz,
Steven Rostedt, Thomas Gleixner, gerald.schaefer, Parisc List,
Greg KH, cohuck, Linux Kernel Mailing List, Ralf Baechle,
David S. Miller, Jan Hoeppner, kbuild-all, Stefan Haberland,
Martin Schwidefsky, linuxppc-dev, Ursula Braun
In-Reply-To: <CAK8P3a1fxWAK94GH0cpzh6CHXgL4uJuDNCGpdJen5ib1HH1xoA@mail.gmail.com>
On Wed, Mar 14, 2018 at 1:52 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> On Wed, Mar 14, 2018 at 4:50 AM, Deepa Dinamani <deepa.kernel@gmail.com> wrote:
>> The file arch/arm64/kernel/process.c needs asm/compat.h also to be
>> included directly since this is included conditionally from
>> include/compat.h. This does seem to be typical of arm64 as I was not
>> completely able to get rid of asm/compat.h includes for arm64 in this
>> series. My plan is to have separate patches to get rid of asm/compat.h
>> includes for the architectures that are not straight forward to keep
>> this series simple.
>> I will fix this and update the series.
>>
>
> I ran across the same thing in two more files during randconfig testing on
> arm64 now, adding this fixup on top for the moment, but maybe there
> is a better way:
I was looking at how Al tested his uaccess patches:
https://www.spinics.net/lists/linux-fsdevel/msg108752.html
He seems to be running the kbuild bot tests on his own git.
Is it possible to verify it this way on the 2038 tree? Or, I could
host a tree also.
Thanks,
Deepa
^ permalink raw reply
* [TESTS] Create 'enabled_flush_types' boot option, and short-circuit LPM
From: Mauricio Faria de Oliveira @ 2018-03-14 22:40 UTC (permalink / raw)
To: linuxppc-dev, mpe, msuchanek
In-Reply-To: <1521067243-19520-1-git-send-email-mauricfo@linux.vnet.ibm.com>
fallback -> fallback:
# dmesg -c | grep rfi-flush
[ 0.000000] rfi-flush (debug): enabled flush types: 0x0
[ 0.000000] rfi-flush: fallback displacement flush available
[ 0.000000] rfi-flush: patched 8 locations (fallback displacement flush)
# echo > /sys/kernel/mobility/migration
# dmesg -c | grep rfi-flush
[ 32.525443] rfi-flush: fallback displacement flush available
[ 32.526269] rfi-flush: patched 8 locations (fallback displacement flush)
fallback -> instructions:
# dmesg -c | grep rfi-flush
[ 0.000000] rfi-flush (debug): enabled flush types: 0x0
[ 0.000000] rfi-flush: fallback displacement flush available
[ 0.000000] rfi-flush: patched 8 locations (fallback displacement flush)
# echo 24 > /sys/kernel/debug/powerpc/rfi_flush
# dmesg -c | grep rfi-flush
[ 30.984247] rfi-flush (debug): enabled flush types: 0xc
# echo > /sys/kernel/mobility/migration
# dmesg -c | grep rfi-flush
[ 51.554357] rfi-flush: fallback displacement flush available
[ 51.554360] rfi-flush: ori type flush available
[ 51.554361] rfi-flush: mttrig type flush available
[ 51.554366] rfi-flush: patched 8 locations (ori+mttrig type flush)
instructions -> instructions:
# dmesg -c | grep rfi-flush
[ 0.000000] rfi-flush (debug): enabled flush types: 0xc
[ 0.000000] rfi-flush: fallback displacement flush available
[ 0.000000] rfi-flush: ori type flush available
[ 0.000000] rfi-flush: mttrig type flush available
[ 0.000000] rfi-flush: patched 8 locations (ori+mttrig type flush)
# echo > /sys/kernel/mobility/migration
# dmesg -c | grep rfi-flush
[ 55.100566] rfi-flush: fallback displacement flush available
[ 55.100570] rfi-flush: ori type flush available
[ 55.100571] rfi-flush: mttrig type flush available
[ 55.100575] rfi-flush: patched 8 locations (ori+mttrig type flush)
instructions -> fallback:
# dmesg -c | grep rfi-flush
[ 0.000000] rfi-flush (debug): enabled flush types: 0xc
[ 0.000000] rfi-flush: fallback displacement flush available
[ 0.000000] rfi-flush: ori type flush available
[ 0.000000] rfi-flush: mttrig type flush available
[ 0.000000] rfi-flush: patched 8 locations (ori+mttrig type flush)
# echo 1111 > /sys/kernel/debug/powerpc/rfi_flush
# dmesg -c | grep rfi-flush
[ 18.730782] rfi-flush (debug): enabled flush types: 0x0
# echo > /sys/kernel/mobility/migration
# dmesg -c | grep rfi-flush
[ 27.120071] rfi-flush: fallback displacement flush available
[ 27.120078] rfi-flush: patched 8 locations (fallback displacement flush)
debugfs switch:
# echo 0 > /sys/kernel/debug/powerpc/rfi_flush
# echo 1 > /sys/kernel/debug/powerpc/rfi_flush
# dmesg -c | grep rfi-flush
[ 106.031993] rfi-flush: patched 8 locations (no flush)
[ 109.670966] rfi-flush: patched 8 locations (fallback displacement flush)
ori type only:
# echo 8 > /sys/kernel/debug/powerpc/rfi_flush
# echo 0 > /sys/kernel/debug/powerpc/rfi_flush
# echo 1 > /sys/kernel/debug/powerpc/rfi_flush
# dmesg -c | grep rfi-flush
[ 308.988958] rfi-flush (debug): enabled flush types: 0x4
[ 314.206548] rfi-flush: patched 8 locations (no flush)
[ 316.349916] rfi-flush: patched 8 locations (ori type flush)
mttrig type only:
# echo 16 > /sys/kernel/debug/powerpc/rfi_flush
# echo 0 > /sys/kernel/debug/powerpc/rfi_flush
# echo 1 > /sys/kernel/debug/powerpc/rfi_flush
# dmesg -c | grep rfi-flush
[ 355.993189] rfi-flush (debug): enabled flush types: 0x8
[ 360.644291] rfi-flush: patched 8 locations (no flush)
[ 365.300547] rfi-flush: patched 8 locations (mttrig type flush)
Signed-off-by: Mauricio Faria de Oliveira <mauricfo@linux.vnet.ibm.com>
---
arch/powerpc/kernel/setup_64.c | 29 +++++++++++++++++++++++++++++
arch/powerpc/platforms/pseries/mobility.c | 4 ++++
2 files changed, 33 insertions(+)
diff --git a/arch/powerpc/kernel/setup_64.c b/arch/powerpc/kernel/setup_64.c
index 4ec4a27..9c9568e 100644
--- a/arch/powerpc/kernel/setup_64.c
+++ b/arch/powerpc/kernel/setup_64.c
@@ -816,6 +816,24 @@ static void *l1d_flush_fallback_area;
static bool no_rfi_flush;
bool rfi_flush;
+static int __init handle_enabled_flush_types(char *p)
+{
+ int rc;
+ enum l1d_flush_type types;
+
+ rc = kstrtoul(p, 0, (long unsigned int *)&types);
+ if (!rc) {
+ enabled_flush_types = types;
+ pr_info("rfi-flush (debug): enabled flush types: 0x%x\n", enabled_flush_types);
+ } else {
+ enabled_flush_types = L1D_FLUSH_NONE;
+ pr_info("rfi-flush (debug): enabled flush types is invalid\n");
+ }
+
+ return rc;
+}
+early_param("enabled_flush_types", handle_enabled_flush_types);
+
static int __init handle_no_rfi_flush(char *p)
{
pr_info("rfi-flush: disabled on command line.");
@@ -883,6 +901,8 @@ static void init_fallback_flush(void)
void setup_rfi_flush(enum l1d_flush_type types, bool enable)
{
+ types |= enabled_flush_types;
+
if (types & L1D_FLUSH_FALLBACK) {
pr_info("rfi-flush: fallback displacement flush available\n");
init_fallback_flush();
@@ -909,6 +929,15 @@ static int rfi_flush_set(void *data, u64 val)
enable = true;
else if (val == 0)
enable = false;
+ else if (val == 1111) {
+ enable = true;
+ enabled_flush_types = 0;
+ pr_info("rfi-flush (debug): enabled flush types: 0x%x\n", enabled_flush_types);
+ } else if (val >= 2) {
+ enable = true;
+ enabled_flush_types = val >> 1;
+ pr_info("rfi-flush (debug): enabled flush types: 0x%x\n", enabled_flush_types);
+ }
else
return -EINVAL;
diff --git a/arch/powerpc/platforms/pseries/mobility.c b/arch/powerpc/platforms/pseries/mobility.c
index 8a8033a..2a8458a 100644
--- a/arch/powerpc/platforms/pseries/mobility.c
+++ b/arch/powerpc/platforms/pseries/mobility.c
@@ -326,6 +326,7 @@ int pseries_devicetree_update(s32 scope)
void post_mobility_fixup(void)
{
+#if 0
int rc;
int activate_fw_token;
@@ -348,6 +349,7 @@ void post_mobility_fixup(void)
printk(KERN_ERR "Post-mobility device tree update "
"failed: %d\n", rc);
+#endif
/* Possibly switch to a new RFI flush type */
pseries_setup_rfi_flush();
@@ -358,6 +360,7 @@ static ssize_t migration_store(struct class *class,
struct class_attribute *attr, const char *buf,
size_t count)
{
+#if 0
u64 streamid;
int rc;
@@ -374,6 +377,7 @@ static ssize_t migration_store(struct class *class,
if (rc)
return rc;
+#endif
post_mobility_fixup();
return count;
}
--
2.7.4
^ permalink raw reply related
* [PATCH v3 5/5] rfi-flush: Call setup_rfi_flush() after LPM migration
From: Mauricio Faria de Oliveira @ 2018-03-14 22:40 UTC (permalink / raw)
To: linuxppc-dev, mpe, msuchanek
In-Reply-To: <1521067243-19520-1-git-send-email-mauricfo@linux.vnet.ibm.com>
From: Michael Ellerman <mpe@ellerman.id.au>
We might have migrated to a machine that uses a different flush type,
or doesn't need flushing at all.
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Signed-off-by: Mauricio Faria de Oliveira <mauricfo@linux.vnet.ibm.com>
---
arch/powerpc/platforms/pseries/mobility.c | 3 +++
arch/powerpc/platforms/pseries/pseries.h | 2 ++
arch/powerpc/platforms/pseries/setup.c | 2 +-
3 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/arch/powerpc/platforms/pseries/mobility.c b/arch/powerpc/platforms/pseries/mobility.c
index 0f7fb71..8a8033a 100644
--- a/arch/powerpc/platforms/pseries/mobility.c
+++ b/arch/powerpc/platforms/pseries/mobility.c
@@ -348,6 +348,9 @@ void post_mobility_fixup(void)
printk(KERN_ERR "Post-mobility device tree update "
"failed: %d\n", rc);
+ /* Possibly switch to a new RFI flush type */
+ pseries_setup_rfi_flush();
+
return;
}
diff --git a/arch/powerpc/platforms/pseries/pseries.h b/arch/powerpc/platforms/pseries/pseries.h
index c73351c..60db2ee 100644
--- a/arch/powerpc/platforms/pseries/pseries.h
+++ b/arch/powerpc/platforms/pseries/pseries.h
@@ -108,4 +108,6 @@ static inline unsigned long cmo_get_page_size(void)
int dlpar_workqueue_init(void);
+void pseries_setup_rfi_flush(void);
+
#endif /* _PSERIES_PSERIES_H */
diff --git a/arch/powerpc/platforms/pseries/setup.c b/arch/powerpc/platforms/pseries/setup.c
index b20d107..f34f908 100644
--- a/arch/powerpc/platforms/pseries/setup.c
+++ b/arch/powerpc/platforms/pseries/setup.c
@@ -459,7 +459,7 @@ static void __init find_and_init_phbs(void)
of_pci_check_probe_only();
}
-static void pseries_setup_rfi_flush(void)
+void pseries_setup_rfi_flush(void)
{
struct h_cpu_char_result result;
enum l1d_flush_type types;
--
2.7.4
^ permalink raw reply related
* [PATCH v3 4/5] rfi-flush: Differentiate enabled and patched flush types
From: Mauricio Faria de Oliveira @ 2018-03-14 22:40 UTC (permalink / raw)
To: linuxppc-dev, mpe, msuchanek
In-Reply-To: <1521067243-19520-1-git-send-email-mauricfo@linux.vnet.ibm.com>
Currently the rfi-flush messages print 'Using <type> flush' for all
enabled_flush_types, but that is not necessarily true -- as now the
fallback flush is always enabled on pseries, but the fixup function
overwrites its nop/branch slot with other flush types, if available.
So, replace the 'Using <type> flush' messages with '<type> flush is
available'.
Also, print the patched flush types in the fixup function, so users
can know what is (not) being used (e.g., the slower, fallback flush,
or no flush type at all if flush is disabled via the debugfs switch).
Suggested-by: Michael Ellerman <mpe@ellerman.id.au>
Signed-off-by: Mauricio Faria de Oliveira <mauricfo@linux.vnet.ibm.com>
---
P.S.: Michael, you suggested only hunk 1. please feel free to discard
hunk 2 if you don't like it.
arch/powerpc/kernel/setup_64.c | 6 +++---
arch/powerpc/lib/feature-fixups.c | 9 ++++++++-
2 files changed, 11 insertions(+), 4 deletions(-)
diff --git a/arch/powerpc/kernel/setup_64.c b/arch/powerpc/kernel/setup_64.c
index d60e2f7..4ec4a27 100644
--- a/arch/powerpc/kernel/setup_64.c
+++ b/arch/powerpc/kernel/setup_64.c
@@ -884,15 +884,15 @@ static void init_fallback_flush(void)
void setup_rfi_flush(enum l1d_flush_type types, bool enable)
{
if (types & L1D_FLUSH_FALLBACK) {
- pr_info("rfi-flush: Using fallback displacement flush\n");
+ pr_info("rfi-flush: fallback displacement flush available\n");
init_fallback_flush();
}
if (types & L1D_FLUSH_ORI)
- pr_info("rfi-flush: Using ori type flush\n");
+ pr_info("rfi-flush: ori type flush available\n");
if (types & L1D_FLUSH_MTTRIG)
- pr_info("rfi-flush: Using mttrig type flush\n");
+ pr_info("rfi-flush: mttrig type flush available\n");
enabled_flush_types = types;
diff --git a/arch/powerpc/lib/feature-fixups.c b/arch/powerpc/lib/feature-fixups.c
index 73697c4..35f80ab 100644
--- a/arch/powerpc/lib/feature-fixups.c
+++ b/arch/powerpc/lib/feature-fixups.c
@@ -153,7 +153,14 @@ void do_rfi_flush_fixups(enum l1d_flush_type types)
patch_instruction(dest + 2, instrs[2]);
}
- printk(KERN_DEBUG "rfi-flush: patched %d locations\n", i);
+ printk(KERN_DEBUG "rfi-flush: patched %d locations (%s flush)\n", i,
+ (types == L1D_FLUSH_NONE) ? "no" :
+ (types == L1D_FLUSH_FALLBACK) ? "fallback displacement" :
+ (types & L1D_FLUSH_ORI) ? (types & L1D_FLUSH_MTTRIG)
+ ? "ori+mttrig type"
+ : "ori type" :
+ (types & L1D_FLUSH_MTTRIG) ? "mttrig type"
+ : "unknown");
}
#endif /* CONFIG_PPC_BOOK3S_64 */
--
2.7.4
^ permalink raw reply related
* [PATCH v3 3/5] rfi-flush: Always enable fallback flush on pseries
From: Mauricio Faria de Oliveira @ 2018-03-14 22:40 UTC (permalink / raw)
To: linuxppc-dev, mpe, msuchanek
In-Reply-To: <1521067243-19520-1-git-send-email-mauricfo@linux.vnet.ibm.com>
From: Michael Ellerman <mpe@ellerman.id.au>
This ensures the fallback flush area is always allocated on pseries,
so in case a LPAR is migrated from a patched to an unpatched system,
it is possible to enable the fallback flush in the target system.
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Signed-off-by: Mauricio Faria de Oliveira <mauricfo@linux.vnet.ibm.com>
---
arch/powerpc/platforms/pseries/setup.c | 10 +---------
1 file changed, 1 insertion(+), 9 deletions(-)
diff --git a/arch/powerpc/platforms/pseries/setup.c b/arch/powerpc/platforms/pseries/setup.c
index 4642e48..b20d107 100644
--- a/arch/powerpc/platforms/pseries/setup.c
+++ b/arch/powerpc/platforms/pseries/setup.c
@@ -468,26 +468,18 @@ static void pseries_setup_rfi_flush(void)
/* Enable by default */
enable = true;
+ types = L1D_FLUSH_FALLBACK;
rc = plpar_get_cpu_characteristics(&result);
if (rc == H_SUCCESS) {
- types = L1D_FLUSH_NONE;
-
if (result.character & H_CPU_CHAR_L1D_FLUSH_TRIG2)
types |= L1D_FLUSH_MTTRIG;
if (result.character & H_CPU_CHAR_L1D_FLUSH_ORI30)
types |= L1D_FLUSH_ORI;
- /* Use fallback if nothing set in hcall */
- if (types == L1D_FLUSH_NONE)
- types = L1D_FLUSH_FALLBACK;
-
if ((!(result.behaviour & H_CPU_BEHAV_L1D_FLUSH_PR)) ||
(!(result.behaviour & H_CPU_BEHAV_FAVOUR_SECURITY)))
enable = false;
- } else {
- /* Default to fallback if case hcall is not available */
- types = L1D_FLUSH_FALLBACK;
}
setup_rfi_flush(types, enable);
--
2.7.4
^ permalink raw reply related
* [PATCH v3 2/5] rfi-flush: Make it possible to call setup_rfi_flush() again
From: Mauricio Faria de Oliveira @ 2018-03-14 22:40 UTC (permalink / raw)
To: linuxppc-dev, mpe, msuchanek
In-Reply-To: <1521067243-19520-1-git-send-email-mauricfo@linux.vnet.ibm.com>
From: Michael Ellerman <mpe@ellerman.id.au>
For PowerVM migration we want to be able to call setup_rfi_flush()
again after we've migrated the partition.
To support that we need to check that we're not trying to allocate the
fallback flush area after memblock has gone away (i.e., boot-time only).
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Signed-off-by: Mauricio Faria de Oliveira <mauricfo@linux.vnet.ibm.com>
---
arch/powerpc/include/asm/setup.h | 2 +-
arch/powerpc/kernel/setup_64.c | 6 +++++-
2 files changed, 6 insertions(+), 2 deletions(-)
diff --git a/arch/powerpc/include/asm/setup.h b/arch/powerpc/include/asm/setup.h
index 469b7fd..bbcdf92 100644
--- a/arch/powerpc/include/asm/setup.h
+++ b/arch/powerpc/include/asm/setup.h
@@ -49,7 +49,7 @@ enum l1d_flush_type {
L1D_FLUSH_MTTRIG = 0x8,
};
-void __init setup_rfi_flush(enum l1d_flush_type, bool enable);
+void setup_rfi_flush(enum l1d_flush_type, bool enable);
void do_rfi_flush_fixups(enum l1d_flush_type types);
#endif /* !__ASSEMBLY__ */
diff --git a/arch/powerpc/kernel/setup_64.c b/arch/powerpc/kernel/setup_64.c
index 3efc01a..d60e2f7 100644
--- a/arch/powerpc/kernel/setup_64.c
+++ b/arch/powerpc/kernel/setup_64.c
@@ -860,6 +860,10 @@ static void init_fallback_flush(void)
u64 l1d_size, limit;
int cpu;
+ /* Only allocate the fallback flush area once (at boot time). */
+ if (l1d_flush_fallback_area)
+ return;
+
l1d_size = ppc64_caches.l1d.size;
limit = min(ppc64_bolted_size(), ppc64_rma_size);
@@ -877,7 +881,7 @@ static void init_fallback_flush(void)
}
}
-void __init setup_rfi_flush(enum l1d_flush_type types, bool enable)
+void setup_rfi_flush(enum l1d_flush_type types, bool enable)
{
if (types & L1D_FLUSH_FALLBACK) {
pr_info("rfi-flush: Using fallback displacement flush\n");
--
2.7.4
^ permalink raw reply related
* [PATCH v3 1/5] rfi-flush: Move the logic to avoid a redo into the debugfs code
From: Mauricio Faria de Oliveira @ 2018-03-14 22:40 UTC (permalink / raw)
To: linuxppc-dev, mpe, msuchanek
In-Reply-To: <1521067243-19520-1-git-send-email-mauricfo@linux.vnet.ibm.com>
From: Michael Ellerman <mpe@ellerman.id.au>
rfi_flush_enable() includes a check to see if we're already
enabled (or disabled), and in that case does nothing.
But that means calling setup_rfi_flush() a 2nd time doesn't actually
work, which is a bit confusing.
Move that check into the debugfs code, where it really belongs.
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Signed-off-by: Mauricio Faria de Oliveira <mauricfo@linux.vnet.ibm.com>
---
arch/powerpc/kernel/setup_64.c | 13 ++++++++-----
1 file changed, 8 insertions(+), 5 deletions(-)
diff --git a/arch/powerpc/kernel/setup_64.c b/arch/powerpc/kernel/setup_64.c
index c388cc3..3efc01a 100644
--- a/arch/powerpc/kernel/setup_64.c
+++ b/arch/powerpc/kernel/setup_64.c
@@ -846,9 +846,6 @@ static void do_nothing(void *unused)
void rfi_flush_enable(bool enable)
{
- if (rfi_flush == enable)
- return;
-
if (enable) {
do_rfi_flush_fixups(enabled_flush_types);
on_each_cpu(do_nothing, NULL, 1);
@@ -902,13 +899,19 @@ void __init setup_rfi_flush(enum l1d_flush_type types, bool enable)
#ifdef CONFIG_DEBUG_FS
static int rfi_flush_set(void *data, u64 val)
{
+ bool enable;
+
if (val == 1)
- rfi_flush_enable(true);
+ enable = true;
else if (val == 0)
- rfi_flush_enable(false);
+ enable = false;
else
return -EINVAL;
+ /* Only do anything if we're changing state */
+ if (enable != rfi_flush)
+ rfi_flush_enable(enable);
+
return 0;
}
--
2.7.4
^ permalink raw reply related
* [PATCH v3 0/5] Setup RFI flush after PowerVM LPM migration
From: Mauricio Faria de Oliveira @ 2018-03-14 22:40 UTC (permalink / raw)
To: linuxppc-dev, mpe, msuchanek
This patchset allows for setup_rfi_flush() to be called again
after PowerVM LPM (live partition mobility) aka LPAR migration,
in order to possibly switch to a different flush method.
The patches are mostly from Michael Ellerman, I have rebased to
powerpc/linux.git merge branch as of commit d8443ef (March 14).
Testcase and results sent as the last email in the series.
v3: add patch 4 to tell flush types are 'available' and not 'Using'.
remove plumbing patch.
v2: add plumbing patch between platforms and setup code to
force init fallback flush.
Mauricio Faria de Oliveira (1):
rfi-flush: Differentiate enabled and patched flush types
Michael Ellerman (4):
rfi-flush: Move the logic to avoid a redo into the debugfs code
rfi-flush: Make it possible to call setup_rfi_flush() again
rfi-flush: Always enable fallback flush on pseries
rfi-flush: Call setup_rfi_flush() after LPM migration
arch/powerpc/include/asm/setup.h | 2 +-
arch/powerpc/kernel/setup_64.c | 25 ++++++++++++++++---------
arch/powerpc/lib/feature-fixups.c | 9 ++++++++-
arch/powerpc/platforms/pseries/mobility.c | 3 +++
arch/powerpc/platforms/pseries/pseries.h | 2 ++
arch/powerpc/platforms/pseries/setup.c | 12 ++----------
6 files changed, 32 insertions(+), 21 deletions(-)
--
2.7.4
^ permalink raw reply
* Re: [RFC] powerpc/xive: Remove irq from queue when it is shutdown
From: Cédric Le Goater @ 2018-03-14 17:47 UTC (permalink / raw)
To: Frederic Barrat, linuxppc-dev, benh; +Cc: andrew.donnellan, vaibhav
In-Reply-To: <20180314165808.26628-1-fbarrat@linux.vnet.ibm.com>
On 03/14/2018 05:58 PM, Frederic Barrat wrote:
> If a driver has called free_irq() but an irq is waiting in a queue, an
> error is logged when a cpu processes it:
>
> irq 232, desc: 0000000044e5941a, depth: 1, count: 9823, unhandled: 0
> ->handle_irq(): 0000000023f2e352,
> handle_bad_irq+0x0/0x2e0
> ->irq_data.chip(): 000000007fd7bf50,
> no_irq_chip+0x0/0x110
> ->action(): (null)
> IRQ_NOREQUEST set
> unexpected IRQ trap at vector e8
>
> In most cases, it's due to a driver bug or a misbehaving device, but
> it can be observed with opencapi with no involvment of a device. AFU
> interrupts are triggered by writing a special page of a process, but
> it's possible for a thread of that process to write to that page as
> well. If that process exits abruptly, the driver will free the AFU
> interrupts resources, but there's no possible quiescing of the
> process, so we may have interrupts in the queue.
>
> This patch adds a scan of the queue when an interrupt is shutdown to
> replace any pending irq with XIVE_BAD_IRQ, since those are ignored.
> Also move when XIVE_BAD_IRQs are ignored closer to reading the queue,
> so that we can reset the CPPR if it's the last interrupt in the queue.
We could also loop on the ESB 'P' bit to wait for the irqs to be handled,
using :
xive_esb_read(irq, XIVE_ESB_GET)
which has no side effect. It looks simpler to me. Is that possible ?
> Signed-off-by: Frederic Barrat <fbarrat@linux.vnet.ibm.com>
> ---
> arch/powerpc/sysdev/xive/common.c | 69 ++++++++++++++++++++++++++++++++++++---
> 1 file changed, 65 insertions(+), 4 deletions(-)
>
> diff --git a/arch/powerpc/sysdev/xive/common.c b/arch/powerpc/sysdev/xive/common.c
> index 3459015092fa..91047bc7c731 100644
> --- a/arch/powerpc/sysdev/xive/common.c
> +++ b/arch/powerpc/sysdev/xive/common.c
> @@ -148,8 +148,16 @@ static u32 xive_scan_interrupts(struct xive_cpu *xc, bool just_peek)
> prio = ffs(xc->pending_prio) - 1;
> DBG_VERBOSE("scan_irq: trying prio %d\n", prio);
>
> - /* Try to fetch */
> - irq = xive_read_eq(&xc->queue[prio], just_peek);
> + /*
> + * Try to fetch.
> + * When not peeking, drop any BAD_IRQ on the floor
> + * now. If we let it reach get_irq() and it's the last
> + * one, then we'd need to rescan again to reset the
> + * CPPR
> + */
> + do {
> + irq = xive_read_eq(&xc->queue[prio], just_peek);
> + } while (irq == XIVE_BAD_IRQ && !just_peek);
>
> /* Found something ? That's it */
> if (irq)
> @@ -282,8 +290,6 @@ static unsigned int xive_get_irq(void)
> irq, xc->pending_prio);
>
> /* Return pending interrupt if any */
> - if (irq == XIVE_BAD_IRQ)
> - return 0;
> return irq;
> }
>
> @@ -592,6 +598,58 @@ static unsigned int xive_irq_startup(struct irq_data *d)
> return 0;
> }
>
> +static void xive_remove_from_queue(unsigned int hw_irq, int cpu)
> +{
> + struct xive_cpu *xc;
> + struct xive_q *q;
> + u32 irq = 0, cur, idx, toggle, prev;
> + u8 prio;
> + int count;
> +
> + xc = per_cpu(xive_cpu, cpu);
> +
> + /*
> + * Only one queue is really in use, but let's try stay generic, and
> + * check all of them
> + */
nevertheless, we could use some helper routines to manipulate
the xive queues. This is beginning to be cryptic.
> + for (prio = 0; prio < XIVE_MAX_QUEUES; prio++) {
> + q = &xc->queue[prio];
> + if (!q->qpage)
> + continue;
> +
> + /*
> + * We can't read a reliable index and toggle without
> + * locking, as another cpu can process an interrupt
> + * and read the queue at any given time. So use the
> + * toggle from the previous index, which should be ok
> + * as long as the queue doesn't overflow.
> + */
> + idx = q->idx;
> + prev = (idx - 1) & q->msk;
> + cur = be32_to_cpup(q->qpage + prev);
> + toggle = (cur >> 31) ^ 1;
> + count = 0;
> + do {
> + count++;
> + cur = be32_to_cpup(q->qpage + idx);
> + if ((cur >> 31) == toggle)
> + irq = 0;
> + else
> + irq = cur & 0x7fffffff;
> +
> + if (irq == hw_irq) {
> + cur &= 1 << 31;
> + cur |= XIVE_BAD_IRQ;
> + *(q->qpage + idx) = cpu_to_be32(cur);
Are we sure that the XIVE controller is not updating the queue at
the same time ?
Thanks,
C.
> + }
> +
> + idx = (idx + 1) & q->msk;
> + if (idx == 0)
> + toggle ^= 1;
> + } while (irq && (count < q->msk));
> + }
> +}
> +
> static void xive_irq_shutdown(struct irq_data *d)
> {
> struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> @@ -624,6 +682,9 @@ static void xive_irq_shutdown(struct irq_data *d)
> get_hard_smp_processor_id(xd->target),
> 0xff, XIVE_BAD_IRQ);
>
> + /* configure_irq() syncs the queue */
> + xive_remove_from_queue(hw_irq, xd->target);
> +
> xive_dec_target_count(xd->target);
> xd->target = XIVE_INVALID_TARGET;
> }
>
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox