Fam10+ extended config space enabling

Fam10+ extended config space enabling

[Jan Beulich]

Robert,

starting with 2.6.27 Linux is doing this, using MSR accesses (these
days in CPU hotplug notification callbacks). Is there a reason this
cannot be done through PCI config space writes (as the register is
aliased), and then only once at boot time (perhaps utilizing some
of the code in arch/x86/kernel/k8.c)?

I'm particularly asking from the Xen perspective, where the MSR
based mechanism won't reliably work (since there's no guarantee
all of Dom0's virtual CPUs would, at initialization time, actually
cover all physical CPUs that Dom0 may ever run on).

Thanks, Jan

Re: Fam10+ extended config space enabling

[Keir Fraser]

On 06/01/2011 10:15, "Jan Beulich" <JBeulich@novell.com> wrote:

> Robert,
> 
> starting with 2.6.27 Linux is doing this, using MSR accesses (these
> days in CPU hotplug notification callbacks). Is there a reason this
> cannot be done through PCI config space writes (as the register is
> aliased), and then only once at boot time (perhaps utilizing some
> of the code in arch/x86/kernel/k8.c)?
> 
> I'm particularly asking from the Xen perspective, where the MSR
> based mechanism won't reliably work (since there's no guarantee
> all of Dom0's virtual CPUs would, at initialization time, actually
> cover all physical CPUs that Dom0 may ever run on).

Just thinking, could we simply do this in Xen, either unconditionally, or as
soon as we detect dom0 doing it for at least one CPU?

 -- Keir

> Thanks, Jan
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Fam10+ extended config space enabling

[Jan Beulich]

>>> On 06.01.11 at 11:38, Keir Fraser <keir@xen.org> wrote:
> On 06/01/2011 10:15, "Jan Beulich" <JBeulich@novell.com> wrote:
> 
>> Robert,
>> 
>> starting with 2.6.27 Linux is doing this, using MSR accesses (these
>> days in CPU hotplug notification callbacks). Is there a reason this
>> cannot be done through PCI config space writes (as the register is
>> aliased), and then only once at boot time (perhaps utilizing some
>> of the code in arch/x86/kernel/k8.c)?
>> 
>> I'm particularly asking from the Xen perspective, where the MSR
>> based mechanism won't reliably work (since there's no guarantee
>> all of Dom0's virtual CPUs would, at initialization time, actually
>> cover all physical CPUs that Dom0 may ever run on).
> 
> Just thinking, could we simply do this in Xen, either unconditionally, or as
> soon as we detect dom0 doing it for at least one CPU?

We certainly could (albeit not unconditional I'd say; this was also
my first thought, with the alternative of adding a specific
hypercall), but it would mean adding special code for something
that can be done in Dom0 (and there even in a way that's suitable
for both native and Xen) if using the PCI config space method for
this is possible.

Jan

Re: Fam10+ extended config space enabling

[Robert Richter]

On 06.01.11 05:38:48, Keir Fraser wrote:
> On 06/01/2011 10:15, "Jan Beulich" <JBeulich@novell.com> wrote:

> > starting with 2.6.27 Linux is doing this, using MSR accesses (these
> > days in CPU hotplug notification callbacks). Is there a reason this
> > cannot be done through PCI config space writes (as the register is
> > aliased), and then only once at boot time (perhaps utilizing some
> > of the code in arch/x86/kernel/k8.c)?

>From reading the documentation this should work too. The msr controls
a Northbridge function and thus is per-node. NB function msrs are
shared between all cores on the node and there is one instance per
node.

IIRC there were several reasons to choose msr access. During early cpu
bringup there are only some basic early pci access functions
available, requiring a little more complex code to scan NB devices. I
think in between there is code available that could be shared. Second,
if we implement the setup using pci access, we must adopt the code
with every new cpu family with new nb devices to support pci ecs with
new cpus. So, new cpus couldn't use pci ecs with an old kernel then.

But the msr 0xC001001F is not architectural. Actually this means the
assumption in the code is wrong, that all cpus from fam10h have IO
ECS. There might be future cpus not supporting it. But in practice all
current cpu families from family 10h support the msr (10h, 11h, 12h,
14h, 15h).

> > I'm particularly asking from the Xen perspective, where the MSR
> > based mechanism won't reliably work (since there's no guarantee
> > all of Dom0's virtual CPUs would, at initialization time, actually
> > cover all physical CPUs that Dom0 may ever run on).
> 
> Just thinking, could we simply do this in Xen, either unconditionally, or as
> soon as we detect dom0 doing it for at least one CPU?

I don't know if xen is capable to emulate per-node msrs, presumable
not. I don't have an overview which other per-node msrs might be worth
to be supported by xen. If so, it might be better to implement support
for per-node msrs.

But I am fine with a setup with pci access for some nb devices, but
would rather leave the setup via msrs as a fallback.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: Fam10+ extended config space enabling

[Jan Beulich]

>>> On 06.01.11 at 12:47, Robert Richter <robert.richter@amd.com> wrote:
> I don't know if xen is capable to emulate per-node msrs, presumable
> not. I don't have an overview which other per-node msrs might be worth
> to be supported by xen. If so, it might be better to implement support
> for per-node msrs.

The question isn't about emulating, but about letting Dom0 control
the whole platform (if e.g. Dom0 is booted with just 2 [virtual] CPUs
on a large system, it's clear that it'll be very unlikely that it manages
to enabled ECS on all nodes when it issues the MSR writes on just
the two physical CPUs underlying two virtual ones happen to run
on).

> But I am fine with a setup with pci access for some nb devices, but
> would rather leave the setup via msrs as a fallback.

Okay, I'll craft something long those lines.

Jan

[PATCH] x86, AMD, PCI: Add AMD northbridge PCI device id for CPU families 12h and 14h

[Robert Richter]

On 06.01.11 07:36:13, Jan Beulich wrote:
> >>> On 06.01.11 at 12:47, Robert Richter <robert.richter@amd.com> wrote:

> > But I am fine with a setup with pci access for some nb devices, but
> > would rather leave the setup via msrs as a fallback.
> 
> Okay, I'll craft something long those lines.

Jan,

the patch below implements the PCI nb device id for AMD cpu families
12h and 14h.

Ingo,

please apply to tip. There are some future use cases that use this PCI
device and we would like to clarify its naming.

Thanks,

-Robert

--

>From f791e16fbbf30d1ab6c66ebb8daa77a6c86bda0b Mon Sep 17 00:00:00 2001
From: Robert Richter <robert.richter@amd.com>
Date: Thu, 6 Jan 2011 15:01:11 +0100
Subject: [PATCH] x86, AMD, PCI: Add AMD northbridge PCI device id for CPU families 12h and 14h

This patch adds the PCI northbridge device id for AMD CPU families 12h
and 14h. Both families have implemented the same PCI northbridge
device.

Signed-off-by: Robert Richter <robert.richter@amd.com>
---
 include/linux/pci_ids.h |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
index cb845c1..dd7d4e2 100644
--- a/include/linux/pci_ids.h
+++ b/include/linux/pci_ids.h
@@ -518,6 +518,7 @@
 #define PCI_DEVICE_ID_AMD_11H_NB_MISC	0x1303
 #define PCI_DEVICE_ID_AMD_11H_NB_LINK	0x1304
 #define PCI_DEVICE_ID_AMD_15H_NB_MISC	0x1603
+#define PCI_DEVICE_ID_AMD_CNB17H_F3	0x1703
 #define PCI_DEVICE_ID_AMD_LANCE		0x2000
 #define PCI_DEVICE_ID_AMD_LANCE_HOME	0x2001
 #define PCI_DEVICE_ID_AMD_SCSI		0x2020
-- 
1.7.3.4




-- 
Advanced Micro Devices, Inc.
Operating System Research Center

[tip:x86/urgent] x86, AMD, PCI: Add AMD northbridge PCI device id for CPU families 12h and 14h

[tip-bot for Robert Richter]

Commit-ID:  ca86828ccd3128513f6d4e200b437deac95408db
Gitweb:     http://git.kernel.org/tip/ca86828ccd3128513f6d4e200b437deac95408db
Author:     Robert Richter <robert.richter@amd.com>
AuthorDate: Thu, 6 Jan 2011 17:51:07 +0100
Committer:  Ingo Molnar <mingo@elte.hu>
CommitDate: Fri, 7 Jan 2011 14:00:14 +0100

x86, AMD, PCI: Add AMD northbridge PCI device id for CPU families 12h and 14h

This patch adds the PCI northbridge device id for AMD CPU
families 12h and 14h. Both families have implemented the same
PCI northbridge device.

There are some future use cases that use this PCI device and
we would like to clarify its naming.

Signed-off-by: Robert Richter <robert.richter@amd.com>
Cc: xen-devel@lists.xensource.com <xen-devel@lists.xensource.com>
Cc: Keir Fraser <keir@xen.org>
Cc: Jan Beulich <JBeulich@novell.com>
LKML-Reference: <20110106165107.GL4739@erda.amd.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
---
 include/linux/pci_ids.h |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
index cb845c1..dd7d4e2 100644
--- a/include/linux/pci_ids.h
+++ b/include/linux/pci_ids.h
@@ -518,6 +518,7 @@
 #define PCI_DEVICE_ID_AMD_11H_NB_MISC	0x1303
 #define PCI_DEVICE_ID_AMD_11H_NB_LINK	0x1304
 #define PCI_DEVICE_ID_AMD_15H_NB_MISC	0x1603
+#define PCI_DEVICE_ID_AMD_CNB17H_F3	0x1703
 #define PCI_DEVICE_ID_AMD_LANCE		0x2000
 #define PCI_DEVICE_ID_AMD_LANCE_HOME	0x2001
 #define PCI_DEVICE_ID_AMD_SCSI		0x2020

[tip:x86/urgent] x86, AMD, PCI: Add AMD northbridge PCI device id for CPU families 12h and 14h

[tip-bot for Robert Richter]

Commit-ID:  ca86828ccd3128513f6d4e200b437deac95408db
Gitweb:     http://git.kernel.org/tip/ca86828ccd3128513f6d4e200b437deac95408db
Author:     Robert Richter <robert.richter@amd.com>
AuthorDate: Thu, 6 Jan 2011 17:51:07 +0100
Committer:  Ingo Molnar <mingo@elte.hu>
CommitDate: Fri, 7 Jan 2011 14:00:14 +0100

x86, AMD, PCI: Add AMD northbridge PCI device id for CPU families 12h and 14h

This patch adds the PCI northbridge device id for AMD CPU
families 12h and 14h. Both families have implemented the same
PCI northbridge device.

There are some future use cases that use this PCI device and
we would like to clarify its naming.

Signed-off-by: Robert Richter <robert.richter@amd.com>
Cc: xen-devel@lists.xensource.com <xen-devel@lists.xensource.com>
Cc: Keir Fraser <keir@xen.org>
Cc: Jan Beulich <JBeulich@novell.com>
LKML-Reference: <20110106165107.GL4739@erda.amd.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
---
 include/linux/pci_ids.h |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
index cb845c1..dd7d4e2 100644
--- a/include/linux/pci_ids.h
+++ b/include/linux/pci_ids.h
@@ -518,6 +518,7 @@
 #define PCI_DEVICE_ID_AMD_11H_NB_MISC	0x1303
 #define PCI_DEVICE_ID_AMD_11H_NB_LINK	0x1304
 #define PCI_DEVICE_ID_AMD_15H_NB_MISC	0x1603
+#define PCI_DEVICE_ID_AMD_CNB17H_F3	0x1703
 #define PCI_DEVICE_ID_AMD_LANCE		0x2000
 #define PCI_DEVICE_ID_AMD_LANCE_HOME	0x2001
 #define PCI_DEVICE_ID_AMD_SCSI		0x2020