* [PATCH 4/4] Add MPC8349E MDS device tree source file to arch/powerpc/boot/dts
From: Kim Phillips @ 2006-08-29 23:13 UTC (permalink / raw)
To: linuxppc-dev
Add MPC8349E MDS device tree source file to arch/powerpc/boot/dts
Signed-off-by: Kim Phillips <kim.phillips@freescale.com>
---
Amended version of "[PATCH 4/4] Add MPC8349E MDS device tree source file to arch/powerpc/boot/dts": PCI senses are now correctly corrected to 8 (level low).
arch/powerpc/boot/dts/mpc8349emds.dts | 328 +++++++++++++++++++++++++++++++++
1 files changed, 328 insertions(+), 0 deletions(-)
diff --git a/arch/powerpc/boot/dts/mpc8349emds.dts b/arch/powerpc/boot/dts/mpc8349emds.dts
new file mode 100644
index 0000000..12f5dbf
--- /dev/null
+++ b/arch/powerpc/boot/dts/mpc8349emds.dts
@@ -0,0 +1,328 @@
+/*
+ * MPC8349E MDS Device Tree Source
+ *
+ * Copyright 2005, 2006 Freescale Semiconductor Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; either version 2 of the License, or (at your
+ * option) any later version.
+ */
+
+/ {
+ model = "MPC8349EMDS";
+ compatible = "MPC834xMDS";
+ #address-cells = <1>;
+ #size-cells = <1>;
+
+ cpus {
+ #cpus = <1>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ PowerPC,8349@0 {
+ device_type = "cpu";
+ reg = <0>;
+ d-cache-line-size = <20>; // 32 bytes
+ i-cache-line-size = <20>; // 32 bytes
+ d-cache-size = <8000>; // L1, 32K
+ i-cache-size = <8000>; // L1, 32K
+ timebase-frequency = <0>; // from bootloader
+ bus-frequency = <0>; // from bootloader
+ clock-frequency = <0>; // from bootloader
+ 32-bit;
+ };
+ };
+
+ memory {
+ device_type = "memory";
+ reg = <00000000 10000000>; // 256MB at 0
+ };
+
+ soc8349@e0000000 {
+ #address-cells = <1>;
+ #size-cells = <1>;
+ #interrupt-cells = <2>;
+ device_type = "soc";
+ ranges = <0 e0000000 00100000>;
+ reg = <e0000000 00000200>;
+ bus-frequency = <0>;
+
+ wdt@200 {
+ device_type = "watchdog";
+ compatible = "mpc83xx_wdt";
+ reg = <200 100>;
+ };
+
+ i2c@3000 {
+ device_type = "i2c";
+ compatible = "fsl-i2c";
+ reg = <3000 100>;
+ interrupts = <e 8>;
+ interrupt-parent = <700>;
+ dfsrr;
+ };
+
+ i2c@3100 {
+ device_type = "i2c";
+ compatible = "fsl-i2c";
+ reg = <3100 100>;
+ interrupts = <f 8>;
+ interrupt-parent = <700>;
+ dfsrr;
+ };
+
+ spi@7000 {
+ device_type = "spi";
+ compatible = "mpc83xx_spi";
+ reg = <7000 1000>;
+ interrupts = <10 8>;
+ interrupt-parent = <700>;
+ mode = <0>;
+ };
+
+ /* phy type (ULPI or SERIAL) are only types supportted for MPH */
+ /* port = 0 or 1 */
+ usb@22000 {
+ device_type = "usb";
+ compatible = "fsl-usb2-mph";
+ reg = <22000 1000>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ interrupt-parent = <700>;
+ interrupts = <27 2>;
+ phy_type = "ulpi";
+ port1;
+ };
+ /* phy type (ULPI, UTMI, UTMI_WIDE, SERIAL) */
+ usb@23000 {
+ device_type = "usb";
+ compatible = "fsl-usb2-dr";
+ reg = <23000 1000>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ interrupt-parent = <700>;
+ interrupts = <26 2>;
+ phy_type = "ulpi";
+ };
+
+ mdio@24520 {
+ device_type = "mdio";
+ compatible = "gianfar";
+ reg = <24520 20>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ linux,phandle = <24520>;
+ ethernet-phy@0 {
+ linux,phandle = <2452000>;
+ interrupt-parent = <700>;
+ interrupts = <11 2>;
+ reg = <0>;
+ device_type = "ethernet-phy";
+ };
+ ethernet-phy@1 {
+ linux,phandle = <2452001>;
+ interrupt-parent = <700>;
+ interrupts = <12 2>;
+ reg = <1>;
+ device_type = "ethernet-phy";
+ };
+ };
+
+ ethernet@24000 {
+ device_type = "network";
+ model = "TSEC";
+ compatible = "gianfar";
+ reg = <24000 1000>;
+ address = [ 00 00 00 00 00 00 ];
+ local-mac-address = [ 00 00 00 00 00 00 ];
+ interrupts = <20 8 21 8 22 8>;
+ interrupt-parent = <700>;
+ phy-handle = <2452000>;
+ };
+
+ ethernet@25000 {
+ #address-cells = <1>;
+ #size-cells = <0>;
+ device_type = "network";
+ model = "TSEC";
+ compatible = "gianfar";
+ reg = <25000 1000>;
+ address = [ 00 00 00 00 00 00 ];
+ local-mac-address = [ 00 00 00 00 00 00 ];
+ interrupts = <23 8 24 8 25 8>;
+ interrupt-parent = <700>;
+ phy-handle = <2452001>;
+ };
+
+ serial@4500 {
+ device_type = "serial";
+ compatible = "ns16550";
+ reg = <4500 100>;
+ clock-frequency = <0>;
+ interrupts = <9 8>;
+ interrupt-parent = <700>;
+ };
+
+ serial@4600 {
+ device_type = "serial";
+ compatible = "ns16550";
+ reg = <4600 100>;
+ clock-frequency = <0>;
+ interrupts = <a 8>;
+ interrupt-parent = <700>;
+ };
+
+ pci@8500 {
+ interrupt-map-mask = <f800 0 0 7>;
+ interrupt-map = <
+
+ /* IDSEL 0x11 */
+ 8800 0 0 1 700 14 8
+ 8800 0 0 2 700 15 8
+ 8800 0 0 3 700 16 8
+ 8800 0 0 4 700 17 8
+
+ /* IDSEL 0x12 */
+ 9000 0 0 1 700 16 8
+ 9000 0 0 2 700 17 8
+ 9000 0 0 3 700 14 8
+ 9000 0 0 4 700 15 8
+
+ /* IDSEL 0x13 */
+ 9800 0 0 1 700 17 8
+ 9800 0 0 2 700 14 8
+ 9800 0 0 3 700 15 8
+ 9800 0 0 4 700 16 8
+
+ /* IDSEL 0x15 */
+ a800 0 0 1 700 14 8
+ a800 0 0 2 700 15 8
+ a800 0 0 3 700 16 8
+ a800 0 0 4 700 17 8
+
+ /* IDSEL 0x16 */
+ b000 0 0 1 700 17 8
+ b000 0 0 2 700 14 8
+ b000 0 0 3 700 15 8
+ b000 0 0 4 700 16 8
+
+ /* IDSEL 0x17 */
+ b800 0 0 1 700 16 8
+ b800 0 0 2 700 17 8
+ b800 0 0 3 700 14 8
+ b800 0 0 4 700 15 8
+
+ /* IDSEL 0x18 */
+ b000 0 0 1 700 15 8
+ b000 0 0 2 700 16 8
+ b000 0 0 3 700 17 8
+ b000 0 0 4 700 14 8>;
+ interrupt-parent = <700>;
+ interrupts = <42 8>;
+ bus-range = <0 0>;
+ ranges = <02000000 0 a0000000 a0000000 0 10000000
+ 42000000 0 80000000 80000000 0 10000000
+ 01000000 0 00000000 e2000000 0 00100000>;
+ clock-frequency = <3f940aa>;
+ #interrupt-cells = <1>;
+ #size-cells = <2>;
+ #address-cells = <3>;
+ reg = <8500 100>;
+ compatible = "83xx";
+ device_type = "pci";
+ };
+
+ pci@8600 {
+ interrupt-map-mask = <f800 0 0 7>;
+ interrupt-map = <
+
+ /* IDSEL 0x11 */
+ 8800 0 0 1 700 14 8
+ 8800 0 0 2 700 15 8
+ 8800 0 0 3 700 16 8
+ 8800 0 0 4 700 17 8
+
+ /* IDSEL 0x12 */
+ 9000 0 0 1 700 16 8
+ 9000 0 0 2 700 17 8
+ 9000 0 0 3 700 14 8
+ 9000 0 0 4 700 15 8
+
+ /* IDSEL 0x13 */
+ 9800 0 0 1 700 17 8
+ 9800 0 0 2 700 14 8
+ 9800 0 0 3 700 15 8
+ 9800 0 0 4 700 16 8
+
+ /* IDSEL 0x15 */
+ a800 0 0 1 700 14 8
+ a800 0 0 2 700 15 8
+ a800 0 0 3 700 16 8
+ a800 0 0 4 700 17 8
+
+ /* IDSEL 0x16 */
+ b000 0 0 1 700 17 8
+ b000 0 0 2 700 14 8
+ b000 0 0 3 700 15 8
+ b000 0 0 4 700 16 8
+
+ /* IDSEL 0x17 */
+ b800 0 0 1 700 16 8
+ b800 0 0 2 700 17 8
+ b800 0 0 3 700 14 8
+ b800 0 0 4 700 15 8
+
+ /* IDSEL 0x18 */
+ b000 0 0 1 700 15 8
+ b000 0 0 2 700 16 8
+ b000 0 0 3 700 17 8
+ b000 0 0 4 700 14 8>;
+ interrupt-parent = <700>;
+ interrupts = <42 8>;
+ bus-range = <0 0>;
+ ranges = <02000000 0 b0000000 b0000000 0 10000000
+ 42000000 0 90000000 90000000 0 10000000
+ 01000000 0 00000000 e2100000 0 00100000>;
+ clock-frequency = <3f940aa>;
+ #interrupt-cells = <1>;
+ #size-cells = <2>;
+ #address-cells = <3>;
+ reg = <8600 100>;
+ compatible = "83xx";
+ device_type = "pci";
+ };
+
+ /* May need to remove if on a part without crypto engine */
+ crypto@30000 {
+ device_type = "crypto";
+ model = "SEC2";
+ compatible = "talitos";
+ reg = <30000 10000>;
+ interrupts = <b 8>;
+ interrupt-parent = <700>;
+ num-channels = <4>;
+ channel-fifo-len = <18>;
+ exec-units-mask = <0000007e>;
+ /* desc mask is for rev2.0,
+ * we need runtime fixup for >2.0 */
+ descriptor-types-mask = <01010ebf>;
+ };
+
+ /* IPIC
+ * interrupts cell = <intr #, sense>
+ * sense values match linux IORESOURCE_IRQ_* defines:
+ * sense == 8: Level, low assertion
+ * sense == 2: Edge, high-to-low change
+ */
+ pic@700 {
+ linux,phandle = <700>;
+ interrupt-controller;
+ #address-cells = <0>;
+ #interrupt-cells = <2>;
+ reg = <700 100>;
+ built-in;
+ device_type = "ipic";
+ };
+ };
+};
--
2006_06_07.01.gittree_pull-dirty
^ permalink raw reply related
* libnuma interleaving oddness
From: Nishanth Aravamudan @ 2006-08-29 23:15 UTC (permalink / raw)
To: clameter, ak; +Cc: linux-mm, lnxninja, linuxppc-dev
[Sorry for the double-post, correcting Christoph's address]
Hi,
While trying to add NUMA-awareness to libhugetlbfs' morecore
functionality (hugepage-backed malloc), I ran into an issue on a
ppc64-box with 8 memory nodes, running SLES10. I am using two functions
from libnuma: numa_available() and numa_interleave_memory(). When I ask
numa_interleave_memory() to interleave over all nodes (numa_all_nodes is
the nodemask from libnuma), it exhausts node 0, then moves to node 1,
then node 2, etc, until the allocations are satisfied. If I custom
generate a nodemask, such that bits 1 through 7 are set, but bit 0 is
not, then I get proper interleaving, where the first hugepage is on node
1, the second is on node 2, etc. Similarly, if I set bits 0 through 6 in
a custom nodemask, interleaving works across the requested 7 nodes. But
it has yet to work across all 8.
I don't know if this is a libnuma bug (I extracted out the code from
libnuma, it looked sane; and even reimplemented it in libhugetlbfs for
testing purposes, but got the same results) or a NUMA kernel bug (mbind
is some hairy code...) or a ppc64 bug or maybe not a bug at all.
Regardless, I'm getting somewhat inconsistent behavior. I can provide
more debugging output, or whatever is requested, but I wasn't sure what
to include. I'm hoping someone has heard of or seen something similar?
The test application I'm using makes some mallopt calls then justs
mallocs large chunks in a loop (4096 * 100 bytes). libhugetlbfs is
LD_PRELOAD'd so that we can override malloc.
Thanks,
Nish
--
Nishanth Aravamudan <nacc@us.ibm.com>
IBM Linux Technology Center
^ permalink raw reply
* Re: undefined reference to pci_io_base
From: Linas Vepstas @ 2006-08-29 23:20 UTC (permalink / raw)
To: Paul Mackerras; +Cc: linuxppc-dev
In-Reply-To: <17652.49904.419167.448821@cargo.ozlabs.ibm.com>
On Wed, Aug 30, 2006 at 08:42:56AM +1000, Paul Mackerras wrote:
> Geoff Levand writes:
>
> > I'm trying to understand the intended design of the inb()
> > and outb() macros in asm-powerpc/io.h. They are causing
> > me some grief when I set CONFIG_PCI=n:
> >
> > drivers/built-in.o: undefined reference to `pci_io_base'
> >
> > This is coming from drivers/char/mem.c, which is always
> > built in, and is referencing pci_io_base through inb()
> > and outb().
>
> Hmmm. inb and outb are designed for accessing PCI I/O space. I guess
> we could turn them into BUG() when CONFIG_PCI=n.
I just looked at drivers/char/mem.c and the code in question
is surrounded by
#if defined(CONFIG_ISA) || !defined(__mc68000__)
which seems just plain wrong.
It should probably be changed to
#if (defined(CONFIG_ISA) || defined(CONFIG_PCI)) && !defined(__mc68000__)
This code surrounds fileops to enale fileio on /dev/port
which maps to inb/outb.
Do we want to enable /dev/port for pci space ??
--linas
^ permalink raw reply
* Re: Help for building linux 2.6.15
From: Wolfgang Denk @ 2006-08-29 23:58 UTC (permalink / raw)
To: wei.li4; +Cc: linuxppc-embedded
In-Reply-To: <20060829171138.y9uuwvfi4gs48ws4@webmail.mcgill.ca>
In message <20060829171138.y9uuwvfi4gs48ws4@webmail.mcgill.ca> you wrote:
>
> I am working on ELDK 4.0 package with my evaluation board: A&M MPC875,
> after configuration for my board, and do 'make zImage', I got the
> following errors:
I cannot find any board support for this board in the 2.6 kernel
trees. Who ported Linux to this hardware? Please go and ask these
guys.
> CC arch/ppc/syslib/m8xx_setup.o
> arch/ppc/syslib/m8xx_setup.c:56: error: 'bd_t' undeclared here (not in
> a function)
Broken code, and/or a broken configuration, if you ask me.
> Can anyone help me? Thanks.
Which board configration are you using? And where is the board support
coming from?
Best regards,
Wolfgang Denk
--
Software Engineering: Embedded and Realtime Systems, Embedded Linux
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Change is the essential process of all existence.
-- Spock, "Let That Be Your Last Battlefield",
stardate 5730.2
^ permalink raw reply
* Re: undefined reference to pci_io_base
From: Geoff Levand @ 2006-08-29 23:57 UTC (permalink / raw)
To: Linas Vepstas; +Cc: linuxppc-dev, Paul Mackerras
In-Reply-To: <20060829232026.GC27372@austin.ibm.com>
Linas Vepstas wrote:
> On Wed, Aug 30, 2006 at 08:42:56AM +1000, Paul Mackerras wrote:
>> Geoff Levand writes:
>>
>> > I'm trying to understand the intended design of the inb()
>> > and outb() macros in asm-powerpc/io.h. They are causing
>> > me some grief when I set CONFIG_PCI=n:
>> >
>> > drivers/built-in.o: undefined reference to `pci_io_base'
>> >
>> > This is coming from drivers/char/mem.c, which is always
>> > built in, and is referencing pci_io_base through inb()
>> > and outb().
>>
>> Hmmm. inb and outb are designed for accessing PCI I/O space. I guess
>> we could turn them into BUG() when CONFIG_PCI=n.
>
> I just looked at drivers/char/mem.c and the code in question
> is surrounded by
>
> #if defined(CONFIG_ISA) || !defined(__mc68000__)
>
> which seems just plain wrong.
>
> It should probably be changed to
>
> #if (defined(CONFIG_ISA) || defined(CONFIG_PCI)) && !defined(__mc68000__)
>
> This code surrounds fileops to enale fileio on /dev/port
> which maps to inb/outb.
>
> Do we want to enable /dev/port for pci space ??
I found the same, but was thinking
#if defined(CONFIG_ISA) && !defined(__mc68000__)
So it is only built when CONFIG_ISA=y. Is there any
reason to always have it whit PCI?
Paul, can I send you that fix, or should I send it to
someone else?
-Geoff
^ permalink raw reply
* Re: undefined reference to pci_io_base
From: Paul Mackerras @ 2006-08-30 0:13 UTC (permalink / raw)
To: Geoff Levand; +Cc: linuxppc-dev
In-Reply-To: <44F4D458.7030707@am.sony.com>
Geoff Levand writes:
> I found the same, but was thinking
>
> #if defined(CONFIG_ISA) && !defined(__mc68000__)
>
> So it is only built when CONFIG_ISA=y. Is there any
> reason to always have it whit PCI?
It's occasionally useful with PCI && !ISA. It's pretty useless with
!PCI && !ISA.
> Paul, can I send you that fix, or should I send it to
> someone else?
Send it to Andrew Morton.
Paul.
^ permalink raw reply
* Re: libnuma interleaving oddness
From: Nishanth Aravamudan @ 2006-08-30 0:21 UTC (permalink / raw)
To: Christoph Lameter; +Cc: linux-mm, lnxninja, ak, linuxppc-dev
In-Reply-To: <Pine.LNX.4.64.0608291655160.22397@schroedinger.engr.sgi.com>
On 29.08.2006 [16:57:35 -0700], Christoph Lameter wrote:
> On Tue, 29 Aug 2006, Nishanth Aravamudan wrote:
>
> > I don't know if this is a libnuma bug (I extracted out the code from
> > libnuma, it looked sane; and even reimplemented it in libhugetlbfs
> > for testing purposes, but got the same results) or a NUMA kernel bug
> > (mbind is some hairy code...) or a ppc64 bug or maybe not a bug at
> > all. Regardless, I'm getting somewhat inconsistent behavior. I can
> > provide more debugging output, or whatever is requested, but I
> > wasn't sure what to include. I'm hoping someone has heard of or seen
> > something similar?
>
> Are you setting the tasks allocation policy before the allocation or
> do you set a vma based policy? The vma based policies will only work
> for anonymous pages.
The order is (with necessary params filled in):
p = mmap( , newsize, RW, PRIVATE, unlinked_hugetlbfs_heap_fd, );
numa_interleave_memory(p, newsize);
mlock(p, newsize); /* causes all the hugepages to be faulted in */
munlock(p,newsize);
>From what I gathered from the numa manpages, the interleave policy
should take effect on the mlock, as that is "fault-time" in this
context. We're forcing the fault, that is.
Does that answer your question? Sorry if I'm unclear, I'm a bit of a
newbie to the VM.
Thanks,
Nish
--
Nishanth Aravamudan <nacc@us.ibm.com>
IBM Linux Technology Center
^ permalink raw reply
* Re: libnuma interleaving oddness
From: Christoph Lameter @ 2006-08-29 23:57 UTC (permalink / raw)
To: Nishanth Aravamudan; +Cc: linux-mm, lnxninja, ak, linuxppc-dev
In-Reply-To: <20060829231545.GY5195@us.ibm.com>
On Tue, 29 Aug 2006, Nishanth Aravamudan wrote:
> I don't know if this is a libnuma bug (I extracted out the code from
> libnuma, it looked sane; and even reimplemented it in libhugetlbfs for
> testing purposes, but got the same results) or a NUMA kernel bug (mbind
> is some hairy code...) or a ppc64 bug or maybe not a bug at all.
> Regardless, I'm getting somewhat inconsistent behavior. I can provide
> more debugging output, or whatever is requested, but I wasn't sure what
> to include. I'm hoping someone has heard of or seen something similar?
Are you setting the tasks allocation policy before the allocation or do
you set a vma based policy? The vma based policies will only work for
anonymous pages.
^ permalink raw reply
* RE: atomic operations in user space
From: Liu Dave-r63238 @ 2006-08-30 2:17 UTC (permalink / raw)
To: Li Yang-r58472, linuxppc-embedded, linuxppc-dev
[snip]
> I surely know all the theories you mentioned clearly. But=20
> please do look at the case I gave. Correct me if I missed=20
> anything. Thanks
>=20
> All the lwarx and stwcx operate on the same address.
>=20
> > Task A Task B
> > lwarx =09
> // Get RESERVATION
> > ......
> > lwarx
> > stwcx
>=20
> // RESERVATION cleared
> >
> > .....
> > .....
> > lwarx
>=20
> // Get RESERVATION again
> > stwcx =09
>=20
> //Note here: RESERVATION is valid, address is the same.
> So the result is commited, no retry for task A
>=20
> > .....
> > stwcx
> //RESERVATION is cleared, retry atomic op for task B
>=20
> Please be noted that reservation is only identified by=20
> reservation bit and address operated on. So different=20
> lwarx's on the same address, may be considered as the same=20
> reservation.
Is this your reason we cannot do atomic operation in user space?
How about the kernel space? You can image it.
The context switching as above also happen in kernel space,
Why we can do atomic operation in kernel space, not do in user space?
You are assuming the context switching cause the reservation broken.
but we can do atomic operation in kernel space. The context switching
really is the execption of processor, If we can clear the wrong
RESERVATION
before exception return, I think we can solve this problem. We can dummy
stwcx. before exception return or the processor automaticly clear the
reservation in exception.=20
Are you missing these important things?
-DAve
^ permalink raw reply
* Re: libnuma interleaving oddness
From: Nishanth Aravamudan @ 2006-08-30 2:26 UTC (permalink / raw)
To: Christoph Lameter; +Cc: linux-mm, lnxninja, ak, linuxppc-dev
In-Reply-To: <20060830002110.GZ5195@us.ibm.com>
On 29.08.2006 [17:21:10 -0700], Nishanth Aravamudan wrote:
> On 29.08.2006 [16:57:35 -0700], Christoph Lameter wrote:
> > On Tue, 29 Aug 2006, Nishanth Aravamudan wrote:
> >
> > > I don't know if this is a libnuma bug (I extracted out the code from
> > > libnuma, it looked sane; and even reimplemented it in libhugetlbfs
> > > for testing purposes, but got the same results) or a NUMA kernel bug
> > > (mbind is some hairy code...) or a ppc64 bug or maybe not a bug at
> > > all. Regardless, I'm getting somewhat inconsistent behavior. I can
> > > provide more debugging output, or whatever is requested, but I
> > > wasn't sure what to include. I'm hoping someone has heard of or seen
> > > something similar?
> >
> > Are you setting the tasks allocation policy before the allocation or
> > do you set a vma based policy? The vma based policies will only work
> > for anonymous pages.
>
> The order is (with necessary params filled in):
>
> p = mmap( , newsize, RW, PRIVATE, unlinked_hugetlbfs_heap_fd, );
>
> numa_interleave_memory(p, newsize);
>
> mlock(p, newsize); /* causes all the hugepages to be faulted in */
>
> munlock(p,newsize);
>
> From what I gathered from the numa manpages, the interleave policy
> should take effect on the mlock, as that is "fault-time" in this
> context. We're forcing the fault, that is.
For some more data, I did some manipulations of libhugetlbfs and came up
with the following:
If I use the default hugepage-aligned hugepage-backed malloc
replacement, I get the following in /proc/pid/numa_maps (excerpt):
20000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.3JbO7R\040(deleted) huge dirty=1 N0=1
21000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.3JbO7R\040(deleted) huge dirty=1 N0=1
...
37000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.3JbO7R\040(deleted) huge dirty=1 N0=1
38000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.3JbO7R\040(deleted) huge dirty=1 N0=1
If I change the nodemask to 1-7, I get:
20000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N1=1
21000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N2=1
22000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N3=1
23000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N4=1
24000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N5=1
25000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N6=1
26000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N7=1
...
35000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N1=1
36000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N2=1
37000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N3=1
38000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N4=1
If I then change our malloc implementation to (unnecessarily) mmap a
size aligned to 4 hugepages, rather aligned to a single hugepage, but
using a nodemask of 0-7, I get:
20000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.PFt0xt\040(deleted) huge dirty=4 N0=1 N1=1 N2=1 N3=1
24000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.PFt0xt\040(deleted) huge dirty=4 N0=1 N1=1 N2=1 N3=1
28000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.PFt0xt\040(deleted) huge dirty=4 N0=1 N1=1 N2=1 N3=1
2c000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.PFt0xt\040(deleted) huge dirty=4 N0=1 N1=1 N2=1 N3=1
30000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.PFt0xt\040(deleted) huge dirty=4 N0=1 N1=1 N2=1 N3=1
34000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.PFt0xt\040(deleted) huge dirty=4 N0=1 N1=1 N2=1 N3=1
38000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.PFt0xt\040(deleted) huge dirty=1 mapped=4 N0=1 N1=1 N2=1 N3=1
It seems rather odd that it's this inconsistent, and that I'm the only
one seeing it as such :)
Thanks,
Nish
--
Nishanth Aravamudan <nacc@us.ibm.com>
IBM Linux Technology Center
^ permalink raw reply
* RE: atomic operations in user space
From: Liu Dave-r63238 @ 2006-08-30 2:27 UTC (permalink / raw)
To: Liu Dave-r63238, Li Yang-r58472, linuxppc-embedded, linuxppc-dev
In-Reply-To: <995B09A8299C2C44B59866F6391D263511BB47@zch01exm21.fsl.freescale.net>
> [snip]
> > I surely know all the theories you mentioned clearly. But=20
> please do=20
> > look at the case I gave. Correct me if I missed anything. Thanks
> >=20
> > All the lwarx and stwcx operate on the same address.
> >=20
> > > Task A Task B
> > > lwarx =09
> > // Get RESERVATION
> > > ......
> > > lwarx
> > > stwcx
> >=20
> > // RESERVATION cleared
> > >
> > > .....
> > > .....
> > > lwarx
> >=20
> > // Get RESERVATION again
> > > stwcx =09
> >=20
> > //Note here: RESERVATION is valid, address is the same.
> > So the result is commited, no retry for task A
> >=20
> > > .....
> > > stwcx
> > //RESERVATION is cleared, retry atomic op for task B
> >=20
> > Please be noted that reservation is only identified by=20
> reservation bit=20
> > and address operated on. So different lwarx's on the same address,=20
> > may be considered as the same reservation.
>=20
> Is this your reason we cannot do atomic operation in user space?
>=20
> How about the kernel space? You can image it.
> The context switching as above also happen in kernel space,=20
> Why we can do atomic operation in kernel space, not do in user space?
>=20
> You are assuming the context switching cause the reservation broken.
> but we can do atomic operation in kernel space. The context=20
> switching really is the execption of processor, If we can=20
> clear the wrong RESERVATION before exception return, I think=20
> we can solve this problem. We can dummy stwcx. before=20
> exception return or the processor automaticly clear the=20
> reservation in exception.=20
>=20
> Are you missing these important things?
>=20
> -DAve
I got it. I noticed that all of execption return in kernel did stwcx.
to clear the wrong reserved bit. See the source code.
.globl ret_from_except_full
ret_from_except_full:
REST_NVGPRS(r1)
/* fall through */
.globl ret_from_except
ret_from_except:
......
restore:
lwz r0,GPR0(r1)
lwz r2,GPR2(r1)
REST_4GPRS(3, r1)
REST_2GPRS(7, r1)
lwz r10,_XER(r1)
lwz r11,_CTR(r1)
mtspr SPRN_XER,r10
mtctr r11
PPC405_ERR77(0,r1)
stwcx. r0,0,r1 /* to clear the reservation */
......
rfi
-DAve
=20
^ permalink raw reply
* RE: RE: atomic operations in user space
From: Li Yang-r58472 @ 2006-08-30 2:28 UTC (permalink / raw)
To: Liu Dave-r63238, linuxppc-embedded, linuxppc-dev
In-Reply-To: <995B09A8299C2C44B59866F6391D263511BB47@zch01exm21.fsl.freescale.net>
> -----Original Message-----
> From: Liu Dave-r63238
> Sent: Wednesday, August 30, 2006 10:17 AM
> To: Li Yang-r58472; linuxppc-embedded@ozlabs.org;
linuxppc-dev@ozlabs.org
> Subject: RE: atomic operations in user space
>=20
> [snip]
> > I surely know all the theories you mentioned clearly. But
> > please do look at the case I gave. Correct me if I missed
> > anything. Thanks
> >
> > All the lwarx and stwcx operate on the same address.
> >
> > > Task A Task B
> > > lwarx
> > // Get RESERVATION
> > > ......
> > > lwarx
> > > stwcx
> >
> > // RESERVATION cleared
> > >
> > > .....
> > > .....
> > > lwarx
> >
> > // Get RESERVATION again
> > > stwcx
> >
> > //Note here: RESERVATION is valid, address is the same.
> > So the result is commited, no retry for task A
> >
> > > .....
> > > stwcx
> > //RESERVATION is cleared, retry atomic op for task B
> >
> > Please be noted that reservation is only identified by
> > reservation bit and address operated on. So different
> > lwarx's on the same address, may be considered as the same
> > reservation.
>=20
> Is this your reason we cannot do atomic operation in user space?
>=20
> How about the kernel space? You can image it.
> The context switching as above also happen in kernel space,
> Why we can do atomic operation in kernel space, not do in user space?
>
There are substantial different between kernel and user control path.
First, interrupt can't be interrupted by user process. Second, context
switch can be explicitly controlled in kernel, but not in user space.
> You are assuming the context switching cause the reservation broken.
> but we can do atomic operation in kernel space. The context switching
> really is the execption of processor, If we can clear the wrong
RESERVATION
> before exception return, I think we can solve this problem. We can
dummy
> stwcx. before exception return or the processor automaticly clear the
> reservation in exception.
I assume stwcx is a costing instruction, and I don't see such code
indeed.
- Leo
^ permalink raw reply
* RE: atomic operations in user space
From: Li Yang-r58472 @ 2006-08-30 2:40 UTC (permalink / raw)
To: Liu Dave-r63238, linuxppc-embedded, linuxppc-dev
In-Reply-To: <995B09A8299C2C44B59866F6391D263511BB4D@zch01exm21.fsl.freescale.net>
> -----Original Message-----
> From: Liu Dave-r63238
> Sent: Wednesday, August 30, 2006 10:27 AM
> To: Liu Dave-r63238; Li Yang-r58472; linuxppc-embedded@ozlabs.org;
> linuxppc-dev@ozlabs.org
> Subject: RE: atomic operations in user space
>=20
> > [snip]
> > > I surely know all the theories you mentioned clearly. But
> > please do
> > > look at the case I gave. Correct me if I missed anything. Thanks
> > >
> > > All the lwarx and stwcx operate on the same address.
> > >
> > > > Task A Task B
> > > > lwarx
> > > // Get RESERVATION
> > > > ......
> > > > lwarx
> > > > stwcx
> > >
> > > // RESERVATION cleared
> > > >
> > > > .....
> > > > .....
> > > > lwarx
> > >
> > > // Get RESERVATION again
> > > > stwcx
> > >
> > > //Note here: RESERVATION is valid, address is the same.
> > > So the result is commited, no retry for task A
> > >
> > > > .....
> > > > stwcx
> > > //RESERVATION is cleared, retry atomic op for task B
> > >
> > > Please be noted that reservation is only identified by
> > reservation bit
> > > and address operated on. So different lwarx's on the same
address,
> > > may be considered as the same reservation.
> >
> > Is this your reason we cannot do atomic operation in user space?
> >
> > How about the kernel space? You can image it.
> > The context switching as above also happen in kernel space,
> > Why we can do atomic operation in kernel space, not do in user
space?
> >
> > You are assuming the context switching cause the reservation broken.
> > but we can do atomic operation in kernel space. The context
> > switching really is the execption of processor, If we can
> > clear the wrong RESERVATION before exception return, I think
> > we can solve this problem. We can dummy stwcx. before
> > exception return or the processor automaticly clear the
> > reservation in exception.
> >
> > Are you missing these important things?
> >
> > -DAve
>=20
> I got it. I noticed that all of execption return in kernel did stwcx.
> to clear the wrong reserved bit. See the source code.
>=20
> .globl ret_from_except_full
> ret_from_except_full:
> REST_NVGPRS(r1)
> /* fall through */
>=20
> .globl ret_from_except
> ret_from_except:
> ......
>=20
> restore:
> lwz r0,GPR0(r1)
> lwz r2,GPR2(r1)
> REST_4GPRS(3, r1)
> REST_2GPRS(7, r1)
>=20
> lwz r10,_XER(r1)
> lwz r11,_CTR(r1)
> mtspr SPRN_XER,r10
> mtctr r11
>=20
> PPC405_ERR77(0,r1)
> stwcx. r0,0,r1 /* to clear the reservation */
Ya, you found the point. There is no problem for me about this
question.
- Leo
^ permalink raw reply
* RE: RE: atomic operations in user space
From: Liu Dave-r63238 @ 2006-08-30 2:42 UTC (permalink / raw)
To: Li Yang-r58472, linuxppc-embedded, linuxppc-dev
In-Reply-To: <4879B0C6C249214CBE7AB04453F84E4D0FC82B@zch01exm20.fsl.freescale.net>
[snip]
> >=20
> > Is this your reason we cannot do atomic operation in user space?
> >=20
> > How about the kernel space? You can image it.
> > The context switching as above also happen in kernel space,=20
> Why we can=20
> > do atomic operation in kernel space, not do in user space?
> >
>=20
> There are substantial different between kernel and user=20
> control path. First, interrupt can't be interrupted by user=20
> process. Second, context switch can be explicitly controlled=20
> in kernel, but not in user space.
I agree this, but from the processor's view, the context switch
is the same to user space and kernel space. The exception
control flow only happen at exception interrupt.=20
What is different you point ?
> > You are assuming the context switching cause the reservation broken.
> > but we can do atomic operation in kernel space. The=20
> context switching=20
> > really is the execption of processor, If we can clear the wrong=20
> > RESERVATION before exception return, I think we can solve this=20
> > problem. We can dummy stwcx. before exception return or the=20
> processor=20
> > automaticly clear the reservation in exception.
>=20
> I assume stwcx is a costing instruction, and I don't see such=20
> code indeed.
^ permalink raw reply
* RE: RE: atomic operations in user space
From: Li Yang-r58472 @ 2006-08-30 2:52 UTC (permalink / raw)
To: Liu Dave-r63238, linuxppc-embedded, linuxppc-dev
In-Reply-To: <995B09A8299C2C44B59866F6391D263511BB58@zch01exm21.fsl.freescale.net>
> -----Original Message-----
> From: Liu Dave-r63238
> Sent: Wednesday, August 30, 2006 10:43 AM
> To: Li Yang-r58472; 'linuxppc-embedded@ozlabs.org';
'linuxppc-dev@ozlabs.org'
> Subject: RE: RE: atomic operations in user space
>=20
> [snip]
> > >
> > > Is this your reason we cannot do atomic operation in user space?
> > >
> > > How about the kernel space? You can image it.
> > > The context switching as above also happen in kernel space,
> > Why we can
> > > do atomic operation in kernel space, not do in user space?
> > >
> >
> > There are substantial different between kernel and user
> > control path. First, interrupt can't be interrupted by user
> > process. Second, context switch can be explicitly controlled
> > in kernel, but not in user space.
>=20
> I agree this, but from the processor's view, the context switch
> is the same to user space and kernel space. The exception
> control flow only happen at exception interrupt.
Exception is special and tiny part of the kernel, which should be
programmed carefully not to break any thing. Anyway, as you found,
clear reservation in exception do solve all the problems.
>=20
> What is different you point ?
>=20
> > > You are assuming the context switching cause the reservation
broken.
> > > but we can do atomic operation in kernel space. The
> > context switching
> > > really is the execption of processor, If we can clear the wrong
> > > RESERVATION before exception return, I think we can solve this
> > > problem. We can dummy stwcx. before exception return or the
> > processor
> > > automaticly clear the reservation in exception.
> >
> > I assume stwcx is a costing instruction, and I don't see such
> > code indeed.
^ permalink raw reply
* RE: RE: atomic operations in user space
From: Liu Dave-r63238 @ 2006-08-30 2:55 UTC (permalink / raw)
To: Li Yang-r58472, linuxppc-embedded, linuxppc-dev
In-Reply-To: <4879B0C6C249214CBE7AB04453F84E4D0FC836@zch01exm20.fsl.freescale.net>
> > [snip]
> > > >
> > > > Is this your reason we cannot do atomic operation in user space?
> > > >
> > > > How about the kernel space? You can image it.
> > > > The context switching as above also happen in kernel space,
> > > Why we can
> > > > do atomic operation in kernel space, not do in user space?
> > > >
> > >
> > > There are substantial different between kernel and user control=20
> > > path. First, interrupt can't be interrupted by user process. =20
> > > Second, context switch can be explicitly controlled in=20
> kernel, but=20
> > > not in user space.
> >=20
> > I agree this, but from the processor's view, the context=20
> switch is the=20
> > same to user space and kernel space. The exception control=20
> flow only=20
> > happen at exception interrupt.
> > What is different you point ?
>=20
> Exception is special and tiny part of the kernel, which=20
> should be programmed carefully not to break any thing. =20
> Anyway, as you found, clear reservation in exception do solve=20
> all the problems.
Can we do atomic operation in user space as kernel space?
> > > > You are assuming the context switching cause the=20
> reservation broken.
> > > > but we can do atomic operation in kernel space. The
> > > context switching
> > > > really is the execption of processor, If we can clear the wrong=20
> > > > RESERVATION before exception return, I think we can solve this=20
> > > > problem. We can dummy stwcx. before exception return or the
> > > processor
> > > > automaticly clear the reservation in exception.
> > >
> > > I assume stwcx is a costing instruction, and I don't see=20
> such code=20
> > > indeed.
^ permalink raw reply
* Re: libnuma interleaving oddness
From: Christoph Lameter @ 2006-08-30 4:26 UTC (permalink / raw)
To: Nishanth Aravamudan; +Cc: linux-mm, lnxninja, ak, linuxppc-dev
In-Reply-To: <20060830022621.GA5195@us.ibm.com>
On Tue, 29 Aug 2006, Nishanth Aravamudan wrote:
> If I use the default hugepage-aligned hugepage-backed malloc
> replacement, I get the following in /proc/pid/numa_maps (excerpt):
>
> 20000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.3JbO7R\040(deleted) huge dirty=1 N0=1
> 21000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.3JbO7R\040(deleted) huge dirty=1 N0=1
> ...
> 37000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.3JbO7R\040(deleted) huge dirty=1 N0=1
> 38000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.3JbO7R\040(deleted) huge dirty=1 N0=1
Is this with nodemask set to [0]?
> If I change the nodemask to 1-7, I get:
>
> 20000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N1=1
> 21000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N2=1
> 22000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N3=1
> 23000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N4=1
> 24000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N5=1
> 25000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N6=1
> 26000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N7=1
> ...
> 35000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N1=1
> 36000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N2=1
> 37000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N3=1
> 38000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N4=1
So interleave has an effect.
Are you using cpusets? Or are you only using memory policies? What is the
default policy of the task you are running?
> If I then change our malloc implementation to (unnecessarily) mmap a
> size aligned to 4 hugepages, rather aligned to a single hugepage, but
> using a nodemask of 0-7, I get:
>
> 20000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.PFt0xt\040(deleted) huge dirty=4 N0=1 N1=1 N2=1 N3=1
> 24000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.PFt0xt\040(deleted) huge dirty=4 N0=1 N1=1 N2=1 N3=1
> 28000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.PFt0xt\040(deleted) huge dirty=4 N0=1 N1=1 N2=1 N3=1
> 2c000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.PFt0xt\040(deleted) huge dirty=4 N0=1 N1=1 N2=1 N3=1
> 30000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.PFt0xt\040(deleted) huge dirty=4 N0=1 N1=1 N2=1 N3=1
> 34000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.PFt0xt\040(deleted) huge dirty=4 N0=1 N1=1 N2=1 N3=1
> 38000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.PFt0xt\040(deleted) huge dirty=1 mapped=4 N0=1 N1=1 N2=1 N3=1
Hmm... Strange. Interleaving should continue after the last one....
^ permalink raw reply
* Re: libnuma interleaving oddness
From: Nishanth Aravamudan @ 2006-08-30 5:31 UTC (permalink / raw)
To: Christoph Lameter; +Cc: linux-mm, lnxninja, ak, linuxppc-dev
In-Reply-To: <Pine.LNX.4.64.0608292123230.23009@schroedinger.engr.sgi.com>
On 29.08.2006 [21:26:58 -0700], Christoph Lameter wrote:
> On Tue, 29 Aug 2006, Nishanth Aravamudan wrote:
>
> > If I use the default hugepage-aligned hugepage-backed malloc
> > replacement, I get the following in /proc/pid/numa_maps (excerpt):
> >
> > 20000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.3JbO7R\040(deleted) huge dirty=1 N0=1
> > 21000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.3JbO7R\040(deleted) huge dirty=1 N0=1
> > ...
> > 37000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.3JbO7R\040(deleted) huge dirty=1 N0=1
> > 38000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.3JbO7R\040(deleted) huge dirty=1 N0=1
>
> Is this with nodemask set to [0]?
nodemask was set to 0xFF, effectively, bits 0-7 set, all others cleared.
Just to make sure that I'm not misunderstanding, that's what the
interleave=0-7 also indicates, right? That the particular memory area
was specified to interleave over those nodes, if possible, and then at
the end of each line are the nodes that it actually was placed on?
> > If I change the nodemask to 1-7, I get:
> >
> > 20000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N1=1
> > 21000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N2=1
> > 22000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N3=1
> > 23000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N4=1
> > 24000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N5=1
> > 25000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N6=1
> > 26000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N7=1
> > ...
> > 35000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N1=1
> > 36000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N2=1
> > 37000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N3=1
> > 38000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N4=1
>
> So interleave has an effect.
Yup, exactly -- and that's the confusing part. I was willing to write it
off as being some sort of mistake on my part, but all I have to do is
clear any one bit between 0 and 7, and I get the interleaving I expect.
That's what leads me to conclude there is a bug, but after a lot of
looking at libnuma and the mbind() system call, I couldn't see the
problem.
> Are you using cpusets? Or are you only using memory policies? What is
> the default policy of the task you are running?
No cpusets, only memory policies. The test application that is
exhibiting this behavior is *really* simple, and doesn't specifically
set a memory policy, so I assume it's MPOL_DEFAULT?
> > If I then change our malloc implementation to (unnecessarily) mmap a
> > size aligned to 4 hugepages, rather aligned to a single hugepage,
> > but using a nodemask of 0-7, I get:
> >
> > 20000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.PFt0xt\040(deleted) huge dirty=4 N0=1 N1=1 N2=1 N3=1
> > 24000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.PFt0xt\040(deleted) huge dirty=4 N0=1 N1=1 N2=1 N3=1
> > 28000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.PFt0xt\040(deleted) huge dirty=4 N0=1 N1=1 N2=1 N3=1
> > 2c000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.PFt0xt\040(deleted) huge dirty=4 N0=1 N1=1 N2=1 N3=1
> > 30000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.PFt0xt\040(deleted) huge dirty=4 N0=1 N1=1 N2=1 N3=1
> > 34000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.PFt0xt\040(deleted) huge dirty=4 N0=1 N1=1 N2=1 N3=1
> > 38000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.PFt0xt\040(deleted) huge dirty=1 mapped=4 N0=1 N1=1 N2=1 N3=1
>
> Hmm... Strange. Interleaving should continue after the last one....
"last one" being the last allocation, or the last node? My understanding
of what is happening in this case is that interleave is working, but in
a way different from the immediately previous example. Here we're
interleaving within the allocation, so each of the 4 hugepages goes on a
different node. When the next allocation comes through, we start back
over at node 0 (given the previous results, I would have thought it
would have gone N0,N1,N2,N3 then N4,N5,N6,N7 then back to N0,N1,N2,N3).
Also, note that in this last case, in case I wasn't clear before, I was
artificially inflating our consumption of hugepages per allocation, just
to see what happened.
I should also mention this is the SuSE kernel, too, so 2.6.16-ish. If
there are sufficient changes in this area between there and mainline, I
can try and get the box rebooted into 2.6.18-rc5.
Thanks,
Nish
--
Nishanth Aravamudan <nacc@us.ibm.com>
IBM Linux Technology Center
^ permalink raw reply
* Re: libnuma interleaving oddness
From: Tim Pepper @ 2006-08-30 5:40 UTC (permalink / raw)
To: Christoph Lameter; +Cc: linuxppc-dev, Nishanth Aravamudan, ak, linux-mm
In-Reply-To: <Pine.LNX.4.64.0608292123230.23009@schroedinger.engr.sgi.com>
On 8/29/06, Christoph Lameter <clameter@sgi.com> wrote:
> On Tue, 29 Aug 2006, Nishanth Aravamudan wrote:
>
> > If I use the default hugepage-aligned hugepage-backed malloc
> > replacement, I get the following in /proc/pid/numa_maps (excerpt):
> >
> > 20000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.3JbO7R\040(deleted) huge dirty=1 N0=1
> > 21000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.3JbO7R\040(deleted) huge dirty=1 N0=1
> > ...
> > 37000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.3JbO7R\040(deleted) huge dirty=1 N0=1
> > 38000000 interleave=0-7 file=/libhugetlbfs/libhugetlbfs.tmp.3JbO7R\040(deleted) huge dirty=1 N0=1
>
> Is this with nodemask set to [0]?
The above is with a nodemask of 0-7. Just removing node 0 from the mask causes
interleaving to start as below:
> > If I change the nodemask to 1-7, I get:
> >
> > 20000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N1=1
> > 21000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N2=1
> > 22000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N3=1
> > 23000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N4=1
> > 24000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N5=1
> > 25000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N6=1
> > 26000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N7=1
> > ...
> > 35000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N1=1
> > 36000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N2=1
> > 37000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N3=1
> > 38000000 interleave=1-7 file=/libhugetlbfs/libhugetlbfs.tmp.Eh9Bmp\040(deleted) huge dirty=1 N4=1
>
> So interleave has an effect.
>
> Are you using cpusets? Or are you only using memory policies? What is the
> default policy of the task you are running?
Just memory policies with the default task policy...really simple
code. The current incantation basically does setup in the form of:
numa_available();
nodemask_zero(&nodemask);
for (i = 0; i <= maxnode; i++)
nodemask_set(&nodemask, i);
and then creates mmaps followed by:
numa_interleave_memory(p, size, &nodemask);
mlock(p, size)
munlock(p, size);
to get the page faulted in.
> Hmm... Strange. Interleaving should continue after the last one....
That's what we thought...good to know we're not crazy. We've spent a
lot of time looking at libnuma and the userspace side of things trying
to figure out if we were somehow passing an invalid nodemask into the
kernel, but we've pretty well convinced ourselves that is not the
case. The kernel side of things (eg: sys_mbind() codepath) isn't
exactly obvious...code inspection's been a bit gruelling...need to do
kernel side probing to see what codepaths we're actually hitting.
An interesting additional point: Nish's code originally wasn't using
libnuma and I wrote a simple little mmapping test program using
libnuma to compare results (thinking userspace issue). My code worked
fine. He rewrote to use libnuma and I rewrote to not use libnuma
thinking we'd find the problem in between. Yet my code still gets
interleaving and his does not. The only real difference between our
code is that mine basically does:
mmap(...many hugepages...)
and Nish's effectively is doing:
foreach(1..n) { mmap(...many/n hugepages...)}
if that pseudocode makes sense. As above, when he changes his mmap to
grab more than one hugepage of memory at a time he starts seeing
interleaving.
Tim
^ permalink raw reply
* Re: Linux v2.6.18-rc5
From: Paul Mackerras @ 2006-08-30 6:13 UTC (permalink / raw)
To: Olaf Hering
Cc: linuxppc-dev, Linus Torvalds, Nathan Lynch,
Linux Kernel Mailing List
In-Reply-To: <20060829155216.GA25861@aepfle.de>
Olaf,
This patch should fix it. The problem was that I was comparing a
32-bit quantity with a 64-bit quantity, and consequently time wasn't
advancing. This makes us use a 64-bit quantity on all platforms,
which ends up simplifying the code since we can now get rid of the
tb_last_stamp variable (which actually fixes another bug that Ben H
and I noticed while going carefully through the code).
This works fine on my G4 tibook. Let me know how it goes on your
machines.
Paul.
diff --git a/arch/powerpc/kernel/time.c b/arch/powerpc/kernel/time.c
index 18e59e4..a124499 100644
--- a/arch/powerpc/kernel/time.c
+++ b/arch/powerpc/kernel/time.c
@@ -125,15 +125,8 @@ static long timezone_offset;
unsigned long ppc_proc_freq;
unsigned long ppc_tb_freq;
-u64 tb_last_jiffy __cacheline_aligned_in_smp;
-unsigned long tb_last_stamp;
-
-/*
- * Note that on ppc32 this only stores the bottom 32 bits of
- * the timebase value, but that's enough to tell when a jiffy
- * has passed.
- */
-DEFINE_PER_CPU(unsigned long, last_jiffy);
+static u64 tb_last_jiffy __cacheline_aligned_in_smp;
+static DEFINE_PER_CPU(u64, last_jiffy);
#ifdef CONFIG_VIRT_CPU_ACCOUNTING
/*
@@ -458,7 +451,7 @@ void do_gettimeofday(struct timeval *tv)
do {
seq = read_seqbegin_irqsave(&xtime_lock, flags);
sec = xtime.tv_sec;
- nsec = xtime.tv_nsec + tb_ticks_since(tb_last_stamp);
+ nsec = xtime.tv_nsec + tb_ticks_since(tb_last_jiffy);
} while (read_seqretry_irqrestore(&xtime_lock, seq, flags));
usec = nsec / 1000;
while (usec >= 1000000) {
@@ -700,7 +693,6 @@ #endif
tb_next_jiffy = tb_last_jiffy + tb_ticks_per_jiffy;
if (per_cpu(last_jiffy, cpu) >= tb_next_jiffy) {
tb_last_jiffy = tb_next_jiffy;
- tb_last_stamp = per_cpu(last_jiffy, cpu);
do_timer(regs);
timer_recalc_offset(tb_last_jiffy);
timer_check_rtc();
@@ -749,7 +741,7 @@ void __init smp_space_timers(unsigned in
int i;
unsigned long half = tb_ticks_per_jiffy / 2;
unsigned long offset = tb_ticks_per_jiffy / max_cpus;
- unsigned long previous_tb = per_cpu(last_jiffy, boot_cpuid);
+ u64 previous_tb = per_cpu(last_jiffy, boot_cpuid);
/* make sure tb > per_cpu(last_jiffy, cpu) for all cpus always */
previous_tb -= tb_ticks_per_jiffy;
@@ -830,7 +822,7 @@ #endif
* and therefore the (jiffies - wall_jiffies) computation
* has been removed.
*/
- tb_delta = tb_ticks_since(tb_last_stamp);
+ tb_delta = tb_ticks_since(tb_last_jiffy);
tb_delta = mulhdu(tb_delta, do_gtod.varp->tb_to_xs); /* in xsec */
new_nsec -= SCALE_XSEC(tb_delta, 1000000000);
@@ -950,8 +942,7 @@ void __init time_init(void)
if (__USE_RTC()) {
/* 601 processor: dec counts down by 128 every 128ns */
ppc_tb_freq = 1000000000;
- tb_last_stamp = get_rtcl();
- tb_last_jiffy = tb_last_stamp;
+ tb_last_jiffy = get_rtcl();
} else {
/* Normal PowerPC with timebase register */
ppc_md.calibrate_decr();
@@ -959,7 +950,7 @@ void __init time_init(void)
ppc_tb_freq / 1000000, ppc_tb_freq % 1000000);
printk(KERN_DEBUG "time_init: processor frequency = %lu.%.6lu MHz\n",
ppc_proc_freq / 1000000, ppc_proc_freq % 1000000);
- tb_last_stamp = tb_last_jiffy = get_tb();
+ tb_last_jiffy = get_tb();
}
tb_ticks_per_jiffy = ppc_tb_freq / HZ;
@@ -1036,7 +1027,7 @@ void __init time_init(void)
do_gtod.varp = &do_gtod.vars[0];
do_gtod.var_idx = 0;
do_gtod.varp->tb_orig_stamp = tb_last_jiffy;
- __get_cpu_var(last_jiffy) = tb_last_stamp;
+ __get_cpu_var(last_jiffy) = tb_last_jiffy;
do_gtod.varp->stamp_xsec = (u64) xtime.tv_sec * XSEC_PER_SEC;
do_gtod.tb_ticks_per_sec = tb_ticks_per_sec;
do_gtod.varp->tb_to_xs = tb_to_xs;
diff --git a/include/asm-powerpc/time.h b/include/asm-powerpc/time.h
index dcde441..5785ac4 100644
--- a/include/asm-powerpc/time.h
+++ b/include/asm-powerpc/time.h
@@ -30,10 +30,6 @@ extern unsigned long tb_ticks_per_usec;
extern unsigned long tb_ticks_per_sec;
extern u64 tb_to_xs;
extern unsigned tb_to_us;
-extern unsigned long tb_last_stamp;
-extern u64 tb_last_jiffy;
-
-DECLARE_PER_CPU(unsigned long, last_jiffy);
struct rtc_time;
extern void to_tm(int tim, struct rtc_time * tm);
^ permalink raw reply related
* How to turn off MMU in MPC8540
From: Reddy Suneel-ASR125 @ 2006-08-30 6:25 UTC (permalink / raw)
To: linuxppc-embedded
[-- Attachment #1: Type: text/plain, Size: 83 bytes --]
Hi
Can anyone know how to turn off MMU in MPC8540?
Thanks®ards
Suneel
[-- Attachment #2: Type: text/html, Size: 849 bytes --]
^ permalink raw reply
* Gabe Johnson is out of the office.
From: Gabe.Johnson @ 2006-08-30 6:28 UTC (permalink / raw)
To: linuxppc-embedded
I will be out of the office starting 08/30/2006 and will not return until
09/05/2006.
I will respond to your message when I return.
^ permalink raw reply
* RE: How to turn off MMU in MPC8540
From: Li Yang-r58472 @ 2006-08-30 6:31 UTC (permalink / raw)
To: Reddy Suneel-ASR125, linuxppc-embedded
In-Reply-To: <405ECA8A30557F439A723E52D50838F9D8295F@ZMY16EXM66.ds.mot.com>
> -----Original Message-----
> From:=20
> linuxppc-embedded-bounces+leoli=3Dfreescale.com@ozlabs.org=20
> [mailto:linuxppc-embedded-bounces+leoli=3Dfreescale.com@ozlabs.o
> rg] On Behalf Of Reddy Suneel-ASR125
> Sent: Wednesday, August 30, 2006 2:25 PM
> To: linuxppc-embedded@ozlabs.org
> Subject: How to turn off MMU in MPC8540
>=20
> Hi
> =20
> Can anyone know how to turn off MMU in MPC8540?
I heard that it can't be turned off on e500.
^ permalink raw reply
* RE: How to turn off MMU in MPC8540
From: Liu Dave-r63238 @ 2006-08-30 6:31 UTC (permalink / raw)
To: Reddy Suneel-ASR125, linuxppc-embedded
In-Reply-To: <405ECA8A30557F439A723E52D50838F9D8295F@ZMY16EXM66.ds.mot.com>
[-- Attachment #1: Type: text/plain, Size: 531 bytes --]
Turn off MMU of e500? hmmm...
Do you want to get real address space?
You can emulate this.
-DAve
________________________________
From: linuxppc-embedded-bounces+daveliu=freescale.com@ozlabs.org
[mailto:linuxppc-embedded-bounces+daveliu=freescale.com@ozlabs.org] On
Behalf Of Reddy Suneel-ASR125
Sent: Wednesday, August 30, 2006 2:25 PM
To: linuxppc-embedded@ozlabs.org
Subject: How to turn off MMU in MPC8540
Hi
Can anyone know how to turn off MMU in MPC8540?
Thanks®ards
Suneel
[-- Attachment #2: Type: text/html, Size: 2236 bytes --]
^ permalink raw reply
* Re: RE: atomic operations in user space
From: Olof Johansson @ 2006-08-30 6:33 UTC (permalink / raw)
To: Liu Dave-r63238; +Cc: linuxppc-dev, linuxppc-embedded
In-Reply-To: <995B09A8299C2C44B59866F6391D263511BB62@zch01exm21.fsl.freescale.net>
On Wed, Aug 30, 2006 at 10:55:39AM +0800, Liu Dave-r63238 wrote:
> Can we do atomic operation in user space as kernel space?
Yes.
-Olof
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox