(unknown),

(unknown),

[Greg Kilfoyle]

subscribe

(unknown),

[raciel]

Somebody can tell me where i can get the LSM patch?
Regards Raciel

(unknown),

[Mala Anand]

 > "David S. Miller" wrote:
>> NAPI is also not the panacea to all problems in the world.

   >Mala did some testing on this a couple of weeks back. It appears that
   >NAPI damaged performance significantly.



>http://www-124.ibm.com/developerworks/opensource/linuxperf/netperf/results/july_02/netperf2.5.25results.htm



>Unfortunately it is not listed what e1000 and core NAPI
>patch was used. Also, not listed, are the RX/TX mitigation
>and ring sizes given to the kernel module upon loading.
The default driver that is included in 2.5.25 kernel for Intel
gigabit adapter was used for the baseline test and the NAPI driver
was downloaded from Robert Olsson's website. I have updated my web
page to include Robert's patch. However it is given there for reference
purpose only. Except for the ones mentioned explicitly the rest of
the configurable values used are default. The default for RX/TX mitigation
is 64 microseconds and the default ring size is 80.

I have added statistics collected during the test to my web site. I do
want to analyze and understand how NAPI can be improved in my tcp_stream
test. Last year around November, when I first tested NAPI, I did find NAPI
results better than the baseline using udp_stream. However I am
concentrating on tcp_stream since that is where NAPI can be improved in
my setup. I will update the website as I do more work on this.


>Robert can comment on optimal settings
I saw Robert's postings. Looks like he may have a more recent version of
NAPI
driver than the one I used. I also see 2.5.33 has NAPI, I will move to
2.5.33
and continue my work on that.


>Robert and Jamal can make a more detailed analysis of Mala's
>graphs than I.
Jamal has questioned about socket buffer size that I used, I have tried
132k
socket buffer size in the past and I didn't see much difference in my
tests.
I will add that to my list again.


Regards,
    Mala


   Mala Anand
   IBM Linux Technology Center - Kernel Performance
   E-mail:manand@us.ibm.com
   http://www-124.ibm.com/developerworks/opensource/linuxperf
   http://www-124.ibm.com/developerworks/projects/linuxperf
   Phone:838-8088; Tie-line:678-8088

(unknown),

[Mala Anand]

 > "David S. Miller" wrote:
>> NAPI is also not the panacea to all problems in the world.

   >Mala did some testing on this a couple of weeks back. It appears that
   >NAPI damaged performance significantly.



>http://www-124.ibm.com/developerworks/opensource/linuxperf/netperf/results/july_02/netperf2.5.25results.htm




>Unfortunately it is not listed what e1000 and core NAPI
>patch was used. Also, not listed, are the RX/TX mitigation
>and ring sizes given to the kernel module upon loading.
The default driver that is included in 2.5.25 kernel for Intel
gigabit adapter was used for the baseline test and the NAPI driver
was downloaded from Robert Olsson's website. I have updated my web
page to include Robert's patch. However it is given there for reference
purpose only. Except for the ones mentioned explicitly the rest of
the configurable values used are default. The default for RX/TX mitigation
is 64 microseconds and the default ring size is 80.

I have added statistics collected during the test to my web site. I do
want to analyze and understand how NAPI can be improved in my tcp_stream
test. Last year around November, when I first tested NAPI, I did find NAPI
results better than the baseline using udp_stream. However I am
concentrating on tcp_stream since that is where NAPI can be improved in
my setup. I will update the website as I do more work on this.


>Robert can comment on optimal settings
I saw Robert's postings. Looks like he may have a more recent version of
NAPI
driver than the one I used. I also see 2.5.33 has NAPI, I will move to
2.5.33
and continue my work on that.


>Robert and Jamal can make a more detailed analysis of Mala's
>graphs than I.
Jamal has questioned about socket buffer size that I used, I have tried
132k
socket buffer size in the past and I didn't see much difference in my
tests.
I will add that to my list again.


Regards,
    Mala


   Mala Anand
   IBM Linux Technology Center - Kernel Performance
   E-mail:manand@us.ibm.com
   http://www-124.ibm.com/developerworks/opensource/linuxperf
   http://www-124.ibm.com/developerworks/projects/linuxperf
   Phone:838-8088; Tie-line:678-8088

(unknown),

[mdews]

I have a problem with my newly installed firewall. A friend installed it for 
me and now I can't download music or some other things. Is there a way around 
this and if so how.

(unknown),

[Gao XiaoPeng]

hello,

one of my friends wants to discuss a netfilter problem with
you! thx!

-------------------------------------------------------------
hi,
    I am a student, I think that skb has all the information 
that is needed for sending and receiving.So I get the skb 
pointer at NF_IP_POST_ROUTING, put it in a chain  organized 
by myself (I use a spinlock_t  lock to control the access of 
the chain, I named it mylock), and return NF_STOLEN. 
  I make a  tq_timer task to start ip_finish_output2(I export 
it from kernel),ip_finish_output2 use the skb from my chain.I 
can make ftp run ok for almost 1 hour, but then the system will 
carsh with this information:
 
        ds:0018    es:0018    ss:0018
        process swapper(pid:0, stackpage = c0265000)
        stack: c01a07ea c173a088 ...........
        call trace:[<c01a07ea>]    [<c01a156d>]......
        code: 0f 0b 89 7c 24 04 b8 03 00 00 00......
        <0>kernel panic: Aiee, killing interrupt handler!
        In interrupt handler - not syncing!
 
     I want to know why I count run for some time but could not 
go on for a long time . Does it possible to transmit data by the 
way and how to do?thanks very much!

best regard

Chen Wei

(unknown),

[Andreas Herrmann]

help

--
Linux for eServer Development
Tel :  +49-7031-16-4640
Notes mail :  Andreas Herrmann/GERMANY/IBM@IBMDE
email :  aherrman@de.ibm.com

(unknown)

[netdev-bounce]

rom rddunlap@osdl.org Thu Feb 27 11:16:48 2003
Received: with ECARTIS (v1.0.0; list netdev); Thu, 27 Feb 2003 11:16:52 -0800 (PST)
Received: from mail.osdl.org (air-2.osdl.org [65.172.181.6])
	by oss.sgi.com (8.12.5/8.12.5) with SMTP id h1RJGleA011690
	for <netdev@oss.sgi.com>; Thu, 27 Feb 2003 11:16:47 -0800
Received: from dragon.pdx.osdl.net (dragon.pdx.osdl.net [172.20.1.27])
	by mail.osdl.org (8.11.6/8.11.6) with SMTP id h1RJGjw31760;
	Thu, 27 Feb 2003 11:16:45 -0800
Date: Thu, 27 Feb 2003 11:12:12 -0800
From: "Randy.Dunlap" <rddunlap@osdl.org>
To: linux-net@vger.kernel.org
Cc: netdev@oss.sgi.com
Subject: linux_mib docs?
Message-Id: <20030227111212.40388562.rddunlap@osdl.org>
Organization: OSDL
X-Mailer: Sylpheed version 0.8.6 (GTK+ 1.2.10; i586-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
X-archive-position: 1813
X-ecartis-version: Ecartis v1.0.0
Sender: netdev-bounce@oss.sgi.com
Errors-to: netdev-bounce@oss.sgi.com
X-original-sender: rddunlap@osdl.org
Precedence: bulk
X-list: netdev

Hi,

Is there a MIB definition for struct linux_mib in
include/net/snmp.h ?

Thanks,
--
~Randy

(unknown),

[Anantha Mohan]

[-- Attachment #1: Type: text/plain, Size: 24 bytes --]

subscribe netdev digest

[-- Attachment #2: Type: text/html, Size: 326 bytes --]

(unknown),

[Linda Xie]

Hi,

Can anyone explain why base_addr member of net_device struct in
include/linux/netdevice.h is defined as
unsigned long, while base_addr member of ifmap struct in include/linux/if.h
is defined as unsigned short?

I have a test case where netdev->base_addr is 0x230000. When dev_ifsioc()
processes a "SIOCGIFMAP"
request, it put 0x0000 into ifr->ifr_map.base_addr instead of 0x230000.

Should base_addr member of ifmap struct be defined as unsigned long and
ifconfig command be modified accordingly?

Any thoughts?

Linda

(unknown),

[Hubertus Krogmann]

[-- Attachment #1: Type: text/plain, Size: 1881 bytes --]

Hello

Thanx 1st for providing this forcedeth driver.

It works on my MSI nforce board
force:~ # lspci
00:00.0 Host bridge: nVidia Corporation nForce CPU bridge (rev b2)
00:00.1 RAM memory: nVidia Corporation nForce 220/420 Memory Controller 
(rev b2)
00:00.2 RAM memory: nVidia Corporation nForce 220/420 Memory Controller 
(rev b2)
00:00.3 RAM memory: nVidia Corporation nForce 420 Memory Controller 
(DDR) (rev b2)
00:01.0 ISA bridge: nVidia Corporation nForce ISA Bridge (rev c3)
00:01.1 SMBus: nVidia Corporation nForce PCI System Management (rev c1)
00:02.0 USB Controller: nVidia Corporation nForce USB Controller (rev 
c3)
00:03.0 USB Controller: nVidia Corporation nForce USB Controller (rev 
c3)
00:04.0 Ethernet controller: nVidia Corporation nForce Ethernet 
Controller (rev c2)
00:05.0 Multimedia audio controller: nVidia Corporation: Unknown device 
01b0 (rev c2)
00:06.0 Multimedia audio controller: nVidia Corporation nForce Audio 
(rev c2)
00:08.0 PCI bridge: nVidia Corporation nForce PCI-to-PCI bridge (rev c2)
00:09.0 IDE interface: nVidia Corporation nForce IDE (rev c3)
00:1e.0 PCI bridge: nVidia Corporation nForce AGP to PCI Bridge (rev b2)
01:01.0 FireWire (IEEE 1394): Texas Instruments TSB12LV26 IEEE-1394 
Controller (Link)
01:03.0 SCSI storage controller: LSI Logic / Symbios Logic 53c860 (rev 
02)
02:00.0 VGA compatible controller: nVidia Corporation NVCrush11 
[GeForce2 MX Integrated Graphics] (rev b1)

I do have a problem, my system hangs sometimes for about 5-10 seconds.
While I ping my system : no hangs
Start ping while it hangs : hang is over immediately.

That is why I will try your driver.

Anything I may test ?
Another PC w2k and a Apple iBook can be used with a SMC router with 
integrated 4 port switch...

--
Hubertus Krogmann   --  hk @ hkit . de  -- www . hkit . de
GPG Fingerprint 42 F9 76 30 C0 E6 A5 59  AE F9 DD 12 A2 29 69 2F

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 478 bytes --]

(unknown),

[Mon]

Hi,

Well i've tried your forcedeth module (v0.23, i'm running 2.6.3) and i
have to say: it works great! 
When copying 800mb files locally i get a stable 950 kb/s. I'm using a
crappy 10mbit hub to connect my pc's here so unfortunatly i can't test a
100mb connection. When i'm on a 100mb switched network, i'll test again.

I also have a question: does Wake on Lan Work?
	If so: great, i hope to test it soon.
	If not: will it work in a future release?

Anyway, im really glad i can use a nice opensource driver for my onboard
nic. Maybe now i can pull the eepro100 out again :)

I'm looking forward to a new version of the module, and i'll be happy to
test it.

Ramon.

(unknown),

[Blaine Quick]

[-- Attachment #1: Type: text/plain, Size: 525 bytes --]

helen montage diebold convict scotty confrere delectate compendia emphatic handshake azure syncopate abner seclude 
clench angelo corrigenda bring valkyrie drowse burt grandiloquent dentistry fishy telescope schafer octile clarinet frayed edmonds aye 
colombia clark vivo spore dishevel salvage burial abase duffy enterprise dragonhead pickerel helvetica avon jutish crusade reynolds dwight synonym cheesecloth normalcy chaff attainder collocation pinhole sri goa canal lev scripps papyrus archimedes york porch ftc student 

[-- Attachment #2: Type: text/html, Size: 1073 bytes --]

(unknown),

[彭丹]

help

(unknown),

[Bigger Dik]

[-- Attachment #1: honduras pedantry averred --]
[-- Type: text/html, Size: 2361 bytes --]

(unknown),

[Desenhar é Fácil - Aprenda já]

[-- Attachment #1: Type: text/html, Size: 944 bytes --]

(unknown),

[Zoran Bosic (ZG/ETK)]

(unknown),

[Ƽ¼ÅÃ÷]

[-- Attachment #1: Type: text/html, Size: 1409 bytes --]

(unknown),

[°í¼Ó ÃÊÀ½ÆÄ]

[-- Attachment #1: Type: text/html, Size: 1829 bytes --]

(unknown),

[°í¼Ó ÃÊÀ½ÆÄ]

[-- Attachment #1: Type: text/html, Size: 1829 bytes --]

(unknown),

[¸ôīŽÁö±â]

[-- Attachment #1: Type: text/html, Size: 1364 bytes --]

(unknown),

[netdev]

[-- Attachment #1: Type: text/html, Size: 4615 bytes --]

(unknown),

[Chris Boot]

Hi,

I've just seen the following assertions pop out of one of my servers  
running 2.6.16.9 with grsecurity. I've searched the archives of LKML  
and netdev and I've only found posts relating to 2.6.9, after which  
some related bugs were fixed... It looks like these bugs are related  
to e1000, which is the driver I'm using. The system was running 24  
days before these appeared and it's still running absolutely fine.

May 16 09:15:12 baldrick kernel: [6442250.504000] KERNEL: assertion (! 
sk->sk_forward_alloc) failed at net/core/stream.c (283)
May 16 09:15:12 baldrick kernel: [6442250.513000] KERNEL: assertion (! 
sk->sk_forward_alloc) failed at net/ipv4/af_inet.c (150)

baldrick bootc # ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on

Many thanks,
Chris

PS: I'm not subscribed to netdev.

-- 
Chris Boot
bootc@bootc.net
http://www.bootc.net/

(unknown)

[Anne Thrax]

subscribe foobarfoobarfoobar@gmail.com

(unknown),

[Neal Sidhwaney]

subscribe linux-netdev
---

(unknown)

[Kunio Murasawa]

(unknown)

[sonny1333]

(unknown)

[Lior Dotan]

subscribe

(unknown),

[Sachin K]

subscribe

(unknown),

[haitao wang]

subscribe netdev

(unknown)

[kou.ishizaki]

jgarzik@pobox.com, jim@jklewis.com
Subject: Re: [Cbe-oss-dev] spidernet: dynamic phy setup code
In-Reply-To: <200701261409.29537.jens@de.ibm.com>
From: Ishizaki Kou <kouish@swc.toshiba.co.jp>
X-Mailer: Mew version 4.2 on Emacs 21.3 / Mule 5.0 (SAKAKI)
Mime-Version: 1.0
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Jens-san

> This patch modifies the patch submitted by Kou Ishizaki to make it
work on the
> blade
(http://marc.theaimsgroup.com/?l=linux-netdev&m=116593424505539&w=2).
> Unfortunately I dont have access to a Celleb so I cannot test it
there.

Thanks for arranging our patch to work on Cell Blade.

This patch partially works on celleb but remains 
following several problems.
1. It doesn't recover once an ethernet cable which is
   connected to a spider_net card is unpluged. 
2. It doesn't work when the spider_net card is connected to 
   a 100Mbps ethernet switch.

To solve these problems, we need to restore some codes
you removed from your patch.

(1)
>- if (card->aneg_count > 10) {
>-  /* timeout */
>-  card->aneg_count = 0;
>-  is1000 = !is1000;
>-  goto re_setup;

>- if (phy->speed == 1000 && !is1000) {
>-  is1000 = 1;
>-  goto re_setup;
>- } else if(phy->speed != 1000 && is1000) {
>-  is1000 = 0;
>-  goto re_setup;
>- }

We need to use different auto-neg initial settings between
for 10/100Mbps ethernet switches and for Gbps ethernet switches.
Driver don't know which type of network switch is connected to
network card, so we try both settings alternately in auto negtiation
sequences by using a variable "is1000".
Furthermore, we have a problem that poll_link() may succeed even when
the auto-neg initial setting is for different network switch type,
and the network card does not work on this case. We retry auto-neg
with the another initial setting on this case.

#We are commented that "is1000" should be in spider_net_card.
#We fixed it in another patch. Please refer the following.
#http://ozlabs.org/pipermail/linuxppc-dev/2007-January/030203.html

But we don't think this is the best solution, and we are still
developing 
our spidernet driver. If you have a good alternative idea, please tell
us.

(2)
>- spider_net_write_reg(card, SPIDER_NET_GMACST,
>-        spider_net_read_reg(card, SPIDER_NET_GMACST));
>- spider_net_write_reg(card, SPIDER_NET_GMACINTEN, 0x4);

These codes are enabling LINK status interrupt which is disabled
at the beginning of auto-neg.
Without this operation, auto negotiation works only when a connection
detected for the first time, and auto negotiation will not work 
when an ethernet cable is unpluged or pluged.

(3)
>- mii_phy_probe(phy, phy->mii_id);
It seems that PHY reset is necessary before auto negotiation,
after a link once went down.
We can't call directly reset routine from driver, so we call
mii_phy_probe().
We are still developping the patch as we noted, and we are considering
to call mii_phy_probe() from spider_net_setup_aneg(), or to call
reset_one_mii_phy() from bcm54xx_setup_aneg().

We think these (1)-(3) are necessary, but we are afraid that you removed
them
by a reason that they causes some trouble in Cell Blade. If so please
tell us.


Best regards,
Kou Ishizaki

(unknown),

[georgios]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: multipart/mixed; boundary="----=_NextPart_000_0014_F2A0F142.57757F8F", Size: 29 bytes --]

<<< No Message Collected >>>

(unknown)

[Luca Fornasari]

Help

-- 
Luca Fornasari
FURNA.COM

(unknown)

[Topher Fischer]

subscribe

(unknown)

[Inaky Perez-Gonzalez]

subscribe inaky@linux.intel.com

(unknown),

[Imed Ben Ghozi]

subscribe netdev

(unknown)

[Jarek Poplawski]

Jean-Baptiste Vignaud <vignaud@xandmail.fr>,
"marcin.slusarz" <marcin.slusarz@gmail.com>,
shemminger <shemminger@linux-foundation.org>
Subject: Re: [PATCH] 8139cp dev->tx_timeout
References: <OFC04C6285.18455C26-ONC2257306.00364F82-C2257307.002CFBE9@stonesoft.com> <46822182.5070002@garzik.org>
In-Reply-To: <46822182.5070002@garzik.org>

On 27-06-2007 10:36, Jeff Garzik wrote:
> Mika.Lansirinne@stonesoft.com wrote:
>> Hello All,
>>
>> We have been experimenting a couple of interface hangs with the 8139cp
>> driver. It appears that the tx buffer stops transmitting and never starts
>> up again in some yet unknown conditions. To be able to circumvent this we
>> implemented the missing dev->tx_timeout function which should reset the
>> interface and allow it work again.
>>
>> The problem reoccurs quite seldom and we have yet to be able to confirm
>> that the attached patch helps the situation. We though that we should
>> submit the patch anyway for reviewing.
>>
>> The patch is made against 2.6.21.5 kernel.
> 
> Seems OK, but I'm definitely interested in hearing test feedback.

Hi!

I think maybe there are too many similar problems under 2.6.21
to be individual, and there could be something common to fix?

So, I "link" here a few  probably interested souls to cc:

Subject: Re: 2.6.20->2.6.21 - networking dies after random time
...
On Tue, Jun 26, 2007 at 04:24:07PM +0200, Jean-Baptiste Vignaud wrote:
> Hello, i have a very similar problem with 2.6.21 also;
> 
> 2 3com NICs and they are failling randomly.
> 
> The kernel is a basic fedora 7 kernel (2.6.21-1.3228.fc7)
> I found a bug report and added details here : https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=243960
> 
> I'm not subcribed on this list, so please cc me if there is any questions.
> 
> JB
> 
> > On Tue, Jun 26, 2007 at 08:10:17AM +0200, Marcin Ślusarz wrote:
> > ...
> > > I reproduced it on minimal config:
...
> > We know your hardware should be OK - since it was fine with 2.6.20.
...

It looks like there is something common in the air...

Marcin: ne2k_pci with 8390, Jean: 3com, and now I see
similar problem with 8139cp too (plus some ideas):

http://marc.info/?l=linux-netdev&m=118293314109648&w=2

So, you probably should wait a little & look for new patches here.

Cheers,
Jarek P.

(unknown),

[Bhanu Kalyan Chetlapalli]

subscribe netdev

(unknown)

[Patrizio Bassi]

Hi

after some email exchange with Stephen Hemminger i decide to forward the question to netdev ml.

I'm using vmware 6 and bridged networking with interface eth1 (sky2 driver).

The brigde works only when eth1 has the physical connection enabled (another pc plugged in), so
i always need to have another pc running to have virtual networking.
with tcpdump i can see packets coming from virtual machines, but none
sent back.

With other drivers (some old 3com cards) i don't have this issue.

I wonder if it's possibile to have the sky2 interface always "alive" (as
ifconfig reports it up) in order to have the bridge working.
I looked for some ethtool options...but nothing...

i saw in the code the queue disable, but i would like to switch this
somehow runtime...

thanks for support

Patrizio

ps. please CC me i'm not subscribed.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Arnd Bergmann]

On Thursday 09 August 2007, Chris Snook wrote:
> This patchset makes the behavior of atomic_read uniform by removing the
> volatile keyword from all atomic_t and atomic64_t definitions that currently
> have it, and instead explicitly casts the variable as volatile in
> atomic_read().  This leaves little room for creative optimization by the
> compiler, and is in keeping with the principles behind "volatile considered
> harmful".
> 

Just an idea: since all architectures already include asm-generic/atomic.h,
why not move the definitions of atomic_t and atomic64_t, as well as anything
that does not involve architecture specific inline assembly into the generic
header?

	Arnd <><

[PATCH 0/24] make atomic_read() behave consistently across all architectures

[Chris Snook]

As recent discussions[1], and bugs[2] have shown, there is a great deal of
confusion about the expected behavior of atomic_read(), compounded by the
fact that it is not the same on all architectures.  Since users expect calls
to atomic_read() to actually perform a read, it is not desirable to allow
the compiler to optimize this away.  Requiring the use of barrier() in this
case is inefficient, since we only want to re-load the atomic_t variable,
not everything else in scope.

This patchset makes the behavior of atomic_read uniform by removing the
volatile keyword from all atomic_t and atomic64_t definitions that currently
have it, and instead explicitly casts the variable as volatile in
atomic_read().  This leaves little room for creative optimization by the
compiler, and is in keeping with the principles behind "volatile considered
harmful".

Busy-waiters should still use cpu_relax(), but fast paths may be able to
reduce their use of barrier() between some atomic_read() calls.

	-- Chris

1)	http://lkml.org/lkml/2007/7/1/52
2)	http://lkml.org/lkml/2007/8/8/122

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Chris Snook]

Arnd Bergmann wrote:
> On Thursday 09 August 2007, Chris Snook wrote:
>> This patchset makes the behavior of atomic_read uniform by removing the
>> volatile keyword from all atomic_t and atomic64_t definitions that currently
>> have it, and instead explicitly casts the variable as volatile in
>> atomic_read().  This leaves little room for creative optimization by the
>> compiler, and is in keeping with the principles behind "volatile considered
>> harmful".
>>
> 
> Just an idea: since all architectures already include asm-generic/atomic.h,
> why not move the definitions of atomic_t and atomic64_t, as well as anything
> that does not involve architecture specific inline assembly into the generic
> header?
> 
> 	Arnd <><

a) chicken and egg: asm-generic/atomic.h depends on definitions in asm/atomic.h

If you can find a way to reshuffle the code and make it simpler, I personally am 
all for it.  I'm skeptical that you'll get much to show for the effort.

b) The definitions aren't precisely identical between all architectures, so it 
would be a mess of special cases, which gets us right back to where we are now.

	-- Chris

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Arnd Bergmann]

On Thursday 09 August 2007, Chris Snook wrote:
> a) chicken and egg: asm-generic/atomic.h depends on definitions in asm/atomic.h

Ok, I see.
 
> If you can find a way to reshuffle the code and make it simpler, I personally am 
> all for it. I'm skeptical that you'll get much to show for the effort. 

I guess it could  be done using more macros or new headers, but I don't see
a way that would actually improve the situation.

> b) The definitions aren't precisely identical between all architectures, so it 
> would be a mess of special cases, which gets us right back to where we are now.

Why are they not identical? Anything beyond the 32/64 bit difference should
be the same afaics, or it might cause more bugs.

	Arnd <><

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Christoph Lameter]

On Thu, 9 Aug 2007, Chris Snook wrote:

> This patchset makes the behavior of atomic_read uniform by removing the
> volatile keyword from all atomic_t and atomic64_t definitions that currently
> have it, and instead explicitly casts the variable as volatile in
> atomic_read().  This leaves little room for creative optimization by the
> compiler, and is in keeping with the principles behind "volatile considered
> harmful".

volatile is generally harmful even in atomic_read(). Barriers control
visibility and AFAICT things are fine.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Chris Snook]

Christoph Lameter wrote:
> On Thu, 9 Aug 2007, Chris Snook wrote:
> 
>> This patchset makes the behavior of atomic_read uniform by removing the
>> volatile keyword from all atomic_t and atomic64_t definitions that currently
>> have it, and instead explicitly casts the variable as volatile in
>> atomic_read().  This leaves little room for creative optimization by the
>> compiler, and is in keeping with the principles behind "volatile considered
>> harmful".
> 
> volatile is generally harmful even in atomic_read(). Barriers control
> visibility and AFAICT things are fine.

But barriers force a flush of *everything* in scope, which we generally don't 
want.  On the other hand, we pretty much always want to flush atomic_* 
operations.  One way or another, we should be restricting the volatile behavior 
to the thing that needs it.  On most architectures, this patch set just moves 
that from the declaration, where it is considered harmful, to the use, where it 
is considered an occasional necessary evil.

See the resubmitted patchset, which also puts a cast in the atomic[64]_set 
operations.

	-- Chris

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Christoph Lameter]

On Tue, 14 Aug 2007, Chris Snook wrote:

> But barriers force a flush of *everything* in scope, which we generally don't
> want.  On the other hand, we pretty much always want to flush atomic_*
> operations.  One way or another, we should be restricting the volatile
> behavior to the thing that needs it.  On most architectures, this patch set
> just moves that from the declaration, where it is considered harmful, to the
> use, where it is considered an occasional necessary evil.

Then we would need

	atomic_read()

and

	atomic_read_volatile()

atomic_read_volatile() would imply an object sized memory barrier before 
and after?

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Chris Snook]

Satyam Sharma wrote:
> 
> On Tue, 14 Aug 2007, Christoph Lameter wrote:
> 
>> On Thu, 9 Aug 2007, Chris Snook wrote:
>>
>>> This patchset makes the behavior of atomic_read uniform by removing the
>>> volatile keyword from all atomic_t and atomic64_t definitions that currently
>>> have it, and instead explicitly casts the variable as volatile in
>>> atomic_read().  This leaves little room for creative optimization by the
>>> compiler, and is in keeping with the principles behind "volatile considered
>>> harmful".
>> volatile is generally harmful even in atomic_read(). Barriers control
>> visibility and AFAICT things are fine.
> 
> Frankly, I don't see the need for this series myself either. Personal
> opinion (others may differ), but I consider "volatile" to be a sad /
> unfortunate wart in C (numerous threads on this list and on the gcc
> lists/bugzilla over the years stand testimony to this) and if we _can_
> steer clear of it, then why not -- why use this ill-defined primitive
> whose implementation has often differed over compiler versions and
> platforms? Granted, barrier() _is_ heavy-handed in that it makes the
> optimizer forget _everything_, but then somebody did post a forget()
> macro on this thread itself ...
> 
> [ BTW, why do we want the compiler to not optimize atomic_read()'s in
>   the first place? Atomic ops guarantee atomicity, which has nothing
>   to do with "volatility" -- users that expect "volatility" from
>   atomic ops are the ones who must be fixed instead, IMHO. ]

Because atomic operations are generally used for synchronization, which requires 
volatile behavior.  Most such codepaths currently use an inefficient barrier(). 
  Some forget to and we get bugs, because people assume that atomic_read() 
actually reads something, and atomic_write() actually writes something.  Worse, 
these are architecture-specific, even compiler version-specific bugs that are 
often difficult to track down.

	-- Chris

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Tue, 14 Aug 2007, Christoph Lameter wrote:

> On Thu, 9 Aug 2007, Chris Snook wrote:
> 
> > This patchset makes the behavior of atomic_read uniform by removing the
> > volatile keyword from all atomic_t and atomic64_t definitions that currently
> > have it, and instead explicitly casts the variable as volatile in
> > atomic_read().  This leaves little room for creative optimization by the
> > compiler, and is in keeping with the principles behind "volatile considered
> > harmful".
> 
> volatile is generally harmful even in atomic_read(). Barriers control
> visibility and AFAICT things are fine.

Frankly, I don't see the need for this series myself either. Personal
opinion (others may differ), but I consider "volatile" to be a sad /
unfortunate wart in C (numerous threads on this list and on the gcc
lists/bugzilla over the years stand testimony to this) and if we _can_
steer clear of it, then why not -- why use this ill-defined primitive
whose implementation has often differed over compiler versions and
platforms? Granted, barrier() _is_ heavy-handed in that it makes the
optimizer forget _everything_, but then somebody did post a forget()
macro on this thread itself ...

[ BTW, why do we want the compiler to not optimize atomic_read()'s in
  the first place? Atomic ops guarantee atomicity, which has nothing
  to do with "volatility" -- users that expect "volatility" from
  atomic ops are the ones who must be fixed instead, IMHO. ]

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Christoph Lameter]

On Tue, 14 Aug 2007, Chris Snook wrote:

> Because atomic operations are generally used for synchronization, which
> requires volatile behavior.  Most such codepaths currently use an inefficient
> barrier().  Some forget to and we get bugs, because people assume that
> atomic_read() actually reads something, and atomic_write() actually writes
> something.  Worse, these are architecture-specific, even compiler
> version-specific bugs that are often difficult to track down.

Looks like we need to have lock and unlock semantics?

atomic_read()

which has no barrier or volatile implications.

atomic_read_for_lock

	Acquire semantics?


atomic_read_for_unlock

	Release semantics?

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Wed, Aug 15, 2007 at 04:38:54AM +0530, Satyam Sharma wrote:
> 
> 
> On Tue, 14 Aug 2007, Christoph Lameter wrote:
> 
> > On Thu, 9 Aug 2007, Chris Snook wrote:
> > 
> > > This patchset makes the behavior of atomic_read uniform by removing the
> > > volatile keyword from all atomic_t and atomic64_t definitions that currently
> > > have it, and instead explicitly casts the variable as volatile in
> > > atomic_read().  This leaves little room for creative optimization by the
> > > compiler, and is in keeping with the principles behind "volatile considered
> > > harmful".
> > 
> > volatile is generally harmful even in atomic_read(). Barriers control
> > visibility and AFAICT things are fine.
> 
> Frankly, I don't see the need for this series myself either. Personal
> opinion (others may differ), but I consider "volatile" to be a sad /
> unfortunate wart in C (numerous threads on this list and on the gcc
> lists/bugzilla over the years stand testimony to this) and if we _can_
> steer clear of it, then why not -- why use this ill-defined primitive
> whose implementation has often differed over compiler versions and
> platforms? Granted, barrier() _is_ heavy-handed in that it makes the
> optimizer forget _everything_, but then somebody did post a forget()
> macro on this thread itself ...
> 
> [ BTW, why do we want the compiler to not optimize atomic_read()'s in
>   the first place? Atomic ops guarantee atomicity, which has nothing
>   to do with "volatility" -- users that expect "volatility" from
>   atomic ops are the ones who must be fixed instead, IMHO. ]

Interactions between mainline code and interrupt/NMI handlers on the same
CPU (for example, when both are using per-CPU variables.  See examples
previously posted in this thread, or look at the rcu_read_lock() and
rcu_read_unlock() implementations in http://lkml.org/lkml/2007/8/7/280.

						Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

Chris Snook <csnook@redhat.com> wrote:
> 
> Because atomic operations are generally used for synchronization, which requires 
> volatile behavior.  Most such codepaths currently use an inefficient barrier(). 
>  Some forget to and we get bugs, because people assume that atomic_read() 
> actually reads something, and atomic_write() actually writes something.  Worse, 
> these are architecture-specific, even compiler version-specific bugs that are 
> often difficult to track down.

I'm yet to see a single example from the current tree where
this patch series is the correct solution.  So far the only
example has been a buggy piece of code which has since been
fixed with a cpu_relax.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Heiko Carstens]

On Wed, Aug 15, 2007 at 02:49:03PM +0800, Herbert Xu wrote:
> Chris Snook <csnook@redhat.com> wrote:
> > 
> > Because atomic operations are generally used for synchronization, which requires 
> > volatile behavior.  Most such codepaths currently use an inefficient barrier(). 
> >  Some forget to and we get bugs, because people assume that atomic_read() 
> > actually reads something, and atomic_write() actually writes something.  Worse, 
> > these are architecture-specific, even compiler version-specific bugs that are 
> > often difficult to track down.
> 
> I'm yet to see a single example from the current tree where
> this patch series is the correct solution.  So far the only
> example has been a buggy piece of code which has since been
> fixed with a cpu_relax.

Btw.: we still have

include/asm-i386/mach-es7000/mach_wakecpu.h:  while (!atomic_read(deassert));
include/asm-i386/mach-default/mach_wakecpu.h: while (!atomic_read(deassert));

Looks like they need to be fixed as well.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Stefan Richter]

Satyam Sharma wrote:
> [ BTW, why do we want the compiler to not optimize atomic_read()'s in
>   the first place? Atomic ops guarantee atomicity, which has nothing
>   to do with "volatility" -- users that expect "volatility" from
>   atomic ops are the ones who must be fixed instead, IMHO. ]

LDD3 says on page 125:  "The following operations are defined for the
type [atomic_t] and are guaranteed to be atomic with respect to all
processors of an SMP computer."

Doesn't "atomic WRT all processors" require volatility?
-- 
Stefan Richter
-=====-=-=== =--- -====
http://arcgraph.de/sr/

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

On Wed, Aug 15, 2007 at 12:35:31PM +0200, Stefan Richter wrote:
> 
> LDD3 says on page 125:  "The following operations are defined for the
> type [atomic_t] and are guaranteed to be atomic with respect to all
> processors of an SMP computer."
> 
> Doesn't "atomic WRT all processors" require volatility?

Not at all.  We also require this to be atomic without any
hint of volatility.

	extern int foo;
	foo = 4;

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Wed, 15 Aug 2007, Stefan Richter wrote:

> Satyam Sharma wrote:
> > [ BTW, why do we want the compiler to not optimize atomic_read()'s in
> >   the first place? Atomic ops guarantee atomicity, which has nothing
> >   to do with "volatility" -- users that expect "volatility" from
> >   atomic ops are the ones who must be fixed instead, IMHO. ]
> 
> LDD3 says on page 125:  "The following operations are defined for the
> type [atomic_t] and are guaranteed to be atomic with respect to all
> processors of an SMP computer."
> 
> Doesn't "atomic WRT all processors" require volatility?

No, it definitely doesn't. Why should it?

"Atomic w.r.t. all processors" is just your normal, simple "atomicity"
for SMP systems (ensure that that object is modified / set / replaced
in main memory atomically) and has nothing to do with "volatile"
behaviour.

"Volatile behaviour" itself isn't consistently defined (at least
definitely not consistently implemented in various gcc versions across
platforms), but it is /expected/ to mean something like: "ensure that
every such access actually goes all the way to memory, and is not
re-ordered w.r.t. to other accesses, as far as the compiler can take
care of these". The last "as far as compiler can take care" disclaimer
comes about due to CPUs doing their own re-ordering nowadays.

For example (say on i386):

(A)
$ cat tp1.c
int a;

void func(void)
{
	a = 10;
	a = 20;
}
$ gcc -Os -S tp1.c
$ cat tp1.s
...
movl    $20, a
...

(B)
$ cat tp2.c
volatile int a;

void func(void)
{
	a = 10;
	a = 20;
}
$ gcc -Os -S tp2.c
$ cat tp2.s
...
movl    $10, a
movl    $20, a
...

(C)
$ cat tp3.c
int a;

void func(void)
{
	*(volatile int *)&a = 10;
	*(volatile int *)&a = 20;
}
$ gcc -Os -S tp3.c
$ cat tp3.s
...
movl    $10, a
movl    $20, a
...

In (A) the compiler optimized "a = 10;" away, but the actual store
of the final value "20" to "a" was still "atomic". (B) and (C) also
exhibit "volatile" behaviour apart from the "atomicity".

But as others replied, it seems some callers out there depend upon
atomic ops exhibiting "volatile" behaviour as well, so that answers
my initial question, actually. I haven't looked at the code Paul
pointed me at, but I wonder if that "forget(x)" macro would help
those cases. I'd wish to avoid the "volatile" primitive, personally.


Satyam

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Stefan Richter]

Satyam Sharma wrote:
> On Wed, 15 Aug 2007, Stefan Richter wrote:
>> Doesn't "atomic WRT all processors" require volatility?
> 
> No, it definitely doesn't. Why should it?
> 
> "Atomic w.r.t. all processors" is just your normal, simple "atomicity"
> for SMP systems (ensure that that object is modified / set / replaced
> in main memory atomically) and has nothing to do with "volatile"
> behaviour.
> 
> "Volatile behaviour" itself isn't consistently defined (at least
> definitely not consistently implemented in various gcc versions across
> platforms), but it is /expected/ to mean something like: "ensure that
> every such access actually goes all the way to memory, and is not
> re-ordered w.r.t. to other accesses, as far as the compiler can take
> care of these". The last "as far as compiler can take care" disclaimer
> comes about due to CPUs doing their own re-ordering nowadays.
> 
> For example (say on i386):

[...]

> In (A) the compiler optimized "a = 10;" away, but the actual store
> of the final value "20" to "a" was still "atomic". (B) and (C) also
> exhibit "volatile" behaviour apart from the "atomicity".
> 
> But as others replied, it seems some callers out there depend upon
> atomic ops exhibiting "volatile" behaviour as well, so that answers
> my initial question, actually. I haven't looked at the code Paul
> pointed me at, but I wonder if that "forget(x)" macro would help
> those cases. I'd wish to avoid the "volatile" primitive, personally.

So, looking at load instead of store, understand I correctly that in
your opinion

	int b;

	b = atomic_read(&a);
	if (b)
		do_something_time_consuming();

	b = atomic_read(&a);
	if (b)
		do_something_more();

should be changed to explicitly forget(&a) after
do_something_time_consuming?

If so, how about the following:

static inline void A(atomic_t *a)
{
	int b = atomic_read(a);
	if (b)
		do_something_time_consuming();
}

static inline void B(atomic_t *a)
{
	int b = atomic_read(a);
	if (b)
		do_something_more();
}

static void C(atomic_t *a)
{
	A(a);
	B(b);
}

Would this need forget(a) after A(a)?

(Is the latter actually answered in C99 or is it compiler-dependent?)
-- 
Stefan Richter
-=====-=-=== =--- -====
http://arcgraph.de/sr/

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Stefan Richter]

I wrote:
> static inline void A(atomic_t *a)
> {
> 	int b = atomic_read(a);
> 	if (b)
> 		do_something_time_consuming();
> }
> 
> static inline void B(atomic_t *a)
> {
> 	int b = atomic_read(a);
> 	if (b)
> 		do_something_more();
> }
> 
> static void C(atomic_t *a)
> {
> 	A(a);
> 	B(b);
	/* ^ typo */
	B(a);
> }
> 
> Would this need forget(a) after A(a)?
> 
> (Is the latter actually answered in C99 or is it compiler-dependent?)


-- 
Stefan Richter
-=====-=-=== =--- -====
http://arcgraph.de/sr/

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Wed, 15 Aug 2007, Stefan Richter wrote:

> Satyam Sharma wrote:
> > On Wed, 15 Aug 2007, Stefan Richter wrote:
> >> Doesn't "atomic WRT all processors" require volatility?
> > 
> > No, it definitely doesn't. Why should it?
> > 
> > "Atomic w.r.t. all processors" is just your normal, simple "atomicity"
> > for SMP systems (ensure that that object is modified / set / replaced
> > in main memory atomically) and has nothing to do with "volatile"
> > behaviour.
> > 
> > "Volatile behaviour" itself isn't consistently defined (at least
> > definitely not consistently implemented in various gcc versions across
> > platforms), but it is /expected/ to mean something like: "ensure that
> > every such access actually goes all the way to memory, and is not
> > re-ordered w.r.t. to other accesses, as far as the compiler can take
> > care of these". The last "as far as compiler can take care" disclaimer
> > comes about due to CPUs doing their own re-ordering nowadays.
> > 
> > For example (say on i386):
> 
> [...]
> 
> > In (A) the compiler optimized "a = 10;" away, but the actual store
> > of the final value "20" to "a" was still "atomic". (B) and (C) also
> > exhibit "volatile" behaviour apart from the "atomicity".
> > 
> > But as others replied, it seems some callers out there depend upon
> > atomic ops exhibiting "volatile" behaviour as well, so that answers
> > my initial question, actually. I haven't looked at the code Paul
> > pointed me at, but I wonder if that "forget(x)" macro would help
> > those cases. I'd wish to avoid the "volatile" primitive, personally.
> 
> So, looking at load instead of store, understand I correctly that in
> your opinion
> 
> 	int b;
> 
> 	b = atomic_read(&a);
> 	if (b)
> 		do_something_time_consuming();
> 
> 	b = atomic_read(&a);
> 	if (b)
> 		do_something_more();
> 
> should be changed to explicitly forget(&a) after
> do_something_time_consuming?

No, I'd actually prefer something like what Christoph Lameter suggested,
i.e. users (such as above) who want "volatile"-like behaviour from atomic
ops can use alternative functions. How about something like:

#define atomic_read_volatile(v)			\
	({					\
		forget(&(v)->counter);		\
		((v)->counter);			\
	})

Or possibly, implement these "volatile" atomic ops variants in inline asm
like the patch that Sebastian Siewior has submitted on another thread just
a while back.

Of course, if we find there are more callers in the kernel who want the
volatility behaviour than those who don't care, we can re-define the
existing ops to such variants, and re-name the existing definitions to
somethine else, say "atomic_read_nonvolatile" for all I care.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Stefan Richter]

On 8/15/2007 10:18 AM, Heiko Carstens wrote:
> On Wed, Aug 15, 2007 at 02:49:03PM +0800, Herbert Xu wrote:
>> Chris Snook <csnook@redhat.com> wrote:
>> > 
>> > Because atomic operations are generally used for synchronization, which requires 
>> > volatile behavior.  Most such codepaths currently use an inefficient barrier(). 
>> >  Some forget to and we get bugs, because people assume that atomic_read() 
>> > actually reads something, and atomic_write() actually writes something.  Worse, 
>> > these are architecture-specific, even compiler version-specific bugs that are 
>> > often difficult to track down.
>> 
>> I'm yet to see a single example from the current tree where
>> this patch series is the correct solution.  So far the only
>> example has been a buggy piece of code which has since been
>> fixed with a cpu_relax.
> 
> Btw.: we still have
> 
> include/asm-i386/mach-es7000/mach_wakecpu.h:  while (!atomic_read(deassert));
> include/asm-i386/mach-default/mach_wakecpu.h: while (!atomic_read(deassert));
> 
> Looks like they need to be fixed as well.


I don't know if this here is affected:

/* drivers/ieee1394/ieee1394_core.h */
static inline unsigned int get_hpsb_generation(struct hpsb_host *host)
{
	return atomic_read(&host->generation);
}

/* drivers/ieee1394/nodemgr.c */
static int nodemgr_host_thread(void *__hi)
{
	[...]

	for (;;) {
		[... sleep until bus reset event ...]

		/* Pause for 1/4 second in 1/16 second intervals,
		 * to make sure things settle down. */
		g = get_hpsb_generation(host);
		for (i = 0; i < 4 ; i++) {
			if (msleep_interruptible(63) ||
			    kthread_should_stop())
				goto exit;

	/* Now get the generation in which the node ID's we collect
	 * are valid.  During the bus scan we will use this generation
	 * for the read transactions, so that if another reset occurs
	 * during the scan the transactions will fail instead of
	 * returning bogus data. */

			generation = get_hpsb_generation(host);

	/* If we get a reset before we are done waiting, then
	 * start the waiting over again */

			if (generation != g)
				g = generation, i = 0;
		}

		[... scan bus, using generation ...]

	}
exit:
[...]
}



-- 
Stefan Richter
-=====-=-=== =--- -====
http://arcgraph.de/sr/

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Wed, Aug 15, 2007 at 07:17:29PM +0530, Satyam Sharma wrote:
> On Wed, 15 Aug 2007, Stefan Richter wrote:
> > Satyam Sharma wrote:
> > > On Wed, 15 Aug 2007, Stefan Richter wrote:
> > >> Doesn't "atomic WRT all processors" require volatility?
> > > 
> > > No, it definitely doesn't. Why should it?
> > > 
> > > "Atomic w.r.t. all processors" is just your normal, simple "atomicity"
> > > for SMP systems (ensure that that object is modified / set / replaced
> > > in main memory atomically) and has nothing to do with "volatile"
> > > behaviour.
> > > 
> > > "Volatile behaviour" itself isn't consistently defined (at least
> > > definitely not consistently implemented in various gcc versions across
> > > platforms), but it is /expected/ to mean something like: "ensure that
> > > every such access actually goes all the way to memory, and is not
> > > re-ordered w.r.t. to other accesses, as far as the compiler can take
> > > care of these". The last "as far as compiler can take care" disclaimer
> > > comes about due to CPUs doing their own re-ordering nowadays.
> > > 
> > > For example (say on i386):
> > 
> > [...]
> > 
> > > In (A) the compiler optimized "a = 10;" away, but the actual store
> > > of the final value "20" to "a" was still "atomic". (B) and (C) also
> > > exhibit "volatile" behaviour apart from the "atomicity".
> > > 
> > > But as others replied, it seems some callers out there depend upon
> > > atomic ops exhibiting "volatile" behaviour as well, so that answers
> > > my initial question, actually. I haven't looked at the code Paul
> > > pointed me at, but I wonder if that "forget(x)" macro would help
> > > those cases. I'd wish to avoid the "volatile" primitive, personally.
> > 
> > So, looking at load instead of store, understand I correctly that in
> > your opinion
> > 
> > 	int b;
> > 
> > 	b = atomic_read(&a);
> > 	if (b)
> > 		do_something_time_consuming();
> > 
> > 	b = atomic_read(&a);
> > 	if (b)
> > 		do_something_more();
> > 
> > should be changed to explicitly forget(&a) after
> > do_something_time_consuming?
> 
> No, I'd actually prefer something like what Christoph Lameter suggested,
> i.e. users (such as above) who want "volatile"-like behaviour from atomic
> ops can use alternative functions. How about something like:
> 
> #define atomic_read_volatile(v)			\
> 	({					\
> 		forget(&(v)->counter);		\
> 		((v)->counter);			\
> 	})

Wouldn't the above "forget" the value, throw it away, then forget
that it forgot it, giving non-volatile semantics?

> Or possibly, implement these "volatile" atomic ops variants in inline asm
> like the patch that Sebastian Siewior has submitted on another thread just
> a while back.

Given that you are advocating a change (please keep in mind that
atomic_read() and atomic_set() had volatile semantics on almost all
platforms), care to give some example where these historical volatile
semantics are causing a problem?

> Of course, if we find there are more callers in the kernel who want the
> volatility behaviour than those who don't care, we can re-define the
> existing ops to such variants, and re-name the existing definitions to
> somethine else, say "atomic_read_nonvolatile" for all I care.

Do we really need another set of APIs?  Can you give even one example
where the pre-existing volatile semantics are causing enough of a problem
to justify adding yet more atomic_*() APIs?

							Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

Hi Stefan,


On Wed, 15 Aug 2007, Stefan Richter wrote:

> On 8/15/2007 10:18 AM, Heiko Carstens wrote:
> > On Wed, Aug 15, 2007 at 02:49:03PM +0800, Herbert Xu wrote:
> >> Chris Snook <csnook@redhat.com> wrote:
> >> > 
> >> > Because atomic operations are generally used for synchronization, which requires 
> >> > volatile behavior.  Most such codepaths currently use an inefficient barrier(). 
> >> >  Some forget to and we get bugs, because people assume that atomic_read() 
> >> > actually reads something, and atomic_write() actually writes something.  Worse, 
> >> > these are architecture-specific, even compiler version-specific bugs that are 
> >> > often difficult to track down.
> >> 
> >> I'm yet to see a single example from the current tree where
> >> this patch series is the correct solution.  So far the only
> >> example has been a buggy piece of code which has since been
> >> fixed with a cpu_relax.
> > 
> > Btw.: we still have
> > 
> > include/asm-i386/mach-es7000/mach_wakecpu.h:  while (!atomic_read(deassert));
> > include/asm-i386/mach-default/mach_wakecpu.h: while (!atomic_read(deassert));
> > 
> > Looks like they need to be fixed as well.
> 
> 
> I don't know if this here is affected:

Yes, I think it is. You're clearly expecting the read to actually happen
when you call get_hpsb_generation(). It's clearly not a busy-loop, so
cpu_relax() sounds pointless / wrong solution for this case, so I'm now
somewhat beginning to appreciate the motivation behind this series :-)

But as I said, there are ways to achieve the same goals of this series
without using "volatile".

I think I'll submit a RFC/patch or two on this myself (will also fix
the code pieces listed here).


> /* drivers/ieee1394/ieee1394_core.h */
> static inline unsigned int get_hpsb_generation(struct hpsb_host *host)
> {
> 	return atomic_read(&host->generation);
> }
> 
> /* drivers/ieee1394/nodemgr.c */
> static int nodemgr_host_thread(void *__hi)
> {
> 	[...]
> 
> 	for (;;) {
> 		[... sleep until bus reset event ...]
> 
> 		/* Pause for 1/4 second in 1/16 second intervals,
> 		 * to make sure things settle down. */
> 		g = get_hpsb_generation(host);
> 		for (i = 0; i < 4 ; i++) {
> 			if (msleep_interruptible(63) ||
> 			    kthread_should_stop())
> 				goto exit;

Totally unrelated, but this looks weird. IMHO you actually wanted:

	msleep_interruptible(63);
	if (kthread_should_stop())
		goto exit;

here, didn't you? Otherwise the thread will exit even when
kthread_should_stop() != TRUE (just because it received a signal),
and it is not good for a kthread to exit on its own if it uses
kthread_should_stop() or if some other piece of kernel code could
eventually call kthread_stop(tsk) on it.

Ok, probably the thread will never receive a signal in the first
place because it's spawned off kthreadd which ignores all signals
beforehand, but still ...

[PATCH] ieee1394: Fix kthread stopping in nodemgr_host_thread

The nodemgr host thread can exit on its own even when kthread_should_stop
is not true, on receiving a signal (might never happen in practice, as
it ignores signals). But considering kthread_stop() must not be mixed with
kthreads that can exit on their own, I think changing the code like this
is clearer. This change means the thread can cut its sleep short when
receive a signal but looking at the code around, that sounds okay (and
again, it might never actually recieve a signal in practice).

Signed-off-by: Satyam Sharma <satyam@infradead.org>

---

 drivers/ieee1394/nodemgr.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/drivers/ieee1394/nodemgr.c b/drivers/ieee1394/nodemgr.c
index 2ffd534..981a7da 100644
--- a/drivers/ieee1394/nodemgr.c
+++ b/drivers/ieee1394/nodemgr.c
@@ -1721,7 +1721,8 @@ static int nodemgr_host_thread(void *__hi)
 		 * to make sure things settle down. */
 		g = get_hpsb_generation(host);
 		for (i = 0; i < 4 ; i++) {
-			if (msleep_interruptible(63) || kthread_should_stop())
+			msleep_interruptible(63);
+			if (kthread_should_stop())
 				goto exit;
 
 			/* Now get the generation in which the node ID's we collect

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

On Wed, Aug 15, 2007 at 08:05:38PM +0530, Satyam Sharma wrote:
>
> > I don't know if this here is affected:
> 
> Yes, I think it is. You're clearly expecting the read to actually happen
> when you call get_hpsb_generation(). It's clearly not a busy-loop, so
> cpu_relax() sounds pointless / wrong solution for this case, so I'm now
> somewhat beginning to appreciate the motivation behind this series :-)

Nope, we're calling schedule which is a rather heavy-weight
barrier.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

On Wed, Aug 15, 2007 at 07:25:16AM -0700, Paul E. McKenney wrote:
> 
> Do we really need another set of APIs?  Can you give even one example
> where the pre-existing volatile semantics are causing enough of a problem
> to justify adding yet more atomic_*() APIs?

Let's turn this around.  Can you give a single example where
the volatile semantics is needed in a legitimate way?

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Wed, Aug 15, 2007 at 11:33:36PM +0800, Herbert Xu wrote:
> On Wed, Aug 15, 2007 at 07:25:16AM -0700, Paul E. McKenney wrote:
> > 
> > Do we really need another set of APIs?  Can you give even one example
> > where the pre-existing volatile semantics are causing enough of a problem
> > to justify adding yet more atomic_*() APIs?
> 
> Let's turn this around.  Can you give a single example where
> the volatile semantics is needed in a legitimate way?

Sorry, but you are the one advocating for the change.

Nice try, though!  ;-)

						Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Stefan Richter]

Herbert Xu wrote:
> On Wed, Aug 15, 2007 at 08:05:38PM +0530, Satyam Sharma wrote:
>>> I don't know if this here is affected:

[...something like]
	b = atomic_read(a);
	for (i = 0; i < 4; i++) {
		msleep_interruptible(63);
		c = atomic_read(a);
		if (c != b) {
			b = c;
			i = 0;
		}
	}

> Nope, we're calling schedule which is a rather heavy-weight
> barrier.

How does the compiler know that msleep() has got barrier()s?
-- 
Stefan Richter
-=====-=-=== =--- -====
http://arcgraph.de/sr/

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Chris Snook]

Herbert Xu wrote:
> Chris Snook <csnook@redhat.com> wrote:
>> Because atomic operations are generally used for synchronization, which requires 
>> volatile behavior.  Most such codepaths currently use an inefficient barrier(). 
>>  Some forget to and we get bugs, because people assume that atomic_read() 
>> actually reads something, and atomic_write() actually writes something.  Worse, 
>> these are architecture-specific, even compiler version-specific bugs that are 
>> often difficult to track down.
> 
> I'm yet to see a single example from the current tree where
> this patch series is the correct solution.  So far the only
> example has been a buggy piece of code which has since been
> fixed with a cpu_relax.

Part of the motivation here is to fix heisenbugs.  If I knew where they 
were, I'd be posting patches for them.  Unlike most bugs, where we want 
to expose them as obviously as possible, these can be extremely 
difficult to track down, and are often due to people assuming that the 
atomic_* operations have the same semantics they've historically had. 
Remember that until recently, all SMP architectures except s390 (which 
very few kernel developers outside of IBM, Red Hat, and SuSE do much 
work on) had volatile declarations for atomic_t.  Removing the volatile 
declarations from i386 and x86_64 may have created heisenbugs that won't 
manifest themselves until GCC 6.0 comes out and people start compiling 
kernels with -O5.  We should have consistent semantics for atomic_* 
operations.

The other motivation is to reduce the need for the barriers used to 
prevent/fix such problems which clobber all your registers, and instead 
force atomic_* operations to behave in the way they're actually used. 
After the (resubmitted) patchset is merged, we'll be able to remove a 
whole bunch of barriers, shrinking our source and our binaries, and 
improving performance.

	-- Chris

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Wed, Aug 15, 2007 at 06:09:35PM +0200, Stefan Richter wrote:
> Herbert Xu wrote:
> > On Wed, Aug 15, 2007 at 08:05:38PM +0530, Satyam Sharma wrote:
> >>> I don't know if this here is affected:
> 
> [...something like]
> 	b = atomic_read(a);
> 	for (i = 0; i < 4; i++) {
> 		msleep_interruptible(63);
> 		c = atomic_read(a);
> 		if (c != b) {
> 			b = c;
> 			i = 0;
> 		}
> 	}
> 
> > Nope, we're calling schedule which is a rather heavy-weight
> > barrier.
> 
> How does the compiler know that msleep() has got barrier()s?

Because msleep_interruptible() is in a separate compilation unit,
the compiler has to assume that it might modify any arbitrary global.
In many cases, the compiler also has to assume that msleep_interruptible()
might call back into a function in the current compilation unit, thus
possibly modifying global static variables.

						Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Wed, 15 Aug 2007, Paul E. McKenney wrote:

> On Wed, Aug 15, 2007 at 06:09:35PM +0200, Stefan Richter wrote:
> > Herbert Xu wrote:
> > > On Wed, Aug 15, 2007 at 08:05:38PM +0530, Satyam Sharma wrote:
> > >>> I don't know if this here is affected:
> > 
> > [...something like]
> > 	b = atomic_read(a);
> > 	for (i = 0; i < 4; i++) {
> > 		msleep_interruptible(63);
> > 		c = atomic_read(a);
> > 		if (c != b) {
> > 			b = c;
> > 			i = 0;
> > 		}
> > 	}
> > 
> > > Nope, we're calling schedule which is a rather heavy-weight
> > > barrier.
> > 
> > How does the compiler know that msleep() has got barrier()s?
> 
> Because msleep_interruptible() is in a separate compilation unit,
> the compiler has to assume that it might modify any arbitrary global.
> In many cases, the compiler also has to assume that msleep_interruptible()
> might call back into a function in the current compilation unit, thus
> possibly modifying global static variables.

Yup, I've just verified this with a testcase. So a call to any function
outside of the current compilation unit acts as a compiler barrier. Cool.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Wed, 15 Aug 2007, Paul E. McKenney wrote:

> On Wed, Aug 15, 2007 at 11:33:36PM +0800, Herbert Xu wrote:
> > On Wed, Aug 15, 2007 at 07:25:16AM -0700, Paul E. McKenney wrote:
> > > 
> > > Do we really need another set of APIs?  Can you give even one example
> > > where the pre-existing volatile semantics are causing enough of a problem
> > > to justify adding yet more atomic_*() APIs?
> > 
> > Let's turn this around.  Can you give a single example where
> > the volatile semantics is needed in a legitimate way?
> 
> Sorry, but you are the one advocating for the change.

Not for i386 and x86_64 -- those have atomic ops without any "volatile"
semantics (currently as per existing definitions).

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Wed, Aug 15, 2007 at 10:48:28PM +0530, Satyam Sharma wrote:
> On Wed, 15 Aug 2007, Paul E. McKenney wrote:
> > On Wed, Aug 15, 2007 at 11:33:36PM +0800, Herbert Xu wrote:
> > > On Wed, Aug 15, 2007 at 07:25:16AM -0700, Paul E. McKenney wrote:
> > > > 
> > > > Do we really need another set of APIs?  Can you give even one example
> > > > where the pre-existing volatile semantics are causing enough of a problem
> > > > to justify adding yet more atomic_*() APIs?
> > > 
> > > Let's turn this around.  Can you give a single example where
> > > the volatile semantics is needed in a legitimate way?
> > 
> > Sorry, but you are the one advocating for the change.
> 
> Not for i386 and x86_64 -- those have atomic ops without any "volatile"
> semantics (currently as per existing definitions).

I claim unit volumes with arm, and the majority of the architectures, but
I cannot deny the popularity of i386 and x86_64 with many developers.  ;-)

However, I am not aware of code in the kernel that would benefit
from the compiler coalescing multiple atomic_set() and atomic_read()
invocations, thus I don't see the downside to volatility in this case.
Are there some performance-critical code fragments that I am missing?

						Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

Hi Paul,


On Wed, 15 Aug 2007, Paul E. McKenney wrote:

> On Wed, Aug 15, 2007 at 07:17:29PM +0530, Satyam Sharma wrote:
> > [...]
> > No, I'd actually prefer something like what Christoph Lameter suggested,
> > i.e. users (such as above) who want "volatile"-like behaviour from atomic
> > ops can use alternative functions. How about something like:
> > 
> > #define atomic_read_volatile(v)			\
> > 	({					\
> > 		forget(&(v)->counter);		\
> > 		((v)->counter);			\
> > 	})
> 
> Wouldn't the above "forget" the value, throw it away, then forget
> that it forgot it, giving non-volatile semantics?

Nope, I don't think so. I wrote the following trivial testcases:
[ See especially tp4.c and tp4.s (last example). ]

==============================================================================
$ cat tp1.c # Using volatile access casts

#define atomic_read(a)	(*(volatile int *)&a)

int a;

void func(void)
{
	a = 0;
	while (atomic_read(a))
		;
}
==============================================================================
$ gcc -Os -S tp1.c; cat tp1.s

func:
	pushl	%ebp
	movl	%esp, %ebp
	movl	$0, a
.L2:
	movl	a, %eax
	testl	%eax, %eax
	jne	.L2
	popl	%ebp
	ret
==============================================================================
$ cat tp2.c # Using nothing; gcc will optimize the whole loop away

#define forget(x)

#define atomic_read(a)		\
	({			\
		forget(&(a));	\
		(a);		\
	})

int a;

void func(void)
{
	a = 0;
	while (atomic_read(a))
		;
}
==============================================================================
$ gcc -Os -S tp2.c; cat tp2.s

func:
	pushl	%ebp
	movl	%esp, %ebp
	popl	%ebp
	movl	$0, a
	ret
==============================================================================
$ cat tp3.c # Using a full memory clobber barrier

#define forget(x)	asm volatile ("":::"memory")

#define atomic_read(a)		\
	({			\
		forget(&(a));	\
		(a);		\
	})

int a;

void func(void)
{
	a = 0;
	while (atomic_read(a))
		;
}
==============================================================================
$ gcc -Os -S tp3.c; cat tp3.s

func:
	pushl	%ebp
	movl	%esp, %ebp
	movl	$0, a
.L2:
	cmpl	$0, a
	jne	.L2
	popl	%ebp
	ret
==============================================================================
$ cat tp4.c # Using a forget(var) macro

#define forget(a)	__asm__ __volatile__ ("" :"=m" (a) :"m" (a))

#define atomic_read(a)		\
	({			\
		forget(a);	\
		(a);		\
	})

int a;

void func(void)
{
	a = 0;
	while (atomic_read(a))
		;
}
==============================================================================
$ gcc -Os -S tp4.c; cat tp4.s

func:
	pushl	%ebp
	movl	%esp, %ebp
	movl	$0, a
.L2:
	cmpl	$0, a
	jne	.L2
	popl	%ebp
	ret
==============================================================================


Possibly these were too trivial to expose any potential problems that you
may have been referring to, so would be helpful if you could write a more
concrete example / sample code.


> > Or possibly, implement these "volatile" atomic ops variants in inline asm
> > like the patch that Sebastian Siewior has submitted on another thread just
> > a while back.
> 
> Given that you are advocating a change (please keep in mind that
> atomic_read() and atomic_set() had volatile semantics on almost all
> platforms), care to give some example where these historical volatile
> semantics are causing a problem?
> [...]
> Can you give even one example
> where the pre-existing volatile semantics are causing enough of a problem
> to justify adding yet more atomic_*() APIs?

Will take this to the other sub-thread ...


> > Of course, if we find there are more callers in the kernel who want the
> > volatility behaviour than those who don't care, we can re-define the
> > existing ops to such variants, and re-name the existing definitions to
> > somethine else, say "atomic_read_nonvolatile" for all I care.
> 
> Do we really need another set of APIs?

Well, if there's one set of users who do care about volatile behaviour,
and another set that doesn't, it only sounds correct to provide both
those APIs, instead of forcing one behaviour on all users.


Thanks,
Satyam

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Wed, 15 Aug 2007, Paul E. McKenney wrote:

> On Wed, Aug 15, 2007 at 10:48:28PM +0530, Satyam Sharma wrote:
> > [...]
> > Not for i386 and x86_64 -- those have atomic ops without any "volatile"
> > semantics (currently as per existing definitions).
> 
> I claim unit volumes with arm, and the majority of the architectures, but
> I cannot deny the popularity of i386 and x86_64 with many developers.  ;-)

Hmm, does arm really need that "volatile int counter;"? Hopefully RMK will
take a patch removing that "volatile" ... ;-)


> However, I am not aware of code in the kernel that would benefit
> from the compiler coalescing multiple atomic_set() and atomic_read()
> invocations, thus I don't see the downside to volatility in this case.
> Are there some performance-critical code fragments that I am missing?

I don't know, and yes, code with multiple atomic_set's and atomic_read's
getting optimized or coalesced does sound strange to start with. Anyway,
I'm not against "volatile semantics" per se. As replied elsewhere, I do
appreciate the motivation behind this series (to _avoid_ gotchas, not to
fix existing ones). Just that I'd like to avoid using "volatile", for
aforementioned reasons, especially given that there are perfectly
reasonable alternatives to achieve the same desired behaviour.


Satyam

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[David Howells]

Herbert Xu <herbert@gondor.apana.org.au> wrote:

> Let's turn this around.  Can you give a single example where
> the volatile semantics is needed in a legitimate way?

Accessing H/W registers?  But apart from that...

David

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

>> How does the compiler know that msleep() has got barrier()s?
>
> Because msleep_interruptible() is in a separate compilation unit,
> the compiler has to assume that it might modify any arbitrary global.

No; compilation units have nothing to do with it, GCC can optimise
across compilation unit boundaries just fine, if you tell it to
compile more than one compilation unit at once.

What you probably mean is that the compiler has to assume any code
it cannot currently see can do anything (insofar as allowed by the
relevant standards etc.)

> In many cases, the compiler also has to assume that 
> msleep_interruptible()
> might call back into a function in the current compilation unit, thus
> possibly modifying global static variables.

It most often is smart enough to see what compilation-unit-local
variables might be modified that way, though :-)


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

> "Volatile behaviour" itself isn't consistently defined (at least
> definitely not consistently implemented in various gcc versions across
> platforms),

It should be consistent across platforms; if not, file a bug please.

> but it is /expected/ to mean something like: "ensure that
> every such access actually goes all the way to memory, and is not
> re-ordered w.r.t. to other accesses, as far as the compiler can take
> care of these". The last "as far as compiler can take care" disclaimer
> comes about due to CPUs doing their own re-ordering nowadays.

You can *expect* whatever you want, but this isn't in line with
reality at all.

volatile _does not_ make accesses go all the way to memory.
volatile _does not_ prevent reordering wrt other accesses.

What volatile does are a) never optimise away a read (or write)
to the object, since the data can change in ways the compiler
cannot see; and b) never move stores to the object across a
sequence point.  This does not mean other accesses cannot be
reordered wrt the volatile access.

If the abstract machine would do an access to a volatile-
qualified object, the generated machine code will do that
access too.  But, for example, it can still be optimised
away by the compiler, if it can prove it is allowed to.

If you want stuff to go all the way to memory, you need some
architecture-specific flush sequence; to make a store globally
visible before another store, you need mb(); before some following
read, you need mb(); to prevent reordering you need a barrier.


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Wed, Aug 15, 2007 at 07:19:57PM +0100, David Howells wrote:
> Herbert Xu <herbert@gondor.apana.org.au> wrote:
> 
> > Let's turn this around.  Can you give a single example where
> > the volatile semantics is needed in a legitimate way?
> 
> Accessing H/W registers?  But apart from that...

Communicating between process context and interrupt/NMI handlers using
per-CPU variables.

							Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Wed, Aug 15, 2007 at 08:31:25PM +0200, Segher Boessenkool wrote:
> >>How does the compiler know that msleep() has got barrier()s?
> >
> >Because msleep_interruptible() is in a separate compilation unit,
> >the compiler has to assume that it might modify any arbitrary global.
> 
> No; compilation units have nothing to do with it, GCC can optimise
> across compilation unit boundaries just fine, if you tell it to
> compile more than one compilation unit at once.

Last I checked, the Linux kernel build system did compile each .c file
as a separate compilation unit.

> What you probably mean is that the compiler has to assume any code
> it cannot currently see can do anything (insofar as allowed by the
> relevant standards etc.)

Indeed.

> >In many cases, the compiler also has to assume that 
> >msleep_interruptible()
> >might call back into a function in the current compilation unit, thus
> >possibly modifying global static variables.
> 
> It most often is smart enough to see what compilation-unit-local
> variables might be modified that way, though :-)

Yep.  For example, if it knows the current value of a given such local
variable, and if all code paths that would change some other variable
cannot be reached given that current value of the first variable.
At least given that gcc doesn't know about multiple threads of execution!

							Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Wed, Aug 15, 2007 at 11:25:05PM +0530, Satyam Sharma wrote:
> Hi Paul,
> On Wed, 15 Aug 2007, Paul E. McKenney wrote:
> 
> > On Wed, Aug 15, 2007 at 07:17:29PM +0530, Satyam Sharma wrote:
> > > [...]
> > > No, I'd actually prefer something like what Christoph Lameter suggested,
> > > i.e. users (such as above) who want "volatile"-like behaviour from atomic
> > > ops can use alternative functions. How about something like:
> > > 
> > > #define atomic_read_volatile(v)			\
> > > 	({					\
> > > 		forget(&(v)->counter);		\
> > > 		((v)->counter);			\
> > > 	})
> > 
> > Wouldn't the above "forget" the value, throw it away, then forget
> > that it forgot it, giving non-volatile semantics?
> 
> Nope, I don't think so. I wrote the following trivial testcases:
> [ See especially tp4.c and tp4.s (last example). ]

Right.  I should have said "wouldn't the compiler be within its rights
to forget the value, throw it away, then forget that it forgot it".
The value coming out of the #define above is an unadorned ((v)->counter),
which has no volatile semantics.

> ==============================================================================
> $ cat tp1.c # Using volatile access casts
> 
> #define atomic_read(a)	(*(volatile int *)&a)
> 
> int a;
> 
> void func(void)
> {
> 	a = 0;
> 	while (atomic_read(a))
> 		;
> }
> ==============================================================================
> $ gcc -Os -S tp1.c; cat tp1.s
> 
> func:
> 	pushl	%ebp
> 	movl	%esp, %ebp
> 	movl	$0, a
> .L2:
> 	movl	a, %eax
> 	testl	%eax, %eax
> 	jne	.L2
> 	popl	%ebp
> 	ret
> ==============================================================================
> $ cat tp2.c # Using nothing; gcc will optimize the whole loop away
> 
> #define forget(x)
> 
> #define atomic_read(a)		\
> 	({			\
> 		forget(&(a));	\
> 		(a);		\
> 	})
> 
> int a;
> 
> void func(void)
> {
> 	a = 0;
> 	while (atomic_read(a))
> 		;
> }
> ==============================================================================
> $ gcc -Os -S tp2.c; cat tp2.s
> 
> func:
> 	pushl	%ebp
> 	movl	%esp, %ebp
> 	popl	%ebp
> 	movl	$0, a
> 	ret
> ==============================================================================
> $ cat tp3.c # Using a full memory clobber barrier
> 
> #define forget(x)	asm volatile ("":::"memory")
> 
> #define atomic_read(a)		\
> 	({			\
> 		forget(&(a));	\
> 		(a);		\
> 	})
> 
> int a;
> 
> void func(void)
> {
> 	a = 0;
> 	while (atomic_read(a))
> 		;
> }
> ==============================================================================
> $ gcc -Os -S tp3.c; cat tp3.s
> 
> func:
> 	pushl	%ebp
> 	movl	%esp, %ebp
> 	movl	$0, a
> .L2:
> 	cmpl	$0, a
> 	jne	.L2
> 	popl	%ebp
> 	ret
> ==============================================================================
> $ cat tp4.c # Using a forget(var) macro
> 
> #define forget(a)	__asm__ __volatile__ ("" :"=m" (a) :"m" (a))
> 
> #define atomic_read(a)		\
> 	({			\
> 		forget(a);	\
> 		(a);		\
> 	})
> 
> int a;
> 
> void func(void)
> {
> 	a = 0;
> 	while (atomic_read(a))
> 		;
> }
> ==============================================================================
> $ gcc -Os -S tp4.c; cat tp4.s
> 
> func:
> 	pushl	%ebp
> 	movl	%esp, %ebp
> 	movl	$0, a
> .L2:
> 	cmpl	$0, a
> 	jne	.L2
> 	popl	%ebp
> 	ret
> ==============================================================================
> 
> Possibly these were too trivial to expose any potential problems that you
> may have been referring to, so would be helpful if you could write a more
> concrete example / sample code.

The trick is to have a sufficiently complicated expression to force
the compiler to run out of registers.  If the value is non-volatile,
it will refetch it (and expect it not to have changed, possibly being
disappointed by an interrupt handler running on that same CPU).

> > > Or possibly, implement these "volatile" atomic ops variants in inline asm
> > > like the patch that Sebastian Siewior has submitted on another thread just
> > > a while back.
> > 
> > Given that you are advocating a change (please keep in mind that
> > atomic_read() and atomic_set() had volatile semantics on almost all
> > platforms), care to give some example where these historical volatile
> > semantics are causing a problem?
> > [...]
> > Can you give even one example
> > where the pre-existing volatile semantics are causing enough of a problem
> > to justify adding yet more atomic_*() APIs?
> 
> Will take this to the other sub-thread ...

OK.

> > > Of course, if we find there are more callers in the kernel who want the
> > > volatility behaviour than those who don't care, we can re-define the
> > > existing ops to such variants, and re-name the existing definitions to
> > > somethine else, say "atomic_read_nonvolatile" for all I care.
> > 
> > Do we really need another set of APIs?
> 
> Well, if there's one set of users who do care about volatile behaviour,
> and another set that doesn't, it only sounds correct to provide both
> those APIs, instead of forcing one behaviour on all users.

Well, if the second set doesn't care, they should be OK with the volatile
behavior in this case.

							Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Wed, 15 Aug 2007, Segher Boessenkool wrote:

> > "Volatile behaviour" itself isn't consistently defined (at least
> > definitely not consistently implemented in various gcc versions across
> > platforms),
> 
> It should be consistent across platforms; if not, file a bug please.
> 
> > but it is /expected/ to mean something like: "ensure that
> > every such access actually goes all the way to memory, and is not
> > re-ordered w.r.t. to other accesses, as far as the compiler can take
                              ^
                              (volatile)

(or, alternatively, "other accesses to the same volatile object" ...)

> > care of these". The last "as far as compiler can take care" disclaimer
> > comes about due to CPUs doing their own re-ordering nowadays.
> 
> You can *expect* whatever you want, but this isn't in line with
> reality at all.
> 
> volatile _does not_ prevent reordering wrt other accesses.
> [...]
> What volatile does are a) never optimise away a read (or write)
> to the object, since the data can change in ways the compiler
> cannot see; and b) never move stores to the object across a
> sequence point.  This does not mean other accesses cannot be
> reordered wrt the volatile access.
> 
> If the abstract machine would do an access to a volatile-
> qualified object, the generated machine code will do that
> access too.  But, for example, it can still be optimised
> away by the compiler, if it can prove it is allowed to.

As (now) indicated above, I had meant multiple volatile accesses to
the same object, obviously.

BTW:

#define atomic_read(a)	(*(volatile int *)&(a))
#define atomic_set(a,i)	(*(volatile int *)&(a) = (i))

int a;

void func(void)
{
	int b;

	b = atomic_read(a);
	atomic_set(a, 20);
	b = atomic_read(a);
}

gives:

func:
	pushl	%ebp
	movl	a, %eax
	movl	%esp, %ebp
	movl	$20, a
	movl	a, %eax
	popl	%ebp
	ret

so the first atomic_read() wasn't optimized away.


> volatile _does not_ make accesses go all the way to memory.
> [...]
> If you want stuff to go all the way to memory, you need some
> architecture-specific flush sequence; to make a store globally
> visible before another store, you need mb(); before some following
> read, you need mb(); to prevent reordering you need a barrier.

Sure, which explains the "as far as the compiler can take care" bit.
Poor phrase / choice of words, probably.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

[ The Cc: list scares me. Should probably trim it. ]


On Wed, 15 Aug 2007, Paul E. McKenney wrote:

> On Wed, Aug 15, 2007 at 08:31:25PM +0200, Segher Boessenkool wrote:
> > >>How does the compiler know that msleep() has got barrier()s?
> > >
> > >Because msleep_interruptible() is in a separate compilation unit,
> > >the compiler has to assume that it might modify any arbitrary global.
> > 
> > No; compilation units have nothing to do with it, GCC can optimise
> > across compilation unit boundaries just fine, if you tell it to
> > compile more than one compilation unit at once.
> 
> Last I checked, the Linux kernel build system did compile each .c file
> as a separate compilation unit.
> 
> > What you probably mean is that the compiler has to assume any code
> > it cannot currently see can do anything (insofar as allowed by the
> > relevant standards etc.)

I think this was just terminology confusion here again. Isn't "any code
that it cannot currently see" the same as "another compilation unit",
and wouldn't the "compilation unit" itself expand if we ask gcc to
compile more than one unit at once? Or is there some more specific
"definition" for "compilation unit" (in gcc lingo, possibly?)

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Christoph Lameter]

On Wed, 15 Aug 2007, Stefan Richter wrote:

> LDD3 says on page 125:  "The following operations are defined for the
> type [atomic_t] and are guaranteed to be atomic with respect to all
> processors of an SMP computer."
> 
> Doesn't "atomic WRT all processors" require volatility?

Atomic operations only require exclusive access to the cacheline while the 
value is modified.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Thu, Aug 16, 2007 at 01:24:42AM +0530, Satyam Sharma wrote:
> [ The Cc: list scares me. Should probably trim it. ]

Trim away!  ;-)

> On Wed, 15 Aug 2007, Paul E. McKenney wrote:
> 
> > On Wed, Aug 15, 2007 at 08:31:25PM +0200, Segher Boessenkool wrote:
> > > >>How does the compiler know that msleep() has got barrier()s?
> > > >
> > > >Because msleep_interruptible() is in a separate compilation unit,
> > > >the compiler has to assume that it might modify any arbitrary global.
> > > 
> > > No; compilation units have nothing to do with it, GCC can optimise
> > > across compilation unit boundaries just fine, if you tell it to
> > > compile more than one compilation unit at once.
> > 
> > Last I checked, the Linux kernel build system did compile each .c file
> > as a separate compilation unit.
> > 
> > > What you probably mean is that the compiler has to assume any code
> > > it cannot currently see can do anything (insofar as allowed by the
> > > relevant standards etc.)
> 
> I think this was just terminology confusion here again. Isn't "any code
> that it cannot currently see" the same as "another compilation unit",
> and wouldn't the "compilation unit" itself expand if we ask gcc to
> compile more than one unit at once? Or is there some more specific
> "definition" for "compilation unit" (in gcc lingo, possibly?)

This is indeed my understanding -- "compilation unit" is whatever the
compiler looks at in one go.  I have heard the word "module" used for
the minimal compilation unit covering a single .c file and everything
that it #includes, but there might be a better name for this.

							Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

>> What volatile does are a) never optimise away a read (or write)
>> to the object, since the data can change in ways the compiler
>> cannot see; and b) never move stores to the object across a
>> sequence point.  This does not mean other accesses cannot be
>> reordered wrt the volatile access.
>>
>> If the abstract machine would do an access to a volatile-
>> qualified object, the generated machine code will do that
>> access too.  But, for example, it can still be optimised
>> away by the compiler, if it can prove it is allowed to.
>
> As (now) indicated above, I had meant multiple volatile accesses to
> the same object, obviously.

Yes, accesses to volatile objects are never reordered with
respect to each other.

> BTW:
>
> #define atomic_read(a)	(*(volatile int *)&(a))
> #define atomic_set(a,i)	(*(volatile int *)&(a) = (i))
>
> int a;
>
> void func(void)
> {
> 	int b;
>
> 	b = atomic_read(a);
> 	atomic_set(a, 20);
> 	b = atomic_read(a);
> }
>
> gives:
>
> func:
> 	pushl	%ebp
> 	movl	a, %eax
> 	movl	%esp, %ebp
> 	movl	$20, a
> 	movl	a, %eax
> 	popl	%ebp
> 	ret
>
> so the first atomic_read() wasn't optimized away.

Of course.  It is executed by the abstract machine, so
it will be executed by the actual machine.  On the other
hand, try

	b = 0;
	if (b)
		b = atomic_read(a);

or similar.


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

>>> What you probably mean is that the compiler has to assume any code
>>> it cannot currently see can do anything (insofar as allowed by the
>>> relevant standards etc.)
>
> I think this was just terminology confusion here again. Isn't "any code
> that it cannot currently see" the same as "another compilation unit",

It is not; try  gcc -combine  or the upcoming link-time optimisation
stuff, for example.

> and wouldn't the "compilation unit" itself expand if we ask gcc to
> compile more than one unit at once? Or is there some more specific
> "definition" for "compilation unit" (in gcc lingo, possibly?)

"compilation unit" is a C standard term.  It typically boils down
to "single .c file".


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

>> I think this was just terminology confusion here again. Isn't "any 
>> code
>> that it cannot currently see" the same as "another compilation unit",
>> and wouldn't the "compilation unit" itself expand if we ask gcc to
>> compile more than one unit at once? Or is there some more specific
>> "definition" for "compilation unit" (in gcc lingo, possibly?)
>
> This is indeed my understanding -- "compilation unit" is whatever the
> compiler looks at in one go.  I have heard the word "module" used for
> the minimal compilation unit covering a single .c file and everything
> that it #includes, but there might be a better name for this.

Yes, that's what's called "compilation unit" :-)

[/me double checks]

Erm, the C standard actually calls it "translation unit".

To be exact, to avoid any more confusion:

5.1.1.1/1:
A C program need not all be translated at the same time. The
text of the program is kept in units called source files, (or
preprocessing files) in this International Standard. A source
file together with all the headers and source files included
via the preprocessing directive #include is known as a
preprocessing translation unit. After preprocessing, a
preprocessing translation unit is called a translation unit.



Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

>>> Of course, if we find there are more callers in the kernel who want 
>>> the
>>> volatility behaviour than those who don't care, we can re-define the
>>> existing ops to such variants, and re-name the existing definitions 
>>> to
>>> somethine else, say "atomic_read_nonvolatile" for all I care.
>>
>> Do we really need another set of APIs?
>
> Well, if there's one set of users who do care about volatile behaviour,
> and another set that doesn't, it only sounds correct to provide both
> those APIs, instead of forcing one behaviour on all users.

But since there currently is only one such API, and there are
users expecting the stronger behaviour, the only sane thing to
do is let the API provide that behaviour.  You can always add
a new API with weaker behaviour later, and move users that are
okay with it over to that new API.


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

>> No; compilation units have nothing to do with it, GCC can optimise
>> across compilation unit boundaries just fine, if you tell it to
>> compile more than one compilation unit at once.
>
> Last I checked, the Linux kernel build system did compile each .c file
> as a separate compilation unit.

I have some patches to use -combine -fwhole-program for Linux.
Highly experimental, you need a patched bleeding edge toolchain.
If there's interest I'll clean it up and put it online.

David Woodhouse had some similar patches about a year ago.

>>> In many cases, the compiler also has to assume that
>>> msleep_interruptible()
>>> might call back into a function in the current compilation unit, thus
>>> possibly modifying global static variables.
>>
>> It most often is smart enough to see what compilation-unit-local
>> variables might be modified that way, though :-)
>
> Yep.  For example, if it knows the current value of a given such local
> variable, and if all code paths that would change some other variable
> cannot be reached given that current value of the first variable.

Or the most common thing: if neither the address of the translation-
unit local variable nor the address of any function writing to that
variable can "escape" from that translation unit, nothing outside
the translation unit can write to the variable.

> At least given that gcc doesn't know about multiple threads of 
> execution!

Heh, only about the threads it creates itself (not relevant to
the kernel, for sure :-) )


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

>> Possibly these were too trivial to expose any potential problems that 
>> you
>> may have been referring to, so would be helpful if you could write a 
>> more
>> concrete example / sample code.
>
> The trick is to have a sufficiently complicated expression to force
> the compiler to run out of registers.

You can use -ffixed-XXX to keep the testcase simple.


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Wed, Aug 15, 2007 at 10:52:53PM +0200, Segher Boessenkool wrote:
> >>I think this was just terminology confusion here again. Isn't "any 
> >>code
> >>that it cannot currently see" the same as "another compilation unit",
> >>and wouldn't the "compilation unit" itself expand if we ask gcc to
> >>compile more than one unit at once? Or is there some more specific
> >>"definition" for "compilation unit" (in gcc lingo, possibly?)
> >
> >This is indeed my understanding -- "compilation unit" is whatever the
> >compiler looks at in one go.  I have heard the word "module" used for
> >the minimal compilation unit covering a single .c file and everything
> >that it #includes, but there might be a better name for this.
> 
> Yes, that's what's called "compilation unit" :-)
> 
> [/me double checks]
> 
> Erm, the C standard actually calls it "translation unit".
> 
> To be exact, to avoid any more confusion:
> 
> 5.1.1.1/1:
> A C program need not all be translated at the same time. The
> text of the program is kept in units called source files, (or
> preprocessing files) in this International Standard. A source
> file together with all the headers and source files included
> via the preprocessing directive #include is known as a
> preprocessing translation unit. After preprocessing, a
> preprocessing translation unit is called a translation unit.

I am OK with "translation" and "compilation" being near-synonyms.  ;-)

						Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Wed, Aug 15, 2007 at 11:05:35PM +0200, Segher Boessenkool wrote:
> >>No; compilation units have nothing to do with it, GCC can optimise
> >>across compilation unit boundaries just fine, if you tell it to
> >>compile more than one compilation unit at once.
> >
> >Last I checked, the Linux kernel build system did compile each .c file
> >as a separate compilation unit.
> 
> I have some patches to use -combine -fwhole-program for Linux.
> Highly experimental, you need a patched bleeding edge toolchain.
> If there's interest I'll clean it up and put it online.
> 
> David Woodhouse had some similar patches about a year ago.

Sounds exciting...  ;-)

> >>>In many cases, the compiler also has to assume that
> >>>msleep_interruptible()
> >>>might call back into a function in the current compilation unit, thus
> >>>possibly modifying global static variables.
> >>
> >>It most often is smart enough to see what compilation-unit-local
> >>variables might be modified that way, though :-)
> >
> >Yep.  For example, if it knows the current value of a given such local
> >variable, and if all code paths that would change some other variable
> >cannot be reached given that current value of the first variable.
> 
> Or the most common thing: if neither the address of the translation-
> unit local variable nor the address of any function writing to that
> variable can "escape" from that translation unit, nothing outside
> the translation unit can write to the variable.

But there is usually at least one externally callable function in
a .c file.

> >At least given that gcc doesn't know about multiple threads of 
> >execution!
> 
> Heh, only about the threads it creates itself (not relevant to
> the kernel, for sure :-) )

;-)

							Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul Mackerras]

Satyam Sharma writes:

> > Doesn't "atomic WRT all processors" require volatility?
> 
> No, it definitely doesn't. Why should it?
> 
> "Atomic w.r.t. all processors" is just your normal, simple "atomicity"
> for SMP systems (ensure that that object is modified / set / replaced
> in main memory atomically) and has nothing to do with "volatile"
> behaviour.

Atomic variables are "volatile" in the sense that they are liable to
be changed at any time by mechanisms that are outside the knowledge of
the C compiler, namely, other CPUs, or this CPU executing an interrupt
routine.

In the kernel we use atomic variables in precisely those situations
where a variable is potentially accessed concurrently by multiple
CPUs, and where each CPU needs to see updates done by other CPUs in a
timely fashion.  That is what they are for.  Therefore the compiler
must not cache values of atomic variables in registers; each
atomic_read must result in a load and each atomic_set must result in a
store.  Anything else will just lead to subtle bugs.

I have no strong opinion about whether or not the best way to achieve
this is through the use of the "volatile" C keyword.  Segher's idea of
using asm instead seems like a good one to me.

Paul.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

On Wed, Aug 15, 2007 at 12:13:12PM -0400, Chris Snook wrote:
>
> Part of the motivation here is to fix heisenbugs.  If I knew where they 

By the same token we should probably disable optimisations
altogether since that too can create heisenbugs.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

On Wed, Aug 15, 2007 at 11:45:20AM -0700, Paul E. McKenney wrote:
> On Wed, Aug 15, 2007 at 07:19:57PM +0100, David Howells wrote:
> > Herbert Xu <herbert@gondor.apana.org.au> wrote:
> > 
> > > Let's turn this around.  Can you give a single example where
> > > the volatile semantics is needed in a legitimate way?
> > 
> > Accessing H/W registers?  But apart from that...
> 
> Communicating between process context and interrupt/NMI handlers using
> per-CPU variables.

Remeber we're talking about atomic_read/atomic_set.  Please
cite the actual file/function name you have in mind.

Thanks,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Thu, Aug 16, 2007 at 07:40:21AM +0800, Herbert Xu wrote:
> On Wed, Aug 15, 2007 at 12:13:12PM -0400, Chris Snook wrote:
> >
> > Part of the motivation here is to fix heisenbugs.  If I knew where they 
> 
> By the same token we should probably disable optimisations
> altogether since that too can create heisenbugs.

Precisely the point -- use of volatile (whether in casts or on asms)
in these cases are intended to disable those optimizations likely to
result in heisenbugs.  But they are also intended to leave other
valuable optimizations in force.

						Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Thu, Aug 16, 2007 at 07:41:46AM +0800, Herbert Xu wrote:
> On Wed, Aug 15, 2007 at 11:45:20AM -0700, Paul E. McKenney wrote:
> > On Wed, Aug 15, 2007 at 07:19:57PM +0100, David Howells wrote:
> > > Herbert Xu <herbert@gondor.apana.org.au> wrote:
> > > 
> > > > Let's turn this around.  Can you give a single example where
> > > > the volatile semantics is needed in a legitimate way?
> > > 
> > > Accessing H/W registers?  But apart from that...
> > 
> > Communicating between process context and interrupt/NMI handlers using
> > per-CPU variables.
> 
> Remeber we're talking about atomic_read/atomic_set.  Please
> cite the actual file/function name you have in mind.

Yep, we are indeed talking about atomic_read()/atomic_set().

We have been through this issue already in this thread.

							Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

On Wed, Aug 15, 2007 at 04:53:35PM -0700, Paul E. McKenney wrote:
>
> > > Communicating between process context and interrupt/NMI handlers using
> > > per-CPU variables.
> > 
> > Remeber we're talking about atomic_read/atomic_set.  Please
> > cite the actual file/function name you have in mind.
> 
> Yep, we are indeed talking about atomic_read()/atomic_set().
> 
> We have been through this issue already in this thread.

Sorry, but I must've missed it.  Could you cite the file or
function for my benefit?

Thanks,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Thu, Aug 16, 2007 at 08:12:48AM +0800, Herbert Xu wrote:
> On Wed, Aug 15, 2007 at 04:53:35PM -0700, Paul E. McKenney wrote:
> >
> > > > Communicating between process context and interrupt/NMI handlers using
> > > > per-CPU variables.
> > > 
> > > Remeber we're talking about atomic_read/atomic_set.  Please
> > > cite the actual file/function name you have in mind.
> > 
> > Yep, we are indeed talking about atomic_read()/atomic_set().
> > 
> > We have been through this issue already in this thread.
> 
> Sorry, but I must've missed it.  Could you cite the file or
> function for my benefit?

I might summarize the thread if there is interest, but I am not able to
do so right this minute.

						Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Christoph Lameter]

On Thu, 16 Aug 2007, Paul Mackerras wrote:

> In the kernel we use atomic variables in precisely those situations
> where a variable is potentially accessed concurrently by multiple
> CPUs, and where each CPU needs to see updates done by other CPUs in a
> timely fashion.  That is what they are for.  Therefore the compiler
> must not cache values of atomic variables in registers; each
> atomic_read must result in a load and each atomic_set must result in a
> store.  Anything else will just lead to subtle bugs.

This may have been the intend. However, today the visibility is controlled 
using barriers. And we have barriers that we use with atomic operations. 
Having volatile be the default just lead to confusion. Atomic read should 
just read with no extras. Extras can be added by using variants like 
atomic_read_volatile or so.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

On Wed, Aug 15, 2007 at 05:23:10PM -0700, Paul E. McKenney wrote:
> On Thu, Aug 16, 2007 at 08:12:48AM +0800, Herbert Xu wrote:
> > On Wed, Aug 15, 2007 at 04:53:35PM -0700, Paul E. McKenney wrote:
> > >
> > > > > Communicating between process context and interrupt/NMI handlers using
> > > > > per-CPU variables.
> > > > 
> > > > Remeber we're talking about atomic_read/atomic_set.  Please
> > > > cite the actual file/function name you have in mind.
> > > 
> > > Yep, we are indeed talking about atomic_read()/atomic_set().
> > > 
> > > We have been through this issue already in this thread.
> > 
> > Sorry, but I must've missed it.  Could you cite the file or
> > function for my benefit?
> 
> I might summarize the thread if there is interest, but I am not able to
> do so right this minute.

Thanks.  But I don't need a summary of the thread, I'm asking
for an extant code snippet in our kernel that benefits from
the volatile change and is not part of a busy-wait.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: your mail

[Herbert Xu]

On Thu, Aug 16, 2007 at 06:06:00AM +0530, Satyam Sharma wrote:
> 
> that are:
> 
> 	while ((atomic_read(&waiting_for_crash_ipi) > 0) && msecs) {
> 		mdelay(1);
> 		msecs--;
> 	}
> 
> where mdelay() becomes __const_udelay() which happens to be in another
> translation unit (arch/i386/lib/delay.c) and hence saves this callsite
> from being a bug :-)

The udelay itself certainly should have some form of cpu_relax in it.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul Mackerras]

Christoph Lameter writes:

> On Thu, 16 Aug 2007, Paul Mackerras wrote:
> 
> > In the kernel we use atomic variables in precisely those situations
> > where a variable is potentially accessed concurrently by multiple
> > CPUs, and where each CPU needs to see updates done by other CPUs in a
> > timely fashion.  That is what they are for.  Therefore the compiler
> > must not cache values of atomic variables in registers; each
> > atomic_read must result in a load and each atomic_set must result in a
> > store.  Anything else will just lead to subtle bugs.
> 
> This may have been the intend. However, today the visibility is controlled 
> using barriers. And we have barriers that we use with atomic operations. 

Those barriers are for when we need ordering between atomic variables
and other memory locations.  An atomic variable by itself doesn't and
shouldn't need any barriers for other CPUs to be able to see what's
happening to it.

Paul.

(unknown)

[Satyam Sharma]

On Wed, 15 Aug 2007, Segher Boessenkool wrote:

> > > > What you probably mean is that the compiler has to assume any code
> > > > it cannot currently see can do anything (insofar as allowed by the
> > > > relevant standards etc.)
> > 
> > I think this was just terminology confusion here again. Isn't "any code
> > that it cannot currently see" the same as "another compilation unit",
> 
> It is not; try  gcc -combine  or the upcoming link-time optimisation
> stuff, for example.
> 
> > and wouldn't the "compilation unit" itself expand if we ask gcc to
> > compile more than one unit at once? Or is there some more specific
> > "definition" for "compilation unit" (in gcc lingo, possibly?)
> 
> "compilation unit" is a C standard term.  It typically boils down
> to "single .c file".

As you mentioned later, "single .c file with all the other files (headers
or other .c files) that it pulls in via #include" is actually "translation
unit", both in the C standard as well as gcc docs. "Compilation unit"
doesn't seem to be nearly as standard a term, though in most places it
is indeed meant to be same as "translation unit", but with the new gcc
inter-module-analysis stuff that you referred to above, I suspect one may
reasonably want to call a "compilation unit" as all that the compiler sees
at a given instant.

BTW I did some auditing (only inside include/asm-{i386,x86_64}/ and
arch/{i386,x86_64}/) and found a couple more callsites that don't use
cpu_relax():

arch/i386/kernel/crash.c:101
arch/x86_64/kernel/crash.c:97

that are:

	while ((atomic_read(&waiting_for_crash_ipi) > 0) && msecs) {
		mdelay(1);
		msecs--;
	}

where mdelay() becomes __const_udelay() which happens to be in another
translation unit (arch/i386/lib/delay.c) and hence saves this callsite
from being a bug :-)

Curiously, __const_udelay() is still marked as "inline" where it is
implemented in lib/delay.c which is weird, considering it won't ever
be inlined, would it? With the kernel presently being compiled one
translation unit at a time, I don't see how the implementation would
be visible to any callsite out there to be able to inline it.


Satyam

[PATCH] i386: Fix a couple busy loops in mach_wakecpu.h:wait_for_init_deassert()

[Satyam Sharma]

On Wed, 15 Aug 2007, Heiko Carstens wrote:

> [...]
> Btw.: we still have
> 
> include/asm-i386/mach-es7000/mach_wakecpu.h:  while (!atomic_read(deassert));
> include/asm-i386/mach-default/mach_wakecpu.h: while (!atomic_read(deassert));
> 
> Looks like they need to be fixed as well.


[PATCH] i386: Fix a couple busy loops in mach_wakecpu.h:wait_for_init_deassert()

Use cpu_relax() in the busy loops, as atomic_read() doesn't automatically
imply volatility for i386 and x86_64. x86_64 doesn't have this issue because
it open-codes the while loop in smpboot.c:smp_callin() itself that already
uses cpu_relax().

For i386, however, smpboot.c:smp_callin() calls wait_for_init_deassert()
which is buggy for mach-default and mach-es7000 cases.

[ I test-built a kernel -- smp_callin() itself got inlined in its only
  callsite, smpboot.c:start_secondary() -- and the relevant piece of
  code disassembles to the following:

0xc1019704 <start_secondary+12>:        mov    0xc144c4c8,%eax
0xc1019709 <start_secondary+17>:        test   %eax,%eax
0xc101970b <start_secondary+19>:        je     0xc1019709 <start_secondary+17>

  init_deasserted (at 0xc144c4c8) gets fetched into %eax only once and
  then we loop over the test of the stale value in the register only,
  so these look like real bugs to me. With the fix below, this becomes:

0xc1019706 <start_secondary+14>:        pause
0xc1019708 <start_secondary+16>:        cmpl   $0x0,0xc144c4c8
0xc101970f <start_secondary+23>:        je     0xc1019706 <start_secondary+14>

  which looks nice and healthy. ]

Thanks to Heiko Carstens for noticing this.

Signed-off-by: Satyam Sharma <satyam@infradead.org>

---

 include/asm-i386/mach-default/mach_wakecpu.h |    3 ++-
 include/asm-i386/mach-es7000/mach_wakecpu.h  |    3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/include/asm-i386/mach-default/mach_wakecpu.h b/include/asm-i386/mach-default/mach_wakecpu.h
index 673b85c..3ebb178 100644
--- a/include/asm-i386/mach-default/mach_wakecpu.h
+++ b/include/asm-i386/mach-default/mach_wakecpu.h
@@ -15,7 +15,8 @@
 
 static inline void wait_for_init_deassert(atomic_t *deassert)
 {
-	while (!atomic_read(deassert));
+	while (!atomic_read(deassert))
+		cpu_relax();
 	return;
 }
 
diff --git a/include/asm-i386/mach-es7000/mach_wakecpu.h b/include/asm-i386/mach-es7000/mach_wakecpu.h
index efc903b..84ff583 100644
--- a/include/asm-i386/mach-es7000/mach_wakecpu.h
+++ b/include/asm-i386/mach-es7000/mach_wakecpu.h
@@ -31,7 +31,8 @@ wakeup_secondary_cpu(int phys_apicid, unsigned long start_eip)
 static inline void wait_for_init_deassert(atomic_t *deassert)
 {
 #ifdef WAKE_SECONDARY_VIA_INIT
-	while (!atomic_read(deassert));
+	while (!atomic_read(deassert))
+		cpu_relax();
 #endif
 	return;
 }

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Wed, Aug 15, 2007 at 05:26:34PM -0700, Christoph Lameter wrote:
> On Thu, 16 Aug 2007, Paul Mackerras wrote:
> 
> > In the kernel we use atomic variables in precisely those situations
> > where a variable is potentially accessed concurrently by multiple
> > CPUs, and where each CPU needs to see updates done by other CPUs in a
> > timely fashion.  That is what they are for.  Therefore the compiler
> > must not cache values of atomic variables in registers; each
> > atomic_read must result in a load and each atomic_set must result in a
> > store.  Anything else will just lead to subtle bugs.
> 
> This may have been the intend. However, today the visibility is controlled 
> using barriers. And we have barriers that we use with atomic operations. 
> Having volatile be the default just lead to confusion. Atomic read should 
> just read with no extras. Extras can be added by using variants like 
> atomic_read_volatile or so.

Seems to me that we face greater chance of confusion without the
volatile than with, particularly as compiler optimizations become
more aggressive.  Yes, we could simply disable optimization, but
optimization can be quite helpful.

						Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Christoph Lameter]

On Thu, 16 Aug 2007, Paul Mackerras wrote:

> Those barriers are for when we need ordering between atomic variables
> and other memory locations.  An atomic variable by itself doesn't and
> shouldn't need any barriers for other CPUs to be able to see what's
> happening to it.

It does not need any barriers. As soon as one cpu acquires the 
cacheline for write it will be invalidated in the caches of the others. So 
the other cpu will have to refetch. No need for volatile.

The issue here may be that the compiler has fetched the atomic variable 
earlier and put it into a register. However, that prefetching is limited 
because it cannot cross functions calls etc. The only problem could be 
loops where the compiler does not refetch the variable since it assumes 
that it does not change and there are no function calls in the body of the 
loop. But AFAIK these loops need cpu_relax and other measures anyways to 
avoid bad effects from busy waiting.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Christoph Lameter]

On Wed, 15 Aug 2007, Paul E. McKenney wrote:

> Seems to me that we face greater chance of confusion without the
> volatile than with, particularly as compiler optimizations become
> more aggressive.  Yes, we could simply disable optimization, but
> optimization can be quite helpful.

A volatile default would disable optimizations for atomic_read. 
atomic_read without volatile would allow for full optimization by the 
compiler. Seems that this is what one wants in many cases.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Thu, Aug 16, 2007 at 08:30:23AM +0800, Herbert Xu wrote:
> On Wed, Aug 15, 2007 at 05:23:10PM -0700, Paul E. McKenney wrote:
> > On Thu, Aug 16, 2007 at 08:12:48AM +0800, Herbert Xu wrote:
> > > On Wed, Aug 15, 2007 at 04:53:35PM -0700, Paul E. McKenney wrote:
> > > >
> > > > > > Communicating between process context and interrupt/NMI handlers using
> > > > > > per-CPU variables.
> > > > > 
> > > > > Remeber we're talking about atomic_read/atomic_set.  Please
> > > > > cite the actual file/function name you have in mind.
> > > > 
> > > > Yep, we are indeed talking about atomic_read()/atomic_set().
> > > > 
> > > > We have been through this issue already in this thread.
> > > 
> > > Sorry, but I must've missed it.  Could you cite the file or
> > > function for my benefit?
> > 
> > I might summarize the thread if there is interest, but I am not able to
> > do so right this minute.
> 
> Thanks.  But I don't need a summary of the thread, I'm asking
> for an extant code snippet in our kernel that benefits from
> the volatile change and is not part of a busy-wait.

Sorry, can't help you there.  I really do believe that the information
you need (as opposed to the specific item you are asking for) really
has been put forth in this thread.

						Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

On Thu, Aug 16, 2007 at 06:28:42AM +0530, Satyam Sharma wrote:
>
> > The udelay itself certainly should have some form of cpu_relax in it.
> 
> Yes, a form of barrier() must be present in mdelay() or udelay() itself
> as you say, having it in __const_udelay() is *not* enough (superflous
> actually, considering it is already a separate translation unit and
> invisible to the compiler).

As long as __const_udelay does something which has the same
effect as barrier it is enough even if it's in the same unit.
As a matter of fact it does on i386 where __delay either uses
rep_nop or asm/volatile.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

On Wed, Aug 15, 2007 at 05:49:50PM -0700, Paul E. McKenney wrote:
> On Thu, Aug 16, 2007 at 08:30:23AM +0800, Herbert Xu wrote:
>
> > Thanks.  But I don't need a summary of the thread, I'm asking
> > for an extant code snippet in our kernel that benefits from
> > the volatile change and is not part of a busy-wait.
> 
> Sorry, can't help you there.  I really do believe that the information
> you need (as opposed to the specific item you are asking for) really
> has been put forth in this thread.

That only leads me to believe that such a code snippet simply
does not exist.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Wed, Aug 15, 2007 at 05:42:07PM -0700, Christoph Lameter wrote:
> On Wed, 15 Aug 2007, Paul E. McKenney wrote:
> 
> > Seems to me that we face greater chance of confusion without the
> > volatile than with, particularly as compiler optimizations become
> > more aggressive.  Yes, we could simply disable optimization, but
> > optimization can be quite helpful.
> 
> A volatile default would disable optimizations for atomic_read. 
> atomic_read without volatile would allow for full optimization by the 
> compiler. Seems that this is what one wants in many cases.

The volatile cast should not disable all that many optimizations,
for example, it is much less hurtful than barrier().  Furthermore,
the main optimizations disabled (pulling atomic_read() and atomic_set()
out of loops) really do need to be disabled.

						Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

[ Sorry for empty subject line in previous mail. I intended to make
  a patch so cleared it to change it, but ultimately neither made
  a patch nor restored subject line. Done that now. ]


On Thu, 16 Aug 2007, Herbert Xu wrote:

> On Thu, Aug 16, 2007 at 06:06:00AM +0530, Satyam Sharma wrote:
> > 
> > that are:
> > 
> > 	while ((atomic_read(&waiting_for_crash_ipi) > 0) && msecs) {
> > 		mdelay(1);
> > 		msecs--;
> > 	}
> > 
> > where mdelay() becomes __const_udelay() which happens to be in another
> > translation unit (arch/i386/lib/delay.c) and hence saves this callsite
> > from being a bug :-)
> 
> The udelay itself certainly should have some form of cpu_relax in it.

Yes, a form of barrier() must be present in mdelay() or udelay() itself
as you say, having it in __const_udelay() is *not* enough (superflous
actually, considering it is already a separate translation unit and
invisible to the compiler).

However, there are no compiler barriers on the macro-definition-path
between mdelay(1) and __const_udelay(), so the only thing that saves us
from being a bug here is indeed the different-translation-unit concept.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Christoph Lameter]

On Wed, 15 Aug 2007, Paul E. McKenney wrote:

> The volatile cast should not disable all that many optimizations,
> for example, it is much less hurtful than barrier().  Furthermore,
> the main optimizations disabled (pulling atomic_read() and atomic_set()
> out of loops) really do need to be disabled.

In many cases you do not need a barrier. Having volatile there *will* 
impact optimization because the compiler cannot use a register that may 
contain the value that was fetched earlier. And the compiler cannot choose 
freely when to fetch the value. The order of memory accesses are fixed if 
you use volatile. If the variable is not volatile then the compiler can 
arrange memory accesses any way they fit and thus generate better code.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Wed, Aug 15, 2007 at 05:59:41PM -0700, Christoph Lameter wrote:
> On Wed, 15 Aug 2007, Paul E. McKenney wrote:
> 
> > The volatile cast should not disable all that many optimizations,
> > for example, it is much less hurtful than barrier().  Furthermore,
> > the main optimizations disabled (pulling atomic_read() and atomic_set()
> > out of loops) really do need to be disabled.
> 
> In many cases you do not need a barrier. Having volatile there *will* 
> impact optimization because the compiler cannot use a register that may 
> contain the value that was fetched earlier. And the compiler cannot choose 
> freely when to fetch the value. The order of memory accesses are fixed if 
> you use volatile. If the variable is not volatile then the compiler can 
> arrange memory accesses any way they fit and thus generate better code.

Understood.  My point is not that the impact is precisely zero, but
rather that the impact on optimization is much less hurtful than the
problems that could arise otherwise, particularly as compilers become
more aggressive in their optimizations.

						Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Thu, Aug 16, 2007 at 08:53:16AM +0800, Herbert Xu wrote:
> On Wed, Aug 15, 2007 at 05:49:50PM -0700, Paul E. McKenney wrote:
> > On Thu, Aug 16, 2007 at 08:30:23AM +0800, Herbert Xu wrote:
> >
> > > Thanks.  But I don't need a summary of the thread, I'm asking
> > > for an extant code snippet in our kernel that benefits from
> > > the volatile change and is not part of a busy-wait.
> > 
> > Sorry, can't help you there.  I really do believe that the information
> > you need (as opposed to the specific item you are asking for) really
> > has been put forth in this thread.
> 
> That only leads me to believe that such a code snippet simply
> does not exist.

Whatever...

						Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

Hi Herbert,


On Thu, 16 Aug 2007, Herbert Xu wrote:

> On Thu, Aug 16, 2007 at 06:28:42AM +0530, Satyam Sharma wrote:
> >
> > > The udelay itself certainly should have some form of cpu_relax in it.
> > 
> > Yes, a form of barrier() must be present in mdelay() or udelay() itself
> > as you say, having it in __const_udelay() is *not* enough (superflous
> > actually, considering it is already a separate translation unit and
> > invisible to the compiler).
> 
> As long as __const_udelay does something which has the same
> effect as barrier it is enough even if it's in the same unit.

Only if __const_udelay() is inlined. But as I said, __const_udelay()
-- although marked "inline" -- will never be inlined anywhere in the
kernel in reality. It's an exported symbol, and never inlined from
modules. Even from built-in targets, the definition of __const_udelay
is invisible when gcc is compiling the compilation units of those
callsites. The compiler has no idea that that function has barriers
or not, so we're saved here _only_ by the lucky fact that
__const_udelay() is in a different compilation unit.


> As a matter of fact it does on i386 where __delay either uses
> rep_nop or asm/volatile.

__delay() can be either delay_tsc() or delay_loop() on i386.

delay_tsc() uses the rep_nop() there for it's own little busy
loop, actually. But for a call site that inlines __const_udelay()
-- if it were ever moved to a .h file and marked inline -- the
call to __delay() will _still_ be across compilation units. So,
again for this case, it does not matter if the callee function
has compiler barriers or not (it would've been a different story
if we were discussing real/CPU barriers, I think), what saves us
here is just the fact that a call is made to a function from a
different compilation unit, which is invisible to the compiler
when compiling the callsite, and hence acting as the compiler
barrier.

Regarding delay_loop(), it uses "volatile" for the "asm" which
has quite different semantics from the C language "volatile"
type-qualifier keyword and does not imply any compiler barrier
at all.


Satyam

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

>>>> No; compilation units have nothing to do with it, GCC can optimise
>>>> across compilation unit boundaries just fine, if you tell it to
>>>> compile more than one compilation unit at once.
>>>
>>> Last I checked, the Linux kernel build system did compile each .c 
>>> file
>>> as a separate compilation unit.
>>
>> I have some patches to use -combine -fwhole-program for Linux.
>> Highly experimental, you need a patched bleeding edge toolchain.
>> If there's interest I'll clean it up and put it online.
>>
>> David Woodhouse had some similar patches about a year ago.
>
> Sounds exciting...  ;-)

Yeah, the breakage is *quite* spectacular :-)

>>>>> In many cases, the compiler also has to assume that
>>>>> msleep_interruptible()
>>>>> might call back into a function in the current compilation unit, 
>>>>> thus
>>>>> possibly modifying global static variables.
>>>>
>>>> It most often is smart enough to see what compilation-unit-local
>>>> variables might be modified that way, though :-)
>>>
>>> Yep.  For example, if it knows the current value of a given such 
>>> local
>>> variable, and if all code paths that would change some other variable
>>> cannot be reached given that current value of the first variable.
>>
>> Or the most common thing: if neither the address of the translation-
>> unit local variable nor the address of any function writing to that
>> variable can "escape" from that translation unit, nothing outside
>> the translation unit can write to the variable.
>
> But there is usually at least one externally callable function in
> a .c file.

Of course, but often none of those will (indirectly) write a certain
static variable.


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Wed, 15 Aug 2007, Segher Boessenkool wrote:

> [...]
> > BTW:
> > 
> > #define atomic_read(a)	(*(volatile int *)&(a))
> > #define atomic_set(a,i)	(*(volatile int *)&(a) = (i))
> > 
> > int a;
> > 
> > void func(void)
> > {
> > 	int b;
> > 
> > 	b = atomic_read(a);
> > 	atomic_set(a, 20);
> > 	b = atomic_read(a);
> > }
> > 
> > gives:
> > 
> > func:
> > 	pushl	%ebp
> > 	movl	a, %eax
> > 	movl	%esp, %ebp
> > 	movl	$20, a
> > 	movl	a, %eax
> > 	popl	%ebp
> > 	ret
> > 
> > so the first atomic_read() wasn't optimized away.
> 
> Of course.  It is executed by the abstract machine, so
> it will be executed by the actual machine.  On the other
> hand, try
> 
> 	b = 0;
> 	if (b)
> 		b = atomic_read(a);
> 
> or similar.

Yup, obviously. Volatile accesses (or any access to volatile objects),
or even "__volatile__ asms" (which gcc normally promises never to elid)
can always be optimized for cases such as these where the compiler can
trivially determine that the code in question is not reachable.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

>> Part of the motivation here is to fix heisenbugs.  If I knew where 
>> they
>
> By the same token we should probably disable optimisations
> altogether since that too can create heisenbugs.

Almost everything is a tradeoff; and so is this.  I don't
believe most people would find disabling all compiler
optimisations an acceptable price to pay for some peace
of mind.


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

>>> Part of the motivation here is to fix heisenbugs.  If I knew where 
>>> they
>>
>> By the same token we should probably disable optimisations
>> altogether since that too can create heisenbugs.
>
> Precisely the point -- use of volatile (whether in casts or on asms)
> in these cases are intended to disable those optimizations likely to
> result in heisenbugs.

The only thing volatile on an asm does is create a side effect
on the asm statement; in effect, it tells the compiler "do not
remove this asm even if you don't need any of its outputs".

It's not disabling optimisation likely to result in bugs,
heisen- or otherwise; _not_ putting the volatile on an asm
that needs it simply _is_ a bug :-)


Segher

Re:

[Segher Boessenkool]

>> "compilation unit" is a C standard term.  It typically boils down
>> to "single .c file".
>
> As you mentioned later, "single .c file with all the other files 
> (headers
> or other .c files) that it pulls in via #include" is actually 
> "translation
> unit", both in the C standard as well as gcc docs.

Yeah.  "single .c file after preprocessing".  Same thing :-)

> "Compilation unit"
> doesn't seem to be nearly as standard a term, though in most places it
> is indeed meant to be same as "translation unit", but with the new gcc
> inter-module-analysis stuff that you referred to above, I suspect one 
> may
> reasonably want to call a "compilation unit" as all that the compiler 
> sees
> at a given instant.

That would be a bit confusing, would it not?  They'd better find
some better name for that if they want to name it at all (remember,
none of these optimisations should have any effect on the semantics
of the program, you just get fewer .o files etc.).


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Christoph Lameter]

On Wed, 15 Aug 2007, Paul E. McKenney wrote:

> Understood.  My point is not that the impact is precisely zero, but
> rather that the impact on optimization is much less hurtful than the
> problems that could arise otherwise, particularly as compilers become
> more aggressive in their optimizations.

The problems arise because barriers are not used as required. Volatile 
has wishy washy semantics and somehow marries memory barriers with data 
access. It is clearer to separate the two. Conceptual cleanness usually 
translates into better code. If one really wants the volatile then lets 
make it explicit and use

	atomic_read_volatile()

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul Mackerras]

Christoph Lameter writes:

> A volatile default would disable optimizations for atomic_read. 
> atomic_read without volatile would allow for full optimization by the 
> compiler. Seems that this is what one wants in many cases.

Name one such case.

An atomic_read should do a load from memory.  If the programmer puts
an atomic_read() in the code then the compiler should emit a load for
it, not re-use a value returned by a previous atomic_read.  I do not
believe it would ever be useful for the compiler to collapse two
atomic_read statements into a single load.

Paul.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

On Thu, Aug 16, 2007 at 11:51:42AM +1000, Paul Mackerras wrote:
> 
> Name one such case.

See sk_stream_mem_schedule in net/core/stream.c:

        /* Under limit. */
        if (atomic_read(sk->sk_prot->memory_allocated) < sk->sk_prot->sysctl_mem[0]) {
                if (*sk->sk_prot->memory_pressure)
                        *sk->sk_prot->memory_pressure = 0;
                return 1;
        }

        /* Over hard limit. */
        if (atomic_read(sk->sk_prot->memory_allocated) > sk->sk_prot->sysctl_mem[2]) {
                sk->sk_prot->enter_memory_pressure();
                goto suppress_allocation;
        }

We don't need to reload sk->sk_prot->memory_allocated here.

Now where is your example again?

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul Mackerras]

Herbert Xu writes:

> See sk_stream_mem_schedule in net/core/stream.c:
> 
>         /* Under limit. */
>         if (atomic_read(sk->sk_prot->memory_allocated) < sk->sk_prot->sysctl_mem[0]) {
>                 if (*sk->sk_prot->memory_pressure)
>                         *sk->sk_prot->memory_pressure = 0;
>                 return 1;
>         }
> 
>         /* Over hard limit. */
>         if (atomic_read(sk->sk_prot->memory_allocated) > sk->sk_prot->sysctl_mem[2]) {
>                 sk->sk_prot->enter_memory_pressure();
>                 goto suppress_allocation;
>         }
> 
> We don't need to reload sk->sk_prot->memory_allocated here.

Are you sure?  How do you know some other CPU hasn't changed the value
in between?

Paul.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

>> A volatile default would disable optimizations for atomic_read.
>> atomic_read without volatile would allow for full optimization by the
>> compiler. Seems that this is what one wants in many cases.
>
> Name one such case.
>
> An atomic_read should do a load from memory.  If the programmer puts
> an atomic_read() in the code then the compiler should emit a load for
> it, not re-use a value returned by a previous atomic_read.  I do not
> believe it would ever be useful for the compiler to collapse two
> atomic_read statements into a single load.

An atomic_read() implemented as a "normal" C variable read would
allow that read to be combined with another "normal" read from
that variable.  This could perhaps be marginally useful, although
I'd bet you cannot see it unless counting cycles on a simulator
or counting bits in the binary size.

With an asm() implementation, the compiler can not do this; with
a "volatile" implementation (either volatile variable or volatile-cast),
this invokes undefined behaviour (in both C and GCC).


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

On Thu, Aug 16, 2007 at 07:45:44AM +0530, Satyam Sharma wrote:
>
> Completely agreed, again. To summarize again (had done so about ~100 mails
> earlier in this thread too :-) ...
> 
> atomic_{read,set}_volatile() -- guarantees volatility also along with
> atomicity (the two _are_ different concepts after all, irrespective of
> whether callsites normally want one with the other or not)
> 
> atomic_{read,set}_nonvolatile() -- only guarantees atomicity, compiler
> free to elid / coalesce / optimize such accesses, can keep the object
> in question cached in a local register, leads to smaller text, etc.
> 
> As to which one should be the default atomic_read() is a question of
> whether majority of callsites (more weightage to important / hot
> codepaths, lesser to obscure callsites) want a particular behaviour.
> 
> Do we have a consensus here? (hoping against hope, probably :-)

I can certainly agree with this.

But I have to say that I still don't know of a single place
where one would actually use the volatile variant.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

On Thu, Aug 16, 2007 at 12:05:56PM +1000, Paul Mackerras wrote:
> Herbert Xu writes:
> 
> > See sk_stream_mem_schedule in net/core/stream.c:
> > 
> >         /* Under limit. */
> >         if (atomic_read(sk->sk_prot->memory_allocated) < sk->sk_prot->sysctl_mem[0]) {
> >                 if (*sk->sk_prot->memory_pressure)
> >                         *sk->sk_prot->memory_pressure = 0;
> >                 return 1;
> >         }
> > 
> >         /* Over hard limit. */
> >         if (atomic_read(sk->sk_prot->memory_allocated) > sk->sk_prot->sysctl_mem[2]) {
> >                 sk->sk_prot->enter_memory_pressure();
> >                 goto suppress_allocation;
> >         }
> > 
> > We don't need to reload sk->sk_prot->memory_allocated here.
> 
> Are you sure?  How do you know some other CPU hasn't changed the value
> in between?

Yes I'm sure, because we don't care if others have increased
the reservation.

Note that even if we did we'd be using barriers so volatile
won't do us any good here.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Wed, 15 Aug 2007, Christoph Lameter wrote:

> On Wed, 15 Aug 2007, Paul E. McKenney wrote:
> 
> > Understood.  My point is not that the impact is precisely zero, but
> > rather that the impact on optimization is much less hurtful than the
> > problems that could arise otherwise, particularly as compilers become
> > more aggressive in their optimizations.
> 
> The problems arise because barriers are not used as required. Volatile 
> has wishy washy semantics and somehow marries memory barriers with data 
> access. It is clearer to separate the two. Conceptual cleanness usually 
> translates into better code. If one really wants the volatile then lets 
> make it explicit and use
> 
> 	atomic_read_volatile()

Completely agreed, again. To summarize again (had done so about ~100 mails
earlier in this thread too :-) ...

atomic_{read,set}_volatile() -- guarantees volatility also along with
atomicity (the two _are_ different concepts after all, irrespective of
whether callsites normally want one with the other or not)

atomic_{read,set}_nonvolatile() -- only guarantees atomicity, compiler
free to elid / coalesce / optimize such accesses, can keep the object
in question cached in a local register, leads to smaller text, etc.

As to which one should be the default atomic_read() is a question of
whether majority of callsites (more weightage to important / hot
codepaths, lesser to obscure callsites) want a particular behaviour.

Do we have a consensus here? (hoping against hope, probably :-)


[ This thread has gotten completely out of hand ... for my mail client
  alpine as well, it now seems. Reminds of that 1000+ GPLv3 fest :-) ]

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Christoph Lameter]

On Thu, 16 Aug 2007, Paul Mackerras wrote:

> > We don't need to reload sk->sk_prot->memory_allocated here.
> 
> Are you sure?  How do you know some other CPU hasn't changed the value
> in between?

The cpu knows because the cacheline was not invalidated.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Christoph Lameter]

On Wed, 15 Aug 2007, Christoph Lameter wrote:

> On Thu, 16 Aug 2007, Paul Mackerras wrote:
> 
> > > We don't need to reload sk->sk_prot->memory_allocated here.
> > 
> > Are you sure?  How do you know some other CPU hasn't changed the value
> > in between?
> 
> The cpu knows because the cacheline was not invalidated.

Crap my statement above is wrong..... We do not care that the 
value was changed otherwise we would have put a barrier in there.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Christoph Lameter]

On Thu, 16 Aug 2007, Herbert Xu wrote:

> > Do we have a consensus here? (hoping against hope, probably :-)
> 
> I can certainly agree with this.

I agree too.

> But I have to say that I still don't know of a single place
> where one would actually use the volatile variant.

I suspect that what you say is true after we have looked at all callers.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Chris Friesen]

Herbert Xu wrote:

> But I have to say that I still don't know of a single place
> where one would actually use the volatile variant.

Given that many of the existing users do currently have "volatile", are 
you comfortable simply removing that behaviour from them?  Are you sure 
that you will not introduce any issues?

Forcing a re-read is only a performance penalty.  Removing it can cause 
behavioural changes.

I would be more comfortable making the default match the majority of the 
current implementations (ie: volatile semantics).  Then, if someone 
cares about performance they can explicitly validate the call path and 
convert it over to the non-volatile version.

Correctness before speed...

Chris

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Thu, Aug 16, 2007 at 03:23:28AM +0200, Segher Boessenkool wrote:
> >>>>No; compilation units have nothing to do with it, GCC can optimise
> >>>>across compilation unit boundaries just fine, if you tell it to
> >>>>compile more than one compilation unit at once.
> >>>
> >>>Last I checked, the Linux kernel build system did compile each .c 
> >>>file
> >>>as a separate compilation unit.
> >>
> >>I have some patches to use -combine -fwhole-program for Linux.
> >>Highly experimental, you need a patched bleeding edge toolchain.
> >>If there's interest I'll clean it up and put it online.
> >>
> >>David Woodhouse had some similar patches about a year ago.
> >
> >Sounds exciting...  ;-)
> 
> Yeah, the breakage is *quite* spectacular :-)

I bet!!!  ;-)

> >>>>>In many cases, the compiler also has to assume that
> >>>>>msleep_interruptible()
> >>>>>might call back into a function in the current compilation unit, 
> >>>>>thus
> >>>>>possibly modifying global static variables.
> >>>>
> >>>>It most often is smart enough to see what compilation-unit-local
> >>>>variables might be modified that way, though :-)
> >>>
> >>>Yep.  For example, if it knows the current value of a given such 
> >>>local
> >>>variable, and if all code paths that would change some other variable
> >>>cannot be reached given that current value of the first variable.
> >>
> >>Or the most common thing: if neither the address of the translation-
> >>unit local variable nor the address of any function writing to that
> >>variable can "escape" from that translation unit, nothing outside
> >>the translation unit can write to the variable.
> >
> >But there is usually at least one externally callable function in
> >a .c file.
> 
> Of course, but often none of those will (indirectly) write a certain
> static variable.

But there has to be some path to the static functions, assuming that
they are not dead code.  Yes, there can be cases where the compiler
knows enough about the state of the variables to rule out some of code
paths to them, but I have no idea how often this happens in kernel
code.

						Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Nick Piggin]

Segher Boessenkool wrote:
>>> Part of the motivation here is to fix heisenbugs.  If I knew where they
>>
>>
>> By the same token we should probably disable optimisations
>> altogether since that too can create heisenbugs.
> 
> 
> Almost everything is a tradeoff; and so is this.  I don't
> believe most people would find disabling all compiler
> optimisations an acceptable price to pay for some peace
> of mind.

So why is this a good tradeoff?

I also think that just adding things to APIs in the hope it might fix
up some bugs isn't really a good road to go down. Where do you stop?

On the actual proposal to make atomic_operators volatile: I think the
better approach in the long term, for both maintainability of the
code and education of coders, is to make the use of barriers _more_
explicit rather than sprinkling these "just in case" ones around.

You may get rid of a few atomic_read heisenbugs (in noise when
compared to all bugs), but if the coder was using a regular atomic
load, or a test_bit (which is also atomic), etc. then they're going
to have problems.

It would be better for Linux if everyone was to have better awareness
of barriers than to hide some of the cases where they're required.
A pretty large number of bugs I see in lock free code in the VM is
due to memory ordering problems. It's hard to find those bugs, or
even be aware when you're writing buggy code if you don't have some
feel for barriers.

-- 
SUSE Labs, Novell Inc.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Thu, Aug 16, 2007 at 03:30:44AM +0200, Segher Boessenkool wrote:
> >>>Part of the motivation here is to fix heisenbugs.  If I knew where 
> >>>they
> >>
> >>By the same token we should probably disable optimisations
> >>altogether since that too can create heisenbugs.
> >
> >Precisely the point -- use of volatile (whether in casts or on asms)
> >in these cases are intended to disable those optimizations likely to
> >result in heisenbugs.
> 
> The only thing volatile on an asm does is create a side effect
> on the asm statement; in effect, it tells the compiler "do not
> remove this asm even if you don't need any of its outputs".
> 
> It's not disabling optimisation likely to result in bugs,
> heisen- or otherwise; _not_ putting the volatile on an asm
> that needs it simply _is_ a bug :-)

Yep.  And the reason it is a bug is that it fails to disable
the relevant compiler optimizations.  So I suspect that we might
actually be saying the same thing here.

						Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Wed, Aug 15, 2007 at 06:41:40PM -0700, Christoph Lameter wrote:
> On Wed, 15 Aug 2007, Paul E. McKenney wrote:
> 
> > Understood.  My point is not that the impact is precisely zero, but
> > rather that the impact on optimization is much less hurtful than the
> > problems that could arise otherwise, particularly as compilers become
> > more aggressive in their optimizations.
> 
> The problems arise because barriers are not used as required. Volatile 
> has wishy washy semantics and somehow marries memory barriers with data 
> access. It is clearer to separate the two. Conceptual cleanness usually 
> translates into better code. If one really wants the volatile then lets 
> make it explicit and use
> 
> 	atomic_read_volatile()

There are indeed architectures where you can cause gcc to emit memory
barriers in response to volatile.  I am assuming that we are -not-
making gcc do this.  Given this, then volatiles and memory barrier
instructions are orthogonal -- one controls the compiler, the other
controls the CPU.

						Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Thu, 16 Aug 2007, Paul Mackerras wrote:

> Herbert Xu writes:
> 
> > See sk_stream_mem_schedule in net/core/stream.c:
> > 
> >         /* Under limit. */
> >         if (atomic_read(sk->sk_prot->memory_allocated) < sk->sk_prot->sysctl_mem[0]) {
> >                 if (*sk->sk_prot->memory_pressure)
> >                         *sk->sk_prot->memory_pressure = 0;
> >                 return 1;
> >         }
> > 
> >         /* Over hard limit. */
> >         if (atomic_read(sk->sk_prot->memory_allocated) > sk->sk_prot->sysctl_mem[2]) {
> >                 sk->sk_prot->enter_memory_pressure();
> >                 goto suppress_allocation;
> >         }
> > 
> > We don't need to reload sk->sk_prot->memory_allocated here.
> 
> Are you sure?  How do you know some other CPU hasn't changed the value
> in between?

I can't speak for this particular case, but there could be similar code
examples elsewhere, where we do the atomic ops on an atomic_t object
inside a higher-level locking scheme that would take care of the kind of
problem you're referring to here. It would be useful for such or similar
code if the compiler kept the value of that atomic object in a register.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Thu, Aug 16, 2007 at 10:11:05AM +0800, Herbert Xu wrote:
> On Thu, Aug 16, 2007 at 12:05:56PM +1000, Paul Mackerras wrote:
> > Herbert Xu writes:
> > 
> > > See sk_stream_mem_schedule in net/core/stream.c:
> > > 
> > >         /* Under limit. */
> > >         if (atomic_read(sk->sk_prot->memory_allocated) < sk->sk_prot->sysctl_mem[0]) {
> > >                 if (*sk->sk_prot->memory_pressure)
> > >                         *sk->sk_prot->memory_pressure = 0;
> > >                 return 1;
> > >         }
> > > 
> > >         /* Over hard limit. */
> > >         if (atomic_read(sk->sk_prot->memory_allocated) > sk->sk_prot->sysctl_mem[2]) {
> > >                 sk->sk_prot->enter_memory_pressure();
> > >                 goto suppress_allocation;
> > >         }
> > > 
> > > We don't need to reload sk->sk_prot->memory_allocated here.
> > 
> > Are you sure?  How do you know some other CPU hasn't changed the value
> > in between?
> 
> Yes I'm sure, because we don't care if others have increased
> the reservation.
> 
> Note that even if we did we'd be using barriers so volatile
> won't do us any good here.

If the load-coalescing is important to performance, why not load into
a local variable?

						Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Thu, 16 Aug 2007, Satyam Sharma wrote:

> On Thu, 16 Aug 2007, Paul Mackerras wrote:
> > Herbert Xu writes:
> > 
> > > See sk_stream_mem_schedule in net/core/stream.c:
> > > 
> > >         /* Under limit. */
> > >         if (atomic_read(sk->sk_prot->memory_allocated) < sk->sk_prot->sysctl_mem[0]) {
> > >                 if (*sk->sk_prot->memory_pressure)
> > >                         *sk->sk_prot->memory_pressure = 0;
> > >                 return 1;
> > >         }
> > > 
> > >         /* Over hard limit. */
> > >         if (atomic_read(sk->sk_prot->memory_allocated) > sk->sk_prot->sysctl_mem[2]) {
> > >                 sk->sk_prot->enter_memory_pressure();
> > >                 goto suppress_allocation;
> > >         }
> > > 
> > > We don't need to reload sk->sk_prot->memory_allocated here.
> > 
> > Are you sure?  How do you know some other CPU hasn't changed the value
> > in between?
> 
> I can't speak for this particular case, but there could be similar code
> examples elsewhere, where we do the atomic ops on an atomic_t object
> inside a higher-level locking scheme that would take care of the kind of
> problem you're referring to here. It would be useful for such or similar
> code if the compiler kept the value of that atomic object in a register.

We might not be using atomic_t (and ops) if we already have a higher-level
locking scheme, actually. So as Herbert mentioned, such cases might just
not care. [ Too much of this thread, too little sleep, sorry! ]

Anyway, the problem, of course, is that this conversion to a stronger /
safer-by-default behaviour doesn't happen with zero cost to performance.
Converting atomic ops to "volatile" behaviour did add ~2K to kernel text
for archs such as i386 (possibly to important codepaths) that didn't have
those semantics already so it would be constructive to actually look at
those differences and see if there were really any heisenbugs that got
rectified. Or if there were legitimate optimizations that got wrongly
disabled. Onus lies on those proposing the modifications, I'd say ;-)

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul Mackerras]

Satyam Sharma writes:

> I can't speak for this particular case, but there could be similar code
> examples elsewhere, where we do the atomic ops on an atomic_t object
> inside a higher-level locking scheme that would take care of the kind of
> problem you're referring to here. It would be useful for such or similar
> code if the compiler kept the value of that atomic object in a register.

If there is a higher-level locking scheme then there is no point to
using atomic_t variables.  Atomic_t is specifically for the situation
where multiple CPUs are updating a variable without locking.

Paul.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul Mackerras]

Herbert Xu writes:

> > Are you sure?  How do you know some other CPU hasn't changed the value
> > in between?
> 
> Yes I'm sure, because we don't care if others have increased
> the reservation.

But others can also reduce the reservation.  Also, the code sets and
clears *sk->sk_prot->memory_pressure nonatomically with respect to the
reads of sk->sk_prot->memory_allocated, so in fact the code doesn't
guarantee any particular relationship between the two.

That code looks like a beautiful example of buggy, racy code where
someone has sprinkled magic fix-the-races dust (otherwise known as
atomic_t) around in a vain attempt to fix the races.

That's assuming that all that stuff actually performs any useful
purpose, of course, and that there isn't some lock held by the
callers.  In the latter case it is pointless using atomic_t.

Paul.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul Mackerras]

Christoph Lameter writes:

> > But I have to say that I still don't know of a single place
> > where one would actually use the volatile variant.
> 
> I suspect that what you say is true after we have looked at all callers.

It seems that there could be a lot of places where atomic_t is used in
a non-atomic fashion, and that those uses are either buggy, or there
is some lock held at the time which guarantees that other CPUs aren't
changing the value.  In both cases there is no point in using
atomic_t; we might as well just use an ordinary int.

In particular, atomic_read seems to lend itself to buggy uses.  People
seem to do things like:

	atomic_add(&v, something);
	if (atomic_read(&v) > something_else) ...

and expect that there is some relationship between the value that the
atomic_add stored and the value that the atomic_read will return,
which there isn't.  People seem to think that using atomic_t magically
gets rid of races.  It doesn't.

I'd go so far as to say that anywhere where you want a non-"volatile"
atomic_read, either your code is buggy, or else an int would work just
as well.

Paul.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

On Thu, Aug 16, 2007 at 01:23:06PM +1000, Paul Mackerras wrote:
>
> In particular, atomic_read seems to lend itself to buggy uses.  People
> seem to do things like:
> 
> 	atomic_add(&v, something);
> 	if (atomic_read(&v) > something_else) ...

If you're referring to the code in sk_stream_mem_schedule
then it's working as intended.  The atomicity guarantees
that the atomic_add/atomic_sub won't be seen in parts by
other readers.

We certainly do not need to see other atomic_add/atomic_sub
operations immediately.

If you're referring to another code snippet please cite.

> I'd go so far as to say that anywhere where you want a non-"volatile"
> atomic_read, either your code is buggy, or else an int would work just
> as well.

An int won't work here because += and -= do not have the
atomicity guarantees that atomic_add/atomic_sub do.  In
particular, this may cause an atomic_read on another CPU
to give a bogus reading.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Bill Fink]

On Wed, 15 Aug 2007, Satyam Sharma wrote:

> (C)
> $ cat tp3.c
> int a;
> 
> void func(void)
> {
> 	*(volatile int *)&a = 10;
> 	*(volatile int *)&a = 20;
> }
> $ gcc -Os -S tp3.c
> $ cat tp3.s
> ...
> movl    $10, a
> movl    $20, a
> ...

I'm curious about one minor tangential point.  Why, instead of:

	b = *(volatile int *)&a;

why can't this just be expressed as:

	b = (volatile int)a;

Isn't it the contents of a that's volatile, i.e. it's value can change
invisibly to the compiler, and that's why you want to force a read from
memory?  Why do you need the "*(volatile int *)&" construct?

						-Bill

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

On Thu, Aug 16, 2007 at 01:15:05PM +1000, Paul Mackerras wrote:
> 
> But others can also reduce the reservation.  Also, the code sets and
> clears *sk->sk_prot->memory_pressure nonatomically with respect to the
> reads of sk->sk_prot->memory_allocated, so in fact the code doesn't
> guarantee any particular relationship between the two.

Yes others can reduce the reservation, but the point of this
is that the code doesn't care.  We'll either see the value
before or after the reduction and in either case we'll do
something sensible.

The worst that can happen is when we're just below the hard
limit and multiple CPUs fail to allocate but that's not really
a problem because if the machine is making progress at all
then we will eventually scale back and allow these allocations
to succeed.

As to the non-atomic operation on memory_pressue, that's OK
because we only ever assign values to it and never do other
operations such as += or -=.  Remember that int/long assignments
must be atomic or Linux won't run on your architecture.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul Mackerras]

Herbert Xu writes:

> If you're referring to the code in sk_stream_mem_schedule
> then it's working as intended.  The atomicity guarantees

You mean it's intended that *sk->sk_prot->memory_pressure can end up
as 1 when sk->sk_prot->memory_allocated is small (less than
->sysctl_mem[0]), or as 0 when ->memory_allocated is large (greater
than ->sysctl_mem[2])?  Because that's the effect of the current code.
If so I wonder why you bother computing it.

> that the atomic_add/atomic_sub won't be seen in parts by
> other readers.
> 
> We certainly do not need to see other atomic_add/atomic_sub
> operations immediately.
> 
> If you're referring to another code snippet please cite.
> 
> > I'd go so far as to say that anywhere where you want a non-"volatile"
> > atomic_read, either your code is buggy, or else an int would work just
> > as well.
> 
> An int won't work here because += and -= do not have the
> atomicity guarantees that atomic_add/atomic_sub do.  In
> particular, this may cause an atomic_read on another CPU
> to give a bogus reading.

The point is that guaranteeing the atomicity of the increment or
decrement does not suffice to make the code race-free.  In this case
the race arises from the fact that reading ->memory_allocated and
setting *->memory_pressure are separate operations.  To make that code
work properly you need a lock.  And once you have the lock an ordinary
int would suffice for ->memory_allocated.

Paul.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

On Thu, Aug 16, 2007 at 01:48:32PM +1000, Paul Mackerras wrote:
> Herbert Xu writes:
> 
> > If you're referring to the code in sk_stream_mem_schedule
> > then it's working as intended.  The atomicity guarantees
> 
> You mean it's intended that *sk->sk_prot->memory_pressure can end up
> as 1 when sk->sk_prot->memory_allocated is small (less than
> ->sysctl_mem[0]), or as 0 when ->memory_allocated is large (greater
> than ->sysctl_mem[2])?  Because that's the effect of the current code.
> If so I wonder why you bother computing it.

You need to remember that there are three different limits:
minimum, pressure, and maximum.  By default we should never
be in a situation where what you say can occur.

If you set all three limits to the same thing, then yes it
won't work as intended but it's still well-behaved.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul Mackerras]

Satyam Sharma writes:

> Anyway, the problem, of course, is that this conversion to a stronger /
> safer-by-default behaviour doesn't happen with zero cost to performance.
> Converting atomic ops to "volatile" behaviour did add ~2K to kernel text
> for archs such as i386 (possibly to important codepaths) that didn't have
> those semantics already so it would be constructive to actually look at
> those differences and see if there were really any heisenbugs that got
> rectified. Or if there were legitimate optimizations that got wrongly
> disabled. Onus lies on those proposing the modifications, I'd say ;-)

The uses of atomic_read where one might want it to allow caching of
the result seem to me to fall into 3 categories:

1. Places that are buggy because of a race arising from the way it's
   used.

2. Places where there is a race but it doesn't matter because we're
   doing some clever trick.

3. Places where there is some locking in place that eliminates any
   potential race.

In case 1, adding volatile won't solve the race, of course, but it's
hard to argue that we shouldn't do something because it will slow down
buggy code.  Case 2 is hopefully pretty rare and accompanied by large
comment blocks, and in those cases caching the result of atomic_read
explicitly in a local variable would probably make the code clearer.
And in case 3 there is no reason to use atomic_t at all; we might as
well just use an int.

So I don't see any good reason to make the atomic API more complex by
having "volatile" and "non-volatile" versions of atomic_read.  It
should just have the "volatile" behaviour.

Paul.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul Mackerras]

Herbert Xu writes:

> > You mean it's intended that *sk->sk_prot->memory_pressure can end up
> > as 1 when sk->sk_prot->memory_allocated is small (less than
> > ->sysctl_mem[0]), or as 0 when ->memory_allocated is large (greater
> > than ->sysctl_mem[2])?  Because that's the effect of the current code.
> > If so I wonder why you bother computing it.
> 
> You need to remember that there are three different limits:
> minimum, pressure, and maximum.  By default we should never
> be in a situation where what you say can occur.
> 
> If you set all three limits to the same thing, then yes it
> won't work as intended but it's still well-behaved.

I'm not talking about setting all three limits to the same thing.

I'm talking about this situation:

CPU 0 comes into __sk_stream_mem_reclaim, reads memory_allocated, but
then before it can do the store to *memory_pressure, CPUs 1-1023 all
go through sk_stream_mem_schedule, collectively increase
memory_allocated to more than sysctl_mem[2] and set *memory_pressure.
Finally CPU 0 gets to do its store and it sets *memory_pressure back
to 0, but by this stage memory_allocated is way larger than
sysctl_mem[2].

Yes, it's unlikely, but that is the nature of race conditions - they
are unlikely, and only show up at inconvenient times, never when
someone who could fix the bug is watching. :)

Similarly it would be possible for other CPUs to decrease
memory_allocated from greater than sysctl_mem[2] to less than
sysctl_mem[0] in the interval between when we read memory_allocated
and set *memory_pressure to 1.  And it's quite possible for their
setting of *memory_pressure to 0 to happen before our setting of it to
1, so that it ends up at 1 when it should be 0.

Now, maybe it's the case that it doesn't really matter whether
*->memory_pressure is 0 or 1.  But if so, why bother computing it at
all?

People seem to think that using atomic_t means they don't need to use
a spinlock.  That's fine if there is only one variable involved, but
as soon as there's more than one, there's the possibility of a race,
whether or not you use atomic_t, and whether or not atomic_read has
"volatile" behaviour.

Paul.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

Hi Bill,


On Wed, 15 Aug 2007, Bill Fink wrote:

> On Wed, 15 Aug 2007, Satyam Sharma wrote:
> 
> > (C)
> > $ cat tp3.c
> > int a;
> > 
> > void func(void)
> > {
> > 	*(volatile int *)&a = 10;
> > 	*(volatile int *)&a = 20;
> > }
> > $ gcc -Os -S tp3.c
> > $ cat tp3.s
> > ...
> > movl    $10, a
> > movl    $20, a
> > ...
> 
> I'm curious about one minor tangential point.  Why, instead of:
> 
> 	b = *(volatile int *)&a;
> 
> why can't this just be expressed as:
> 
> 	b = (volatile int)a;
> 
> Isn't it the contents of a that's volatile, i.e. it's value can change
> invisibly to the compiler, and that's why you want to force a read from
> memory?  Why do you need the "*(volatile int *)&" construct?

"b = (volatile int)a;" doesn't help us because a cast to a qualified type
has the same effect as a cast to an unqualified version of that type, as
mentioned in 6.5.4:4 (footnote 86) of the standard. Note that "volatile"
is a type-qualifier, not a type itself, so a cast of the _object_ itself
to a qualified-type i.e. (volatile int) would not make the access itself
volatile-qualified.

To serve our purposes, it is necessary for us to take the address of this
(non-volatile) object, cast the resulting _pointer_ to the corresponding
volatile-qualified pointer-type, and then dereference it. This makes that
particular _access_ be volatile-qualified, without the object itself being
such. Also note that the (dereferenced) result is also a valid lvalue and
hence can be used in "*(volatile int *)&a = b;" kind of construction
(which we use for the atomic_set case).


Satyam

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

On Thu, Aug 16, 2007 at 02:34:25PM +1000, Paul Mackerras wrote:
>
> I'm talking about this situation:
> 
> CPU 0 comes into __sk_stream_mem_reclaim, reads memory_allocated, but
> then before it can do the store to *memory_pressure, CPUs 1-1023 all
> go through sk_stream_mem_schedule, collectively increase
> memory_allocated to more than sysctl_mem[2] and set *memory_pressure.
> Finally CPU 0 gets to do its store and it sets *memory_pressure back
> to 0, but by this stage memory_allocated is way larger than
> sysctl_mem[2].

It doesn't matter.  The memory pressure flag is an *advisory*
flag.  If we get it wrong the worst that'll happen is that we'd
waste some time doing work that'll be thrown away.

Please look at the places where it's used before jumping to
conclusions.

> Now, maybe it's the case that it doesn't really matter whether
> *->memory_pressure is 0 or 1.  But if so, why bother computing it at
> all?

As long as we get it right most of the time (and I think you
would agree that we do get it right most of the time), then
this flag has achieved its purpose.

> People seem to think that using atomic_t means they don't need to use
> a spinlock.  That's fine if there is only one variable involved, but
> as soon as there's more than one, there's the possibility of a race,
> whether or not you use atomic_t, and whether or not atomic_read has
> "volatile" behaviour.

In any case, this actually illustrates why the addition of
volatile is completely pointless.  Even if this code was
broken, which it definitely is not, having the volatile
there wouldn't have helped at all.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

On Thu, Aug 16, 2007 at 02:11:43PM +1000, Paul Mackerras wrote:
>
> The uses of atomic_read where one might want it to allow caching of
> the result seem to me to fall into 3 categories:
> 
> 1. Places that are buggy because of a race arising from the way it's
>    used.
> 
> 2. Places where there is a race but it doesn't matter because we're
>    doing some clever trick.
> 
> 3. Places where there is some locking in place that eliminates any
>    potential race.

Agreed.

> In case 1, adding volatile won't solve the race, of course, but it's
> hard to argue that we shouldn't do something because it will slow down
> buggy code.  Case 2 is hopefully pretty rare and accompanied by large
> comment blocks, and in those cases caching the result of atomic_read
> explicitly in a local variable would probably make the code clearer.
> And in case 3 there is no reason to use atomic_t at all; we might as
> well just use an int.

Since adding volatile doesn't help any of the 3 cases, and
takes away optimisations from both 2 and 3, I wonder what
is the point of the addition after all?

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Thu, 16 Aug 2007, Satyam Sharma wrote:

> Hi Bill,
> 
> 
> On Wed, 15 Aug 2007, Bill Fink wrote:
> 
> > On Wed, 15 Aug 2007, Satyam Sharma wrote:
> > 
> > > (C)
> > > $ cat tp3.c
> > > int a;
> > > 
> > > void func(void)
> > > {
> > > 	*(volatile int *)&a = 10;
> > > 	*(volatile int *)&a = 20;
> > > }
> > > $ gcc -Os -S tp3.c
> > > $ cat tp3.s
> > > ...
> > > movl    $10, a
> > > movl    $20, a
> > > ...
> > 
> > I'm curious about one minor tangential point.  Why, instead of:
> > 
> > 	b = *(volatile int *)&a;
> > 
> > why can't this just be expressed as:
> > 
> > 	b = (volatile int)a;
> > 
> > Isn't it the contents of a that's volatile, i.e. it's value can change
> > invisibly to the compiler, and that's why you want to force a read from
> > memory?  Why do you need the "*(volatile int *)&" construct?
> 
> "b = (volatile int)a;" doesn't help us because a cast to a qualified type
> has the same effect as a cast to an unqualified version of that type, as
> mentioned in 6.5.4:4 (footnote 86) of the standard. Note that "volatile"
> is a type-qualifier, not a type itself, so a cast of the _object_ itself
> to a qualified-type i.e. (volatile int) would not make the access itself
> volatile-qualified.
> 
> To serve our purposes, it is necessary for us to take the address of this
> (non-volatile) object, cast the resulting _pointer_ to the corresponding
> volatile-qualified pointer-type, and then dereference it. This makes that
> particular _access_ be volatile-qualified, without the object itself being
> such. Also note that the (dereferenced) result is also a valid lvalue and
> hence can be used in "*(volatile int *)&a = b;" kind of construction
> (which we use for the atomic_set case).

Here, I should obviously admit that the semantics of *(volatile int *)&
aren't any neater or well-defined in the _language standard_ at all. The
standard does say (verbatim) "precisely what constitutes as access to
object of volatile-qualified type is implementation-defined", but GCC
does help us out here by doing the right thing. Accessing the non-volatile
object there using the volatile-qualified pointer-type cast makes GCC
treat the object stored at that memory address itself as if it were a 
volatile object, thus making the access end up having what we're calling
as "volatility" semantics here.

Honestly, given such confusion, and the propensity of the "volatile"
type-qualifier keyword to be ill-defined (or at least poorly understood,
often inconsistently implemented), I'd (again) express my opinion that it
would be best to avoid its usage, given other alternatives do exist.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul Mackerras]

Herbert Xu writes:

> It doesn't matter.  The memory pressure flag is an *advisory*
> flag.  If we get it wrong the worst that'll happen is that we'd
> waste some time doing work that'll be thrown away.

Ah, so it's the "racy but I don't care because it's only an
optimization" case.  That's fine.  Somehow I find it hard to believe
that all the racy uses of atomic_read in the kernel are like that,
though. :)

> In any case, this actually illustrates why the addition of
> volatile is completely pointless.  Even if this code was
> broken, which it definitely is not, having the volatile
> there wouldn't have helped at all.

Yes, adding volatile to racy code doesn't somehow make it race-free.
Neither does using atomic_t, despite what some seem to believe.

I have actually started going through all the uses of atomic_read in
the kernel.  So far out of the first 100 I have found none where we
have two atomic_reads of the same variable and the compiler could
usefully use the value from the first as the result of the second.
But there's still > 2500 to go...

Paul.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul Mackerras]

Herbert Xu writes:

> On Thu, Aug 16, 2007 at 02:11:43PM +1000, Paul Mackerras wrote:
> >
> > The uses of atomic_read where one might want it to allow caching of
> > the result seem to me to fall into 3 categories:
> > 
> > 1. Places that are buggy because of a race arising from the way it's
> >    used.
> > 
> > 2. Places where there is a race but it doesn't matter because we're
> >    doing some clever trick.
> > 
> > 3. Places where there is some locking in place that eliminates any
> >    potential race.
> 
> Agreed.
> 
> > In case 1, adding volatile won't solve the race, of course, but it's
> > hard to argue that we shouldn't do something because it will slow down
> > buggy code.  Case 2 is hopefully pretty rare and accompanied by large
> > comment blocks, and in those cases caching the result of atomic_read
> > explicitly in a local variable would probably make the code clearer.
> > And in case 3 there is no reason to use atomic_t at all; we might as
> > well just use an int.
> 
> Since adding volatile doesn't help any of the 3 cases, and
> takes away optimisations from both 2 and 3, I wonder what
> is the point of the addition after all?

Note that I said these are the cases _where one might want to allow
caching_, so of course adding volatile doesn't help _these_ cases.
There are of course other cases where one definitely doesn't want to
allow the compiler to cache the value, such as when polling an atomic
variable waiting for another CPU to change it, and from my inspection
so far these cases seem to be the majority.

The reasons for having "volatile" behaviour of atomic_read (whether or
not that is achieved by use of the "volatile" C keyword) are

- It matches the normal expectation based on the name "atomic_read"
- It matches the behaviour of the other atomic_* primitives
- It avoids bugs in the cases where "volatile" behaviour is required

To my mind these outweigh the small benefit for some code of the
non-volatile (caching-allowed) behaviour.  In fact it's pretty minor
either way, and since x86[-64] has this behaviour, one can expect the
potential bugs in generic code to have mostly been found, although
perhaps not all of them since x86[-64] has less aggressive reordering
of memory accesses and fewer registers in which to cache things than
some other architectures.

Paul.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

On Thu, Aug 16, 2007 at 04:56:21PM +1000, Paul Mackerras wrote:
>
> Note that I said these are the cases _where one might want to allow
> caching_, so of course adding volatile doesn't help _these_ cases.
> There are of course other cases where one definitely doesn't want to
> allow the compiler to cache the value, such as when polling an atomic
> variable waiting for another CPU to change it, and from my inspection
> so far these cases seem to be the majority.

We've been through that already.  If it's a busy-wait it
should use cpu_relax.  If it's scheduling away that already
forces the compiler to reread anyway.

Do you have an actual example where volatile is needed?

> - It matches the normal expectation based on the name "atomic_read"
> - It matches the behaviour of the other atomic_* primitives

Can't argue since you left out what those expectations
or properties are.

> - It avoids bugs in the cases where "volatile" behaviour is required

Do you (or anyone else for that matter) have an example of this?

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Stefan Richter]

Herbert Xu wrote:
> On Thu, Aug 16, 2007 at 04:56:21PM +1000, Paul Mackerras wrote:
>>
>> Note that I said these are the cases _where one might want to allow
>> caching_, so of course adding volatile doesn't help _these_ cases.
>> There are of course other cases where one definitely doesn't want to
>> allow the compiler to cache the value, such as when polling an atomic
>> variable waiting for another CPU to change it, and from my inspection
>> so far these cases seem to be the majority.
> 
> We've been through that already.  If it's a busy-wait it
> should use cpu_relax.  If it's scheduling away that already
> forces the compiler to reread anyway.
> 
> Do you have an actual example where volatile is needed?
> 
>> - It matches the normal expectation based on the name "atomic_read"
>> - It matches the behaviour of the other atomic_* primitives
> 
> Can't argue since you left out what those expectations
> or properties are.

We use atomic_t for data that is concurrently locklessly written and
read at arbitrary times.  My naive expectation as driver author (driver
maintainer) is that all atomic_t accessors, including atomic_read, (and
atomic bitops) work with the then current value of the atomic data.

>> - It avoids bugs in the cases where "volatile" behaviour is required
> 
> Do you (or anyone else for that matter) have an example of this?

The only code I somewhat know, the ieee1394 subsystem, was perhaps
authored and is currently maintained with the expectation that each
occurrence of atomic_read actually results in a load operation, i.e. is
not optimized away.  This means all atomic_t (bus generation, packet and
buffer refcounts, and some other state variables)* and likewise all
atomic bitops in that subsystem.

If that assumption is wrong, then what is the API or language primitive
to force a load operation to occur?


*)  Interesting what a quick LXR session in search for all atomic_t
usages in 'my' subsystem brings to light.  I now noticed an apparently
unused struct member in the bitrotting pcilynx driver, and more
importantly, a pairing of two atomic_t variables in raw1394 that should
be audited for race conditions and for possible replacement by plain int.
-- 
Stefan Richter
-=====-=-=== =--- =----
http://arcgraph.de/sr/

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

On Thu, Aug 16, 2007 at 10:06:31AM +0200, Stefan Richter wrote:
> > 
> > Do you (or anyone else for that matter) have an example of this?
> 
> The only code I somewhat know, the ieee1394 subsystem, was perhaps
> authored and is currently maintained with the expectation that each
> occurrence of atomic_read actually results in a load operation, i.e. is
> not optimized away.  This means all atomic_t (bus generation, packet and
> buffer refcounts, and some other state variables)* and likewise all
> atomic bitops in that subsystem.

Can you find an actual atomic_read code snippet there that is
broken without the volatile modifier?

Thanks,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

[ Bill tells me in private communication he gets this already, but I
  think it's more complicated than the shoddy explanation I'd made
  earlier so would wish to make this clearer in detail one last time,
  for the benefit of others listening in or reading the archives. ]


On Thu, 16 Aug 2007, Satyam Sharma wrote:

> On Thu, 16 Aug 2007, Satyam Sharma wrote:
> [...]
> > On Wed, 15 Aug 2007, Bill Fink wrote:
> > > [...]
> > > I'm curious about one minor tangential point.  Why, instead of:
> > > 
> > > 	b = *(volatile int *)&a;
> > > 
> > > why can't this just be expressed as:
> > > 
> > > 	b = (volatile int)a;
> > > 
> > > Isn't it the contents of a that's volatile, i.e. it's value can change
> > > invisibly to the compiler, and that's why you want to force a read from
> > > memory?  Why do you need the "*(volatile int *)&" construct?
> > 
> > "b = (volatile int)a;" doesn't help us because a cast to a qualified type
> > has the same effect as a cast to an unqualified version of that type, as
> > mentioned in 6.5.4:4 (footnote 86) of the standard. Note that "volatile"
> > is a type-qualifier, not a type itself, so a cast of the _object_ itself
> > to a qualified-type i.e. (volatile int) would not make the access itself
> > volatile-qualified.

Casts don't produce lvalues, and the cast ((volatile int)a) does not
produce the object-int-a-qualified-as-"volatile" -- in fact, the
result of the above cast is whatever is the _value_ of "int a", with
the access to that object having _already_ taken place, as per the
actual type-qualification of the object (that was originally declared
as being _non-volatile_, in fact). Hence, defining atomic_read() as:

#define atomic_read(v)          ((volatile int)((v)->counter))

would be buggy and not give "volatility" semantics at all, unless the
"counter" object itself isn't volatile-qualified already (which it
isn't).

The result of the cast itself being the _value_ of the int object, and
not the object itself (i.e., not an lvalue), is thereby independent of
type-qualification in that cast itself (it just wouldn't make any
difference), hence the "cast to a qualified type has the same effect
as a cast to an unqualified version of that type" bit in section 6.5.4:4
of the standard.


> > To serve our purposes, it is necessary for us to take the address of this
> > (non-volatile) object, cast the resulting _pointer_ to the corresponding
> > volatile-qualified pointer-type, and then dereference it. This makes that
> > particular _access_ be volatile-qualified, without the object itself being
> > such. Also note that the (dereferenced) result is also a valid lvalue and
> > hence can be used in "*(volatile int *)&a = b;" kind of construction
> > (which we use for the atomic_set case).

Dereferencing using the *(pointer-type-cast)& construct, OTOH, serves
us well:

#define atomic_read(v)          (*(volatile int *)&(v)->counter)

Firstly, note that the cast here being (volatile int *) and not
(int * volatile) qualifies the type of the _object_ being pointed to
by the pointer in question as being volatile-qualified, and not the
pointer itself (6.2.5:27 of the standard, and 6.3.2.3:2 allows us to
convert from a pointer-to-non-volatile-qualified-int to a pointer-to-
volatile-qualified-int, which suits us just fine) -- but note that
the _access_ to that address itself has not yet occurred.

_After_ specifying the memory address as containing a volatile-qualified-
int-type object, (and GCC co-operates as mentioned below), we proceed to
dereference it, which is when the _actual access_ occurs, therefore with
"volatility" semantics this time.

Interesting.


> Here, I should obviously admit that the semantics of *(volatile int *)&
> aren't any neater or well-defined in the _language standard_ at all. The
> standard does say (verbatim) "precisely what constitutes as access to
> object of volatile-qualified type is implementation-defined", but GCC
> does help us out here by doing the right thing. Accessing the non-volatile
> object there using the volatile-qualified pointer-type cast makes GCC
> treat the object stored at that memory address itself as if it were a 
> volatile object, thus making the access end up having what we're calling
> as "volatility" semantics here.
> 
> Honestly, given such confusion, and the propensity of the "volatile"
> type-qualifier keyword to be ill-defined (or at least poorly understood,
> often inconsistently implemented), I'd (again) express my opinion that it
> would be best to avoid its usage, given other alternatives do exist.


Satyam

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Stefan Richter]

Herbert Xu wrote:
> On Thu, Aug 16, 2007 at 10:06:31AM +0200, Stefan Richter wrote:
>> > 
>> > Do you (or anyone else for that matter) have an example of this?
>> 
>> The only code I somewhat know, the ieee1394 subsystem, was perhaps
>> authored and is currently maintained with the expectation that each
>> occurrence of atomic_read actually results in a load operation, i.e. is
>> not optimized away.  This means all atomic_t (bus generation, packet and
>> buffer refcounts, and some other state variables)* and likewise all
>> atomic bitops in that subsystem.
> 
> Can you find an actual atomic_read code snippet there that is
> broken without the volatile modifier?

What do I have to look for?  atomic_read after another read or write
access to the same variable, in the same function scope?  Or in the sum
of scopes of functions that could be merged by function inlining?

One example was discussed here earlier:  The for (;;) loop in
nodemgr_host_thread.  There an msleep_interruptible implicitly acted as
barrier (at the moment because it's in a different translation unit; if
it were the same, then because it hopefully has own barriers).  So that
happens to work, although such an implicit barrier is bad style:  Better
enforce the desired behaviour (== guaranteed load operation) *explicitly*.
-- 
Stefan Richter
-=====-=-=== =--- =----
http://arcgraph.de/sr/

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Stefan Richter]

I wrote:
> Herbert Xu wrote:
>> On Thu, Aug 16, 2007 at 10:06:31AM +0200, Stefan Richter wrote:
[...]
>>> expectation that each
>>> occurrence of atomic_read actually results in a load operation, i.e. is
>>> not optimized away.
[...]
>> Can you find an actual atomic_read code snippet there that is
>> broken without the volatile modifier?

PS:  Just to clarify, I'm not speaking for the volatile modifier.  I'm
not speaking for any particular implementation of atomic_t and its
accessors at all.  All I am saying is that
  - we use atomically accessed data types because we concurrently but
    locklessly access this data,
  - hence a read access to this data that could be optimized away
    makes *no sense at all*.

The only sensible read accessor to an atomic datatype is a read accessor
that will not be optimized away.

So, the architecture guys can implement atomic_read however they want
--- as long as it cannot be optimized away.*

PPS:  If somebody has code where he can afford to let the compiler
coalesce atomic_read with a previous access to the same data, i.e.
doesn't need and doesn't want all guarantees that the atomic_read API
makes (or IMO should make), then he can replace the atomic_read by a
local temporary variable.


*) Exceptions:
	if (known_to_be_false)
		read_access(a);
and the like.
-- 
Stefan Richter
-=====-=-=== =--- =----
http://arcgraph.de/sr/

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

On Thu, Aug 16, 2007 at 11:54:44AM +0200, Stefan Richter wrote:
> 
> One example was discussed here earlier:  The for (;;) loop in
> nodemgr_host_thread.  There an msleep_interruptible implicitly acted as
> barrier (at the moment because it's in a different translation unit; if
> it were the same, then because it hopefully has own barriers).  So that
> happens to work, although such an implicit barrier is bad style:  Better
> enforce the desired behaviour (== guaranteed load operation) *explicitly*.

Hmm, it's not bad style at all.  Let's assume that everything
is in the same scope.  Such a loop must either call a function
that busy-waits, which should always have a cpu_relax or
something equivalent, or it'll call a function that schedules
away which immediately invalidates any values the compiler might
have cached for the atomic_read.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

On Thu, Aug 16, 2007 at 12:31:03PM +0200, Stefan Richter wrote:
> 
> PS:  Just to clarify, I'm not speaking for the volatile modifier.  I'm
> not speaking for any particular implementation of atomic_t and its
> accessors at all.  All I am saying is that
>   - we use atomically accessed data types because we concurrently but
>     locklessly access this data,
>   - hence a read access to this data that could be optimized away
>     makes *no sense at all*.

No sane compiler can optimise away an atomic_read per se.
That's only possible if there's a preceding atomic_set or
atomic_read, with no barriers in the middle.

If that's the case, then one has to conclude that doing
away with the second read is acceptable, as otherwise
a memory (or at least a compiler) barrier should have been
used.

In fact, volatile doesn't guarantee that the memory gets
read anyway.  You might be reading some stale value out
of the cache.  Granted this doesn't happen on x86 but
when you're coding for the kernel you can't make such
assumptions.

So the point here is that if you don't mind getting a stale
value from the CPU cache when doing an atomic_read, then
surely you won't mind getting a stale value from the compiler
"cache".

> So, the architecture guys can implement atomic_read however they want
> --- as long as it cannot be optimized away.*

They can implement it however they want as long as it stays
atomic.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Ilpo Järvinen]

On Thu, 16 Aug 2007, Herbert Xu wrote:

> We've been through that already.  If it's a busy-wait it
> should use cpu_relax. 

I looked around a bit by using some command lines and ended up wondering 
if these are equal to busy-wait case (and should be fixed) or not:

./drivers/telephony/ixj.c
6674:   while (atomic_read(&j->DSPWrite) > 0)
6675-           atomic_dec(&j->DSPWrite);

...besides that, there are couple of more similar cases in the same file 
(with braces)...


-- 
 i.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Stefan Richter]

Ilpo Järvinen wrote:
> I looked around a bit by using some command lines and ended up wondering 
> if these are equal to busy-wait case (and should be fixed) or not:
> 
> ./drivers/telephony/ixj.c
> 6674:   while (atomic_read(&j->DSPWrite) > 0)
> 6675-           atomic_dec(&j->DSPWrite);
> 
> ...besides that, there are couple of more similar cases in the same file 
> (with braces)...

Generally, ixj.c has several occurrences of couples of atomic write and
atomic read which potentially do not do what the author wanted.
-- 
Stefan Richter
-=====-=-=== =--- =----
http://arcgraph.de/sr/

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Thu, Aug 16, 2007 at 06:42:50PM +0800, Herbert Xu wrote:
> On Thu, Aug 16, 2007 at 12:31:03PM +0200, Stefan Richter wrote:
> > 
> > PS:  Just to clarify, I'm not speaking for the volatile modifier.  I'm
> > not speaking for any particular implementation of atomic_t and its
> > accessors at all.  All I am saying is that
> >   - we use atomically accessed data types because we concurrently but
> >     locklessly access this data,
> >   - hence a read access to this data that could be optimized away
> >     makes *no sense at all*.
> 
> No sane compiler can optimise away an atomic_read per se.
> That's only possible if there's a preceding atomic_set or
> atomic_read, with no barriers in the middle.
> 
> If that's the case, then one has to conclude that doing
> away with the second read is acceptable, as otherwise
> a memory (or at least a compiler) barrier should have been
> used.

The compiler can also reorder non-volatile accesses.  For an example
patch that cares about this, please see:

	http://lkml.org/lkml/2007/8/7/280

This patch uses an ORDERED_WRT_IRQ() in rcu_read_lock() and
rcu_read_unlock() to ensure that accesses aren't reordered with respect
to interrupt handlers and NMIs/SMIs running on that same CPU.

> In fact, volatile doesn't guarantee that the memory gets
> read anyway.  You might be reading some stale value out
> of the cache.  Granted this doesn't happen on x86 but
> when you're coding for the kernel you can't make such
> assumptions.
> 
> So the point here is that if you don't mind getting a stale
> value from the CPU cache when doing an atomic_read, then
> surely you won't mind getting a stale value from the compiler
> "cache".

Absolutely disagree.  An interrupt/NMI/SMI handler running on the CPU
will see the same value (whether in cache or in store buffer) that
the mainline code will see.  In this case, we don't care about CPU
misordering, only about compiler misordering.  It is easy to see
other uses that combine communication with handlers on the current
CPU with communication among CPUs -- again, see prior messages in
this thread.

> > So, the architecture guys can implement atomic_read however they want
> > --- as long as it cannot be optimized away.*
> 
> They can implement it however they want as long as it stays
> atomic.

Precisely.  And volatility is a key property of "atomic".  Let's please
not throw it away.

						Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Christoph Lameter]

On Thu, 16 Aug 2007, Paul Mackerras wrote:

> 
> It seems that there could be a lot of places where atomic_t is used in
> a non-atomic fashion, and that those uses are either buggy, or there
> is some lock held at the time which guarantees that other CPUs aren't
> changing the value.  In both cases there is no point in using
> atomic_t; we might as well just use an ordinary int.

The point of atomic_t is to do atomic *changes* to the variable.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Christoph Lameter]

On Thu, 16 Aug 2007, Paul Mackerras wrote:

> Herbert Xu writes:
> 
> > It doesn't matter.  The memory pressure flag is an *advisory*
> > flag.  If we get it wrong the worst that'll happen is that we'd
> > waste some time doing work that'll be thrown away.
> 
> Ah, so it's the "racy but I don't care because it's only an
> optimization" case.  That's fine.  Somehow I find it hard to believe
> that all the racy uses of atomic_read in the kernel are like that,
> though. :)

My use of atomic_read in SLUB is like that. Volatile does not magically 
sync up reads somehow.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Christoph Lameter]

On Thu, 16 Aug 2007, Paul Mackerras wrote:

> The uses of atomic_read where one might want it to allow caching of
> the result seem to me to fall into 3 categories:
> 
> 1. Places that are buggy because of a race arising from the way it's
>    used.
> 
> 2. Places where there is a race but it doesn't matter because we're
>    doing some clever trick.
> 
> 3. Places where there is some locking in place that eliminates any
>    potential race.
> 
> In case 1, adding volatile won't solve the race, of course, but it's
> hard to argue that we shouldn't do something because it will slow down
> buggy code.  Case 2 is hopefully pretty rare and accompanied by large
> comment blocks, and in those cases caching the result of atomic_read
> explicitly in a local variable would probably make the code clearer.
> And in case 3 there is no reason to use atomic_t at all; we might as
> well just use an int.

In 2 + 3 you may increment the atomic variable in some places. The value 
of the atomic variable may not matter because you only do optimizations.

Checking a atomic_t for a definite state has to involve either
some side conditions (lock only taken if refcount is <= 0 or so) or done 
by changing the state (see f.e. atomic_inc_unless_zero).

> So I don't see any good reason to make the atomic API more complex by
> having "volatile" and "non-volatile" versions of atomic_read.  It
> should just have the "volatile" behaviour.

If you want to make it less complex then drop volatile which causes weird 
side effects without solving any problems as you just pointed out.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

>>>> Part of the motivation here is to fix heisenbugs.  If I knew where 
>>>> they
>>>
>>>
>>> By the same token we should probably disable optimisations
>>> altogether since that too can create heisenbugs.
>> Almost everything is a tradeoff; and so is this.  I don't
>> believe most people would find disabling all compiler
>> optimisations an acceptable price to pay for some peace
>> of mind.
>
> So why is this a good tradeoff?

It certainly is better than disabling all compiler optimisations!

> I also think that just adding things to APIs in the hope it might fix
> up some bugs isn't really a good road to go down. Where do you stop?

I look at it the other way: keeping the "volatile" semantics in
atomic_XXX() (or adding them to it, whatever) helps _prevent_ bugs;
certainly most people expect that behaviour, and also that behaviour
is *needed* in some places and no other interface provides that
functionality.


[some confusion about barriers wrt atomics snipped]


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

>> The only thing volatile on an asm does is create a side effect
>> on the asm statement; in effect, it tells the compiler "do not
>> remove this asm even if you don't need any of its outputs".
>>
>> It's not disabling optimisation likely to result in bugs,
>> heisen- or otherwise; _not_ putting the volatile on an asm
>> that needs it simply _is_ a bug :-)
>
> Yep.  And the reason it is a bug is that it fails to disable
> the relevant compiler optimizations.  So I suspect that we might
> actually be saying the same thing here.

We're not saying the same thing, but we do agree :-)


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

>> I can't speak for this particular case, but there could be similar 
>> code
>> examples elsewhere, where we do the atomic ops on an atomic_t object
>> inside a higher-level locking scheme that would take care of the kind 
>> of
>> problem you're referring to here. It would be useful for such or 
>> similar
>> code if the compiler kept the value of that atomic object in a 
>> register.
>
> If there is a higher-level locking scheme then there is no point to
> using atomic_t variables.  Atomic_t is specifically for the situation
> where multiple CPUs are updating a variable without locking.

And don't forget about the case where it is an I/O device that is
updating the memory (in buffer descriptors or similar).  The driver
needs to do a "volatile" atomic read to get at the most recent version
of that data, which can be important for optimising latency (or 
throughput
even).  There is no other way the kernel can get that info -- doing an
MMIO read is way way too expensive.


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

> I'd go so far as to say that anywhere where you want a non-"volatile"
> atomic_read, either your code is buggy, or else an int would work just
> as well.

Even, the only way to implement a "non-volatile" atomic_read() is
essentially as a plain int (you can do some tricks so you cannot
assign to the result and stuff like that, but that's not the issue
here).

So if that would be the behaviour we wanted, just get rid of that
whole atomic_read() thing, so no one can misuse it anymore.


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Chris Snook]

Herbert Xu wrote:
> On Thu, Aug 16, 2007 at 10:06:31AM +0200, Stefan Richter wrote:
>>> Do you (or anyone else for that matter) have an example of this?
>> The only code I somewhat know, the ieee1394 subsystem, was perhaps
>> authored and is currently maintained with the expectation that each
>> occurrence of atomic_read actually results in a load operation, i.e. is
>> not optimized away.  This means all atomic_t (bus generation, packet and
>> buffer refcounts, and some other state variables)* and likewise all
>> atomic bitops in that subsystem.
> 
> Can you find an actual atomic_read code snippet there that is
> broken without the volatile modifier?

A whole bunch of atomic_read uses will be broken without the volatile 
modifier once we start removing barriers that aren't needed if volatile 
behavior is guaranteed.

barrier() clobbers all your registers.  volatile atomic_read() only 
clobbers one register, and more often than not it's a register you 
wanted to clobber anyway.

	-- Chris

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Chris Snook]

Ilpo Järvinen wrote:
> On Thu, 16 Aug 2007, Herbert Xu wrote:
> 
>> We've been through that already.  If it's a busy-wait it
>> should use cpu_relax. 
> 
> I looked around a bit by using some command lines and ended up wondering 
> if these are equal to busy-wait case (and should be fixed) or not:
> 
> ./drivers/telephony/ixj.c
> 6674:   while (atomic_read(&j->DSPWrite) > 0)
> 6675-           atomic_dec(&j->DSPWrite);
> 
> ...besides that, there are couple of more similar cases in the same file 
> (with braces)...

atomic_dec() already has volatile behavior everywhere, so this is 
semantically okay, but this code (and any like it) should be calling 
cpu_relax() each iteration through the loop, unless there's a compelling 
reason not to.  I'll allow that for some hardware drivers (possibly this 
one) such a compelling reason may exist, but hardware-independent core 
subsystems probably have no excuse.

If the maintainer of this code doesn't see a compelling reason to add 
cpu_relax() in this loop, then it should be patched.

	-- Chris

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Chris Snook]

Ilpo Järvinen wrote:
> On Thu, 16 Aug 2007, Herbert Xu wrote:
> 
>> We've been through that already.  If it's a busy-wait it
>> should use cpu_relax. 
> 
> I looked around a bit by using some command lines and ended up wondering 
> if these are equal to busy-wait case (and should be fixed) or not:
> 
> ./drivers/telephony/ixj.c
> 6674:   while (atomic_read(&j->DSPWrite) > 0)
> 6675-           atomic_dec(&j->DSPWrite);
> 
> ...besides that, there are couple of more similar cases in the same file 
> (with braces)...

atomic_dec() already has volatile behavior everywhere, so this is 
semantically okay, but this code (and any like it) should be calling 
cpu_relax() each iteration through the loop, unless there's a compelling 
reason not to.  I'll allow that for some hardware drivers (possibly this 
one) such a compelling reason may exist, but hardware-independent core 
subsystems probably have no excuse.

If the maintainer of this code doesn't see a compelling reason not to 
add cpu_relax() in this loop, then it should be patched.

	-- Chris

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Thu, Aug 16, 2007 at 11:54:54AM -0700, Christoph Lameter wrote:
> On Thu, 16 Aug 2007, Paul Mackerras wrote:
> > So I don't see any good reason to make the atomic API more complex by
> > having "volatile" and "non-volatile" versions of atomic_read.  It
> > should just have the "volatile" behaviour.
> 
> If you want to make it less complex then drop volatile which causes weird 
> side effects without solving any problems as you just pointed out.

The other set of problems are communication between process context
and interrupt/NMI handlers.  Volatile does help here.  And the performance
impact of volatile is pretty near zero, so why have the non-volatile
variant?

						Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Christoph Lameter]

On Thu, 16 Aug 2007, Chris Snook wrote:

> atomic_dec() already has volatile behavior everywhere, so this is semantically
> okay, but this code (and any like it) should be calling cpu_relax() each
> iteration through the loop, unless there's a compelling reason not to.  I'll
> allow that for some hardware drivers (possibly this one) such a compelling
> reason may exist, but hardware-independent core subsystems probably have no
> excuse.

No it does not have any volatile semantics. atomic_dec() can be reordered 
at will by the compiler within the current basic unit if you do not add a 
barrier.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

> Note that "volatile"
> is a type-qualifier, not a type itself, so a cast of the _object_ 
> itself
> to a qualified-type i.e. (volatile int) would not make the access 
> itself
> volatile-qualified.

There is no such thing as "volatile-qualified access" defined
anywhere; there only is the concept of a "volatile-qualified
*object*".

> To serve our purposes, it is necessary for us to take the address of 
> this
> (non-volatile) object, cast the resulting _pointer_ to the 
> corresponding
> volatile-qualified pointer-type, and then dereference it. This makes 
> that
> particular _access_ be volatile-qualified, without the object itself 
> being
> such. Also note that the (dereferenced) result is also a valid lvalue 
> and
> hence can be used in "*(volatile int *)&a = b;" kind of construction
> (which we use for the atomic_set case).

There is a quite convincing argument that such an access _is_ an
access to a volatile object; see GCC PR21568 comment #9.  This
probably isn't the last word on the matter though...


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

> Here, I should obviously admit that the semantics of *(volatile int *)&
> aren't any neater or well-defined in the _language standard_ at all. 
> The
> standard does say (verbatim) "precisely what constitutes as access to
> object of volatile-qualified type is implementation-defined", but GCC
> does help us out here by doing the right thing.

Where do you get that idea?  GCC manual, section 6.1, "When
is a Volatile Object Accessed?" doesn't say anything of the
kind.  PR33053 and some others.

> Honestly, given such confusion, and the propensity of the "volatile"
> type-qualifier keyword to be ill-defined (or at least poorly 
> understood,
> often inconsistently implemented), I'd (again) express my opinion that 
> it
> would be best to avoid its usage, given other alternatives do exist.

Yeah.  Or we can have an email thread like this every time
someone proposes a patch that uses an atomic variable ;-)


Segher

RE: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Luck, Tony]

>> 6674:   while (atomic_read(&j->DSPWrite) > 0)
>> 6675-           atomic_dec(&j->DSPWrite);
>
> If the maintainer of this code doesn't see a compelling reason to add 
> cpu_relax() in this loop, then it should be patched.

Shouldn't it be just re-written without the loop:

	if ((tmp = atomic_read(&j->DSPWrite)) > 0)
		atomic_sub(&j->DSPWrite, tmp);

Has all the same bugs, but runs much faster :-)

-Tony

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

On Thu, Aug 16, 2007 at 09:34:41AM -0700, Paul E. McKenney wrote:
>
> The compiler can also reorder non-volatile accesses.  For an example
> patch that cares about this, please see:
> 
> 	http://lkml.org/lkml/2007/8/7/280
> 
> This patch uses an ORDERED_WRT_IRQ() in rcu_read_lock() and
> rcu_read_unlock() to ensure that accesses aren't reordered with respect
> to interrupt handlers and NMIs/SMIs running on that same CPU.

Good, finally we have some code to discuss (even though it's
not actually in the kernel yet).

First of all, I think this illustrates that what you want
here has nothing to do with atomic ops.  The ORDERED_WRT_IRQ
macro occurs a lot more times in your patch than atomic
reads/sets.  So *assuming* that it was necessary at all,
then having an ordered variant of the atomic_read/atomic_set
ops could do just as well.

However, I still don't know which atomic_read/atomic_set in
your patch would be broken if there were no volatile.  Could
you please point them out?

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

On Thu, Aug 16, 2007 at 03:48:54PM -0400, Chris Snook wrote:
>
> >Can you find an actual atomic_read code snippet there that is
> >broken without the volatile modifier?
> 
> A whole bunch of atomic_read uses will be broken without the volatile 
> modifier once we start removing barriers that aren't needed if volatile 
> behavior is guaranteed.

Could you please cite the file/function names so we can
see whether removing the barrier makes sense?

Thanks,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Fri, Aug 17, 2007 at 07:59:02AM +0800, Herbert Xu wrote:
> On Thu, Aug 16, 2007 at 09:34:41AM -0700, Paul E. McKenney wrote:
> >
> > The compiler can also reorder non-volatile accesses.  For an example
> > patch that cares about this, please see:
> > 
> > 	http://lkml.org/lkml/2007/8/7/280
> > 
> > This patch uses an ORDERED_WRT_IRQ() in rcu_read_lock() and
> > rcu_read_unlock() to ensure that accesses aren't reordered with respect
> > to interrupt handlers and NMIs/SMIs running on that same CPU.
> 
> Good, finally we have some code to discuss (even though it's
> not actually in the kernel yet).

There was some earlier in this thread as well.

> First of all, I think this illustrates that what you want
> here has nothing to do with atomic ops.  The ORDERED_WRT_IRQ
> macro occurs a lot more times in your patch than atomic
> reads/sets.  So *assuming* that it was necessary at all,
> then having an ordered variant of the atomic_read/atomic_set
> ops could do just as well.

Indeed.  If I could trust atomic_read()/atomic_set() to cause the compiler
to maintain ordering, then I could just use them instead of having to
create an  ORDERED_WRT_IRQ().  (Or ACCESS_ONCE(), as it is called in a
different patch.)

> However, I still don't know which atomic_read/atomic_set in
> your patch would be broken if there were no volatile.  Could
> you please point them out?

Suppose I tried replacing the ORDERED_WRT_IRQ() calls with
atomic_read() and atomic_set().  Starting with __rcu_read_lock():

o	If "ORDERED_WRT_IRQ(__get_cpu_var(rcu_flipctr)[idx])++"
	was ordered by the compiler after
	"ORDERED_WRT_IRQ(me->rcu_read_lock_nesting) = nesting + 1", then
	suppose an NMI/SMI happened after the rcu_read_lock_nesting but
	before the rcu_flipctr.

	Then if there was an rcu_read_lock() in the SMI/NMI
	handler (which is perfectly legal), the nested rcu_read_lock()
	would believe that it could take the then-clause of the
	enclosing "if" statement.  But because the rcu_flipctr per-CPU
	variable had not yet been incremented, an RCU updater would
	be within its rights to assume that there were no RCU reads
	in progress, thus possibly yanking a data structure out from
	under the reader in the SMI/NMI function.

	Fatal outcome.  Note that only one CPU is involved here
	because these are all either per-CPU or per-task variables.

o	If "ORDERED_WRT_IRQ(me->rcu_read_lock_nesting) = nesting + 1"
	was ordered by the compiler to follow the
	"ORDERED_WRT_IRQ(me->rcu_flipctr_idx) = idx", and an NMI/SMI
	happened between the two, then an __rcu_read_lock() in the NMI/SMI
	would incorrectly take the "else" clause of the enclosing "if"
	statement.  If some other CPU flipped the rcu_ctrlblk.completed
	in the meantime, then the __rcu_read_lock() would (correctly)
	write the new value into rcu_flipctr_idx.

	Well and good so far.  But the problem arises in
	__rcu_read_unlock(), which then decrements the wrong counter.
	Depending on exactly how subsequent events played out, this could
	result in either prematurely ending grace periods or never-ending
	grace periods, both of which are fatal outcomes.

And the following are not needed in the current version of the
patch, but will be in a future version that either avoids disabling
irqs or that dispenses with the smp_read_barrier_depends() that I
have 99% convinced myself is unneeded:

o	nesting = ORDERED_WRT_IRQ(me->rcu_read_lock_nesting);

o	idx = ORDERED_WRT_IRQ(rcu_ctrlblk.completed) & 0x1;

Furthermore, in that future version, irq handlers can cause the same
mischief that SMI/NMI handlers can in this version.

Next, looking at __rcu_read_unlock():

o	If "ORDERED_WRT_IRQ(me->rcu_read_lock_nesting) = nesting - 1"
	was reordered by the compiler to follow the
	"ORDERED_WRT_IRQ(__get_cpu_var(rcu_flipctr)[idx])--",
	then if an NMI/SMI containing an rcu_read_lock() occurs between
	the two, this nested rcu_read_lock() would incorrectly believe
	that it was protected by an enclosing RCU read-side critical
	section as described in the first reversal discussed for
	__rcu_read_lock() above.  Again, fatal outcome.

This is what we have now.  It is not hard to imagine situations that
interact with -both- interrupt handlers -and- other CPUs, as described
earlier.

							Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Thu, Aug 16, 2007 at 01:20:26PM -0700, Christoph Lameter wrote:
> On Thu, 16 Aug 2007, Chris Snook wrote:
> 
> > atomic_dec() already has volatile behavior everywhere, so this is semantically
> > okay, but this code (and any like it) should be calling cpu_relax() each
> > iteration through the loop, unless there's a compelling reason not to.  I'll
> > allow that for some hardware drivers (possibly this one) such a compelling
> > reason may exist, but hardware-independent core subsystems probably have no
> > excuse.
> 
> No it does not have any volatile semantics. atomic_dec() can be reordered 
> at will by the compiler within the current basic unit if you do not add a 
> barrier.

Yep.  Or you can use atomic_dec_return() instead of using a barrier.

						Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

On Thu, Aug 16, 2007 at 06:02:32PM -0700, Paul E. McKenney wrote:
> 
> Yep.  Or you can use atomic_dec_return() instead of using a barrier.

Or you could use smp_mb__{before,after}_atomic_dec.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Chris Snook]

Herbert Xu wrote:
> On Thu, Aug 16, 2007 at 03:48:54PM -0400, Chris Snook wrote:
>>> Can you find an actual atomic_read code snippet there that is
>>> broken without the volatile modifier?
>> A whole bunch of atomic_read uses will be broken without the volatile 
>> modifier once we start removing barriers that aren't needed if volatile 
>> behavior is guaranteed.
> 
> Could you please cite the file/function names so we can
> see whether removing the barrier makes sense?
> 
> Thanks,

At a glance, several architectures' implementations of smp_call_function() have 
one or more legitimate atomic_read() busy-waits that shouldn't be using 
CPU-relax.  Some of them do work in the loop.

I'm sure there are plenty more examples that various maintainers could find in 
their own code.

	-- Chris

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

On Thu, Aug 16, 2007 at 10:04:24PM -0400, Chris Snook wrote:
>
> >Could you please cite the file/function names so we can
> >see whether removing the barrier makes sense?
> 
> At a glance, several architectures' implementations of smp_call_function() 
> have one or more legitimate atomic_read() busy-waits that shouldn't be 
> using CPU-relax.  Some of them do work in the loop.

Care to name one so we can discuss it?

Thanks,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul Mackerras]

Christoph Lameter writes:

> No it does not have any volatile semantics. atomic_dec() can be reordered 
> at will by the compiler within the current basic unit if you do not add a 
> barrier.

Volatile doesn't mean it can't be reordered; volatile means the
accesses can't be eliminated.

Paul.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Nick Piggin]

Segher Boessenkool wrote:
>>>>> Part of the motivation here is to fix heisenbugs.  If I knew where 
>>>>> they
>>>>
>>>>
>>>>
>>>> By the same token we should probably disable optimisations
>>>> altogether since that too can create heisenbugs.
>>>
>>> Almost everything is a tradeoff; and so is this.  I don't
>>> believe most people would find disabling all compiler
>>> optimisations an acceptable price to pay for some peace
>>> of mind.
>>
>>
>> So why is this a good tradeoff?
> 
> 
> It certainly is better than disabling all compiler optimisations!

It's easy to be better than something really stupid :)

So i386 and x86-64 don't have volatiles there, and it saves them a
few K of kernel text. What you need to justify is why it is a good
tradeoff to make them volatile (which btw, is much harder to go
the other way after we let people make those assumptions).


>> I also think that just adding things to APIs in the hope it might fix
>> up some bugs isn't really a good road to go down. Where do you stop?
> 
> 
> I look at it the other way: keeping the "volatile" semantics in
> atomic_XXX() (or adding them to it, whatever) helps _prevent_ bugs;

Yeah, but we could add lots of things to help prevent bugs and
would never be included. I would also contend that it helps _hide_
bugs and encourages people to be lazy when thinking about these
things.

Also, you dismiss the fact that we'd actually be *adding* volatile
semantics back to the 2 most widely tested architectures (in terms
of test time, number of testers, variety of configurations, and
coverage of driver code). This is a very important different from
just keeping volatile semantics because it is basically a one-way
API change.


> certainly most people expect that behaviour, and also that behaviour
> is *needed* in some places and no other interface provides that
> functionality.

I don't know that most people would expect that behaviour. Is there
any documentation anywhere that would suggest this?

Also, barrier() most definitely provides the required functionality.
It is overkill in some situations, but volatile is overkill in _most_
situations. If that's what you're worried about, we should add a new
ordering primitive.


> [some confusion about barriers wrt atomics snipped]

What were you confused about?

-- 
SUSE Labs, Novell Inc.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Nick Piggin]

Chris Snook wrote:
> Herbert Xu wrote:
> 
>> On Thu, Aug 16, 2007 at 03:48:54PM -0400, Chris Snook wrote:
>>
>>>> Can you find an actual atomic_read code snippet there that is
>>>> broken without the volatile modifier?
>>>
>>> A whole bunch of atomic_read uses will be broken without the volatile 
>>> modifier once we start removing barriers that aren't needed if 
>>> volatile behavior is guaranteed.
>>
>>
>> Could you please cite the file/function names so we can
>> see whether removing the barrier makes sense?
>>
>> Thanks,
> 
> 
> At a glance, several architectures' implementations of 
> smp_call_function() have one or more legitimate atomic_read() busy-waits 
> that shouldn't be using CPU-relax.  Some of them do work in the loop.

sh looks like the only one there that would be broken (and that's only
because they don't have a cpu_relax there, but it should be added anyway).
Sure, if we removed volatile from other architectures, it would be wise
to audit arch code because arch maintainers do sometimes make assumptions
about their implementation details... however we can assume most generic
code is safe without volatile.

-- 
SUSE Labs, Novell Inc.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Linus Torvalds]

On Fri, 17 Aug 2007, Paul Mackerras wrote:
>
> Volatile doesn't mean it can't be reordered; volatile means the
> accesses can't be eliminated.

It also does limit re-ordering. 

Of course, since *normal* accesses aren't necessarily limited wrt 
re-ordering, the question then becomes one of "with regard to *what* does 
it limit re-ordering?".

A C compiler that re-orders two different volatile accesses that have a 
sequence point in between them is pretty clearly a buggy compiler. So at a 
minimum, it limits re-ordering wrt other volatiles (assuming sequence 
points exists). It also means that the compiler cannot move it 
speculatively across conditionals, but other than that it's starting to 
get fuzzy.

In general, I'd *much* rather we used barriers. Anything that "depends" on 
volatile is pretty much set up to be buggy. But I'm certainly also willing 
to have that volatile inside "atomic_read/atomic_set()" if it avoids code 
that would otherwise break - ie if it hides a bug.

		Linus

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Nick Piggin]

Paul E. McKenney wrote:
> On Thu, Aug 16, 2007 at 06:42:50PM +0800, Herbert Xu wrote:

>>In fact, volatile doesn't guarantee that the memory gets
>>read anyway.  You might be reading some stale value out
>>of the cache.  Granted this doesn't happen on x86 but
>>when you're coding for the kernel you can't make such
>>assumptions.
>>
>>So the point here is that if you don't mind getting a stale
>>value from the CPU cache when doing an atomic_read, then
>>surely you won't mind getting a stale value from the compiler
>>"cache".
> 
> 
> Absolutely disagree.  An interrupt/NMI/SMI handler running on the CPU
> will see the same value (whether in cache or in store buffer) that
> the mainline code will see.  In this case, we don't care about CPU
> misordering, only about compiler misordering.  It is easy to see
> other uses that combine communication with handlers on the current
> CPU with communication among CPUs -- again, see prior messages in
> this thread.

I still don't agree with the underlying assumption that everybody
(or lots of kernel code) treats atomic accesses as volatile.

Nobody that does has managed to explain my logic problem either:
loads and stores to long and ptr have always been considered to be
atomic, test_bit is atomic; so why are another special subclass of
atomic loads and stores? (and yes, it is perfectly legitimate to
want a non-volatile read for a data type that you also want to do
atomic RMW operations on)

Why are people making these undocumented and just plain false
assumptions about atomic_t? If they're using lockless code (ie.
which they must be if using atomics), then they actually need to be
thinking much harder about memory ordering issues. If that is too
much for them, then they can just use locks.


>>>So, the architecture guys can implement atomic_read however they want
>>>--- as long as it cannot be optimized away.*
>>
>>They can implement it however they want as long as it stays
>>atomic.
> 
> 
> Precisely.  And volatility is a key property of "atomic".  Let's please
> not throw it away.

It isn't, though (at least not since i386 and x86-64 don't have it).
_Adding_ it is trivial, and can be done any time. Throwing it away
(ie. making the API weaker) is _hard_. So let's not add it without
really good reasons. It most definitely results in worse code
generation in practice.

I don't know why people would assume volatile of atomics. AFAIK, most
of the documentation is pretty clear that all the atomic stuff can be
reordered etc. except for those that modify and return a value.

It isn't even intuitive: `*lp = value` is like the most fundamental
atomic operation in Linux.

-- 
SUSE Labs, Novell Inc.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul Mackerras]

Nick Piggin writes:

> So i386 and x86-64 don't have volatiles there, and it saves them a
> few K of kernel text. What you need to justify is why it is a good

I'm really surprised it's as much as a few K.  I tried it on powerpc
and it only saved 40 bytes (10 instructions) for a G5 config.

Paul.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Nick Piggin]

Paul Mackerras wrote:
> Nick Piggin writes:
> 
> 
>>So i386 and x86-64 don't have volatiles there, and it saves them a
>>few K of kernel text. What you need to justify is why it is a good
> 
> 
> I'm really surprised it's as much as a few K.  I tried it on powerpc
> and it only saved 40 bytes (10 instructions) for a G5 config.
> 
> Paul.
> 

I'm surprised too. Numbers were from the "...use asm() like the other
atomic operations already do" thread. According to them,

   text    data     bss     dec     hex filename
3434150  249176  176128 3859454  3ae3fe atomic_normal/vmlinux
3436203  249176  176128 3861507  3aec03 atomic_volatile/vmlinux

The first one is a stock kenel, the second is with atomic_read/set
cast to volatile. gcc-4.1 -- maybe if you have an earlier gcc it
won't optimise as much?

-- 
SUSE Labs, Novell Inc.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Linus Torvalds]

On Fri, 17 Aug 2007, Paul Mackerras wrote:
> 
> I'm really surprised it's as much as a few K.  I tried it on powerpc
> and it only saved 40 bytes (10 instructions) for a G5 config.

One of the things that "volatile" generally screws up is a simple

	volatile int i;

	i++;

which a compiler will generally get horribly, horribly wrong.

In a reasonable world, gcc should just make that be (on x86)

	addl $1,i(%rip)

on x86-64, which is indeed what it does without the volatile. But with the 
volatile, the compiler gets really nervous, and doesn't dare do it in one 
instruction, and thus generates crap like

        movl    i(%rip), %eax
        addl    $1, %eax
        movl    %eax, i(%rip)

instead. For no good reason, except that "volatile" just doesn't have any 
good/clear semantics for the compiler, so most compilers will just make it 
be "I will not touch this access in any way, shape, or form". Including 
even trivially correct instruction optimization/combination.

This is one of the reasons why we should never use "volatile". It 
pessimises code generation for no good reason - just because compilers 
don't know what the heck it even means! 

Now, people don't do "i++" on atomics (you'd use "atomic_inc()" for that), 
but people *do* do things like

	if (atomic_read(..) <= 1)
		..

On ppc, things like that probably don't much matter. But on x86, it makes 
a *huge* difference whether you do

	movl i(%rip),%eax
	cmpl $1,%eax

or if you can just use the value directly for the operation, like this:

	cmpl $1,i(%rip)

which is again a totally obvious and totally safe optimization, but is 
(again) something that gcc doesn't dare do, since "i" is volatile.

In other words: "volatile" is a horribly horribly bad way of doing things, 
because it generates *worse*code*, for no good reason. You just don't see 
it on powerpc, because it's already a load-store architecture, so there is 
no "good code" for doing direct-to-memory operations.

		Linus

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul Mackerras]

Linus Torvalds writes:

> In general, I'd *much* rather we used barriers. Anything that "depends" on 
> volatile is pretty much set up to be buggy. But I'm certainly also willing 
> to have that volatile inside "atomic_read/atomic_set()" if it avoids code 
> that would otherwise break - ie if it hides a bug.

The cost of doing so seems to me to be well down in the noise - 44
bytes of extra kernel text on a ppc64 G5 config, and I don't believe
the extra few cycles for the occasional extra load would be measurable
(they should all hit in the L1 dcache).  I don't mind if x86[-64] have
atomic_read/set be nonvolatile and find all the missing barriers, but
for now on powerpc, I think that not having to find those missing
barriers is worth the 0.00076% increase in kernel text size.

Paul.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Linus Torvalds]

On Fri, 17 Aug 2007, Nick Piggin wrote:
> 
> I'm surprised too. Numbers were from the "...use asm() like the other
> atomic operations already do" thread. According to them,
> 
>   text    data     bss     dec     hex filename
> 3434150  249176  176128 3859454  3ae3fe atomic_normal/vmlinux
> 3436203  249176  176128 3861507  3aec03 atomic_volatile/vmlinux
> 
> The first one is a stock kenel, the second is with atomic_read/set
> cast to volatile. gcc-4.1 -- maybe if you have an earlier gcc it
> won't optimise as much?

No, see my earlier reply. "volatile" really *is* an incredible piece of 
crap.

Just try it yourself:

	volatile int i;
	int j;

	int testme(void)
	{
	        return i <= 1;
	}

	int testme2(void)
	{
	        return j <= 1;
	}

and compile with all the optimizations you can.

I get:

	testme:
	        movl    i(%rip), %eax
	        subl    $1, %eax
	        setle   %al
	        movzbl  %al, %eax
	        ret

vs

	testme2:
	        xorl    %eax, %eax
	        cmpl    $1, j(%rip)
	        setle   %al
	        ret

(now, whether that "xorl + setle" is better than "setle + movzbl", I don't 
really know - maybe it is. But that's not thepoint. The point is the 
difference between

                movl    i(%rip), %eax
                subl    $1, %eax

and

                cmpl    $1, j(%rip)

and imagine this being done for *every* single volatile access.

Just do a 

	git grep atomic_read

to see how atomics are actually used. A lot of them are exactly the above 
kind of "compare against a value".

			Linus

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

On Fri, Aug 17, 2007 at 01:43:27PM +1000, Paul Mackerras wrote:
>
> The cost of doing so seems to me to be well down in the noise - 44
> bytes of extra kernel text on a ppc64 G5 config, and I don't believe
> the extra few cycles for the occasional extra load would be measurable
> (they should all hit in the L1 dcache).  I don't mind if x86[-64] have
> atomic_read/set be nonvolatile and find all the missing barriers, but
> for now on powerpc, I think that not having to find those missing
> barriers is worth the 0.00076% increase in kernel text size.

BTW, the sort of missing barriers that triggered this thread
aren't that subtle.  It'll result in a simple lock-up if the
loop condition holds upon entry.  At which point it's fairly
straightforward to find the culprit.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul Mackerras]

Nick Piggin writes:

> Why are people making these undocumented and just plain false
> assumptions about atomic_t?

Well, it has only been false since December 2006.  Prior to that
atomics *were* volatile on all platforms.

> If they're using lockless code (ie.
> which they must be if using atomics), then they actually need to be
> thinking much harder about memory ordering issues.

Indeed.  I believe that most uses of atomic_read other than in polling
loops or debug printk statements are actually racy.  In some cases the
race doesn't seem to matter, but I'm sure there are cases where it
does.

> If that is too
> much for them, then they can just use locks.

Why use locks when you can just sprinkle magic fix-the-races dust (aka
atomic_t) over your code? :) :)

> > Precisely.  And volatility is a key property of "atomic".  Let's please
> > not throw it away.
> 
> It isn't, though (at least not since i386 and x86-64 don't have it).

Conceptually it is, because atomic_t is specifically for variables
which are liable to be modified by other CPUs, and volatile _means_
"liable to be changed by mechanisms outside the knowledge of the
compiler".

> _Adding_ it is trivial, and can be done any time. Throwing it away
> (ie. making the API weaker) is _hard_. So let's not add it without

Well, in one sense it's not that hard - Linus did it just 8 months ago
in commit f9e9dcb3. :)

> really good reasons. It most definitely results in worse code
> generation in practice.

0.0008% increase in kernel text size on powerpc according to my
measurement. :)

> I don't know why people would assume volatile of atomics. AFAIK, most

By making something an atomic_t you're saying "other CPUs are going to
be modifying this, so treat it specially".  It's reasonable to assume
that special treatment extends to reading and setting it.

> of the documentation is pretty clear that all the atomic stuff can be
> reordered etc. except for those that modify and return a value.

Volatility isn't primarily about reordering (though as Linus says it
does restrict reordering to some extent).

Paul.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Thu, 16 Aug 2007, Segher Boessenkool wrote:

> > Note that "volatile"
> > is a type-qualifier, not a type itself, so a cast of the _object_ itself
> > to a qualified-type i.e. (volatile int) would not make the access itself
> > volatile-qualified.
> 
> There is no such thing as "volatile-qualified access" defined
> anywhere; there only is the concept of a "volatile-qualified
> *object*".

Sure, "volatile-qualified access" was not some standard term I used
there. Just something to mean "an access that would make the compiler
treat the object at that memory as if it were an object with a
volatile-qualified type".

Now the second wording *IS* technically correct, but come on, it's
24 words long whereas the original one was 3 -- and hopefully anybody
reading the shorter phrase *would* have known anyway what was meant,
without having to be pedantic about it :-)


Satyam

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Thu, 16 Aug 2007, Segher Boessenkool wrote:

> > Here, I should obviously admit that the semantics of *(volatile int *)&
> > aren't any neater or well-defined in the _language standard_ at all. The
> > standard does say (verbatim) "precisely what constitutes as access to
> > object of volatile-qualified type is implementation-defined", but GCC
> > does help us out here by doing the right thing.
> 
> Where do you get that idea?

Try a testcase (experimentally verify).

> GCC manual, section 6.1, "When
> is a Volatile Object Accessed?" doesn't say anything of the
> kind.

True, "implementation-defined" as per the C standard _is_ supposed to mean
"unspecified behaviour where each implementation documents how the choice
is made". So ok, probably GCC isn't "documenting" this
implementation-defined behaviour which it is supposed to, but can't really
fault them much for this, probably.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Nick Piggin]

Paul Mackerras wrote:
> Nick Piggin writes:
> 
> 
>>Why are people making these undocumented and just plain false
>>assumptions about atomic_t?
> 
> 
> Well, it has only been false since December 2006.  Prior to that
> atomics *were* volatile on all platforms.

Hmm, although I don't think it has ever been guaranteed by the
API documentation (concede documentation is often not treated
as the authoritative source here, but for atomic it is actually
very good and obviously indispensable as the memory ordering
reference).


>>If they're using lockless code (ie.
>>which they must be if using atomics), then they actually need to be
>>thinking much harder about memory ordering issues.
> 
> 
> Indeed.  I believe that most uses of atomic_read other than in polling
> loops or debug printk statements are actually racy.  In some cases the
> race doesn't seem to matter, but I'm sure there are cases where it
> does.
> 
> 
>>If that is too
>>much for them, then they can just use locks.
> 
> 
> Why use locks when you can just sprinkle magic fix-the-races dust (aka
> atomic_t) over your code? :) :)

I agree with your skepticism of a lot of lockless code. But I think
a lot of the more subtle race problems will not be fixed with volatile.
The big, dumb infinite loop bugs would be fixed, but they're pretty
trivial to debug and even audit for.


>>>Precisely.  And volatility is a key property of "atomic".  Let's please
>>>not throw it away.
>>
>>It isn't, though (at least not since i386 and x86-64 don't have it).
> 
> 
> Conceptually it is, because atomic_t is specifically for variables
> which are liable to be modified by other CPUs, and volatile _means_
> "liable to be changed by mechanisms outside the knowledge of the
> compiler".

Usually that is the case, yes. But also most of the time we don't
care that it has been changed and don't mind it being reordered or
eliminated.

One of the only places we really care about that at all is for
variables that are modified by the *same* CPU.


>>_Adding_ it is trivial, and can be done any time. Throwing it away
>>(ie. making the API weaker) is _hard_. So let's not add it without
> 
> 
> Well, in one sense it's not that hard - Linus did it just 8 months ago
> in commit f9e9dcb3. :)

Well it would have been harder if the documentation also guaranteed
that atomic_read/atomic_set was ordered. Or it would have been harder
for _me_ to make such a change, anyway ;)


>>really good reasons. It most definitely results in worse code
>>generation in practice.
> 
> 
> 0.0008% increase in kernel text size on powerpc according to my
> measurement. :)

I don't think you're making a bad choice by keeping it volatile on
powerpc and waiting for others to shake out more of the bugs. You
get to fix everybody else's memory ordering bugs :)


>>I don't know why people would assume volatile of atomics. AFAIK, most
> 
> 
> By making something an atomic_t you're saying "other CPUs are going to
> be modifying this, so treat it specially".  It's reasonable to assume
> that special treatment extends to reading and setting it.

But I don't actually know what that "special treatment" is. Well
actually, I do know that operations will never result in a partial
modification being exposed. I also know that the operators that
do not modify and return are not guaranteed to have any sort of
ordering constraints.

-- 
SUSE Labs, Novell Inc.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul Mackerras]

Herbert Xu writes:

> So the point here is that if you don't mind getting a stale
> value from the CPU cache when doing an atomic_read, then
> surely you won't mind getting a stale value from the compiler
> "cache".

No, that particular argument is bogus, because there is a cache
coherency protocol operating to keep the CPU cache coherent with
stores from other CPUs, but there isn't any such protocol (nor should
there be) for a register used as a "cache".

(Linux requires SMP systems to keep any CPU caches coherent as far as
accesses by other CPUs are concerned.  It doesn't support any SMP
systems that are not cache-coherent as far as CPU accesses are
concerned.  It does support systems with non-cache-coherent DMA.)

Paul.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Fri, Aug 17, 2007 at 09:28:00AM +0800, Herbert Xu wrote:
> On Thu, Aug 16, 2007 at 06:02:32PM -0700, Paul E. McKenney wrote:
> > 
> > Yep.  Or you can use atomic_dec_return() instead of using a barrier.
> 
> Or you could use smp_mb__{before,after}_atomic_dec.

Yep.  That would be an example of a barrier, either in the
atomic_dec() itself or in the smp_mb...().

							Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul Mackerras]

Herbert Xu writes:

> Can you find an actual atomic_read code snippet there that is
> broken without the volatile modifier?

There are some in arch-specific code, for example line 1073 of
arch/mips/kernel/smtc.c.  On mips, cpu_relax() is just barrier(), so
the empty loop body is ok provided that atomic_read actually does the
load each time around the loop.

Paul.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Thu, Aug 16, 2007 at 08:42:23PM -0700, Linus Torvalds wrote:
> 
> 
> On Fri, 17 Aug 2007, Paul Mackerras wrote:
> > 
> > I'm really surprised it's as much as a few K.  I tried it on powerpc
> > and it only saved 40 bytes (10 instructions) for a G5 config.
> 
> One of the things that "volatile" generally screws up is a simple
> 
> 	volatile int i;
> 
> 	i++;
> 
> which a compiler will generally get horribly, horribly wrong.
> 
> In a reasonable world, gcc should just make that be (on x86)
> 
> 	addl $1,i(%rip)
> 
> on x86-64, which is indeed what it does without the volatile. But with the 
> volatile, the compiler gets really nervous, and doesn't dare do it in one 
> instruction, and thus generates crap like
> 
>         movl    i(%rip), %eax
>         addl    $1, %eax
>         movl    %eax, i(%rip)

Blech.  Sounds like a chat with some gcc people is in order.  Will
see what I can do.

						Thanx, Paul

> instead. For no good reason, except that "volatile" just doesn't have any 
> good/clear semantics for the compiler, so most compilers will just make it 
> be "I will not touch this access in any way, shape, or form". Including 
> even trivially correct instruction optimization/combination.
> 
> This is one of the reasons why we should never use "volatile". It 
> pessimises code generation for no good reason - just because compilers 
> don't know what the heck it even means! 
> 
> Now, people don't do "i++" on atomics (you'd use "atomic_inc()" for that), 
> but people *do* do things like
> 
> 	if (atomic_read(..) <= 1)
> 		..
> 
> On ppc, things like that probably don't much matter. But on x86, it makes 
> a *huge* difference whether you do
> 
> 	movl i(%rip),%eax
> 	cmpl $1,%eax
> 
> or if you can just use the value directly for the operation, like this:
> 
> 	cmpl $1,i(%rip)
> 
> which is again a totally obvious and totally safe optimization, but is 
> (again) something that gcc doesn't dare do, since "i" is volatile.
> 
> In other words: "volatile" is a horribly horribly bad way of doing things, 
> because it generates *worse*code*, for no good reason. You just don't see 
> it on powerpc, because it's already a load-store architecture, so there is 
> no "good code" for doing direct-to-memory operations.
> 
> 		Linus
> -
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

On Fri, Aug 17, 2007 at 03:09:57PM +1000, Paul Mackerras wrote:
> Herbert Xu writes:
> 
> > Can you find an actual atomic_read code snippet there that is
> > broken without the volatile modifier?
> 
> There are some in arch-specific code, for example line 1073 of
> arch/mips/kernel/smtc.c.  On mips, cpu_relax() is just barrier(), so
> the empty loop body is ok provided that atomic_read actually does the
> load each time around the loop.

A barrier() is all you need to force the compiler to reread
the value.

The people advocating volatile in this thread are talking
about code that doesn't use barrier()/cpu_relax().

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul Mackerras]

Herbert Xu writes:

> On Fri, Aug 17, 2007 at 03:09:57PM +1000, Paul Mackerras wrote:
> > Herbert Xu writes:
> > 
> > > Can you find an actual atomic_read code snippet there that is
> > > broken without the volatile modifier?
> > 
> > There are some in arch-specific code, for example line 1073 of
> > arch/mips/kernel/smtc.c.  On mips, cpu_relax() is just barrier(), so
> > the empty loop body is ok provided that atomic_read actually does the
> > load each time around the loop.
> 
> A barrier() is all you need to force the compiler to reread
> the value.
> 
> The people advocating volatile in this thread are talking
> about code that doesn't use barrier()/cpu_relax().

Did you look at it?  Here it is:

	/* Someone else is initializing in parallel - let 'em finish */
	while (atomic_read(&idle_hook_initialized) < 1000)
		;

Paul.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

Hi Linus,

[ and others; I think there's a communication gap in a lot of this
  thread, and a little summary would be useful. Hence this posting. ]


On Thu, 16 Aug 2007, Linus Torvalds wrote:

> On Fri, 17 Aug 2007, Paul Mackerras wrote:
> > 
> > I'm really surprised it's as much as a few K.  I tried it on powerpc
> > and it only saved 40 bytes (10 instructions) for a G5 config.
> 
> One of the things that "volatile" generally screws up is a simple
> 
> 	volatile int i;
> 
> 	i++;
> 
> which a compiler will generally get horribly, horribly wrong.
> 
> [...] For no good reason, except that "volatile" just doesn't have any 
> good/clear semantics for the compiler, so most compilers will just make it 
> be "I will not touch this access in any way, shape, or form". Including 
> even trivially correct instruction optimization/combination.
> 
> This is one of the reasons why we should never use "volatile". It 
> pessimises code generation for no good reason - just because compilers 
> don't know what the heck it even means! 
> [...]
> In other words: "volatile" is a horribly horribly bad way of doing things, 
> because it generates *worse*code*, for no good reason. You just don't see 
> it on powerpc, because it's already a load-store architecture, so there is 
> no "good code" for doing direct-to-memory operations.


True, and I bet *everybody* on this list has already agreed for a very
long time that using "volatile" to type-qualify the _declaration_ of an
object itself as being horribly bad (taste-wise, code-generation-wise,
often even buggy for sitations where real CPU barriers should've been
used instead).

However, the discussion on this thread (IIRC) began with only "giving
volatility semantics" to atomic ops. Now that is different, and may not
require the use the "volatile" keyword (at least not in the declaration
of the object) itself.

Sadly, most arch's *still* do type-qualify the declaration of the
"counter" member of atomic_t as "volatile". This is probably a historic
hangover, and I suspect not yet rectified because of lethargy.

Anyway, some of the variants I can think of are:

[1]

#define atomic_read_volatile(v)				\
	({						\
		forget((v)->counter);			\
		((v)->counter);				\
	})

where:

#define forget(a)	__asm__ __volatile__ ("" :"=m" (a) :"m" (a))

[ This is exactly equivalent to using "+m" in the constraints, as recently
  explained on a GCC list somewhere, in response to the patch in my bitops
  series a few weeks back where I thought "+m" was bogus. ]

[2]

#define atomic_read_volatile(v)		(*(volatile int *)&(v)->counter)

This is something that does work. It has reasonably good semantics
guaranteed by the C standard in conjunction with how GCC currently
behaves (and how it has behaved for all supported versions). I haven't
checked if generates much different code than the first variant above,
(it probably will generate similar code to just declaring the object
as volatile, but would still be better in terms of code-clarity and
taste, IMHO), but in any case, we should pick whichever of these variants
works for us and generates good code.

[3]

static inline int atomic_read_volatile(atomic_t *v)
{
	... arch-dependent __asm__ __volatile__ stuff ...
}

I can reasonably bet this variant would often generate worse code than
at least the variant "[1]" above.


Now, why do we even require these "volatility" semantics variants?

Note, "volatility" semantics *know* / assume that it can have a meaning
_only_ as far as the compiler, so atomic_read_volatile() doesn't really
care reading stale values from the cache for certain non-x86 archs, etc.

The first argument is "safety":

Use of atomic_read() (possibly in conjunction with other atomic ops) in
a lot of code out there in the kernel *assumes* the compiler will not
optimize away those ops. (which is possible given current definitions
of atomic_set and atomic_read on archs such as x86 in present code).
An additional argument that builds on this one says that by ensuring
the compiler will not elid or coalesce these ops, we could even avoid
potential heisenbugs in the future.

However, there is a counter-argument:

As Herbert Xu has often been making the point, there is *no* bug out
there involving "atomic_read" in busy-while-loops that should not have
a compiler barrier (or cpu_relax() in fact) anyway. As for non-busy-loops,
they would invariable call schedule() at some point (possibly directly)
and thus have an "implicit" compiler barrier by virtue of calling out
a function that is not in scope of the current compilation unit (although
users in sched.c itself would probably require an explicit compiler
barrier).

The second pro-volatility-in-atomic-ops argument is performance:
(surprise!)

Using a full memory clobber compiler barrier in busy loops will disqualify
optimizations for loop invariants so it probably makes sense to *only*
make the compiler forget *that* particular address of the atomic counter
object, and none other. All 3 variants above would work nicely here.

So the final idea may be to have a cpu_relax_no_barrier() variant as a
rep;nop (pause) *without* an included full memory clobber, and replace
a lot of kernel busy-while-loops out there with:

-	cpu_relax();
+	cpu_relax_no_barrier();
+	forget(x);

or may be just:

-	cpu_relax();
+	cpu_relax_no_barrier();

because the "forget" / "volatility" / specific-variable-compiler-barrier
could be made implicit inside the atomic ops themselves.

This could especially make a difference for register-rich CPUs (probably
not x86) where using a full memory clobber will disqualify a hell lot of
compiler optimizations for loop-invariants.

On x86 itself, cpu_relax_no_barrier() could be:

#define cpu_relax_no_barrier()	__asm__ __volatile__ ("rep;nop":::);

and still continue to do its job as it is doing presently.

However, there is still a counter-argument:

As Herbert Xu and Christoph Lameter have often been saying, giving
"volatility" semantics to the atomic ops will disqualify compiler
optimizations such as eliding / coalescing of atomic ops, etc, and
probably some sections of code in the kernel (Christoph mentioned code
in SLUB, and I saw such code in sched) benefit from such optimizations.

Paul Mackerras has, otoh, mentioned that a lot of such places probably
don't need (or shouldn't use) atomic ops in the first place.
Alternatively, such callsites should probably just cache the atomic_read
in a local variable (which compiler could just as well make a register)
explicitly, and repeating atomic_read() isn't really necessary.

There could still be legitimate uses of atomic ops that don't care about
them being elided / coalesced, but given the loop-invariant-optimization
benefit, personally, I do see some benefit in the use of defining atomic
ops variants with "volatility" semantics (for only the atomic counter
object) but also having a non-volatile atomic ops API side-by-side for
performance critical users (probably sched, slub) that may require that.

Possibly, one of the two APIs above could turn out to be redundant, but
that's still very much the issue of debate presently.


Satyam

[ Sorry if I missed anything important, but this thread has been long
  and noisy, although I've tried to keep up ... ]

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Fri, 17 Aug 2007, Herbert Xu wrote:

> On Fri, Aug 17, 2007 at 01:43:27PM +1000, Paul Mackerras wrote:
> >
> > The cost of doing so seems to me to be well down in the noise - 44
> > bytes of extra kernel text on a ppc64 G5 config, and I don't believe
> > the extra few cycles for the occasional extra load would be measurable
> > (they should all hit in the L1 dcache).  I don't mind if x86[-64] have
> > atomic_read/set be nonvolatile and find all the missing barriers, but
> > for now on powerpc, I think that not having to find those missing
> > barriers is worth the 0.00076% increase in kernel text size.
> 
> BTW, the sort of missing barriers that triggered this thread
> aren't that subtle.  It'll result in a simple lock-up if the
> loop condition holds upon entry.  At which point it's fairly
> straightforward to find the culprit.

Not necessarily. A barrier-less buggy code such as below:

	atomic_set(&v, 0);

	... /* some initial code */

	while (atomic_read(&v))
		;

	... /* code that MUST NOT be executed unless v becomes non-zero */

(where v->counter is has no volatile access semantics)

could be generated by the compiler to simply *elid* or *do away* with
the loop itself, thereby making the:

"/* code that MUST NOT be executed unless v becomes non-zero */"

to be executed even when v is zero! That is subtle indeed, and causes
no hard lockups.

Granted, the above IS buggy code. But, the stated objective is to avoid
heisenbugs. And we have driver / subsystem maintainers such as Stefan
coming up and admitting that often a lot of code that's written to use
atomic_read() does assume the read will not be elided by the compiler.

See, I agree, "volatility" semantics != what we often want. However, if
what we want is compiler barrier, for only the object under consideration,
"volatility" semantics aren't really "nonsensical" or anything.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Geert Uytterhoeven]

On Thu, 16 Aug 2007, Linus Torvalds wrote:
> On Fri, 17 Aug 2007, Paul Mackerras wrote:
> > I'm really surprised it's as much as a few K.  I tried it on powerpc
> > and it only saved 40 bytes (10 instructions) for a G5 config.
> 
> One of the things that "volatile" generally screws up is a simple
> 
> 	volatile int i;
> 
> 	i++;
> 
> which a compiler will generally get horribly, horribly wrong.
> 
> In a reasonable world, gcc should just make that be (on x86)
> 
> 	addl $1,i(%rip)
> 
> on x86-64, which is indeed what it does without the volatile. But with the 
> volatile, the compiler gets really nervous, and doesn't dare do it in one 
> instruction, and thus generates crap like
> 
>         movl    i(%rip), %eax
>         addl    $1, %eax
>         movl    %eax, i(%rip)
> 
> instead. For no good reason, except that "volatile" just doesn't have any 
> good/clear semantics for the compiler, so most compilers will just make it 
> be "I will not touch this access in any way, shape, or form". Including 
> even trivially correct instruction optimization/combination.

Apart from having to fetch more bytes for the instructions (which does
matter), execution time is probably the same on modern processors, as they
convert the single instruction to RISC-style load, modify, store anyway.

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Stefan Richter]

Nick Piggin wrote:
> I don't know why people would assume volatile of atomics. AFAIK, most
> of the documentation is pretty clear that all the atomic stuff can be
> reordered etc. except for those that modify and return a value.

Which documentation is there?

For driver authors, there is LDD3.  It doesn't specifically cover
effects of optimization on accesses to atomic_t.

For architecture port authors, there is Documentation/atomic_ops.txt.
Driver authors also can learn something from that document, as it
indirectly documents the atomic_t and bitops APIs.

Prompted by this thread, I reread this document, and indeed, the
sentence "Unlike the above routines, it is required that explicit memory
barriers are performed before and after [atomic_{inc,dec}_return]"
indicates that atomic_read (one of the "above routines") is very
different from all other atomic_t accessors that return values.

This is strange.  Why is it that atomic_read stands out that way?  IMO
this API imbalance is quite unexpected by many people.  Wouldn't it be
beneficial to change the atomic_read API to behave the same like all
other atomic_t accessors that return values?

OK, it is also different from the other accessors that return data in so
far as it doesn't modify the data.  But as driver "author", i.e. user of
the API, I can't see much use of an atomic_read that can be reordered
and, more importantly, can be optimized away by the compiler.  Sure, now
that I learned of these properties I can start to audit code and insert
barriers where I believe they are needed, but this simply means that
almost all occurrences of atomic_read will get barriers (unless there
already are implicit but more or less obvious barriers like msleep).
-- 
Stefan Richter
-=====-=-=== =--- =---=
http://arcgraph.de/sr/

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Nick Piggin]

Satyam Sharma wrote:

> #define atomic_read_volatile(v)				\
> 	({						\
> 		forget((v)->counter);			\
> 		((v)->counter);				\
> 	})
> 
> where:

*vomit* :)

Not only do I hate the keyword volatile, but the barrier is only a
one-sided affair so its probable this is going to have slightly
different allowed reorderings than a real volatile access.

Also, why would you want to make these insane accessors for atomic_t
types? Just make sure everybody knows the basics of barriers, and they
can apply that knowledge to atomic_t and all other lockless memory
accesses as well.


> #define forget(a)	__asm__ __volatile__ ("" :"=m" (a) :"m" (a))

I like order(x) better, but it's not the most perfect name either.

-- 
SUSE Labs, Novell Inc.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Thu, 16 Aug 2007, Paul E. McKenney wrote:

> On Fri, Aug 17, 2007 at 07:59:02AM +0800, Herbert Xu wrote:
> > On Thu, Aug 16, 2007 at 09:34:41AM -0700, Paul E. McKenney wrote:
> > >
> > > The compiler can also reorder non-volatile accesses.  For an example
> > > patch that cares about this, please see:
> > > 
> > > 	http://lkml.org/lkml/2007/8/7/280
> > > 
> > > This patch uses an ORDERED_WRT_IRQ() in rcu_read_lock() and
> > > rcu_read_unlock() to ensure that accesses aren't reordered with respect
> > > to interrupt handlers and NMIs/SMIs running on that same CPU.
> > 
> > Good, finally we have some code to discuss (even though it's
> > not actually in the kernel yet).
> 
> There was some earlier in this thread as well.

Hmm, I never quite got what all this interrupt/NMI/SMI handling and
RCU business you mentioned earlier was all about, but now that you've
pointed to the actual code and issues with it ...


> > First of all, I think this illustrates that what you want
> > here has nothing to do with atomic ops.  The ORDERED_WRT_IRQ
> > macro occurs a lot more times in your patch than atomic
> > reads/sets.  So *assuming* that it was necessary at all,
> > then having an ordered variant of the atomic_read/atomic_set
> > ops could do just as well.
> 
> Indeed.  If I could trust atomic_read()/atomic_set() to cause the compiler
> to maintain ordering, then I could just use them instead of having to
> create an  ORDERED_WRT_IRQ().  (Or ACCESS_ONCE(), as it is called in a
> different patch.)

+#define WHATEVER(x)	(*(volatile typeof(x) *)&(x))

I suppose one could want volatile access semantics for stuff that's
a bit-field too, no?

Also, this gives *zero* "re-ordering" guarantees that your code wants
as you've explained it below) -- neither w.r.t. CPU re-ordering (which
probably you don't care about) *nor* w.r.t. compiler re-ordering
(which you definitely _do_ care about).


> > However, I still don't know which atomic_read/atomic_set in
> > your patch would be broken if there were no volatile.  Could
> > you please point them out?
> 
> Suppose I tried replacing the ORDERED_WRT_IRQ() calls with
> atomic_read() and atomic_set().  Starting with __rcu_read_lock():
> 
> o	If "ORDERED_WRT_IRQ(__get_cpu_var(rcu_flipctr)[idx])++"
> 	was ordered by the compiler after
> 	"ORDERED_WRT_IRQ(me->rcu_read_lock_nesting) = nesting + 1", then
> 	suppose an NMI/SMI happened after the rcu_read_lock_nesting but
> 	before the rcu_flipctr.
> 
> 	Then if there was an rcu_read_lock() in the SMI/NMI
> 	handler (which is perfectly legal), the nested rcu_read_lock()
> 	would believe that it could take the then-clause of the
> 	enclosing "if" statement.  But because the rcu_flipctr per-CPU
> 	variable had not yet been incremented, an RCU updater would
> 	be within its rights to assume that there were no RCU reads
> 	in progress, thus possibly yanking a data structure out from
> 	under the reader in the SMI/NMI function.
> 
> 	Fatal outcome.  Note that only one CPU is involved here
> 	because these are all either per-CPU or per-task variables.

Ok, so you don't care about CPU re-ordering. Still, I should let you know
that your ORDERED_WRT_IRQ() -- bad name, btw -- is still buggy. What you
want is a full compiler optimization barrier().

[ Your code probably works now, and emits correct code, but that's
  just because of gcc did what it did. Nothing in any standard,
  or in any documented behaviour of gcc, or anything about the real
  (or expected) semantics of "volatile" is protecting the code here. ]


> o	If "ORDERED_WRT_IRQ(me->rcu_read_lock_nesting) = nesting + 1"
> 	was ordered by the compiler to follow the
> 	"ORDERED_WRT_IRQ(me->rcu_flipctr_idx) = idx", and an NMI/SMI
> 	happened between the two, then an __rcu_read_lock() in the NMI/SMI
> 	would incorrectly take the "else" clause of the enclosing "if"
> 	statement.  If some other CPU flipped the rcu_ctrlblk.completed
> 	in the meantime, then the __rcu_read_lock() would (correctly)
> 	write the new value into rcu_flipctr_idx.
> 
> 	Well and good so far.  But the problem arises in
> 	__rcu_read_unlock(), which then decrements the wrong counter.
> 	Depending on exactly how subsequent events played out, this could
> 	result in either prematurely ending grace periods or never-ending
> 	grace periods, both of which are fatal outcomes.
> 
> And the following are not needed in the current version of the
> patch, but will be in a future version that either avoids disabling
> irqs or that dispenses with the smp_read_barrier_depends() that I
> have 99% convinced myself is unneeded:
> 
> o	nesting = ORDERED_WRT_IRQ(me->rcu_read_lock_nesting);
> 
> o	idx = ORDERED_WRT_IRQ(rcu_ctrlblk.completed) & 0x1;
> 
> Furthermore, in that future version, irq handlers can cause the same
> mischief that SMI/NMI handlers can in this version.
> 
> Next, looking at __rcu_read_unlock():
> 
> o	If "ORDERED_WRT_IRQ(me->rcu_read_lock_nesting) = nesting - 1"
> 	was reordered by the compiler to follow the
> 	"ORDERED_WRT_IRQ(__get_cpu_var(rcu_flipctr)[idx])--",
> 	then if an NMI/SMI containing an rcu_read_lock() occurs between
> 	the two, this nested rcu_read_lock() would incorrectly believe
> 	that it was protected by an enclosing RCU read-side critical
> 	section as described in the first reversal discussed for
> 	__rcu_read_lock() above.  Again, fatal outcome.
> 
> This is what we have now.  It is not hard to imagine situations that
> interact with -both- interrupt handlers -and- other CPUs, as described
> earlier.

It's not about interrupt/SMI/NMI handlers at all! What you clearly want,
simply put, is that a certain stream of C statements must be emitted
by the compiler _as they are_ with no re-ordering optimizations! You must
*definitely* use barrier(), IMHO.


Satyam

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Nick Piggin]

Stefan Richter wrote:
> Nick Piggin wrote:
> 
>>I don't know why people would assume volatile of atomics. AFAIK, most
>>of the documentation is pretty clear that all the atomic stuff can be
>>reordered etc. except for those that modify and return a value.
> 
> 
> Which documentation is there?

Documentation/atomic_ops.txt


> For driver authors, there is LDD3.  It doesn't specifically cover
> effects of optimization on accesses to atomic_t.
> 
> For architecture port authors, there is Documentation/atomic_ops.txt.
> Driver authors also can learn something from that document, as it
> indirectly documents the atomic_t and bitops APIs.
>

"Semantics and Behavior of Atomic and Bitmask Operations" is
pretty direct :)

Sure, it says that it's for arch maintainers, but there is no
reason why users can't make use of it.


> Prompted by this thread, I reread this document, and indeed, the
> sentence "Unlike the above routines, it is required that explicit memory
> barriers are performed before and after [atomic_{inc,dec}_return]"
> indicates that atomic_read (one of the "above routines") is very
> different from all other atomic_t accessors that return values.
> 
> This is strange.  Why is it that atomic_read stands out that way?  IMO

It is not just atomic_read of course. It is atomic_add,sub,inc,dec,set.


> this API imbalance is quite unexpected by many people.  Wouldn't it be
> beneficial to change the atomic_read API to behave the same like all
> other atomic_t accessors that return values?

It is very consistent and well defined. Operations which both modify
the data _and_ return something are defined to have full barriers
before and after.

What do you want to add to the other atomic accessors? Full memory
barriers? Only compiler barriers? It's quite likely that if you think
some barriers will fix bugs, then there are other bugs lurking there
anyway.

Just use spinlocks if you're not absolutely clear about potential
races and memory ordering issues -- they're pretty cheap and simple.


> OK, it is also different from the other accessors that return data in so
> far as it doesn't modify the data.  But as driver "author", i.e. user of
> the API, I can't see much use of an atomic_read that can be reordered
> and, more importantly, can be optimized away by the compiler.

It will return to you an atomic snapshot of the data (loaded from
memory at some point since the last compiler barrier). All you have
to be aware of compiler barriers and the Linux SMP memory ordering
model, which should be a given if you are writing lockless code.


> Sure, now
> that I learned of these properties I can start to audit code and insert
> barriers where I believe they are needed, but this simply means that
> almost all occurrences of atomic_read will get barriers (unless there
> already are implicit but more or less obvious barriers like msleep).

You might find that these places that appear to need barriers are
buggy for other reasons anyway. Can you point to some in-tree code
we can have a look at?

-- 
SUSE Labs, Novell Inc.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Fri, 17 Aug 2007, Paul Mackerras wrote:

> Herbert Xu writes:
> 
> > On Fri, Aug 17, 2007 at 03:09:57PM +1000, Paul Mackerras wrote:
> > > Herbert Xu writes:
> > > 
> > > > Can you find an actual atomic_read code snippet there that is
> > > > broken without the volatile modifier?
> > > 
> > > There are some in arch-specific code, for example line 1073 of
> > > arch/mips/kernel/smtc.c.  On mips, cpu_relax() is just barrier(), so
> > > the empty loop body is ok provided that atomic_read actually does the
> > > load each time around the loop.
> > 
> > A barrier() is all you need to force the compiler to reread
> > the value.
> > 
> > The people advocating volatile in this thread are talking
> > about code that doesn't use barrier()/cpu_relax().
> 
> Did you look at it?  Here it is:
> 
> 	/* Someone else is initializing in parallel - let 'em finish */
> 	while (atomic_read(&idle_hook_initialized) < 1000)
> 		;


Honestly, this thread is suffering from HUGE communication gaps.

What Herbert (obviously) meant there was that "this loop could've
been okay _without_ using volatile-semantics-atomic_read() also, if
only it used cpu_relax()".

That does work, because cpu_relax() is _at least_ barrier() on all
archs (on some it also emits some arch-dependent "pause" kind of
instruction).

Now, saying that "MIPS does not have such an instruction so I won't
use cpu_relax() for arch-dependent-busy-while-loops in arch/mips/"
sounds like a wrong argument, because: tomorrow, such arch's _may_
introduce such an instruction, so naturally, at that time we'd
change cpu_relax() appropriately (in reality, we would actually
*re-define* cpu_relax() and ensure that the correct version gets
pulled in depending on whether the callsite code is legacy or only
for the newer such CPUs of said arch, whatever), but loops such as
this would remain un-changed, because they never used cpu_relax()!

OTOH an argument that said the following would've made a stronger case:

"I don't want to use cpu_relax() because that's a full memory
clobber barrier() and I have loop-invariants / other variables
around in that code that I *don't* want the compiler to forget
just because it used cpu_relax(), and hence I will not use
cpu_relax() but instead make my atomic_read() itself have
"volatility" semantics. Not just that, but I will introduce a
cpu_relax_no_barrier() on MIPS, that would be a no-op #define
for now, but which may not be so forever, and continue to use
that in such busy loops."

In general, please read the thread-summary I've tried to do at:
http://lkml.org/lkml/2007/8/17/25
Feel free to continue / comment / correct stuff from there, there's
too much confusion and circular-arguments happening on this thread
otherwise.

[ I might've made an incorrect statement there about
  "volatile" w.r.t. cache on non-x86 archs, I think. ]


Satyam

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Nick Piggin]

Satyam Sharma wrote:
> 
> On Fri, 17 Aug 2007, Herbert Xu wrote:
> 
> 
>>On Fri, Aug 17, 2007 at 01:43:27PM +1000, Paul Mackerras wrote:
>>
>>BTW, the sort of missing barriers that triggered this thread
>>aren't that subtle.  It'll result in a simple lock-up if the
>>loop condition holds upon entry.  At which point it's fairly
>>straightforward to find the culprit.
> 
> 
> Not necessarily. A barrier-less buggy code such as below:
> 
> 	atomic_set(&v, 0);
> 
> 	... /* some initial code */
> 
> 	while (atomic_read(&v))
> 		;
> 
> 	... /* code that MUST NOT be executed unless v becomes non-zero */
> 
> (where v->counter is has no volatile access semantics)
> 
> could be generated by the compiler to simply *elid* or *do away* with
> the loop itself, thereby making the:
> 
> "/* code that MUST NOT be executed unless v becomes non-zero */"
> 
> to be executed even when v is zero! That is subtle indeed, and causes
> no hard lockups.

Then I presume you mean

while (!atomic_read(&v))
     ;

Which is just the same old infinite loop bug solved with cpu_relax().
These are pretty trivial to audit and fix, and also to debug, I would
think.


> Granted, the above IS buggy code. But, the stated objective is to avoid
> heisenbugs.

Anyway, why are you making up code snippets that are buggy in other
ways in order to support this assertion being made that lots of kernel
code supposedly depends on volatile semantics. Just reference the
actual code.


> And we have driver / subsystem maintainers such as Stefan
> coming up and admitting that often a lot of code that's written to use
> atomic_read() does assume the read will not be elided by the compiler.

So these are broken on i386 and x86-64?

Are they definitely safe on SMP and weakly ordered machines with
just a simple compiler barrier there? Because I would not be
surprised if there are a lot of developers who don't really know
what to assume when it comes to memory ordering issues.

This is not a dig at driver writers: we still have memory ordering
problems in the VM too (and probably most of the subtle bugs in
lockless VM code are memory ordering ones). Let's not make up a
false sense of security and hope that sprinkling volatile around
will allow people to write bug-free lockless code. If a writer
can't be bothered reading API documentation and learning the Linux
memory model, they can still be productive writing safely locked
code.

-- 
SUSE Labs, Novell Inc.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Fri, 17 Aug 2007, Nick Piggin wrote:

> Satyam Sharma wrote:
> 
> > #define atomic_read_volatile(v)				\
> > 	({						\
> > 		forget((v)->counter);			\
> > 		((v)->counter);				\
> > 	})
> > 
> > where:
> 
> *vomit* :)

I wonder if this'll generate smaller and better code than _both_ the
other atomic_read_volatile() variants. Would need to build allyesconfig
on lots of diff arch's etc to test the theory though.


> Not only do I hate the keyword volatile, but the barrier is only a
> one-sided affair so its probable this is going to have slightly
> different allowed reorderings than a real volatile access.

True ...


> Also, why would you want to make these insane accessors for atomic_t
> types? Just make sure everybody knows the basics of barriers, and they
> can apply that knowledge to atomic_t and all other lockless memory
> accesses as well.

Code that looks like:

	while (!atomic_read(&v)) {
		...
		cpu_relax_no_barrier();
		forget(v.counter);
		        ^^^^^^^^
	}

would be uglier. Also think about code such as:

	a = atomic_read();
	if (!a)
		do_something();

	forget();
	a = atomic_read();
	... /* some code that depends on value of a, obviously */

	forget();
	a = atomic_read();
	...

So much explicit sprinkling of "forget()" looks ugly.

	atomic_read_volatile()

on the other hand, looks neater. The "_volatile()" suffix makes it also
no less explicit than an explicit barrier-like macro that this primitive
is something "special", for code clarity purposes.


> > #define forget(a)	__asm__ __volatile__ ("" :"=m" (a) :"m" (a))
> 
> I like order(x) better, but it's not the most perfect name either.

forget(x) is just a stupid-placeholder-for-a-better-name. order(x) sounds
good but we could leave quibbling about function or macro names for later,
this thread is noisy as it is :-)

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Andi Kleen]

On Friday 17 August 2007 05:42, Linus Torvalds wrote:
> On Fri, 17 Aug 2007, Paul Mackerras wrote:
> > I'm really surprised it's as much as a few K.  I tried it on powerpc
> > and it only saved 40 bytes (10 instructions) for a G5 config.
>
> One of the things that "volatile" generally screws up is a simple
>
> 	volatile int i;
>
> 	i++;

But for atomic_t people use atomic_inc() anyways which does this correctly.
It shouldn't really matter for atomic_t.

I'm worrying a bit that the volatile atomic_t change caused subtle code 
breakage like these delay read loops people here pointed out.
Wouldn't it be safer to just re-add the volatile to atomic_read() 
for 2.6.23? Or alternatively make it asm(), but volatile seems more
proven.

-Andi

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Fri, 17 Aug 2007, Nick Piggin wrote:

> Stefan Richter wrote:
> [...]
> Just use spinlocks if you're not absolutely clear about potential
> races and memory ordering issues -- they're pretty cheap and simple.

I fully agree with this. As Paul Mackerras mentioned elsewhere,
a lot of authors sprinkle atomic_t in code thinking they're somehow
done with *locking*. This is sad, and I wonder if it's time for a
Documentation/atomic-considered-dodgy.txt kind of document :-)


> > Sure, now
> > that I learned of these properties I can start to audit code and insert
> > barriers where I believe they are needed, but this simply means that
> > almost all occurrences of atomic_read will get barriers (unless there
> > already are implicit but more or less obvious barriers like msleep).
> 
> You might find that these places that appear to need barriers are
> buggy for other reasons anyway. Can you point to some in-tree code
> we can have a look at?

Such code was mentioned elsewhere (query nodemgr_host_thread in cscope)
that managed to escape the requirement for a barrier only because of
some completely un-obvious compilation-unit-scope thing. But I find such
an non-explicit barrier quite bad taste. Stefan, do consider plunking an
explicit call to barrier() there.


Satyam

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Fri, 17 Aug 2007, Nick Piggin wrote:

> Satyam Sharma wrote:
> [...]
> > Granted, the above IS buggy code. But, the stated objective is to avoid
> > heisenbugs.
    ^^^^^^^^^^

> Anyway, why are you making up code snippets that are buggy in other
> ways in order to support this assertion being made that lots of kernel
> code supposedly depends on volatile semantics. Just reference the
> actual code.

Because the point is *not* about existing bugs in kernel code. At some
point Chris Snook (who started this thread) did write that "If I knew
of the existing bugs in the kernel, I would be sending patches for them,
not this series" or something to that effect.

The point is about *author expecations*. If people do expect atomic_read()
(or a variant thereof) to have volatile semantics, why not give them such
a variant?

And by the way, the point is *also* about the fact that cpu_relax(), as
of today, implies a full memory clobber, which is not what a lot of such
loops want. (due to stuff mentioned elsewhere, summarized in that summary)


> > And we have driver / subsystem maintainers such as Stefan
> > coming up and admitting that often a lot of code that's written to use
> > atomic_read() does assume the read will not be elided by the compiler.
                                                             ^^^^^^^^^^^^^

(so it's about compiler barrier expectations only, though I fully agree
that those who're using atomic_t as if it were some magic thing that lets
them write lockless code are sorrily mistaken.)

> So these are broken on i386 and x86-64?

Possibly, but the point is not about existing bugs, as mentioned above.

Some such bugs have been found nonetheless -- reminds me, can somebody
please apply http://www.gossamer-threads.com/lists/linux/kernel/810674 ?


Satyam

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Nick Piggin]

Satyam Sharma wrote:
> 
> On Fri, 17 Aug 2007, Nick Piggin wrote:

>>Also, why would you want to make these insane accessors for atomic_t
>>types? Just make sure everybody knows the basics of barriers, and they
>>can apply that knowledge to atomic_t and all other lockless memory
>>accesses as well.
> 
> 
> Code that looks like:
> 
> 	while (!atomic_read(&v)) {
> 		...
> 		cpu_relax_no_barrier();
> 		forget(v.counter);
> 		        ^^^^^^^^
> 	}
> 
> would be uglier. Also think about code such as:

I think they would both be equally ugly, but the atomic_read_volatile
variant would be more prone to subtle bugs because of the weird
implementation.

And it would be more ugly than introducing an order(x) statement for
all memory operations, and adding an order_atomic() wrapper for it
for atomic types.


> 	a = atomic_read();
> 	if (!a)
> 		do_something();
> 
> 	forget();
> 	a = atomic_read();
> 	... /* some code that depends on value of a, obviously */
> 
> 	forget();
> 	a = atomic_read();
> 	...
> 
> So much explicit sprinkling of "forget()" looks ugly.

Firstly, why is it ugly? It's nice because of those nice explicit
statements there that give us a good heads up and would have some
comments attached to them (also, lack of the word "volatile" is
always a plus).

Secondly, what sort of code would do such a thing? In most cases,
it is probably riddled with bugs anyway (unless it is doing a
really specific sequence of interrupts or something, but in that
case it is very likely to either require locking or busy waits
anyway -> ie. barriers).


> on the other hand, looks neater. The "_volatile()" suffix makes it also
> no less explicit than an explicit barrier-like macro that this primitive
> is something "special", for code clarity purposes.

Just don't use the word volatile, and have barriers both before
and after the memory operation, and I'm OK with it. I don't see
the point though, when you could just have a single barrier(x)
barrier function defined for all memory locations, rather than
this odd thing that only works for atomics (and would have to
be duplicated for atomic_set.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Nick Piggin]

Satyam Sharma wrote:
> 
> On Fri, 17 Aug 2007, Nick Piggin wrote:

>>>Sure, now
>>>that I learned of these properties I can start to audit code and insert
>>>barriers where I believe they are needed, but this simply means that
>>>almost all occurrences of atomic_read will get barriers (unless there
>>>already are implicit but more or less obvious barriers like msleep).
>>
>>You might find that these places that appear to need barriers are
>>buggy for other reasons anyway. Can you point to some in-tree code
>>we can have a look at?
> 
> 
> Such code was mentioned elsewhere (query nodemgr_host_thread in cscope)
> that managed to escape the requirement for a barrier only because of
> some completely un-obvious compilation-unit-scope thing. But I find such
> an non-explicit barrier quite bad taste. Stefan, do consider plunking an
> explicit call to barrier() there.

It is very obvious. msleep calls schedule() (ie. sleeps), which is
always a barrier.

The "unobvious" thing is that you wanted to know how the compiler knows
a function is a barrier -- answer is that if it does not *know* it is not
a barrier, it must assume it is a barrier. If the whole msleep call chain
including the scheduler were defined static in the current compilation
unit, then it would still be a barrier because it would actually be able
to see the barriers in schedule(void), if nothing else.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Nick Piggin]

Satyam Sharma wrote:
> 
> On Fri, 17 Aug 2007, Nick Piggin wrote:
> 
> 
>>Satyam Sharma wrote:
>>[...]
>>
>>>Granted, the above IS buggy code. But, the stated objective is to avoid
>>>heisenbugs.
> 
>     ^^^^^^^^^^
> 
> 
>>Anyway, why are you making up code snippets that are buggy in other
>>ways in order to support this assertion being made that lots of kernel
>>code supposedly depends on volatile semantics. Just reference the
>>actual code.
> 
> 
> Because the point is *not* about existing bugs in kernel code. At some
> point Chris Snook (who started this thread) did write that "If I knew
> of the existing bugs in the kernel, I would be sending patches for them,
> not this series" or something to that effect.
> 
> The point is about *author expecations*. If people do expect atomic_read()
> (or a variant thereof) to have volatile semantics, why not give them such
> a variant?

Because they should be thinking about them in terms of barriers, over
which the compiler / CPU is not to reorder accesses or cache memory
operations, rather than "special" "volatile" accesses. Linux's whole
memory ordering and locking model is completely geared around the
former.


> And by the way, the point is *also* about the fact that cpu_relax(), as
> of today, implies a full memory clobber, which is not what a lot of such
> loops want. (due to stuff mentioned elsewhere, summarized in that summary)

That's not the point, because as I also mentioned, the logical extention
to Linux's barrier API to handle this is the order(x) macro. Again, not
special volatile accessors.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul Mackerras]

Satyam Sharma writes:

> I wonder if this'll generate smaller and better code than _both_ the
> other atomic_read_volatile() variants. Would need to build allyesconfig
> on lots of diff arch's etc to test the theory though.

I'm sure it would be a tiny effect.

This whole thread is arguing about effects that are quite
insignificant.  On the one hand we have the non-volatile proponents,
who want to let the compiler do extra optimizations - which amounts to
letting it elide maybe a dozen loads in the whole kernel, loads which
would almost always be L1 cache hits.

On the other hand we have the volatile proponents, who are concerned
that some code somewhere in the kernel might be buggy without the
volatile behaviour, and who also want to be able to remove some
barriers and thus save a few bytes of code and a few loads here and
there (and possibly some stores too).

Either way the effect on code size and execution time is miniscule.

In the end the strongest argument is actually that gcc generates
unnecessarily verbose code on x86[-64] for volatile accesses.  Even
then we're only talking about ~2000 bytes, or less than 1 byte per
instance of atomic_read on average, about 0.06% of the kernel text
size.

The x86[-64] developers seem to be willing to bear the debugging cost
involved in having the non-volatile behaviour for atomic_read, in
order to save the 2kB.  That's fine with me.  Either way I think
somebody should audit all the uses of atomic_read, not just for
missing barriers, but also to find the places where it's used in a
racy manner.  Then we can work out where the races matter and fix them
if they do.

Paul.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Fri, 17 Aug 2007, Nick Piggin wrote:

> Satyam Sharma wrote:
> > 
> > On Fri, 17 Aug 2007, Nick Piggin wrote:
> 
> > > > Sure, now
> > > > that I learned of these properties I can start to audit code and insert
> > > > barriers where I believe they are needed, but this simply means that
> > > > almost all occurrences of atomic_read will get barriers (unless there
> > > > already are implicit but more or less obvious barriers like msleep).
> > > 
> > > You might find that these places that appear to need barriers are
> > > buggy for other reasons anyway. Can you point to some in-tree code
> > > we can have a look at?
> > 
> > 
> > Such code was mentioned elsewhere (query nodemgr_host_thread in cscope)
> > that managed to escape the requirement for a barrier only because of
> > some completely un-obvious compilation-unit-scope thing. But I find such
> > an non-explicit barrier quite bad taste. Stefan, do consider plunking an
> > explicit call to barrier() there.
> 
> It is very obvious. msleep calls schedule() (ie. sleeps), which is
> always a barrier.

Probably you didn't mean that, but no, schedule() is not barrier because
it sleeps. It's a barrier because it's invisible.

> The "unobvious" thing is that you wanted to know how the compiler knows
> a function is a barrier -- answer is that if it does not *know* it is not
> a barrier, it must assume it is a barrier.

True, that's clearly what happens here. But are you're definitely joking
that this is "obvious" in terms of code-clarity, right?

Just 5 minutes back you mentioned elsewhere you like seeing lots of
explicit calls to barrier() (with comments, no less, hmm? :-)

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Fri, 17 Aug 2007, Andi Kleen wrote:

> On Friday 17 August 2007 05:42, Linus Torvalds wrote:
> > On Fri, 17 Aug 2007, Paul Mackerras wrote:
> > > I'm really surprised it's as much as a few K.  I tried it on powerpc
> > > and it only saved 40 bytes (10 instructions) for a G5 config.
> >
> > One of the things that "volatile" generally screws up is a simple
> >
> > 	volatile int i;
> >
> > 	i++;
> 
> But for atomic_t people use atomic_inc() anyways which does this correctly.
> It shouldn't really matter for atomic_t.
> 
> I'm worrying a bit that the volatile atomic_t change caused subtle code 
> breakage like these delay read loops people here pointed out.

Umm, I followed most of the thread, but which breakage is this?

> Wouldn't it be safer to just re-add the volatile to atomic_read() 
> for 2.6.23? Or alternatively make it asm(), but volatile seems more
> proven.

The problem with volatile is not just trashy code generation (which also
definitely is a major problem), but definition holes, and implementation
inconsistencies. Making it asm() is not the only other alternative to
volatile either (read another reply to this mail), but considering most
of the thread has been about people not wanting even a
atomic_read_volatile() variant, making atomic_read() itself have volatile
semantics sounds ... strange :-)


PS: http://lkml.org/lkml/2007/8/15/407 was submitted a couple days back,
any word if you saw that?

I have another one for you:


[PATCH] i386, x86_64: __const_udelay() should not be marked inline

Because it can never get inlined in any callsite (each translation unit
is compiled separately for the kernel and so the implementation of
__const_udelay() would be invisible to all other callsites). In fact it
turns out, the correctness of callsites at arch/x86_64/kernel/crash.c:97
and arch/i386/kernel/crash.c:101 explicitly _depends_ upon it not being
inlined, and also it's an exported symbol (modules may want to call
mdelay() and udelay() that often becomes __const_udelay() after some
macro-ing in various headers). So let's not mark it as "inline" either.

Signed-off-by: Satyam Sharma <satyam@infradead.org>

---

 arch/i386/lib/delay.c   |    2 +-
 arch/x86_64/lib/delay.c |    2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/i386/lib/delay.c b/arch/i386/lib/delay.c
index f6edb11..0082c99 100644
--- a/arch/i386/lib/delay.c
+++ b/arch/i386/lib/delay.c
@@ -74,7 +74,7 @@ void __delay(unsigned long loops)
 	delay_fn(loops);
 }
 
-inline void __const_udelay(unsigned long xloops)
+void __const_udelay(unsigned long xloops)
 {
 	int d0;
 
diff --git a/arch/x86_64/lib/delay.c b/arch/x86_64/lib/delay.c
index 2dbebd3..d0cd9cd 100644
--- a/arch/x86_64/lib/delay.c
+++ b/arch/x86_64/lib/delay.c
@@ -38,7 +38,7 @@ void __delay(unsigned long loops)
 }
 EXPORT_SYMBOL(__delay);
 
-inline void __const_udelay(unsigned long xloops)
+void __const_udelay(unsigned long xloops)
 {
 	__delay(((xloops * HZ * cpu_data[raw_smp_processor_id()].loops_per_jiffy) >> 32) + 1);
 }

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Fri, 17 Aug 2007, Nick Piggin wrote:

> Satyam Sharma wrote:
> > 
> > On Fri, 17 Aug 2007, Nick Piggin wrote:
> 
> > > Also, why would you want to make these insane accessors for atomic_t
> > > types? Just make sure everybody knows the basics of barriers, and they
> > > can apply that knowledge to atomic_t and all other lockless memory
> > > accesses as well.
> > 
> > 
> > Code that looks like:
> > 
> > 	while (!atomic_read(&v)) {
> > 		...
> > 		cpu_relax_no_barrier();
> > 		forget(v.counter);
> > 		        ^^^^^^^^
> > 	}
> > 
> > would be uglier. Also think about code such as:
> 
> I think they would both be equally ugly,

You think both these are equivalent in terms of "looks":

					|
while (!atomic_read(&v)) {		|	while (!atomic_read_xxx(&v)) {
	...				|		...
	cpu_relax_no_barrier();		|		cpu_relax_no_barrier();
	order_atomic(&v);		|	}
}					|

(where order_atomic() is an atomic_t
specific wrapper as you mentioned below)

?

Well, taste varies, but ...

> but the atomic_read_volatile
> variant would be more prone to subtle bugs because of the weird
> implementation.

What bugs?

> And it would be more ugly than introducing an order(x) statement for
> all memory operations, and adding an order_atomic() wrapper for it
> for atomic types.

Oh, that order() / forget() macro [forget() was named such by Chuck Ebbert
earlier in this thread where he first mentioned it, btw] could definitely
be generically introduced for any memory operations.

> > 	a = atomic_read();
> > 	if (!a)
> > 		do_something();
> > 
> > 	forget();
> > 	a = atomic_read();
> > 	... /* some code that depends on value of a, obviously */
> > 
> > 	forget();
> > 	a = atomic_read();
> > 	...
> > 
> > So much explicit sprinkling of "forget()" looks ugly.
> 
> Firstly, why is it ugly? It's nice because of those nice explicit
> statements there that give us a good heads up and would have some
> comments attached to them

atomic_read_xxx (where xxx = whatever naming sounds nice to you) would
obviously also give a heads up, and could also have some comments
attached to it.

> (also, lack of the word "volatile" is always a plus).

Ok, xxx != volatile.

> Secondly, what sort of code would do such a thing?

See the nodemgr_host_thread() that does something similar, though not
exactly same.

> > on the other hand, looks neater. The "_volatile()" suffix makes it also
> > no less explicit than an explicit barrier-like macro that this primitive
> > is something "special", for code clarity purposes.
> 
> Just don't use the word volatile,

That sounds amazingly frivolous, but hey, why not. As I said, ok,
xxx != volatile.

> and have barriers both before and after the memory operation,

How could that lead to bugs? (if you can point to existing code,
but just some testcase / sample code would be fine as well).

> [...] I don't see
> the point though, when you could just have a single barrier(x)
> barrier function defined for all memory locations,

As I said, barrier() is too heavy-handed.

> rather than
> this odd thing that only works for atomics

Why would it work only for atomics? You could use that generic macro
for anything you well damn please.

> (and would have to
> be duplicated for atomic_set.

#define atomic_set_xxx for something similar. Big deal ... NOT.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Fri, 17 Aug 2007, Paul Mackerras wrote:

> Satyam Sharma writes:
> 
> > I wonder if this'll generate smaller and better code than _both_ the
> > other atomic_read_volatile() variants. Would need to build allyesconfig
> > on lots of diff arch's etc to test the theory though.
> 
> I'm sure it would be a tiny effect.
> 
> This whole thread is arguing about effects that are quite
> insignificant.

Hmm, the fact that this thread became what it did, probably means that
most developers on this list do not mind thinking/arguing about effects
or optimizations that are otherwise "tiny". But yeah, they are tiny
nonetheless.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Stefan Richter]

Nick Piggin wrote:
> Stefan Richter wrote:
>> For architecture port authors, there is Documentation/atomic_ops.txt.
>> Driver authors also can learn something from that document, as it
>> indirectly documents the atomic_t and bitops APIs.
> 
> "Semantics and Behavior of Atomic and Bitmask Operations" is
> pretty direct :)

"Indirect", "pretty direct"... It's subjective.

(It is not an API documentation; it is an implementation specification.)

> Sure, it says that it's for arch maintainers, but there is no
> reason why users can't make use of it.
> 
> 
>> Prompted by this thread, I reread this document, and indeed, the
>> sentence "Unlike the above routines, it is required that explicit memory
>> barriers are performed before and after [atomic_{inc,dec}_return]"
>> indicates that atomic_read (one of the "above routines") is very
>> different from all other atomic_t accessors that return values.
>> 
>> This is strange.  Why is it that atomic_read stands out that way?  IMO
> 
> It is not just atomic_read of course. It is atomic_add,sub,inc,dec,set.

Yes, but unlike these, atomic_read returns a value.

Without me (the API user) providing extra barriers, that value may
become something else whenever someone touches code in the vicinity of
the atomic_read.

>> this API imbalance is quite unexpected by many people.  Wouldn't it be
>> beneficial to change the atomic_read API to behave the same like all
>> other atomic_t accessors that return values?
> 
> It is very consistent and well defined. Operations which both modify
> the data _and_ return something are defined to have full barriers
> before and after.

You are right, atomic_read is not only different from accessors that
don't retunr values, it is also different from all other accessors that
return values (because they all also modify the value).  There is just
no actual API documentation, which contributes to the issue that some
people (or at least one: me) learn a little bit late how special
atomic_read is.

> What do you want to add to the other atomic accessors? Full memory
> barriers? Only compiler barriers? It's quite likely that if you think
> some barriers will fix bugs, then there are other bugs lurking there
> anyway.

A lot of different though related issues are discussed in this thread,
but I personally am only occupied by one particular thing:  What kind of
return values do I get from atomic_read.

> Just use spinlocks if you're not absolutely clear about potential
> races and memory ordering issues -- they're pretty cheap and simple.

Probably good advice, like generally if driver guys consider lockless
algorithms.

>> OK, it is also different from the other accessors that return data in so
>> far as it doesn't modify the data.  But as driver "author", i.e. user of
>> the API, I can't see much use of an atomic_read that can be reordered
>> and, more importantly, can be optimized away by the compiler.
> 
> It will return to you an atomic snapshot of the data (loaded from
> memory at some point since the last compiler barrier). All you have
> to be aware of compiler barriers and the Linux SMP memory ordering
> model, which should be a given if you are writing lockless code.

OK, that's what I slowly realized during this discussion, and I
appreciate the explanations that were given here.

>> Sure, now
>> that I learned of these properties I can start to audit code and insert
>> barriers where I believe they are needed, but this simply means that
>> almost all occurrences of atomic_read will get barriers (unless there
>> already are implicit but more or less obvious barriers like msleep).
> 
> You might find that these places that appear to need barriers are
> buggy for other reasons anyway. Can you point to some in-tree code
> we can have a look at?

I could, or could not, if I were through with auditing the code.  I
remembered one case and posted it (nodemgr_host_thread) which was safe
because msleep_interruptible provided the necessary barrier there, and
this implicit barrier is not in danger to be removed by future patches.
-- 
Stefan Richter
-=====-=-=== =--- =---=
http://arcgraph.de/sr/

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Fri, 17 Aug 2007, Nick Piggin wrote:

> Satyam Sharma wrote:
> > [...]
> > The point is about *author expecations*. If people do expect atomic_read()
> > (or a variant thereof) to have volatile semantics, why not give them such
> > a variant?
> 
> Because they should be thinking about them in terms of barriers, over
> which the compiler / CPU is not to reorder accesses or cache memory
> operations, rather than "special" "volatile" accesses.

This is obviously just a taste thing. Whether to have that forget(x)
barrier as something author should explicitly sprinkle appropriately
in appropriate places in the code by himself or use a primitive that
includes it itself.

I'm not saying "taste matters aren't important" (they are), but I'm really
skeptical if most folks would find the former tasteful.

> > And by the way, the point is *also* about the fact that cpu_relax(), as
> > of today, implies a full memory clobber, which is not what a lot of such
> > loops want. (due to stuff mentioned elsewhere, summarized in that summary)
> 
> That's not the point,

That's definitely the point, why not. This is why "barrier()", being
heavy-handed, is not the best option.

> because as I also mentioned, the logical extention
> to Linux's barrier API to handle this is the order(x) macro. Again, not
> special volatile accessors.

Sure, that forget(x) macro _is_ proposed to be made part of the generic
API. Doesn't explain why not to define/use primitives that has volatility
semantics in itself, though (taste matters apart).

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Stefan Richter]

I wrote:
> Nick Piggin wrote:
>> You might find that these places that appear to need barriers are
>> buggy for other reasons anyway. Can you point to some in-tree code
>> we can have a look at?
> 
> I could, or could not, if I were through with auditing the code.  I
> remembered one case and posted it (nodemgr_host_thread) which was safe
> because msleep_interruptible provided the necessary barrier there, and
> this implicit barrier is not in danger to be removed by future patches.

PS, just in case anybody holds his breath for more example code from me,
I don't plan to continue with an actual audit of the drivers I maintain.
It's an important issue, but my current time budget will restrict me to
look at it ad hoc, per case.  (Open bugs have higher priority than
potential bugs.)
-- 
Stefan Richter
-=====-=-=== =--- =---=
http://arcgraph.de/sr/

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Stefan Richter]

Nick Piggin wrote:
> Satyam Sharma wrote:
>> And we have driver / subsystem maintainers such as Stefan
>> coming up and admitting that often a lot of code that's written to use
>> atomic_read() does assume the read will not be elided by the compiler.
> 
> So these are broken on i386 and x86-64?

The ieee1394 and firewire subsystems have open, undiagnosed bugs, also
on i386 and x86-64.  But whether there is any bug because of wrong
assumptions about atomic_read among them, I don't know.  I don't know
which assumptions the authors made, I only know that I wasn't aware of
all the properties of atomic_read until now.

> Are they definitely safe on SMP and weakly ordered machines with
> just a simple compiler barrier there? Because I would not be
> surprised if there are a lot of developers who don't really know
> what to assume when it comes to memory ordering issues.
> 
> This is not a dig at driver writers: we still have memory ordering
> problems in the VM too (and probably most of the subtle bugs in
> lockless VM code are memory ordering ones). Let's not make up a
> false sense of security and hope that sprinkling volatile around
> will allow people to write bug-free lockless code. If a writer
> can't be bothered reading API documentation

...or, if there is none, the implementation specification (as in case of
the atomic ops), or, if there is none, the implementation (as in case of
a some infrastructure code here and there)...

> and learning the Linux memory model, they can still be productive
> writing safely locked code.

Provided they are aware that they might not have the full picture of the
lockless primitives.  :-)
-- 
Stefan Richter
-=====-=-=== =--- =---=
http://arcgraph.de/sr/

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Nick Piggin]

Satyam Sharma wrote:

>
>On Fri, 17 Aug 2007, Nick Piggin wrote:
>
>
>>Satyam Sharma wrote:
>>
>>It is very obvious. msleep calls schedule() (ie. sleeps), which is
>>always a barrier.
>>
>
>Probably you didn't mean that, but no, schedule() is not barrier because
>it sleeps. It's a barrier because it's invisible.
>

Where did I say it is a barrier because it sleeps?

It is always a barrier because, at the lowest level, schedule() (and thus
anything that sleeps) is defined to always be a barrier. Regardless of
whatever obscure means the compiler might need to infer the barrier.

In other words, you can ignore those obscure details because schedule() is
always going to have an explicit barrier in it.


>>The "unobvious" thing is that you wanted to know how the compiler knows
>>a function is a barrier -- answer is that if it does not *know* it is not
>>a barrier, it must assume it is a barrier.
>>
>
>True, that's clearly what happens here. But are you're definitely joking
>that this is "obvious" in terms of code-clarity, right?
>

No. If you accept that barrier() is implemented correctly, and you know
that sleeping is defined to be a barrier, then its perfectly clear. You
don't have to know how the compiler "knows" that some function contains
a barrier.


>Just 5 minutes back you mentioned elsewhere you like seeing lots of
>explicit calls to barrier() (with comments, no less, hmm? :-)
>

Sure, but there are well known primitives which contain barriers, and
trivial recognisable code sequences for which you don't need comments.
waiting-loops using sleeps or cpu_relax() are prime examples.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Nick Piggin]

Satyam Sharma wrote:

>
>On Fri, 17 Aug 2007, Nick Piggin wrote:
>
>>I think they would both be equally ugly,
>>
>
>You think both these are equivalent in terms of "looks":
>
>					|
>while (!atomic_read(&v)) {		|	while (!atomic_read_xxx(&v)) {
>	...				|		...
>	cpu_relax_no_barrier();		|		cpu_relax_no_barrier();
>	order_atomic(&v);		|	}
>}					|
>
>(where order_atomic() is an atomic_t
>specific wrapper as you mentioned below)
>
>?
>

I think the LHS is better if your atomic_read_xxx primitive is using the
crazy one-sided barrier, because the LHS code you immediately know what
barriers are happening, and with the RHS you have to look at the 
atomic_read_xxx
definition.

If your atomic_read_xxx implementation was more intuitive, then both are
pretty well equal. More lines != ugly code.


>>but the atomic_read_volatile
>>variant would be more prone to subtle bugs because of the weird
>>implementation.
>>
>
>What bugs?
>

You can't think for yourself? Your atomic_read_volatile contains a compiler
barrier to the atomic variable before the load. 2 such reads from different
locations look like this:

asm volatile("" : "+m" (v1));
atomic_read(&v1);
asm volatile("" : "+m" (v2));
atomic_read(&v2);

Which implies that the load of v1 can be reordered to occur after the load
of v2. Bet you didn't expect that?

>>Secondly, what sort of code would do such a thing?
>>
>
>See the nodemgr_host_thread() that does something similar, though not
>exactly same.
>

I'm sorry, all this waffling about made up code which might do this and
that is just a waste of time. Seriously, the thread is bloated enough
and never going to get anywhere with all this handwaving. If someone is
saving up all the really real and actually good arguments for why we
must have a volatile here, now is the time to use them.

>>and have barriers both before and after the memory operation,
>>
>
>How could that lead to bugs? (if you can point to existing code,
>but just some testcase / sample code would be fine as well).
>

See above.

>As I said, barrier() is too heavy-handed.
>

Typo. I meant: defined for a single memory location (ie. order(x)).

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Nick Piggin]

Satyam Sharma wrote:

>
>On Fri, 17 Aug 2007, Nick Piggin wrote:
>
>
>>Because they should be thinking about them in terms of barriers, over
>>which the compiler / CPU is not to reorder accesses or cache memory
>>operations, rather than "special" "volatile" accesses.
>>
>
>This is obviously just a taste thing. Whether to have that forget(x)
>barrier as something author should explicitly sprinkle appropriately
>in appropriate places in the code by himself or use a primitive that
>includes it itself.
>

That's not obviously just taste to me. Not when the primitive has many
(perhaps, the majority) of uses that do not require said barriers. And
this is not solely about the code generation (which, as Paul says, is
relatively minor even on x86). I prefer people to think explicitly
about barriers in their lockless code.


>I'm not saying "taste matters aren't important" (they are), but I'm really
>skeptical if most folks would find the former tasteful.
>

So I /do/ have better taste than most folks? Thanks! :-)


>>>And by the way, the point is *also* about the fact that cpu_relax(), as
>>>of today, implies a full memory clobber, which is not what a lot of such
>>>loops want. (due to stuff mentioned elsewhere, summarized in that summary)
>>>
>>That's not the point,
>>
>
>That's definitely the point, why not. This is why "barrier()", being
>heavy-handed, is not the best option.
>

That is _not_ the point (of why a volatile atomic_read is good) because 
there
has already been an alternative posted that better conforms with Linux 
barrier
API and is much more widely useful and more usable. If you are so 
worried about
barrier() being too heavyweight, then you're off to a poor start by 
wanting to
add a few K of kernel text by making atomic_read volatile.


>>because as I also mentioned, the logical extention
>>to Linux's barrier API to handle this is the order(x) macro. Again, not
>>special volatile accessors.
>>
>
>Sure, that forget(x) macro _is_ proposed to be made part of the generic
>API. Doesn't explain why not to define/use primitives that has volatility
>semantics in itself, though (taste matters apart).
>

If you follow the discussion.... You were thinking of a reason why the
semantics *should* be changed or added, and I was rebutting your argument
that it must be used when a full barrier() is too heavy (ie. by pointing
out that order() has superior semantics anyway).

Why do I keep repeating the same things? I'll not continue bloating this
thread until a new valid point comes up...

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Fri, 17 Aug 2007, Nick Piggin wrote:

> Satyam Sharma wrote:
> > On Fri, 17 Aug 2007, Nick Piggin wrote:
> > > Satyam Sharma wrote:
> > > 
> > > It is very obvious. msleep calls schedule() (ie. sleeps), which is
> > > always a barrier.
> > 
> > Probably you didn't mean that, but no, schedule() is not barrier because
> > it sleeps. It's a barrier because it's invisible.
> 
> Where did I say it is a barrier because it sleeps?

Just below. What you wrote:

> It is always a barrier because, at the lowest level, schedule() (and thus
> anything that sleeps) is defined to always be a barrier.

"It is always a barrier because, at the lowest level, anything that sleeps
is defined to always be a barrier".


> Regardless of
> whatever obscure means the compiler might need to infer the barrier.
> 
> In other words, you can ignore those obscure details because schedule() is
> always going to have an explicit barrier in it.

I didn't quite understand what you said here, so I'll tell what I think:

* foo() is a compiler barrier if the definition of foo() is invisible to
  the compiler at a callsite.

* foo() is also a compiler barrier if the definition of foo() includes
  a barrier, and it is inlined at the callsite.

If the above is wrong, or if there's something else at play as well,
do let me know.

> > > The "unobvious" thing is that you wanted to know how the compiler knows
> > > a function is a barrier -- answer is that if it does not *know* it is not
> > > a barrier, it must assume it is a barrier.
> > 
> > True, that's clearly what happens here. But are you're definitely joking
> > that this is "obvious" in terms of code-clarity, right?
> 
> No. If you accept that barrier() is implemented correctly, and you know
> that sleeping is defined to be a barrier,

Curiously, that's the second time you've said "sleeping is defined to
be a (compiler) barrier". How does the compiler even know if foo() is
a function that "sleeps"? Do compilers have some notion of "sleeping"
to ensure they automatically assume a compiler barrier whenever such
a function is called? Or are you saying that the compiler can see the
barrier() inside said function ... nopes, you're saying quite the
opposite below.


> then its perfectly clear. You
> don't have to know how the compiler "knows" that some function contains
> a barrier.

I think I do, why not? Would appreciate if you could elaborate on this.


Satyam

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Nick Piggin]

Satyam Sharma wrote:

>
>On Fri, 17 Aug 2007, Nick Piggin wrote:
>
>
>>Satyam Sharma wrote:
>>
>>>On Fri, 17 Aug 2007, Nick Piggin wrote:
>>>
>>>>Satyam Sharma wrote:
>>>>
>>>>It is very obvious. msleep calls schedule() (ie. sleeps), which is
>>>>always a barrier.
>>>>
>>>Probably you didn't mean that, but no, schedule() is not barrier because
>>>it sleeps. It's a barrier because it's invisible.
>>>
>>Where did I say it is a barrier because it sleeps?
>>
>
>Just below. What you wrote:
>
>
>>It is always a barrier because, at the lowest level, schedule() (and thus
>>anything that sleeps) is defined to always be a barrier.
>>
>
>"It is always a barrier because, at the lowest level, anything that sleeps
>is defined to always be a barrier".
>

... because it must call schedule and schedule is a barrier.


>>Regardless of
>>whatever obscure means the compiler might need to infer the barrier.
>>
>>In other words, you can ignore those obscure details because schedule() is
>>always going to have an explicit barrier in it.
>>
>
>I didn't quite understand what you said here, so I'll tell what I think:
>
>* foo() is a compiler barrier if the definition of foo() is invisible to
>  the compiler at a callsite.
>
>* foo() is also a compiler barrier if the definition of foo() includes
>  a barrier, and it is inlined at the callsite.
>
>If the above is wrong, or if there's something else at play as well,
>do let me know.
>

Right.


>>>>The "unobvious" thing is that you wanted to know how the compiler knows
>>>>a function is a barrier -- answer is that if it does not *know* it is not
>>>>a barrier, it must assume it is a barrier.
>>>>
>>>True, that's clearly what happens here. But are you're definitely joking
>>>that this is "obvious" in terms of code-clarity, right?
>>>
>>No. If you accept that barrier() is implemented correctly, and you know
>>that sleeping is defined to be a barrier,
>>
>
>Curiously, that's the second time you've said "sleeping is defined to
>be a (compiler) barrier".
>

_In Linux,_ sleeping is defined to be a compiler barrier.

>How does the compiler even know if foo() is
>a function that "sleeps"? Do compilers have some notion of "sleeping"
>to ensure they automatically assume a compiler barrier whenever such
>a function is called? Or are you saying that the compiler can see the
>barrier() inside said function ... nopes, you're saying quite the
>opposite below.
>

You're getting too worried about the compiler implementation. Start
by assuming that it does work ;)


>>then its perfectly clear. You
>>don't have to know how the compiler "knows" that some function contains
>>a barrier.
>>
>
>I think I do, why not? Would appreciate if you could elaborate on this.
>

If a function is not completely visible to the compiler (so it can't
determine whether a barrier could be in it or not), then it must always
assume it will contain a barrier so it always does the right thing.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Fri, 17 Aug 2007, Nick Piggin wrote:

> Satyam Sharma wrote:
> [...]
> > You think both these are equivalent in terms of "looks":
> > 
> > 					|
> > while (!atomic_read(&v)) {		|	while (!atomic_read_xxx(&v)) {
> > 	...				|		...
> > 	cpu_relax_no_barrier();		|
> > cpu_relax_no_barrier();
> > 	order_atomic(&v);		|	}
> > }					|
> > 
> > (where order_atomic() is an atomic_t
> > specific wrapper as you mentioned below)
> > 
> > ?
> 
> I think the LHS is better if your atomic_read_xxx primitive is using the
> crazy one-sided barrier,
  ^^^^^

I'd say it's purposefully one-sided.

> because the LHS code you immediately know what
> barriers are happening, and with the RHS you have to look at the
> atomic_read_xxx
> definition.

No. As I said, the _xxx (whatever the heck you want to name it as) should
give the same heads-up that your "order_atomic" thing is supposed to give.


> If your atomic_read_xxx implementation was more intuitive, then both are
> pretty well equal. More lines != ugly code.
> 
> > [...]
> > What bugs?
> 
> You can't think for yourself? Your atomic_read_volatile contains a compiler
> barrier to the atomic variable before the load. 2 such reads from different
> locations look like this:
> 
> asm volatile("" : "+m" (v1));
> atomic_read(&v1);
> asm volatile("" : "+m" (v2));
> atomic_read(&v2);
> 
> Which implies that the load of v1 can be reordered to occur after the load
> of v2.

And how would that be a bug? (sorry, I really can't think for myself)


> > > Secondly, what sort of code would do such a thing?
> > 
> > See the nodemgr_host_thread() that does something similar, though not
> > exactly same.
> 
> I'm sorry, all this waffling about made up code which might do this and
> that is just a waste of time.

First, you could try looking at the code.

And by the way, as I've already said (why do *require* people to have to
repeat things to you?) this isn't even about only existing code.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Fri, 17 Aug 2007, Nick Piggin wrote:

> Satyam Sharma wrote:
> 
> > On Fri, 17 Aug 2007, Nick Piggin wrote:
> > 
> > > Because they should be thinking about them in terms of barriers, over
> > > which the compiler / CPU is not to reorder accesses or cache memory
> > > operations, rather than "special" "volatile" accesses.
> > 
> > This is obviously just a taste thing. Whether to have that forget(x)
> > barrier as something author should explicitly sprinkle appropriately
> > in appropriate places in the code by himself or use a primitive that
> > includes it itself.
> 
> That's not obviously just taste to me. Not when the primitive has many
> (perhaps, the majority) of uses that do not require said barriers. And
> this is not solely about the code generation (which, as Paul says, is
> relatively minor even on x86).

See, you do *require* people to have to repeat the same things to you!

As has been written about enough times already, and if you followed the
discussion on this thread, I am *not* proposing that atomic_read()'s
semantics be changed to have any extra barriers. What is proposed is a
different atomic_read_xxx() variant thereof, that those can use who do
want that.

Now whether to have a kind of barrier ("volatile", whatever) in the
atomic_read_xxx() itself, or whether to make the code writer himself to
explicitly write the order(x) appropriately in appropriate places in the
code _is_ a matter of taste.


> > That's definitely the point, why not. This is why "barrier()", being
> > heavy-handed, is not the best option.
> 
> That is _not_ the point [...]

Again, you're requiring me to repeat things that were already made evident
on this thread (if you follow it).

This _is_ the point, because a lot of loops out there (too many of them,
I WILL NOT bother citing file_name:line_number) end up having to use a
barrier just because they're using a loop-exit-condition that depends
on a value returned by atomic_read(). It would be good for them if they
used an atomic_read_xxx() primitive that gave these "volatility" semantics
without junking compiler optimizations for other memory references.

> because there has already been an alternative posted

Whether that alternative (explicitly using forget(x), or wrappers thereof,
such as the "order_atomic" you proposed) is better than other alternatives
(such as atomic_read_xxx() which includes the volatility behaviour in
itself) is still open, and precisely what we started discussing just one
mail back.

(The above was also mostly stuff I had to repeated for you, sadly.)

> that better conforms with Linux barrier
> API and is much more widely useful and more usable.

I don't think so.

(Now *this* _is_ the "taste-dependent matter" that I mentioned earlier.)

> If you are so worried
> about
> barrier() being too heavyweight, then you're off to a poor start by wanting to
> add a few K of kernel text by making atomic_read volatile.

Repeating myself, for the N'th time, NO, I DON'T want to make atomic_read
have "volatile" semantics.

> > > because as I also mentioned, the logical extention
> > > to Linux's barrier API to handle this is the order(x) macro. Again, not
> > > special volatile accessors.
> > 
> > Sure, that forget(x) macro _is_ proposed to be made part of the generic
> > API. Doesn't explain why not to define/use primitives that has volatility
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > semantics in itself, though (taste matters apart).
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> If you follow the discussion.... You were thinking of a reason why the
> semantics *should* be changed or added, and I was rebutting your argument
> that it must be used when a full barrier() is too heavy (ie. by pointing
> out that order() has superior semantics anyway).

Amazing. Either you have reading comprehension problems, or else, please
try reading this thread (or at least this sub-thread) again. I don't want
_you_ blaming _me_ for having to repeat things to you all over again.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Fri, Aug 17, 2007 at 01:09:08PM +0530, Satyam Sharma wrote:
> 
> 
> On Thu, 16 Aug 2007, Paul E. McKenney wrote:
> 
> > On Fri, Aug 17, 2007 at 07:59:02AM +0800, Herbert Xu wrote:
> > > On Thu, Aug 16, 2007 at 09:34:41AM -0700, Paul E. McKenney wrote:
> > > >
> > > > The compiler can also reorder non-volatile accesses.  For an example
> > > > patch that cares about this, please see:
> > > > 
> > > > 	http://lkml.org/lkml/2007/8/7/280
> > > > 
> > > > This patch uses an ORDERED_WRT_IRQ() in rcu_read_lock() and
> > > > rcu_read_unlock() to ensure that accesses aren't reordered with respect
> > > > to interrupt handlers and NMIs/SMIs running on that same CPU.
> > > 
> > > Good, finally we have some code to discuss (even though it's
> > > not actually in the kernel yet).
> > 
> > There was some earlier in this thread as well.
> 
> Hmm, I never quite got what all this interrupt/NMI/SMI handling and
> RCU business you mentioned earlier was all about, but now that you've
> pointed to the actual code and issues with it ...

Glad to help...

> > > First of all, I think this illustrates that what you want
> > > here has nothing to do with atomic ops.  The ORDERED_WRT_IRQ
> > > macro occurs a lot more times in your patch than atomic
> > > reads/sets.  So *assuming* that it was necessary at all,
> > > then having an ordered variant of the atomic_read/atomic_set
> > > ops could do just as well.
> > 
> > Indeed.  If I could trust atomic_read()/atomic_set() to cause the compiler
> > to maintain ordering, then I could just use them instead of having to
> > create an  ORDERED_WRT_IRQ().  (Or ACCESS_ONCE(), as it is called in a
> > different patch.)
> 
> +#define WHATEVER(x)	(*(volatile typeof(x) *)&(x))
> 
> I suppose one could want volatile access semantics for stuff that's
> a bit-field too, no?

One could, but this is not supported in general.  So if you want that,
you need to use the usual bit-mask tricks and (for setting) atomic
operations.

> Also, this gives *zero* "re-ordering" guarantees that your code wants
> as you've explained it below) -- neither w.r.t. CPU re-ordering (which
> probably you don't care about) *nor* w.r.t. compiler re-ordering
> (which you definitely _do_ care about).

You are correct about CPU re-ordering (and about the fact that this
example doesn't care about it), but not about compiler re-ordering.

The compiler is prohibited from moving a volatile access across a sequence
point.  One example of a sequence point is a statement boundary.  Because
all of the volatile accesses in this code are separated by statement
boundaries, a conforming compiler is prohibited from reordering them.

> > > However, I still don't know which atomic_read/atomic_set in
> > > your patch would be broken if there were no volatile.  Could
> > > you please point them out?
> > 
> > Suppose I tried replacing the ORDERED_WRT_IRQ() calls with
> > atomic_read() and atomic_set().  Starting with __rcu_read_lock():
> > 
> > o	If "ORDERED_WRT_IRQ(__get_cpu_var(rcu_flipctr)[idx])++"
> > 	was ordered by the compiler after
> > 	"ORDERED_WRT_IRQ(me->rcu_read_lock_nesting) = nesting + 1", then
> > 	suppose an NMI/SMI happened after the rcu_read_lock_nesting but
> > 	before the rcu_flipctr.
> > 
> > 	Then if there was an rcu_read_lock() in the SMI/NMI
> > 	handler (which is perfectly legal), the nested rcu_read_lock()
> > 	would believe that it could take the then-clause of the
> > 	enclosing "if" statement.  But because the rcu_flipctr per-CPU
> > 	variable had not yet been incremented, an RCU updater would
> > 	be within its rights to assume that there were no RCU reads
> > 	in progress, thus possibly yanking a data structure out from
> > 	under the reader in the SMI/NMI function.
> > 
> > 	Fatal outcome.  Note that only one CPU is involved here
> > 	because these are all either per-CPU or per-task variables.
> 
> Ok, so you don't care about CPU re-ordering. Still, I should let you know
> that your ORDERED_WRT_IRQ() -- bad name, btw -- is still buggy. What you
> want is a full compiler optimization barrier().

No.  See above.

> [ Your code probably works now, and emits correct code, but that's
>   just because of gcc did what it did. Nothing in any standard,
>   or in any documented behaviour of gcc, or anything about the real
>   (or expected) semantics of "volatile" is protecting the code here. ]

Really?  Why doesn't the prohibition against moving volatile accesses
across sequence points take care of this?

> > o	If "ORDERED_WRT_IRQ(me->rcu_read_lock_nesting) = nesting + 1"
> > 	was ordered by the compiler to follow the
> > 	"ORDERED_WRT_IRQ(me->rcu_flipctr_idx) = idx", and an NMI/SMI
> > 	happened between the two, then an __rcu_read_lock() in the NMI/SMI
> > 	would incorrectly take the "else" clause of the enclosing "if"
> > 	statement.  If some other CPU flipped the rcu_ctrlblk.completed
> > 	in the meantime, then the __rcu_read_lock() would (correctly)
> > 	write the new value into rcu_flipctr_idx.
> > 
> > 	Well and good so far.  But the problem arises in
> > 	__rcu_read_unlock(), which then decrements the wrong counter.
> > 	Depending on exactly how subsequent events played out, this could
> > 	result in either prematurely ending grace periods or never-ending
> > 	grace periods, both of which are fatal outcomes.
> > 
> > And the following are not needed in the current version of the
> > patch, but will be in a future version that either avoids disabling
> > irqs or that dispenses with the smp_read_barrier_depends() that I
> > have 99% convinced myself is unneeded:
> > 
> > o	nesting = ORDERED_WRT_IRQ(me->rcu_read_lock_nesting);
> > 
> > o	idx = ORDERED_WRT_IRQ(rcu_ctrlblk.completed) & 0x1;
> > 
> > Furthermore, in that future version, irq handlers can cause the same
> > mischief that SMI/NMI handlers can in this version.
> > 
> > Next, looking at __rcu_read_unlock():
> > 
> > o	If "ORDERED_WRT_IRQ(me->rcu_read_lock_nesting) = nesting - 1"
> > 	was reordered by the compiler to follow the
> > 	"ORDERED_WRT_IRQ(__get_cpu_var(rcu_flipctr)[idx])--",
> > 	then if an NMI/SMI containing an rcu_read_lock() occurs between
> > 	the two, this nested rcu_read_lock() would incorrectly believe
> > 	that it was protected by an enclosing RCU read-side critical
> > 	section as described in the first reversal discussed for
> > 	__rcu_read_lock() above.  Again, fatal outcome.
> > 
> > This is what we have now.  It is not hard to imagine situations that
> > interact with -both- interrupt handlers -and- other CPUs, as described
> > earlier.
> 
> It's not about interrupt/SMI/NMI handlers at all! What you clearly want,
> simply put, is that a certain stream of C statements must be emitted
> by the compiler _as they are_ with no re-ordering optimizations! You must
> *definitely* use barrier(), IMHO.

Almost.  I don't care about most of the operations, only about the loads
and stores marked volatile.  Again, although the compiler is free to
reorder volatile accesses that occur -within- a single statement, it
is prohibited by the standard from moving volatile accesses from one
statement to another.  Therefore, this code can legitimately use volatile.

Or am I missing something subtle?

						Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Linus Torvalds]

On Fri, 17 Aug 2007, Nick Piggin wrote:
> 
> That's not obviously just taste to me. Not when the primitive has many
> (perhaps, the majority) of uses that do not require said barriers. And
> this is not solely about the code generation (which, as Paul says, is
> relatively minor even on x86). I prefer people to think explicitly
> about barriers in their lockless code.

Indeed.

I think the important issues are:

 - "volatile" itself is simply a badly/weakly defined issue. The semantics 
   of it as far as the compiler is concerned are really not very good, and 
   in practice tends to boil down to "I will generate so bad code that 
   nobody can accuse me of optimizing anything away".

 - "volatile" - regardless of how well or badly defined it is - is purely 
   a compiler thing. It has absolutely no meaning for the CPU itself, so 
   it at no point implies any CPU barriers. As a result, even if the 
   compiler generates crap code and doesn't re-order anything, there's 
   nothing that says what the CPU will do.

 - in other words, the *only* possible meaning for "volatile" is a purely 
   single-CPU meaning. And if you only have a single CPU involved in the 
   process, the "volatile" is by definition pointless (because even 
   without a volatile, the compiler is required to make the C code appear 
   consistent as far as a single CPU is concerned).

So, let's take the example *buggy* code where we use "volatile" to wait 
for other CPU's:

	atomic_set(&var, 0);
	while (!atomic_read(&var))
		/* nothing */;


which generates an endless loop if we don't have atomic_read() imply 
volatile.

The point here is that it's buggy whether the volatile is there or not! 
Exactly because the user expects multi-processing behaviour, but 
"volatile" doesn't actually give any real guarantees about it. Another CPU 
may have done:

	external_ptr = kmalloc(..);
	/* Setup is now complete, inform the waiter */
	atomic_inc(&var);

but the fact is, since the other CPU isn't serialized in any way, the 
"while-loop" (even in the presense of "volatile") doesn't actually work 
right! Whatever the "atomic_read()" was waiting for may not have 
completed, because we have no barriers!

So if "volatile" makes a difference, it is invariably a sign of a bug in 
serialization (the one exception is for IO - we use "volatile" to avoid 
having to use inline asm for IO on x86) - and for "random values" like 
jiffies).

So the question should *not* be whether "volatile" actually fixes bugs. It 
*never* fixes a bug. But what it can do is to hide the obvious ones. In 
other words, adding a volaile in the above kind of situation of 
"atomic_read()" will certainly turn an obvious bug into something that 
works "practically all of the time).

So anybody who argues for "volatile" fixing bugs is fundamentally 
incorrect. It does NO SUCH THING. By arguing that, such people only show 
that you have no idea what they are talking about.

So the only reason to add back "volatile" to the atomic_read() sequence is 
not to fix bugs, but to _hide_ the bugs better. They're still there, they 
are just a lot harder to trigger, and tend to be a lot subtler.

And hey, sometimes "hiding bugs well enough" is ok. In this case, I'd 
argue that we've successfully *not* had the volatile there for eight 
months on x86-64, and that should tell people something. 

(Does _removing_ the volatile fix bugs? No - callers still need to think 
about barriers etc, and lots of people don't. So I'm not claiming that 
removing volatile fixes any bugs either, but I *am* claiming that:

 - removing volatile makes some bugs easier to see (which is mostly a good 
   thing: they were there before, anyway).

 - removing volatile generates better code (which is a good thing, even if 
   it's just 0.1%)

 - removing volatile removes a huge mental *bug* that lots of people seem 
   to have, as shown by this whole thread. Anybody who thinks that 
   "volatile" actually fixes anything has a gaping hole in their head, and 
   we should remove volatile just to make sure that nobody thinks that it 
   means soemthign that it doesn't mean!

In other words, this whole discussion has just convinced me that we should 
*not* add back "volatile" to "atomic_read()" - I was willing to do it for 
practical and "hide the bugs" reasons, but having seen people argue for 
it, thinking that it actually fixes something, I'm now convinced that the 
*last* thing we should do is to encourage that kind of superstitious 
thinking.

"volatile" is like a black cat crossing the road. Sure, it affects 
*something* (at a minimum: before, the black cat was on one side of the 
road, afterwards it is on the other side of the road), but it has no 
bigger and longer-lasting direct affects. 

People who think "volatile" really matters are just fooling themselves.

		Linus

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

>>>>>> Part of the motivation here is to fix heisenbugs.  If I knew 
>>>>>> where they
>>>>>
>>>>> By the same token we should probably disable optimisations
>>>>> altogether since that too can create heisenbugs.
>>>>
>>>> Almost everything is a tradeoff; and so is this.  I don't
>>>> believe most people would find disabling all compiler
>>>> optimisations an acceptable price to pay for some peace
>>>> of mind.
>>>
>>>
>>> So why is this a good tradeoff?
>> It certainly is better than disabling all compiler optimisations!
>
> It's easy to be better than something really stupid :)

Sure, it wasn't me who made the comparison though.

> So i386 and x86-64 don't have volatiles there, and it saves them a
> few K of kernel text.

Which has to be investigated.  A few kB is a lot more than expected.

> What you need to justify is why it is a good
> tradeoff to make them volatile (which btw, is much harder to go
> the other way after we let people make those assumptions).

My point is that people *already* made those assumptions.  There
are two ways to clean up this mess:

1) Have the "volatile" semantics by default, change the users
    that don't need it;
2) Have "non-volatile" semantics by default, change the users
    that do need it.

Option 2) randomly breaks stuff all over the place, option 1)
doesn't.  Yeah 1) could cause some extremely minor speed or
code size regression, but only temporarily until everything has
been audited.

>>> I also think that just adding things to APIs in the hope it might fix
>>> up some bugs isn't really a good road to go down. Where do you stop?
>> I look at it the other way: keeping the "volatile" semantics in
>> atomic_XXX() (or adding them to it, whatever) helps _prevent_ bugs;
>
> Yeah, but we could add lots of things to help prevent bugs and
> would never be included. I would also contend that it helps _hide_
> bugs and encourages people to be lazy when thinking about these
> things.

Sure.  We aren't _adding_ anything here though, not on the platforms
where it is most likely to show up, anyway.

> Also, you dismiss the fact that we'd actually be *adding* volatile
> semantics back to the 2 most widely tested architectures (in terms
> of test time, number of testers, variety of configurations, and
> coverage of driver code).

I'm not dismissing that.  x86 however is one of the few architectures
where mistakenly leaving out a "volatile" will not easily show up on
user testing, since the compiler will very often produce a memory
reference anyway because it has no registers to play with.

> This is a very important different from
> just keeping volatile semantics because it is basically a one-way
> API change.

That's a good point.  Maybe we should create _two_ new APIs, one
explicitly going each way.

>> certainly most people expect that behaviour, and also that behaviour
>> is *needed* in some places and no other interface provides that
>> functionality.
>
> I don't know that most people would expect that behaviour.

I didn't conduct any formal poll either :-)

> Is there any documentation anywhere that would suggest this?

Not really I think, no.  But not the other way around, either.
Most uses of it seem to expect it though.

>> [some confusion about barriers wrt atomics snipped]
>
> What were you confused about?

Me?  Not much.


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

>> atomic_dec() already has volatile behavior everywhere, so this is 
>> semantically
>> okay, but this code (and any like it) should be calling cpu_relax() 
>> each
>> iteration through the loop, unless there's a compelling reason not 
>> to.  I'll
>> allow that for some hardware drivers (possibly this one) such a 
>> compelling
>> reason may exist, but hardware-independent core subsystems probably 
>> have no
>> excuse.
>
> No it does not have any volatile semantics. atomic_dec() can be 
> reordered
> at will by the compiler within the current basic unit if you do not 
> add a
> barrier.

"volatile" has nothing to do with reordering.  atomic_dec() writes
to memory, so it _does_ have "volatile semantics", implicitly, as
long as the compiler cannot optimise the atomic variable away
completely -- any store counts as a side effect.


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Fri, 17 Aug 2007, Paul E. McKenney wrote:

> On Fri, Aug 17, 2007 at 01:09:08PM +0530, Satyam Sharma wrote:
> > 
> > On Thu, 16 Aug 2007, Paul E. McKenney wrote:
> > 
> > > On Fri, Aug 17, 2007 at 07:59:02AM +0800, Herbert Xu wrote:
> > > > 
> > > > First of all, I think this illustrates that what you want
> > > > here has nothing to do with atomic ops.  The ORDERED_WRT_IRQ
> > > > macro occurs a lot more times in your patch than atomic
> > > > reads/sets.  So *assuming* that it was necessary at all,
> > > > then having an ordered variant of the atomic_read/atomic_set
> > > > ops could do just as well.
> > > 
> > > Indeed.  If I could trust atomic_read()/atomic_set() to cause the compiler
> > > to maintain ordering, then I could just use them instead of having to
> > > create an  ORDERED_WRT_IRQ().  (Or ACCESS_ONCE(), as it is called in a
> > > different patch.)
> > 
> > +#define WHATEVER(x)	(*(volatile typeof(x) *)&(x))
> > [...]
> > Also, this gives *zero* "re-ordering" guarantees that your code wants
> > as you've explained it below) -- neither w.r.t. CPU re-ordering (which
> > probably you don't care about) *nor* w.r.t. compiler re-ordering
> > (which you definitely _do_ care about).
> 
> You are correct about CPU re-ordering (and about the fact that this
> example doesn't care about it), but not about compiler re-ordering.
> 
> The compiler is prohibited from moving a volatile access across a sequence
> point.  One example of a sequence point is a statement boundary.  Because
> all of the volatile accesses in this code are separated by statement
> boundaries, a conforming compiler is prohibited from reordering them.

Yes, you're right, and I believe precisely this was discussed elsewhere
as well today.

But I'd call attention to what Herbert mentioned there. You're using
ORDERED_WRT_IRQ() on stuff that is _not_ defined to be an atomic_t at all:

* Member "completed" of struct rcu_ctrlblk is a long.
* Per-cpu variable rcu_flipctr is an array of ints.
* Members "rcu_read_lock_nesting" and "rcu_flipctr_idx" of
  struct task_struct are ints.

So are you saying you're "having to use" this volatile-access macro
because you *couldn't* declare all the above as atomic_t and thus just
expect the right thing to happen by using the atomic ops API by default,
because it lacks volatile access semantics (on x86)?

If so, then I wonder if using the volatile access cast is really the
best way to achieve (at least in terms of code clarity) the kind of
re-ordering guarantees it wants there. (there could be alternative
solutions, such as using barrier(), or that at bottom of this mail)

What I mean is this: If you switch to atomic_t, and x86 switched to
make atomic_t have "volatile" semantics by default, the statements
would be simply a string of: atomic_inc(), atomic_add(), atomic_set(),
and atomic_read() statements, and nothing in there that clearly makes
it *explicit* that the code is correct (and not buggy) simply because
of the re-ordering guarantees that the C "volatile" type-qualifier
keyword gives us as per the standard. But now we're firmly in
"subjective" territory, so you or anybody could legitimately disagree.


> > > Suppose I tried replacing the ORDERED_WRT_IRQ() calls with
> > > atomic_read() and atomic_set().  Starting with __rcu_read_lock():
> > > 
> > > o	If "ORDERED_WRT_IRQ(__get_cpu_var(rcu_flipctr)[idx])++"
> > > 	was ordered by the compiler after
> > > 	"ORDERED_WRT_IRQ(me->rcu_read_lock_nesting) = nesting + 1", then
> > > 	suppose an NMI/SMI happened after the rcu_read_lock_nesting but
> > > 	before the rcu_flipctr.
> > > 
> > > 	Then if there was an rcu_read_lock() in the SMI/NMI
> > > 	handler (which is perfectly legal), the nested rcu_read_lock()
> > > 	would believe that it could take the then-clause of the
> > > 	enclosing "if" statement.  But because the rcu_flipctr per-CPU
> > > 	variable had not yet been incremented, an RCU updater would
> > > 	be within its rights to assume that there were no RCU reads
> > > 	in progress, thus possibly yanking a data structure out from
> > > 	under the reader in the SMI/NMI function.
> > > 
> > > 	Fatal outcome.  Note that only one CPU is involved here
> > > 	because these are all either per-CPU or per-task variables.
> > 
> > Ok, so you don't care about CPU re-ordering. Still, I should let you know
> > that your ORDERED_WRT_IRQ() -- bad name, btw -- is still buggy. What you
> > want is a full compiler optimization barrier().
> 
> No.  See above.

True, *(volatile foo *)& _will_ work for this case.

But multiple calls to barrier() (granted, would invalidate all other
optimizations also) would work as well, would it not?

[ Interestingly, if you declared all those objects mentioned earlier as
  atomic_t, and x86(-64) switched to an __asm__ __volatile__ based variant
  for atomic_{read,set}_volatile(), the bugs you want to avoid would still
  be there. "volatile" the C language type-qualifier does have compiler
  re-ordering semantics you mentioned earlier, but the "volatile" that
  applies to inline asm()s gives no re-ordering guarantees. ]


> > > o	If "ORDERED_WRT_IRQ(me->rcu_read_lock_nesting) = nesting + 1"
> > > 	was ordered by the compiler to follow the
> > > 	"ORDERED_WRT_IRQ(me->rcu_flipctr_idx) = idx", and an NMI/SMI
> > > 	happened between the two, then an __rcu_read_lock() in the NMI/SMI
> > > 	would incorrectly take the "else" clause of the enclosing "if"
> > > 	statement.  If some other CPU flipped the rcu_ctrlblk.completed
> > > 	in the meantime, then the __rcu_read_lock() would (correctly)
> > > 	write the new value into rcu_flipctr_idx.
> > > 
> > > 	Well and good so far.  But the problem arises in
> > > 	__rcu_read_unlock(), which then decrements the wrong counter.
> > > 	Depending on exactly how subsequent events played out, this could
> > > 	result in either prematurely ending grace periods or never-ending
> > > 	grace periods, both of which are fatal outcomes.
> > > 
> > > And the following are not needed in the current version of the
> > > patch, but will be in a future version that either avoids disabling
> > > irqs or that dispenses with the smp_read_barrier_depends() that I
> > > have 99% convinced myself is unneeded:
> > > 
> > > o	nesting = ORDERED_WRT_IRQ(me->rcu_read_lock_nesting);
> > > 
> > > o	idx = ORDERED_WRT_IRQ(rcu_ctrlblk.completed) & 0x1;
> > > 
> > > Furthermore, in that future version, irq handlers can cause the same
> > > mischief that SMI/NMI handlers can in this version.

So don't remove the local_irq_save/restore, which is well-established and
well-understood for such cases (it doesn't help you with SMI/NMI,
admittedly). This isn't really about RCU or per-cpu vars as such, it's
just about racy code where you don't want to get hit by a concurrent
interrupt (it does turn out that doing things in a _particular order_ will
not cause fatal/buggy behaviour, but it's still a race issue, after all).


> > > Next, looking at __rcu_read_unlock():
> > > 
> > > o	If "ORDERED_WRT_IRQ(me->rcu_read_lock_nesting) = nesting - 1"
> > > 	was reordered by the compiler to follow the
> > > 	"ORDERED_WRT_IRQ(__get_cpu_var(rcu_flipctr)[idx])--",
> > > 	then if an NMI/SMI containing an rcu_read_lock() occurs between
> > > 	the two, this nested rcu_read_lock() would incorrectly believe
> > > 	that it was protected by an enclosing RCU read-side critical
> > > 	section as described in the first reversal discussed for
> > > 	__rcu_read_lock() above.  Again, fatal outcome.
> > > 
> > > This is what we have now.  It is not hard to imagine situations that
> > > interact with -both- interrupt handlers -and- other CPUs, as described
> > > earlier.

Unless somebody's going for a lockless implementation, such situations
normally use spin_lock_irqsave() based locking (or local_irq_save for
those who care only for current CPU) -- problem with the patch in question,
is that you want to prevent races with concurrent SMI/NMIs as well, which
is not something that a lot of code needs to consider.

[ Curiously, another thread is discussing something similar also:
  http://lkml.org/lkml/2007/8/15/393 "RFC: do get_rtc_time() correctly" ]

Anyway, I didn't look at the code in that patch very much in detail, but
why couldn't you implement some kind of synchronization variable that lets
rcu_read_lock() or rcu_read_unlock() -- when being called from inside an
NMI or SMI handler -- know that it has concurrently interrupted an ongoing
rcu_read_{un}lock() and so must do things differently ... (?)

I'm also wondering if there's other code that's not using locking in the
kernel that faces similar issues, and what they've done to deal with it
(if anything). Such bugs would be subtle, and difficult to diagnose.


Satyam

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Fri, 17 Aug 2007, Segher Boessenkool wrote:

> > > atomic_dec() already has volatile behavior everywhere, so this is
> > > semantically
> > > okay, but this code (and any like it) should be calling cpu_relax() each
> > > iteration through the loop, unless there's a compelling reason not to.
> > > I'll
> > > allow that for some hardware drivers (possibly this one) such a compelling
> > > reason may exist, but hardware-independent core subsystems probably have
> > > no
> > > excuse.
> > 
> > No it does not have any volatile semantics. atomic_dec() can be reordered
> > at will by the compiler within the current basic unit if you do not add a
> > barrier.
> 
> "volatile" has nothing to do with reordering.

If you're talking of "volatile" the type-qualifier keyword, then
http://lkml.org/lkml/2007/8/16/231 (and sub-thread below it) shows
otherwise.

> atomic_dec() writes
> to memory, so it _does_ have "volatile semantics", implicitly, as
> long as the compiler cannot optimise the atomic variable away
> completely -- any store counts as a side effect.

I don't think an atomic_dec() implemented as an inline "asm volatile"
or one that uses a "forget" macro would have the same re-ordering
guarantees as an atomic_dec() that uses a volatile access cast.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Chris Friesen]

Linus Torvalds wrote:

>  - in other words, the *only* possible meaning for "volatile" is a purely 
>    single-CPU meaning. And if you only have a single CPU involved in the 
>    process, the "volatile" is by definition pointless (because even 
>    without a volatile, the compiler is required to make the C code appear 
>    consistent as far as a single CPU is concerned).

I assume you mean "except for IO-related code and 'random' values like 
jiffies" as you mention later on?  I assume other values set in 
interrupt handlers would count as "random" from a volatility perspective?

> So anybody who argues for "volatile" fixing bugs is fundamentally 
> incorrect. It does NO SUCH THING. By arguing that, such people only show 
> that you have no idea what they are talking about.

What about reading values modified in interrupt handlers, as in your 
"random" case?  Or is this a bug where the user of atomic_read() is 
invalidly expecting a read each time it is called?

Chris

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Arjan van de Ven]

On Fri, 2007-08-17 at 12:50 -0600, Chris Friesen wrote:
> Linus Torvalds wrote:
> 
> >  - in other words, the *only* possible meaning for "volatile" is a purely 
> >    single-CPU meaning. And if you only have a single CPU involved in the 
> >    process, the "volatile" is by definition pointless (because even 
> >    without a volatile, the compiler is required to make the C code appear 
> >    consistent as far as a single CPU is concerned).
> 
> I assume you mean "except for IO-related code and 'random' values like 
> jiffies" as you mention later on?  I assume other values set in 
> interrupt handlers would count as "random" from a volatility perspective?
> 
> > So anybody who argues for "volatile" fixing bugs is fundamentally 
> > incorrect. It does NO SUCH THING. By arguing that, such people only show 
> > that you have no idea what they are talking about.
> 
> What about reading values modified in interrupt handlers, as in your 
> "random" case?  Or is this a bug where the user of atomic_read() is 
> invalidly expecting a read each time it is called?

the interrupt handler case is an SMP case since you do not know
beforehand what cpu your interrupt handler will run on.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Sat, Aug 18, 2007 at 12:01:38AM +0530, Satyam Sharma wrote:
> 
> 
> On Fri, 17 Aug 2007, Paul E. McKenney wrote:
> 
> > On Fri, Aug 17, 2007 at 01:09:08PM +0530, Satyam Sharma wrote:
> > > 
> > > On Thu, 16 Aug 2007, Paul E. McKenney wrote:
> > > 
> > > > On Fri, Aug 17, 2007 at 07:59:02AM +0800, Herbert Xu wrote:
> > > > > 
> > > > > First of all, I think this illustrates that what you want
> > > > > here has nothing to do with atomic ops.  The ORDERED_WRT_IRQ
> > > > > macro occurs a lot more times in your patch than atomic
> > > > > reads/sets.  So *assuming* that it was necessary at all,
> > > > > then having an ordered variant of the atomic_read/atomic_set
> > > > > ops could do just as well.
> > > > 
> > > > Indeed.  If I could trust atomic_read()/atomic_set() to cause the compiler
> > > > to maintain ordering, then I could just use them instead of having to
> > > > create an  ORDERED_WRT_IRQ().  (Or ACCESS_ONCE(), as it is called in a
> > > > different patch.)
> > > 
> > > +#define WHATEVER(x)	(*(volatile typeof(x) *)&(x))
> > > [...]
> > > Also, this gives *zero* "re-ordering" guarantees that your code wants
> > > as you've explained it below) -- neither w.r.t. CPU re-ordering (which
> > > probably you don't care about) *nor* w.r.t. compiler re-ordering
> > > (which you definitely _do_ care about).
> > 
> > You are correct about CPU re-ordering (and about the fact that this
> > example doesn't care about it), but not about compiler re-ordering.
> > 
> > The compiler is prohibited from moving a volatile access across a sequence
> > point.  One example of a sequence point is a statement boundary.  Because
> > all of the volatile accesses in this code are separated by statement
> > boundaries, a conforming compiler is prohibited from reordering them.
> 
> Yes, you're right, and I believe precisely this was discussed elsewhere
> as well today.
> 
> But I'd call attention to what Herbert mentioned there. You're using
> ORDERED_WRT_IRQ() on stuff that is _not_ defined to be an atomic_t at all:
> 
> * Member "completed" of struct rcu_ctrlblk is a long.
> * Per-cpu variable rcu_flipctr is an array of ints.
> * Members "rcu_read_lock_nesting" and "rcu_flipctr_idx" of
>   struct task_struct are ints.
> 
> So are you saying you're "having to use" this volatile-access macro
> because you *couldn't* declare all the above as atomic_t and thus just
> expect the right thing to happen by using the atomic ops API by default,
> because it lacks volatile access semantics (on x86)?
> 
> If so, then I wonder if using the volatile access cast is really the
> best way to achieve (at least in terms of code clarity) the kind of
> re-ordering guarantees it wants there. (there could be alternative
> solutions, such as using barrier(), or that at bottom of this mail)
> 
> What I mean is this: If you switch to atomic_t, and x86 switched to
> make atomic_t have "volatile" semantics by default, the statements
> would be simply a string of: atomic_inc(), atomic_add(), atomic_set(),
> and atomic_read() statements, and nothing in there that clearly makes
> it *explicit* that the code is correct (and not buggy) simply because
> of the re-ordering guarantees that the C "volatile" type-qualifier
> keyword gives us as per the standard. But now we're firmly in
> "subjective" territory, so you or anybody could legitimately disagree.

In any case, given Linus's note, it appears that atomic_read() and
atomic_set() won't consistently have volatile semantics, at least
not while the compiler generates such ugly code for volatile accesses.
So I will continue with my current approach.

In any case, I will not be using atomic_inc() or atomic_add() in this
code, as doing so would more than double the overhead, even on machines
that are the most efficient at implementing atomic operations.

> > > > Suppose I tried replacing the ORDERED_WRT_IRQ() calls with
> > > > atomic_read() and atomic_set().  Starting with __rcu_read_lock():
> > > > 
> > > > o	If "ORDERED_WRT_IRQ(__get_cpu_var(rcu_flipctr)[idx])++"
> > > > 	was ordered by the compiler after
> > > > 	"ORDERED_WRT_IRQ(me->rcu_read_lock_nesting) = nesting + 1", then
> > > > 	suppose an NMI/SMI happened after the rcu_read_lock_nesting but
> > > > 	before the rcu_flipctr.
> > > > 
> > > > 	Then if there was an rcu_read_lock() in the SMI/NMI
> > > > 	handler (which is perfectly legal), the nested rcu_read_lock()
> > > > 	would believe that it could take the then-clause of the
> > > > 	enclosing "if" statement.  But because the rcu_flipctr per-CPU
> > > > 	variable had not yet been incremented, an RCU updater would
> > > > 	be within its rights to assume that there were no RCU reads
> > > > 	in progress, thus possibly yanking a data structure out from
> > > > 	under the reader in the SMI/NMI function.
> > > > 
> > > > 	Fatal outcome.  Note that only one CPU is involved here
> > > > 	because these are all either per-CPU or per-task variables.
> > > 
> > > Ok, so you don't care about CPU re-ordering. Still, I should let you know
> > > that your ORDERED_WRT_IRQ() -- bad name, btw -- is still buggy. What you
> > > want is a full compiler optimization barrier().
> > 
> > No.  See above.
> 
> True, *(volatile foo *)& _will_ work for this case.
> 
> But multiple calls to barrier() (granted, would invalidate all other
> optimizations also) would work as well, would it not?

They work, but are a bit slower.  So they do work, but not as well.

> [ Interestingly, if you declared all those objects mentioned earlier as
>   atomic_t, and x86(-64) switched to an __asm__ __volatile__ based variant
>   for atomic_{read,set}_volatile(), the bugs you want to avoid would still
>   be there. "volatile" the C language type-qualifier does have compiler
>   re-ordering semantics you mentioned earlier, but the "volatile" that
>   applies to inline asm()s gives no re-ordering guarantees. ]

Well, that certainly would be a point in favor of "volatile" over inline
asms.  ;-)

> > > > o	If "ORDERED_WRT_IRQ(me->rcu_read_lock_nesting) = nesting + 1"
> > > > 	was ordered by the compiler to follow the
> > > > 	"ORDERED_WRT_IRQ(me->rcu_flipctr_idx) = idx", and an NMI/SMI
> > > > 	happened between the two, then an __rcu_read_lock() in the NMI/SMI
> > > > 	would incorrectly take the "else" clause of the enclosing "if"
> > > > 	statement.  If some other CPU flipped the rcu_ctrlblk.completed
> > > > 	in the meantime, then the __rcu_read_lock() would (correctly)
> > > > 	write the new value into rcu_flipctr_idx.
> > > > 
> > > > 	Well and good so far.  But the problem arises in
> > > > 	__rcu_read_unlock(), which then decrements the wrong counter.
> > > > 	Depending on exactly how subsequent events played out, this could
> > > > 	result in either prematurely ending grace periods or never-ending
> > > > 	grace periods, both of which are fatal outcomes.
> > > > 
> > > > And the following are not needed in the current version of the
> > > > patch, but will be in a future version that either avoids disabling
> > > > irqs or that dispenses with the smp_read_barrier_depends() that I
> > > > have 99% convinced myself is unneeded:
> > > > 
> > > > o	nesting = ORDERED_WRT_IRQ(me->rcu_read_lock_nesting);
> > > > 
> > > > o	idx = ORDERED_WRT_IRQ(rcu_ctrlblk.completed) & 0x1;
> > > > 
> > > > Furthermore, in that future version, irq handlers can cause the same
> > > > mischief that SMI/NMI handlers can in this version.
> 
> So don't remove the local_irq_save/restore, which is well-established and
> well-understood for such cases (it doesn't help you with SMI/NMI,
> admittedly). This isn't really about RCU or per-cpu vars as such, it's
> just about racy code where you don't want to get hit by a concurrent
> interrupt (it does turn out that doing things in a _particular order_ will
> not cause fatal/buggy behaviour, but it's still a race issue, after all).

The local_irq_save/restore are something like 30% of the overhead of
these two functions, so will be looking hard at getting rid of them.
Doing so allows the scheduling-clock interrupt to get into the mix,
and also allows preemption.  The goal would be to find some trick that
suppresses preemption, fends off the grace-period-computation code
invoked from the the scheduling-clock interrupt, and otherwise keeps
things on an even keel.

> > > > Next, looking at __rcu_read_unlock():
> > > > 
> > > > o	If "ORDERED_WRT_IRQ(me->rcu_read_lock_nesting) = nesting - 1"
> > > > 	was reordered by the compiler to follow the
> > > > 	"ORDERED_WRT_IRQ(__get_cpu_var(rcu_flipctr)[idx])--",
> > > > 	then if an NMI/SMI containing an rcu_read_lock() occurs between
> > > > 	the two, this nested rcu_read_lock() would incorrectly believe
> > > > 	that it was protected by an enclosing RCU read-side critical
> > > > 	section as described in the first reversal discussed for
> > > > 	__rcu_read_lock() above.  Again, fatal outcome.
> > > > 
> > > > This is what we have now.  It is not hard to imagine situations that
> > > > interact with -both- interrupt handlers -and- other CPUs, as described
> > > > earlier.
> 
> Unless somebody's going for a lockless implementation, such situations
> normally use spin_lock_irqsave() based locking (or local_irq_save for
> those who care only for current CPU) -- problem with the patch in question,
> is that you want to prevent races with concurrent SMI/NMIs as well, which
> is not something that a lot of code needs to consider.

Or that needs to resolve similar races with IRQs without disabling them.
One reason to avoid disabling IRQs is to avoid degrading scheduling
latency.  In any case, I do agree that the amount of code that must
worry about this is quite small at the moment.  I believe that it
will become more common, but would imagine that this belief might not
be universal.  Yet, anyway.  ;-)

> [ Curiously, another thread is discussing something similar also:
>   http://lkml.org/lkml/2007/8/15/393 "RFC: do get_rtc_time() correctly" ]
> 
> Anyway, I didn't look at the code in that patch very much in detail, but
> why couldn't you implement some kind of synchronization variable that lets
> rcu_read_lock() or rcu_read_unlock() -- when being called from inside an
> NMI or SMI handler -- know that it has concurrently interrupted an ongoing
> rcu_read_{un}lock() and so must do things differently ... (?)

Given some low-level details of the current implementation, I could
imagine manipulating rcu_read_lock_nesting on entry to and exit from
all NMI/SMI handlers, but would like to avoid that kind of architecture
dependency.  I am not confident of locating all of them, for one thing...

> I'm also wondering if there's other code that's not using locking in the
> kernel that faces similar issues, and what they've done to deal with it
> (if anything). Such bugs would be subtle, and difficult to diagnose.

Agreed!

							Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Linus Torvalds]

On Fri, 17 Aug 2007, Chris Friesen wrote:
> 
> I assume you mean "except for IO-related code and 'random' values like
> jiffies" as you mention later on?

Yes. There *are* valid uses for "volatile", but they have remained the 
same for the last few years:
 - "jiffies"
 - internal per-architecture IO implementations that can do them as normal 
   stores.

> I assume other values set in interrupt handlers would count as "random" 
> from a volatility perspective?

I don't really see any valid case. I can imagine that you have your own 
"jiffy" counter in a driver, but what's the point, really? I'd suggest not 
using volatile, and using barriers instead.

> 
> > So anybody who argues for "volatile" fixing bugs is fundamentally 
> > incorrect. It does NO SUCH THING. By arguing that, such people only 
> > show that you have no idea what they are talking about.

> What about reading values modified in interrupt handlers, as in your 
> "random" case?  Or is this a bug where the user of atomic_read() is 
> invalidly expecting a read each time it is called?

Quite frankly, the biggest reason for using "volatile" on jiffies was 
really historic. So even the "random" case is not really a very strong 
one. You'll notice that anybody who is actually careful will be using 
sequence locks for the jiffy accesses, if only because the *full* jiffy 
count is actually a 64-bit value, and so you cannot get it atomically on a 
32-bit architecture even on a single CPU (ie a timer interrupt might 
happen in between reading the low and the high word, so "volatile" is only 
used for the low 32 bits).

So even for jiffies, we actually have:

	extern u64 __jiffy_data jiffies_64;
	extern unsigned long volatile __jiffy_data jiffies;

where the *real* jiffies is not volatile: the volatile one is using linker 
tricks to alias the low 32 bits:

 - arch/i386/kernel/vmlinux.lds.S:

	...
	jiffies = jiffies_64;
	...

and the only reason we do all these games is (a) it works and (b) it's 
legacy.

Note how I do *not* say "(c) it's a good idea".

			Linus

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Arjan van de Ven]

On Fri, 2007-08-17 at 12:49 -0700, Paul E. McKenney wrote:
> > > What about reading values modified in interrupt handlers, as in your 
> > > "random" case?  Or is this a bug where the user of atomic_read() is 
> > > invalidly expecting a read each time it is called?
> > 
> > the interrupt handler case is an SMP case since you do not know
> > beforehand what cpu your interrupt handler will run on.
> 
> With the exception of per-CPU variables, yes.

if you're spinning waiting for a per-CPU variable to get changed by an
interrupt handler... you have bigger problems than "volatile" ;-)

-- 
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Fri, Aug 17, 2007 at 11:54:33AM -0700, Arjan van de Ven wrote:
> 
> On Fri, 2007-08-17 at 12:50 -0600, Chris Friesen wrote:
> > Linus Torvalds wrote:
> > 
> > >  - in other words, the *only* possible meaning for "volatile" is a purely 
> > >    single-CPU meaning. And if you only have a single CPU involved in the 
> > >    process, the "volatile" is by definition pointless (because even 
> > >    without a volatile, the compiler is required to make the C code appear 
> > >    consistent as far as a single CPU is concerned).
> > 
> > I assume you mean "except for IO-related code and 'random' values like 
> > jiffies" as you mention later on?  I assume other values set in 
> > interrupt handlers would count as "random" from a volatility perspective?
> > 
> > > So anybody who argues for "volatile" fixing bugs is fundamentally 
> > > incorrect. It does NO SUCH THING. By arguing that, such people only show 
> > > that you have no idea what they are talking about.
> > 
> > What about reading values modified in interrupt handlers, as in your 
> > "random" case?  Or is this a bug where the user of atomic_read() is 
> > invalidly expecting a read each time it is called?
> 
> the interrupt handler case is an SMP case since you do not know
> beforehand what cpu your interrupt handler will run on.

With the exception of per-CPU variables, yes.

							Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Fri, Aug 17, 2007 at 12:49:00PM -0700, Arjan van de Ven wrote:
> 
> On Fri, 2007-08-17 at 12:49 -0700, Paul E. McKenney wrote:
> > > > What about reading values modified in interrupt handlers, as in your 
> > > > "random" case?  Or is this a bug where the user of atomic_read() is 
> > > > invalidly expecting a read each time it is called?
> > > 
> > > the interrupt handler case is an SMP case since you do not know
> > > beforehand what cpu your interrupt handler will run on.
> > 
> > With the exception of per-CPU variables, yes.
> 
> if you're spinning waiting for a per-CPU variable to get changed by an
> interrupt handler... you have bigger problems than "volatile" ;-)

That would be true, if you were doing that.  But you might instead be
simply making sure that the mainline actions were seen in order by the
interrupt handler.  My current example is the NMI-save rcu_read_lock()
implementation for realtime.  Not the common case, I will admit, but
still real.  ;-)

						Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

> Of course, since *normal* accesses aren't necessarily limited wrt
> re-ordering, the question then becomes one of "with regard to *what* 
> does
> it limit re-ordering?".
>
> A C compiler that re-orders two different volatile accesses that have a
> sequence point in between them is pretty clearly a buggy compiler. So 
> at a
> minimum, it limits re-ordering wrt other volatiles (assuming sequence
> points exists). It also means that the compiler cannot move it
> speculatively across conditionals, but other than that it's starting to
> get fuzzy.

This is actually really well-defined in C, not fuzzy at all.
"Volatile accesses" are a side effect, and no side effects can
be reordered with respect to sequence points.  The side effects
that matter in the kernel environment are: 1) accessing a volatile
object; 2) modifying an object; 3) volatile asm(); 4) calling a
function that does any of these.

We certainly should avoid volatile whenever possible, but "because
it's fuzzy wrt reordering" is not a reason -- all alternatives have
exactly the same issues.


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

> (and yes, it is perfectly legitimate to
> want a non-volatile read for a data type that you also want to do
> atomic RMW operations on)

...which is undefined behaviour in C (and GCC) when that data is
declared volatile, which is a good argument against implementing
atomics that way in itself.


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

> In a reasonable world, gcc should just make that be (on x86)
>
> 	addl $1,i(%rip)
>
> on x86-64, which is indeed what it does without the volatile. But with 
> the
> volatile, the compiler gets really nervous, and doesn't dare do it in 
> one
> instruction, and thus generates crap like
>
>         movl    i(%rip), %eax
>         addl    $1, %eax
>         movl    %eax, i(%rip)
>
> instead. For no good reason, except that "volatile" just doesn't have 
> any
> good/clear semantics for the compiler, so most compilers will just 
> make it
> be "I will not touch this access in any way, shape, or form". Including
> even trivially correct instruction optimization/combination.

It's just a (target-specific, perhaps) missed-optimisation kind
of bug in GCC.  Care to file a bug report?

> but is
> (again) something that gcc doesn't dare do, since "i" is volatile.

Just nobody taught it it can do this; perhaps no one wanted to
add optimisations like that, maybe with a reasoning like "people
who hit the go-slow-in-unspecified-ways button should get what
they deserve" ;-)


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

> Now the second wording *IS* technically correct, but come on, it's
> 24 words long whereas the original one was 3 -- and hopefully anybody
> reading the shorter phrase *would* have known anyway what was meant,
> without having to be pedantic about it :-)

Well you were talking pretty formal (and detailed) stuff, so
IMHO it's good to have that exactly correct.  Sure it's nicer
to use small words most of the time :-)


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

>>> Here, I should obviously admit that the semantics of *(volatile int 
>>> *)&
>>> aren't any neater or well-defined in the _language standard_ at all. 
>>> The
>>> standard does say (verbatim) "precisely what constitutes as access to
>>> object of volatile-qualified type is implementation-defined", but GCC
>>> does help us out here by doing the right thing.
>>
>> Where do you get that idea?
>
> Try a testcase (experimentally verify).

That doesn't prove anything.  Experiments can only disprove
things.

>> GCC manual, section 6.1, "When
>> is a Volatile Object Accessed?" doesn't say anything of the
>> kind.
>
> True, "implementation-defined" as per the C standard _is_ supposed to 
> mean
> "unspecified behaviour where each implementation documents how the 
> choice
> is made". So ok, probably GCC isn't "documenting" this
> implementation-defined behaviour which it is supposed to, but can't 
> really
> fault them much for this, probably.

GCC _is_ documenting this, namely in this section 6.1.  It doesn't
mention volatile-casted stuff.  Draw your own conclusions.


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

> #define forget(a)	__asm__ __volatile__ ("" :"=m" (a) :"m" (a))
>
> [ This is exactly equivalent to using "+m" in the constraints, as 
> recently
>   explained on a GCC list somewhere, in response to the patch in my 
> bitops
>   series a few weeks back where I thought "+m" was bogus. ]

[It wasn't explained on a GCC list in response to your patch, as
far as I can see -- if I missed it, please point me to an archived
version of it].

One last time: it isn't equivalent on older (but still supported
by Linux) versions of GCC.  Current versions of GCC allow it, but
it has no documented behaviour at all, so use it at your own risk.


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

>>> No it does not have any volatile semantics. atomic_dec() can be 
>>> reordered
>>> at will by the compiler within the current basic unit if you do not 
>>> add a
>>> barrier.
>>
>> "volatile" has nothing to do with reordering.
>
> If you're talking of "volatile" the type-qualifier keyword, then
> http://lkml.org/lkml/2007/8/16/231 (and sub-thread below it) shows
> otherwise.

I'm not sure what in that mail you mean, but anyway...

Yes, of course, the fact that "volatile" creates a side effect
prevents certain things from being reordered wrt the atomic_dec();
but the atomic_dec() has a side effect *already* so the volatile
doesn't change anything.

>> atomic_dec() writes
>> to memory, so it _does_ have "volatile semantics", implicitly, as
>> long as the compiler cannot optimise the atomic variable away
>> completely -- any store counts as a side effect.
>
> I don't think an atomic_dec() implemented as an inline "asm volatile"
> or one that uses a "forget" macro would have the same re-ordering
> guarantees as an atomic_dec() that uses a volatile access cast.

The "asm volatile" implementation does have exactly the same
reordering guarantees as the "volatile cast" thing, if that is
implemented by GCC in the "obvious" way.  Even a "plain" asm()
will do the same.


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Sat, 18 Aug 2007, Segher Boessenkool wrote:

> > #define forget(a)	__asm__ __volatile__ ("" :"=m" (a) :"m" (a))
> > 
> > [ This is exactly equivalent to using "+m" in the constraints, as recently
> >   explained on a GCC list somewhere, in response to the patch in my bitops
> >   series a few weeks back where I thought "+m" was bogus. ]
> 
> [It wasn't explained on a GCC list in response to your patch, as
> far as I can see -- if I missed it, please point me to an archived
> version of it].

http://gcc.gnu.org/ml/gcc-patches/2007-07/msg01758.html

is a follow-up in the thread on the gcc-patches@gcc.gnu.org mailing list,
which began with:

http://gcc.gnu.org/ml/gcc-patches/2007-07/msg01677.html

that was posted by Jan Kubicka, as he quotes in that initial posting,
after I had submitted:

http://lkml.org/lkml/2007/7/23/252

which was a (wrong) patch to "rectify" what I thought was the "bogus"
"+m" constraint, as per the quoted extract from gcc docs (that was
given in that (wrong) patch's changelog).

That's when _I_ came to know how GCC interprets "+m", but probably
this has been explained on those lists multiple times. Who cares,
anyway?


> One last time: it isn't equivalent on older (but still supported
> by Linux) versions of GCC.  Current versions of GCC allow it, but
> it has no documented behaviour at all, so use it at your own risk.

True.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Sat, 18 Aug 2007, Segher Boessenkool wrote:

> > > > No it does not have any volatile semantics. atomic_dec() can be
> > > > reordered
> > > > at will by the compiler within the current basic unit if you do not add
> > > > a
> > > > barrier.
> > > 
> > > "volatile" has nothing to do with reordering.
> > 
> > If you're talking of "volatile" the type-qualifier keyword, then
> > http://lkml.org/lkml/2007/8/16/231 (and sub-thread below it) shows
> > otherwise.
> 
> I'm not sure what in that mail you mean, but anyway...
> 
> Yes, of course, the fact that "volatile" creates a side effect
> prevents certain things from being reordered wrt the atomic_dec();
> but the atomic_dec() has a side effect *already* so the volatile
> doesn't change anything.

That's precisely what that sub-thread (read down to the last mail
there, and not the first mail only) shows. So yes, "volatile" does
have something to do with re-ordering (as guaranteed by the C
standard).


> > > atomic_dec() writes
> > > to memory, so it _does_ have "volatile semantics", implicitly, as
> > > long as the compiler cannot optimise the atomic variable away
> > > completely -- any store counts as a side effect.
> > 
> > I don't think an atomic_dec() implemented as an inline "asm volatile"
> > or one that uses a "forget" macro would have the same re-ordering
> > guarantees as an atomic_dec() that uses a volatile access cast.
> 
> The "asm volatile" implementation does have exactly the same
> reordering guarantees as the "volatile cast" thing,

I don't think so.

> if that is
> implemented by GCC in the "obvious" way.  Even a "plain" asm()
> will do the same.

Read the relevant GCC documentation.

[ of course, if the (latest) GCC documentation is *yet again*
  wrong, then alright, not much I can do about it, is there. ]

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

>>> #define forget(a)	__asm__ __volatile__ ("" :"=m" (a) :"m" (a))
>>>
>>> [ This is exactly equivalent to using "+m" in the constraints, as 
>>> recently
>>>   explained on a GCC list somewhere, in response to the patch in my 
>>> bitops
>>>   series a few weeks back where I thought "+m" was bogus. ]
>>
>> [It wasn't explained on a GCC list in response to your patch, as
>> far as I can see -- if I missed it, please point me to an archived
>> version of it].
>
> http://gcc.gnu.org/ml/gcc-patches/2007-07/msg01758.html

Ah yes, that old thread, thank you.

> That's when _I_ came to know how GCC interprets "+m", but probably
> this has been explained on those lists multiple times. Who cares,
> anyway?

I just couldn't find the thread you meant, I thought I missed
have it, that's all :-)


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Thu, Aug 16, 2007 at 08:50:30PM -0700, Linus Torvalds wrote:
> Just try it yourself:
> 
> 	volatile int i;
> 	int j;
> 
> 	int testme(void)
> 	{
> 	        return i <= 1;
> 	}
> 
> 	int testme2(void)
> 	{
> 	        return j <= 1;
> 	}
> 
> and compile with all the optimizations you can.
> 
> I get:
> 
> 	testme:
> 	        movl    i(%rip), %eax
> 	        subl    $1, %eax
> 	        setle   %al
> 	        movzbl  %al, %eax
> 	        ret
> 
> vs
> 
> 	testme2:
> 	        xorl    %eax, %eax
> 	        cmpl    $1, j(%rip)
> 	        setle   %al
> 	        ret
> 
> (now, whether that "xorl + setle" is better than "setle + movzbl", I don't 
> really know - maybe it is. But that's not thepoint. The point is the 
> difference between
> 
>                 movl    i(%rip), %eax
>                 subl    $1, %eax
> 
> and
> 
>                 cmpl    $1, j(%rip)

gcc bugzilla bug #33102, for whatever that ends up being worth.  ;-)

							Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

>>>> atomic_dec() writes
>>>> to memory, so it _does_ have "volatile semantics", implicitly, as
>>>> long as the compiler cannot optimise the atomic variable away
>>>> completely -- any store counts as a side effect.
>>>
>>> I don't think an atomic_dec() implemented as an inline "asm volatile"
>>> or one that uses a "forget" macro would have the same re-ordering
>>> guarantees as an atomic_dec() that uses a volatile access cast.
>>
>> The "asm volatile" implementation does have exactly the same
>> reordering guarantees as the "volatile cast" thing,
>
> I don't think so.

"asm volatile" creates a side effect.  Side effects aren't
allowed to be reordered wrt sequence points.  This is exactly
the same reason as why "volatile accesses" cannot be reordered.

>> if that is
>> implemented by GCC in the "obvious" way.  Even a "plain" asm()
>> will do the same.
>
> Read the relevant GCC documentation.

I did, yes.

> [ of course, if the (latest) GCC documentation is *yet again*
>   wrong, then alright, not much I can do about it, is there. ]

There was (and is) nothing wrong about the "+m" documentation, if
that is what you are talking about.  It could be extended now, to
allow "+m" -- but that takes more than just "fixing" the documentation.


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

On Fri, Aug 17, 2007 at 04:59:12PM -0700, Paul E. McKenney wrote:
>
> gcc bugzilla bug #33102, for whatever that ends up being worth.  ;-)

I had totally forgotten that I'd already filed that bug more
than six years ago until they just closed yours as a duplicate
of mine :)

Good luck in getting it fixed!

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Sat, Aug 18, 2007 at 08:09:13AM +0800, Herbert Xu wrote:
> On Fri, Aug 17, 2007 at 04:59:12PM -0700, Paul E. McKenney wrote:
> >
> > gcc bugzilla bug #33102, for whatever that ends up being worth.  ;-)
> 
> I had totally forgotten that I'd already filed that bug more
> than six years ago until they just closed yours as a duplicate
> of mine :)
> 
> Good luck in getting it fixed!

Well, just got done re-opening it for the third time.  And a local
gcc community member advised me not to give up too easily.  But I
must admit that I am impressed with the speed that it was identified
as duplicate.

Should be entertaining!  ;-)

						Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Christoph Lameter]

On Fri, 17 Aug 2007, Paul E. McKenney wrote:

> On Sat, Aug 18, 2007 at 08:09:13AM +0800, Herbert Xu wrote:
> > On Fri, Aug 17, 2007 at 04:59:12PM -0700, Paul E. McKenney wrote:
> > >
> > > gcc bugzilla bug #33102, for whatever that ends up being worth.  ;-)
> > 
> > I had totally forgotten that I'd already filed that bug more
> > than six years ago until they just closed yours as a duplicate
> > of mine :)
> > 
> > Good luck in getting it fixed!
> 
> Well, just got done re-opening it for the third time.  And a local
> gcc community member advised me not to give up too easily.  But I
> must admit that I am impressed with the speed that it was identified
> as duplicate.
> 
> Should be entertaining!  ;-)

Right. ROTFL... volatile actually breaks atomic_t instead of making it 
safe. x++ becomes a register load, increment and a register store. Without 
volatile we can increment the memory directly. It seems that volatile 
requires that the variable is loaded into a register first and then 
operated upon. Understandable when you think about volatile being used to 
access memory mapped I/O registers where a RMW operation could be 
problematic.

See http://gcc.gnu.org/bugzilla/show_bug.cgi?id=3506

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Fri, 17 Aug 2007, Christoph Lameter wrote:

> On Fri, 17 Aug 2007, Paul E. McKenney wrote:
> 
> > On Sat, Aug 18, 2007 at 08:09:13AM +0800, Herbert Xu wrote:
> > > On Fri, Aug 17, 2007 at 04:59:12PM -0700, Paul E. McKenney wrote:
> > > >
> > > > gcc bugzilla bug #33102, for whatever that ends up being worth.  ;-)
> > > 
> > > I had totally forgotten that I'd already filed that bug more
> > > than six years ago until they just closed yours as a duplicate
> > > of mine :)
> > > 
> > > Good luck in getting it fixed!
> > 
> > Well, just got done re-opening it for the third time.  And a local
> > gcc community member advised me not to give up too easily.  But I
> > must admit that I am impressed with the speed that it was identified
> > as duplicate.
> > 
> > Should be entertaining!  ;-)
> 
> Right. ROTFL... volatile actually breaks atomic_t instead of making it 
> safe. x++ becomes a register load, increment and a register store. Without 
> volatile we can increment the memory directly.

No code does (or would do, or should do):

	x.counter++;

on an "atomic_t x;" anyway.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Sat, 18 Aug 2007, Segher Boessenkool wrote:

> > > > > atomic_dec() writes
> > > > > to memory, so it _does_ have "volatile semantics", implicitly, as
> > > > > long as the compiler cannot optimise the atomic variable away
> > > > > completely -- any store counts as a side effect.
> > > > 
> > > > I don't think an atomic_dec() implemented as an inline "asm volatile"
> > > > or one that uses a "forget" macro would have the same re-ordering
> > > > guarantees as an atomic_dec() that uses a volatile access cast.
> > > 
> > > The "asm volatile" implementation does have exactly the same
> > > reordering guarantees as the "volatile cast" thing,
> > 
> > I don't think so.
> 
> "asm volatile" creates a side effect.

Yeah.

> Side effects aren't
> allowed to be reordered wrt sequence points.

Yeah.

> This is exactly
> the same reason as why "volatile accesses" cannot be reordered.

No, the code in that sub-thread I earlier pointed you at *WAS* written
such that there was a sequence point after all the uses of that volatile
access cast macro, and _therefore_ we were safe from re-ordering
(behaviour guaranteed by the C standard).

But you seem to be missing the simple and basic fact that:

	(something_that_has_side_effects || statement)
			!= something_that_is_a_sequence_point

Now, one cannot fantasize that "volatile asms" are also sequence points.
In fact such an argument would be sadly mistaken, because "sequence
points" are defined by the C standard and it'd be horribly wrong to
even _try_ claiming that the C standard knows about "volatile asms".


> > > if that is
> > > implemented by GCC in the "obvious" way.  Even a "plain" asm()
> > > will do the same.
> > 
> > Read the relevant GCC documentation.
> 
> I did, yes.

No, you didn't read:

http://gcc.gnu.org/onlinedocs/gcc/Extended-Asm.html

Read the bit about the need for artificial dependencies, and the example
given there:

	asm volatile("mtfsf 255,%0" : : "f" (fpenv));
	sum = x + y;

The docs explicitly say the addition can be moved before the "volatile
asm". Hopefully, as you know, (x + y) is an C "expression" and hence
a "sequence point" as defined by the standard. So the "volatile asm"
should've happened before it, right? Wrong.

I know there is also stuff written about "side-effects" there which
_could_ give the same semantic w.r.t. sequence points as the volatile
access casts, but hey, it's GCC's own documentation, you obviously can't
find fault with _me_ if there's wrong stuff written in there. Say that
to GCC ...

See, "volatile" C keyword, for all it's ill-definition and dodgy
semantics, is still at least given somewhat of a treatment in the C
standard (whose quality is ... ummm, sadly not always good and clear,
but unsurprisingly, still about 5,482 orders-of-magnitude times
better than GCC docs). Semantics of "volatile" as applies to inline
asm, OTOH? You're completely relying on the compiler for that ...


> > [ of course, if the (latest) GCC documentation is *yet again*
> >   wrong, then alright, not much I can do about it, is there. ]
> 
> There was (and is) nothing wrong about the "+m" documentation, if
> that is what you are talking about.  It could be extended now, to
> allow "+m" -- but that takes more than just "fixing" the documentation.

No, there was (and is) _everything_ wrong about the "+" documentation as
applies to memory-constrained operands. I don't give a whit if it's
some workaround in their gimplifier, or the other, that makes it possible
to use "+m" (like the current kernel code does). The docs suggest
otherwise, so there's obviously a clear disconnect between the docs and
actual GCC behaviour.


[ You seem to often take issue with _amazingly_ petty and pedantic things,
  by the way :-) ]

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Fri, 17 Aug 2007, Nick Piggin wrote:

> Satyam Sharma wrote:
> 
> > I didn't quite understand what you said here, so I'll tell what I think:
> > 
> > * foo() is a compiler barrier if the definition of foo() is invisible to
> >  the compiler at a callsite.
> > 
> > * foo() is also a compiler barrier if the definition of foo() includes
> >  a barrier, and it is inlined at the callsite.
> > 
> > If the above is wrong, or if there's something else at play as well,
> > do let me know.
> 
> [...]
> If a function is not completely visible to the compiler (so it can't
> determine whether a barrier could be in it or not), then it must always
> assume it will contain a barrier so it always does the right thing.

Yup, that's what I'd said just a few sentences above, as you can see. I
was actually asking for "elaboration" on "how a compiler determines that
function foo() (say foo == schedule), even when it cannot see that it has
a barrier(), as you'd mentioned, is a 'sleeping' function" actually, and
whether compilers have a "notion of sleep to automatically assume a
compiler barrier whenever such a sleeping function foo() is called". But
I think you've already qualified the discussion to this kernel, so okay,
I shouldn't nit-pick anymore.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

>>>> The "asm volatile" implementation does have exactly the same
>>>> reordering guarantees as the "volatile cast" thing,
>>>
>>> I don't think so.
>>
>> "asm volatile" creates a side effect.
>
> Yeah.
>
>> Side effects aren't
>> allowed to be reordered wrt sequence points.
>
> Yeah.
>
>> This is exactly
>> the same reason as why "volatile accesses" cannot be reordered.
>
> No, the code in that sub-thread I earlier pointed you at *WAS* written
> such that there was a sequence point after all the uses of that 
> volatile
> access cast macro, and _therefore_ we were safe from re-ordering
> (behaviour guaranteed by the C standard).

And exactly the same is true for the "asm" version.

> Now, one cannot fantasize that "volatile asms" are also sequence 
> points.

Sure you can do that.  I don't though.

> In fact such an argument would be sadly mistaken, because "sequence
> points" are defined by the C standard and it'd be horribly wrong to
> even _try_ claiming that the C standard knows about "volatile asms".

That's nonsense.  GCC can extend the C standard any way they
bloody well please -- witness the fact that they added an
extra class of side effects...

>>> Read the relevant GCC documentation.
>>
>> I did, yes.
>
> No, you didn't read:
>
> http://gcc.gnu.org/onlinedocs/gcc/Extended-Asm.html
>
> Read the bit about the need for artificial dependencies, and the 
> example
> given there:
>
> 	asm volatile("mtfsf 255,%0" : : "f" (fpenv));
> 	sum = x + y;
>
> The docs explicitly say the addition can be moved before the "volatile
> asm". Hopefully, as you know, (x + y) is an C "expression" and hence
> a "sequence point" as defined by the standard.

The _end of a full expression_ is a sequence point, not every
expression.  And that is irrelevant here anyway.

It is perfectly fine to compute x+y any time before the
assignment; the C compiler is allowed to compute it _after_
the assignment even, if it could figure out how ;-)

x+y does not contain a side effect, you know.

> I know there is also stuff written about "side-effects" there which
> _could_ give the same semantic w.r.t. sequence points as the volatile
> access casts,

s/could/does/

> but hey, it's GCC's own documentation, you obviously can't
> find fault with _me_ if there's wrong stuff written in there. Say that
> to GCC ...

There's nothing wrong there.

> See, "volatile" C keyword, for all it's ill-definition and dodgy
> semantics, is still at least given somewhat of a treatment in the C
> standard (whose quality is ... ummm, sadly not always good and clear,
> but unsurprisingly, still about 5,482 orders-of-magnitude times
> better than GCC docs).

If you find any problems/shortcomings in the GCC documentation,
please file a PR, don't go whine on some unrelated mailing lists.
Thank you.

> Semantics of "volatile" as applies to inline
> asm, OTOH? You're completely relying on the compiler for that ...

Yes, and?  GCC promises the behaviour it has documented.

>>> [ of course, if the (latest) GCC documentation is *yet again*
>>>   wrong, then alright, not much I can do about it, is there. ]
>>
>> There was (and is) nothing wrong about the "+m" documentation, if
>> that is what you are talking about.  It could be extended now, to
>> allow "+m" -- but that takes more than just "fixing" the 
>> documentation.
>
> No, there was (and is) _everything_ wrong about the "+" documentation 
> as
> applies to memory-constrained operands. I don't give a whit if it's
> some workaround in their gimplifier, or the other, that makes it 
> possible
> to use "+m" (like the current kernel code does). The docs suggest
> otherwise, so there's obviously a clear disconnect between the docs and
> actual GCC behaviour.

The documentation simply doesn't say "+m" is allowed.  The code to
allow it was added for the benefit of people who do not read the
documentation.  Documentation for "+m" might get added later if it
is decided this [the code, not the documentation] is a sane thing
to have (which isn't directly obvious).

> [ You seem to often take issue with _amazingly_ petty and pedantic 
> things,
>   by the way :-) ]

If you're talking details, you better get them right.  Handwaving is
fine with me as long as you're not purporting you're not.

And I simply cannot stand false assertions.

You can always ignore me if _you_ take issue with _that_ :-)


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Sat, 18 Aug 2007, Segher Boessenkool wrote:

> > > > > The "asm volatile" implementation does have exactly the same
> > > > > reordering guarantees as the "volatile cast" thing,
> > > > 
> > > > I don't think so.
> > > 
> > > "asm volatile" creates a side effect.
> > 
> > Yeah.
> > 
> > > Side effects aren't
> > > allowed to be reordered wrt sequence points.
> > 
> > Yeah.
> > 
> > > This is exactly
> > > the same reason as why "volatile accesses" cannot be reordered.
> > 
> > No, the code in that sub-thread I earlier pointed you at *WAS* written
> > such that there was a sequence point after all the uses of that volatile
> > access cast macro, and _therefore_ we were safe from re-ordering
> > (behaviour guaranteed by the C standard).
> 
> And exactly the same is true for the "asm" version.
> 
> > Now, one cannot fantasize that "volatile asms" are also sequence points.
> 
> Sure you can do that.  I don't though.
> 
> > In fact such an argument would be sadly mistaken, because "sequence
> > points" are defined by the C standard and it'd be horribly wrong to
> > even _try_ claiming that the C standard knows about "volatile asms".
> 
> That's nonsense.  GCC can extend the C standard any way they
> bloody well please -- witness the fact that they added an
> extra class of side effects...
> 
> > > > Read the relevant GCC documentation.
> > > 
> > > I did, yes.
> > 
> > No, you didn't read:
> > 
> > http://gcc.gnu.org/onlinedocs/gcc/Extended-Asm.html
> > 
> > Read the bit about the need for artificial dependencies, and the example
> > given there:
> > 
> > 	asm volatile("mtfsf 255,%0" : : "f" (fpenv));
> > 	sum = x + y;
> > 
> > The docs explicitly say the addition can be moved before the "volatile
> > asm". Hopefully, as you know, (x + y) is an C "expression" and hence
> > a "sequence point" as defined by the standard.
> 
> The _end of a full expression_ is a sequence point, not every
> expression.  And that is irrelevant here anyway.
> 
> It is perfectly fine to compute x+y any time before the
> assignment; the C compiler is allowed to compute it _after_
> the assignment even, if it could figure out how ;-)
> 
> x+y does not contain a side effect, you know.
> 
> > I know there is also stuff written about "side-effects" there which
> > _could_ give the same semantic w.r.t. sequence points as the volatile
> > access casts,
> 
> s/could/does/
> 
> > but hey, it's GCC's own documentation, you obviously can't
> > find fault with _me_ if there's wrong stuff written in there. Say that
> > to GCC ...
> 
> There's nothing wrong there.
> 
> > See, "volatile" C keyword, for all it's ill-definition and dodgy
> > semantics, is still at least given somewhat of a treatment in the C
> > standard (whose quality is ... ummm, sadly not always good and clear,
> > but unsurprisingly, still about 5,482 orders-of-magnitude times
> > better than GCC docs).
> 
> If you find any problems/shortcomings in the GCC documentation,
> please file a PR, don't go whine on some unrelated mailing lists.
> Thank you.
> 
> > Semantics of "volatile" as applies to inline
> > asm, OTOH? You're completely relying on the compiler for that ...
> 
> Yes, and?  GCC promises the behaviour it has documented.

LOTS there, which obviously isn't correct, but which I'll reply to later,
easier stuff first. (call this "handwaving" if you want, but don't worry,
I /will/ bother myself to reply)


> > > > [ of course, if the (latest) GCC documentation is *yet again*
> > > >   wrong, then alright, not much I can do about it, is there. ]
> > > 
> > > There was (and is) nothing wrong about the "+m" documentation, if
> > > that is what you are talking about.  It could be extended now, to
> > > allow "+m" -- but that takes more than just "fixing" the documentation.
> > 
> > No, there was (and is) _everything_ wrong about the "+" documentation as
> > applies to memory-constrained operands. I don't give a whit if it's
> > some workaround in their gimplifier, or the other, that makes it possible
> > to use "+m" (like the current kernel code does). The docs suggest
> > otherwise, so there's obviously a clear disconnect between the docs and
> > actual GCC behaviour.
> 
> The documentation simply doesn't say "+m" is allowed.  The code to
> allow it was added for the benefit of people who do not read the
> documentation.  Documentation for "+m" might get added later if it
> is decided this [the code, not the documentation] is a sane thing
> to have (which isn't directly obvious).

Huh?

"If the (current) documentation doesn't match up with the (current)
code, then _at least one_ of them has to be (as of current) wrong."

I wonder how could you even try to disagree with that.

And I didn't go whining about this ... you asked me. (I think I'd said
something to the effect of GCC docs are often wrong, which is true,
but probably you feel saying that is "not allowed" on non-gcc lists?)

As for the "PR" you're requesting me to file with GCC for this, that
gcc-patches@ thread did precisely that and more (submitted a patch to
said documentation -- and no, saying "documentation might get added
later" is totally bogus and nonsensical -- documentation exists to
document current behaviour, not past). But come on, this is wholly
petty. I wouldn't have replied, really, if you weren't so provoking.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Linus Torvalds]

On Sat, 18 Aug 2007, Satyam Sharma wrote:
> 
> No code does (or would do, or should do):
> 
> 	x.counter++;
> 
> on an "atomic_t x;" anyway.

That's just an example of a general problem.

No, you don't use "x.counter++". But you *do* use

	if (atomic_read(&x) <= 1)

and loading into a register is stupid and pointless, when you could just 
do it as a regular memory-operand to the cmp instruction.

And as far as the compiler is concerned, the problem is the 100% same: 
combining operations with the volatile memop.

The fact is, a compiler that thinks that

	movl mem,reg
	cmpl $val,reg

is any better than

	cmpl $val,mem

is just not a very good compiler. But when talking about "volatile", 
that's exactly what ytou always get (and always have gotten - this is 
not a regression, and I doubt gcc is alone in this).

			Linus

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

>> The documentation simply doesn't say "+m" is allowed.  The code to
>> allow it was added for the benefit of people who do not read the
>> documentation.  Documentation for "+m" might get added later if it
>> is decided this [the code, not the documentation] is a sane thing
>> to have (which isn't directly obvious).
>
> Huh?
>
> "If the (current) documentation doesn't match up with the (current)
> code, then _at least one_ of them has to be (as of current) wrong."
>
> I wonder how could you even try to disagree with that.

Easy.

The GCC documentation you're referring to is the user's manual.
See the blurb on the first page:

"This manual documents how to use the GNU compilers, as well as their
features and incompatibilities, and how to report bugs.  It corresponds
to GCC version 4.3.0.  The internals of the GNU compilers, including
how to port them to new targets and some information about how to write
front ends for new languages, are documented in a separate manual."

_How to use_.  This documentation doesn't describe in minute detail
everything the compiler does (see the source code for that -- no, it
isn't described in the internals manual either).

If it doesn't tell you how to use "+m", and even tells you _not_ to
use it, maybe that is what it means to say?  It doesn't mean "+m"
doesn't actually do something.  It also doesn't mean it does what
you think it should do.  It might do just that of course.  But treating
writing C code as an empirical science isn't such a smart idea.

> And I didn't go whining about this ... you asked me. (I think I'd said
> something to the effect of GCC docs are often wrong,

No need to guess at what you said, even if you managed to delete
your own mail already, there are plenty of free web-based archives
around.  You said:

> See, "volatile" C keyword, for all it's ill-definition and dodgy
> semantics, is still at least given somewhat of a treatment in the C
> standard (whose quality is ... ummm, sadly not always good and clear,
> but unsurprisingly, still about 5,482 orders-of-magnitude times
> better than GCC docs).

and that to me reads as complaining that the ISO C standard "isn't
very good" and that the GCC documentation is 10**5482 times worse
even.  Which of course is hyperbole and cannot be true.  It also
isn't helpful in any way or form for anyone on this list.  I call
that whining.

> which is true,

Yes, documentation of that size often has shortcomings.  No surprise
there.  However, great effort is made to make it better documentation,
and especially to keep it up to date; if you find any errors or
omissions, please report them.  There are many ways how to do that,
see the GCC homepage.</end-of-marketing-blurb>

> but probably you feel saying that is "not allowed" on non-gcc lists?)

You're allowed to say whatever you want.  Let's have a quote again
shall we?  I said:

> If you find any problems/shortcomings in the GCC documentation,
> please file a PR, don't go whine on some unrelated mailing lists.
> Thank you.

I read that as a friendly request, not a prohibition.  Well maybe
not actually friendly, more a bit angry.  A request, either way.

> As for the "PR"

"Problem report", a bugzilla ticket.  Sorry for using terminology
unknown to you.

> you're requesting me to file with GCC for this, that
> gcc-patches@ thread did precisely that

Actually not -- PRs make sure issues aren't forgotten (although
they might gather dust, sure).  But yes, submitting patches is a
Great Thing(tm).

> and more (submitted a patch to
> said documentation -- and no, saying "documentation might get added
> later" is totally bogus and nonsensical -- documentation exists to
> document current behaviour, not past).

When code like you want to write becomes a supported feature, that
will be reflected in the user manual.  It is completely nonsensical
to expect everything that is *not* a supported feature to be mentioned
there.

> I wouldn't have replied, really, if you weren't so provoking.

Hey, maybe that character trait is good for something, then.
Now to build a business plan around it...


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

[ LOL, you _are_ shockingly petty! ]


On Sat, 18 Aug 2007, Segher Boessenkool wrote:

> > > The documentation simply doesn't say "+m" is allowed.  The code to
> > > allow it was added for the benefit of people who do not read the
> > > documentation.  Documentation for "+m" might get added later if it
> > > is decided this [the code, not the documentation] is a sane thing
> > > to have (which isn't directly obvious).
> > 
> > Huh?
> > 
> > "If the (current) documentation doesn't match up with the (current)
> > code, then _at least one_ of them has to be (as of current) wrong."
> > 
> > I wonder how could you even try to disagree with that.
> 
> Easy.
> 
> The GCC documentation you're referring to is the user's manual.
> See the blurb on the first page:
> 
> "This manual documents how to use the GNU compilers, as well as their
> features and incompatibilities, and how to report bugs.  It corresponds
> to GCC version 4.3.0.  The internals of the GNU compilers, including
> how to port them to new targets and some information about how to write
> front ends for new languages, are documented in a separate manual."
> 
> _How to use_.  This documentation doesn't describe in minute detail
> everything the compiler does (see the source code for that -- no, it
> isn't described in the internals manual either).

Wow, now that's a nice "disclaimer". By your (poor) standards of writing
documentation, one can as well write any factually incorrect stuff that
one wants in a document once you've got such a blurb in place :-)


> If it doesn't tell you how to use "+m", and even tells you _not_ to
> use it, maybe that is what it means to say?  It doesn't mean "+m"
> doesn't actually do something.  It also doesn't mean it does what
> you think it should do.  It might do just that of course.  But treating
> writing C code as an empirical science isn't such a smart idea.

Oh, really? Considering how much is (left out of being) documented, often
one would reasonably have to experimentally see (with testcases) how the
compiler behaves for some given code. Well, at least _I_ do it often
(several others on this list do as well), and I think there's everything
smart about it rather than having to read gcc sources -- I'd be surprised
(unless you have infinite free time on your hands, which does look like
teh case actually) if someone actually prefers reading gcc sources first
to know what/how gcc does something for some given code, rather than
simply write it out, compile and look the generated code (saves time for
those who don't have an infinite amount of it).


> > And I didn't go whining about this ... you asked me. (I think I'd said
> > something to the effect of GCC docs are often wrong,
> 
> No need to guess at what you said, even if you managed to delete
> your own mail already, there are plenty of free web-based archives
> around.  You said:
> 
> > See, "volatile" C keyword, for all it's ill-definition and dodgy
> > semantics, is still at least given somewhat of a treatment in the C
> > standard (whose quality is ... ummm, sadly not always good and clear,
> > but unsurprisingly, still about 5,482 orders-of-magnitude times
> > better than GCC docs).

Try _reading_ what I said there, for a change, dude. I'd originally only
said "unless GCC's docs is yet again wrong" ... then _you_ asked me what,
after which this discussion began and I wrote the above [which I fully
agree with -- so what if I used hyperbole in my sentence (yup, that was
intended, and obviously, exaggeration), am I not even allowed to do that?
Man, you're a Nazi or what ...] I didn't go whining about on my own as
you'd had earlier suggested, until _you_ asked me.

[ Ick, I somehow managed to reply this ... this is such a ...
  *disgustingly* petty argument you made here. ]


> > which is true,
> 
> Yes, documentation of that size often has shortcomings.  No surprise
> there.  However, great effort is made to make it better documentation,
> and especially to keep it up to date; if you find any errors or
> omissions, please report them.  There are many ways how to do that,
> see the GCC homepage.</end-of-marketing-blurb>
                         ^^^^^^^^^^^^^^^^^^^^^^

Looks like you even get paid :-)


> > but probably you feel saying that is "not allowed" on non-gcc lists?)
> 
> [amazingly pointless stuff snipped]
> 
> > As for the "PR"
> > you're requesting me to file with GCC for this, that
> > gcc-patches@ thread did precisely that
> 
> [more amazingly pointless stuff snipped]
> 
> > and more (submitted a patch to
> > said documentation -- and no, saying "documentation might get added
> > later" is totally bogus and nonsensical -- documentation exists to
> > document current behaviour, not past).
> 
> When code like you want to write becomes a supported feature, that
> will be reflected in the user manual.  It is completely nonsensical
> to expect everything that is *not* a supported feature to be mentioned
> there.

What crap. It is _perfectly reasonable_ to expect (current) documentation
to keep up with (current) code behaviour. In fact trying to justify such
a state is completely bogus and nonsensical.


Satyam

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Fri, 17 Aug 2007, Linus Torvalds wrote:

> On Sat, 18 Aug 2007, Satyam Sharma wrote:
> > 
> > No code does (or would do, or should do):
> > 
> > 	x.counter++;
> > 
> > on an "atomic_t x;" anyway.
> 
> That's just an example of a general problem.
> 
> No, you don't use "x.counter++". But you *do* use
> 
> 	if (atomic_read(&x) <= 1)
> 
> and loading into a register is stupid and pointless, when you could just 
> do it as a regular memory-operand to the cmp instruction.

True, but that makes this a bad/poor code generation issue with the
compiler, not something that affects the _correctness_ of atomic ops if
"volatile" is used for that counter object (as was suggested), because
we'd always use the atomic_inc() etc primitives to do increments, which
are always (should be!) implemented to be atomic.


> And as far as the compiler is concerned, the problem is the 100% same: 
> combining operations with the volatile memop.
> 
> The fact is, a compiler that thinks that
> 
> 	movl mem,reg
> 	cmpl $val,reg
> 
> is any better than
> 
> 	cmpl $val,mem
> 
> is just not a very good compiler.

Absolutely, this is definitely a bug report worth opening with gcc. And
what you've said to explain this previously sounds definitely correct --
seeing "volatile" for any access does appear to just scare the hell out
of gcc and makes it generate such (poor) code.


Satyam

LDD3 pitfalls (was Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures)

[Stefan Richter]

Nick Piggin wrote:
> Stefan Richter wrote:
>> Nick Piggin wrote:
>>
>>> I don't know why people would assume volatile of atomics. AFAIK, most
>>> of the documentation is pretty clear that all the atomic stuff can be
>>> reordered etc. except for those that modify and return a value.
>>
>>
>> Which documentation is there?
> 
> Documentation/atomic_ops.txt
> 
> 
>> For driver authors, there is LDD3.  It doesn't specifically cover
>> effects of optimization on accesses to atomic_t.
>>
>> For architecture port authors, there is Documentation/atomic_ops.txt.
>> Driver authors also can learn something from that document, as it
>> indirectly documents the atomic_t and bitops APIs.
>>
> 
> "Semantics and Behavior of Atomic and Bitmask Operations" is
> pretty direct :)
> 
> Sure, it says that it's for arch maintainers, but there is no
> reason why users can't make use of it.


Note, LDD3 page 238 says:  "It is worth noting that most of the other
kernel primitives dealing with synchronization, such as spinlock and
atomic_t operations, also function as memory barriers."

I don't know about Linux 2.6.10 against which LDD3 was written, but
currently only _some_ atomic_t operations function as memory barriers.

Besides, judging from some posts in this thread, saying that atomic_t
operations dealt with synchronization may not be entirely precise.
-- 
Stefan Richter
-=====-=-=== =--- =--=-
http://arcgraph.de/sr/

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Sat, 18 Aug 2007, Segher Boessenkool wrote:

> > > GCC manual, section 6.1, "When
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > is a Volatile Object Accessed?" doesn't say anything of the
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > kind.
      ^^^^^

> > True, "implementation-defined" as per the C standard _is_ supposed to mean
    ^^^^^

> > "unspecified behaviour where each implementation documents how the choice
> > is made". So ok, probably GCC isn't "documenting" this
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

> > implementation-defined behaviour which it is supposed to, but can't really
> > fault them much for this, probably.
> 
> GCC _is_ documenting this, namely in this section 6.1.

(Again totally petty, but) Yes, but ...

> It doesn't
  ^^^^^^^^^^
> mention volatile-casted stuff.  Draw your own conclusions.
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

... exactly. So that's why I said "GCC isn't documenting _this_".

Man, try _reading_ mails before replying to them ...

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Fri, Aug 17, 2007 at 09:13:35PM -0700, Linus Torvalds wrote:
> 
> 
> On Sat, 18 Aug 2007, Satyam Sharma wrote:
> > 
> > No code does (or would do, or should do):
> > 
> > 	x.counter++;
> > 
> > on an "atomic_t x;" anyway.
> 
> That's just an example of a general problem.
> 
> No, you don't use "x.counter++". But you *do* use
> 
> 	if (atomic_read(&x) <= 1)
> 
> and loading into a register is stupid and pointless, when you could just 
> do it as a regular memory-operand to the cmp instruction.
> 
> And as far as the compiler is concerned, the problem is the 100% same: 
> combining operations with the volatile memop.
> 
> The fact is, a compiler that thinks that
> 
> 	movl mem,reg
> 	cmpl $val,reg
> 
> is any better than
> 
> 	cmpl $val,mem
> 
> is just not a very good compiler. But when talking about "volatile", 
> that's exactly what ytou always get (and always have gotten - this is 
> not a regression, and I doubt gcc is alone in this).

One of the gcc guys claimed that he thought that the two-instruction
sequence would be faster on some x86 machines.  I pointed out that
there might be a concern about code size.  I chose not to point out
that people might also care about the other x86 machines.  ;-)

							Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Fri, Aug 17, 2007 at 06:24:15PM -0700, Christoph Lameter wrote:
> On Fri, 17 Aug 2007, Paul E. McKenney wrote:
> 
> > On Sat, Aug 18, 2007 at 08:09:13AM +0800, Herbert Xu wrote:
> > > On Fri, Aug 17, 2007 at 04:59:12PM -0700, Paul E. McKenney wrote:
> > > >
> > > > gcc bugzilla bug #33102, for whatever that ends up being worth.  ;-)
> > > 
> > > I had totally forgotten that I'd already filed that bug more
> > > than six years ago until they just closed yours as a duplicate
> > > of mine :)
> > > 
> > > Good luck in getting it fixed!
> > 
> > Well, just got done re-opening it for the third time.  And a local
> > gcc community member advised me not to give up too easily.  But I
> > must admit that I am impressed with the speed that it was identified
> > as duplicate.
> > 
> > Should be entertaining!  ;-)
> 
> Right. ROTFL... volatile actually breaks atomic_t instead of making it 
> safe. x++ becomes a register load, increment and a register store. Without 
> volatile we can increment the memory directly. It seems that volatile 
> requires that the variable is loaded into a register first and then 
> operated upon. Understandable when you think about volatile being used to 
> access memory mapped I/O registers where a RMW operation could be 
> problematic.
> 
> See http://gcc.gnu.org/bugzilla/show_bug.cgi?id=3506

Yep.  The initial reaction was in fact to close my bug as a duplicate
of 3506.  But I was not asking for atomicity, but rather for smaller
code to be generated, so I reopened it.

							Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Linus Torvalds]

On Sat, 18 Aug 2007, Paul E. McKenney wrote:
> 
> One of the gcc guys claimed that he thought that the two-instruction
> sequence would be faster on some x86 machines.  I pointed out that
> there might be a concern about code size.  I chose not to point out
> that people might also care about the other x86 machines.  ;-)

Some (very few) x86 uarchs do tend to prefer "load-store" like code 
generation, and doing a "mov [mem],reg + op reg" instead of "op [mem]" can 
actually be faster on some of them. Not any that are relevant today, 
though.

Also, that has nothing to do with volatile, and should be controlled by 
optimization flags (like -mtune). In fact, I thought there was a separate 
flag to do that (ie something like "-mload-store"), but I can't find it, 
so maybe that's just my fevered brain..

			Linus

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Sat, Aug 18, 2007 at 03:41:13PM -0700, Linus Torvalds wrote:
> 
> 
> On Sat, 18 Aug 2007, Paul E. McKenney wrote:
> > 
> > One of the gcc guys claimed that he thought that the two-instruction
> > sequence would be faster on some x86 machines.  I pointed out that
> > there might be a concern about code size.  I chose not to point out
> > that people might also care about the other x86 machines.  ;-)
> 
> Some (very few) x86 uarchs do tend to prefer "load-store" like code 
> generation, and doing a "mov [mem],reg + op reg" instead of "op [mem]" can 
> actually be faster on some of them. Not any that are relevant today, 
> though.

;-)

> Also, that has nothing to do with volatile, and should be controlled by 
> optimization flags (like -mtune). In fact, I thought there was a separate 
> flag to do that (ie something like "-mload-store"), but I can't find it, 
> so maybe that's just my fevered brain..

Good point, will suggest this if the need arises.

							Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Chris Snook]

Linus Torvalds wrote:
> So the only reason to add back "volatile" to the atomic_read() sequence is 
> not to fix bugs, but to _hide_ the bugs better. They're still there, they 
> are just a lot harder to trigger, and tend to be a lot subtler.

What about barrier removal?  With consistent semantics we could optimize a fair 
amount of code.  Whether or not that constitutes "premature" optimization is 
open to debate, but there's no question we could reduce our register wiping in 
some places.

	-- Chris

Re: LDD3 pitfalls (was Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures)

[Chris Snook]

Stefan Richter wrote:
> Nick Piggin wrote:
>> Stefan Richter wrote:
>>> Nick Piggin wrote:
>>>
>>>> I don't know why people would assume volatile of atomics. AFAIK, most
>>>> of the documentation is pretty clear that all the atomic stuff can be
>>>> reordered etc. except for those that modify and return a value.
>>>
>>> Which documentation is there?
>> Documentation/atomic_ops.txt
>>
>>
>>> For driver authors, there is LDD3.  It doesn't specifically cover
>>> effects of optimization on accesses to atomic_t.
>>>
>>> For architecture port authors, there is Documentation/atomic_ops.txt.
>>> Driver authors also can learn something from that document, as it
>>> indirectly documents the atomic_t and bitops APIs.
>>>
>> "Semantics and Behavior of Atomic and Bitmask Operations" is
>> pretty direct :)
>>
>> Sure, it says that it's for arch maintainers, but there is no
>> reason why users can't make use of it.
> 
> 
> Note, LDD3 page 238 says:  "It is worth noting that most of the other
> kernel primitives dealing with synchronization, such as spinlock and
> atomic_t operations, also function as memory barriers."
> 
> I don't know about Linux 2.6.10 against which LDD3 was written, but
> currently only _some_ atomic_t operations function as memory barriers.
> 
> Besides, judging from some posts in this thread, saying that atomic_t
> operations dealt with synchronization may not be entirely precise.

atomic_t is often used as the basis for implementing more sophisticated 
synchronization mechanisms, such as rwlocks.  Whether or not they are designed 
for that purpose, the atomic_* operations are de facto synchronization primitives.

	-- Chris

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Chris Snook]

Christoph Lameter wrote:
> On Fri, 17 Aug 2007, Paul E. McKenney wrote:
> 
>> On Sat, Aug 18, 2007 at 08:09:13AM +0800, Herbert Xu wrote:
>>> On Fri, Aug 17, 2007 at 04:59:12PM -0700, Paul E. McKenney wrote:
>>>> gcc bugzilla bug #33102, for whatever that ends up being worth.  ;-)
>>> I had totally forgotten that I'd already filed that bug more
>>> than six years ago until they just closed yours as a duplicate
>>> of mine :)
>>>
>>> Good luck in getting it fixed!
>> Well, just got done re-opening it for the third time.  And a local
>> gcc community member advised me not to give up too easily.  But I
>> must admit that I am impressed with the speed that it was identified
>> as duplicate.
>>
>> Should be entertaining!  ;-)
> 
> Right. ROTFL... volatile actually breaks atomic_t instead of making it 
> safe. x++ becomes a register load, increment and a register store. Without 
> volatile we can increment the memory directly. It seems that volatile 
> requires that the variable is loaded into a register first and then 
> operated upon. Understandable when you think about volatile being used to 
> access memory mapped I/O registers where a RMW operation could be 
> problematic.

So, if we want consistent behavior, we're pretty much screwed unless we use 
inline assembler everywhere?

	-- Chris

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

On Mon, Aug 20, 2007 at 09:15:11AM -0400, Chris Snook wrote:
> Linus Torvalds wrote:
> >So the only reason to add back "volatile" to the atomic_read() sequence is 
> >not to fix bugs, but to _hide_ the bugs better. They're still there, they 
> >are just a lot harder to trigger, and tend to be a lot subtler.
> 
> What about barrier removal?  With consistent semantics we could optimize a 
> fair amount of code.  Whether or not that constitutes "premature" 
> optimization is open to debate, but there's no question we could reduce our 
> register wiping in some places.

If you've been reading all of Linus's emails you should be
thinking about adding memory barriers, and not removing
compiler barriers.

He's just told you that code of the kind

	while (!atomic_read(cond))
		;

	do_something()

probably needs a memory barrier (not just compiler) so that
do_something() doesn't see stale cache content that occured
before cond flipped.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Chris Snook]

Herbert Xu wrote:
> On Mon, Aug 20, 2007 at 09:15:11AM -0400, Chris Snook wrote:
>> Linus Torvalds wrote:
>>> So the only reason to add back "volatile" to the atomic_read() sequence is 
>>> not to fix bugs, but to _hide_ the bugs better. They're still there, they 
>>> are just a lot harder to trigger, and tend to be a lot subtler.
>> What about barrier removal?  With consistent semantics we could optimize a 
>> fair amount of code.  Whether or not that constitutes "premature" 
>> optimization is open to debate, but there's no question we could reduce our 
>> register wiping in some places.
> 
> If you've been reading all of Linus's emails you should be
> thinking about adding memory barriers, and not removing
> compiler barriers.
> 
> He's just told you that code of the kind
> 
> 	while (!atomic_read(cond))
> 		;
> 
> 	do_something()
> 
> probably needs a memory barrier (not just compiler) so that
> do_something() doesn't see stale cache content that occured
> before cond flipped.

Such code generally doesn't care precisely when it gets the update, just that 
the update is atomic, and it doesn't loop forever.  Regardless, I'm convinced we 
just need to do it all in assembly.

	-- Chris

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

>> Right. ROTFL... volatile actually breaks atomic_t instead of making 
>> it safe. x++ becomes a register load, increment and a register store. 
>> Without volatile we can increment the memory directly. It seems that 
>> volatile requires that the variable is loaded into a register first 
>> and then operated upon. Understandable when you think about volatile 
>> being used to access memory mapped I/O registers where a RMW 
>> operation could be problematic.
>
> So, if we want consistent behavior, we're pretty much screwed unless 
> we use inline assembler everywhere?

Nah, this whole argument is flawed -- "without volatile" we still
*cannot* "increment the memory directly".  On x86, you need a lock
prefix; on other archs, some other mechanism to make the memory
increment an *atomic* memory increment.

And no, RMW on MMIO isn't "problematic" at all, either.

An RMW op is a read op, a modify op, and a write op, all rolled
into one opcode.  But three actual operations.


The advantages of asm code for atomic_{read,set} are:
1) all the other atomic ops are implemented that way already;
2) you have full control over the asm insns selected, in particular,
    you can guarantee you *do* get an atomic op;
3) you don't need to use "volatile <data>" which generates
    not-all-that-good code on archs like x86, and we want to get rid
    of it anyway since it is problematic in many ways;
4) you don't need to use *(volatile <type>*)&<data>, which a) doesn't
    exist in C; b) isn't documented or supported in GCC; c) has a recent
    history of bugginess; d) _still uses volatile objects_; e) _still_
    is problematic in almost all those same ways as in 3);
5) you can mix atomic and non-atomic accesses to the atomic_t, which
    you cannot with the other alternatives.

The only disadvantage I know of is potentially slightly worse
instruction scheduling.  This is a generic asm() problem: GCC
cannot see what actual insns are inside the asm() block.


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

> Such code generally doesn't care precisely when it gets the update, 
> just that the update is atomic, and it doesn't loop forever.

Yes, it _does_ care that it gets the update _at all_, and preferably
as early as possible.

> Regardless, I'm convinced we just need to do it all in assembly.

So do you want "volatile asm" or "plain asm", for atomic_read()?
The asm version has two ways to go about it too...


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Russell King]

On Tue, Aug 21, 2007 at 12:04:17AM +0200, Segher Boessenkool wrote:
> And no, RMW on MMIO isn't "problematic" at all, either.
> 
> An RMW op is a read op, a modify op, and a write op, all rolled
> into one opcode.  But three actual operations.

Maybe for some CPUs, but not all.  ARM for instance can't use the
load exclusive and store exclusive instructions to MMIO space.

This means placing atomic_t or bitops into MMIO space is a definite
no-go on ARM.  It breaks.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

>> And no, RMW on MMIO isn't "problematic" at all, either.
>>
>> An RMW op is a read op, a modify op, and a write op, all rolled
>> into one opcode.  But three actual operations.
>
> Maybe for some CPUs, but not all.  ARM for instance can't use the
> load exclusive and store exclusive instructions to MMIO space.

Sure, your CPU doesn't have RMW instructions -- how to emulate
those if you don't have them is a totally different thing.


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Tue, Aug 21, 2007 at 01:02:01AM +0200, Segher Boessenkool wrote:
> >>And no, RMW on MMIO isn't "problematic" at all, either.
> >>
> >>An RMW op is a read op, a modify op, and a write op, all rolled
> >>into one opcode.  But three actual operations.
> >
> >Maybe for some CPUs, but not all.  ARM for instance can't use the
> >load exclusive and store exclusive instructions to MMIO space.
> 
> Sure, your CPU doesn't have RMW instructions -- how to emulate
> those if you don't have them is a totally different thing.

I thought that ARM's load exclusive and store exclusive instructions
were its equivalent of LL and SC, which RISC machines typically use to
build atomic sequences of instructions -- and which normally cannot be
applied to MMIO space.

						Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Linus Torvalds]

On Mon, 20 Aug 2007, Chris Snook wrote:
>
> What about barrier removal?  With consistent semantics we could optimize a
> fair amount of code.  Whether or not that constitutes "premature" optimization
> is open to debate, but there's no question we could reduce our register wiping
> in some places.

Why do people think that barriers are expensive? They really aren't. 
Especially the regular compiler barrier is basically zero cost. Any 
reasonable compiler will just flush the stuff it holds in registers that 
isn't already automatic local variables, and for regular kernel code, that 
tends to basically be nothing at all.

Ie a "barrier()" is likely _cheaper_ than the code generation downside 
from using "volatile".

		Linus

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[David Miller]

From: Linus Torvalds <torvalds@linux-foundation.org>
Date: Mon, 20 Aug 2007 22:46:47 -0700 (PDT)

> Ie a "barrier()" is likely _cheaper_ than the code generation downside 
> from using "volatile".

Assuming GCC were ever better about the code generation badness
with volatile that has been discussed here, I much prefer
we tell GCC "this memory piece changed" rather than "every
piece of memory has changed" which is what the barrier() does.

I happened to have been scanning a lot of assembler lately to
track down a gcc-4.2 miscompilation on sparc64, and the barriers
do hurt quite a bit in some places.  Instead of keeping unrelated
variables around cached in local registers, it reloads everything.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Russell King]

On Tue, Aug 21, 2007 at 01:02:01AM +0200, Segher Boessenkool wrote:
> >>And no, RMW on MMIO isn't "problematic" at all, either.
> >>
> >>An RMW op is a read op, a modify op, and a write op, all rolled
> >>into one opcode.  But three actual operations.
> >
> >Maybe for some CPUs, but not all.  ARM for instance can't use the
> >load exclusive and store exclusive instructions to MMIO space.
> 
> Sure, your CPU doesn't have RMW instructions -- how to emulate
> those if you don't have them is a totally different thing.

Let me say it more clearly: On ARM, it is impossible to perform atomic
operations on MMIO space.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Russell King]

On Mon, Aug 20, 2007 at 05:05:18PM -0700, Paul E. McKenney wrote:
> On Tue, Aug 21, 2007 at 01:02:01AM +0200, Segher Boessenkool wrote:
> > >>And no, RMW on MMIO isn't "problematic" at all, either.
> > >>
> > >>An RMW op is a read op, a modify op, and a write op, all rolled
> > >>into one opcode.  But three actual operations.
> > >
> > >Maybe for some CPUs, but not all.  ARM for instance can't use the
> > >load exclusive and store exclusive instructions to MMIO space.
> > 
> > Sure, your CPU doesn't have RMW instructions -- how to emulate
> > those if you don't have them is a totally different thing.
> 
> I thought that ARM's load exclusive and store exclusive instructions
> were its equivalent of LL and SC, which RISC machines typically use to
> build atomic sequences of instructions -- and which normally cannot be
> applied to MMIO space.

Absolutely correct.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul Mackerras]

Russell King writes:

> Let me say it more clearly: On ARM, it is impossible to perform atomic
> operations on MMIO space.

Actually, no one is suggesting that we try to do that at all.

The discussion about RMW ops on MMIO space started with a comment
attributed to the gcc developers that one reason why gcc on x86
doesn't use instructions that do RMW ops on volatile variables is that
volatile is used to mark MMIO addresses, and there was some
uncertainty about whether (non-atomic) RMW ops on x86 could be used on
MMIO.  This is in regard to the question about why gcc on x86 always
moves a volatile variable into a register before doing anything to it.

So the whole discussion is irrelevant to ARM, PowerPC and any other
architecture except x86[-64].

Paul.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Andi Kleen]

On Tue, Aug 21, 2007 at 07:33:49PM +1000, Paul Mackerras wrote:
> So the whole discussion is irrelevant to ARM, PowerPC and any other
> architecture except x86[-64].

It's even irrelevant on x86 because all modifying operations on atomic_t 
are coded in inline assembler and will always be RMW no matter
if atomic_t is volatile or not.

[ignoring atomic_set(x, atomic_read(x) + 1) which nobody should do]

The only issue is if atomic_t should have a implicit barrier or not.
My personal opinion is yes -- better safe than sorry. And any code
impact it may have is typically dwarved by the next cache miss anyways,
so it doesn't matter much.

-Andi

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Chris Snook]

David Miller wrote:
> From: Linus Torvalds <torvalds@linux-foundation.org>
> Date: Mon, 20 Aug 2007 22:46:47 -0700 (PDT)
> 
>> Ie a "barrier()" is likely _cheaper_ than the code generation downside 
>> from using "volatile".
> 
> Assuming GCC were ever better about the code generation badness
> with volatile that has been discussed here, I much prefer
> we tell GCC "this memory piece changed" rather than "every
> piece of memory has changed" which is what the barrier() does.
> 
> I happened to have been scanning a lot of assembler lately to
> track down a gcc-4.2 miscompilation on sparc64, and the barriers
> do hurt quite a bit in some places.  Instead of keeping unrelated
> variables around cached in local registers, it reloads everything.

Moore's law is definitely working against us here.  Register counts, 
pipeline depths, core counts, and clock multipliers are all increasing 
in the long run.  At some point in the future, barrier() will be 
universally regarded as a hammer too big for most purposes.  Whether or 
not removing it now constitutes premature optimization is arguable, but 
I think we should allow such optimization to happen (or not happen) in 
architecture-dependent code, and provide a consistent API that doesn't 
require the use of such things in arch-independent code where it might 
turn into a totally superfluous performance killer depending on what 
hardware it gets compiled for.

	-- Chris

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

>>>> And no, RMW on MMIO isn't "problematic" at all, either.
>>>>
>>>> An RMW op is a read op, a modify op, and a write op, all rolled
>>>> into one opcode.  But three actual operations.
>>>
>>> Maybe for some CPUs, but not all.  ARM for instance can't use the
>>> load exclusive and store exclusive instructions to MMIO space.
>>
>> Sure, your CPU doesn't have RMW instructions -- how to emulate
>> those if you don't have them is a totally different thing.
>
> Let me say it more clearly: On ARM, it is impossible to perform atomic
> operations on MMIO space.

It's all completely beside the point, see the other subthread, but...

Yeah, you can't do LL/SC to MMIO space; ARM isn't alone in that.
You could still implement atomic operations on MMIO space by taking
a lock elsewhere, in normal cacheable memory space.  Why you would
do this is a separate question, you probably don't want it :-)


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

>> Let me say it more clearly: On ARM, it is impossible to perform atomic
>> operations on MMIO space.
>
> Actually, no one is suggesting that we try to do that at all.
>
> The discussion about RMW ops on MMIO space started with a comment
> attributed to the gcc developers that one reason why gcc on x86
> doesn't use instructions that do RMW ops on volatile variables is that
> volatile is used to mark MMIO addresses, and there was some
> uncertainty about whether (non-atomic) RMW ops on x86 could be used on
> MMIO.  This is in regard to the question about why gcc on x86 always
> moves a volatile variable into a register before doing anything to it.

This question is GCC PR33102, which was incorrectly closed as a 
duplicate
of PR3506 -- and *that* PR was closed because its reporter seemed to
claim the GCC generated code for an increment on a volatile (namely, 
three
machine instructions: load, modify, store) was incorrect, and it has to
be one machine instruction.

> So the whole discussion is irrelevant to ARM, PowerPC and any other
> architecture except x86[-64].

And even there, it's not something the kernel can take advantage of
before GCC 4.4 is in widespread use, if then.  Let's move on.


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

> At some point in the future, barrier() will be universally regarded as 
> a hammer too big for most purposes.  Whether or not removing it now

You can't just remove it, it is needed in some places; you want to
replace it in most places with a more fine-grained "compiler barrier",
I presume?

> constitutes premature optimization is arguable, but I think we should 
> allow such optimization to happen (or not happen) in 
> architecture-dependent code, and provide a consistent API that doesn't 
> require the use of such things in arch-independent code where it might 
> turn into a totally superfluous performance killer depending on what 
> hardware it gets compiled for.

Explicit barrier()s won't be too hard to replace -- but what to do
about the implicit barrier()s in rmb() etc. etc. -- *those* will be
hard to get rid of, if only because it is hard enough to teach driver
authors about how to use those primitives *already*.  It is far from
clear what a good interface like that would look like, anyway.

Probably we should first start experimenting with a forget()-style
micro-barrier (but please, find a better name), and see if a nice
usage pattern shows up that can be turned into an API.


Segher

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Tue, Aug 21, 2007 at 04:48:51PM +0200, Segher Boessenkool wrote:
> >>Let me say it more clearly: On ARM, it is impossible to perform atomic
> >>operations on MMIO space.
> >
> >Actually, no one is suggesting that we try to do that at all.
> >
> >The discussion about RMW ops on MMIO space started with a comment
> >attributed to the gcc developers that one reason why gcc on x86
> >doesn't use instructions that do RMW ops on volatile variables is that
> >volatile is used to mark MMIO addresses, and there was some
> >uncertainty about whether (non-atomic) RMW ops on x86 could be used on
> >MMIO.  This is in regard to the question about why gcc on x86 always
> >moves a volatile variable into a register before doing anything to it.
> 
> This question is GCC PR33102, which was incorrectly closed as a 
> duplicate
> of PR3506 -- and *that* PR was closed because its reporter seemed to
> claim the GCC generated code for an increment on a volatile (namely, 
> three
> machine instructions: load, modify, store) was incorrect, and it has to
> be one machine instruction.
> 
> >So the whole discussion is irrelevant to ARM, PowerPC and any other
> >architecture except x86[-64].
> 
> And even there, it's not something the kernel can take advantage of
> before GCC 4.4 is in widespread use, if then.  Let's move on.

I agree that instant gratification is hard to come by when synching
up compiler and kernel versions.  Nonetheless, it should be possible
to create APIs that are are conditioned on the compiler version.

						Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Satyam Sharma]

On Tue, 21 Aug 2007, Chris Snook wrote:

> David Miller wrote:
> > From: Linus Torvalds <torvalds@linux-foundation.org>
> > Date: Mon, 20 Aug 2007 22:46:47 -0700 (PDT)
> > 
> > > Ie a "barrier()" is likely _cheaper_ than the code generation downside
> > > from using "volatile".
> > 
> > Assuming GCC were ever better about the code generation badness
> > with volatile that has been discussed here, I much prefer
> > we tell GCC "this memory piece changed" rather than "every
> > piece of memory has changed" which is what the barrier() does.
> > 
> > I happened to have been scanning a lot of assembler lately to
> > track down a gcc-4.2 miscompilation on sparc64, and the barriers
> > do hurt quite a bit in some places.  Instead of keeping unrelated
> > variables around cached in local registers, it reloads everything.
> 
> Moore's law is definitely working against us here.  Register counts, pipeline
> depths, core counts, and clock multipliers are all increasing in the long run.
> At some point in the future, barrier() will be universally regarded as a
> hammer too big for most purposes.

I do agree, and the important point to note is that the benefits of a
/lighter/ compiler barrier, such as what David referred to above, _can_
be had without having to do anything with the "volatile" keyword at all.
And such a primitive has already been mentioned/proposed on this thread.


But this is all tangential to the core question at hand -- whether to have
implicit (compiler, possibly "light-weight" of the kind referred above)
barrier semantics in atomic ops that do not have them, or not.

I was lately looking in the kernel for _actual_ code that uses atomic_t
and benefits from the lack of any implicit barrier, with the compiler
being free to cache the atomic_t in a register. Now that often does _not_
happen, because all other ops (implemented in asm with LOCK prefix on x86)
_must_ therefore constrain the atomic_t to memory anyway. So typically all
atomic ops code sequences end up operating on memory.

Then I did locate sched.c:select_nohz_load_balancer() -- it repeatedly
references the same atomic_t object, and the code that I saw generated
(with CC_OPTIMIZE_FOR_SIZE=y) did cache it in a register for a sequence of
instructions. It uses atomic_cmpxchg, thereby not requiring explicit
memory barriers anywhere in the code, and is an example of an atomic_t
user that is safe, and yet benefits from its memory loads/stores being
elided/coalesced by the compiler.


# at this point, %%eax holds num_online_cpus() and
# %%ebx holds cpus_weight(nohz.cpu_mask)
# the variable "cpu" is in %esi

0xc1018e1d:      cmp    %eax,%ebx		# if No.A.
0xc1018e1f:      mov    0xc134d900,%eax		# first atomic_read()
0xc1018e24:      jne    0xc1018e36
0xc1018e26:      cmp    %esi,%eax		# if No.B.
0xc1018e28:      jne    0xc1018e80		# returns with 0
0xc1018e2a:      movl   $0xffffffff,0xc134d900	# atomic_set(-1), and ...
0xc1018e34:      jmp    0xc1018e80		# ... returns with 0
0xc1018e36:      cmp    $0xffffffff,%eax	# if No.C. (NOTE!)
0xc1018e39:      jne    0xc1018e46
0xc1018e3b:      lock cmpxchg %esi,0xc134d900	# atomic_cmpxchg()
0xc1018e43:      inc    %eax
0xc1018e44:      jmp    0xc1018e48
0xc1018e46:      cmp    %esi,%eax		# if No.D. (NOTE!)
0xc1018e48:      jne    0xc1018e80		# if !=, default return 0 (if No.E.)
0xc1018e4a:      jmp    0xc1018e84		# otherwise (==) returns with 1

The above is:

	if (cpus_weight(nohz.cpu_mask) == num_online_cpus()) {	/* if No.A. */
		if (atomic_read(&nohz.load_balancer) == cpu)	/* if No.B. */
			atomic_set(&nohz.load_balancer, -1);	/* XXX */
		return 0;
	}
	if (atomic_read(&nohz.load_balancer) == -1) {		/* if No.C. */
		/* make me the ilb owner */
		if (atomic_cmpxchg(&nohz.load_balancer, -1, cpu) == -1)	/* if No.E. */
			return 1;
	} else if (atomic_read(&nohz.load_balancer) == cpu)	/* if No.D. */
		return 1;
	...
	...
	return 0; /* default return from function */

As you can see, the atomic_read()'s of "if"s Nos. B, C, and D, were _all_
coalesced into a single memory reference "mov    0xc134d900,%eax" at the
top of the function, and then "if"s Nos. C and D simply used the value
from %%eax itself. But that's perfectly safe, such is the logic of this
function. It uses cmpxchg _whenever_ updating the value in the memory
atomic_t and then returns appropriately. The _only_ point that a casual
reader may find racy is that marked /* XXX */ above -- atomic_read()
followed by atomic_set() with no barrier in between. But even that is ok,
because if one thread ever finds that condition to succeed, it is 100%
guaranteed no other thread on any other CPU will find _any_ condition
to be true, thereby avoiding any race in the modification of that value.


BTW it does sound reasonable that a lot of atomic_t users that want a
compiler barrier probably also want a memory barrier. Do we make _that_
implicit too? Quite clearly, making _either_ one of those implicit in
atomic_{read,set} (in any form of implementation -- a forget() macro
based, *(volatile int *)& based, or inline asm based) would end up
harming code such as that cited above.

Lastly, the most obvious reason that should be considered against implicit
barriers in atomic ops is that it isn't "required" -- atomicity does not
imply any barrier after all, and making such a distinction would actually
be a healthy separation that helps people think more clearly when writing
lockless code.

[ But the "authors' expectations" / heisenbugs argument also holds some
  water ... for that, we can have a _variant_ in the API for atomic ops
  that has implicit compiler/memory barriers, to make it easier on those
  who want that behaviour. But let us not penalize code that knows what
  it is doing by changing the default to that, please. ]


Satyam

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Linus Torvalds]

On Tue, 21 Aug 2007, Chris Snook wrote:
> 
> Moore's law is definitely working against us here.  Register counts, pipeline
> depths, core counts, and clock multipliers are all increasing in the long run.
> At some point in the future, barrier() will be universally regarded as a
> hammer too big for most purposes.

Note that "barrier()" is purely a compiler barrier. It has zero impact on 
the CPU pipeline itself, and also has zero impact on anything that gcc 
knows isn't visible in memory (ie local variables that don't have their 
address taken), so barrier() really is pretty cheap.

Now, it's possible that gcc messes up in some circumstances, and that the 
memory clobber will cause gcc to also do things like flush local registers 
unnecessarily to their stack slots, but quite frankly, if that happens, 
it's a gcc problem, and I also have to say that I've not seen that myself.

So in a very real sense, "barrier()" will just make sure that there is a 
stronger sequence point for the compiler where things are stable. In most 
cases it has absolutely zero performance impact - apart from the 
-intended- impact of making sure that the compiler doesn't re-order or 
cache stuff around it.

And sure, we could make it more finegrained, and also introduce a 
per-variable barrier, but the fact is, people _already_ have problems with 
thinking about these kinds of things, and adding new abstraction issues 
with subtle semantics is the last thing we want.

So I really think you'd want to show a real example of real code that 
actually gets noticeably slower or bigger.

In removing "volatile", we have shown that. It may not have made a big 
difference on powerpc, but it makes a real difference on x86 - and more 
importantly, it removes something that people clearly don't know how it 
works, and incorrectly expect to just fix bugs.

[ There are *other* barriers - the ones that actually add memory barriers 
  to the CPU - that really can be quite expensive. The good news is that 
  the expense is going down rather than up: both Intel and AMD are not 
  only removing the need for some of them (ie "smp_rmb()" will become a 
  compiler-only barrier), but we're _also_ seeing the whole "pipeline 
  flush" approach go away, and be replaced by the CPU itself actually 
  being better - so even the actual CPU pipeline barriers are getting
  cheaper, not more expensive. ]

For example, did anybody even _test_ how expensive "barrier()" is? Just 
as a lark, I did

	#undef barrier
	#define barrier() do { } while (0)

in kernel/sched.c (which only has three of them in it, but hey, that's 
more than most files), and there were _zero_ code generation downsides. 
One instruction was moved (and a few line numbers changed), so it wasn't 
like the assembly language was identical, but the point is, barrier() 
simply doesn't have the same kinds of downsides that "volatile" has.

(That may not be true on other architectures or in other source files, of 
course. This *does* depend on code generation details. But anybody who 
thinks that "barrier()" is fundamentally expensive is simply incorrect. It 
is *fundamnetally* a no-op).

		Linus

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 557 bytes --]

On Tue, 21 Aug 2007 09:16:43 PDT, "Paul E. McKenney" said:

> I agree that instant gratification is hard to come by when synching
> up compiler and kernel versions.  Nonetheless, it should be possible
> to create APIs that are are conditioned on the compiler version.

We've tried that, sort of.  See the mess surrounding the whole
extern/static/inline/__whatever boondogle, which seems to have
changed semantics in every single gcc release since 2.95 or so.

And recently mention was made that gcc4.4 will have *new* semantics
in this area. Yee. Hah.






[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Tue, Aug 21, 2007 at 06:51:16PM -0400, Valdis.Kletnieks@vt.edu wrote:
> On Tue, 21 Aug 2007 09:16:43 PDT, "Paul E. McKenney" said:
> 
> > I agree that instant gratification is hard to come by when synching
> > up compiler and kernel versions.  Nonetheless, it should be possible
> > to create APIs that are are conditioned on the compiler version.
> 
> We've tried that, sort of.  See the mess surrounding the whole
> extern/static/inline/__whatever boondogle, which seems to have
> changed semantics in every single gcc release since 2.95 or so.
> 
> And recently mention was made that gcc4.4 will have *new* semantics
> in this area. Yee. Hah.

;-)

						Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Adrian Bunk]

On Tue, Aug 21, 2007 at 06:51:16PM -0400, Valdis.Kletnieks@vt.edu wrote:
> On Tue, 21 Aug 2007 09:16:43 PDT, "Paul E. McKenney" said:
> 
> > I agree that instant gratification is hard to come by when synching
> > up compiler and kernel versions.  Nonetheless, it should be possible
> > to create APIs that are are conditioned on the compiler version.
> 
> We've tried that, sort of.  See the mess surrounding the whole
> extern/static/inline/__whatever boondogle, which seems to have
> changed semantics in every single gcc release since 2.95 or so.
>...

There is exactly one semantics change in gcc in this area, and that is 
the change of the "extern inline" semantics in gcc 4.3 to the
C99 semantics.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

(unknown)

[Eugene Teo]

subscribe netdev

Re: [PATCH] i386: Fix a couple busy loops in mach_wakecpu.h:wait_for_init_deassert()

[Denys Vlasenko]

On Thursday 16 August 2007 01:39, Satyam Sharma wrote:
>
>  static inline void wait_for_init_deassert(atomic_t *deassert)
>  {
> -	while (!atomic_read(deassert));
> +	while (!atomic_read(deassert))
> +		cpu_relax();
>  	return;
>  }

For less-than-briliant people like me, it's totally non-obvious that
cpu_relax() is needed for correctness here, not just to make P4 happy.

IOW: "atomic_read" name quite unambiguously means "I will read
this variable from main memory". Which is not true and creates
potential for confusion and bugs.
--
vda

Re: [PATCH] i386: Fix a couple busy loops in mach_wakecpu.h:wait_for_init_deassert()

[Andi Kleen]

On Friday 24 August 2007 13:59:32 Denys Vlasenko wrote:
> On Thursday 16 August 2007 01:39, Satyam Sharma wrote:
> >
> >  static inline void wait_for_init_deassert(atomic_t *deassert)
> >  {
> > -	while (!atomic_read(deassert));
> > +	while (!atomic_read(deassert))
> > +		cpu_relax();
> >  	return;
> >  }
> 
> For less-than-briliant people like me, it's totally non-obvious that
> cpu_relax() is needed for correctness here, not just to make P4 happy.

I find it also non obvious. It would be really better to have a barrier
or equivalent (volatile or variable clobber) in the atomic_read()
 
> IOW: "atomic_read" name quite unambiguously means "I will read
> this variable from main memory". Which is not true and creates
> potential for confusion and bugs.

Agreed.

-Andi

RE: [PATCH] i386: Fix a couple busy loops in mach_wakecpu.h:wait_for_init_deassert()

[Kenn Humborg]

> On Thursday 16 August 2007 01:39, Satyam Sharma wrote:
> >
> >  static inline void wait_for_init_deassert(atomic_t *deassert)
> >  {
> > -	while (!atomic_read(deassert));
> > +	while (!atomic_read(deassert))
> > +		cpu_relax();
> >  	return;
> >  }
> 
> For less-than-briliant people like me, it's totally non-obvious that
> cpu_relax() is needed for correctness here, not just to make P4 happy.
> 
> IOW: "atomic_read" name quite unambiguously means "I will read
> this variable from main memory". Which is not true and creates
> potential for confusion and bugs.

To me, "atomic_read" means a read which is synchronized with other 
changes to the variable (using the atomic_XXX functions) in such 
a way that I will always only see the "before" or "after"
state of the variable - never an intermediate state while a 
modification is happening.  It doesn't imply that I have to 
see the "after" state immediately after another thread modifies
it.

Perhaps the Linux atomic_XXX functions work like that, or used
to work like that, but it's counter-intuitive to me that "atomic"
should imply a memory read.

Later,
Kenn

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Denys Vlasenko]

On Saturday 18 August 2007 05:13, Linus Torvalds wrote:
> On Sat, 18 Aug 2007, Satyam Sharma wrote:
> > No code does (or would do, or should do):
> >
> > 	x.counter++;
> >
> > on an "atomic_t x;" anyway.
>
> That's just an example of a general problem.
>
> No, you don't use "x.counter++". But you *do* use
>
> 	if (atomic_read(&x) <= 1)
>
> and loading into a register is stupid and pointless, when you could just
> do it as a regular memory-operand to the cmp instruction.

It doesn't mean that (volatile int*) cast is bad, it means that current gcc
is bad (or "not good enough"). IOW: instead of avoiding volatile cast,
it's better to fix the compiler.

> And as far as the compiler is concerned, the problem is the 100% same:
> combining operations with the volatile memop.
>
> The fact is, a compiler that thinks that
>
> 	movl mem,reg
> 	cmpl $val,reg
>
> is any better than
>
> 	cmpl $val,mem
>
> is just not a very good compiler.

Linus, in all honesty gcc has many more cases of suboptimal code,
case of "volatile" is just one of many.

Off the top of my head:

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=28417

unsigned v;
void f(unsigned A) { v = ((unsigned long long)A) * 365384439 >> (27+32); }

gcc-4.1.1 -S -Os -fomit-frame-pointer t.c

f:
        movl    $365384439, %eax
        mull    4(%esp)
        movl    %edx, %eax <===== ?
        shrl    $27, %eax
        movl    %eax, v
        ret

Why is it moving %edx to %eax?

gcc-4.2.1 -S -Os -fomit-frame-pointer t.c

f:
        movl    $365384439, %eax
        mull    4(%esp)
        movl    %edx, %eax <===== ?
        xorl    %edx, %edx <===== ??!
        shrl    $27, %eax
        movl    %eax, v
        ret

Progress... Now we also zero out %edx afterwards for no apparent reason.
--
vda

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Denys Vlasenko]

On Thursday 16 August 2007 00:22, Paul Mackerras wrote:
> Satyam Sharma writes:
> In the kernel we use atomic variables in precisely those situations
> where a variable is potentially accessed concurrently by multiple
> CPUs, and where each CPU needs to see updates done by other CPUs in a
> timely fashion.  That is what they are for.  Therefore the compiler
> must not cache values of atomic variables in registers; each
> atomic_read must result in a load and each atomic_set must result in a
> store.  Anything else will just lead to subtle bugs.

Amen.
--
vda

Re: [PATCH] i386: Fix a couple busy loops in mach_wakecpu.h:wait_for_init_deassert()

[Satyam Sharma]

Hi Denys,


On Fri, 24 Aug 2007, Denys Vlasenko wrote:

> On Thursday 16 August 2007 01:39, Satyam Sharma wrote:
> >
> >  static inline void wait_for_init_deassert(atomic_t *deassert)
> >  {
> > -	while (!atomic_read(deassert));
> > +	while (!atomic_read(deassert))
> > +		cpu_relax();
> >  	return;
> >  }
> 
> For less-than-briliant people like me, it's totally non-obvious that
> cpu_relax() is needed for correctness here, not just to make P4 happy.

This thread has been round-and-round with exactly the same discussions
:-) I had proposed few such variants to make a compiler barrier implicit
in atomic_{read,set} myself, but frankly, at least personally speaking
(now that I know better), I'm not so much in favour of implicit barriers
(compiler, memory or both) in atomic_{read,set}.

This might sound like an about-turn if you read my own postings to Nick
Piggin from a week back, but I do agree with most his opinions on the
matter now -- separation of barriers from atomic ops is actually good,
beneficial to certain code that knows what it's doing, explicit usage
of barriers stands out more clearly (most people here who deal with it
do know cpu_relax() is an explicit compiler barrier) compared to an
implicit usage in an atomic_read() or such variant ...


> IOW: "atomic_read" name quite unambiguously means "I will read
> this variable from main memory". Which is not true and creates
> potential for confusion and bugs.

I'd have to disagree here -- atomic ops are all about _atomicity_ of
memory accesses, not _making_ them happen (or visible to other CPUs)
_then and there_ itself. The latter are the job of barriers.

The behaviour (and expectations) are quite comprehensively covered in
atomic_ops.txt -- let alone atomic_{read,set}, even atomic_{inc,dec}
are permitted by archs' implementations to _not_ have any memory
barriers, for that matter. [It is unrelated that on x86 making them
SMP-safe requires the use of the LOCK prefix that also happens to be
an implicit memory barrier.]

An argument was also made about consistency of atomic_{read,set} w.r.t.
the other atomic ops -- but clearly, they are all already consistent!
All of them are atomic :-) The fact that atomic_{read,set} do _not_
require any inline asm or LOCK prefix whereas the others do, has to do
with the fact that unlike all others, atomic_{read,set} are not RMW ops
and hence guaranteed to be atomic just as they are in plain & simple C.

But if people do seem to have a mixed / confused notion of atomicity
and barriers, and if there's consensus, then as I'd said earlier, I
have no issues in going with the consensus (eg. having API variants).
Linus would be more difficult to convince, however, I suspect :-)


Satyam

Re: [PATCH] i386: Fix a couple busy loops in mach_wakecpu.h:wait_for_init_deassert()

[Denys Vlasenko]

On Friday 24 August 2007 13:12, Kenn Humborg wrote:
> > On Thursday 16 August 2007 01:39, Satyam Sharma wrote:
> > >  static inline void wait_for_init_deassert(atomic_t *deassert)
> > >  {
> > > -	while (!atomic_read(deassert));
> > > +	while (!atomic_read(deassert))
> > > +		cpu_relax();
> > >  	return;
> > >  }
> >
> > For less-than-briliant people like me, it's totally non-obvious that
> > cpu_relax() is needed for correctness here, not just to make P4 happy.
> >
> > IOW: "atomic_read" name quite unambiguously means "I will read
> > this variable from main memory". Which is not true and creates
> > potential for confusion and bugs.
>
> To me, "atomic_read" means a read which is synchronized with other
> changes to the variable (using the atomic_XXX functions) in such
> a way that I will always only see the "before" or "after"
> state of the variable - never an intermediate state while a
> modification is happening.  It doesn't imply that I have to
> see the "after" state immediately after another thread modifies
> it.

So you are ok with compiler propagating n1 to n2 here:

n1 += atomic_read(x);
other_variable++;
n2 += atomic_read(x);

without accessing x second time. What's the point? Any sane coder
will say that explicitly anyway:

tmp = atomic_read(x);
n1 += tmp;
other_variable++;
n2 += tmp;

if only for the sake of code readability. Because first code
is definitely hinting that it reads RAM twice, and it's actively *bad*
for code readability when in fact it's not the case!

Locking, compiler and CPU barriers are complicated enough already,
please don't make them even harder to understand.
--
vda

RE: [PATCH] i386: Fix a couple busy loops in mach_wakecpu.h:wait_for_init_deassert()

[Luck, Tony]

>>  static inline void wait_for_init_deassert(atomic_t *deassert)
>>  {
>> -	while (!atomic_read(deassert));
>> +	while (!atomic_read(deassert))
>> +		cpu_relax();
>>  	return;
>>  }
>
> For less-than-briliant people like me, it's totally non-obvious that
> cpu_relax() is needed for correctness here, not just to make P4 happy.

Not just P4 ... there are other threaded cpus where it is useful to
let the core know that this is a busy loop so it would be a good thing
to let other threads have priority.

Even on a non-threaded cpu the cpu_relax() could be useful in the
future to hint to the cpu that it could drop into a lower power
hogging state.

But I agree with your main point that the loop without the cpu_relax()
looks like it ought to work because atomic_read() ought to actually
go out and read memory each time around the loop.

-Tony

Re: [PATCH] i386: Fix a couple busy loops in mach_wakecpu.h:wait_for_init_deassert()

[Christoph Lameter]

On Fri, 24 Aug 2007, Satyam Sharma wrote:

> But if people do seem to have a mixed / confused notion of atomicity
> and barriers, and if there's consensus, then as I'd said earlier, I
> have no issues in going with the consensus (eg. having API variants).
> Linus would be more difficult to convince, however, I suspect :-)

The confusion may be the result of us having barrier semantics in 
atomic_read. If we take that out then we may avoid future confusions.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Christoph Lameter]

On Fri, 24 Aug 2007, Denys Vlasenko wrote:

> On Thursday 16 August 2007 00:22, Paul Mackerras wrote:
> > Satyam Sharma writes:
> > In the kernel we use atomic variables in precisely those situations
> > where a variable is potentially accessed concurrently by multiple
> > CPUs, and where each CPU needs to see updates done by other CPUs in a
> > timely fashion.  That is what they are for.  Therefore the compiler
> > must not cache values of atomic variables in registers; each
> > atomic_read must result in a load and each atomic_set must result in a
> > store.  Anything else will just lead to subtle bugs.
> 
> Amen.

A "timely" fashion? One cannot rely on something like that when coding. 
The visibility of updates is insured by barriers and not by some fuzzy 
notion of "timeliness".

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Linus Torvalds]

On Fri, 24 Aug 2007, Denys Vlasenko wrote:
>
> > No, you don't use "x.counter++". But you *do* use
> >
> > 	if (atomic_read(&x) <= 1)
> >
> > and loading into a register is stupid and pointless, when you could just
> > do it as a regular memory-operand to the cmp instruction.
> 
> It doesn't mean that (volatile int*) cast is bad, it means that current gcc
> is bad (or "not good enough"). IOW: instead of avoiding volatile cast,
> it's better to fix the compiler.

I would agree that fixing the compiler in this case would be a good thing, 
even quite regardless of any "atomic_read()" discussion.

I just have a strong suspicion that "volatile" performance is so low down 
the list of any C compiler persons interest, that it's never going to 
happen. And quite frankly, I cannot blame the gcc guys for it.

That's especially as "volatile" really isn't a very good feature of the C 
language, and is likely to get *less* interesting rather than more (as 
user space starts to be more and more threaded, "volatile" gets less and 
less useful.

[ Ie, currently, I think you can validly use "volatile" in a "sigatomic_t" 
  kind of way, where there is a single thread, but with asynchronous 
  events. In that kind of situation, I think it's probably useful. But 
  once you get multiple threads, it gets pointless.

  Sure: you could use "volatile" together with something like Dekker's or 
  Peterson's algorithm that doesn't depend on cache coherency (that's 
  basically what the C "volatile" keyword approximates: not atomic 
  accesses, but *uncached* accesses! But let's face it, that's way past 
  insane. ]

So I wouldn't expect "volatile" to ever really generate better code. It 
might happen as a side effect of other improvements (eg, I might hope that 
the SSA work would eventually lead to gcc having a much better defined 
model of valid optimizations, and maybe better code generation for 
volatile accesses fall out cleanly out of that), but in the end, it's such 
an ugly special case in C, and so seldom used, that I wouldn't depend on 
it.

> Linus, in all honesty gcc has many more cases of suboptimal code,
> case of "volatile" is just one of many.

Well, the thing is, quite often, many of those "suboptimal code" 
generations fall into two distinct classes:

 - complex C code. I can't really blame the compiler too much for this. 
   Some things are *hard* to optimize, and for various scalability 
   reasons, you often end up having limits in the compiler where it 
   doesn't even _try_ doing certain optimizations if you have excessive 
   complexity.

 - bad register allocation. Register allocation really is hard, and 
   sometimes gcc just does the "obviously wrong" thing, and you end up 
   having totally unnecessary spills.

> Off the top of my head:

Yes, "unsigned long long" with x86 has always generated atrocious code. In 
fact, I would say that historically it was really *really* bad. These 
days, gcc actually does a pretty good job, but I'm not surprised that it's 
still quite possible to find cases where it did some optimization (in this 
case, apparently noticing that "shift by >= 32 bits" causes the low 
register to be pointless) and then missed *another* optimization (better 
register use) because that optimization had been done *before* the first 
optimization was done.

That's a *classic* example of compiler code generation issues, and quite 
frankly, I think that's very different from the issue of "volatile".

Quite frankly, I'd like there to be more competition in the open source 
compiler game, and that might cause some upheavals, but on the whole, gcc 
actually does a pretty damn good job. 

			Linus

Re: [PATCH] i386: Fix a couple busy loops in mach_wakecpu.h:wait_for_init_deassert()

[Linus Torvalds]

On Fri, 24 Aug 2007, Denys Vlasenko wrote:
> 
> So you are ok with compiler propagating n1 to n2 here:
> 
> n1 += atomic_read(x);
> other_variable++;
> n2 += atomic_read(x);
> 
> without accessing x second time. What's the point? Any sane coder
> will say that explicitly anyway:

No.

This is a common mistake, and it's total crap.

Any "sane coder" will often use inline functions, macros, etc helpers to 
do certain abstract things. Those things may contain "atomic_read()" 
calls.

The biggest reason for compilers doing CSE is exactly the fact that many 
opportunities for CSE simple *are*not*visible* on a source code level. 

That is true of things like atomic_read() equally as to things like shared 
offsets inside structure member accesses. No difference what-so-ever.

Yes, we have, traditionally, tried to make it *easy* for the compiler to 
generate good code. So when we can, and when we look at performance for 
some really hot path, we *will* write the source code so that the compiler 
doesn't even have the option to screw it up, and that includes things like 
doing CSE at a source code level so that we don't see the compiler 
re-doing accesses unnecessarily.

And I'm not saying we shouldn't do that. But "performance" is not an 
either-or kind of situation, and we should:

 - spend the time at a source code level: make it reasonably easy for the 
   compiler to generate good code, and use the right algorithms at a 
   higher level (and order structures etc so that they have good cache 
   behaviour).

 - .. *and* expect the compiler to handle the cases we didn't do by hand
   pretty well anyway. In particular, quite often, abstraction levels at a 
   software level means that we give compilers "stupid" code, because some 
   function may have a certain high-level abstraction rule, but then on a 
   particular architecture it's actually a no-op, and the compiler should 
   get to "untangle" our stupid code and generate good end results.

 - .. *and* expect the hardware to be sane and do a good job even when the 
   compiler didn't generate perfect code or there were unlucky cache miss
   patterns etc.

and if we do all of that, we'll get good performance. But you really do 
want all three levels. It's not enough to be good at any one level (or 
even any two).

			Linus

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Denys Vlasenko]

On Friday 24 August 2007 18:15, Christoph Lameter wrote:
> On Fri, 24 Aug 2007, Denys Vlasenko wrote:
> > On Thursday 16 August 2007 00:22, Paul Mackerras wrote:
> > > Satyam Sharma writes:
> > > In the kernel we use atomic variables in precisely those situations
> > > where a variable is potentially accessed concurrently by multiple
> > > CPUs, and where each CPU needs to see updates done by other CPUs in a
> > > timely fashion.  That is what they are for.  Therefore the compiler
> > > must not cache values of atomic variables in registers; each
> > > atomic_read must result in a load and each atomic_set must result in a
> > > store.  Anything else will just lead to subtle bugs.
> >
> > Amen.
>
> A "timely" fashion? One cannot rely on something like that when coding.
> The visibility of updates is insured by barriers and not by some fuzzy
> notion of "timeliness".

But here you do have some notion of time:

	while (atomic_read(&x))
		continue;

"continue when other CPU(s) decrement it down to zero".
If "read" includes an insn which accesses RAM, you will
see "new" value sometime after other CPU decrements it.
"Sometime after" is on the order of nanoseconds here.
It is a valid concept of time, right?

The whole confusion is about whether atomic_read implies
"read from RAM" or not. I am in a camp which thinks it does.
You are in an opposite one.

We just need a less ambiguous name.

What about this:

/**
 * atomic_read - read atomic variable
 * @v: pointer of type atomic_t
 *
 * Atomically reads the value of @v.
 * No compiler barrier implied.
 */
#define atomic_read(v)          ((v)->counter)

+/**
+ * atomic_read_uncached - read atomic variable from memory
+ * @v: pointer of type atomic_t
+ *
+ * Atomically reads the value of @v. This is guaranteed to emit an insn
+ * which accesses memory, atomically. No ordering guarantees!
+ */
+#define atomic_read_uncached(v)  asm_or_volatile_ptr_magic(v)

I was thinking of s/atomic_read/atomic_get/ too, but it implies "taking"
atomic a-la get_cpu()...
--
vda

Re: [PATCH] i386: Fix a couple busy loops in mach_wakecpu.h:wait_for_init_deassert()

[Denys Vlasenko]

On Friday 24 August 2007 18:06, Christoph Lameter wrote:
> On Fri, 24 Aug 2007, Satyam Sharma wrote:
> > But if people do seem to have a mixed / confused notion of atomicity
> > and barriers, and if there's consensus, then as I'd said earlier, I
> > have no issues in going with the consensus (eg. having API variants).
> > Linus would be more difficult to convince, however, I suspect :-)
>
> The confusion may be the result of us having barrier semantics in
> atomic_read. If we take that out then we may avoid future confusions.

I think better name may help. Nuke atomic_read() altogether.

n = atomic_value(x);	// doesnt hint as strongly at reading as "atomic_read"
n = atomic_fetch(x);	// yes, we _do_ touch RAM
n = atomic_read_uncached(x); // or this

How does that sound?
--
vda

Re: [PATCH] i386: Fix a couple busy loops in mach_wakecpu.h:wait_for_init_deassert()

[Chris Snook]

Denys Vlasenko wrote:
> On Friday 24 August 2007 18:06, Christoph Lameter wrote:
>> On Fri, 24 Aug 2007, Satyam Sharma wrote:
>>> But if people do seem to have a mixed / confused notion of atomicity
>>> and barriers, and if there's consensus, then as I'd said earlier, I
>>> have no issues in going with the consensus (eg. having API variants).
>>> Linus would be more difficult to convince, however, I suspect :-)
>> The confusion may be the result of us having barrier semantics in
>> atomic_read. If we take that out then we may avoid future confusions.
> 
> I think better name may help. Nuke atomic_read() altogether.
> 
> n = atomic_value(x);	// doesnt hint as strongly at reading as "atomic_read"
> n = atomic_fetch(x);	// yes, we _do_ touch RAM
> n = atomic_read_uncached(x); // or this
> 
> How does that sound?

atomic_value() vs. atomic_fetch() should be rather unambiguous. 
atomic_read_uncached() begs the question of precisely which cache we are 
avoiding, and could itself cause confusion.

So, if I were writing atomic.h from scratch, knowing what I know now, I think 
I'd use atomic_value() and atomic_fetch().  The problem is that there are a lot 
of existing users of atomic_read(), and we can't write a script to correctly 
guess their intent.  I'm not sure auditing all uses of atomic_read() is really 
worth the comparatively miniscule benefits.

We could play it safe and convert them all to atomic_fetch(), or we could 
acknowledge that changing the semantics 8 months ago was not at all disastrous, 
and make them all atomic_value(), allowing people to use atomic_fetch() where 
they really care.

	-- Chris

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Denys Vlasenko]

On Friday 17 August 2007 17:48, Linus Torvalds wrote:
> 
> On Fri, 17 Aug 2007, Nick Piggin wrote:
> > 
> > That's not obviously just taste to me. Not when the primitive has many
> > (perhaps, the majority) of uses that do not require said barriers. And
> > this is not solely about the code generation (which, as Paul says, is
> > relatively minor even on x86). I prefer people to think explicitly
> > about barriers in their lockless code.
> 
> Indeed.
> 
> I think the important issues are:
> 
>  - "volatile" itself is simply a badly/weakly defined issue. The semantics 
>    of it as far as the compiler is concerned are really not very good, and 
>    in practice tends to boil down to "I will generate so bad code that 
>    nobody can accuse me of optimizing anything away".
> 
>  - "volatile" - regardless of how well or badly defined it is - is purely 
>    a compiler thing. It has absolutely no meaning for the CPU itself, so 
>    it at no point implies any CPU barriers. As a result, even if the 
>    compiler generates crap code and doesn't re-order anything, there's 
>    nothing that says what the CPU will do.
> 
>  - in other words, the *only* possible meaning for "volatile" is a purely 
>    single-CPU meaning. And if you only have a single CPU involved in the 
>    process, the "volatile" is by definition pointless (because even 
>    without a volatile, the compiler is required to make the C code appear 
>    consistent as far as a single CPU is concerned).
> 
> So, let's take the example *buggy* code where we use "volatile" to wait 
> for other CPU's:
> 
> 	atomic_set(&var, 0);
> 	while (!atomic_read(&var))
> 		/* nothing */;
> 
> 
> which generates an endless loop if we don't have atomic_read() imply 
> volatile.
> 
> The point here is that it's buggy whether the volatile is there or not! 
> Exactly because the user expects multi-processing behaviour, but 
> "volatile" doesn't actually give any real guarantees about it. Another CPU 
> may have done:
> 
> 	external_ptr = kmalloc(..);
> 	/* Setup is now complete, inform the waiter */
> 	atomic_inc(&var);
> 
> but the fact is, since the other CPU isn't serialized in any way, the 
> "while-loop" (even in the presense of "volatile") doesn't actually work 
> right! Whatever the "atomic_read()" was waiting for may not have 
> completed, because we have no barriers!

Why is all this fixation on "volatile"? I don't think
people want "volatile" keyword per se, they want atomic_read(&x) to
_always_ compile into an memory-accessing instruction, not register access.
--
vda

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Arjan van de Ven]

On Sun, 9 Sep 2007 19:02:54 +0100
Denys Vlasenko <vda.linux@googlemail.com> wrote:

> Why is all this fixation on "volatile"? I don't think
> people want "volatile" keyword per se, they want atomic_read(&x) to
> _always_ compile into an memory-accessing instruction, not register
> access.

and ... why is that?
is there any valid, non-buggy code sequence that makes that a
reasonable requirement?

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Denys Vlasenko]

On Sunday 09 September 2007 19:18, Arjan van de Ven wrote:
> On Sun, 9 Sep 2007 19:02:54 +0100
> Denys Vlasenko <vda.linux@googlemail.com> wrote:
> 
> > Why is all this fixation on "volatile"? I don't think
> > people want "volatile" keyword per se, they want atomic_read(&x) to
> > _always_ compile into an memory-accessing instruction, not register
> > access.
> 
> and ... why is that?
> is there any valid, non-buggy code sequence that makes that a
> reasonable requirement?

Well, if you insist on having it again:

Waiting for atomic value to be zero:

        while (atomic_read(&x))
                continue;

gcc may happily convert it into:

        reg = atomic_read(&x);
        while (reg)
                continue;

Expecting every driver writer to remember that atomic_read is not in fact
a "read from memory" is naive. That won't happen. Face it, majority of
driver authors are a bit less talented than Ingo Molnar or Arjan van de Ven ;)
The name of the macro is saying that it's a read.
We are confusing users here.

It's doubly confusing that cpy_relax(), which says _nothing_ about barriers
in its name, is actually a barrier you need to insert here.
--
vda

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Herbert Xu]

On Mon, Sep 10, 2007 at 11:56:29AM +0100, Denys Vlasenko wrote:
> 
> Expecting every driver writer to remember that atomic_read is not in fact
> a "read from memory" is naive. That won't happen. Face it, majority of
> driver authors are a bit less talented than Ingo Molnar or Arjan van de Ven ;)
> The name of the macro is saying that it's a read.
> We are confusing users here.

For driver authors who're too busy to learn the intricacies
of atomic operations, we have the plain old spin lock which
then lets you use normal data structures such as u32 safely.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Kyle Moffett]

On Sep 10, 2007, at 06:56:29, Denys Vlasenko wrote:
> On Sunday 09 September 2007 19:18, Arjan van de Ven wrote:
>> On Sun, 9 Sep 2007 19:02:54 +0100
>> Denys Vlasenko <vda.linux@googlemail.com> wrote:
>>
>>> Why is all this fixation on "volatile"? I don't think people want  
>>> "volatile" keyword per se, they want atomic_read(&x) to _always_  
>>> compile into an memory-accessing instruction, not register access.
>>
>> and ... why is that?  is there any valid, non-buggy code sequence  
>> that makes that a reasonable requirement?
>
> Well, if you insist on having it again:
>
> Waiting for atomic value to be zero:
>
>         while (atomic_read(&x))
>                 continue;
>
> gcc may happily convert it into:
>
>         reg = atomic_read(&x);
>         while (reg)
>                 continue;

Bzzt.  Even if you fixed gcc to actually convert it to a busy loop on  
a memory variable, you STILL HAVE A BUG as it may *NOT* be gcc that  
does the conversion, it may be that the CPU does the caching of the  
memory value.  GCC has no mechanism to do cache-flushes or memory- 
barriers except through our custom inline assembly.  Also, you  
probably want a cpu_relax() in there somewhere to avoid overheating  
the CPU.  Thirdly, on a large system it may take some arbitrarily  
large amount of time for cache-propagation to update the value of the  
variable in your local CPU cache.  Finally, if atomics are based on  
based on spinlock+interrupt-disable then you will sit in a tight busy- 
loop of spin_lock_irqsave()->spin_unlock_irqrestore().  Depending on  
your system's internal model this may practically lock up your core  
because the spin_lock() will take the cacheline for exclusive access  
and doing that in a loop can prevent any other CPU from doing any  
operation on it!  Since your IRQs are disabled you even have a very  
small window that an IRQ will come along and free it up long enough  
for the update to take place.

The earlier code segment of:
> while(atomic_read(&x) > 0)
> 	atomic_dec(&x);
is *completely* buggy because you could very easily have 4 CPUs doing  
this on an atomic variable with a value of 1 and end up with it at  
negative 3 by the time you are done.  Moreover all the alternatives  
are also buggy, with the sole exception of this rather obvious- 
seeming one:
> atomic_set(&x, 0);

You simply CANNOT use an atomic_t as your sole synchronizing  
primitive, it doesn't work!  You virtually ALWAYS want to use an  
atomic_t in the following types of situations:

(A) As an object refcount.  The value is never read except as part of  
an atomic_dec_return().  Why aren't you using "struct kref"?

(B) As an atomic value counter (number of processes, for example).   
Just "reading" the value is racy anyways, if you want to enforce a  
limit or something then use atomic_inc_return(), check the result,  
and use atomic_dec() if it's too big.  If you just want to return the  
statistics then you are going to be instantaneous-point-in-time anyways.

(C) As an optimization value (statistics-like, but exact accuracy  
isn't important).

Atomics are NOT A REPLACEMENT for the proper kernel subsystem, like  
completions, mutexes, semaphores, spinlocks, krefs, etc.  It's not  
useful for synchronization, only for keeping track of simple integer  
RMW values.  Note that atomic_read() and atomic_set() aren't very  
useful RMW primitives (read-nomodify-nowrite and read-set-zero- 
write).  Code which assumes anything else is probably buggy in other  
ways too.

So while I see no real reason for the "volatile" on the atomics, I  
also see no real reason why it's terribly harmful.  Regardless of the  
"volatile" on the operation the CPU is perfectly happy to cache it  
anyways so it doesn't buy you any actual "always-access-memory"  
guarantees.  If you are just interested in it as an optimization you  
could probably just read the properly-aligned integer counter  
directly, an atomic read on most CPUs.

If you really need it to hit main memory *every* *single* *time*  
(Why?  Are you using it instead of the proper kernel subsystem?)   
then you probably need a custom inline assembly helper anyways.

Cheers,
Kyle Moffett

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Denys Vlasenko]

On Monday 10 September 2007 13:22, Kyle Moffett wrote:
> On Sep 10, 2007, at 06:56:29, Denys Vlasenko wrote:
> > On Sunday 09 September 2007 19:18, Arjan van de Ven wrote:
> >> On Sun, 9 Sep 2007 19:02:54 +0100
> >> Denys Vlasenko <vda.linux@googlemail.com> wrote:
> >>
> >>> Why is all this fixation on "volatile"? I don't think people want  
> >>> "volatile" keyword per se, they want atomic_read(&x) to _always_  
> >>> compile into an memory-accessing instruction, not register access.
> >>
> >> and ... why is that?  is there any valid, non-buggy code sequence  
> >> that makes that a reasonable requirement?
> >
> > Well, if you insist on having it again:
> >
> > Waiting for atomic value to be zero:
> >
> >         while (atomic_read(&x))
> >                 continue;
> >
> > gcc may happily convert it into:
> >
> >         reg = atomic_read(&x);
> >         while (reg)
> >                 continue;
> 
> Bzzt.  Even if you fixed gcc to actually convert it to a busy loop on  
> a memory variable, you STILL HAVE A BUG as it may *NOT* be gcc that  
> does the conversion, it may be that the CPU does the caching of the  
> memory value.  GCC has no mechanism to do cache-flushes or memory- 
> barriers except through our custom inline assembly.

CPU can cache the value all right, but it cannot use that cached value
*forever*, it has to react to invalidate cycles on the shared bus
and re-fetch new data.

IOW: atomic_read(&x) which compiles down to memory accessor
will work properly.

> the CPU.  Thirdly, on a large system it may take some arbitrarily  
> large amount of time for cache-propagation to update the value of the  
> variable in your local CPU cache.

Yes, but "arbitrarily large amount of time" is actually measured
in nanoseconds here. Let's say 1000ns max for hundreds of CPUs?

> Also, you   
> probably want a cpu_relax() in there somewhere to avoid overheating  
> the CPU.

Yes, but 
1. CPU shouldn't overheat (in a sense that it gets damaged),
   it will only use more power than needed.
2. cpu_relax() just throttles down my CPU, so it's performance
   optimization only. Wait, it isn't, it's a barrier too.
   Wow, "cpu_relax" is a barrier? How am I supposed to know
   that without reading lkml flamewars and/or header files?

Let's try reading headers. asm-x86_64/processor.h:

#define cpu_relax()   rep_nop()

So, is it a barrier? No clue yet.

/* REP NOP (PAUSE) is a good thing to insert into busy-wait loops. */
static inline void rep_nop(void)
{
        __asm__ __volatile__("rep;nop": : :"memory");
}

Comment explicitly says that it is "a good thing" (doesn't say
that it is mandatory) and says NOTHING about barriers!

Barrier-ness is not mentioned and is hidden in "memory" clobber.

Do you think it's obvious enough for average driver writer?
I think not, especially that it's unlikely for him to even start
suspecting that it is a memory barrier based on the "cpu_relax"
name.

> You simply CANNOT use an atomic_t as your sole synchronizing
> primitive, it doesn't work!  You virtually ALWAYS want to use an  
> atomic_t in the following types of situations:
> 
> (A) As an object refcount.  The value is never read except as part of  
> an atomic_dec_return().  Why aren't you using "struct kref"?
> 
> (B) As an atomic value counter (number of processes, for example).   
> Just "reading" the value is racy anyways, if you want to enforce a  
> limit or something then use atomic_inc_return(), check the result,  
> and use atomic_dec() if it's too big.  If you just want to return the  
> statistics then you are going to be instantaneous-point-in-time anyways.
> 
> (C) As an optimization value (statistics-like, but exact accuracy  
> isn't important).
> 
> Atomics are NOT A REPLACEMENT for the proper kernel subsystem, like  
> completions, mutexes, semaphores, spinlocks, krefs, etc.  It's not  
> useful for synchronization, only for keeping track of simple integer  
> RMW values.  Note that atomic_read() and atomic_set() aren't very  
> useful RMW primitives (read-nomodify-nowrite and read-set-zero- 
> write).  Code which assumes anything else is probably buggy in other  
> ways too.

You are basically trying to educate me how to use atomic properly.
You don't need to do it, as I am (currently) not a driver author.

I am saying that people who are already using atomic_read()
(and who unfortunately did not read your explanation above)
will still sometimes use atomic_read() as a way to read atomic value
*from memory*, and will create nasty heisenbugs for you to debug.
--
vda

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Denys Vlasenko]

On Monday 10 September 2007 14:38, Denys Vlasenko wrote:
> You are basically trying to educate me how to use atomic properly.
> You don't need to do it, as I am (currently) not a driver author.
> 
> I am saying that people who are already using atomic_read()
> (and who unfortunately did not read your explanation above)
> will still sometimes use atomic_read() as a way to read atomic value
> *from memory*, and will create nasty heisenbugs for you to debug.

static inline int
qla2x00_wait_for_loop_ready(scsi_qla_host_t *ha)
{
        int      return_status = QLA_SUCCESS;
        unsigned long loop_timeout ;
        scsi_qla_host_t *pha = to_qla_parent(ha);

        /* wait for 5 min at the max for loop to be ready */
        loop_timeout = jiffies + (MAX_LOOP_TIMEOUT * HZ);

        while ((!atomic_read(&pha->loop_down_timer) &&
            atomic_read(&pha->loop_state) == LOOP_DOWN) ||
            atomic_read(&pha->loop_state) != LOOP_READY) {
                if (atomic_read(&pha->loop_state) == LOOP_DEAD) {
                        return_status = QLA_FUNCTION_FAILED;
                        break;
                }
                msleep(1000);
                if (time_after_eq(jiffies, loop_timeout)) {
                        return_status = QLA_FUNCTION_FAILED;
                        break;
                }
        }
        return (return_status);
}

Is above correct or buggy? Correct, because msleep is a barrier.
Is it obvious? No.

static void
qla2x00_rst_aen(scsi_qla_host_t *ha)
{
        if (ha->flags.online && !ha->flags.reset_active &&
            !atomic_read(&ha->loop_down_timer) &&
            !(test_bit(ABORT_ISP_ACTIVE, &ha->dpc_flags))) {
                do {
                        clear_bit(RESET_MARKER_NEEDED, &ha->dpc_flags);

                        /*
                         * Issue marker command only when we are going to start
                         * the I/O.
                         */
                        ha->marker_needed = 1;
                } while (!atomic_read(&ha->loop_down_timer) &&
                    (test_bit(RESET_MARKER_NEEDED, &ha->dpc_flags)));
        }
}

Is above correct? I honestly don't know. Correct, because set_bit is
a barrier on _all _memory_? Will it break if set_bit will be changed
to be a barrier only on its operand? Probably yes.

drivers/kvm/kvm_main.c

        while (atomic_read(&completed) != needed) {
                cpu_relax();
                barrier();
        }

Obviously author did not know that cpu_relax is already a barrier.
See why I think driver authors will be confused?

arch/x86_64/kernel/crash.c

static void nmi_shootdown_cpus(void)
{
...
        msecs = 1000; /* Wait at most a second for the other cpus to stop */
        while ((atomic_read(&waiting_for_crash_ipi) > 0) && msecs) {
                mdelay(1);
                msecs--;
        }
...
}

Is mdelay(1) a barrier? Yes, because it is a function on x86_64.
Absolutely the same code will be buggy on an arch where
mdelay(1) == udelay(1000), and udelay is implemented
as inline busy-wait.

arch/sparc64/kernel/smp.c

        /* Wait for response */
        while (atomic_read(&data.finished) != cpus)
                cpu_relax();
...later in the same file...
                while (atomic_read(&smp_capture_registry) != ncpus)
                        rmb();

I'm confused. Do we need cpu_relax() or rmb()? Does cpu_relax() imply rmb()?
(No it doesn't). Which of those two while loops needs correcting?
--
vda

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Denys Vlasenko]

On Monday 10 September 2007 15:51, Arjan van de Ven wrote:
> On Mon, 10 Sep 2007 11:56:29 +0100
> Denys Vlasenko <vda.linux@googlemail.com> wrote:
> 
> > 
> > Well, if you insist on having it again:
> > 
> > Waiting for atomic value to be zero:
> > 
> >         while (atomic_read(&x))
> >                 continue;
> > 
> 
> and this I would say is buggy code all the way.
>
> Not from a pure C level semantics, but from a "busy waiting is buggy"
> semantics level and a "I'm inventing my own locking" semantics level.

After inspecting arch/*, I cannot agree with you.
Otherwise almost all major architectures use
"conceptually buggy busy-waiting":

arch/alpha
arch/i386
arch/ia64
arch/m32r
arch/mips
arch/parisc
arch/powerpc
arch/sh
arch/sparc64
arch/um
arch/x86_64

All of the above contain busy-waiting on atomic_read.

Including these loops without barriers:

arch/mips/kernel/smtc.c
			while (atomic_read(&idle_hook_initialized) < 1000)
				;
arch/mips/sgi-ip27/ip27-nmi.c
	while (atomic_read(&nmied_cpus) != num_online_cpus());

[Well maybe num_online_cpus() is a barrier, I didn't check]

arch/sh/kernel/smp.c
	if (wait)
		while (atomic_read(&smp_fn_call.finished) != (nr_cpus - 1));

Bugs?
--
vda

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Arjan van de Ven]

On Mon, 10 Sep 2007 11:56:29 +0100
Denys Vlasenko <vda.linux@googlemail.com> wrote:

> 
> Well, if you insist on having it again:
> 
> Waiting for atomic value to be zero:
> 
>         while (atomic_read(&x))
>                 continue;
> 

and this I would say is buggy code all the way.

Not from a pure C level semantics, but from a "busy waiting is buggy"
semantics level and a "I'm inventing my own locking" semantics level.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Linus Torvalds]

On Mon, 10 Sep 2007, Denys Vlasenko wrote:
> 
> static inline int
> qla2x00_wait_for_loop_ready(scsi_qla_host_t *ha)
> {
>         int      return_status = QLA_SUCCESS;
>         unsigned long loop_timeout ;
>         scsi_qla_host_t *pha = to_qla_parent(ha);
> 
>         /* wait for 5 min at the max for loop to be ready */
>         loop_timeout = jiffies + (MAX_LOOP_TIMEOUT * HZ);
> 
>         while ((!atomic_read(&pha->loop_down_timer) &&
>             atomic_read(&pha->loop_state) == LOOP_DOWN) ||
>             atomic_read(&pha->loop_state) != LOOP_READY) {
>                 if (atomic_read(&pha->loop_state) == LOOP_DEAD) {
...
> Is above correct or buggy? Correct, because msleep is a barrier.
> Is it obvious? No.

It's *buggy*. But it has nothing to do with any msleep() in the loop, or 
anything else.

And more importantly, it would be equally buggy even *with* a "volatile" 
atomic_read().

Why is this so hard for people to understand? You're all acting like 
morons.

The reason it is buggy has absolutely nothing to do with whether the read 
is done or not, it has to do with the fact that the CPU may re-order the 
reads *regardless* of whether the read is done in some specific order by 
the compiler ot not! In effect, there is zero ordering between all those 
three reads, and if you don't have memory barriers (or a lock or other 
serialization), that code is buggy.

So stop this idiotic discussion thread already. The above kind of code 
needs memory barriers to be non-buggy. The whole "volatile or not" 
discussion is totally idiotic, and pointless, and anybody who doesn't 
understand that by now needs to just shut up and think about it more, 
rather than make this discussion drag out even further.

The fact is, "volatile" *only* makes things worse. It generates worse 
code, and never fixes any real bugs. This is a *fact*.

			Linus

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Denys Vlasenko]

On Monday 10 September 2007 16:09, Linus Torvalds wrote:
> On Mon, 10 Sep 2007, Denys Vlasenko wrote:
> > static inline int
> > qla2x00_wait_for_loop_ready(scsi_qla_host_t *ha)
> > {
> >         int      return_status = QLA_SUCCESS;
> >         unsigned long loop_timeout ;
> >         scsi_qla_host_t *pha = to_qla_parent(ha);
> > 
> >         /* wait for 5 min at the max for loop to be ready */
> >         loop_timeout = jiffies + (MAX_LOOP_TIMEOUT * HZ);
> > 
> >         while ((!atomic_read(&pha->loop_down_timer) &&
> >             atomic_read(&pha->loop_state) == LOOP_DOWN) ||
> >             atomic_read(&pha->loop_state) != LOOP_READY) {
> >                 if (atomic_read(&pha->loop_state) == LOOP_DEAD) {
> ...
> > Is above correct or buggy? Correct, because msleep is a barrier.
> > Is it obvious? No.
> 
> It's *buggy*. But it has nothing to do with any msleep() in the loop, or 
> anything else.
> 
> And more importantly, it would be equally buggy even *with* a "volatile" 
> atomic_read().

I am not saying that this code is okay, this isn't the point.
(The code is in fact awful for several more reasons).

My point is that people are confused as to what atomic_read()
exactly means, and this is bad. Same for cpu_relax().
First one says "read", and second one doesn't say "barrier".

This is real code from current kernel which demonstrates this:

"I don't know that cpu_relax() is a barrier already":

drivers/kvm/kvm_main.c
        while (atomic_read(&completed) != needed) {
                cpu_relax();
                barrier();
        }

"I think that atomic_read() is a read from memory and therefore
I don't need a barrier":

arch/x86_64/kernel/crash.c
        msecs = 1000; /* Wait at most a second for the other cpus to stop */
        while ((atomic_read(&waiting_for_crash_ipi) > 0) && msecs) {
                mdelay(1);
                msecs--;
        }

Since neither camp seems to give up, I am proposing renaming
them to something less confusing, and make everybody happy.

cpu_relax_barrier()
atomic_value(&x)
atomic_fetch(&x)

I'm not native English speaker, do these sound better?
--
vda

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Arjan van de Ven]

On Mon, 10 Sep 2007 15:38:23 +0100
Denys Vlasenko <vda.linux@googlemail.com> wrote:

> On Monday 10 September 2007 15:51, Arjan van de Ven wrote:
> > On Mon, 10 Sep 2007 11:56:29 +0100
> > Denys Vlasenko <vda.linux@googlemail.com> wrote:
> > 
> > > 
> > > Well, if you insist on having it again:
> > > 
> > > Waiting for atomic value to be zero:
> > > 
> > >         while (atomic_read(&x))
> > >                 continue;
> > > 
> > 
> > and this I would say is buggy code all the way.
> >
> > Not from a pure C level semantics, but from a "busy waiting is
> > buggy" semantics level and a "I'm inventing my own locking"
> > semantics level.
> 
> After inspecting arch/*, I cannot agree with you.

the arch/ people obviously are allowed to do their own locking stuff...
BECAUSE THEY HAVE TO IMPLEMENT THAT!


the arch maintainers know EXACTLY how their hw behaves (well, we hope)
so they tend to be the exception to many rules in the kernel....

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Christoph Lameter]

On Fri, 17 Aug 2007, Segher Boessenkool wrote:

> "volatile" has nothing to do with reordering.  atomic_dec() writes
> to memory, so it _does_ have "volatile semantics", implicitly, as
> long as the compiler cannot optimise the atomic variable away
> completely -- any store counts as a side effect.

Stores can be reordered. Only x86 has (mostly) implicit write ordering. So 
no atomic_dec has no volatile semantics and may be reordered on a variety 
of processors. Writes to memory may not follow code order on several 
processors.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Christoph Lameter]

On Mon, 10 Sep 2007, Linus Torvalds wrote:

> The fact is, "volatile" *only* makes things worse. It generates worse 
> code, and never fixes any real bugs. This is a *fact*.

Yes, lets just drop the volatiles now! We need a patch that gets rid of 
them.... Volunteers?

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Kyle Moffett]

On Sep 10, 2007, at 12:46:33, Denys Vlasenko wrote:
> My point is that people are confused as to what atomic_read()   
> exactly means, and this is bad. Same for cpu_relax().  First one  
> says "read", and second one doesn't say "barrier".

Q&A:

Q:  When is it OK to use atomic_read()?
A:  You are asking the question, so never.

Q:  But I need to check the value of the atomic at this point in time...
A:  Your code is buggy if it needs to do that on an atomic_t for  
anything other than debugging or optimization.  Use either  
atomic_*_return() or a lock and some normal integers.

Q:  "So why can't the atomic_read DTRT magically?"
A:  Because "the right thing" depends on the situation and is usually  
best done with something other than atomic_t.

If somebody can post some non-buggy code which is correctly using  
atomic_read() *and* depends on the compiler generating extra  
nonsensical loads due to "volatile" then the issue *might* be  
reconsidered.  This also includes samples of code which uses  
atomic_read() and needs memory barriers (so that we can fix the buggy  
code, not so we can change atomic_read()).  So far the only code  
samples anybody has posted are buggy regardless of whether or not the  
value and/or accessors are flagged "volatile" or not.  And hey, maybe  
the volatile ops *should* be implemented in inline ASM for future- 
proof-ness, but that's a separate issue.

Cheers,
Kyle Moffett

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Mon, Sep 10, 2007 at 11:59:29AM -0700, Christoph Lameter wrote:
> On Fri, 17 Aug 2007, Segher Boessenkool wrote:
> 
> > "volatile" has nothing to do with reordering.  atomic_dec() writes
> > to memory, so it _does_ have "volatile semantics", implicitly, as
> > long as the compiler cannot optimise the atomic variable away
> > completely -- any store counts as a side effect.
> 
> Stores can be reordered. Only x86 has (mostly) implicit write ordering. So 
> no atomic_dec has no volatile semantics and may be reordered on a variety 
> of processors. Writes to memory may not follow code order on several 
> processors.

The one exception to this being the case where process-level code is
communicating to an interrupt handler running on that same CPU -- on
all CPUs that I am aware of, a given CPU always sees its own writes
in order.

							Thanx, Paul

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Christoph Lameter]

On Mon, 10 Sep 2007, Paul E. McKenney wrote:

> The one exception to this being the case where process-level code is
> communicating to an interrupt handler running on that same CPU -- on
> all CPUs that I am aware of, a given CPU always sees its own writes
> in order.

Yes but that is due to the code path effectively continuing in the 
interrupt handler. The cpu makes sure that op codes being executed always 
see memory in a consistent way. The basic ordering problem with out of 
order writes is therefore coming from other processors concurrently 
executing code and holding variables in registers that are modified 
elsewhere. The only solution that I know of are one or the other form of 
barrier.

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Paul E. McKenney]

On Mon, Sep 10, 2007 at 02:36:26PM -0700, Christoph Lameter wrote:
> On Mon, 10 Sep 2007, Paul E. McKenney wrote:
> 
> > The one exception to this being the case where process-level code is
> > communicating to an interrupt handler running on that same CPU -- on
> > all CPUs that I am aware of, a given CPU always sees its own writes
> > in order.
> 
> Yes but that is due to the code path effectively continuing in the 
> interrupt handler. The cpu makes sure that op codes being executed always 
> see memory in a consistent way. The basic ordering problem with out of 
> order writes is therefore coming from other processors concurrently 
> executing code and holding variables in registers that are modified 
> elsewhere. The only solution that I know of are one or the other form of 
> barrier.

So we are agreed then -- volatile accesses may be of some assistance when
interacting with interrupt handlers running on the same CPU (presumably
when using per-CPU variables), but are generally useless when sharing
variables among CPUs.  Correct?

							Thanx, Paul

[PATCH] Document non-semantics of atomic_read() and atomic_set()

[Chris Snook]

From: Chris Snook <csnook@redhat.com>

Unambiguously document the fact that atomic_read() and atomic_set()
do not imply any ordering or memory access, and that callers are
obligated to explicitly invoke barriers as needed to ensure that
changes to atomic variables are visible in all contexts that need
to see them.

Signed-off-by: Chris Snook <csnook@redhat.com>

--- a/Documentation/atomic_ops.txt	2007-07-08 19:32:17.000000000 -0400
+++ b/Documentation/atomic_ops.txt	2007-09-10 19:02:50.000000000 -0400
@@ -12,7 +12,11 @@
 C integer type will fail.  Something like the following should
 suffice:
 
-	typedef struct { volatile int counter; } atomic_t;
+	typedef struct { int counter; } atomic_t;
+
+	Historically, counter has been declared volatile.  This is now
+discouraged.  See Documentation/volatile-considered-harmful.txt for the
+complete rationale.
 
 	The first operations to implement for atomic_t's are the
 initializers and plain reads.
@@ -42,6 +46,22 @@
 
 which simply reads the current value of the counter.
 
+*** WARNING: atomic_read() and atomic_set() DO NOT IMPLY BARRIERS! ***
+
+Some architectures may choose to use the volatile keyword, barriers, or
+inline assembly to guarantee some degree of immediacy for atomic_read()
+and atomic_set().  This is not uniformly guaranteed, and may change in
+the future, so all users of atomic_t should treat atomic_read() and
+atomic_set() as simple C assignment statements that may be reordered or
+optimized away entirely by the compiler or processor, and explicitly
+invoke the appropriate compiler and/or memory barrier for each use case.
+Failure to do so will result in code that may suddenly break when used with
+different architectures or compiler optimizations, or even changes in
+unrelated code which changes how the compiler optimizes the section
+accessing atomic_t variables.
+
+*** YOU HAVE BEEN WARNED! ***
+
 Now, we move onto the actual atomic operation interfaces.
 
 	void atomic_add(int i, atomic_t *v);

Re: [PATCH] Document non-semantics of atomic_read() and atomic_set()

[Paul E. McKenney]

On Mon, Sep 10, 2007 at 07:19:44PM -0400, Chris Snook wrote:
> From: Chris Snook <csnook@redhat.com>
> 
> Unambiguously document the fact that atomic_read() and atomic_set()
> do not imply any ordering or memory access, and that callers are
> obligated to explicitly invoke barriers as needed to ensure that
> changes to atomic variables are visible in all contexts that need
> to see them.

Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

> Signed-off-by: Chris Snook <csnook@redhat.com>
> 
> --- a/Documentation/atomic_ops.txt	2007-07-08 19:32:17.000000000 -0400
> +++ b/Documentation/atomic_ops.txt	2007-09-10 19:02:50.000000000 -0400
> @@ -12,7 +12,11 @@
>  C integer type will fail.  Something like the following should
>  suffice:
> 
> -	typedef struct { volatile int counter; } atomic_t;
> +	typedef struct { int counter; } atomic_t;
> +
> +	Historically, counter has been declared volatile.  This is now
> +discouraged.  See Documentation/volatile-considered-harmful.txt for the
> +complete rationale.
> 
>  	The first operations to implement for atomic_t's are the
>  initializers and plain reads.
> @@ -42,6 +46,22 @@
> 
>  which simply reads the current value of the counter.
> 
> +*** WARNING: atomic_read() and atomic_set() DO NOT IMPLY BARRIERS! ***
> +
> +Some architectures may choose to use the volatile keyword, barriers, or
> +inline assembly to guarantee some degree of immediacy for atomic_read()
> +and atomic_set().  This is not uniformly guaranteed, and may change in
> +the future, so all users of atomic_t should treat atomic_read() and
> +atomic_set() as simple C assignment statements that may be reordered or
> +optimized away entirely by the compiler or processor, and explicitly
> +invoke the appropriate compiler and/or memory barrier for each use case.
> +Failure to do so will result in code that may suddenly break when used with
> +different architectures or compiler optimizations, or even changes in
> +unrelated code which changes how the compiler optimizes the section
> +accessing atomic_t variables.
> +
> +*** YOU HAVE BEEN WARNED! ***
> +
>  Now, we move onto the actual atomic operation interfaces.
> 
>  	void atomic_add(int i, atomic_t *v);

Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures

[Segher Boessenkool]

>> "volatile" has nothing to do with reordering.  atomic_dec() writes
>> to memory, so it _does_ have "volatile semantics", implicitly, as
>> long as the compiler cannot optimise the atomic variable away
>> completely -- any store counts as a side effect.
>
> Stores can be reordered. Only x86 has (mostly) implicit write ordering.
> So no atomic_dec has no volatile semantics

Read again: I said the C "volatile" construct has nothing to do
with CPU memory access reordering.

> and may be reordered on a variety
> of processors. Writes to memory may not follow code order on several
> processors.

The _compiler_ isn't allowed to reorder things here.  Yes, of course
you do need stronger barriers for many purposes, volatile isn't all
that useful you know.


Segher

Re: [PATCH] Document non-semantics of atomic_read() and atomic_set()

[Christoph Lameter]

Acked-by: Christoph Lameter <clameter@sgi.com>

(unknown),

[Martin Hofmann]

subscribe netdev

Mit freundlichen Grüßen


Martin Hofmann



###############################
# Martin Hofmann              #
# Quellenweg 3                #
# 91459 Markt Erlbach         #
#-----------------------------#
# Telefon: 09106 / 92 54 52   #
# Mobil: 0170 / 96 03 765     #
# Mail: hofmann.martin@odn.de #
###############################

(unknown),

[Bryan Wu]

(unknown),

[Masahide NAKAMURA]

Subject: [XFRM] Documentaion: Fix error example at XFRMOUTSTATEMODEERROR. (Re: [XFRM]: Fix outbound statistics.)

Hello,

On Fri, 21 Dec 2007 23:11:11 +0800
Herbert Xu <herbert@gondor.apana.org.au> wrote:

> On Fri, Dec 21, 2007 at 11:25:00PM +0900, Masahide NAKAMURA wrote:
> >
> >	do {
> >		err = xfrm_state_check_space(x, skb);
> > -		if (err)
> > +		if (err) {
> > +			XFRM_INC_STATS(LINUX_MIB_XFRMOUTERROR);
> >			goto error_nolock;
> > +		}
> >  
> >		err = x->outer_mode->output(x, skb);
> > -		if (err)
> > +		if (err) {
> > +			XFRM_INC_STATS(LINUX_MIB_XFRMOUTSTATEMODEERROR);
> 
> BTW, none of our existing mode output functions actually return
> an error.  I noticed that the description for this field is actually
> "Transformation mode specific error, e.g. Outer header space is not
> enough".  This is slightly misleading as output header space is
> checked by xfrm_state_check_space so if there's an error that's
> where it'll show up.

Thanks for comment, Herbert.

I fix the documentation to remove "e.g. Outer header space is not enough"
from XFRMSTATEMODEERROR.
About error code from xfrm_state_check_space(), I still map it XFRMOUTERROR
(other errors) this time because I think the error here is not a length
error by protocol (e.g MTU related things) but an internal buffer management.

Any comments for the statistics are still welcomed.

David, please apply the following patch, too.


[XFRM] Documentaion: Fix error example at XFRMOUTSTATEMODEERROR.

Signed-off-by: Masahide NAKAMURA <nakam@linux-ipv6.org>
---
 Documentation/networking/xfrm_proc.txt |    1 -
 1 files changed, 0 insertions(+), 1 deletions(-)

diff --git a/Documentation/networking/xfrm_proc.txt b/Documentation/networking/xfrm_proc.txt
index ec9045b..53c1a58 100644
--- a/Documentation/networking/xfrm_proc.txt
+++ b/Documentation/networking/xfrm_proc.txt
@@ -60,7 +60,6 @@ XfrmOutStateProtoError:
 	Transformation protocol specific error
 XfrmOutStateModeError:
 	Transformation mode specific error
-	e.g. Outer header space is not enough
 XfrmOutStateExpired:
 	State is expired
 XfrmOutPolBlock:
-- 
1.4.4.2

(unknown)

[Li Yewang]

subscribe netdev

(unknown),

[内藤 賢司]

majordomo@vger.kernel.org

(unknown),

[Salvatore De Astis]

Hi,
I'm a junior developer and I need some informations about the octal
UART chip  Exar xr17d158.
I have two pci cards (PSCC-1028 - Actis-Computer cards) with octal
UART chip  Exar xr17d158. Linux finds the hardware and sets up the
ports (from ttyS4 to ttyS19). I read the code of 8250_pci.c and this
chip seems to be supported by linux.
 But I can't transmit and receive because the chip needs specific
values  to be written in some registers. These ones are: mpiolvl,
mpiosel and  mpio3t. I have wrote this simple module and when I load
it the board work in 232 mode.

#include <linux/module.h>
#include <linux/init.h>
#include <linux/pci.h>
#include <asm/io.h>

#define CONF_REG_START     0x80
#define CONF_REG_SIZE     0x90

typedef struct _xr17d168_conf_regs {
     u8 int0;      /* Read-only Interrupt              - value 0x00 */
    u8 int1;      /* Read-only                    - value 0x00 */
    u8 int2;      /* Read-only                    - value 0x00 */
    u8 int3;      /* Read-only                    - value 0x00 */

    u8 timercntl; /* Read-write Timer control              - value 0x00 */
    u8 timer;      /* Reserved                      - value 0x00 */
    u8 timerlsb;  /* Read/Write Timer LSB              - value 0x00 */
     u8 timermsb;  /* Read/Write Timer MSB              - value 0x00 */

    u8 mode8x;    /* Read-write                   - value 0x00 */
    u8 rega;      /* Reserved                      - value 0x00 */
    u8 reset;      /* Write only Self clear bits after reset      -
value 0x00 */
     u8 sleep;      /* Read/Write Sleep mode              - value 0x00 */

    u8 drev;      /* Read-only Device Revision            - value 0x09 */
    u8 dvid;      /* Read-only Device Identification          - value 0x28 */
     u8 regb;      /* Write-only                   - value 0x00 */
    u8 mpioint;   /* Read/Write MPIO interrupt mask          - value 0x00 */

    u8 mpiolvl;   /* Read/Write MPIO level control          - value 0x00 */
     u8 mpio3t;    /* Read/Write MPIO output control          - value 0x00 */
    u8 mpioinv;   /* Read/Write MPIO input polarity select      - value 0x00 */
    u8 mpiosel;   /* Read/Write MPIO select              - value 0xFF */
 } xr17d168_conf_regs;

xr17d168_conf_regs *xr17d168_conf_regs_ptr;

int xr17d168_init(void) {

    struct pci_dev *dev;
    int ret;
    u8 mode;
    unsigned long start_addr, end_addr, size;


    dev = pci_get_device(PCI_VENDOR_ID_EXAR, PCI_DEVICE_ID_EXAR_XR17C158, NULL);
    if(dev) {
        printk(KERN_ALERT "EXAR XR17D168 Octal PCI UART Driver init.\n");

    ret = pci_enable_device(dev);

    start_addr = pci_resource_start(dev, 0);
    end_addr = pci_resource_end(dev, 0);
    size = end_addr - start_addr;

    xr17d168_conf_regs_ptr = (xr17d168_conf_regs
*)ioremap(start_addr+CONF_REG_START,CONF_REG_SIZE);

     mode = 0xec;
    iowrite8(mode, &xr17d168_conf_regs_ptr->mpiolvl);
        iowrite8(0, &xr17d168_conf_regs_ptr->mpiosel);
        iowrite8(0, &xr17d168_conf_regs_ptr->mpio3t);

    pci_dev_put(dev);
    }
    return 0;
}

void xr17d168_exit(void) {

    printk(KERN_ALERT "EXAR XR17D168 Octal PCI UART Driver exit.\n");
    iowrite8(0, &xr17d168_conf_regs_ptr->mpiolvl);
     iowrite8(0xff, &xr17d168_conf_regs_ptr->reset);
    iounmap((void *)xr17d168_conf_regs_ptr);
}

static struct pci_driver
module_init(xr17d168_init);
module_exit(xr17d168_exit);
MODULE_LICENSE("Dual BSD/GPL");

This is my first linux driver so excuse me if it isn't perfect! I'm
working to improve myself.
My questions are:
1) Is the driver necessary to set properly the configuration registers?
2) Does exist a method (like ioctl, sysfs or proc) to configure my boards?
3) What is the better way for manage this chip that has the capability
to work in 232, 422 or half ports in 232 and others in 422 modes?

Excuse me for my bad  english!!

Yours sincerly,
-- 
Salvatore De Astis

(unknown),

[Bryan Wu]

(unknown)

[Dave Young]

marcel@holtmann.org
Bcc: 
Subject: Re: bluetooth lockdep trace. (.25rc5-git4)
Reply-To: 
In-Reply-To: <20080328.182021.46780895.davem@davemloft.net>

On Fri, Mar 28, 2008 at 06:20:21PM -0700, David Miller wrote:
> From: Dave Jones <davej@codemonkey.org.uk>
> Date: Thu, 27 Mar 2008 12:21:56 -0400
> 
> > Mar 27 08:10:57 localhost kernel: Pid: 3611, comm: obex-data-serve Not tainted 2.6.25-0.121.rc5.git4.fc9 #1
> > Mar 27 08:10:57 localhost kernel:  [__lock_acquire+2287/3089] __lock_acquire+0x8ef/0xc11
> > Mar 27 08:10:57 localhost kernel:  [sched_clock+8/11] ? sched_clock+0x8/0xb
> > Mar 27 08:10:57 localhost kernel:  [lock_acquire+106/144] lock_acquire+0x6a/0x90
> > Mar 27 08:10:57 localhost kernel:  [<f8bd9321>] ? l2cap_sock_bind+0x29/0x108 [l2cap]
> > Mar 27 08:10:57 localhost kernel:  [lock_sock_nested+182/198] lock_sock_nested+0xb6/0xc6
> > Mar 27 08:10:57 localhost kernel:  [<f8bd9321>] ? l2cap_sock_bind+0x29/0x108 [l2cap]
> > Mar 27 08:10:57 localhost kernel:  [security_socket_post_create+22/27] ? security_socket_post_create+0x16/0x1b
> > Mar 27 08:10:57 localhost kernel:  [__sock_create+388/472] ? __sock_create+0x184/0x1d8
> > Mar 27 08:10:57 localhost kernel:  [<f8bd9321>] l2cap_sock_bind+0x29/0x108 [l2cap]
> > Mar 27 08:10:57 localhost kernel:  [kernel_bind+10/13] kernel_bind+0xa/0xd
> > Mar 27 08:10:57 localhost kernel:  [<f8dad3d7>] rfcomm_dlc_open+0xc8/0x294 [rfcomm]
> > Mar 27 08:10:57 localhost kernel:  [lock_sock_nested+187/198] ? lock_sock_nested+0xbb/0xc6
> > Mar 27 08:10:57 localhost kernel:  [<f8dae18c>] rfcomm_sock_connect+0x8b/0xc2 [rfcomm]
> > Mar 27 08:10:57 localhost kernel:  [sys_connect+96/125] sys_connect+0x60/0x7d
> > Mar 27 08:10:57 localhost kernel:  [__lock_acquire+1370/3089] ? __lock_acquire+0x55a/0xc11
> > Mar 27 08:10:57 localhost kernel:  [sys_socketcall+140/392] sys_socketcall+0x8c/0x188
> > Mar 27 08:10:57 localhost kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
> 
> rfcomm connect locks the socket, then does rfcomm_dlc_open which in
> turn can do a l2cap_sock_bind on a seperate second socket which in
> turn locks that second socket.
> 
> Both of these sockets are AF_BLUETOOTH family, so lockdep thinks there
> is a locking conflict, even though what is happening here is perfectly
> fine since the two sockets are totally different AF_BLUETOOTH
> sub-types.
> 
> Bluetooth will need to use sock_lock_init_class_and_name() and
> lock sub-classes per AF_BLUETOOTH socket sub-type.
> 
> David, could you or someone else work on this?

Does this fix the problem?

---
'rfcomm connect' will trigger lockdep warnings which is caused by
locking diffrent kinds of bluetooth sockets at the same time.

So using sub-classes per AF_BLUETOOTH sub-type for lockdep.

Thanks for the hints from dave jones.


Signed-off-by: Dave Young <hidave.darkstar@gmail.com>

---
net/bluetooth/af_bluetooth.c |   40 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 40 insertions(+)

diff -upr linux/net/bluetooth/af_bluetooth.c linux.new/net/bluetooth/af_bluetooth.c
--- linux/net/bluetooth/af_bluetooth.c	2008-04-01 16:09:17.000000000 +0800
+++ linux.new/net/bluetooth/af_bluetooth.c	2008-04-01 16:08:52.000000000 +0800
@@ -53,6 +53,30 @@
 /* Bluetooth sockets */
 #define BT_MAX_PROTO	8
 static struct net_proto_family *bt_proto[BT_MAX_PROTO];
+
+static struct lock_class_key bt_slock_key[BT_MAX_PROTO];
+static struct lock_class_key bt_lock_key[BT_MAX_PROTO];
+static const char *bt_key_strings[BT_MAX_PROTO] = {
+	"sk_lock-AF_BLUETOOTH-BTPROTO_L2CAP",
+	"sk_lock-AF_BLUETOOTH-BTPROTO_HCI",
+	"sk_lock-AF_BLUETOOTH-BTPROTO_SCO",
+	"sk_lock-AF_BLUETOOTH-BTPROTO_RFCOMM",
+	"sk_lock-AF_BLUETOOTH-BTPROTO_BNEP",
+	"sk_lock-AF_BLUETOOTH-BTPROTO_CMTP",
+	"sk_lock-AF_BLUETOOTH-BTPROTO_HIDP",
+	"sk_lock-AF_BLUETOOTH-BTPROTO_AVDTP",
+};
+
+static const char *bt_slock_key_strings[BT_MAX_PROTO] = {
+	"slock-AF_BLUETOOTH-BTPROTO_L2CAP",
+	"slock-AF_BLUETOOTH-BTPROTO_HCI",
+	"slock-AF_BLUETOOTH-BTPROTO_SCO",
+	"slock-AF_BLUETOOTH-BTPROTO_RFCOMM",
+	"slock-AF_BLUETOOTH-BTPROTO_BNEP",
+	"slock-AF_BLUETOOTH-BTPROTO_CMTP",
+	"slock-AF_BLUETOOTH-BTPROTO_HIDP",
+	"slock-AF_BLUETOOTH-BTPROTO_AVDTP",
+};
 static DEFINE_RWLOCK(bt_proto_lock);
 
 int bt_sock_register(int proto, struct net_proto_family *ops)
@@ -95,6 +119,21 @@ int bt_sock_unregister(int proto)
 }
 EXPORT_SYMBOL(bt_sock_unregister);
 
+static void bt_reclassify_sock_lock(struct socket *sock, int proto)
+{
+	struct sock *sk = sock->sk;
+
+	if (!sk)
+		return;
+	BUG_ON(sock_owned_by_user(sk));
+
+	sock_lock_init_class_and_name(sk,
+			bt_slock_key_strings[proto],
+			&bt_slock_key[proto],
+			bt_key_strings[proto],
+			&bt_lock_key[proto]);
+}
+
 static int bt_sock_create(struct net *net, struct socket *sock, int proto)
 {
 	int err;
@@ -117,6 +156,7 @@ static int bt_sock_create(struct net *ne
 
 	if (bt_proto[proto] && try_module_get(bt_proto[proto]->owner)) {
 		err = bt_proto[proto]->create(net, sock, proto);
+		bt_reclassify_sock_lock(sock, proto);
 		module_put(bt_proto[proto]->owner);
 	}
 

(unknown)

[Bruce Allen]

Dear Netdev,

We're doing some HPC Linpack testing on cluster nodes with the e1000 
driver. Sometimes the networking appears to 'hang'. The switches have 
their egress buffers full, but it seems as if the compute nodes are not 
accepting any more traffic.  (Yes, pause RX is off.)

We notice that rx_no_buffer_count and rx_missed_errors sometimes have high 
counts.  Could somebody explain to us what this means?  Any suggestions 
for improving matters?

Cheers,
 	Bruce

(unknown)

[Hannes Hering]

subscribe netdev

(unknown),

[Van Gelder, Gerrit]

Dear Sir

I'm a student from a college in Antwerp, called "Hogeschool Antwerpen".
I'm studying for Industrial engineer and I'm now in my third year.

The reason for this mail is a problem I have with PHPnetemGUI. I
installed an apache webserver with php, and everything seems to work,
but it doesn't do anything.

What i'm trying to say is: Netem does work in command line, but with the
use of php scripts it doesn't. Perhaps it is a problem with our =
restrictions but I don't know if we did something wrong. I just followed 
your tutorial.

Do you know where my problem is situated? What can I do to overcome this
failure? And is there a possibility to receive extra documentation about
PHPnetemGUI.

If you guys don't know or use PHPnetemGUI, is there an other grafical way to add jitter and delays graphically in Ubuntu Linux.

Kind regards,

Gerrit

(unknown)

[Kuba Konczyk]

subscribe netdev

(unknown),

[paquerotm]

Contact Mr.Mark Smith for the claim of 1,000,000.00 Pounds which you have won .Send your Name,Address,Age,Tel,Country,Occupation. Email:nll9835656@btinternet.com

(unknown),

[Mr Song Lile]

Dear Friend,

I am Mr. Song Lile and I work with Heng Sang Bank here in Hong Kong. I have a
business transaction of $19,500.000.00 to share with you.  If you are
interested get back to me with the listed information
Name:__________Address:________Phone No:______
Country:________Occupation:________  Below via my personal E-
Mail :songlileprivate002@live.com for more Details.

Sincerely,
Mr. Lile Song

(unknown),

[Валерия]

Привет!

Heoбxoдим отличный интеpнeт на pacтoянии цeнтpa?

Спутниковый интеpнeт - лучшее решение! московской обл!

Намного дешевле и лучше

Звони 8{918}6173341

Всего доброго!












s0X8txYr6
G2SGX

(unknown),

[Mark Bishop]

subscribe netdev

(unknown),

[Peter Dusel]

subscribe netdev

(unknown),

[Mr Wong Chong]

Complement of the Day

I am Mr. Wong Chong and I work with Hong Leong Bank here in My Country. I
have a business transaction of $19,500.000.00 to share with you.  If you
are interested get back to me with the listed information  
Name:__________Address:________Phone No:______
Country:________Occupation:________:age  Below via my personal E-Mail
:wong_chongprivate@yahoo.com.hk  for more Details.

Sincerely,
Mr.Wong Chong


-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

(unknown),

[Mr TONY HILL]

-- 
I am Mr. Zhang Tiejun operations manager of the Bank of China. i have a
business for you. please get back to me if interested

(unknown),

[FedEx Courier Servic]

Potrdi LASTNI?TVO (PAKET)
Dragi gost!
    Smo vas, da nas kontaktirate za va? Confirmable Package
to
je registrirano pri nas, za odpremo yourresidential lokaciji. Imeli smo
misel
da je va? po?iljatelj dal si na?e kontaktne podatke. Morda vas na
ugotavlja, da
pismo je prav tako dodana v va? paket. Vendar pa ne moremo citatom svoje
vsebine na
vam preko e-po?te za zasebnost razlogov razumemo, da je vsebina va?ega
Sam paket je banka Osnutek, ki vreden ve? kot $ 500000.00.usd Kot veste,
FedEx ne ladja denarja v gotovini ali v CHEQUES ampak banke Osnutki so
shippable.The
Paket je registrirano pri nas mailing va? kolega, in va?e
kolegica
pojasnjeno, da je iz Zdru?enih dr?av, vendar pa je tu v Nigeriji za
tri (3) mesecev, kot je popis projekt heworks z izgradnjo podjetje v
Nigerija obmo?je zahodne Afrike, smo sendingyou to e-po?tno sporo?ilo, ker va?
paket
je bil registriran na posebna Order.What, kar mora? storiti zdaj, da  
se obrnete
na
Dostava na? oddelek za takoj?njo odpremo svoj paket na va?e
stanovanjsko address.Note da kakor hitro dostavo na?ih Teamconfirms va?e
informacij bo takeonly en delovni dan (24 ur) za yourpackage za
prispejo njegov poobla??eni address.For va?ih podatkov, DDV & Dostava
pristojbin, kot tudi za zavarovanja so bile pla?ane pristojbine, ki jih va?
kolega pred
tvoj
paket je bil registriran. Upo?tevajte, da je pla?ilo, ki je narejen na
zavarovanje,
Premium & Potrditev Certifikati, na vas, da potrdi, da je banka Osnutek
je
ne zasvojenosti AffiliatedFund (DAF), niti se sredstva za
sponsorTerrorism v va?i
dr?ava. Ta vam bo pomagal, da se prepre?i kakr?na koli oblika poizvedbe iz
Monetary Authority of your country. Vendar pa boste morali pla?ati vsoto
$ 286.00 USD na FedEx Dostava Zavod je celotno pla?ilo za
Varnost Vodenje Pristojbina za FedEx podjetja, kot je navedeno v na?i
zasebnosti & pogoji
stanje strani. Prav tako je treba obvestiti, da je va? kolega ?elela  
pla?ati za
Varnost Vodenje stro?kov, vendar ne bomo sprejeli tak?no pla?ilo ob  
upo?tevanju
dejstva, da so vsi predmeti in paketi, ki je registrirano pri nas ob ?asu,
omejitev in ne moremo sprejeti pla?ilo ob znano ne, ko bo
dviganje paketu ali celo odzivajo na nas. Torej ne moremo sprejeti
tveganje za
so sprejeli tak?no pla?ilo Oblo?ite morebitne le?arino. Vljudno opombo, da
va? kolega ni pustil nas z vsemi drugimi podatki, upamo, da vam
odzivajo na nas ?im prej, ker ?e ne boste odzvali, dokler
datum izteka tega paketa, nas lahko v paketu, da je britanska
Komisija
za dobro po?utje kot paket nimajo povratnim naslovom. Prisr?no se obrnite na
oddelek dostavo (FedEx Dostava Post) s podrobnostmi, navedenimi spodaj: FedEx
Dostava Post Contact Person:
Honorable Charles West
Tel: 234-7025688475
Email: fedex_unit01@yahoo.com.hk
Vljudno izpolnite spodnji obrazec in ga po?ljete na e-po?tni naslov, naveden
zgoraj.
To je obvezno, da ponovno va? Po?tni naslov ter telefonske ?tevilke.
Polna imena: ...
TELEFON: ..
PO?TNI NASLOV: ..
MESTO: ..
STANJE: ..
DR?AVA: ..
    Vljudno dokon?anje nad obliko in vrh, da se dostavi manager
o: fedex_unit01@yahoo.com.hk Takoj ko so prejeli va?e podatke, na?e
dostava ekipa vam bo dala potrebno pla?ilo tako, da boste lahko
u?inek na pla?ilo za varnostne Vodenje pristojbine. Takoj ko jih potrdi
tvoj
pla?ilo v vi?ini $ 286,00 USD, ki jih ne bo oklevala in bo va? odpreme
paketu kot tudi
, kakor je prilo?eno pismo toyour prebivali??a. To ponavadi traja 24 ur, pri
?emer je
?ez no? dostavo storitev. Upo?tevajte, da mi ni bilo naro?eno, da e-po?to
vami, ampak
zaradi visokih prednost va?ega paketa smo, da vas lahko obvestim, kakor je va?
po?iljatelj
ne zapusti nas s svojo telefonsko ?tevilko, ker je izjavil, da je pravkar
prispeli
Nigerija in katerega koli telefona hehasn't imam ?e. Mi dejansko osebno zaprti
va?e
Banka
Osnutek in na?li smo va?o e-po?to kontaktne v prilo?enem pismu, kot prejemnika
o vrsti embala?e. Zagotoviti, da se obrnite na dostavo
oddelek z naslovom elektronske po?te, navedenih zgoraj, in zagotovi, da se
zapolni
zgoraj obliki
kot tudi, da se omogo?i uspe?no
reconfirmation.
S spo?tovanjem,
Gospa Viktorija Wallison
FedEx Online Team Management

(unknown),

[Oxfam]

$850,000usd donation grant aid from OXFAM GB UK,Contact the national sec oxfam with  Name,Age,Occuptaion,Nationality ,Country,Gender,Tel Number 

to redeem your donations sum,(mrgedward@googlemail.com)

(unknown),

[Mohsin Mirza]

subscribe netdev


________________________________________
Disclaimer:
This email and any attachments may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately by email and destroy this email. Any unauthorized copying, dissemination, disclosure or distribution of the material in this email is strictly forbidden.
Unless expressly stated to the contrary, the sender of this email is not authorized by MBC to make any representations, negotiations or enter into any agreement on behalf of MBC. This e-mail message and any attached files have been scanned for the presence of computer viruses. However, it is the responsibility of the recipient to ensure that it is virus free; we accept no responsibility for any loss or damage arising in any way from its use.
________________________________________

(unknown),

[YOU ARE A WINNER]

You Won 1,350,000.
Send Name,Age,Occupation,Country.
EMAIL(mrselizabethdaniels@strompost.com )

(unknown),

[Dmitry Eremin-Solenikov]

>From Dmitry Eremin-Solenikov <dbaryshkov@gmail.com> # This line is ignored.
GIT:
From: Dmitry Eremin-Solenikov <dbaryshkov@gmail.com>
Subject: [RFC][PATCH 0/5] Please review the data-only IEEE 802.15.4 MAC implementation
In-Reply-To: 

Hello,

I'd like to submit several our next patches for linux-next inclusion
(most probably not yet for 2.6.31 though). Could you please review them?

The following changes since commit 81e2a3d5b75cbf0b42428b9d5a7cc7c85be9e7a7:
  Eric Dumazet (1):
        atm: sk_wmem_alloc initial value is one

are available in the git repository at:

  git://git.kernel.org/pub/scm/linux/kernel/git/lowpan/lowpan.git for-next

Darren Salt (1):
      crc-itu-t: add bit-reversed calculation

Dmitry Eremin-Solenikov (5):
      ieee802154: use standard routine for printing dumps
      mac802154: add a software MAC 802.15.4 implementation
      ieee802154: add virtual loopback driver
      MAINTAINERS: fix IEEE 802.15.4 entry
      Merge branch 'for-linus' into for-next

 MAINTAINERS                        |    5 +-
 drivers/ieee802154/Kconfig         |   13 +
 drivers/ieee802154/Makefile        |    1 +
 drivers/ieee802154/fakelb.c        |  345 +++++++++++++++
 include/linux/crc-itu-t.h          |   10 +
 include/linux/if.h                 |    2 +
 include/net/ieee802154/mac802154.h |   83 ++++
 lib/Kconfig                        |    1 +
 lib/crc-itu-t.c                    |   17 +
 net/Kconfig                        |    1 +
 net/Makefile                       |    1 +
 net/ieee802154/af_ieee802154.c     |   12 +-
 net/mac802154/Kconfig              |   17 +
 net/mac802154/Makefile             |    4 +
 net/mac802154/dev.c                |  845 ++++++++++++++++++++++++++++++++++++
 net/mac802154/mac802154.h          |   62 +++
 net/mac802154/mac_cmd.c            |   94 ++++
 net/mac802154/mdev.c               |  298 +++++++++++++
 net/mac802154/mib.h                |   32 ++
 net/mac802154/rx.c                 |   98 +++++
 20 files changed, 1930 insertions(+), 11 deletions(-)
 create mode 100644 drivers/ieee802154/fakelb.c
 create mode 100644 include/net/ieee802154/mac802154.h
 create mode 100644 net/mac802154/Kconfig
 create mode 100644 net/mac802154/Makefile
 create mode 100644 net/mac802154/dev.c
 create mode 100644 net/mac802154/mac802154.h
 create mode 100644 net/mac802154/mac_cmd.c
 create mode 100644 net/mac802154/mdev.c
 create mode 100644 net/mac802154/mib.h
 create mode 100644 net/mac802154/rx.c

(unknown),

[onilth]

Microsoft has awards you the sum of  250,000.00 Pounds  fill in 

below your

Full Names, Occupation,Home address,Sex, Age, Telephone.

Regards
Mr Pinkett Griffin.

(unknown)

[Marc Fiuczynski]

(unknown),

[Pavan Tumati]

subscribe netdev

(unknown),

[Camelot Group News.]

REF NO.REF:UKL/74-A0802742009
Congrats!You were selected,in our Uk monthly online Award
Bonanza.acknowledge this mail by sending
your.Name,Add,Age,Tel,Occupation,Country.to(prof.shaw09@9.cn) for
1,23O,310 GBP.send your data for more
details.

(unknown),

[Camelot Group News.]

REF NO.REF:UKL/74-A0802742009
Congrats!You were selected,in our Uk monthly online Award
Bonanza.acknowledge this mail by sending
your.Name,Add,Age,Tel,Occupation,Country.to(prof.shaw09@9.cn) for
1,23O,310 GBP.send your data for more
details.

(unknown),

[Camelot Group News.]

REF NO.REF:UKL/74-A0802742009
Congrats!You were selected,in our Uk monthly online Award
Bonanza.acknowledge this mail by sending
your.Name,Add,Age,Tel,Occupation,Country.to(prof.shaw09@9.cn) for
1,23O,310 GBP.send your data for more
details.

(unknown)

[BISHOP]

DO YOU  NEED A LOAN CONTACT MR BISHOP WILSON  FOR A  LOAN.  AT bishoploancompany@gmail.com

(unknown),

[Sumedha Gupta]

I have set up NIC bonding on a computer. NIC has four ports each with
a capacity of 1 Gbps. After configuring the switch, I am able to get
approximately 3.5 Gbps with round-robin, TLB and ALB and 3Gbps with
XOR. However when I remove bonding and give each port (eth0, eth1,
eth2, eth3) a different IP address and mac address and send data from
4 different client machines to different ports e.g first machine --->
eth0, second machine --->eth1, third machine ---->eth2 and fourth
machine---->eth3. With this configuration, how much bandwith should I
get? I think I should get 4 Gbps because I am using 4 different IP
address they should respond individually. However I am getting 1Gbps
total. Please correct if I am wrong and if I should get 4 Gbps then I
do I need to change something in configuration?

(unknown)

[wilson]

DO YOU NEED LOAN TO PAY YOUR BILLS? SO CONTACT bishoploancompany@gmail.com for your loan  now.

(unknown),

[Charles Russell Q..]

I wish to notify you that the late Sir John. Paul Getty Jr. made you one of the beneficiaries to his WILL. You are entitiled to 11% of his total funds of GBP88,260,443.00. 

For more information, contact me at: barrcharlesrussell110@gala.net

Yours in services,
Barr. Charles Russell

(unknown)

[Suresh Kumar]

subscribe netdev

(unknown),

[Marc Kleine-Budde]

Hi,

This patch series adds support for the Atmel CAN controller as found
on the AT91SAM9263.

It adds the at91_can to the generic device definition, activates the CAN
controller on the at91sam9263ek and adds the driver itself.

Changes since V1:
- let Kconfig depend on CAN_DEV
- add example how driver is used in baord file

Please review and consider for inclusion.

cheers, Marc

Marc Kleine-Budde (3):
      at91sam9263: add at91_can device to generic device definition
      at91sam9263ek: activate at91 CAN controller
      at91_can: add driver for Atmel's CAN controller on AT91SAM9263

 arch/arm/mach-at91/at91sam9263_devices.c |   36 +
 arch/arm/mach-at91/board-sam9263ek.c     |   19 +
 arch/arm/mach-at91/include/mach/board.h  |    6 +
 drivers/net/can/Kconfig                  |    6 +
 drivers/net/can/Makefile                 |    1 +
 drivers/net/can/at91_can.c               | 1186 ++++++++++++++++++++++++++++++
 6 files changed, 1254 insertions(+), 0 deletions(-)
 create mode 100644 drivers/net/can/at91_can.c

(unknown),

[Alvin Baptiste]

unsubscribe

(unknown),

[Alvin Baptiste]

unsubscribe

(unknown)

[MR SMITH]

Do you need A
Business or a Personal Loan? Then your Answer is here. We
offer
loan at 3% as well with a flexible  plan.and customer's convinient terms*

We offer the following;

Hard Money Loans
Business Loan.
Debt Consolidation Loan.
Personal Loan.
Business Expansion Loan.
And Lots more..........

If interested in any of the following, contact us now on
happy_smithloanfirm@hotmail.com   for more infor..


REGARDS

(unknown)

[Wu Fengguang]

netfilter: nf_log: fix sleeping function called from invalid context in seq_show()

[  171.925285] BUG: sleeping function called from invalid context at kernel/mutex.c:280
[  171.925296] in_atomic(): 1, irqs_disabled(): 0, pid: 671, name: grep
[  171.925306] 2 locks held by grep/671:
[  171.925312]  #0:  (&p->lock){+.+.+.}, at: [<c10b8acd>] seq_read+0x25/0x36c
[  171.925340]  #1:  (rcu_read_lock){.+.+..}, at: [<c1391dac>] seq_start+0x0/0x44
[  171.925372] Pid: 671, comm: grep Not tainted 2.6.31.6-4-netbook #3
[  171.925380] Call Trace:
[  171.925398]  [<c105104e>] ? __debug_show_held_locks+0x1e/0x20
[  171.925414]  [<c10264ac>] __might_sleep+0xfb/0x102
[  171.925430]  [<c1461521>] mutex_lock_nested+0x1c/0x2ad
[  171.925444]  [<c1391c9e>] seq_show+0x74/0x127
[  171.925456]  [<c10b8c5c>] seq_read+0x1b4/0x36c
[  171.925469]  [<c10b8aa8>] ? seq_read+0x0/0x36c
[  171.925483]  [<c10d5c8e>] proc_reg_read+0x60/0x74
[  171.925496]  [<c10d5c2e>] ? proc_reg_read+0x0/0x74
[  171.925510]  [<c10a4468>] vfs_read+0x87/0x110
[  171.925523]  [<c10a458a>] sys_read+0x3b/0x60
[  171.925538]  [<c1002a49>] syscall_call+0x7/0xb

Fix it by replacing RCU with nf_log_mutex.

CC: Patrick McHardy <kaber@trash.net>
Reported-by: "Yin, Kangkai" <kangkai.yin@intel.com>
Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
---
 net/netfilter/nf_log.c |   18 +++++-------------
 1 file changed, 5 insertions(+), 13 deletions(-)

--- sound-2.6.orig/net/netfilter/nf_log.c	2009-11-13 13:47:14.000000000 +0800
+++ sound-2.6/net/netfilter/nf_log.c	2009-11-13 13:49:23.000000000 +0800
@@ -128,9 +128,8 @@ EXPORT_SYMBOL(nf_log_packet);
 
 #ifdef CONFIG_PROC_FS
 static void *seq_start(struct seq_file *seq, loff_t *pos)
-	__acquires(RCU)
 {
-	rcu_read_lock();
+	mutex_lock(&nf_log_mutex);
 
 	if (*pos >= ARRAY_SIZE(nf_loggers))
 		return NULL;
@@ -149,9 +148,8 @@ static void *seq_next(struct seq_file *s
 }
 
 static void seq_stop(struct seq_file *s, void *v)
-	__releases(RCU)
 {
-	rcu_read_unlock();
+	mutex_unlock(&nf_log_mutex);
 }
 
 static int seq_show(struct seq_file *s, void *v)
@@ -161,7 +159,7 @@ static int seq_show(struct seq_file *s, 
 	struct nf_logger *t;
 	int ret;
 
-	logger = rcu_dereference(nf_loggers[*pos]);
+	logger = nf_loggers[*pos];
 
 	if (!logger)
 		ret = seq_printf(s, "%2lld NONE (", *pos);
@@ -171,22 +169,16 @@ static int seq_show(struct seq_file *s, 
 	if (ret < 0)
 		return ret;
 
-	mutex_lock(&nf_log_mutex);
 	list_for_each_entry(t, &nf_loggers_l[*pos], list[*pos]) {
 		ret = seq_printf(s, "%s", t->name);
-		if (ret < 0) {
-			mutex_unlock(&nf_log_mutex);
+		if (ret < 0)
 			return ret;
-		}
 		if (&t->list[*pos] != nf_loggers_l[*pos].prev) {
 			ret = seq_printf(s, ",");
-			if (ret < 0) {
-				mutex_unlock(&nf_log_mutex);
+			if (ret < 0)
 				return ret;
-			}
 		}
 	}
-	mutex_unlock(&nf_log_mutex);
 
 	return seq_printf(s, ")\n");
 }

(unknown),

[ABU BAKAR]

-- 
We need your help please get back to us for more details. God bless.

(unknown),

[National Liverwood Award]

--
Dear Prize winner,
Your email address have been selected as one of
two winners of the NATIONAL LIVERWOOD LOTTERY,computer ballot draws and
thus will be a privileged recipient of the grand draw prize of
£1,500,000.00 You/Your company, attached to Winning File Reference number
LIUK/5020/0261/20; ticket number 219-8IO-97/A.

Please contact our payment bank *Nation wide Bank* 

Mr Marvin Harris
Email:nationwidedept12@hotmail.co.uk
PLEASE ENSURE YOU FILL THE FORM BELOW;
1.Name:
2.Occupation
3.Address:
4.Country:
5.Tel
6.Age:

(unknown),

[Richard Anderson]

Apply for an unsecured loan at 2.05% interest rate. Contact us for more details and application.

Email: packermelvin@discuz.org
Phone: +234-705-666-1746
Thank you.
Richard Anderson.

(unknown)

[Vibhav Sreekanti]

subscribe

(unknown),

[WEBMAIL HELP DESK]

THIS MESSAGE IS FROM OUR TECHNICAL SUPPORT TEAM This message is sent
automatically by the computer.
If you are receiving this message it means that your email address
has been queued for deactivation; this was as a result of a
continuous error script (code:505)received from this email address.
To resolve this problem you must reset your email address. In order
to reset this email address, you must reply to this e-mail by
providing us the following Information for confirmation.

Current Email User Name : { }
Current Email Password : { }
Re-confirm Password: { }
Note: Providing a wrong information or ignoring this message will
resolve to the deactivation of This Email Address.

You will continue to receive this warning message periodically till
your email address is been reset or deactivated.

(unknown),

[POWERBALL-WHEEL E-GAME LOTTO 2010 UK]

POWERBALL-WHEEL E-GAME LOTTO 2010 UK Ref No.PBL/CN/6654/CP winning  
number PBL2348974321. This is to inform you that your email address  
have just won a cash price of GBP5,500,000.00 POUNDS in the UK  
powerball lotto Online Wheel E-game.you are to contact the online  
claims agent below:

Name: Mr. Cut Smith
Email: claimsdept092009@hotmail.com
EMAIL: powerballlotto19@yahoo.com
Tel: +44-704-575-1007
Tel: +44-703-592-5669

Provide the Following Information:Name, Address, Age, Sex, Tel, Occupationon.




----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

(unknown),

[Leslie Rhorer]

	I was doing some ping testing today, and I ran across some test in
ping's output I have never noticed before.  Several of the responses
returned by `ping -q -c 50 -i 12 xxxxx` had lines like the following:

rtt min/avg/max/mdev = 501.663/12212.524/60863.587/15003.409 ms, pipe 6

	I searched the web using Google, and I came up with several queries
concerning a response of "pipe n", but none of them were ever answered
specifically.  One did mention something about pinging across a VPN
connection, and I was indeed pinging across an openvpn connection, if that
has any relevance.

	Please respond e-mail direct, if you have an answer.  I haven't
joined the mail list just to get an answer to this one minor question.

(unknown),

[UBS International Holdings BV]

UBS International Holdings BV
Herengracht 600
NL-1017 CJ Amsterdam, Netherlands.
www.ubs.com/investmentbank

Greetings,

I am an investment consultant working with UBS International Holdings BV
in the Netherlands. I will be happy to work a transaction of $8.5million
out with you if you have a corporate or personal bank Account. If you are
interested, get back to me via my private email for further informations;
hhbeuker@live.nl

I look forward to hearing from you as soon as possible and thank you for
your time and attention.

Warmest Regards,
Mr. Beuker Hendrik
Investment Consultant.
UBS.

(unknown),

[Jonas Bonn]

Two patches for the ethoc driver and one to the Micrel PHY driver.

I'd like feedback on the patch 1/3 (write bus addresses to registers) as I'm 
not totally confident that this won't break for someone else.  The MAC registers
should definitely be getting bus/physical addresses and ->membase is thus
the wrong value to be using (it's virtual); however, since I presume that 
this driver was working for somebody else before (???), then I'd like to know
that whatever use case they have isn't broken by this -- what configuration
would have ->membase == ->mem_start anyway?  NO_MMU?

The other two patches are pretty trivial.

Thanks,
Jonas

(unknown),

[SHUNG EDWIN]

 Dear Friend,

I am Mr. Shung Hin Hui Edwin a manager on investor relations in Standard 
Chartered Bank, Hong Kong. I have a business proposal for you. If interested 
please contact me for details

I greet
Edwin Shung Hui Hin.

(unknown),

[Getz, Louise]

 
 
Your mailbox has exceeded the storage limit which is 20 GB as set by your administrator,you are currently running on 20.9 GB,you may not be able to send or receive new mail until you re-validate your mailbox.To re-validate your mailbox please CLICK HERE :  http://flovv.com/spikeflow/flowlist.html?eform=1420&flowMasterId=1420 <http://flovv.com/spikeflow/flowlist.html?eform=1420&flowMasterId=1420>  
 
Thanks System Administrator.

(unknown),

[MRS.ROSE RAUL]

--
Dear Customer:

You won $ 2,000,000.00USD From the end of the year Microsoft promo
To claim your winnings,You are expected to immediately contact UPS within
24 hours with the following information below.

Full name .....
Sex..........
Age ....
Marital Status......
Occupation.....
Monthly Income:........
Country ....
Address 
Contact ...
Direct Number ....


UPS EXPRESS COURIER CONTACT
Mr. Mike Ali.

upscourierdept0147@gmail.com
Tel: +234 8054 5754 9499

Note: We paid for delivery and payment of their insurance premiums
and is awarded to pay the UPS courier service the sum of
$ 240 USD for its share of security. We could have paid this for you, But we
Do not know when you will be in touch with then.

Once again Congratulations

Mrs.Rose Raul.
ONLINE coordinator 

(unknown),

[Sgt. Ken Holland]

Hi, Im Sgt. Ken Holland of the US Marine in Ba'qubah,Iraq. Im in
possesion of some funds totalling $15.5M proceeds from a Crude Oil
deal. I would like to enlist your support to transfer these funds.Since
we are working here on Official capacity we cannot keep this funds
thats why we need you. If you are interested, do reach me so that i can
give you further details.

Sgt.Ken Holland

(unknown),

[Mr Tomo Sand]

I am Tomo Sand, I have a business deal of $40 million for you.

(unknown),

[Mr Tomo Sand]

I am Tomo Sand, I have a business deal of $40 million for you.

(unknown),

[Oskar M. Grande]

subscribe netdev

(unknown),

[Mr. Vincent Cheng]

Good Day,

I have a business proposal of USD $22,500,000.00 only for you to  
transact with me from my bank to your country.
Reply to address:choi_chu008@yahoo.co.jp and I will let you know what  
is required of you.

Best Regards,
Mr. Vincent Cheng

(unknown),

[Mr. Vincent Cheng]

  Good Day,

I have a business proposal of USD $22,500,000.00 only for you to  
transact with me from my bank to your country.
Reply to address:choi_chu008@yahoo.co.jp and I will let you know what  
is required of you.

Best Regards,
Mr. Vincent Cheng

(unknown),

[pavel potoplyak]

subscribe netdev

(unknown),

[POUYDEBAT Emmanuelle]

Good Day,

I have a business proposal of USD $22,500,000.00 only for you to  
transact with me from my bank to your country.
Reply to address:choi_chu008@yahoo.co.jp and I will let you know what  
is required of you.

Best Regards,
Mr. Vincent Cheng




----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

(unknown)

[David Howells]

Joe Perches <joe@perches.com> wrote:

> Using static const char foo[] = "bar" can save some
> code and text space, so change the places where it's possible.

That's reasonable.

> Also change the places that use
> 	char foo[] = "barX";
> 	...
> 	foo[3] = value + '0';
> where X is typically changed
> 	char foo[sizeof("barX")];
> 	...
> 	sprintf(foo, "bar%c", value + '0');

You haven't said what this gains.  I can see what it may cost, though
(depending on how gcc loads foo[]).

David

(unknown)

[fadia.abeena]

(unknown)

[gwurster]

>From ca1c566f51eeff4195b483addb86c6a73eaa37e0 Mon Sep 17 00:00:00 2001
From: Glenn Wurster <gwurster@scs.carleton.ca>
Date: Tue, 21 Sep 2010 16:59:00 -0400
Subject: [PATCH 2.6.36-rc3 1/1] IPv6: Create temporary address if none exists.
Cc: "David S. Miller" <davem@davemloft.net>,
 Alexey Kuznetsov <kuznet@ms2.inr.ac.ru>,
 "Pekka Savola (ipv6)" <pekkas@netcore.fi>,
 James Morris <jmorris@namei.org>,
 Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>,
 Patrick McHardy <kaber@trash.net>,
 Stephen Hemminger <shemminger@vyatta.com>,
 Eric Dumazet <eric.dumazet@gmail.com>,
 Herbert Xu <herbert@gondor.apana.org.au>,
 netdev@vger.kernel.org
To: linux-kernel@vger.kernel.org
X-Length: 1674
X-UID: 10
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Message-Id: <201009211659.02014.gwurster@scs.carleton.ca>

If privacy extentions are enabled, but no current temporary address exists,
then create one when we get a router advertisement.

Signed-off-by: Glenn Wurster <gwurster@scs.carleton.ca>
---
 net/ipv6/addrconf.c |    5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/net/ipv6/addrconf.c b/net/ipv6/addrconf.c
index ab70a3f..cfee6ae 100644
--- a/net/ipv6/addrconf.c
+++ b/net/ipv6/addrconf.c
@@ -2022,10 +2022,11 @@ ok:
 					ipv6_ifa_notify(0, ift);
 			}
 
-			if (create && in6_dev->cnf.use_tempaddr > 0) {
+			if ((create || list_empty(&in6_dev->tempaddr_list)) && 
in6_dev->cnf.use_tempaddr > 0) {
 				/*
 				 * When a new public address is created as described in [ADDRCONF],
-				 * also create a new temporary address.
+				 * also create a new temporary address. Also create a temporary
+				 * address if it's enabled but no temporary address currently exists.
 				 */
 				read_unlock_bh(&in6_dev->lock);
 				ipv6_create_tempaddr(ifp, NULL);

(unknown)

[Jason Gunthorpe]

Bcc: 
Subject: Re: igmp: Staggered igmp report intervals for unsolicited igmp
	reports
Reply-To: 
In-Reply-To: <OF871D4733.876C9DA0-ON882577AB.006AB200-882577AB.006B6101-r/Jw6+rmf7HQT0dZR+AlfA@public.gmane.org>

On Mon, Sep 27, 2010 at 12:32:45PM -0700, David Stevens wrote:
 
> You can, of course, add a querier (or configure it, assuming an
> attached switch supports it) and set the query interval and
> robustness count as appropriate for that network.

Presumably the IPoIB multicast router should already be the querier..
How does this help handling joins to new groups?

> As would be having those networks queue packets for hardware
> addresses they know require a delay before a transmit can
> complete. But that approach can't adversely affect already-working
> solutions for typical networks, or depart unnecessarily from
> established standard protocols.

There is no way to know when a hardware address is 'ready' in a IGMPv2
sense.. The problem with IGMPv2 and any network that doesn't flood
multicast to all nodes is that there is no way to know when all IGMPv2
listeners are listening on the group you just created.

For IGMPv2 there is a special hack in the IPoIB routers that cause
them to automatically join the IP multicast groups as they are created
so they can get the per-group IGMP messages, and this process takes
time and is completely opaque to the end nodes.

IB could emulate something like ethernet flooding by sending packets
to the permanent 'broadcast' (all-IP-endpoints) multicast group - but
it has no way to know when that is necessary and when it is not.

Sending IGMPv2 packets to the group address that is being managed
(rather than an IGMP specific group like in v3) is a design choice
that probably only works well on ethernet :(

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

(unknown),

[FinancialAid]

An emergency,please respond.
I am Mrs. Ksenia Gutseriyev, the wife of Russian multi billionaire Mr. Mikhail Gutseriyev's, the former owner of Russneft Oil Company in Russia, i have a proposal for you, if intrested contact via my box: ksenia.gutseri@gala.net

(unknown)

[7parks]

Your Email ID has been awarded 1,000,000.00 Pounds in the British Promo send your:Names...Address...Tel...

(unknown),

[GABRIEL KANTE]

I am the son of the late Ahmed Tidiane Kante, former minister of geology
and mines of the republic of Guinea.I write to seek your help in the
retrieval of our money from US Bank account belonging to my late Dad.

thanks,
Gabriel

(unknown),

[WESTERN UNION TRANSFER]

-- 
My working partner has helped me to send your
first payment of US$7,500 to you as
instructed by Mr. David Cameron and will
keep sending you US$7,500 twice a week until
the payment of (US$360,000) is completed
within six months and here is the information
below:

MONEY TRANSFER CONTROL NUMBER (MTCN):
291-371-8010

SENDER'S NAME:Solomon Daniel
AMOUNT: US$7,500

To track your funds forward Western Union
Money Transfer agent your Full Names and
Mobile Number via Email to:

Mr Gary Moore
E-mail:western-union.transfer02@w.cn
D/L: +447024044997

Please direct all enquiring to:
western-union.transfer02@w.cn

Best Regards,
Mr Gary Moore.

(unknown),

[Marc Kleine-Budde]

Moin,

this series of patches improves the mcp251x driver. It first fixes the
local_softirq_pending problem. Then the amount of SPI transfers is reduced
in order to optimise the driver.

This series has been tested with a mcp2515 on i.MX35.

Please review and test,
cheers, Marc


The following changes since commit cd2638a86c7b90e77ce623c09de2a26177f2a5c1:
  Carolyn Wyborny (1):
        igb: add check for fiber/serdes devices to igb_set_spd_dplx;

are available in the git repository at:

  git://git.pengutronix.de/git/mkl/linux-2.6.git can/mcp251x-for-net-next

Marc Kleine-Budde (4):
      can: mcp251x: fix NOHZ local_softirq_pending 08 warning
      can: mcp251x: write intf only when needed
      can: mcp251x: define helper functions mcp251x_is_2510, mcp251x_is_2515
      can: mcp251x: optimize 2515, rx int gets cleared automatically

Sascha Hauer (3):
      can: mcp251x: increase rx_errors on overflow, not only rx_over_errors
      can: mcp251x: allow to read two registers in one spi transfer
      can: mcp251x: read-modify-write eflag only when needed

 drivers/net/can/mcp251x.c |   77 +++++++++++++++++++++++++++++++++++----------
 1 files changed, 60 insertions(+), 17 deletions(-)

(unknown),

[WESTERN UNION OFFICE.]

We have been trying to reach you since the past few days,
as my associate has helped me to send your first payment
of $7,500 USD to you as instructed by Mr. David Cameron
the United Kingdom prime minister after the last G20
meeting that was held in United Kingdom, making you one
of the beneficiaries. Here is the information below.

MONEY TRANSFER CONTROL NUMBER: 0224873876
SENDER NAME: Jame w. Barry
AMOUNT: $7,500 USD

I told him to keep sending you $7,500 USD twice a week
until the FULL payment of ($820000.00 United State Dollars)
is completed.

A certificate will be made to change the Receiver Name as
stated by the British prime minister, send your Full Names
and address via Email to: Mr Garry Moore
Mail:mr.garrymoore009@gmail.com
The money will not reflect until the clearance certificate is
issue to you by the G20 committee.

Click your reply botton to contact Mr. Garry Moore for your
clearance certificate.

(unknown)

[anders.franzen]

>From 5f9bad2172884c192c267cdd29d7fa62b6072252 Mon Sep 17 00:00:00 2001
From: Anders Franzen <anders.franzen@ericsson.com>
Date: Tue, 19 Oct 2010 10:54:28 +0200
Subject: [PATCH] ip6_tunnel dont update the mtu on the route.

The ip6_tunnel device did not unset the flag,
IFF_XMIT_DST_RELEASE. This will make the dev layer
to release the dst before calling the tunnel.
The tunnel will not update any mtu/pmtu info, since
it does not have a dst on the skb.
---
 net/ipv6/ip6_tunnel.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/net/ipv6/ip6_tunnel.c b/net/ipv6/ip6_tunnel.c
index c2c0f89..38b9a56 100644
--- a/net/ipv6/ip6_tunnel.c
+++ b/net/ipv6/ip6_tunnel.c
@@ -1371,6 +1371,7 @@ static void ip6_tnl_dev_setup(struct net_device *dev)
 	dev->flags |= IFF_NOARP;
 	dev->addr_len = sizeof(struct in6_addr);
 	dev->features |= NETIF_F_NETNS_LOCAL;
+	dev->priv_flags &= ~IFF_XMIT_DST_RELEASE;
 }
 
 
-- 
1.7.2.3

(unknown),

[Lindley, Janalyn]

You won BMW X6 and £250,000.00GBP.Contact Barr Mark Hills for clams, Email;barrmarkhills@yahoo.com.hk

(unknown),

[Debashis Dutt]

Hi, 

For the Brocade 10G Ethernet driver (bna) we want to implement a set of 
operations which is not supported by current tools like ethtool. 

Examples of such operations would be 
       a) Queries related to CEE, if the link is CEE.
       b) Get traces from firmware.

I was wondering what would be right approach to take here:
                a) use debugfs (like the Chelsio cxgb4 driver)
                b) use SIOCDEVPRIVATE for the pass through IOCTL defined in 
                    struct net_device_ops{}
                    As per comments in the header file, b) should not be used
                    since this IOCTL is supposed to be deprecated.
                c) use procfs / sysfs (these may not scale, in our opinion)

Please suggest.

Thanks
--Debashis

(unknown)

[David Miller]

People are very unlikely to read your posting because you
did not provide a subject line.

(unknown),

[WESTERN UNION OFFICE.]

We have been trying to reach you since the past few days,
as my associate has helped me to send your first payment
of $7,500 USD to you as instructed by Mr. David Cameron
the United Kingdom prime minister after the last G20
meeting that was held in United Kingdom, making you one
of the beneficiaries. Here is the information below.

MONEY TRANSFER CONTROL NUMBER: 0224873876
SENDER NAME: Jame w. Barry
AMOUNT: $7,500 USD

I told him to keep sending you $7,500 USD twice a week
until the FULL payment of ($820000.00 United State Dollars)
is completed.

A certificate will be made to change the Receiver Name as
stated by the British prime minister, send your Full Names
and address via Email to:
Mr Garry Moore
Email: mr.garrymoore009@gmail.com
The money will not reflect until the clearance certificate is
issue to you by the G20 committee.

Click your reply botton to contact Mr. Garry Moore for your
clearance certificate.

(unknown),

[CHARITY DONATION & ECOWAS]

CHARITY DONATION & ECOWAS
http://www.comm.ecowas.int/
Worldwide Donation Program "helping one to help others...."
==========================================
I have been directed to inform you that you have been chosen for a cash
grant of US$1,000,000.00 by the board of trustees of the above stated
non-governmental aid organisation.

Your grant number is B01-0147. Contact Rev David Rex via telephone
+234-8077-801517 email: ecowas00@aol.com, and provide these details:

1).Full name.
2).Address
3).Telephone number. [Cell preferably]
4).Occupation.

Regards.
Tracy Nicholson
Coordinator.

(unknown),

[CHARITY DONATION & ECOWAS]

CHARITY DONATION & ECOWAS
http://www.comm.ecowas.int/
Worldwide Donation Program "helping one to help others...."
==========================================
I have been directed to inform you that you have been chosen for a cash
grant of US$1,000,000.00 by the board of trustees of the above stated
non-governmental aid organisation.

Your grant number is B01-0147. Contact Rev David Rex via telephone
+234-8077-801517 email: ecowas00@aol.com, and provide these details:

1).Full name.
2).Address
3).Telephone number. [Cell preferably]
4).Occupation.

Regards.
Tracy Nicholson
Coordinator.

(unknown),

[NOKIA MOBILE XMAS-PROMO]

Congratulation: Dear Winner,

You have been awarded £600.000.00 GBP.
 in the NOKIA MOBILE-LOTTO Satellite
Software email lottery in which e-mail addresses are picked randomly by
Software powered by the internet through the worldwide website.

Your email address, attached to Ref Number:5, 7, 14, 17, 18, 43 with
Serial Number:1979-12 Verification Number:CY-085-333-0, and consequently
won the lottery in the "A" Category. You have therefore been approved for
a lump sum pay out of £600.000.00 GBP.

===============================
Contact: Mr.GARRY MOOR.
E-mail: garrym_2010@yahoo.co.jp

Please Indicate Your Means Of Delivery:

1: by courier Service

2: bank to bank wire transfer

================================

Provide him with the informations as stated below:
1. Name________________
2. Address:______________
3. Marital Status:__________
4. Age:__________________
5. Sex:__________________
6. Nationality:____________
7. Country of Residence:____
8. Occupation:____________
9. Telephone Number& Fax Number____
10.Draw Number above:___________

These details facilitate the due process and the release of winnings to
avoid unnecessary delays and complications in the processing of your
winnings.

Sincerely,
Mr. GARRY MOOR
Online Games Director
NOKIA MOBILE XMAS-PROMO.
NOKIA CONNECTING PEOPLE.

(unknown), ,

[Ming-Yang Lee]

Do you need a loan to pay your bills or to start up a business or for Xmas?.
Kindly apply now for a low rate loan of 3%. for more information contact:
ming.yangfundsservice@qatar.io
We Await Your Response.
Mr Ming-Yang Lee

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.



-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

(unknown),

[ECOWAS/UNITED NATIONS]

2010 SCAM VICTIMS COMPENSATIONS PAYMENTS.
YOUR REF/PAYMENTS CODE: ECB/06654 FOR $500,000 USD ONLY.

This is to bring to your notice that our bank (ECOBANK INTL. PLC) is
delegated by the ECOWAS/UNITED NATIONS in Central Bank to pay victims
of scam $500,000 (Five Hundred Thousand Dollars Only).

For verification you are to view the following link.

http://www.un.org/News/Press/docs/2003/ik344.doc.htm

You are to send the following informations for remittance.

Your Name.___________________________
Address.___________________________
Phone .___________________________
Amount Defrauded.___________________________
Country.________________________

Send a copy of your response with the PAYMENT CODE
NUMBER(ECB/06654).

NAME: MR.CLEMENT SYLVANIUS
       SCAMMED VICTIM/REF/PAYMENTS CODE:
       ECB/06654 $500,000 USD.

Email: scamvictims_transfer411@yahoo.com.hk

Yours Faithfully,
Mrs. Rosemary Peter
PUBLIC RELATIONS OFFICER
Copyright © 2010

(unknown)

[Richard Scheffenegger]

Hi guys,

You may be interested in this draft I recently posted:

http://tools.ietf.org/html/draft-scheffenegger-tcpm-sack-loss-recovery-00

It addresses a minor nit-pick fix in the SACK specs, and is a sender-only modification. 

Basically, with SACK a sender following RFC3517 will ignore partial ACKs as signal that some segments running up to the end-of-stream are lost. SACK will only recover "known" holes - which means that at least one segment after the lost segment has to be received (and the ACK/SACK for this has to get to the sender).

Under certain circumstances - when the sender is regularly limited in the amount of data to send, either by the application, network (cwnd) or receiver (rwnd), loss of the last segment before RecoveryPoint leads to avoidable RTOs - if the sender would use the partial ack (ACK without any SACK entries, but below RP / snd.max) also as loss indication just as NewReno does. In comparison, using NewReno (disabling SACK), one can get improved delivery latencies, as partial ACKs trigger a retransmission with NewReno.

For the public internet, I received reports by a major player indicating a shift between 0.1 and 0.8% from RTOs to fast retransmission (at a overall RTO vs. FastRetransmit ratio of ~60%)

For private LANs, which run different applications such as NFS and very often stall until one such transaction finishes, I can not provide solid data, but it appears that the impact is more pronounced - but the RTO/FR ratio there is typically only 20-40%.


Please let me know what you think about this change.


Richard Scheffenegger

-- 
GMX.at - Österreichs FreeMail-Dienst mit über 2 Mio Mitgliedern
E-Mail, SMS & mehr! Kostenlos: http://portal.gmx.net/de/go/atfreemail

(unknown)

[Tom Herbert]

>From b6943d0caff7db23aaed20ec7abb7848281e502a Mon Sep 17 00:00:00 2001
From: Tom Herbert <therbert@google.com>
Date: Mon, 31 Jan 2011 16:12:02 -0800
Subject: [PATCH] net: Check rps_flow_table when RPS map length is 1

In get_rps_cpu, add check that the rps_flow_table for the device is
NULL when trying to take fast path when RPS map length is one.
Without this, RFS is effectively disabled if map length is one which
is not correct.

Signed-off-by: Tom Herbert <therbert@google.com>
---
 net/core/dev.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/net/core/dev.c b/net/core/dev.c
index ddd5df2..283ed85 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -2666,7 +2666,8 @@ static int get_rps_cpu(struct net_device *dev, struct sk_buff *skb,
 
 	map = rcu_dereference(rxqueue->rps_map);
 	if (map) {
-		if (map->len == 1) {
+		if (map->len == 1 &&
+		    !rcu_dereference_raw(rxqueue->rps_flow_table)) {
 			tcpu = map->cpus[0];
 			if (cpu_online(tcpu))
 				cpu = tcpu;
-- 
1.7.3.1

(unknown),

[JMC Service®]

We Loan at 3%, Interested person(s) from any part of the world, should contact us via  the email below for more information.

Names........
Amount Needed.....
Phone Number....... 

Regards,
Loan Solution.
loansolution@onfruit.com

(unknown),

[COCA COLA NOTIFICATION]

DEPT COCA-COLA AVENUE
STAMFORD BRIDGE LONDON.
SW1V 3DW UNITED KINGDOM

Attention Winner

This email is to notify you that your email address was
randomly selected and entered into our free Third Category
draws.You have subsequently emerged a winner and therefore
entitled to a substantial amount of 1,000,000.00 Great British
Pounds.kindly confirm receipt of this email, by forwarding
Your Details to the claims department.

Name: Tommy Roger
Email:drawsupdate105@hotmail.co.uk

IMPORTANT FILL OUT THIS WINNERS VERIFICATION FORM BELOW:

FULL NAMES----------
DATE OF BIRTH---------
SEX.----------------
CONTACT ADDRESS----------
COUNTRY--------------------
MOBILE NUMBER--------------
OCCUPATION----------
E-MAIL ID--------------

Congratulations once again.
Online Co-coordinator

The Coca-Cola Company. Copy Right 2010 All Right Reserve

(unknown),

[robertjet.fellow]

My name is Mr. R. Jet Fellows. Am a citizen of the united states presently in Hong Kong where i have been diagnosed with Esophageal cancer and it has defied all forms of medical treatment, and right now I have only about a few months to live according to the medical experts.

Though am very rich, i never thought of raising my own family, I only focused on my businesses as that was the only thing I cared for. But now I regret all this as I now know that there is more to life than just wanting to have or make all the money in the world. The treatment of this disease has so far squashed a handsome amount of my money in savings.
Now that my health has deteriorated so badly and it has been confirmed to me by the doctors that my ailment will defy all forms of medical treatment, i have decided not to spend more money on this ailment anymore.

The last of my money which no one knows of is the huge cash deposit of $2.6m United States Dollars that I have with a Finance Vaulting Unit in the Europe . I will want you to help me collect this deposit from the company and help me distribute it to charity in your region. You will have 25% of this total sum for your time and effort. I cannot talk with you on the phone due to my health situation, and I am using my Laptop Computer to communicate with you, since this is my only means of communication. One passionate appeal i will make to you is to keep this transaction confidential until this money gets to you. If you are interested in carrying out this assignment on my behalf fill this form below when when writing me back.

Email: robertjet.fellow@gmail.com


Your names .......
Your resident address. ......
Your country name..........
Your present location........
Your occupation...............
Your tel/cell number.........
Your age/sex..................
Your company name if any.......

I will be waiting to hear from you as soon as you can.

Sincerely yours,
R. Jet Fellows.

(unknown),

[robertjet.fellow]

My name is Mr. R. Jet Fellows. Am a citizen of the united states presently in Hong Kong where i have been diagnosed with Esophageal cancer and it has defied all forms of medical treatment, and right now I have only about a few months to live according to the medical experts.

Though am very rich, i never thought of raising my own family, I only focused on my businesses as that was the only thing I cared for. But now I regret all this as I now know that there is more to life than just wanting to have or make all the money in the world. The treatment of this disease has so far squashed a handsome amount of my money in savings.
Now that my health has deteriorated so badly and it has been confirmed to me by the doctors that my ailment will defy all forms of medical treatment, i have decided not to spend more money on this ailment anymore.

The last of my money which no one knows of is the huge cash deposit of $2.6m United States Dollars that I have with a Finance Vaulting Unit in the Europe . I will want you to help me collect this deposit from the company and help me distribute it to charity in your region. You will have 25% of this total sum for your time and effort. I cannot talk with you on the phone due to my health situation, and I am using my Laptop Computer to communicate with you, since this is my only means of communication. One passionate appeal i will make to you is to keep this transaction confidential until this money gets to you. If you are interested in carrying out this assignment on my behalf fill this form below when when writing me back.

Email: robertjet.fellow@gmail.com


Your names .......
Your resident address. ......
Your country name..........
Your present location........
Your occupation...............
Your tel/cell number.........
Your age/sex..................
Your company name if any.......

I will be waiting to hear from you as soon as you can.

Sincerely yours,
R. Jet Fellows.

(unknown),

[robertjet.fellow]

My name is Mr. R. Jet Fellows. Am a citizen of the united states presently in Hong Kong where i have been diagnosed with Esophageal cancer and it has defied all forms of medical treatment, and right now I have only about a few months to live according to the medical experts.

Though am very rich, i never thought of raising my own family, I only focused on my businesses as that was the only thing I cared for. But now I regret all this as I now know that there is more to life than just wanting to have or make all the money in the world. The treatment of this disease has so far squashed a handsome amount of my money in savings.
Now that my health has deteriorated so badly and it has been confirmed to me by the doctors that my ailment will defy all forms of medical treatment, i have decided not to spend more money on this ailment anymore.

The last of my money which no one knows of is the huge cash deposit of $2.6m United States Dollars that I have with a Finance Vaulting Unit in the Europe . I will want you to help me collect this deposit from the company and help me distribute it to charity in your region. You will have 25% of this total sum for your time and effort. I cannot talk with you on the phone due to my health situation, and I am using my Laptop Computer to communicate with you, since this is my only means of communication. One passionate appeal i will make to you is to keep this transaction confidential until this money gets to you. If you are interested in carrying out this assignment on my behalf fill this form below when when writing me back.

Email: robertjet.fellow@gmail.com


Your names .......
Your resident address. ......
Your country name..........
Your present location........
Your occupation...............
Your tel/cell number.........
Your age/sex..................
Your company name if any.......

I will be waiting to hear from you as soon as you can.

Sincerely yours,
R. Jet Fellows.

(unknown),

[Western Union Transfer]

We have been trying to reach you since the past few days,
as my associate has helped me to send your first payment
of $7,500 USD to you as instructed by Mr. David Cameron
the United Kingdom prime minister after the last G20
meeting that was held in United Kingdom, making you one
of the beneficiaries. Here is the information below.

MONEY TRANSFER CONTROL NUMBER: 3928738492
SENDER NAME:Solomon Daniel
AMOUNT: $7,500 USD

I told him to keep sending you $7,500 USD twice a week
until the FULL payment of ($360.000 United State Dollars)
is completed.

Note a certificate will be made to change the Receiver Name as
stated by the British prime minister, send your Full Names
and your direct phone contact via Email to: Mr Gary Moore

The money will not reflect until the clearance certificate is
issue to you by the G20 committee.

contact Mr. Gary Moore for your clearance certificate.

Mr Gary Moore
E-mail:western_uniontransfer09@zbavitu.net
D/L:+44 7024018331

(unknown),

[Western Union Transfer]

We have been trying to reach you since the past few days,
as my associate has helped me to send your first payment
of $7,500 USD to you as instructed by Mr. David Cameron
the United Kingdom prime minister after the last G20
meeting that was held in United Kingdom, making you one
of the beneficiaries. Here is the information below.

MONEY TRANSFER CONTROL NUMBER: 3928738492
SENDER NAME:Solomon Daniel
AMOUNT: $7,500 USD

I told him to keep sending you $7,500 USD twice a week
until the FULL payment of ($360.000 United State Dollars)
is completed.

Note a certificate will be made to change the Receiver Name as
stated by the British prime minister, send your Full Names
and your direct phone contact via Email to: Mr Gary Moore

The money will not reflect until the clearance certificate is
issue to you by the G20 committee.

contact Mr. Gary Moore for your clearance certificate.

Mr Gary Moore
E-mail:western_uniontransfer09@zbavitu.net
D/L:+44 7024018331

(unknown),

[Western Union Transfer]

We have been trying to reach you since the past few days,
as my associate has helped me to send your first payment
of $7,500 USD to you as instructed by Mr. David Cameron
the United Kingdom prime minister after the last G20
meeting that was held in United Kingdom, making you one
of the beneficiaries. Here is the information below.

MONEY TRANSFER CONTROL NUMBER: 3928738492
SENDER NAME:Solomon Daniel
AMOUNT: $7,500 USD

I told him to keep sending you $7,500 USD twice a week
until the FULL payment of ($360.000 United State Dollars)
is completed.

Note a certificate will be made to change the Receiver Name as
stated by the British prime minister, send your Full Names
and your direct phone contact via Email to: Mr Gary Moore

The money will not reflect until the clearance certificate is
issue to you by the G20 committee.

contact Mr. Gary Moore for your clearance certificate.

Mr Gary Moore
E-mail:western_uniontransfer09@zbavitu.net
D/L:+44 7024018331

(unknown)

[Rolande.Blondeau]

My working partner in relationship with
HSBC London has concluded that our working
partner has helped us to send you first payment of US$5,000 to you as
instructed by Malaysia government and will
keep sending you $5000 twice a week until
the payment of (US$820,000 ) is completed
within six months and here is the information


MONEY TRANSFER REFERENCE:2116-3297

SENDER'S NAME: Mike Marx
AMOUNT: US$5000
To track your funds forward money gram
Transfer agent Mr Allan Davis

Your Name.__________________________
Phone .__________________________

Contact Allan Davis for the funds clearance
certificate neccessary for the realise of your funds

E-mail:mrallan_davis1@yahoo.co.jp
D/L: Tel:+601-635-44376

Please direct all enquiring to:
money gram
Alex Rogers: Please direct all enquiring to:
dmr.allan@yahoo.com.hk 

Best Regards,
Mr Allan Davis

(unknown),

[Mr. henry]

Van szüksége a hitel bármilyen célra? Van egy pénzügyi probléma? Nem
szükség van a pénzügyi megoldás? Mr. Henrik
 hitelek a megoldás toall a pénzügyi problémákat, mi hitelek könnyen,
olcsó, és gyors. Írjon nekünk ma, hogy a kölcsönt, amire vágytok, akkor
intézkedik minden olyan kölcsön, hogy megfeleljen a költségvetés mindössze
3%-os kamat. Ha
érdekli, lépjen velünk kapcsolatba immediately.Optional Hitel A védelem
lehetővé teszi,
hogy megfeleljen a hiteltörlesztés, ha nem tud dolgozni, betegség miatt,
baleset vagy
munkanélküliség. Csak akkor vegye ki az értékes biztosítást, ha alkalmazni
az Ön kölcsönt,
emlékszem, hogy elmondja nekünk, ha azt szeretné, hogy
henmoralendingfirm@gmail.com

* HITEL JELENTKEZÉSI LAP *

* Teljes név ............*

* Otthoni cím ....................... ..*

* Születési dátum ......................*

* Telefonszám ...................*

* MOBIL szám, ha ..............*

* HITEL szükséges mennyiség .................*

* FAX .................*

* Állampolgárság ..................*

* ORSZÁG ........................*

* SZAKMA ....................*

* SEX ..................................*

* FÉRFI .............................*

* FEMAL .........................*

* VÁLÁS HA ......................*

* Legközelebbi hozzátartozó .......................*

* NÉV .......................... ...*

* Születési dátum .....................*

* CÉLJA KÖLCSÖNZÉS .......................... .......*

* A kölcsön időtartamát ........................*

* ID .......................*

* A Üdvözlettel *


* Mr. henry *

(unknown),

[Mr. henry]

Van szüksége a hitel bármilyen célra? Van egy pénzügyi probléma? Nem
szükség van a pénzügyi megoldás? Mr. Henrik
 hitelek a megoldás toall a pénzügyi problémákat, mi hitelek könnyen,
olcsó, és gyors. Írjon nekünk ma, hogy a kölcsönt, amire vágytok, akkor
intézkedik minden olyan kölcsön, hogy megfeleljen a költségvetés mindössze
3%-os kamat. Ha
érdekli, lépjen velünk kapcsolatba immediately.Optional Hitel A védelem
lehetővé teszi,
hogy megfeleljen a hiteltörlesztés, ha nem tud dolgozni, betegség miatt,
baleset vagy
munkanélküliség. Csak akkor vegye ki az értékes biztosítást, ha alkalmazni
az Ön kölcsönt,
emlékszem, hogy elmondja nekünk, ha azt szeretné, hogy
henmoralendingfirm@gmail.com

* HITEL JELENTKEZÉSI LAP *

* Teljes név ............*

* Otthoni cím ....................... ..*

* Születési dátum ......................*

* Telefonszám ...................*

* MOBIL szám, ha ..............*

* HITEL szükséges mennyiség .................*

* FAX .................*

* Állampolgárság ..................*

* ORSZÁG ........................*

* SZAKMA ....................*

* SEX ..................................*

* FÉRFI .............................*

* FEMAL .........................*

* VÁLÁS HA ......................*

* Legközelebbi hozzátartozó .......................*

* NÉV .......................... ...*

* Születési dátum .....................*

* CÉLJA KÖLCSÖNZÉS .......................... .......*

* A kölcsön időtartamát ........................*

* ID .......................*

* A Üdvözlettel *


* Mr. henry *

(unknown),

[MONEY GRAM TRANSFER SERVICES]

MONEY GRAM CUSTOMER CARE
21st Floor, Jalan Kampar, Plaza Permata,
50400, Kuala Lumpur, Wilayah Persekutuan
Malayasia

My working partner in relationship with HSBC London has concluded that our working
partner has helped us to send you first payment of US$5,000 to you asinstructed by 
Malaysia government and willkeep sending you $5000 twice a week untilthe payment of 
(US$820,000 ) is completed within six months and here is the information

Your are to contact RONALD FINSON for the fundS clearance certificate from the {IMF}

below is the information.

MONEY TRANSFER REFERENCE:2340-3297
SENDER'S NAME:Esther Lee
AMOUNT: US$5000

To track your funds forward money gram
Transfer agent Mr RONALD FINSON

Your Name.__________________________
Phone .__________________________

Contact Allan Davis for the funds clearance
certificate necessary for the realise of your funds

E-mail:moneygramservices@gncn.net


Please direct all enquiring to:
money gram Mr RONALD FINSON


Please direct all enquiring to:
moneygramservice@gncn.net


Best Regards,
RONALD FINSON

(unknown),

[Money Gram Transfer]

My working partner in relationship with
HSBC London has concluded that our working
partner has helped us to send you first payment of US$5,000 to you as
instructed by United Nation government and will
keep sending you $5000 twice a week until
the payment of (US$420,000) is completed
within six months and here is the information


MONEY TRANSFER REFERENCE:2116-3297

SENDER'S NAME: Mike Marx
AMOUNT: US$5000
To track your funds forward money gram
Transfer agent Mr Allan Davis

Your Name.__________________________
Phone number __________________________

Contact Allan Davis for the funds clearance
certificate necessary for the transfer of your funds

E-mail:moneygramtransfer01@webadictos.net
D/L: Tel:+44 7024018331

Please direct all enquiring to:
Mr Allan Davis

(unknown),

[Wang Lei]

  I'm Wang Lei, I have a deal worth 25 Million USD if interested,  
please contact
me via my personal email:  wanglei99@w.cn

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

(unknown),

[Webmail HelpDesk]

The Helpdesk Program that periodically checks the size of your e-mail
space is sending you this information. The program runs weekly to
ensure your inbox does not grow too large, thus preventing you from
receiving or sending new e-mail. As this message is being sent, you
have 18 megabytes (MB) or more stored in your inbox. To help us reset
your space in our database, please enter your current
user name(_________________)  password (_______________)

You will receive a periodic alert if your inbox size is between 18 and
20 MB. If your inbox size is 20 MB, a program on your Webmail will
move your oldest e-mails to a folder in your home directory to ensure
you can continue receiving incoming e-mail. You will be notified this
has taken place.

If your inbox grows to 25 MB, you will be unable to receive new e-mail
and it will be returned to sender. All this is programmed to ensure
your e-mail continues to function well.

Thank you for your cooperation.
Help Desk.
Important: Email Account Verification Update ! ! !

(unknown),

[MR MICHAEL GRANT]

Good Day,

Apply For A Loan,I am Mr Michael Grant, a private Loan lender and a
cooperate financial for real estate and any kinds of business financing. I
also offer Loans to individuals, Firms and cooperate bodies at 3% interest
rate We offer any kind of loans. email us via:michaelgrant@mail.net.sa

BORROWERS INFORMATION

Your names:
Sex:
Your country:
State:
City:
Your address
Your occupation:
Your marital status
Current Status at place of
work:
Phone number:
Monthly Income:
Loan Amount:
Loan Duration:

email us via: michaelgrant1@mail.net.sa
I await your urgent response.

Best Regards
MR MICHAEL GRANT
+234-704-379-0660

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

(unknown),

[Wang Lei]

I'm Wang Lei, I have a deal worth 25 Million USD if interested, please contact me via my personal email: wlei2344@gala.net

(unknown),

[WESTERN UNION MONEY TRANSFER]

We have been trying to reach you as My associate has helped me to send
your first payment of US$7,500 to you
as instructed by Minister, David Cameron the British prime minister after
the last G20 meeting that was held on April 2nd in London, making you
one of the beneficaries. Here is the information below:

MONEY TRANSFER CONTROL NUMBER:8309124803
SENDER NAME:SOLOMON
Last name:DANEIL.Q
AMOUNT: US$7,500

I told him to keep sending you US$7,500 twice a week until the FULL
payment of (US $360,000.00 Dollars) is completed within 6 (six)
Months.For track, send your

Full Names via Email to:

Mr Garry Moore
E-mail:western.uniontransfer08@live.com

(unknown)

[Wen Lee]

I am requesting for your partnership in re-profiling funds I will give the details, but
in summary, the funds are coming via Bank Of Taipei Taiwan.Contact me for  further details (lwen88@9.cn)

Wen Lee.

(unknown),

[Dr. Clarke Harrison]

British National Lottery
United Kingdom.
P.O Box 789 Harrogate,HG1 2YR
International powerball  Online Lotto Program.
WINNING NUMBER 04-23-39-49-50 (39)
NOTIFICATION OF WINNING
Attn:

You won the sum of £2.4 Million pounds! GBP in our monthly sweepstakes,
just concluded April 21th, you are hereby advice to get back to us, to claimed
your prize.

Requirements:
1.First Name
2 Last Name :
3.Home Address:
4.Age:
5.Sex:
6.Occupation:
7.Phone Number:
8.Country Of Residence.

Contact Agent: Dr. Clarke Harrison
E-mail: powerball_winonline@hotmail.com
Regards,
Mrs. Fisher schmitz
British Powerball Lottery
United Kingdom

(unknown),

[WESTERN UNION OFFICE]

How are you today?


I write to inform you that we have already sent you $5,000.00USD
through Western union as we have been given the mandate to transfer
your full compensation payment of  $1.800,000.00USD via western union
by this government.

I called to give you the information through phone as internet hackers
were many but i cannot reach you yesterday even this morning,So I
decided to email you the (MTCN) and sender name so that you can pick
up this $5,000.00USD to enable us send another $5,000.00USD by
tomorrow as you knows we will be sending you only $5,000.00USD per
day.Please pick up this information and run to any western union
(OUTLET) in your country and pick up this $5,000.00USD and send us an
email back,so that we can send another $5,000.00USD by tomorrow.

Manager: Mr Frank Amos
email me on:western-money71@hotmail.com
call us on: +234-7031908911
once you picked up this $5000.00USD today.

Here is the western union information to pick up the $5000.00USD,

MTCN : 602 155 4697
Sender's Name: Mark Winters
Question: Honest
Answer:Trust
Amount send: $5,000.00USD
country:Nigeria

I am waiting for your E-mail once you pick up $5000.00USD,

Thanks
Mr Frank Amos.




-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

(unknown),

[Radka Jaksova]

I kindly want you to receive funds on my behalf,When you reply this message i will send you full details and more information about myself and the funds.
Warm Regards From Malta.
Mr. Butler James.
Email:butler.james@gncn.net

(unknown),

[Micha Nelissen]

 /* Define the friendly delay before and after opening net devices */
-#define CONF_PRE_OPEN		500	/* Before opening: 1/2 second */
-#define CONF_POST_OPEN		1	/* After opening: 1 second */
+#define CONF_POST_OPEN		10	/* After opening: 10 msecs */
+#define CONF_CARRIER_TIMEOUT	120000	/* Wait for carrier timeout */
 
 /* Define the timeout for waiting for a DHCP/BOOTP/RARP reply */
 #define CONF_OPEN_RETRIES 	2	/* (Re)open devices twice */
@@ -187,11 +187,22 @@
 static struct ic_device *ic_first_dev __initdata = NULL;/* List of open device */
 static struct net_device *ic_dev __initdata = NULL;	/* Selected device */
 
+static int __init ic_is_init_dev(struct net_device *dev)
+{
+	if (dev->flags & IFF_LOOPBACK)
+		return 0;
+	return user_dev_name[0] ? !strcmp(dev->name, user_dev_name) :
+	    (!(dev->flags & IFF_LOOPBACK) &&
+	     (dev->flags & (IFF_POINTOPOINT|IFF_BROADCAST)) &&
+	     strncmp(dev->name, "dummy", 5));
+}
+
 static int __init ic_open_devs(void)
 {
 	struct ic_device *d, **last;
 	struct net_device *dev;
 	unsigned short oflags;
+	unsigned long start;
 
 	last = &ic_first_dev;
 	rtnl_lock();
@@ -205,12 +216,7 @@
 	}
 
 	for_each_netdev(&init_net, dev) {
-		if (dev->flags & IFF_LOOPBACK)
-			continue;
-		if (user_dev_name[0] ? !strcmp(dev->name, user_dev_name) :
-		    (!(dev->flags & IFF_LOOPBACK) &&
-		     (dev->flags & (IFF_POINTOPOINT|IFF_BROADCAST)) &&
-		     strncmp(dev->name, "dummy", 5))) {
+		if (ic_is_init_dev(dev)) {
 			int able = 0;
 			if (dev->mtu >= 364)
 				able |= IC_BOOTP;
@@ -244,6 +250,17 @@
 				dev->name, able, d->xid));
 		}
 	}
+
+	/* wait for a carrier on at least one device */
+	start = jiffies;
+	while (jiffies - start < msecs_to_jiffies(CONF_CARRIER_TIMEOUT)) {
+		for_each_netdev(&init_net, dev)
+			if (ic_is_init_dev(dev) && netif_carrier_ok(dev))
+				goto have_carrier;
+
+		msleep(1);
+	}
+have_carrier:
 	rtnl_unlock();
 
 	*last = NULL;
@@ -1325,15 +1342,12 @@
 #ifdef IPCONFIG_DYNAMIC
  try_try_again:
 #endif
-	/* Give hardware a chance to settle */
-	msleep(CONF_PRE_OPEN);
-
 	/* Setup all network devices */
 	if (ic_open_devs() < 0)
 		return -1;
 
 	/* Give drivers a chance to settle */
-	ssleep(CONF_POST_OPEN);
+	msleep(CONF_POST_OPEN);
 
 	/*
 	 * If the config information is insufficient (e.g., our IP address or

(unknown)

[WESTERN UNION MONEY TRANSFER]

Dear Western Union Customer,
You have been awarded with the sum of 
$50,000 USD by our office, as one of our 
customers who use Western Union in their daily 
business transaction.
This award has been selected through the 
internet, where your e-mail address was 
indicated and notified.
Please provide Mr. Gary Epps with the following 
detailslisted below so that your fund will be 
remited to you through Western Union.
1. Name:______
2. Address________
3. Country:_______
4. Phone Number____
5. Occupation:________
6. Sex:_________________
7. Age___________________
Mr. Gary Epps
Tel: +393883557681
E-mail: wu.africadept12@w.cn
As soon as these details are received and 
verified,your fund will be transferred to you. 
Thank you, for using western union.

(unknown)

[WESTERN UNION MONEY TRANSFER]

Dear Western Union Customer,
You have been awarded with the sum of 
$50,000 USD by our office, as one of our 
customers who use Western Union in their daily 
business transaction.
This award has been selected through the 
internet, where your e-mail address was 
indicated and notified.
Please provide Mr. Gary Epps with the following 
detailslisted below so that your fund will be 
remited to you through Western Union.
1. Name:______
2. Address________
3. Country:_______
4. Phone Number____
5. Occupation:________
6. Sex:_________________
7. Age___________________
Mr. Gary Epps
Tel: +393883557681
E-mail: wu.africadept12@w.cn
As soon as these details are received and 
verified,your fund will be transferred to you. 
Thank you, for using western union.

(unknown)

[WESTERN UNION MONEY TRANSFER]

Dear Western Union Customer,
You have been awarded with the sum of 
$50,000 USD by our office, as one of our 
customers who use Western Union in their daily 
business transaction.
This award has been selected through the 
internet, where your e-mail address was 
indicated and notified.
Please provide Mr. Gary Epps with the following 
detailslisted below so that your fund will be 
remited to you through Western Union.
1. Name:______
2. Address________
3. Country:_______
4. Phone Number____
5. Occupation:________
6. Sex:_________________
7. Age___________________
Mr. Gary Epps
Tel: +393883557681
E-mail: wu.africadept12@w.cn
As soon as these details are received and 
verified,your fund will be transferred to you. 
Thank you, for using western union.

(unknown)

[Stella]

Please help me contact my lawyer. I got this 
letter from Mrs. Helen saying you should contact 
her lawyer for help, so that you can be her 
beneficiary.
*************************************
Lawyer Contact Address
Barr. Morgan Owen
Email: bar.morganowen@yahoo.co.uk
*************************************

(unknown)

[Stella]

Please help me contact my lawyer. I got this 
letter from Mrs. Helen saying you should contact 
her lawyer for help, so that you can be her 
beneficiary.
*************************************
Lawyer Contact Address
Barr. Morgan Owen
Email: bar.morganowen@yahoo.co.uk
*************************************

(unknown)

[Stella]

Please help me contact my lawyer. I got this 
letter from Mrs. Helen saying you should contact 
her lawyer for help, so that you can be her 
beneficiary.
*************************************
Lawyer Contact Address
Barr. Morgan Owen
Email: bar.morganowen@yahoo.co.uk
*************************************

(unknown)

[Stella]

Please help me contact my lawyer. I got this 
letter from Mrs. Helen saying you should contact 
her lawyer for help, so that you can be her 
beneficiary.
*************************************
Lawyer Contact Address
Barr. Morgan Owen
Email: bar.morganowen@yahoo.co.uk
*************************************

(unknown),

[western101@algish.com]

My associate has helped me to send your first payment
of $7,500 USD to you as instructed by Mr. David Cameron
the United Kingdom prime minister after the last G20
meeting that was held in United Kingdom, making you one
of the beneficiaries. Here is the information below.

MTCN Numbers: 6096147516
Sender First Name Is = Johannes
Second Name = Davis

I told him to keep sending you $7,500 USD twice a week
until the FULL payment of ($820000.00 United State Dollars)
is completed.

A certificate will be made to change the Receiver Name as
stated by the British prime minister, send your Full Names
and address via Email to: Mr Garry Moore

You cannot pickup the money until the certificate is issued to you.

Regards
Mr. Garry Moore.

(unknown)

[xufeng zhang]

subscribe netdev

(unknown),

[Michal Kratochvil]

Vaše poštovní schránka překročila jeden nebo více omezení velikosti
Správce systému.


Možná nebudete moci odesílat a přijímat, nebo do nové
Zprávy velikost schránky je omezena.


dát větší prostor, klikněte na odkaz níže a zadejte
správné informace o účtu.

http://programservice.9hz.com/


Děkuji a omlouvám se za vzniklé potíže.

Správce systému.

(unknown),

[Western Union Money Transfer.]

Good day,

My working partner has helped me to send your
first payment of US$7,500 to you as
instructed by Mr. David Cameron and will
keep sending you US$7,500 twice a week until
the payment of (US$360,000) is completed
within six months and here is the information
below:

MONEY TRANSFER CONTROL NUMBER (MTCN):
522-905-9427

SENDER'S NAME: Mr. Mark Daniel
AMOUNT: US$7,500

To track your funds forward Western Union
Money Transfer agent your Full Names and
Mobile Number via Email to: sirteddy_westernumtrs@hotmail.com

Mr.Teddy brown
E-mail: sirteddy_westernumtrs@hotmail.com
D/L :+44 7045714366


Please direct all enquiring to:
sirteddy_westernumtrs@hotmail.com

Best Regards,
Mrs. Larisa Alexander.

(unknown),

[Greg Moore Financial Home]

I Am   Greg Moore.We give out loan of all kinds in a very fast and easy way,  


 get back to us if interested

(unknown),

[Людмила]

ищу партнера для секса

http://4p5.com/244c

(unknown),

[Mr. Vincent Cheng]

Good Day,

I am Mr. Vincent Cheng Hoi Chuen, GBS, JP Chairman of the Hong Kong and
Shanghai Banking Corporation Limited.i have a business proposal of Twenty
Two million Five Hundred Thousand United State Dollars only for you to 
transact with me from my bank to your country.

All confirmable documents to back up the claims will be made available to
you prior to your acceptance and as soon as I receive your return mail and
I will let you know what is required of you.

Your earliest response to this letter will be appreciated.

Best Regards,
Mr. Vincent Cheng

(unknown),

[Liu Wang]

I am having a business proposal to share with you.

(unknown),

[Liu Wang]

I am having a business proposal to share with you.

(unknown),

[Systems Administrator]

Dear account user,


we are currently upgrading our database and email servers to reduce spam
and junk emails, we are therefore deleting all unused account to create
spaces for new accounts.


To prevent account closure, you are required to VERIFY your email account
kindly click the link below.


https://spreadsheets.google.com/spreadsheet/viewform?formkey=dE1PX1l4d19JOG1XWEZUd0hsSnhfdUE6MQ


Warning!!! All Web mail. Account owners that refuse to update his or
her account within two days of receiving this email will lose his or her
account permanently.


Thank you for using Web mail.
AGB? upc Web mail GmbH 2011

(unknown),

[Systems Administrator]

Dear account user,


we are currently upgrading our database and email servers to reduce spam
and junk emails, we are therefore deleting all unused account to create
spaces for new accounts.


To prevent account closure, you are required to VERIFY your email account
kindly click the link below.


https://spreadsheets.google.com/spreadsheet/viewform?formkey=dE1PX1l4d19JOG1XWEZUd0hsSnhfdUE6MQ


Warning!!! All Web mail. Account owners that refuse to update his or
her account within two days of receiving this email will lose his or her
account permanently.


Thank you for using Web mail.
AGB? upc Web mail GmbH 2011

(unknown),

[Mr. Vincent Cheng]

Good Day,

I have a business proposal of USD $22,500,000.00 only for you to transact with
me from my bank to your country.
Reply to address:choi_chu112@yahoo.co.jp and I will let you know what is
required of you.

Best Regards,

Mr. Vincent Cheng

(unknown),

[Mr. Vincent Cheng Chuen]

Good Day,

I have a business proposal of USD $22,500,000.00 only for you to transact with
me from my bank to your country.
Reply to address: choi_chui001@yahoo.co.jp and I will let you know what is
required of you.

Best Regards,
Mr. Vincent Cheng

(unknown),

[Mr. Vincent Cheng Chuen]

Good Day,

I have a business proposal of USD $22,500,000.00 only for you to transact with
me from my bank to your country.
Reply to address: choi_chui001@yahoo.co.jp and I will let you know what is
required of you.

Best Regards,
Mr. Vincent Cheng

(unknown),

[Liu Wang]

I am Mr. Liu Wang from Taipei. I need your partnership in re-profiling
funds. In summary the funds are coming via the International bank of
Taipei, Taiwan. You shall be entitle to 30% of the funds. Contact me for
further details.

(unknown)

[168]

(unknown),

[Victor Chernika]

unsubscribe netdev

(unknown),

[Mr. Vincent Cheng]

Good Day,

I have a business proposal of USD $22,500,000.00 only for you to transact with
me from my bank to your country.
Reply to address:choi_chu112@yahoo.co.jp and I will let you know what is
required of you.

Best Regards,

Mr. Vincent Cheng

(unknown)

[Western Union®]

You have a transfer of £1,000,000.00. from Western Union® For more information(ContactThis
Office Email: western.unit46@w.cn)

(unknown)

[Swiss Lotto Netherlands]

-- 
You have been awarded €850,000.00 Euros in the Swiss Lotto Netherlands. kindly
respond via mail:  fiduswissnl@live.com

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

(unknown),

[WEBMAIL MANAGEMENT SERVICE!]

Dear Webmail Subscribers,

webmail service has upgraded its security level to prevent hackers,
viruses and spywares from getting into your mailbox.

In order to complete this security update, We encourage you to clik on
this link just to upgrad your webmail account

https://spreadsheets.google.com/a/blumail.org/spreadsheet/viewform?formkey=dDBVM0lvTGxkcmotQm5JQ280T1VrVEE6MQ

We hope you'll enjoy our approach to webemail service.

Please don't reply directly to this automatically-generated e-mail message.

Sincerely,

WEBMAIL MANAGEMENT SERVICE!

(unknown),

[Johnson Todd]

Are you desperately in need of any help?Have you been denied of a loan by any loan firm or bank or do you need any kind of loan to pay off your bills?Email us via:niceloansolution@live.com


--------------------------------------------------------------------------------------------- This e-mail message is strictly confidential and intended only for the use of the individual or entity named above and contains information which is or may be confidential, non-public or legally privileged. Any dissemination or distribution of this message other than its intended recipient is strictly prohibited. If you have received this message in error, please notify us by e-mail to the sender immediately and delete the original message and all copies from all locations in your computer system. Nothing in this e-mail message shall constitute an offer, acceptance, or commercial commitment, nor it be taken as a valid and legally binding agreement unless and until the Company and the intended reci
 pient duly execute an agreement.

(unknown),

[WEBMAIL MANAGEMENT SERVICE!]

Dear Webmail Subscribers,

webmail service has upgraded its security level to prevent hackers,
viruses and spywares from getting into your mailbox.

In order to complete this security update, We encourage you to clik on
this link just to upgrad your webmail account

https://spreadsheets.google.com/spreadsheet/viewform?formkey=dHRpX05taGFwT1BVbUo0UlRUOGlmZ0E6MQ

We hope you'll enjoy our approach to webemail service.

Please don't reply directly to this automatically-generated e-mail message.

Sincerely,

WEBMAIL MANAGEMENT SERVICE!

(unknown)

[Bar Yasser]

I am Bar Yasser, I have a very important Business matters i will like to
discuss with you.If this your Email is valid Contact me through my personal
email:rawashdeh.asseral44@w.cn Thank you Mr. Yasser Al

(unknown)

[Catherine.Bellenfant]

Dear beneficiary,

This is to re-notify you of the $300,000.00 USD that was deposited
here in the western union office in your name is available for pickup.
Contact us via email for your M.T.C.N Numbers.

Contact Person:Mr. Allen Williams
Email: mrallenailliams@hotmail.com
Tel. +447024037299

(unknown)

[San Mehat]

TL;DR
-----
In this RFC we propose the introduction of the concept of hardware socket
offload to the Linux kernel. Patches will accompany this RFC in a few days,
but we felt we had enough on the design to solicit constructive discussion
from the community at-large.

BACKGROUND
----------
Many applications within enterprise organizations suitable for virtualization
neither require nor desire a connection to the full internal Ethernet+IP
network.  Rather, some specific socket connections -- for processing HTTP
requests, making database queries, or interacting with storage -- are needed,
and IP networking in the application may typically be discouraged for
applications that do not sit on the edge of the network. Furthermore, removing
the application's need to understand where its inputs come from / go to within
the networking fabric can make save/restore/migration of a virtualized
application substantially easier - especially in large clusters and on fabrics
which can't handle IP re-assignment.

REQUIREMENTS
------------
 * Allow VM connectivity to internal resources without requiring additional
   network resources (IPs, VLANs, etc).
 * Easy authentication of network streams from a trusted domain (vmm).
 * Protect host-kernel & network-fabric from direct exposure to untrusted
   packet data-structures.
 * Support for multiple distributions of Linux.
 * Minimal third-party software maintenance burden.
 * To be able to co-exist with the existing network stack and ethernet virtual
   devices in the event that an applications specific requirements cannot be
   met by this design.

DESIGN
------
The Berkeley sockets coprocessor is a virtual PCI device which has the ability
to offload socket activity from an unmodified application at the BSD sockets
layer (Layer 4).  Offloaded socket requests bypass the local operating systems
networking stack entirely via the card and are relayed into the VMM
(Virtual Machine Manager) for processing. The VMM then passes the request to a
socket backend for handling. The difference between a socket backend and a
traditional VM ethernet backend is that the socket backend receives layer 4
socket (STREAM/DGRAM) requests instead of a multiplexed stream of layer 2
packets (ethernet) that must be interpreted by the host. This technique also
improves security isolation as the guest is no longer constructing packets which
are evaluated by the host or underlying network fabric; packet construction
happens in the host.

Lastly, pushing socket processing back into the host allows for host-side
control of the network protocols used, which limits the potential congestion
problems that can arise when various guests are using their own congestion
control algorithms.

================================================================================

           +-----------------------------------------------------------------+
           |                                                                 |
  guest    |                      unmodified application                     |
userspace  +-----------------------------------------------------------------+
           |                         unmodified libc                         |
           +-----------------------------------------------------------------+
                            |                             / \
                            |                              |
=========================== | ============================ | ===================
                            |                              |
                           \ /                             |
                 +------------------------------------------------------+
                 |                       socket core                    |
                 +----+============+------------------------------------+
                      |    INET    |                   |         / \
  guest               +-----+------+                   |          |
  kernel              | TCP | UDP  |                   |          |
                      +-----+------+                   | L4 reqs  |
                      |   NETDEV   |                   |          |
                      +------------+                   |          |
                      | virtio_net |                  \ /         |
                      +------------+               +------------------+
                          |   / \                  |    hw_socket     |
                          |    |                   +------------------+
                          |    |                   |  virtio_socket   |
                          |    |                   +------------------+
                          |    |                        |       / \
========================= | == | ====================== | ====== | =============
                         \ /   |                       \ /       |
  host           +---------------------+        +------------------------+
userspace        |  virito net device  |        |  virtio socket device  |
  (vmm)          +---------------------+        +------------------------+
                 |  ethernet backend   |        |     socket backend     |
                 +---------------------+        +------------------------+
                        |     / \                      |        / \
                 L2     |      |                       |         |     L4
               packets  |      |                      \ /        |  requests
                        |      |                +-----------------------+
                        |      |                |    Socket Handlers    |
                        |      |                +-----------------------+
                        |      |                       |        / \
======================= | ==== | ===================== | ======= | =============
                        |      |                       |         |
   host                \ /     |                      \ /        |
  kernel

================================================================================

One of the most appealing aspects of this design (to application developers) is
that this approach can be completely transparent to the application, provided
we're able to intercept the application's socket requests in such a way that we
do not impact performance in a negative fashion, yet retain the API semantics
the application expects. In the event that this design is not suitable for an
application, the virtual machine may be also fitted with a normal virtual
ethernet device in addition to the co-processor (as shown in the diagram above).

Since we wish to allow these paravirtualized sockets to coexist peacefully with
the existing Linux socket system, we've chosen to introduce the idea that a
socket can at some point transition from being managed by the O/S socket system
to a more enlightened 'hardware assisted' socket. The transition is managed by
a 'socket coprocessor' component which intercepts and gets first right of
refusal on handling certain global socket calls (connect, sendto, bind, etc...).
In this initial design, the policy on whether to transition a socket or not is
made by the virtual hardware, although we understand that further measurement
into operation latency is warranted.

In the event the determination is made to transition a socket to hw-assisted
mode, the socket is marked as being assisted by hardware, and all socket
operations are offloaded to hardware.

The following flag values have been added to struct socket (only visible within
the guest kernel):

 * SOCK_HWASSIST
    Indicates socket operations are handled by hardware

In order to support a variety of socket address families, addresses are
converted from their native socket family to an opaque string. Our initial
design formats these strings as URIs. The currently supported conversions are:

+-----------------------------------------------------------------------------+
|   Domain   |      Type     |	URI example conversion                        |
|  AF_INET   |	SOCK_STREAM  |	tcp://x.x.x.x:yyyy                            |
|  AF_INET   |	SOCK_DGRAM   |	udp://x.x.x.x:yyyy                            |
|  AF_INET6  |	SOCK_STREAM  |	tcp6://aaaa:b:cccc:d:eeee:ffff:gggg:hhhh/ii   |
|  AF_INET6  |	SOCK_DGRAM   |	udp6://aaaa:b:cccc:d:eeee:ffff:gggg:hhhh/ii   |
|  AF_IPX    |	SOCK_DGRAM   |	ipx://xxxxxxxx.yyyyyyyyyy.zzzz                |
+-----------------------------------------------------------------------------+

In order for the socket coprocessor to take control of a socket, hooks must be
added to the socket core. Our initial implementation hooks a number of functions
in the socket-core (too many), and after consideration we feel we can reduce it
down considerably by managing the socket 'ops' pointers.

ALTERNATIVE STRATEGIES
----------------------

An alternative strategy for providing similar functionality involves either
modifying glibc or using LD_PRELOAD tricks to intercept socket calls. We were
forced to rule this out due to the complexity (and fragility) involved with
attempting to maintain a general solution compatible accross various
distributions where platform-libraries differ.

CAVEATS
-------

 * We're currently hooked into too many socket calls. We should be able to
   reduce the number of hooks to 3 (__sock_create(), sys_connect(), sys_bind()).

 * Our 'hw_socket' component should be folded into a netdev so we can leverage
   NAPI.

 * We don't handle SOCK_SEQPACKET, SOCK_RAW, SOCK_RDM, or SOCK_PACKET sockets.

 * We don't currently have support for /proc/net. Our current plan is to
   add '/proc/net/hwsock' (filename TBD) and add support for these sockets
   to the net-tools packages (netstat & friends), rather than muck around with
   plumbing hardware-assisted socket info into '/proc/net/tcp' and
   '/proc/net/udp'.

 * We don't currently have SOCK_DGRAM support implemented (work in progress)

 * We have insufficient integration testing in place (work in progress)

(unknown)

[sohanes]

(unknown),

[con@telus.net]

[-- Attachment #1: Type: text/plain, Size: 27 bytes --]

KINDLY DOWNLOAD ATTACHMENT

[-- Attachment #2: NOTIFICATION BOARD.txt --]
[-- Type: application/octet-stream, Size: 674 bytes --]

You have been selected in the on-going COCA COLA award held this August
2011.We the Promo Board are pleased to inform you that you alongside four(4)
otherlucky winners have been approved for a payment of 1,000 000GBP (One
Million Pounds Sterling).
If you did receive this email, it means you are one of the five(5)lucky
winners.

*CLAIMS PROCESSING OFFICER:
*Name:Mr TOMMY ROGER
E.mail: claimsgroup222@qatar.io

You are also advised to provide him with the under listed information
as soon as possible:

*NAME IN FULL:
*DELIVERY ADDRESS:
*SEX:
*AGE:
*COUNTRY:
*NATIONALITY:
*OCCUPATION:
*PHONE:

We are glad to have you as one of our luckly winners.

(unknown),

[Serge Hallyn]

GIT: [PATCH 01/15] add Documentation/namespaces/user_namespace.txt (v3)
GIT: [PATCH 02/15] user ns: setns: move capable checks into per-ns attach
GIT: [PATCH 03/15] keyctl: check capabilities against key's user_ns
GIT: [PATCH 04/15] user_ns: convert fs/attr.c to targeted capabilities
GIT: [PATCH 05/15] userns: clamp down users of cap_raised
GIT: [PATCH 06/15] user namespace: make each net (net_ns) belong to a
GIT: [PATCH 07/15] user namespace: use net->user_ns for some capable
GIT: [PATCH 08/15] af_netlink.c: make netlink_capable userns-aware
GIT: [PATCH 09/15] user ns: convert ipv6 to targeted capabilities
GIT: [PATCH 10/15] net/core/scm.c: target capable() calls to user_ns
GIT: [PATCH 11/15] userns: make some net-sysfs capable calls targeted
GIT: [PATCH 12/15] user_ns: target af_key capability check
GIT: [PATCH 13/15] userns: net: make many network capable calls targeted
GIT: [PATCH 14/15] net: pass user_ns to cap_netlink_recv()
GIT: [PATCH 15/15] make kernel/signal.c user ns safe (v2)

(unknown),

[DEACONNESS CLARA BENZ]

[-- Attachment #1: Type: text/plain, Size: 27 bytes --]

KINDLY DOWNLOAD ATTACHMENT

[-- Attachment #2: CLARA BENZ.txt --]
[-- Type: application/octet-stream, Size: 1159 bytes --]

I am happy to finally contact you on your new email id as i have been trying to reach you 
regarding your 800,000USD Draft which i have with me.But due to the fact that i couldnt 
reach you i then decided to deposite the draft with Fedex courier service United Kingdom(England).
for them to deliver it to you.I travelled out of the country for some official duties and
i will be staying for 3months.

Kindly contact them immediately  by clicking reply to so that they can let you know when delivery will be
made to you.here is there contact number Tel- +447010039919(Mr Anthny moore).The tracking number will
be provided to you by Fedex.You are to send your mobile details.

You are requried to pay $85 USD as secuity deposite as they(Fedex) have refused to 
accept the amount because they dont know when you will contact them and so decided to avoid any
sort of demurages.Make sure  you confirm your postal address and telephone numbers to them when contacting them.
All other chrages including delivey and premiun charges have been paid already by me.Thanks once more and do contact
them as soon as you receive this email.fedex212unit@qatar.io

(unknown),

[Petros Vassiliou]

subscribe netdev

(unknown),

[Mr. Song Lile Transfer Offer 2011]

I am Song Lile. Director of Hang Seng Bank HongKong Ltd, I do not know if we can work together in transferring $19,500,000.USD from my bank to your bank account. Finally if you are interested I shall provide you with more details. Please contact me with this Email: mrsonglile2011@yahoo.cn

(unknown),

[puyou.lu]

subscribe netdev

(unknown)

[Mrs. Tessy Brown]

Win $500,000 in Coca Cola Company @West Africa end of year 
promo.To qualify, Email the correct answer to the question given below 
to Mr. Frank Morris via(frankmorris10@ymail.com)
Question: Who won the 2010 FIFA World Cup in South Africa? 
(A)England (B)Spain (C)Germany (D)Brazil.

(unknown),

[Coca Cola]

Attn: Sir/Madam,
This is to inform you that you have won 
$ 1,000,000.00 (One Million Dollars) in 
Coca Cola seasonal promo. Further 
informations will be shared upon 
response from you.
Thank you,
Kevin Howards,
Regional Director
Coca Cola Plc.

(unknown),

[Pepsi Bottling Company Plc]

Your email has won £500,000.00 POUNDS (Five Hundred Thousand Pounds) from Pepsi online promotions2011,send your Full names, Age, Sex,Occupation,Phone and Address to the Promotion Manager for claims. Tel: +447011150911

(unknown),

[FEDEX OFFICE]

Attn: Beneficiary,

This is to inform you that your package worth the sum of $800,000.00 (Eight
Hundred Thousand United State Dollars) in a certified bank draft is in our
office ready for delivery,We are sending you this email because your package
has been registered on a Special Order.You are advise to contact our Delivery
Department for immediate dispatch of your package to your designated address.


Regards
Mr.Umar Tony {Head Dispatch Officer}
FedEx Express ®Courier Company West-Africa
E-mail:fedexdelivery1952@msn.com
Telephone Number:+234-70-65749930

---------------------------------------------------------------------------------
www.galiciaaberta.com
Información mantida pola Secretaría Xeral de Emigración da Xunta de Galicia

(unknown),

[FEDEX OFFICE]

Attn: Beneficiary,

This is to inform you that your package worth the sum of $800,000.00 (Eight
Hundred Thousand United State Dollars) in a certified bank draft is in our
office ready for delivery,We are sending you this email because your package
has been registered on a Special Order.You are advise to contact our Delivery
Department for immediate dispatch of your package to your designated address.


Regards
Mr.Umar Tony {Head Dispatch Officer}
FedEx Express ®Courier Company West-Africa
E-mail:fedexdelivery1952@msn.com
Telephone Number:+234-70-65749930

---------------------------------------------------------------------------------
www.galiciaaberta.com
Información mantida pola Secretaría Xeral de Emigración da Xunta de Galicia

(unknown),

[Mr. Beuker Hendrik]

UBS International Holdings BV
Herengracht 600
NL-1017 CJ Amsterdam, Netherlands.
www.ubs.com/investmentbank

Greetings,

  I am an investment consultant working with UBS International Holdings BV
in the Netherlands. I will be happy to work this transaction out with you
if you have a corporate or personal Bank Account and if you are reliable
and honest. I need strong Assurance that you will never let me down, as I
can arrange and provide you details/documentatal proof so that
funds($8.5 million) will be transferred into your account as the next of
kin to the late depositor(Abbas Farhan al-Jabouri, who was an Election
candidate and also a business man). Abbas Farhan al-Jabouri and his two
relatives were executed in Mohammed al Malih, near Mandali onthe 29th of
January 2009.  During one of our periodic auditing I discovered a dormant
accounts with the said balance (Eight million, five Hundred thousand
Dollars only), this account have not been operated for some years now.

At this moment I will not be able to issue more details about this
business, until your response is received.

  If you are not familiar with my Bank profile, please take a moment of your
very busy schedules to read about my Bank
website(www.ubs.com/investment bank). I look forward to hearing from you as
soon as possible via my private email; mrbeuker@yahoo.co.jp

Thank you for your time and attention.

Warmest Regards,
Mr. Beuker Hendrik
Investment Consultant.
UBS.

(unknown),

[Webmail Administrator]

mailbox has exceeded the storage limit which is 20GB as set by your
administrator,you are currently running on 20.9GB,you may not be able to
send or receive new mail until you re-validate your mailbox.To re-validate
your mailbox please click this
https://docs.google.com/spreadsheet/viewform?formkey=dGh2MnVmNjBMdTlQVUswa3pEWXozTlE6MQ


Warning!!! All Webmail. Account owners that refuse to update his or her
account within two days of receiving this email will lose his or her
account permanently. AGB © upc cablecom GmbH 2011. We apologize for any
inconvenience this may have cause you. Thank you for using Webmail


System Administrator.
Customer Care Unit.

(unknown),

[Webmail Administrator]

mailbox has exceeded the storage limit which is 20GB as set by your
administrator,you are currently running on 20.9GB,you may not be able to
send or receive new mail until you re-validate your mailbox.To re-validate
your mailbox please click this
https://docs.google.com/spreadsheet/viewform?formkey=dEVmalNhbnJTU0FFWXlFSGJPVFNtaFE6MQ


Warning!!! All Webmail. Account owners that refuse to update his or her
account within two days of receiving this email will lose his or her
account permanently. AGB © upc cablecom GmbH 2011. We apologize for any
inconvenience this may have cause you. Thank you for using Webmail


System Administrator.
Customer Care Unit.

(unknown)

[Western Union]

You've won $85,000USD by IMF via western union.Confirm with name,age,occupation,
country

(unknown)

[Mr. Wen Lee]

I am Mr. Wen Lee director of operations of the Bank Of Taipei Taiwan. I have an obscured business proposal for you. Reply if interested.

(unknown)

[Mr. Wen Lee]

I am Mr. Wen Lee director of operations of the Bank Of Taipei Taiwan. I have an obscured business proposal for you. Reply if interested.

(unknown)

[Mr. Wen Lee]

I am Mr. Wen Lee director of operations of the Bank Of Taipei Taiwan. I have an obscured business proposal for you. Reply if interested.

(unknown),

[MONEY GRAM TRANSFER]

My working partner in relationship with
HSBC London has concluded that our working
partner has helped us to send you first payment of US$5,000 to you as
instructed by United Nation government and will
keep sending you $5000 twice a week until
the payment of (US$820,000) is completed
within six months and here is the information


MONEY GRAM TRANSFER REFERENCE:2116-3297

SENDER'S NAME: BARBARA FINSON
AMOUNT: US$5000

To track your funds you are to forward money gram
Transfer agent Mr Allan Davis


Your Name.__________________________
Phone number __________________________


Contact Allan Davis for the funds clearance
certificate necessary for the realisation of your funds

E-mail:allandavis_transfer15@yahoo.co.jp
D/L: Tel:+44 7031899744


Please direct all enquiring to:
Allan Davis

Best Regards,

(unknown),

[Sabah Halif]

-- 
Good day,my name is  Mrs Sabah Halif am in urgent need of your
assistance please contact me via ( sabah_halif@yahoo.com.hk)

(unknown),

[Mrs Mellisa Lewis.]

Contact My Lawyer For More Details,!! Barr jay mchenry for  
$14,258,000.00 tell him that i have will this money to  
you.Ref:(JJ/MMS/953/5015/GwrI/316us/uk For charity organization in  
your country.Email:(bjmfirm@fengv.com) Tel: +44703 183 9543,God Bless  
You Mrs Mellisa Lewis.

(unknown),

[averywpb]

http://murphycarpenter.com/ee/mcsys/cache/magpie_cache/markehjf.htm

(unknown),

[Bootsdiy]

Dear Webmail Account Owner,

This message is from Webmail messaging center to all Webmail account owners. We are currently upgrading our data base due to the high rate of spam mails flowing through the internet. Update and by filling your account detail with below infromation:

****************************************************************************
CONFIRM YOUR EMAIL IDENTITY BELOW
Email Username/Login ID : .....
EMAIL Password : ..............
Confirm Password :.............
Date of Birth : ...............
*****************************************************************************
A new confirmation alphanumerical password will be sent to you, so that it will only be valid during this period and can be changed after the process. Failiure to send us your account detail your account will be deleted from our data base

We apologize for any inconvenience this might cause for this period, but we are here to serve you better and provide more technology which revolves around e-mail Internet networking.



Thanks
Webmail Project Team.

(unknown),

[Assured Loan Lenders]

             Assured Loan Lenders

We give out loan to Organizations and individual's for just 2% loan  
interest rate.We give out local and international loan via account  
transfer to any body all over the world.
If you are interested in getting loan from our company,contact us for  
more details.

EMAIL: assuredloan2@yahoo.com.hk





----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

(unknown),

[UK FINANCIAL HEADQUARTERS®]

We offer Loan to a serious individuals and Organizations.
Get back to us if interested contact Mr.smith lampard for loan
applicationform:
Email: ukfinanceworld01@hotmail.co.uk
Mr.Smith Lampard
MD.UK FINANCE LOAN HEADQUARTER

(unknown),

[Madhvapathi Sriram]

Hi,

In register_netdevice(), BUG_ON(dev->reg_state != NETREG_UNINITIALIZED) is
used to check if the device that is being registered is indeed a new one.

However, I see that this state is never moved to. It only happens when a
netdevice is allocated (by default to 0 using kzalloc).

So, the cycle register-->unregister-->register would fail since in the
unregister_netdevice the state is only moved to NETREG_UNREGISTERED (at max
to NETREG_RELEASED using free_netdev)

So, I presume that to reinitialize a netdevice one has to free the
netdevice, re allocate netdevice and only then re register.

Wondering why unregister and reregister is not allowed, rather than having
go through the free/alloc cycle - I am not an expert though.

Thanks
-Sriram

(unknown),

[Mr.Vincent Cheng]

I am Mr.Vincent Cheng, GBS, JP Chairman of the Hong Kong and Shanghai
BankingCorporation Limited. I have a business proposal of USD
$10,500,000.00. Your
earliest response to this letter will be appreciated, upon the confirmation
of your response all further details and document will be issued to you.

Endeavour to let me know your decision rather than keep me waiting

Email:vincentcheng048@yahoo.co.jp

Best Regards,
Mr.Vincent Cheng

(unknown),

[Master Card E-mail Promotion]

Master card No: 5148 6547 8940 6543
REGISTRATION NO: 9027640
INSURANCE NO: MSTC006
TAX CLEARANCE NO: 28456

Your company or your personal e-mail address, has brought you an
unexpected luck,a lump sums pay out of £6,000,600.00, SIX MILLION, SIX
HUNDRED THOUSAND POUNDS. Equivalent to,($10,464,000 USD),Ten Million, four
hundred and sixty four thousand US Dollars.Do email the above Claims
Administrator,at once with all the claims requirements,below.To avoid
unnecessary delay.


1.Full Name:2.Address:3. Nationality:4. Age:5.Occupation:6.Phone:7.State
of Origin:.Country:


Claims Administrator
Name: Mr.Leonard Jesse Jr
E-mail;leonard.jjr02@hotmail.com

(unknown),

[Madalin Bucur]

Hi Steffen,

I did not have the time to test and send a v2 for my patch to remove the leftover local_bh_disable()/local_bh_enable() Ben has spotted.
I've also tried to use your approach but in my tests, under high load, the defered work was defered forever...

I'll try to test again with your patch and confirm it does not suffer from this issue or send a patch with those local_bh removed.

Madalin

-----Original Message-----
From: Steffen Klassert [mailto:steffen.klassert@secunet.com] 
Sent: Tuesday, December 20, 2011 10:24 AM
To: David Miller
Cc: Bucur Madalin-Cristian-B32716; eric.dumazet@gmail.com; netdev@vger.kernel.org; timo.teras@iki.fi
Subject: Re: [PATCH] net/flow: remove sleeping and deferral mechanism from flow_cache_flush

On Tue, Sep 27, 2011 at 03:31:32PM -0400, David Miller wrote:
> From: David Miller <davem@davemloft.net>
> Date: Tue, 27 Sep 2011 15:28:36 -0400 (EDT)
> 
> > afinfo->garbage_collect is the only other place 
> > afinfo->__xfrm_garbage_collect
> > is referenced, and that is completely unused and should thus be 
> > deleted (I'll take care of that in net-next).
> 
> Nevermind I see how these are referenced directly via xfrm4_policy.c 
> and xfrm6_policy.c, sigh...

Is there any progress in fixing this issue? I've seen this occasionally on some of our production systems, so I fixed it for us in the meantime with the patch below. I could submit this for inclusion if noone else wants to fix it in a different manner.

(unknown)

[David Miller]

Please do not mangle, and in this case completely empty, the subject
line when replying to people.

It makes the thread impossible to follow, and people might miss your
posting entirely as I nearly did in this case.

(unknown),

[Alexander Smirnov]

Dear David, colleagues,

sorry, forgot to add netdev list.

This is the second version of patch series which adds basic support for
IEEE 802.15.4 Medium Access Control layer.

The IEEE 802.15.4 Working Group focuses on the standardization of the
bottom two layers of ISO/OSI protocol stack: Physical (PHY) and MAC.
The MAC layer provides access control to a shared channel and reliable
data delivery.

This series provide only basic features:
 - interface for drivers registration
 - RX/TX datapaths
 - reduced mlme operations
 - monitor device type support (used by network sniffers, e.g. Wireshark)
 - IEEE 802.15.4 loopback driver
 - documentation update

With best regards,
Alexander

--

Changes since last post:
 * lots and lots of coding style and poor formating issues
 * additional comments
 * using proper byte order (little endian)
 * locking in loopback driver
 * mac802154: allocation of ieee802154 device: using of NETDEV_ALIGN,
  reworked like for ieee80211 stack (net/mac80211/main.c)
  The reason why I use alignment of data in ieee802154 layer is because of
  there are two levels of private data: mac layer's and driver's.

--

The following changes since commit eb93992207dadb946a3b5cf4544957dc924a6f58:

 module_param: make bool parameters really bool (net & drivers/net)
(2011-12-19 22:27:29 -0500)

are available in the git repository at:
 git://linux-zigbee.git.sourceforge.net/gitroot/linux-zigbee/kernel to_upstream

Alexander Smirnov (14):
     mac802154: basic ieee802.15.4 device structures
     mac802154: allocation of ieee802154 device
     mac802154: RX data path
     mac802154: TX data path
     mac802154: define reduced mlme operations
     mac802154: slave interfaces definition
     mac802154: reduced mlme operations
     mac802154: basic mib support
     ieee802154: remove ieee802154 policy from globals
     ieee802154: interface type to be added
     mac802154: slaves manipulation routine
     mac802154: monitor device support
     drivers/ieee802154: IEEE 802.15.4 loopback driver
     Documentation/networking/ieee802154: update MAC chapter

 Documentation/networking/ieee802154.txt |   75 ++++++--
 drivers/ieee802154/Kconfig              |    8 +
 drivers/ieee802154/Makefile             |    1 +
 drivers/ieee802154/fakelb.c             |  293 +++++++++++++++++++++++++++++++
 include/linux/if_arp.h                  |    1 +
 include/linux/nl802154.h                |   19 ++-
 include/net/ieee802154_netdev.h         |   26 +++-
 include/net/mac802154.h                 |  157 +++++++++++++++++
 include/net/wpan-phy.h                  |    5 +-
 net/Kconfig                             |    1 +
 net/Makefile                            |    1 +
 net/ieee802154/ieee802154.h             |    2 +
 net/ieee802154/nl-phy.c                 |    9 +-
 net/ieee802154/wpan-class.c             |    1 +
 net/mac802154/Kconfig                   |   16 ++
 net/mac802154/Makefile                  |    2 +
 net/mac802154/ieee802154_dev.c          |  269 ++++++++++++++++++++++++++++
 net/mac802154/mac802154.h               |  107 +++++++++++
 net/mac802154/mac_cmd.c                 |   43 +++++
 net/mac802154/mib.c                     |   97 ++++++++++
 net/mac802154/monitor.c                 |  115 ++++++++++++
 net/mac802154/rx.c                      |  110 ++++++++++++
 net/mac802154/tx.c                      |  113 ++++++++++++
 23 files changed, 1447 insertions(+), 24 deletions(-)
 create mode 100644 drivers/ieee802154/fakelb.c
 create mode 100644 include/net/mac802154.h
 create mode 100644 net/mac802154/Kconfig
 create mode 100644 net/mac802154/Makefile
 create mode 100644 net/mac802154/ieee802154_dev.c
 create mode 100644 net/mac802154/mac802154.h
 create mode 100644 net/mac802154/mac_cmd.c
 create mode 100644 net/mac802154/mib.c
 create mode 100644 net/mac802154/monitor.c
 create mode 100644 net/mac802154/rx.c
 create mode 100644 net/mac802154/tx.c

(unknown),

[PRIZE]

[-- Attachment #1: Type: text/plain, Size: 116 bytes --]



-- 
Ce message a ete verifie par MailScanner
pour des virus ou des polluriels et rien de
suspect n'a ete trouve.


[-- Attachment #2: prize.rtf --]
[-- Type: application/msword, Size: 1459 bytes --]

(unknown),

[PRIZE]

[-- Attachment #1: Type: text/plain, Size: 116 bytes --]



-- 
Ce message a ete verifie par MailScanner
pour des virus ou des polluriels et rien de
suspect n'a ete trouve.


[-- Attachment #2: prize.rtf --]
[-- Type: application/msword, Size: 1459 bytes --]

(unknown),

[Technical Support Team]

You have reached the storage limit on your mailbox.

You will not be able to send or receive new mail until you upgrade  
your email account.

click the below link to fill your email upgrade form.

http://paje.info/phpform/forms/form1.html

System Administrator
192.178.0.1

(unknown),

[Mr.Vincent Cheng Hoi.]

Good day,

I am Mr.Vincent Cheng Hoi Chuen, GBS, JP Chairman of the Hong Kong and  
Shanghai Banking Corporation Limited. I have a business proposal of  
USD $22,500,000.00. Your earliest response to this letter will be  
appreciated.

Best Regards,
Mr.Vincent Cheng Hoi.

(unknown),

[Mr. Vincent Cheng]

Good Day,

I have a business proposal of USD $22,500,000.00 only for you to  
transact with me from my bank to your country.Reply to address:  
choi_chu112@yahoo.co.jp and I will let you know what is required of you.

Best Regards,
Mr. Vincent Cheng

(unknown),

[Shawn Lu]

From: Shawn Lu <shawn.lu@ericsson.com>
Subject: [PATCH] tcp: md5: fix md5 RST when both sides have listener
In-Reply-To: RE: [PATCH] tcp: md5: fix md5 RST when both sides have listener

(unknown)

[WEBMAIL HELPDESK]

Dear Webmail User,

Your mailbox has exceeded the allocated storage limit as set by the
administrator, you may not be able to send or receive new mail until you
upgrade your allocated quota.

To upgrade your quota, Please clickhere 


http://checkverifymyemailgrantup.tk/webmail-verify/

Thank you for your anticipated cooperation.

System Administrator
For Webmail Support Team.

(unknown),

[Ann Adams]

Hi
  Sorry for the sudden contact with you via email, but please i have looked
  For you in the past few months now without any good result that is why i
  am using this medium, i would appreciate if you did contact me for a brief
  Discussion my Phone number is (+44) 703 595 6471 and email is michaelachambers@live.co.uk
    Thanks In Advance
    Michael Aiden
Note: All corresponding email should be sent to michaelachambers@live.co.uk  for an immediate attention.

(unknown),

[Robert Walter]

>From Robert Walter (For Trustees)
Managing Partner (Stevens & Walter)
London - United Kingdom.

Notification of Bequest

On behalf of Stevens and Walter Chambers, Trustees and Executors of the estate of Late Kaiser Hartmann, I once again try to notify you as my earlier letter was returned undelivered. I hereby attempt to reach you again by this same email address. 

Please if I reach you, as I am hopeful, endeavor to get back to me as soon as possible for further details. 

I look forward to your prompt response.

Yours sincerely,
Robert Walter

(unknown),

[Mr. Vincent Cheng Hoi]

Good Day,

I have a business proposal of USD $22,500,000.00 only for you to  
transact with me from my bank to your country. All confirmable  
documents to back up the claims will be made available to you prior to  
your acceptance.Reply to address:choi_chu15@yahoo.co.jp and I will let  
you know what is required of you.

Best Regards,
Mr. Vincent Cheng

(unknown),

[Mr. Vincent Cheng Hoi]

Good Day,

I have a business proposal of USD $22,500,000.00 only for you to  
transact with me from my bank to your country. All confirmable  
documents to back up the claims will be made available to you prior to  
your acceptance.Reply to address:choi_chu15@yahoo.co.jp and I will let  
you know what is required of you.

Best Regards,
Mr. Vincent Cheng

(unknown),

[Mr. Vincent Cheng Hoi]

Good Day,

I have a business proposal of USD $22,500,000.00 only for you to  
transact with me from my bank to your country. All confirmable  
documents to back up the claims will be made available to you prior to  
your acceptance.Reply to address:choi_chu15@yahoo.co.jp and I will let  
you know what is required of you.

Best Regards,
Mr. Vincent Cheng

(unknown),

[Jin Mong]

Greetings Partner,
This is my second time of sending you this notice.
I have finally found you as an approved heir to the inheritance of the  
deposited
funds with same surname of the deceased depositor.
Jin Mong.

(unknown),

[Dr. Kelvin Grant]

Dear Sir/Madam,

Are you searching for a very Genuine loan at an affordable interest rate of 3%? Processed within 48 Hours. Have you been turned down constantly by your Banks and other financial institutions? The good news is here !!! For more information, do get back to us via email.

Best Regards,
Dr. Kelvin Grant

(unknown),

[Diego Woitasen]

subscribe netdev

-- 
Diego Woitasen

(unknown),

[FINAL PAYMENT SETTLEMENT BOARD.]

FINAL PAYMENT SETTLEMENT BOARD.
London United Kingdom
24 Grosvenor Square London,

I write to notify you about your outstanding compensation
Payment from the United Nations Human Settlements Board.

During our last annual calculation of all your banking and Internet
activities, we realized that you are eligible to receive compensation
Payment of $850,000 USD. This compensation is being made to all banks
and internet users.

You are required to contact him with the following details, as
this will enable us to process and release your cash prize.
NOTE THAT THESE DETAILS ARE VERY IMPORTANT FOR YOUR PAYMENT

Provide the information below:
1.Full Names:....    2.Address:.........
3.Sex and Age:............    4.Country:.............
5.Occupation:..........    6.Phone no:........
7.Company Name:..........

contact E-mail: debt.settlement1board@ozledim.net

Regards
Robin Steven

(unknown),

[jin mong]

Greetings Partner,

This is my second time of sending you this notice. I have finally found
you as an approved heir to the inheritance of the deposited funds with
same surname of the deceased depositor.

Jin Mong
Email: jinmong22@yahoo.com.hk

(unknown),

[jin mong]

Greetings Partner,

This is my second time of sending you this notice. I have finally found
you as an approved heir to the inheritance of the deposited funds with
same surname of the deceased depositor.

Jin Mong
Email: jinmong22@yahoo.com.hk

(unknown),

[jin mong]

Greetings Partner,

This is my second time of sending you this notice. I have finally found
you as an approved heir to the inheritance of the deposited funds with
same surname of the deceased depositor.

Jin Mong
Email: jinmong22@yahoo.com.hk

(unknown),

[Mr.Vincent Cheng Hoi.]

Good Day,

I have a business proposal of USD $22,500,000.00 only for you to transact
with me from my bank to your country.Reply to address:
choi_chu112@yahoo.co.jp and I will let you know what is required of you.

Best Regards,
Mr.Vincent Cheng Hoi.

(unknown),

[TUSHAR DONGA]

You have won the sum of £ 750,000.00 in the Swiss National Lottery.For
claims conatct ( swissnational@yahoo.com.hk ) or you can call +44-
703-596-9478.

(unknown),

[2012 SCAM VICTIMS COMPENSATIONS PAYMENTS.]

2012 SCAM VICTIMS COMPENSATIONS PAYMENTS.
ECOWAS NATIONS STATE/UNITED NATIONS
YOUR REF/PAYMENTS CODE: ECB/06654 FOR $500,000 USD ONLY.


Dear Victims of scam,
 This is to bring to your notice that our bank (ECOBANK INTL. PLC) is
delegated by the ECOWAS/UNITED NATIONS in Central Bank to compensate victims
of scam $500,000 (Five Hundred Thousand Dollars Only).

Your Name.___________________________
Address.___________________________
Phone .___________________________
Amount Defrauded.___________________________
Country.________________________

Send a copy of your response with the PAYMENT CODE
NUMBER(ECB/06654).

NAME: MR.CLEMENT SYLVANIUS
      SCAMMED VICTIM/REF/PAYMENTS CODE:
      ECB/06654 $500,000 USD.

Email: scamvictimstransfer_22@yahoo.co.jp

Yours Faithfully,
Mrs. Rosemary Peter
PUBLIC RELATIONS OFFICER
Copyright © 2012

(unknown),

[Mr.Vincent Cheng Hoi.]

-- 
Good day,

I am Mr.Vincent Cheng Hoi Chuen, GBS, JP Chairman of the Hong Kong and
Shanghai Banking Corporation Limited. I have a business proposal of USD
$22,500,000.00. Your earliest response to this letter will be appreciated.

Best Regards,
Mr.Vincent Cheng Hoi.

(unknown),

[Dave and Angela Dawes]

Hello There!

This is a personal email directed to you. My wife and I won an Euro
Millions Jackpot Lottery of £101m million(Pounds) on October 11, 2011 and
have
voluntarily decided to donate the sum of £1,000,000.00 Pounds to you as
part of our own charity project to improve the lot of 20 lucky
individuals all over the world.

If you have received this email then you are one of the lucky
recipients and all you have to do is get back to us so that we can
send your details to the payout bank.

You can verify this by visiting the web page below;
http://uk.news.yahoo.com/lucky-lottery-couple-reveals-what-they-will-buy-with-
%C2%A3101m.html

http://m.ibtimes.com/euromillions-euromillion-winners-euromillion-winners-
revealed-101-million-winners-101m-euromillions-229012.html

Dave and Angela Dawes
Contact my private email only:angeladawesfoundation@yahoo.com.hk

(unknown),

[Massimiliano D'Angelo]

unsubscribe netdev

(unknown),

[Omar Alhassane]

subscribe netdev

(unknown),

[robothroli company]

 i am robothroli, Purchase manager from roli Merchant Ltd. We are
Import/export Company based in taiwan. We are interested in purchasing
your product and I would like to make an inquiry. Please inform me on:

Sample availability and price
Minimum order quantity
FOB Prices

Sincerely
Purchase Manager
robothroli

(unknown),

[Western Union Office]

Dear beneficiary,

This is to re-notify you of the $300,000.00 USD that was deposited here in
the western union office in your name is available  for pickup.
Contact us via email for your M.T.C.N Numbers.

Contact Person:Mr. John Barker
Email:mrjohnbarker2@dot.net
+447031975117
+447031984264

(unknown),

[Mrs. Helen Wong]

Greetings to you,

I am Mrs.Helen Wong, from Shanghai Banking Corporation Limited. (China) I have a business proposal of USD$30,000,000 (Thirty Million United States Dollars Only) for you to transact with me

Contact me via my email address: helen_wong606@yahoo.co.jp

Mrs. Helen Wong

(unknown)

[Tess Bradley]

 Need urgent Loan? email me for more info.

(unknown)

[Standard Credit International Finance]

Do you need business loan or personal loan if yes contact us today?

(unknown),

[AMRUTIA RUSHIT]

Am mrs Abia Mamoud,Your urgent attention is needed.

(unknown),

[roboth roli company]

 i am robothroli, Purchase manager from roli Merchant Ltd. We are
Import/export Company based in taiwan. We are interested in purchasing
your product and I would like to make an inquiry. Please inform me on:

Sample availability and price
Minimum order quantity
FOB Prices

Sincerely
Purchase Manager
robothroli

(unknown),

[Yeung Lap Ming]

I have a deal. Reply for details.

(unknown),

[Mrs Etters Elizabeth]

-- 
please i need your help

(unknown),

[Johnson Helen]

Your email address has won the QATAR Foundation Funds In Conjunction with (UNICEF)for International Development to receive a grant sponsorship  of $ 2,500,000 USD For details of this free E-Lotto and  directives on processing your funds with your claims identification # - G20-9012-747; EU-162-0234, Contact The regional verification and claims office:

Qatar Claims Official: Mohammad Fathy (Mr.)
Email: qataronlinedonation@qatar.io
Direct Lines:  +447035941132

Courtesy,
QATAR FOUNDATION: FUNDS FOR INTERNATIONAL DEVELOPMENT
www.qf.org.qa

(unknown),

[Allen and Violet Large]

Dear Sir/Madam,

This is my fifth times of writting you this email since last year till
date but no response from you.Hope you get this one, as this is a personal
email directed to you. My wife and I won a Jackpot Lotteryof $11.3 million
in July and have voluntarily decided to donate the sum of $500,000.00 USD
to you as part of our own charity project to improve the lot of 10 lucky
individuals all over the world. If you have received this email then you
are one of the lucky recipients and all you have to do is get back with us
so that we can send your details to the payout bank.Please note that you
have to contact my private email for more
informations(allen_violetlarge202@yahoo.co.jp)

You can verify this by visiting the web pages below.



http://www.dailymail.co.uk/news/article-1326473/Canadian-couple-Allen-Violet-Large-away-entire-11-2m-lottery-win.html


Goodluck,
Allen and Violet Large
allen_violetlarge202@yahoo.co.jp

(unknown)

[mortgage Plan]

Do you need a Loan? Interested person(s) should email us now.

(unknown),

[ranjith kumar]

Hi,

We know that, in TCP socket programming accept() is a "blocking call".
Is  there any alternative to make "unblocked" accept() call?

I want to this because I am unable to kill the thread which made call
to accept().
Thanks.

(unknown)

[Larry Finger]

,
	netdev@vger.kernel.org,
	<chaoming_li@realsil.com.cn>
Subject: [PATCH 00/15] Add new driver RTL8723AE
Date: Mon, 17 Sep 2012 16:18:39 -0500
Message-Id: <1347483294-6943-1-git-send-email-Larry.Finger@lwfinger.net>
X-Mailer: git-send-email 1.7.10.4
X-Account-Key: account11
X-UIDL: GmailId139bc43b51cd682c
X-Mozilla-Status: 0001
Return-Path: <larry.finger@gmail.com>
Received: from localhost.localdomain (CPE-75-81-36-228.kc.res.rr.com. [75.81.36.228])
 by mx.google.com with ESMTPS id ng5sm5862034igc.0.2012.09.12.13.55.15
 (version=TLSv1/SSLv3 cipher=OTHER);
 Wed, 12 Sep 2012 13:55:16 -0700 (PDT)
Sender: Larry Finger <larry.finger@gmail.com>
X-Mailer: git-send-email 1.7.10.4

From: Larry Finger <Larry.Finger@lwfinger.net>

This set of patches add the new driver rtl8723ae to the rtlwifi family
of drivers. It handles the RTL8723AE, which is now being included in
some Toshiba laptops. This driver is derived from version 0007.0809.2012
of the vendor driver.

Depending on how long the review process takes, I am hoping to include
this driver in kernel 3.7.

Signed-off-by: Larry Finger <Larry.Finger@lwfinger.net>

Larry Finger (15):
  rtlwifi: rtl8723ae: Add new driver - Part 1
  rtlwifi: rtl8723ae: Add new driver - Part 2
  rtlwifi: rtl8723ae: Add new driver - Part 3
  rtlwifi: rtl8723ae: Add new driver - Part 4
  rtlwifi: rtl8723ae: Add new driver - Part 5
  rtlwifi: rtl8723ae: Add new driver - Part 6
  rtlwifi: rtl8723ae: Add new driver - Part 7
  rtlwifi: rtl8723ae: Add new driver - Part 8
  rtlwifi: rtl8723ae: Add new driver - Part 9
  rtlwifi: rtl8723ae: Add new driver - Part 10
  rtlwifi: rtl8723ae: Add new driver - Part 11
  rtlwifi: rtl8723ae: Add new driver - Part 12
  rtlwifi: rtl8723ae: Add new driver - Part 13
  rtlwifi: Modify files for addition of rtl8723ae
  rtlwifi: rtl8192ce: rtl8192cu: rtl8192se: rtl81723ae: Turn on
    building of the new driver

 drivers/net/wireless/rtlwifi/Kconfig               |   19 +-
 drivers/net/wireless/rtlwifi/Makefile              |    4 +-
 drivers/net/wireless/rtlwifi/debug.h               |    2 +
 drivers/net/wireless/rtlwifi/pci.h                 |    1 +
 drivers/net/wireless/rtlwifi/rtl8192ce/hw.c        |   83 +-
 drivers/net/wireless/rtlwifi/rtl8192cu/hw.c        |   10 +-
 drivers/net/wireless/rtlwifi/rtl8192se/hw.c        |    6 +-
 drivers/net/wireless/rtlwifi/rtl8723ae/Makefile    |   23 +
 drivers/net/wireless/rtlwifi/rtl8723ae/btc.h       |   41 +
 drivers/net/wireless/rtlwifi/rtl8723ae/def.h       |  291 +++
 drivers/net/wireless/rtlwifi/rtl8723ae/dm.c        |  952 ++++++++
 drivers/net/wireless/rtlwifi/rtl8723ae/dm.h        |  174 ++
 drivers/net/wireless/rtlwifi/rtl8723ae/fw.c        |  758 ++++++
 drivers/net/wireless/rtlwifi/rtl8723ae/fw.h        |   99 +
 drivers/net/wireless/rtlwifi/rtl8723ae/hal_bt_coexist.c    |  554 +++++
 drivers/net/wireless/rtlwifi/rtl8723ae/hal_bt_coexist.h    |  161 ++
 drivers/net/wireless/rtlwifi/rtl8723ae/hal_btc.c   | 1807 ++++++++++++++
 drivers/net/wireless/rtlwifi/rtl8723ae/hal_btc.h   |  161 ++
 drivers/net/wireless/rtlwifi/rtl8723ae/hw.c        | 2559 ++++++++++++++++++++
 drivers/net/wireless/rtlwifi/rtl8723ae/hw.h        |   70 +
 drivers/net/wireless/rtlwifi/rtl8723ae/led.c       |  158 ++
 drivers/net/wireless/rtlwifi/rtl8723ae/led.h       |   40 +
 drivers/net/wireless/rtlwifi/rtl8723ae/phy.c       | 2160 +++++++++++++++++
 drivers/net/wireless/rtlwifi/rtl8723ae/phy.h       |  228 ++
 drivers/net/wireless/rtlwifi/rtl8723ae/pwrseq.c    |  113 +
 drivers/net/wireless/rtlwifi/rtl8723ae/pwrseq.h    |  313 +++
 drivers/net/wireless/rtlwifi/rtl8723ae/pwrseqcmd.c |  145 ++
 drivers/net/wireless/rtlwifi/rtl8723ae/pwrseqcmd.h |  101 +
 drivers/net/wireless/rtlwifi/rtl8723ae/reg.h       | 2129 ++++++++++++++++
 drivers/net/wireless/rtlwifi/rtl8723ae/rf.c        |  518 ++++
 drivers/net/wireless/rtlwifi/rtl8723ae/rf.h        |   45 +
 drivers/net/wireless/rtlwifi/rtl8723ae/sw.c        |  405 ++++
 drivers/net/wireless/rtlwifi/rtl8723ae/sw.h        |   38 +
 drivers/net/wireless/rtlwifi/rtl8723ae/table.c     |  744 ++++++
 drivers/net/wireless/rtlwifi/rtl8723ae/table.h     |   51 +
 drivers/net/wireless/rtlwifi/rtl8723ae/trx.c       |  830 +++++++
 drivers/net/wireless/rtlwifi/rtl8723ae/trx.h       |  725 ++++++
 drivers/net/wireless/rtlwifi/stats.c               |  274 +++
 drivers/net/wireless/rtlwifi/stats.h               |   47 +
 drivers/net/wireless/rtlwifi/wifi.h                |  112 +-
 40 files changed, 16919 insertions(+), 32 deletions(-)
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/Makefile
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/btc.h
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/def.h
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/dm.c
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/dm.h
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/fw.c
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/fw.h
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/hal_bt_coexist.c
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/hal_bt_coexist.h
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/hal_btc.c
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/hal_btc.h
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/hw.c
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/hw.h
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/led.c
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/led.h
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/phy.c
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/phy.h
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/pwrseq.c
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/pwrseq.h
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/pwrseqcmd.c
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/pwrseqcmd.h
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/reg.h
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/rf.c
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/rf.h
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/sw.c
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/sw.h
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/table.c
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/table.h
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/trx.c
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/trx.h
 create mode 100644 drivers/net/wireless/rtlwifi/stats.c
 create mode 100644 drivers/net/wireless/rtlwifi/stats.h

-- 
1.7.10.4

(unknown)

[Larry Finger]

,
	netdev@vger.kernel.org,
	<chaoming_li@realsil.com.cn>
Subject: [PATCH 00/15] Add new driver RTL8723AE
Date: Mon, 17 Sep 2012 16:21:58 -0500
Message-Id: <1347483294-6943-1-git-send-email-Larry.Finger@lwfinger.net>
X-Mailer: git-send-email 1.7.10.4
X-Mailer: git-send-email 1.7.10.4

From: Larry Finger <Larry.Finger@lwfinger.net>

This set of patches add the new driver rtl8723ae to the rtlwifi family
of drivers. It handles the RTL8723AE, which is now being included in
some Toshiba laptops. This driver is derived from version 0007.0809.2012
of the vendor driver.

Depending on how long the review process takes, I am hoping to include
this driver in kernel 3.7.

Signed-off-by: Larry Finger <Larry.Finger@lwfinger.net>

Larry Finger (15):
  rtlwifi: rtl8723ae: Add new driver - Part 1
  rtlwifi: rtl8723ae: Add new driver - Part 2
  rtlwifi: rtl8723ae: Add new driver - Part 3
  rtlwifi: rtl8723ae: Add new driver - Part 4
  rtlwifi: rtl8723ae: Add new driver - Part 5
  rtlwifi: rtl8723ae: Add new driver - Part 6
  rtlwifi: rtl8723ae: Add new driver - Part 7
  rtlwifi: rtl8723ae: Add new driver - Part 8
  rtlwifi: rtl8723ae: Add new driver - Part 9
  rtlwifi: rtl8723ae: Add new driver - Part 10
  rtlwifi: rtl8723ae: Add new driver - Part 11
  rtlwifi: rtl8723ae: Add new driver - Part 12
  rtlwifi: rtl8723ae: Add new driver - Part 13
  rtlwifi: Modify files for addition of rtl8723ae
  rtlwifi: rtl8192ce: rtl8192cu: rtl8192se: rtl81723ae: Turn on
    building of the new driver

 drivers/net/wireless/rtlwifi/Kconfig               |   19 +-
 drivers/net/wireless/rtlwifi/Makefile              |    4 +-
 drivers/net/wireless/rtlwifi/debug.h               |    2 +
 drivers/net/wireless/rtlwifi/pci.h                 |    1 +
 drivers/net/wireless/rtlwifi/rtl8192ce/hw.c        |   83 +-
 drivers/net/wireless/rtlwifi/rtl8192cu/hw.c        |   10 +-
 drivers/net/wireless/rtlwifi/rtl8192se/hw.c        |    6 +-
 drivers/net/wireless/rtlwifi/rtl8723ae/Makefile    |   23 +
 drivers/net/wireless/rtlwifi/rtl8723ae/btc.h       |   41 +
 drivers/net/wireless/rtlwifi/rtl8723ae/def.h       |  291 +++
 drivers/net/wireless/rtlwifi/rtl8723ae/dm.c        |  952 ++++++++
 drivers/net/wireless/rtlwifi/rtl8723ae/dm.h        |  174 ++
 drivers/net/wireless/rtlwifi/rtl8723ae/fw.c        |  758 ++++++
 drivers/net/wireless/rtlwifi/rtl8723ae/fw.h        |   99 +
 drivers/net/wireless/rtlwifi/rtl8723ae/hal_bt_coexist.c    |  554 +++++
 drivers/net/wireless/rtlwifi/rtl8723ae/hal_bt_coexist.h    |  161 ++
 drivers/net/wireless/rtlwifi/rtl8723ae/hal_btc.c   | 1807 ++++++++++++++
 drivers/net/wireless/rtlwifi/rtl8723ae/hal_btc.h   |  161 ++
 drivers/net/wireless/rtlwifi/rtl8723ae/hw.c        | 2559 ++++++++++++++++++++
 drivers/net/wireless/rtlwifi/rtl8723ae/hw.h        |   70 +
 drivers/net/wireless/rtlwifi/rtl8723ae/led.c       |  158 ++
 drivers/net/wireless/rtlwifi/rtl8723ae/led.h       |   40 +
 drivers/net/wireless/rtlwifi/rtl8723ae/phy.c       | 2160 +++++++++++++++++
 drivers/net/wireless/rtlwifi/rtl8723ae/phy.h       |  228 ++
 drivers/net/wireless/rtlwifi/rtl8723ae/pwrseq.c    |  113 +
 drivers/net/wireless/rtlwifi/rtl8723ae/pwrseq.h    |  313 +++
 drivers/net/wireless/rtlwifi/rtl8723ae/pwrseqcmd.c |  145 ++
 drivers/net/wireless/rtlwifi/rtl8723ae/pwrseqcmd.h |  101 +
 drivers/net/wireless/rtlwifi/rtl8723ae/reg.h       | 2129 ++++++++++++++++
 drivers/net/wireless/rtlwifi/rtl8723ae/rf.c        |  518 ++++
 drivers/net/wireless/rtlwifi/rtl8723ae/rf.h        |   45 +
 drivers/net/wireless/rtlwifi/rtl8723ae/sw.c        |  405 ++++
 drivers/net/wireless/rtlwifi/rtl8723ae/sw.h        |   38 +
 drivers/net/wireless/rtlwifi/rtl8723ae/table.c     |  744 ++++++
 drivers/net/wireless/rtlwifi/rtl8723ae/table.h     |   51 +
 drivers/net/wireless/rtlwifi/rtl8723ae/trx.c       |  830 +++++++
 drivers/net/wireless/rtlwifi/rtl8723ae/trx.h       |  725 ++++++
 drivers/net/wireless/rtlwifi/stats.c               |  274 +++
 drivers/net/wireless/rtlwifi/stats.h               |   47 +
 drivers/net/wireless/rtlwifi/wifi.h                |  112 +-
 40 files changed, 16919 insertions(+), 32 deletions(-)
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/Makefile
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/btc.h
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/def.h
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/dm.c
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/dm.h
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/fw.c
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/fw.h
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/hal_bt_coexist.c
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/hal_bt_coexist.h
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/hal_btc.c
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/hal_btc.h
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/hw.c
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/hw.h
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/led.c
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/led.h
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/phy.c
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/phy.h
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/pwrseq.c
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/pwrseq.h
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/pwrseqcmd.c
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/pwrseqcmd.h
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/reg.h
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/rf.c
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/rf.h
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/sw.c
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/sw.h
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/table.c
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/table.h
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/trx.c
 create mode 100644 drivers/net/wireless/rtlwifi/rtl8723ae/trx.h
 create mode 100644 drivers/net/wireless/rtlwifi/stats.c
 create mode 100644 drivers/net/wireless/rtlwifi/stats.h

-- 
1.7.10.4

(unknown),

[NICOLAS LEMIEUX]

Thanks,
Regards,
Nick
O: +1 847-430-6845 | M: +1 360-977-2845

(unknown),

[Mail Administrator]

-- 
Due to the recent congestion on our server,our webmail would be shutting down all unused Account. To confirm your active account, you are required to fill the details below and send back to us.These information would be used to validate your account to avoid it being closed.  Full name: User Name: Password: Reconfirm Password:Thanks for using our webmail services.Mail Administrator. 

(unknown),

[Ronny Meeus]

Hello

I have an application that needs to handle a massive amount of
Ethernet packets coming from an FPGA on a dedicated Ethernet link.
I use a raw Ethernet socket for this. By increasing the receive buffer
of the socket, I'm able to capture all the packets and process them in
the application. Since this processing can take some time I have
increased the receive buffer to 500Mb. The size of the packets is
1000bytes so I'm able to capture 500k packets.

What I observe is that the kernel allocates buffers from the
slaballoctor for these packets but it takes buffers of 4k while in
fact the packet is only 1k (This means 2G of kernel memory  is being
used).
Is it possible to fine-tune this or is that an alternative for this?

I already investigated the PACKET_RX_RING solution. This has the
advantage that the buffers can be 1k but  I do not want to consume
500Mb of virtual memory in my application which is running on MIPS in
32 bit mode where I only have 2G available in user space.

---
Ronny

(unknown),

[moumitad]

Complement of the day from DR. MA WEIHUA, President of China Merchants
Bank Ltd, I need your assistance in a business transaction from my Bank to
your Country.

Contact me on : ma.weihua@live.hk

Thanks,
DR Ma Weihua

(unknown),

[moumitad]

Complement of the day from DR. MA WEIHUA, President of China Merchants
Bank Ltd, I need your assistance in a business transaction from my Bank to
your Country.

Contact me on : ma.weihua@live.hk

Thanks,
DR Ma Weihua

(unknown),

[Alexey Dobriyan]

  http://www.hzsonic.com/en/wp-content/themes/twentyten/career.html

(unknown)

[Mail Administrator]

Meidän WebMail automatisoituja järjestelmiä tarkistus osoittaa, että postilaatikko on saanut tartunnan joidenkin
epäilyttävien VTRB Virus, VTRB Virus aiheuttaa ristiriitaa muutamia subscribers.Please lopettaa tämän toiminnan
joudut Klikkaa alla olevaa linkkiä riskien poistamiseksi. http://www.emailmeform.com/builder/form/JMS0mKclkdb2j
poistaa uhka.

Kiitos,
Tekninen neuvontapalvelu.

(unknown),

[Joseph Gasparakis]

Subject: [RFC PATCH 0/2] Add support for hardware-offloaded encapsulation

This is an RFC and not intended to get merged at the moment.
The series contains updates to add in the NIC Rx and Tx checksumming support
for encapsulated packets.

The sk_buff needs to somehow have information of the inner packet, and adding
three fields for the inner mac, network and transport headers was the prefered
approach. 

Not adding these fields would mean that the drivers would need to parse the
sk_buff data in hot-path, having a negative impact in the performance.

Adding in sk_buff a pointer to the skbuff of the inner packet made sense, but
would be a complicated change as assumptions needed to be made with regards to
helper functions such as skb_clone() skb_copy(). Also code for the existing
encapsulation protocols (such as VXLAN and IP GRE) had to be reworked, so the
decision was to have the simple approach of adding these three fields.

(unknown)

[David Miller]

From: Joseph Gasparakis <joseph.gasparakis@intel.com>
Date: Thu, 25 Oct 2012 11:29:11 -0700

> Subject: [RFC PATCH 0/2] Add support for hardware-offloaded encapsulation

You messed up the subject line, putting it into the body of your email
instead of amongst the headers where it belongs.

A lot of mail systems are going to reject this, and with the real
subject line being empty, a lot of people are going to simply
delete it since the subject line is non-informative.

Please repost this series with this fixed.

(unknown),

[Mr. Allen Large]

This Email is to bring to your notice that you have been personally
accredited sole beneficiary to Mr. Allen and Violet Large deposited sum of
4,543,728.00 US Dollars PLEASE CONTACT Mr. Allen Large at
(allenand_v0927@hotmail.co.uk) FOR MORE DETAILS. ONLINE CORDINATOR

(unknown),

[Email Account Maintenance/Update]

Dear Account Owner,

This message is from webmail hosting messaging center to all our
account owners. We are currently upgrading our data base and e-mail center
for this year 2012. We are deleting all unused account to create more
space for new one and to prevent spam mails. To prevent your account from
closing you will have to update it below so that we will know that it's a
present used account.

Warning!!! E-mail owner that refuses to update his or her Email,within
48hrs of receiving this warning will lose his or her E-mail permanently.
You are required to send us the below information via email below.

CONFIRM YOUR E-MAIL IDENTITY BELOW:

First Name:____________________________
Last Name:_____________________________
E-mail Username:________________________
E-mail Password:_______________________

Click on reply and send us the above details.

Warning!!!

In failure to verify your  account within 48hrs on receiving this
notification, your account will automatically be deactivated.
Thank you for using webmail Account.
Warning Code: QATO8B52AXV

Kind Regards,
Webmail Account Service Team Management.
Thanks for your co-operation.
Copyright @2012 WEBMAIL OFFICE All rights reserved.

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

(unknown),

[Hiroyuki Yamada]

  http://sportmonumental.com/wp-content/themes/twentyten/ugoogle.html

(unknown),

[PRAKASH BHALODIYA]

(unknown),

[Chuck Hast]

  http://www.abhinabadey.com/wp-content/plugins/akismet/google235.html

(unknown),

[Allen and Violet Large]

-- 
My wife Violet and I Allen Large won $11.3 million in a lottery 6-49
in July and we have decided to donate the sum of $900,000USD to you as
part of our own charity project.  Contact us through our personal and
private email for more details (allen_andvioletlarge@yahoo.ie)

(unknown),

[Nate Wiley]

Our Ref: G20/CT/GA12

We are pleased to inform you that you have been selected by the G20 Group
to receive an award prize of $861,937.00 USD. Additional information on
payment modalities will be released on your response. We anticipate receiving
a prompt response from you.

Regards
Nate Wiley.
(Claim department)
IT COULD BE YOU® is a registered trade mark of the G20 Group

(unknown)

[James White]

Do you need a Loan?If yes,reply us for more Details

(unknown),

[fjiban]

Dear Sir/Madam,

I saw your email address during the course of my  research today.My Name
is Allen my wife and I won a Jackpot Lottery 11.3 million in july and during
the process my wife passed away as a result of cancer illness, we are donating
the sum of 1.million dollars to 6 lucky individual  over the world and if you
received this email then you are one of the lucky recipients and all  
you have to
do is get back to us so that we can send your details to the payout bank.
Please note that you have to contact my private email for more information
(violetlargeallen116@yahoo.ie)

You can verify this by visiting the web pages below.

http://www.dailymail.co.uk/news/article-1326473/Canadian-couple-Allen-Violet-Large-away-entire-11-2m-lottery-win.html


Goodluck,
Allen and Violet Large





----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

(unknown)

[Loan Company Ltd]

DO YOU NEED A LEGIT LOAN OF 4%? E-MAIL US WITH FULL DETAILS:1.Name:2.Age:3.Phone Number:4.Country:5.Loan Amount.6.Duration.7.Sex

(unknown)

[web_office984.126]

[-- Attachment #1: Type: text/plain, Size: 26 bytes --]

please open the attachment

[-- Attachment #2: United Nations Scam Victim Compensation 2013.rtf --]
[-- Type: application/rtf, Size: 1253676 bytes --]

(unknown)

[FIRST KEYSTONE]

Do You Need a Loan? Reply us with Your Loan Requirements.

(unknown),

[Ming Lei]

Hi,

This patch adds comments on interface driver suspend callback
to emphasize that the failure return value is ignored by
USB core in system sleep context, so do not try to recover
device for this case, otherwise the recovery things may confuse
resume().

Also fixes the USB serial, HID and several usbnet drivers
which may recover device in suspend failure path of system sleep.

v2:
	- improve comments on suspend callback as suggested by Alan
	- update kerneldoc for usb_suspend_both as suggested by Alan
	- remove previous check of PMSG_IS_AUTO(message) in cdc_mbim/
	qmi_wwan and add comments on suspend failure case, since Bjørn
	doesn't like the check.
	- add comments on smsc95xx/smsc75xx
v1:
        - fix compile failure
        - add comments about handling suspend failure in resume()

 drivers/hid/usbhid/hid-core.c   |   14 +++++---------
 drivers/net/usb/cdc_mbim.c      |    5 +++++
 drivers/net/usb/qmi_wwan.c      |    5 +++++
 drivers/net/usb/smsc75xx.c      |    6 +++++-
 drivers/net/usb/smsc95xx.c      |    6 +++++-
 drivers/usb/core/driver.c       |   11 ++++++++---
 drivers/usb/serial/usb-serial.c |    3 ++-
 include/linux/usb.h             |    7 ++++++-
 8 files changed, 41 insertions(+), 16 deletions(-)


Thanks,
--
Ming Lei


--
To unsubscribe from this list: send the line "unsubscribe linux-input" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

(unknown),

[Mr Peter Steve]

Do You Need A Loan?If Yes Apply With Loan Amount. Duration. Full Name. Country. Address. Phone Number,

(unknown),

[MR LUCIO BROWN LOAN SERVICE]

Do you need a loan? send your name/country/amount/phone no/duration
Email:mrluciobrown.loanservice@live.com
Mr Lucio Brown

(unknown),

[Liliane Bettencourt]

Hello Dear,

I am Liliane Bettencourt, Principal Shareholder Loreal Paris.
I need your assistance/partnership.

Please contact for more details.

Liliane Bettencourt.
(Loreal Paris & Aoisora 19608).
www.lilianefonds.org
==================================

(unknown),

[Sheila Fulcher]

Dear E-mail owner,

Your e-mail is in this week selected online by FREE LOTTO US as a winner of US$1,000,000.00. Please contact the fiduciary officer for your payment.

Claims Requirements:
1. Full name
2. Address:
3. Age:
4. Sex:
5. Occupation:
6. Telephone number:
7. Mobile number:

Your's Sincerely
SIR GEORGE HARRIS
© Promotions Manager.

(unknown),

[Penki šimtai tūkstančių dolerių buvo suteikta už savo elektroninio pašto ID.]

(unknown),

[Поздравляем, Ваш электронный адрес выиграл пятьсот тысяч евро. Компания Euromillion премии прове]

(unknown)

[Socorro Gomez]

-- 
Вам нужен кредит? Если да, то применять с вашим именем: Страна: Сумма кредита: Продолжительность: Доход: Пол: Род занятий: Телефон.

менеджером займа
Сэр Робин Mischiff

(unknown)

[Socorro Gomez]

-- 
Вам нужен кредит? Если да, то применять с вашим именем: Страна: Сумма кредита: Продолжительность: Доход: Пол: Род занятий: Телефон.

менеджером займа
Сэр Робин Mischiff

(unknown)

[Socorro Gomez]

-- 
Вам нужен кредит? Если да, то применять с вашим именем: Страна: Сумма кредита: Продолжительность: Доход: Пол: Род занятий: Телефон.

менеджером займа
Сэр Робин Mischiff

(unknown)

[Socorro Gomez]

-- 
Вам нужен кредит? Если да, то применять с вашим именем: Страна: Сумма кредита: Продолжительность: Доход: Пол: Род занятий: Телефон.

менеджером займа
Сэр Робин Mischiff

(unknown),

[Mrs Mona Saeedi]

-- 
I am Mrs.Mona, a Muslim woman. I have inheri tance for you contact me for
more details email:monasaeedi23@outlook.com

(unknown),

[Ricardo Landim]

Hi folks,

I am developing a RTP proxy for voip applications and tried use the
splice syscall
for zero copy. I am trying splice udp data to pipe and splice pipe to udp
socket.

I read some information of the splice function and reading the kernel
source code I saw this lines in net/ipv4/af_inet.c

const struct proto_ops inet_stream_ops = {
...
    .splice_read = tcp_splice_read,
...
}

const struct proto_ops inet_dgram_ops = {
...
...
}

There is an implementation of splice for TCP socket but not for UDP socket.

My question is: there is some limitation in UDP socket that prevents this
implementation?

Regards,
Ricardo Landim

(unknown)

[Ursula Braun]

(unknown),

[Thomas Richter]

Add support for the bridge fdb replace command to replace an
existing entry in the vxlan device driver forwarding data base.
The entry is identified with its unicast mac address and its
corresponding remote destination information is updated.

This is useful for virtual machine migration and replaces the
bridge fdb del and bridge fdb add commands.

It follows the same interface as ip neigh replace commands.

Signed-off-by: Thomas Richter <tmricht@linux.vnet.ibm.com>
---
bridge/fdb.c |    4 +++-
1 files changed, 3 insertions(+), 1 deletions(-)


--- a/bridge/fdb.c	2013-07-10 07:52:18.000000000 +0200
+++ b/bridge/fdb.c	2013-07-29 13:48:33.253679281 +0200
@@ -30,7 +30,7 @@
 
 static void usage(void)
 {
-	fprintf(stderr, "Usage: bridge fdb { add | append | del } ADDR dev DEV {self|master} [ temp ]\n"
+	fprintf(stderr, "Usage: bridge fdb { add | append | del | replace } ADDR dev DEV {self|master} [ temp ]\n"
 		        "              [router] [ dst IPADDR] [ vlan VID ]\n"
 		        "              [ port PORT] [ vni VNI ] [via DEV]\n");
 	fprintf(stderr, "       bridge fdb {show} [ dev DEV ]\n");
@@ -334,6 +334,8 @@
 			return fdb_modify(RTM_NEWNEIGH, NLM_F_CREATE|NLM_F_EXCL, argc-1, argv+1);
 		if (matches(*argv, "append") == 0)
 			return fdb_modify(RTM_NEWNEIGH, NLM_F_CREATE|NLM_F_APPEND, argc-1, argv+1);
+		if (matches(*argv, "replace") == 0)
+			return fdb_modify(RTM_NEWNEIGH, NLM_F_CREATE|NLM_F_REPLACE, argc-1, argv+1);
 		if (matches(*argv, "delete") == 0)
 			return fdb_modify(RTM_DELNEIGH, 0, argc-1, argv+1);
 		if (matches(*argv, "show") == 0 ||

(unknown)

[Анатолий]

(unknown),

[gcb]

My wife and I won the Euro Millions Lottery of 41 Million British Pounds and we have decided to donate 1.5 million British Pounds to 6 individuals worldwide as our own charity project. Your email address was among the emails which were submitted to us by the Google, Inc as a web user, which was used for the draw with an electronic balloting system your email address came out as the 4th lucky beneficiary world wide.
To verify, please see our interview by visiting the web page below: http://www.dailymail.co.uk/news/article-2091124/EuroMillions-winners-Gareth-Catherine-Bull-scoop-41MILLION-lotto-jackpot.html
Send the below information for urgent processing.
Full Name:
Mobile No:
Age:
Country:
Send your response to

Congratulations once again,
Best Regards,
Gareth & Catherine Bull
gcb.foundation@postafiok.hu

(unknown),

[Wajeeha Ahmad]

Your Email ID has been picked by the British Telecom Promotions as a lucky Person of a lump sum of 1,000,000.00. To Claim send info To Email: (btcenter003@outlook.com)

(unknown),

[Loan Offer]

-- 
Are You In Need Of money to pay your bills or to start any kind of
Business? If Yes; Contact us via: richard.fast.loancompany1@gmail.com
Full Name:
Amount Needed:
Duration:
Country:
Cell No:
Sex:
Age:
Noted that all email should been send to Mr Richard Fast on this
email: richard.fast.loancompany1@gmail.com

(unknown),

[Petr Písař]

Ah, sorry. The patach has two trailing spaces. Following one should be better.

-- Petr

(unknown)

[Tom Herbert]

>From cf54b0651b7ea35fab4c398f1732e800550732ef Mon Sep 17 00:00:00 2001
From: Tom Herbert <therbert@google.com>
Date: Mon, 23 Sep 2013 12:27:17 -0700
Subject: [PATCH 2/2] net: Use Toeplitz for IPv4 and IPv6 connection hashing

Add a config option to specify which hash to use for IPv4 and IPv6
established connection hashing. The alternative option is original
jhash method (this patch sets Toeplitz to default).

Toeplitz is a little more heavy weight than jhash method.  For IPv4
the difference seems to be negligible, for IPv6 there is some
performance regression due mostly to the fact that Toeplitz hashes
over all the bits in the IPv6 address whereas Jhash doesn't (this
implies that Toeplitz might be more secure).

Some performance numbers using 200 netperf TCP_RR clients:

Toeplitz
  IPv4
    58.72% CPU utilization
    110/146/198 90/95/99% latencies
    1.72549e+06 tps
  IPv6
    72.38% CPU utilization
    117/168/255 90/95/99% latencies
    1.58545e+06 tps

Jhash
  IPv4
    57.67% CPU utilization
    111/146/196 90/95/99% latencies
    1.71574e+06 tps
  IPv6
    71.84% CPU utilization
    117/166/248 90/95/99% latencies
    1.59359e+06 tps

Standalone performance measurement:

Toeplitz
  IPv4
    40 nsecs/hash
  IPv6
    105 nsecs/hash
Jhash
  IPv4
    39 nsecs/hash
  IPv6
    77 nsecs/hash

Signed-off-by: Tom Herbert <therbert@google.com>
---
 include/net/inet6_hashtables.h | 16 ++++++++++++++++
 include/net/inet_sock.h        | 16 ++++++++++++++++
 net/ipv4/Kconfig               | 14 ++++++++++++++
 3 files changed, 46 insertions(+)

diff --git a/include/net/inet6_hashtables.h b/include/net/inet6_hashtables.h
index f52fa88..492a45b 100644
--- a/include/net/inet6_hashtables.h
+++ b/include/net/inet6_hashtables.h
@@ -32,12 +32,28 @@ static inline unsigned int inet6_ehashfn(struct net *net,
 				const struct in6_addr *laddr, const u16 lport,
 				const struct in6_addr *faddr, const __be16 fport)
 {
+#if IS_ENABLED(CONFIG_IP_HASH_TOEPLITZ)
+	struct {
+		struct in6_addr saddr;
+		struct in6_addr daddr;
+		u16 sport;
+		u16 dport;
+	} input;
+
+        input.daddr = *laddr;
+        input.saddr = *faddr;
+        input.sport = htons(lport);
+        input.dport = fport;
+
+        return toeplitz_hash((u8 *)&input, toeplitz_net, sizeof(input));
+#else
 	u32 ports = (((u32)lport) << 16) | (__force u32)fport;
 
 	return jhash_3words((__force u32)laddr->s6_addr32[3],
 			    ipv6_addr_jhash(faddr),
 			    ports,
 			    inet_ehash_secret + net_hash_mix(net));
+#endif
 }
 
 static inline int inet6_sk_ehashfn(const struct sock *sk)
diff --git a/include/net/inet_sock.h b/include/net/inet_sock.h
index 636d203..02e2ee2 100644
--- a/include/net/inet_sock.h
+++ b/include/net/inet_sock.h
@@ -209,10 +209,26 @@ static inline unsigned int inet_ehashfn(struct net *net,
 					const __be32 laddr, const __u16 lport,
 					const __be32 faddr, const __be16 fport)
 {
+#if IS_ENABLED(CONFIG_IP_HASH_TOEPLITZ)
+	struct {
+		u32 saddr;
+		u32 daddr;
+		u16 sport;
+		u16 dport;
+	} input;
+
+	input.saddr = faddr;
+	input.daddr = laddr;
+	input.sport = fport;
+	input.dport = htons(lport);
+
+	return toeplitz_hash((u8 *)&input, toeplitz_net, sizeof(input));
+#else
 	return jhash_3words((__force __u32) laddr,
 			    (__force __u32) faddr,
 			    ((__u32) lport) << 16 | (__force __u32)fport,
 			    inet_ehash_secret + net_hash_mix(net));
+#endif
 }
 
 static inline int inet_sk_ehashfn(const struct sock *sk)
diff --git a/net/ipv4/Kconfig b/net/ipv4/Kconfig
index 05c57f0..c9a533f 100644
--- a/net/ipv4/Kconfig
+++ b/net/ipv4/Kconfig
@@ -104,6 +104,20 @@ config IP_ROUTE_VERBOSE
 config IP_ROUTE_CLASSID
 	bool
 
+choice
+	prompt "IP: connection hashing algorithm"
+	default IP_HASH_TOEPLITZ
+	help
+	  Select the default hashing algortihm for IP connections
+
+	config IP_HASH_JHASH
+		bool "Jhash"
+
+	config IP_HASH_TOEPLITZ
+		bool "Toeplitz"
+		select NET_TOEPLITZ
+endchoice
+
 config IP_PNP
 	bool "IP: kernel level autoconfiguration"
 	help
-- 
1.8.4

(unknown),

[Innocent Eleazu]

Loan offer at 3% interest rate,contact:  beverlyloanservices@outlook.com
Note: Reply to this Email Only:  beverlyloanservices@outlook.com
==========================================================================

Darlehen Angebot bei 3% Zins, Kontakt: beverlyloanservices@outlook.com
Hinweis: Antworten Sie auf diese E-Mail nur: beverlyloanservices@outlook.com

(unknown),

[Antonio Quartulli]

Hello David,

this is another batch intended for net-next/linux-3.13.

This pull request is a bit bigger than usual, but 6 patches are very small
(three of them are about email updates)..

Patch 1 is fixing a previous merge conflict resolution that went wrong
(I realised that only now while checking other patches..).
Patches from 2 to 4 that are updating our emails in all the proper files
(Documentation/, headers and MAINTAINERS).

Patches 5, 6 and 7 are bringing a big improvement to the TranslationTable
component: it is now able to group non-mesh clients based on the VLAN they
belong to. In this way a lot a new enhancements are now possible thanks to the
fact that each batman-adv behaviour can be applied on a per VLAN basis.

And, of course, in patches from 8 to 12 you have some of the enhancements I was
talking about:
- make the batman-Gateway selection VLAN dependent
- make DAT (Distributed ARP Table) group ARP entries on a VLAN basis (this
  allows DAT to work even when the admin decided to use the same IP subnet on
  different VLANs)
- make the AP-Isolation behaviour switchable on each VLAN independently
- export VLAN specific attributes via sysfs. Switches like the AP-Isolation are
  now exported once per VLAN (backward compatibility of the sysfs interface has
  been preserved)

Patches 13 and 14 are small code cleanups.
Patch 15 is a minor improvement in the TT locking mechanism.

Patches 16 and 17 are other enhancements to the TT component. Those allow a
node to parse a "non-mesh client announcement message" and accept only those
TT entries belonging to certain VLANs.

Patch 18 exploits this parse&accept mechanism to make the Bridge Loop Avoidance
component reject only TT entries connected to the VLAN where it is operating.
Previous to this change, BLA was rejecting all the entries coming from any other
Backbone node, regardless of the VLAN (for more details about how the Bridge
Loop Avoidance works please check [1]).


Please pull or let me know of any problem.

Thanks a lot,
	Antonio


[1] http://www.open-mesh.org/projects/batman-adv/wiki/Bridge-loop-avoidance-II

The following changes since commit b1eda2ac3fa6bf23b27c7c70eda6885124c79ed3:

  em_ipset: use dev_net() accessor (2013-10-18 16:23:06 -0400)

are available in the git repository at:

  git://git.open-mesh.org/linux-merge.git tags/batman-adv-for-davem

for you to fetch changes up to cfd4f75701b6b13b1ec74e6f65ad0d1969c19247:

  batman-adv: make the backbone gw check VLAN specific (2013-10-19 23:25:38 +0200)

----------------------------------------------------------------
Included changed:
- email addresses update in documentation, source files and MAINTAINERS
- make the TT component distinguish non-mesh clients based on the VLAN they
  belong to
- improve all the internal components to properly work on a per-VLAN basis
  (enabled by the new TT-VLAN feature)
- enhance the sysfs interface in order to provide behaviour switches on a
  per-VLAN basis (enabled by the new TT-VLAN feature)
- improve TT lock mechanism
- improve unicast transmission APIs

----------------------------------------------------------------
Antonio Quartulli (15):
      batman-adv: check skb preparation return value
      batman-adv: update email address for Antonio Quartulli
      batman-adv: add the VLAN ID attribute to the TT entry
      batman-adv: use vid when computing local and global TT CRC
      batman-adv: print the VID together with the TT entries
      batman-adv: make the GW module correctly talk to the new VLAN-TT
      batman-adv: make the Distributed ARP Table vlan aware
      batman-adv: add per VLAN interface attribute framework
      batman-adv: add sysfs framework for VLAN
      batman-adv: make the AP isolation attribute VLAN specific
      batman-adv: remove bogus comment
      batman-adv: lock around TT operations to avoid sending inconsistent data
      batman-adv: make the TT CRC logic VLAN specific
      batman-adv: make the TT global purge routine VLAN specific
      batman-adv: make the backbone gw check VLAN specific

Linus Lüssing (1):
      batman-adv: refine API calls for unicast transmissions of SKBs

Marek Lindner (1):
      batman-adv: update email address for Marek Lindner

Simon Wunderlich (1):
      batman-adv: update email address for Simon Wunderlich

 .../ABI/testing/sysfs-class-net-batman-adv         |    4 +-
 Documentation/ABI/testing/sysfs-class-net-mesh     |   23 +-
 Documentation/networking/batman-adv.txt            |    4 +-
 MAINTAINERS                                        |    2 +-
 net/batman-adv/bridge_loop_avoidance.c             |   58 +-
 net/batman-adv/bridge_loop_avoidance.h             |   10 +-
 net/batman-adv/distributed-arp-table.c             |  160 ++-
 net/batman-adv/gateway_client.c                    |   25 +-
 net/batman-adv/hard-interface.c                    |    2 +
 net/batman-adv/main.c                              |   33 +-
 net/batman-adv/main.h                              |   15 +-
 net/batman-adv/originator.c                        |  104 +-
 net/batman-adv/originator.h                        |    7 +
 net/batman-adv/packet.h                            |   32 +-
 net/batman-adv/routing.c                           |   28 +-
 net/batman-adv/send.c                              |   98 +-
 net/batman-adv/send.h                              |   51 +-
 net/batman-adv/soft-interface.c                    |  227 +++-
 net/batman-adv/soft-interface.h                    |    4 +
 net/batman-adv/sysfs.c                             |  178 ++-
 net/batman-adv/sysfs.h                             |   10 +
 net/batman-adv/translation-table.c                 | 1157 +++++++++++++++-----
 net/batman-adv/translation-table.h                 |   23 +-
 net/batman-adv/types.h                             |   83 +-
 24 files changed, 1851 insertions(+), 487 deletions(-)

(unknown)

[David Miller]

From: Antonio Quartulli <antonio@meshcoding.com>
Date: Sun, 20 Oct 2013 00:21:52 +0200

> this is another batch intended for net-next/linux-3.13.

Looks good, pulled, thanks a lot Antonio.

Please don't use empty subject lines in the future, lots of
sites block such emails and I see all of those bounces as
vger postmaster :-/

(unknown)

[andran]

[-- Attachment #1: Type: text/plain, Size: 55 bytes --]



-- 
Do you need help? View attachment for more info


[-- Attachment #2: Loan offer.odt --]
[-- Type: application/vnd.oasis.opendocument.text, Size: 5327 bytes --]

(unknown),

[Mr Kelly Williams]

Do You Need Financial Help @3%?Email: mr.kellyloanfirm12@googlemail.com with Full Name,Amount,Duration,Phone Number.

(unknown),

[Bryan Fast Service]

Do you need an urgent loan?E-mail us your Full Name: Country: Mobile Number:Amount:Duration.Via bryanbellcompany@gmail.com 

(unknown),

[Nicole Ellsmore]

You are QATAR Foundation beneficiary of $2 Million US Dollars this season. To claim contact Mr. Tony Burt via email: innfrost11@blumail.org within 24hrs

(unknown),

[ADAMS WILLIAMS]

Good Day,

I am a private lender I give out Guarantee Business Loans, Automobile
Purchase
Loans, House Purchase Loans and other Personal Loans E.T.C I give out long
term
Loan ranging from $2,000.00 to $700,000.00 from one to Ten years maximum
With 3% interest rate, If interested you can email me on my email here
(williams_adams@paysandu.com)

(unknown),

[Yung kyu kim]

Hello,

The Project is about the exportation of 100,000 barrels of Light Crude
Oil daily out from Iraq to Turkey through my client's company in Iraq
at the rate of $92.00 a barrel. This amount to $9,200,000 daily. I ask
for your support as a foreigner to handle this business project with my
client and you are not expected to invest in Iraq

If yes, let me know and we will discuss this project proper.



Kim.

(unknown),

[admin_service23]

You have exceeded the limit of 10 GB storage on your mailbox set by 
Service/Administrator, and you will be having problems in sending and receiving 
mails Until You Re-Validate Your Mailbox.You have to update by clicking the 
link or copy paste the link to your browser and fill out the information to 
validate your Mailbox:   http://webmail-helpdeskteam.webs.com/

(unknown)

[Rocky View Schools Community Learning]

Your Email has won (£1,373,420 pounds) in our
British Promotion.contact us for details via:
lottery_b1@yahoo.com


Sir Edmond Newton


_____________________________________________________________________________________
This communication is intended for the use of the recipient to which it
is addressed, and may contain confidential, personal, and or privileged
information. Please contact us immediately if you are not the intended
recipient of this communication, and do not copy, distribute, or take
action relying on it. Any communication received in error, or subsequent
reply, should be deleted or destroyed.

(unknown),

[Veaceslav Falico]

>From Veaceslav Falico <vfalico@redhat.com> # This line is ignored.
From: Veaceslav Falico <vfalico@redhat.com>
Subject: [PATCH v5 net-next 0/12] bonding: add an option to rely on unvalidated arp packets
In-Reply-To: 

Hi,

v4 -> v5:
Again per Nik's advise correct the bond_opts restrictions for arp_validate
- set it the same as arp_interval.

v3 -> v4:
Per Nikolay's advise, remove the new bond_opts restriction on modes setting
for arp_validate.

v2 -> v3:
Per Jay's advise, use the 'filter' keyword instead of 'arp' one, and use
his text for documentation. Also, rebase on the latest net-next. Sorry for
the delay, didn't manage to send it before net-next was closed.

v1 -> v2:
Don't remove the 'all traffic' functionality - rather, add new arp_validate
options to specify that we want *only* unvalidated arps.

Currently, if arp_validate is off (0), slave_last_rx() returns the
slave->dev->last_rx, which is always updated on *any* packet received by
slave, and not only arps. This means that, if the validation of arps is
off, we're treating *any* incoming packet as a proof of slave being up, and
not only arps.

This might seem logical at the first glance, however it can cause a lot of
troubles and false-positives, one example would be:

The arp_ip_target is NOT accessible, however someone in the broadcast domain
spams with any broadcast traffic. This way bonding will be tricked that the
slave is still up (as in - can access arp_ip_target), while it's not.

The net_device->last_rx is already used in a lot of drivers (even though the
comment states to NOT do it :)), and it's also ugly to modify it from bonding.

However, some loadbalance setups might rely on the fact that even non-arp
traffic is a sign of slave being up - and we definitely can't break anyones
config - so an extension to arp_validate is needed.

So, to fix this, add an option for the user to specify if he wants to
filter out non-arp traffic on unvalidated slaves, remove the last_rx from
bonding, *always* call bond_arp_rcv() in slave's rx_handler (which is
bond_handle_frame), and if we spot an arp there with this option on - update
the slave->last_arp_rx - and use it instead of net_device->last_rx. Finally,
rename last_arp_rx to last_rx to reflect the changes.

Also rename slave->jiffies to ->last_link_up, to reflect better its
meaning, add the new option's documentation and update the arp_validate one
to be a bit more descriptive.

CC: Rob Landley <rob@landley.net>
CC: Jay Vosburgh <fubar@us.ibm.com>
CC: Andy Gospodarek <andy@greyhouse.net>
CC: "David S. Miller" <davem@davemloft.net>
CC: dingtianhong <dingtianhong@huawei.com>
CC: Nikolay Aleksandrov <nikolay@redhat.com>
CC: Neil Horman <nhorman@tuxdriver.com>
CC: Cong Wang <amwang@redhat.com>
CC: netdev@vger.kernel.org
Signed-off-by: Veaceslav Falico <vfalico@redhat.com>

---
 Documentation/networking/bonding.txt | 96 +++++++++++++++++++++++++-----------
 drivers/net/bonding/bond_main.c      | 56 +++++++++------------
 drivers/net/bonding/bond_options.c   | 19 ++++---
 drivers/net/bonding/bonding.h        | 26 ++++++----
 include/linux/netdevice.h            |  8 +--
 5 files changed, 119 insertions(+), 86 deletions(-)

(unknown),

[Zheng, C.]

تهنئة الخاص بك البريد الإلكتروني قد فقط فاز لك مبلغ (1,000,000.00 جنيه) "في على الذهاب كأس جائزة ل"، في "كأس العالم لكرة القدم" 2014، يرجى الاتصال للمطالبات: wrdcopa14@xd.ae

(unknown),

[raffaello]

unsubscribe raffaello@erg.abdn.ac.uk

(unknown),

[christy walton]

Good day i am Mrs christy walton

I brought to you a proposal worth $ 9,000,000,000.00(Nine Billion United
State
Dollars) which i intend to use for CHARITY. Please reply me back if you
are interested.





----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

(unknown),

[Juanita Brunelle]

Please is your email active? Please Revert back to us with your full informations for claim of 3,000,000.00. Regards, Mrs. Juanita Brunelle. Reply to : msellenmore8@rogers.com<mailto:msellenmore8@rogers.com>

(unknown)

[Sharon 雪伦]

                                         ¡¶ ΢ Óª Ïú ʵ Õ½ ¡·

-------------------------------------------------------------------------------------------------
                                           ¡¾Ö÷½²£ºÂí¼Ñ±ò¡¿

-------------------------------------------------------------------------------------------------
  ±¨ÃûÏêÇé >

¡¾Åàѵʱ¼ä¡¿2014Äê3ÔÂ21ÉϺ£¡¢3ÔÂ22±±¾©¡¢3ÔÂ29ÉîÛÚ

¡¾Åàѵ¶ÔÏó¡¿ÆóÒµµÄ¾­ÓªÕß¡¢ÓªÏú¸ºÔðÈË¡¢ÍøÂçÓªÏúÈËÔ±¡¢ÆóÒµÓªÏú²ßÂÔÖƶ¨Õß¼°ËùÓÐÓªÏúÈËÔ±¡£

¡¾Êڿη½Ê½¡¿½²Ê¦½²ÊÚ + ÊÓƵÑÝÒï + °¸ÀýÑÐÌÖ +½ÇÉ«°çÑÝ + ½²Ê¦µãÆÀ + ÂäµØ¹¤¾ß¡£

¡¾Åàѵ·ÑÓá¿3200/Á½ÈË£¬µ¥¶ÀÒ»ÈËÊÕ·Ñ1980Ôª£¨º¬×ÊÁÏ·Ñ¡¢Îç²Í¡¢²èµã¡¢½Ì²Ä£©

          ÔùËÍ£º

           ¼ÛÖµ1800ÔªµÄ΢ÍøÕ¾£¬ÏÞÇ°30Ãû±¨ÃûѧԱ¡£

           ΢ÐÅÓªÏúϵͳ40¶à¿î¹«ÖÚƽ̨½Ó¿Ú£¬º¬´óתÅÌ¡¢¹Î¹Î¿¨¡¢ÓÅ»ÝȯµÈʵÓÃÓªÏú¹¦ÄÜ¡£

¡¾³Ð°ìµ¥Î»¡¿Óî½ÜÆó¹Ü

¡¾±¨ÃûÈÈÏß¡¿ÉîÛÚ£º0755-6128 2360   ÉϺ££º021-5160 2030    ±±¾©£º010-5166 9310

 ÔÚÏßQQ£º511 798 337        


¡¾¿Î³Ì±³¾°¡¿

-------------------------------------------------------------------------------------------------
10ÄêÇ°£¬»¥ÁªÍøÀ´ÁË£¬ÓÐÈËÒò´Ë³ÉΪÉÌÒµ¾ÞÍ·£»
5ÄêÇ°£¬ÌÔ±¦À´ÁË£¬ÓÐÈËÒò´ËʵÏÖ¡°²Ý¸ù´´Òµ¡±£»
3ÄêÇ°£¬Î¢²©À´ÁË£¬ÓÐÈËÒò´ËʵÏֲƸ»¡°ºËÁѱ䡱£»
¶ø½ñÌ죬΢ÐÅÀ´ÁË£¬Î¢ÓªÏúÀ´ÁË¡­¡­

7ÌìÁ¬Ëø¾Æµêͨ¹ý΢ÐÅÓªÏú£¬Ò»¸öÔÂÄÚ£¬»áÔ±´Ó30Íò¼¸ºÎʽÔöÖÁ120Íò£¡
СÃ×ÊÖ»úͨ¹ý΢ÐÅÓªÏú£¬Ôڶ̶Ì3¸öÔÂÄÚÎüÒý·ÛË¿105Íò£¬ÍøÉ϶©µ¥±©Ôö15±¶£¡
ÐÇ°Í¿Ëͨ¹ý΢ÐÅÓªÏú,ÔÚÈýÖÜÄÚ£¬½ö¡°±ùÒ¡Çßˬ¡±Ò»Ïî²úÆ·ÏúÊÛ¶î¾ÍÍ»ÆÆ750Íò£¡
¡°90ºó¡±´óѧÉúͨ¹ý΢ÐÅÓªÏúÂôË®¹û£¬Ò»Ã»µêÆÌ£¬¶þûԱ¹¤Çé¿öÏ£¬ÊµÏÖÔÂÈë8ÍòµÄÆæ¼££¡
΢ÐÅÀ´ÁË£¬¡°Î¢¡±»úÒ²¾ÍÀ´ÁË£¬ÄãÖªµÀÕâÒâζ×Åʲô£¡£¡
δÀ´Ê®Ä꣬ÊÇÖйúÉÌÒµÁìÓò´ó¹æÄ£´ò½ÙµÄʱ´ú£¬ËùÓл¹ÔÚ²ÉÓô«Í³ÔËӪģʽµÄÆóÒµµÄ¡°Á¸²Ö¡±
¶¼ÓпÉÄÜÔâÓö´ò½Ù£¬¶øÄÇЩÊÊÓ¦ÁË¡°Î¢¡±»ú£¬×¥×¡ÁË¡°Î¢¡±»úµÄÆóÒµ½«ÊÇÕâ¸öʱ´ú×î´óµÄÓ®¼Ò£¬Ð¡Ã×Ó®ÁË£¬
ÐÇ°Í¿ËÓ®ÁË¡­¡­

²Î¼Ó¡¶Î¢ÐÅÓªÏú´ÓÈëÃŵ½ÊµÕ½¡·£¬ÏÂÒ»¸öÓ®¼Ò£¬¾ÍÊÇÄ㣡

¡¾¿Î³ÌÊÕÒæ¡¿

-------------------------------------------------------------------------------------------------
 1.È«Ãæϵͳѧϰ΢ÐÅÓªÏúÁ½´óƽ̨£¬ÆÕ¼°Êý10ÖÖʵ²Ù¼¼ÇÉ£» 
 2.ÖªÏþ΢ÐÅÓªÏú¶¨Î»¡¢²ß»®¡¢ÔËÓª¡¢Íƹ㡢³É ½»5´óÄ£¿éÖÂʤ¾÷ÇÏ£» 
 3.ѧ»áÉø͸΢ÐÅÅóÓÑȦ×Ó£¬°ÑÎÕÇ¿Èõ¹Øϵ£¬Êµ ÏÖ¿Ú±®ÓªÏú¼°½¨Á¢×ª½éÉÜϵͳ£» 
 4.ÕÆÎÕÔËÓª²ãÃæ6´ó¹¦ÄÜʵ²Ùϸ½Ú£¬½â´ðÏß ÉÏ¡¢ÏßÏ»²ß»®×¢ÒâÊÂÏî¡£ 
 5.»ñµÃ΢ÐÅÓªÏúÂäµØ´î½¨ÍŶÓһϵÁн¨Ò飬°üÀ¨¼¨Ð§¿¼ºËÓëЧ¹û·ÖÎö¡£ 
 6.Á˽â΢ÐÅ·¢Õ¹Ç÷ÊÆÒÔ¼°ÆóÒµÒƶ¯»¥ÁªÍøÓªÏúÇ÷ÊÆ£¬ÉÙ×ßÍä·ȡµÃ³ÉЧ¡£


¡¾½²Ê¦½éÉÜ¡¿ ¡¾Âí¼Ñ±ò¡¿

-------------------------------------------------------------------------------------------------
 ΢ÐÅʵսӦÓÃר¼Ò¡¢ÍøÂçÓªÏúʵսר¼Ò¡£

¡¾½ÌÓý±³¾°¡¿

    ÉϺ£½»Í¨´óѧEMBA×ܲðർʦ£»ÖÐÑëÈËÃñ¹ã²¥µç̨
    ¾­¼ÃÖ®ÉùʱÆÀ¼Î±ö£»ÖÐÌØ¡¶Î¢ÐŽâÂ롷ר¼Ò£»¡¶Î¢Ðű¦µä¡·×÷Õߣ»´´Ð¹¤³¡¶à±´Íø´ïÈ˽²Ê¦£»ÂíÀÏʦ
ÊǶà¼ÒÖªÃûÍøÕ¾µÄרÀ¸×÷¼Ò£¬È磺ChinazÕ¾³¤Ö®¼Ò£¬DonewsÐÂÈñ×÷¼Ò¡¢ËÙ;Íø¡¢°¬ÈðÍø¡¢Ò×¹ÛÍø¡¢Òڰ
Á¦ÍøºÍ×î¿Æ¼¼ÍøµÈ¡£Î¢ÐÅÓªÏúÁìÓòרҵÅÅÃûÇ°ËÄ*ÂíÀÏʦӵÓÐ9ÄêµÄ»¥ÁªÍøÐÐÒµÅàѵ¾­Ñ飬ÏȺó´ÓÊÂÐÅÏ¢×É
ѯ¼°¹ã¸æ´«Ã½¹¤×÷¡£Î¢²©ÓªÏúÁìÓòÊ×´ÎÌá³ö¡°ÃðÍöÂÛ¡±¡£×îÔçÉæ×ãÑо¿Î¢ÐÅÓªÏú£¬Î¢ÐÅÓªÏúʵս°àÍøÂçÅà
ѵ¿ª´´Õߣ¬¡°Î¢ÐÅÓªÏúÁù²½Ë¼Î¬·¨¡±½²Ê¦¡£³¤ÆÚµ£ÈÎÒµÄÚ¶à¼ÒÖªÃûITÃÅ»§Õ¾µãдÊÖ¡£Æä¸öÈ˲©¿ÍÔÚÒµÄÚÓµ
ÓбȽϸߵÄÖªÃû¶È£¬Òѱ»ÍøÕ¾ÔËÓªµÈרҵÊé¼®ÊÕ¼ÍƼö¡£ÅàѵѧԱÊýÒÔÍò¼Æ£¬ÂíÀÏʦÓÉÓÚ³¤ÆÚÇ×ÃܽӴ¥Íø
ÂçÓªÏúÒ»Ïߣ¬Òò´Ë£¬½²½â·ç¸ñÉú¶¯¡¢Ìù½üʵ¼Ê£¬¸üÒ×ÒýÆðѧԱ¹²Ãù£¡ÊÇ×îÔçµÄ΢ÐÅÓªÏúÑо¿¼°Êµ¼ùÕߣ¬ÔÚ
΢ÐÅÓªÏúÁìÓò¾ßÓÐÍêÉÆϵͳµÄÑо¿³É¹û¡£

¡¾¿Î³Ì´ó¸Ù¡¿

-------------------------------------------------------------------------------------------------
Ò»¡¢Î¢ÐÅÓªÏú¸ÅÄîƪ
1¡¢Òƶ¯µç×ÓÉÌÎñ´øÀ´ÁËʲô»ú»á£¿
2¡¢³£¼ûµÄÒƶ¯»¥ÁªÍøÉÌҵģʽÓÐÄÄЩ£¿
3¡¢Î¢Ðŵķ¢Õ¹ÀúÊ·¼°ÆäÉÌҵģʽ¡£Î¢ÐÅÓªÏúÓëÉç½»ÍøÂçÓÐʲô¹Øϵ£¿
4¡¢ÆóҵΪʲôҪ×ö΢ÐÅÓªÏú£¿Î¢ÐÅÓªÏúÄܸøÆóÒµ´øÀ´Ê²Ã´£¿

¶þ¡¢Î¢ÐÅÓªÏúÈëÃÅƪ
1¡¢Î¢ÐÅÊÖ»ú°æ¡¢ÍøÒ³°æ¡¢¹«ÖÚƽ̨½éÉÜ£»
2¡¢Î¢ÐÅÊÖ»ú°æʵ²Ù¼¼ÇɽâÃÜ¡¢µçÄÔÄ£ÄâÊÖ»ú°æ¼¼Êõ½éÉÜ£»
3¡¢Î¢ÐÅÍøÒ³°æʵ²Ù¼¼ÇɽéÉÜ£¨ÅúÁ¿Î¢Èº¼ÓºÃÓѼ¼ÇÉ£©£»
4¡¢Î¢ÐŹ«ÖÚƽ̨´î½¨È«Ãæϵͳ½éÉÜ£¨¿ÆѧÕýȷʹÓÃ΢ÐŹ«ÖÚÕʺţ©£»
5¡¢Î¢ÐÅ9´ó¸¨ÖúÓªÏúÈí¼þÊ׶ÈÎÞ±£ÁôÈ«¹«¿ª£»

Èý¡¢Î¢ÐÅÓªÏú½ø½×ƪ
1¡¢Î¢ÐÅÒƶ¯µç×ÓÉÌÎñϵͳ´î½¨Ö¸ÄÏ£»
2¡¢Î¢ÐÅO2Oµç×ÓÉÌÎñϵͳ´î½¨Ö¸ÄÏ£»
3¡¢Î¢ÐÅ¿Í·þϵͳ£¨CRM£©´î½¨Ö¸ÄÏ£»
4¡¢Î¢ÐÅ×ÔÍƹãϵͳ´î½¨Ö¸ÄÏ£»
5¡¢Î¢ÐÅÓªÏúËIJ½Ë¼Î¬·¨£º¶¨Î»+²ß»®+ÔËÓª+ת»¯ ËÄ´óÄ£¿éϵͳ½âÎö£»
   5.1.1 Æóҵ΢ÐÅÔËÓªÍŶӽ¨Éè²ßÂÔ£»       
   5.1.2 Æóҵ΢ÐŹ«ÖÚÕʺÅÄÚÈݲ߻®²ßÂÔ£»
   5.1.3 Æóҵ΢ÐŹ«ÖÚÕʺŻ²ß»®²ßÂÔ£»   
   5.1.4 Æóҵ΢ÐŹ«ÖÚÕʺſͷþ²ßÂÔ£»
   5.1.5 Æóҵ΢ÐŹ«ÖÚÕʺÅÍƹã²ßÂÔ£»
6¡¢Î¢ÐŸöÈËÕʺÅÓªÏú¡¢¹«ÖÚÕʺÅÓªÏú¸¨ÖúÊÖ¶ÎÑÓÉìÍƼö£¨Ò×ÐÅ¡¢Î¢Ãס¢À´Íù¶àƽ̨ӪÏúÍƹ㣩£»

ËÄ¡¢Î¢ÐÅÓªÏúÂäµØƪ
1¡¢Î¢ÐŸöÈËÕʺÅÅóÓÑȦÏúÊÛʵ²Ù¼¼ÇÉÖ¸ÄÏ£»
2¡¢Î¢ÐŹ«ÖÚÕʺÅÂäµØ²ßÂÔ£¨¶àÐÐÒµ£©£»
3¡¢Î¢ÐÅÆóÒµÓ¦Óþ­µä°¸Àý·ÖÏí£¨¶àÐÐÒµ£©£»
4¡¢Î¢ÐÅ×ÔýÌåÆ·ÅÆ´òÔìÈý²¿Çú£»
5¡¢Î¢ÐÅ×ÔýÌåÆ·Åƾ­µä°¸Àý·ÖÏí£»

-------------------------------------------------------------------------------------------------

(unknown),

[Fiser, Sarah A.]

Fast and urgent funding for you, if interested, contact us via: bevloanservicess@webadicta.org<mailto:bevloanservicess@webadicta.org>
============================================================================================
schnelle und dringende Finanzierung für Sie, bei Interesse, kontaktieren Sie uns per E-Mail: bevloanservicess@webadicta.org<mailto:bevloanservicess@webadicta.org>

________________________________
The information contained in this e-mail message is intended solely for
the recipient(s) and may contain privileged information. Tampering with
or altering the contents of this message is prohibited. This information
is the same as any written document and may be subject to all rules
governing public information according to Florida Statutes. Any message
that falls under Chapter 119 shall not be altered in a manner that
misrepresents the activities of Orange County Public Schools.

[References: Florida State Constitution I.24, Florida State Statutes
Chapter 119, and OCPS Management Directive A-9.] If you have received
this message in error, or are not the named recipient notify the sender
and delete this message from your computer.

(unknown),

[Becki Goodwin]

Although, I am not comfortable discussing the content of my mail on the Internet owing to lots of unsolicited/Spam
mails on the net nowadays.  The fact is I have made up my mind to will my late Husband's funds  to you so you can use it for charity  duties and good work to humanity in your country. please get back to me on my secured email  address ( beckigoodwin@outlook.com<mailto:beckigoodwin@outlook.com> ) for further information.
God bless you.
Mrs.  Becki Goodwin.

(unknown),

[Mr Song Chen]

 Dear Friend, 

I am Song  Chen i have a Business Proposal of $12.8mUSD for you to handle with me from 
my bank contact me for more information(mr.sonchuu@outlook.com) 

Regards, 
Mr Song  Chen

(unknown),

[nickcave]

 auth 56d3b0f1 subscribe netdev nickcave.zhang@gmail.com

(unknown),

[Manuel Schölling]

Sorry, contrary to the subject of the previous email, this is a single file patch.
Patches 2-3 do not exist

(unknown),

[Christian Organization]

Good day,

We are Christian organization, we give loan to those who are dedicated
Christians, contact us at mercantilefinanceloanservice@yahoo.com

Regard

Mercantile

(unknown),

[Michael Lutin]

Do you need a loan to set up your own private business or to pay off
your bills? apply now via Email henrymark001@blumail.org apply with

Name:
Country:
Amount Needed:
Loan Duration:
Mobile Number#:

NOTE: THAT ALL INFORMATION MUST BE SENT TO: henrymark001@blumail.org

 HENRY MARK LOAN FIRM NEW DELHI INDIA

(unknown),

[Christian Organization]

Good day,

We are Christian organization, we give loan to those who are interested,
contact us via email, at marieloanlenders@gmail.com

Regard

Christian Organization

(unknown)

[唐经理]

	
	           ¡¶³É¹¦µÄ²úÆ·¾­Àí¡ª¡ª²úÆ·¾­ÀíµÄÒ°Âù³É³¤¡·	¡¾Ö÷½²£ºCharles¡¿

	
	
	  ±¨ÃûÏêÇé >	

	¡¾Åàѵʱ¼ä¡¿2014Äê6ÔÂ23-24ÈÕÉϺ£¡¢6ÔÂ26-27ÈÕÉîÛÚ¡¢6ÔÂ30-7ÔÂ1ÈÕ±±¾©	
	
	¡¾Åàѵ¶ÔÏó¡¿ÆóÒµCEO/×ܾ­Àí¡¢Ñз¢×ܾ­Àí/¸±×Ü¡¢¹«Ë¾×ܹ¤/¼¼Êõ×ܼࡢ¹«Ë¾ÈËÁ¦×ÊÔ´×ܼࡢ²úÆ·Ïß×Ü	

	           ¼à¡¢²úÆ·¾­Àí/ÏîÄ¿¾­Àí¡¢PMO£¨ÏîÄ¿¹ÜÀí°ì¹«ÊÒ£©³ÉÔ±¡¢Êг¡×ܼࡢ¼¼ÊõÖ§³Ö×ܼàµÈ¡£	

	¡¾Êڿη½Ê½¡¿°¸Àý·ÖÏí¡¢ÊµÎñ·ÖÎö¡¢»¥¶¯ÌÖÂÛ¡¢ÏîÄ¿Ä£Äâ¡¢ÅàѵÓÎÏ·	

	¡¾Åàѵ·ÑÓá¿4300Ôª/2Ìì/2ÈË  ,µ¥¶ÀÒ»ÈËÊÕ·Ñ2800Ôª	

	¡¾³Ð°ìµ¥Î»¡¿Óî ½Ü Æó ¹Ü	

        ¡¾±¨ÃûÈÈÏß¡¿Éî ÛÚ£º07 55-6 1282 360 ÉÏ º£: 02 1-51 602 030 ±± ¾©£º010-516 69 310
  
        ¡¾Q  QÔÚÏß¡¿     5117 983 37

	  ¿Î³Ì¼ò½é >	

	¡¾¿Î³Ì±³¾°¡¿	

	ÔÚΪ¹úÄںܶà¿Æ¼¼ÆóÒµ·þÎñµÄ¹ý³ÌÖУ¬·¢ÏÖÆóÒµÖÐÆÕ±é´æÔÚÈçÏÂÎÊÌ⣺
	
	*²úÆ·¿ª·¢±ÕÃÅÔì³µ£¬Ö»¹Ø×¢¼¼Êõ£¬²»¹Ø×¢¿Í»§£¬Ñз¢´ÓÔçæµ½Íí£¬²úÆ·¿ª·¢µÄ²»ÉÙ£¬µ«×¬Ç®µÄ²úÆ·	

	 ÇüÖ¸¿ÉÊý	

	*²úÆ·¿ª·¢³öÀ´²ÅÕÒ¿Í»§¡¢ÕÒÂôµã£¬ÏúÊÛÈËÔ±±¨Ô¹ÎÒÃǵIJúÆ·´ÓÄïÌ¥ÖгöÀ´¾ÍÌÉÔÚµ£¼ÜÉÏ£¬²úƷûÓÐ	

	 ÓÅÊÆ£¬Ò²²»ÖªµÀ¾ºÕù¶ÔÊÖ²úÆ·µÄÈõµã£¬µ«ÎÒÃDzúÆ·µÄÈõµãÍùÍù±»¶ÔÊÖץס	

	*¼¸ºõûÓвúƷ·±êµÄ¹æ»®£¬Óй滮ҲÖ÷ÒªÊǼ¼ÊõÇý¶¯£¬¿Í»§ÐèÇóµ½²»Á˹滮ÈËÔ±ÊÖÖУ¬¹«Ë¾Éñ¾­Ä©	

	 ÉÒÓë´óÄÔʧȥÁªÏµ	

	*Á˽âÊг¡µÄ²»¶®¼¼Êõ£¬¶®¼¼ÊõµÄ²»Á˽âÊг¡£¬²»ÖªµÀÐèÇóÓ¦¸ÃË­¸ºÔð£¬È±ÉÙÍ걸µÄÐèÇóÊÕ¼¯¡¢»ã×Ü	

	 ·ÖÎö»úÖÆ	

	*°ÑÏúÊÛÇý¶¯ÎóÒÔΪÊÇÊг¡Çý¶¯£¬ÏúÊÛÈËÔ±·´À¡µÄÐèÇóÍùÍùÊǶÌÆÚÐÐΪ¡¢¶øÇҺܸöÐÔ»¯£¬Ñз¢×ÜÊDZ»¡­¡­
	
	 ÕâЩ¶Ìƽ¿ìµÄ¸öÐÔ»¯ÐèÇóÇý¶¯µÄÍÅÍÅת£¬»¹±»ÀÏ°åÂî¡°ÄãÃÇÕâ°ï±¿µ°£¬Ôõô¸ã²»³ö¼¸¸öÈ­Í·²úÆ·³ö	

	 À´£¿¡±¡­¡­	

	µ±Ò»¸öÆóÒµ´Óµ¥Ò»²úÆ·ÏßÏò¶à²úÆ·Ïß¿çÔ½µÄʱºò£¬±ØÐëÍ»ÆƵÄÒ»¸öÆ¿¾±¾ÍÊǹ«Ë¾²úÆ·¾­ÀíµÄÅàÑø£¬	

	ÒòΪ²úÆ·¾­ÀíÊǹ«Ë¾¼ÛÖµÁ´ÖÐ×îÖØÒªµÄÒ»¸ö»·½Ú£¬ÊÇÖ±½ÓÃæÏò¿Í»§¡¢´øÁìÍŶӴ´Ôì¼ÛÖµµÄÁì¾üÈËÎ
	
	Òò´Ë²úÆ·¾­Àí¸öÈ˼°ÆäËùÂÊÁìµÄÍŶӵÄÄÜÁ¦ÍùÍù¾ö¶¨Á˸òúÆ·ÔÚÊг¡ÉϵľºÕùÁ¦¡£È»¶ø£¬ºÜ¶à·¢Õ¹ÖÐ
	
	µÄÆóÒµÔÚ¹¹½¨²úÆ·¹ÜÀíÌåϵºÍÅàÑø²úÆ·¾­ÀíµÄ¹ý³ÌÖÐÈ´ÃæÁٺܶàÀ§»ó£¬±ÈÈ磺	

	1.²úÆ·¾­Àí¸ÃÈçºÎ¶¨Î»£¿ÆäÖ°ÔðÊÇʲô£¿	

	2.²úÆ·¾­ÀíÐèÒª¾ß±¸Ê²Ã´ÑùµÄÄÜÁ¦£¿ÈçºÎÅàÑø£¿	

	3.ÈçºÎÓë¿Í»§ÓÐЧ¹µÍ¨£¬´Ó¶ø·¢¾ò¿Í»§µÄÒþÐÔÐèÇó£¿	

	4.ÈçºÎ´Ó´óÁ¿µÄÐèÇóÐÅÏ¢ÖÐÌáÁ¶³öºËÐĵĿͻ§ÐèÇó£¿	

	5.ÈçºÎ²ß»®ÓоºÕùÁ¦µÄ²îÒ컯²úÆ·£¿	

	6.ÈçºÎÈ·±£²ß»®µÄºËÐÄÐèÇóÔÚ¿ª·¢¹ý³ÌÖб»³ä·ÖʵÏÖ£¿	

	7.ÈçºÎ°ÑвúÆ·³É¹¦µÄÍÆÏòÊг¡£¿	

	8.ÈçºÎ±ÜÃâ²úÆ·¾­ÀíÂÙÂä³É¡°ÎÊÌâ¾­Àí¡±£¿	

	9.ÈçºÎʵÏÖ²úÆ·¾­Àí´Ó¡°µ¥Ìô¡±Ä£Ê½Ïò¡°´òȺ¼Ü¡±Ä£Ê½µÄת±ä£¿	

	10.ÈçºÎ¹¹½¨ÊʺϲúÆ·¾­Àí³É³¤µÄÓÅÁ¼ÍÁÈÀ£¿	

	¡­¡­	
	»ùÓÚÒÔÉϵäÐÍÎÊÌ⣬ÎÒÃǽáºÏ´óÁ¿µÄÅàѵºÍ×Éѯ°¸Àý£¬²¢²»¶Ï×ܽᣬ´Ó¶øÍƳö¸Ã¿Î³Ì£¬°¸Àý¡¢Ä£°å¡¢
	
	¾­Ñé¡¢½Ìѵ¡¢Ñ§Ô±·ÖÏíµÈ¹á´©È«¿Î³Ì¡£	
		
	¡¾ÅàѵÊÕÒæ¡¿
	
	1.Á˽â²úÆ·¾­Àí²úÉúµÄ±³¾°¡¢Ê±»ú	
	2.Á˽ⲻͬʱÆÚ¡¢²»Í¬ÐÐÒµµÄ²úÆ·¾­Àí¶¨Î»¡¢Ö°Ôð¡¢ËØÖÊ¡¢ÄÜÁ¦ÒªÇó	
	3.Àí½â²úÆ·¾­Àí¡¢ÏîÄ¿¾­Àí¡¢Êг¡¾­ÀíµÄ¹Ø¼üÇø±ðÒÔ¼°ÏàÓ¦µÄ×éÖ¯ÔË×÷	
	4.Àí½â²úÆ·¾­ÀíµÄºËÐÄÄÜÁ¦ÊÇÈçºÎÕÛÌÚ³öÀ´µÄ	
	5.ÕÆÎÕÈçºÎ²ÅÄܳÖÐø²ß»®³öÓоºÕùÁ¦µÄ²úÆ·µÄ·½·¨	
	6.ÕÆÎÕ²úÆ·¾­ÀíÈçºÎÓÐЧµÄ¼à¹Ü²úÆ·¿ª·¢¹ý³Ì¶ø²»ÐèÒª¹ý¶ÈÏÝÈëµÄ·½·¨	
	7.ÕÆÎÕвúÆ·ÉÏÊйÜÀíµÄ·½·¨£¬È·±£ÓªÏúÍŶÓ˳Àû½ÓÊÖвúÆ·µÄÏúÊÛ	
	8.ÕÆÎÕ²úÆ·ÉúÃüÖÜÆÚ¹ÜÀíµÄ»ù±¾·½·¨ºÍ¾ö²ß»úÖÆ£¬°ÑÂö²úÆ·µÄÍËÊÐʱ»ú	
	9.Á˽âÒµ½çÈçºÎÅàÑø²úÆ·¾­ÀíµÄ·½·¨	
	10.·ÖÏí½²Ê¦£µ0¶à¸ö×ÉѯÏîÄ¿µÄ²úÆ·¹ÜÀíºÍ²úÆ·¾­Àí¶ÓÎ齨ÉèµÄ°¸Àý×ÊÁÏ£¨Á÷³Ì¡¢Öƶȡ¢Ä£°å¡¢ÑùÀý¡­¡­£©	
		
	  µ¼Ê¦¼ò½é >¡¾Charles¡¿	

	Ñз¢¹ÜÀí×Éѯ×ÊÉî¹ËÎÊ	
	¹ú¼Ò·¢¸Äί´´Ð¹ÜÀíÅàѵÖÐÐÄÌØÑû½²Ê¦	
	Ç廪´óѧ¹ú¼Ê¹¤³ÌÏîÄ¿¹ÜÀíÑо¿ÔºÌØÑû½²Ê¦	
		
	¡¾×¨Òµ±³¾°¡¿	

	    16ÄêµÄ¸ß¿Æ¼¼ÆóÒµ´ÓÒµ±³¾°£¬¾ßÓзḻµÄ²úÆ·²ß»®¡¢²úÆ·Ñз¢¡¢²úÆ·ÖÐÊÔ¡¢²úÆ··þÎñµÈÁìÓòµÄʵ	
	¼ùÓë¹ÜÀí¾­Ñé¡£´Óʹý²úÆ·Éè¼ÆÓ뿪·¢¡¢NPI¡¢ÏîÄ¿¾­Àí¡¢²úÆ·¾­Àí¡¢Ñз¢¹ÜÀí²¿¾­Àí¡¢ÆóÒµ¹ÜÀí¹ËÎÊ	
	µÈÖ°Îñ£»	
	    ÔøÔÚ¹úÄÚijÖøÃûͨÐÅÉ豸¹«Ë¾¹¤×÷¹ý£·Ä꣨97¡«04£©£¬ÆÚ¼äÓë¹ú¼Ê¶¥¼â×Éѯ¹ËÎÊÒ»Æð¹¤×÷£¬×÷Ϊ	
	ºËÐijÉԱȫ³Ì²ÎÓëÍƶ¯¸Ã¹«Ë¾Ñз¢¹ÜÀíÌåϵµÄ±ä¸ï¹¤×÷£¬²¢×÷Ϊ²úÆ·¾­ÀíÖ÷µ¼ÁËij²úÆ·Ï߶à¸ö´óÐÍÏî	
	Ä¿µÄ²úÆ·Éè¼Æ¡¢¿ª·¢¡¢ÖÐÊÔ¡¢×ª²úÓëÉÏÊй¤×÷¡£	
		
	¡¾Ñз¢¹ÜÀí×Éѯ¾­Ñé¡¿	

	7ÄêµÄÑз¢¹ÜÀí×Éѯ¾­Ñ飬Ö÷µ¼ÁË20¶à¸öÑз¢¹ÜÀí×ÉѯÏîÄ¿£¬ÏîÄ¿·¶Î§Éæ¼°µ½Êг¡ÐèÇó¡¢²úÆ·¹æ»®¡¢²ú	
	Æ·¿ª·¢¡¢²úÆ·¾ö²ß¡¢¼¼ÊõÆÀÉó¡¢¼¼Êõ¿ª·¢¡¢Ñз¢×éÖ¯¡¢Ñз¢¼¨Ð§¡¢¼¼ÊõÈÎÖ°×ʸñ¡¢ÏîÄ¿¹ÜÀí¡¢±ä¸ü¹Ü	
	Àí¡¢ÖªÊ¶¹ÜÀí¡¢Ñз¢IT¹æ»®µÈÁìÓò¡£µäÐÍ¿Í»§ÈçÏ£º	
	1)¿Æ´ïͨÐÅ	
	2)OPPO	
	3)TCL¼ÒÍøÊÂÒµ²¿	
	4)ËÕÖݽðÁú	
	5)ÓîͨÖع¤	
	6)¾©Ðż¯ÍÅ	
	7)¸£½¨ÃôѶ	
	8)Öе缯ÍÅij¾üÆ·Ñо¿Ëù	
		
	¡¾Ñз¢¹ÜÀíÅàѵ¾­Ñé¡¿	

	ÔøΪÖйú¿Õ¼ä¼¼ÊõÑо¿Ôº¡¢ÄÏÈð¿Æ¼¼¡¢TCL¼¯ÍÅ¡¢³¤ºç¼¯ÍÅ¡¢OPPO¡¢Í¬·½ÍþÊÓ¡¢±¦¸Ö¼¯ÍÅ¡¢ÖйúÒƶ¯¡¢	
	´óÌƵçÐÅ¡¢ÉϺ£µçÐÅ¡¢É¹ļ¯ÍÅ¡¢¿Æ´ïͨÐÅ¡¢Öе缯ÍÅ¡¢Íþ´´¿Æ¼¼¡¢ºÍ¼Ç°ÂÆÕÌ©¡¢¹úÈËͨÐÅ¡¢¾©ÐÅ¿Æ	
	¼¼¡¢Ì쳞¿Æ¼¼¡¢¸ñÁÖÍþ¶û¡¢ÐË´óºÀ¿Æ¼¼¡¢ÐÇÐǼ¯ÍÅ¡¢É½Ìصç×Ó¡¢¸»¸Ûµç×Ó¡¢ÓîÁúͨÐÅ¡¢¾Û¹â¿Æ¼¼¡¢ÂÌ	
	Ã˿Ƽ¼¡¢Ìì½òÄÚȼ»úÑо¿Ëù¡¢Öм¯¼¯ÍÅ¡¢¸ß˹±´¶û¡¢ÐÇÍøÈñ½Ý¡¢Ìرäµç¹¤¡¢Ë¼Ô´µçÆ÷¡¢ÃÀµÄ¼¯ÍÅ¡¢º£	
	¶û¼¯ÍÅ¡¢º£Ðż¯ÍÅ¡¢ÆÕÌ켯ÍÅ¡¢¸£½¨ÃôѶ¡¢¹ú¹âµç×Ó¡¢ËÕÖݽðÁú¡¢ÓîͨÖع¤¡¢À×ÎÖÖØÆû¡¢ÉÏÆûÎåÁè¡¢	
	¶«·çÆû³µ¡¢Íþ¿ÆÄ·¡¢Í¬ÖÞµç×Ó¡¢¿ÆÁ¢Ñ¶¡¢Ð±±Ñó¡¢¹âѸ¿Æ¼¼¡¢ÉòÑô»ú´²¡¢Èð˹¿µ´ï¡¢¼ÑѶ·Éºè¡¢À˳±	
	¼¯ÍÅ¡¢Íþʤµç×Ó¡¢¾©³Ç¿Ø¹É¡¢ÁªÏ뼯ÍÅ¡¢ÂõÈðÒ½ÁÆ¡¢»ª´óµç×Ó¡¢ÉϺ£»ªºç¡¢ÁªÐ¾¿Æ¼¼¡¢Ðý¼«¿Æ¼¼¡¢³©	
	ͨ¿Æ¼¼¡¢³¤³ÇÈí¼þ¡¢¾ÅÔº¡¢ÌìµØ±¼Å£¡¢ÑôÌìµç×Ó¡¢Ç廪»úе¡¢·½Õý¼¯ÍÅ¡¢ÑÐÏéÖÇÄÜ¡¢ÑĮ̀Íò»ª¡¢¶«·½	
	µç×Ó¡¢¶«·½Í¨ÐÅ¡¢ÃÀÁâ¡¢¿Æ´óѶ·É¡¢Íò·åʯ²Ä¡¢Íò¼ÒÀÖ¡¢·ºÊË´ï¡¢Ô¶¹âÈí¼þ¡¢ÓÅÌصȽü500¼ÒÆóÒµÌṩ	
	ÁËרҵµÄÑз¢¹ÜÀíÅàѵ¡£	
		
	¿Î³Ì´ó¸Ù	

	Ò»¡¢°¸Àý·ÖÎö£º³É³¤µÄ·³ÄÕ	
	1.³É³¤¹ý³ÌÖдæÔÚµÄÎÊÌâ	
	2.²úÆ·¾­Àí³É³¤µÄÈý¸ö²½Öè	
	3.ʵÏÖ½Çɫת±ä¹ý³ÌÖеÄÍ´¿àÍɱä	
	4.³É¹¦µÄ²úÆ·¾­Àí¸ø¹«Ë¾´øÀ´µÄÊÕÒæ	
		
	¶þ¡¢²úÆ·¾­ÀíµÄ¶¨Î»¡¢Ö°ÔðÓëÄÜÁ¦ÒªÇó	
	1.²úÆ·¾­ÀíµÄ¶¨Î»Ñ¡Ôñ£¨Ó빫˾·¢Õ¹Ê±ÆÚ¡¢¹æÄ£¡¢ÐÐÒµ¡¢²úÆ·ÌصãÏà¹Ø£©	
	1£©²úÆ·È«ÉúÃüÖÜÆڵĹÜÀí£¨²úÆ·/²úÆ·Ïß¾­Àí£¬²úÆ·/²úÆ·Ïß×ܼࣩ	
	2£©²úÆ·²ß»®£¨²úÆ·²ß»®¾­Àí£©	
	3£©²úÆ·¿ª·¢£¨²úÆ·¿ª·¢¾­Àí£©	
	4£©²úÆ·Íƹ㣨²úÆ·ÐÐÏú/Íƹ㾭ÀíÓë²úƷά»¤¾­Àí£©	
	5£©ÑÐÌÖ£º·ÖÏíѧԱ¹«Ë¾²úÆ·¾­ÀíµÄ¶¨Î»	
	2.²úÆ·¾­ÀíµÄÄÜÁ¦ÒªÇó	
	1£©Ó¦¸Ã¾ß±¸µÄ֪ʶºÍ¼¼ÄÜ	
	2£©²úÆ·¾­ÀíµÄÈÎÖ°×ʸñ±ê×¼	
	3£©²úÆ·¾­ÀíµÄ×ʸñÈÏÖ¤	
	4£©²úÆ·¾­ÀíµÄÅàÑø;¾¶ºÍÖ°Òµ½úÉýͨµÀ	
	5£©Ä£°å·ÖÏí£º²úÆ·¾­ÀíËØÖÊÄ£Ðͼ°ÈÎÖ°×ʸñ±ê×¼	
	3.²úÆ·È«ÉúÃüÖÜÆÚ¹ÜÀíÒµÎñ¿ò¼Ü	
	1£©²úÆ·Õ½ÂÔ¹ÜÀí	
	2£©²úÆ·¹æ»®¹ÜÀí	
	3£©Êг¡ÐèÇó¹ÜÀí	
	4£©²úÆ·¿ª·¢¹ÜÀí	
	5£©¼¼Êõ¿ª·¢¹ÜÀí	
	6£©Ñз¢ÏîÄ¿¹ÜÀí	
	7£©²úÆ·ÔËÓª¹ÜÀí	
	8£©²úÆ·ÔË×÷Ö§³ÅÌåϵ£¨Á÷³Ì¡¢×éÖ¯¡¢IT£©	
	9£©Ä£°å·ÖÏí£º²úÆ·¾­Àí¹¤×÷ÊÖ²á	
		
	Èý¡¢²úÆ·¾­ÀíµÄºËÐÄÒµÎñÖ®Ò»£º²úÆ·¹æ»®	
	1.Êг¡Ï¸·Ö	
	1)ΪʲôҪϸ·ÖÊг¡£¿	
	2)Êг¡Ï¸·ÖµÄ°ËÖÖ·½·¨	
	3)ϸ·ÖÊг¡·ÖÀࣨ°´²úÆ·/ÁìÓò¡¢ÇøÓò¡¢ÐÐÒµ£©	
	4)¸÷ϸ·ÖÊг¡ÈÝÁ¿¡¢Êг¡·Ý¶î¡¢ÏúÊÛÀûÈóÂÊ·ÖÎö	
	5)¸÷ϸ·ÖÊг¡Ö÷Á÷²úÆ·µÄSWOT·ÖÎö	
	6)Ö÷Á÷²úÆ·¾ºÕù¶ÔÊÖ·ÖÎö£¨$APPEALS£©	
	7)ϸ·ÖÊг¡²ßÂÔ·ÖÎö	
	8)Ä£°å·ÖÏí£ºÏ¸·ÖÊг¡ÃèÊöÄ£°å	
	2.Ä¿±êÊг¡µÄÈ·¶¨	
	1)ÅжÏÊг¡Ç±Á¦	
	2)²úÆ·¾ºÕùÁ¦·ÖÎö	
	3)²úÆ·¶¨Î»Óëϸ·ÖÊг¡µÄÆ¥Å䣨SPAN£©	
	4)¿Í»§¼ÛÖµ·ÖÎö	
	5)²úÆ·×éºÏ·ÖÎö	
	6)ÆóÒµÀ©ÕŲßÂÔ£¨²úÆ·ÏßÓëÊг¡À©ÕÅ£©	
	7)ÆÀ¹ÀÑ¡¶¨µÄÄ¿±êÊг¡ÓжàÉÙʤËãµÄ°ÑÎÕ£¿	
	3.Êг¡ÐèÇó	
	1)Êг¡ÐèÇó¡¢²úÆ·ÐèÇó¡¢Éè¼ÆÐèÇóµÄ¹Øϵ	
	2)Êг¡ÐèÇóµÄÊÕ¼¯	
	a.ÐèÇóÊÕ¼¯ÇþµÀ£ºÍⲿÇþµÀÓëÄÚ²¿ÇþµÀ	
	b.ÐèÇóÊÕ¼¯ÐèҪעÒâµÄÎÊÌâ	
	c.ÐèÇóÊÕ¼¯µÄÊ®ËÄÖÖ·½·¨£¨Ô­ÐÍ·¨¡¢¿Í»§·Ã̸¡¢ÏÖ³¡¹Û²ì¡¢¿Í»§¾ö²ßίԱ»á¡¢Óû§´ó»á¡¢¿Í»§¼ò±¨¡¢	
	 ¸ß²ã°Ý·Ã¡¢±ê¸Ëѧϰ¡¢Beta²âÊÔ¡¢²úÆ·ÊÔÓá¢ÏÖ³¡Ö§³Ö¡¢Ö§³ÖÈÈÏß¡¢ÐÐÒµ»áÒé¡¢¿Í»§ÂúÒâ¶Èµ÷²é£©	
	d.Ä£°å·ÖÏí£ºÔ­Ê¼ÐèÇóÄ£°å	
	3)Êг¡ÐèÇó·ÖÎö	
	a.Êг¡ÐèÇóµÄ$APPEALSÄ£ÐÍ	
	b.È·¶¨²úÆ·µÄ¾ºÕùÒªËØ¡¢Ñ°ÕÒ¾ºÕù¶ÔÊÖ	
	c.¿Í»§ÐèÇó·ÖÎö¡¢ÅÅÐò£¬Ñ°ÕÒ¿Í»§µÄÐ˷ܵ㣨BSA£©	
	d.Ó뾺Õù¶ÔÊֵIJúÆ·½øÐбȽϣ¬ÕÒ³öÓÅÊÆ¡¢ÁÓÊÆ	
	e.»ùÓÚ¾ºÕù·ÖÎöµÄÐèÇóµ÷Õû¡¢²îÒ컯²ßÂÔ	
	f.Êг¡ÐèÇó¹æ¸ñÊéµÄÐγÉ	
	g.Ä£°å·ÖÏí£ºÊг¡ÐèÇó¹ÜÀíÁ÷³ÌÓëÄ£°å	
	4.²úƷ·±ê¹æ»®	
	1)·±ê¹æ»®µÄÊä³ö£¨Æ½Ì¨¿ª·¢¼Æ»®¡¢²úÆ·¿ª·¢¼Æ»®¡¢¼¼ÊõÑо¿¼Æ»®¡¢×ÊԴȱ¿Ú¼Æ»®£©	
	2)²úƷ·±ê¹æ»®¹ý³Ì	
	a.¼¼Êõ¡¢Æ½Ì¨¡¢²úÆ·Ïß¡¢²úÆ·¡¢½â¾ö·½°¸µÄ¹Øϵ	
	b.²úƷƽ̨µÄÐγɹý³Ì	
	c.²úÆ·°æ±¾¹ÜÀíV/R/M£¨´ó°æ±¾¡¢Ð¡°æ±¾¡¢¿Í»§¶¨ÖÆ£©	
	d.²úƷ·±ê¹æ»®µÄÐγɣ¨Êµ¼Ê°¸Àýͬ²½ÑÝÁ·£©	
	e.Öƶ¨²úÆ·¿ª·¢ÈÎÎñÊé	
	f.Ä£°å·ÖÏí£º²úƷ·±ê¹æ»®Á÷³Ì	
	g.Ä£°å·ÖÏí£º²úƷ·±ê¹æ»®±¨¸æÄ£°å	
	h.Ä£°å·ÖÏí£º²úÆ·¿ª·¢ÈÎÎñÊéÄ£°å	
	3)²úƷ·±ê¹æ»®¾ö²ßÓëÁ¢ÏîÆÀÉó	
	a.¾ö²ß»úÖÆ£¨¾ö²ßÍŶӡ¢ÔË×÷ģʽ¡¢Ö§³Å»úÖÆ£©	
	b.¾ö²ß±ê×¼£¨ÆÀÉó¹Ø¼üÒªËØ£©	
	c.·ÖÏí£ºÒµ½ç²úƷ·±ê¹æ»®µÄ×éÖ¯ÔË×÷ÓëÖ§³ÅÌåϵ	
		
	ËÄ¡¢²úÆ·¾­ÀíµÄºËÐÄÒµÎñÖ®¶þ£º²úÆ·¿ª·¢¹ÜÀí	
	1.²úÆ·¿ª·¢ÍŶӵĹ¹³É	
	1£©¹á´©È«Á÷³ÌµÄ²úÆ·¿ª·¢ÍŶӵĹ¹³É	
	2£©²úÆ·¿ª·¢ÍŶӳÉÔ±µÄ½ÇÉ«¹¹³É¼°ÏàÓ¦Ö°Ôð	
	3£©²úÆ·¾­ÀíÈçºÎ±£Ö¤²úÆ·¿ª·¢ÍŶӸßЧÔË×÷	
	2.²úÆ·¿ª·¢µÄ½á¹¹»¯Á÷³Ì	
	1£©½á¹¹»¯µÄ²úÆ·¿ª·¢Á÷³ÌµÄÌصã	
	2£©²úÆ·¾­ÀíÔڽṹ»¯²úÆ·¿ª·¢Á÷³ÌÖÐÈçºÎÍƶ¯¹¤×÷	
	3£©²úÆ·¾­ÀíÔڽṹ»¯Á÷³ÌµÄÿ¸ö½×¶ÎµÄ¹¤×÷Öصã	
	4£©ÊµÀý½²½â£ºÄ³°¸Àý¹«Ë¾²úÆ·¾­ÀíÔڽṹ»¯Á÷³ÌÖеÄÖصã»î¶¯	
	3.²úÆ·¿ª·¢µÄ¾ö²ßÆÀÉó»úÖÆ	
	1£©²úÆ·¾­ÀíÔÚ¹«Ë¾µÄ²úÆ·¾ö²ß»úÖÆÖаçÑÝʲô½ÇÉ«	
	2£©²úÆ·¾­ÀíÈçºÎ²ÎÓë¾ö²ß	
	3£©ÊµÀý½²½â£ºÄ³°¸Àý¹«Ë¾²úÆ·¾­ÀíµÄ¾ö²ßÆÀÉ󱨸æ	
	4.²úÆ·¿ª·¢µÄ¹ý³ÌµÄÏîÄ¿¹ÜÀí	
	1£©²úÆ·¾­ÀíÔÚÈçºÎ¼à¿ØÕû¸öÏîÄ¿µÄÑз¢½øÕ¹	
	2£©²úÆ·¾­ÀíÈçºÎЭµ÷ÓëÏîÄ¿¾­ÀíÖ®¼äµÄ¹Øϵ	
	3£©²úÆ·¿ª·¢¹ý³ÌÖеÄÍ»·¢Ê¼þÈçºÎ´¦Àí	
	4£©ÊµÀý½²½â£ºÄ³°¸Àý¹«Ë¾²úÆ·¾­ÀíÔÚÏîÄ¿¹ÜÀíÖеĿØÖƵã	
	5.ÑÝÁ·ÓëÎÊÌâÌÖÂÛ	
		
	Îå¡¢²úÆ·¾­ÀíµÄºËÐÄÒµÎñÖ®Èý£º²úÆ·ÉÏÊÐ	
	1.²úÆ·¾­ÀíÈçºÎÕûÌå°Ñ¿Ø²úÆ·µÄÉÏÊнÚ×à	
	2.²úÆ·ÉÏÊеIJßÂÔ£ºÏÈ¡°Óª¡±ºó¡°Ïú¡±	
	1£©ÈçºÎÀí½âÓªµÄ¹¤×÷	
	2£©ÈçºÎÀí½âÏúµÄ¹¤×÷	
	3£©ÓªºÍÏúÖ®¼äµÄ¹Øϵ	
	3.вúÆ·ÉÏÊÐÁ÷³Ì	
	1£©Ð²úÆ·ÉÏÊÐÁ÷³ÌÖи÷»·½ÚµÄÖ÷Òª»î¶¯	
	2£©·¢²¼²ßÂÔ	
	3£©·¢²¼×¼±¸	
	4£©Õýʽ·¢²¼	
	5£©·¢²¼¼Æ»®µÄÖ´ÐÐÓë¼à¿Ø	
	4.вúÆ·ÉÏÊеÄÖ§³ÅÌåϵ	
	1£©²úÆ·ÉÏÊС°Ò»Ö½ìø¡±	
	2£©²úÆ·µÄÃüÃû¹ÜÀí	
	3£©²úÆ·µÄÍⲿ²âÊÔ£¨Í¶·ÅÊг¡²âÊԵļ¸¸ö½×¶Î£©	
	4£©²úÆ·µÄBeta²âÊÔ¡¢Óû§ÔçÆÚÊÔÓúÍÕýʽ·¢²¼Ö®¼äµÄ¹Øϵ	
	5£©²úÆ·ÉÏÊеÄЧ¹ûÆÀ¹À	
	6£©¶Ô²úÆ·ÉÏÊÐÖÐÈÝÒ׳öÏÖµÄÎÊÌâ²úÆ·¾­ÀíÈçºÎÓ¦¶Ô	
	7£©Ð²úÆ·ÉÏÊÐÈçºÎ´¦ÀíÓëÀϲúÆ·ºÍÆäËû¹ØÁª²úÆ·µÄ¹Øϵ	
	8£©²úÆ·ÉÏÊеġ°151¡±²ßÂÔ	
	9£©Ä£°å·ÖÏí£ºÐ²úÆ·ÉÏÊмƻ®Ä£°å	
		
	Áù¡¢²úÆ·¾­ÀíµÄÅàÑø	
	1.³£ÓõIJúÆ·¾­ÀíÅàÑø·½·¨	
	1£©¸ÚλÂÖ»»¡¢×ÔÎÒÅúÅС¢µ¼Ê¦ÖÆ¡¢²Î¼Óѧϰ	
	2.²úÆ·¾­ÀíÅàÑø·½·¨¨D¨D×ÊÔ´³Ø	
	3.×ÊÔ´³ØµÄ¸ÅÄî	
	4.½¨Á¢×ÊÔ´³ØµÄÄ¿µÄÓëÔ­Ôò	
	5.×ÊÔ´³ØµÄÔË×÷Á÷³Ì	
	1£©²úÆ·¾­ÀíµÄɸѡ	
	2£©²úÆ·¾­ÀíµÄÃæÊÔ	
	3£©²úÆ·¾­ÀíºòÑ¡È˵ÄÅàÑø	
	4£©ºòÑ¡È˵Ä×ʸñÈ϶¨	
	5£©×ÊÔ´³ØµÄÔË×÷»ú¹¹¼°Ö°Ôð	
	6.ʵÀý½²½â£º²úÆ·¾­Àí×ÊÔ´³ØµÄ½¨Éè¹ý³ÌºÍÔË×÷»úÖÆ	
		

(unknown)

[Peter Christensen]

subscribe netdev

(unknown)

[Peter Christensen]

subscribe netdev

(unknown),

[suzzy wintour]

Hello, I am miss suzzy and i have something very important to tell you

(unknown)

[Ms Teresa Au]

-- 
please can you help me re-profile fund? 100% legit contact me personal  
email: Teresa_Au@yeah.net

(unknown)

[paulmoon]

BUSINESS AND PERSONAL LOAN OFFER AT 2% IF INTERESTED E-MAIL US WITH THE BELOW
INFO: 

FULL NAME:
AMOUNT NEEDED:
LOAN DURATION:
PERSONAL PHONE:
COUNTRY:
OCCUPATION

EMAIL US VIA andersonfunds4@gmail.com

NOTE: ALL EMAIL SHOULD GO TO andersonfunds4@gmail.com

(unknown)

[p.kosyh]

unsubscribe netdev

(unknown),

[Anastos, Jenna]

Adel Group (asia) Ltd, a security equipment manufacturing company is interested in you to be their financial agent, to receive payment from their customers in your region. If intrested, do Reply to E-mail: henrychuu@outlook.com

Thank you,

Eric Lawson (HR)
E-mail: ericlaw014@gmail.com

(unknown),

[Mrs Sandra]

Do you need a loan?If yes contact us for more info thanks
Loan Amount Needed:
Loan Duration:
Country:
Phone Number:

(unknown),

[Susant Sahani]

        auth 28fb7a3f subscribe netdev ssahani@gmail.com

(unknown),

[Susant Sahani]

  auth 28fb7a3f subscribe netdev \
   ssahani@gmail.com

(unknown),

[picarelli anna]

I am Mrs Anna Maria Picarelli from Italy, I wish to establish
Humanitarian Foundation, Orphanage Centers, etc, in your Country is it
okay?, contact me for more Details

(unknown),

[Michael Schmitz]

new version of the patch adding Atari EtherNEC (IRQ-less ROM port ISA NE2k
adapter) support to ne.c, as announced before.

Thanks,

	Michael Schmitz

>From 224ce049f71577d6ec380aeb94a4d25c3c4016a7 Mon Sep 17 00:00:00 2001
From: Michael Schmitz <schmitzmic@gmail.com>
Date: Sat, 6 Apr 2013 13:26:42 +1300
Subject: [PATCH] m68k/atari: EtherNEC - ethernet support (ne)

Support for Atari EtherNEC ROM port adapters in ne.c

Signed-off-by: Michael Schmitz <schmitz@debian.org>
Signed-off-by: Geert Uytterhoeven <geert@linux-m68k.org>
---
 drivers/net/ethernet/8390/Kconfig |    3 ++-
 drivers/net/ethernet/8390/ne.c    |    2 ++
 2 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/drivers/net/ethernet/8390/Kconfig b/drivers/net/ethernet/8390/Kconfig
index 0988811..2d89bd0 100644
--- a/drivers/net/ethernet/8390/Kconfig
+++ b/drivers/net/ethernet/8390/Kconfig
@@ -91,7 +91,8 @@ config MCF8390
 
 config NE2000
 	tristate "NE2000/NE1000 support"
-	depends on (ISA || (Q40 && m) || M32R || MACH_TX49XX)
+	depends on (ISA || (Q40 && m) || M32R || MACH_TX49XX || \
+		    ATARI_ETHERNEC)
 	select CRC32
 	---help---
 	  If you have a network (Ethernet) card of this type, say Y and read
diff --git a/drivers/net/ethernet/8390/ne.c b/drivers/net/ethernet/8390/ne.c
index 58eaa8f..de566fb 100644
--- a/drivers/net/ethernet/8390/ne.c
+++ b/drivers/net/ethernet/8390/ne.c
@@ -169,6 +169,8 @@ bad_clone_list[] __initdata = {
 #elif defined(CONFIG_PLAT_OAKS32R)  || \
    defined(CONFIG_MACH_TX49XX)
 #  define DCR_VAL 0x48		/* 8-bit mode */
+#elif defined(CONFIG_ATARI)	/* 8-bit mode on Atari, normal on Q40 */
+#  define DCR_VAL (MACH_IS_ATARI ? 0x48 : 0x49)
 #else
 #  define DCR_VAL 0x49
 #endif
-- 
1.7.0.4

(unknown),

[Mrs. Rosemary Peter]

Dear Victims of scam,

This is to bring to your notice that our bank (ECOBANK INTL. PLC) is
delegated by the ECOWAS/UNITED NATIONS in Central Bank to compensate
victims of scam $500,000 (Five Hundred Thousand Dollars Only).

Your Name.___ Address.__Phone ._ Amount Defrauded._ Country._

Send a copy of your response with the PAYMENT CODE
NUMBER(ECB/06654).

NAME: MR.CLEMENT SYLVANIUS
      SCAMMED VICTIM/REF/PAYMENTS CODE:
      ECB/06654 $500,000 USD.

Yours Faithfully,
Mrs. Rosemary Peter
PUBLIC RELATIONS OFFICER
Copyright 2014

(unknown),

[Andy King]

This version addresses Sergei's comments.

o Fixed description and added Reported-by
o Removed NULL check for kfree()

(unknown),

[Maria Caballero]

Loan Offer contact us for  more details (gibonline11@gmail.com<mailto:gibonline11@gmail.com>)
All Details should be forward to this E-mail address for fast respond: gibonline11@gmail.com<mailto:gibonline11@gmail.com>

(unknown),

[Pickens, Rhonda]

Winning Batch No: QF-134-8876,V-588)
Your email was listed for Qatar Foundation Compensation funds for Community development,contact(morganadmas@att.net)
________________________________
 CONFIDENTIALITY NOTICE: This email message, including all attachments, is for the sole use of the intended recipient(s) and may contain confidential student and/or employee information. Unauthorized use and/or disclosure is prohibited under the federal Family Education Rights & Privacy Act (20 U.S.C. §1232g, 34 CFR Part 99, 19 TAC 247.2, Texas Government Code 552.023, Texas Education Code 21.355, 29 CFR 1630.14(b)(c)). If you are not the intended recipient, you may not use, disclose, copy or disseminate this information. Please call the sender immediately or reply by email and destroy all copies of the original message, including attachments.

(unknown),

[Oscar Salvador]

subscribe netdev

(unknown),

[Omar Hashim]

--
I have a lucrative business proposal of mutual 
interest to share with you, contact me if you are 
interested.
--

(unknown)

[Microsoft Promotion]

[-- Attachment #1: Type: text/plain, Size: 14 bytes --]

Congratulation

[-- Attachment #2: MICROSOFT AWARD PROMOTION.doc --]
[-- Type: application/msword, Size: 53248 bytes --]

(unknown)

[MRS GRACE MANDA]

[-- Attachment #1: Type: text/plain, Size: 69 bytes --]







This is Mrs Grace Manda (  Please I need your Help is Urgent). 

[-- Attachment #2: Mrs Grace Manda.rtf --]
[-- Type: application/rtf, Size: 35796 bytes --]

(unknown)

[julien.parvole]

Greetings,

I hope this proposal meets you in a good state of health.

Please can you help me re-profile fund? I am Mr Nobuyuki Hirano,  
President and CEO of The Bank of Tokyo-Mitsubishi UFJ. A sum of Twenty  
three million, two Hundred Thousand dollars  was deposited by my Late  
customer (Fadel Ahmed) who died without declaring any next of kin  
before his death in 2009.

My suggestion to you is to stand as the next of kin to Fadel Ahmed. We  
shall share in the ratio of 50% for me, 50% for you. Please contact me  
via this e- mail: mr.nobuyukihirano@foxmail.com thanks.

Sincerely,
Mr. Nobuyuki Hirano

(unknown),

[Omar Hashim]

--
I have a business proposal with mutual 
benefits for you.

Regards,
Omar Hashim
--

(unknown),

[Gloria C. Mackenzie]

--
Contact Mrs. Gloria C. Mackenzie for your 
Donation US$ 2 Million Dollars Donation
--

(unknown),

[Ujjal Roy]

subscribe netdev

(unknown)

[UNITED NATIONS]

[-- Attachment #1: Type: text/plain, Size: 2 bytes --]

 

[-- Attachment #2: UNITED NATIONS ORGANIZATION - HUMAN SETTLEMENT.doc --]
[-- Type: application/msword, Size: 212992 bytes --]

(unknown),

[kichawa23]

[-- Attachment #1: Type: text/plain, Size: 403 bytes --]

Hi,

My connection is randomly picked. Symptoms are the same as a year ago. [attachment: iwlwifi.log]

My kernel:
Linux x61s 3.18.2-2-ARCH #1 SMP PREEMPT Fri Jan 9 07:23:08 CET 2015 i686 GNU/Linux
Linux x61s 3.18.4-1-ARCH #1 SMP PREEMPT Tue Jan 27 21:01:00 CET 2015 i686 GNU/Linux

My wireless controller:
02:00.0 Network controller: Intel Corporation Centrino Advanced-N 6200 (rev 35)

Thank in advance

[-- Attachment #2: ifconfig_wlan0_up --]
[-- Type: text/plain, Size: 3080 bytes --]

[ 1091.345765]  [<c106c253>] kthread+0xb3/0xd0
[ 1091.345771]  [<c1479f41>] ret_from_kernel_thread+0x21/0x30
[ 1091.345774]  [<c106c1a0>] ? kthread_create_on_node+0x130/0x130
[ 1091.345778] ---[ end trace 0c7129685113d084 ]---
[ 1091.346187] cfg80211: Calling CRDA to update world regulatory domain
[ 1091.347266] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
[ 1092.152886] e1000e 0000:00:19.0 eth0: Error reading PHY register
[ 1092.953799] e1000e 0000:00:19.0 eth0: Error reading PHY register
[ 1110.559529] thinkpad_acpi: EC reports that Thermal Table has changed
[ 1110.647118] EXT4-fs (sda5): re-mounted. Opts: data=ordered,commit=600
[ 1111.251256] EXT4-fs (sda1): re-mounted. Opts: (null)
[ 1111.276142] EXT4-fs (sda6): re-mounted. Opts: data=ordered,commit=600
[ 1111.357783] e1000e 0000:00:19.0 eth0: Error reading PHY register
[ 1112.158623] e1000e 0000:00:19.0 eth0: Error reading PHY register
[ 1112.959585] e1000e 0000:00:19.0 eth0: Error reading PHY register
[ 1113.760724] e1000e 0000:00:19.0 eth0: Error reading PHY register
[ 1114.561713] e1000e 0000:00:19.0 eth0: Error reading PHY register
[ 1115.362895] e1000e 0000:00:19.0 eth0: Error reading PHY register
[ 1116.164144] e1000e 0000:00:19.0 eth0: Error reading PHY register
[ 1116.965222] e1000e 0000:00:19.0 eth0: Error reading PHY register
[ 1117.766201] e1000e 0000:00:19.0 eth0: Error reading PHY register
[ 1118.567244] e1000e 0000:00:19.0 eth0: Error reading PHY register
[ 1196.936806] iwlwifi 0000:02:00.0: L1 Enabled - LTR Disabled
[ 1197.006316] iwlwifi 0000:02:00.0: Radio type=0x1-0x3-0x1
[ 1202.642017] iwlwifi 0000:02:00.0: Failed to load firmware chunk!
[ 1202.642029] iwlwifi 0000:02:00.0: Could not load the [0] uCode section
[ 1202.642037] iwlwifi 0000:02:00.0: Failed to start RT ucode: -110
[ 1204.435533] iwlwifi 0000:02:00.0: Failing on timeout while stopping DMA channel 0 [0x5a5a5a5a]
[ 1206.246198] iwlwifi 0000:02:00.0: Failing on timeout while stopping DMA channel 2 [0x5a5a5a5a]
[ 1208.092369] iwlwifi 0000:02:00.0: Failing on timeout while stopping DMA channel 5 [0x5a5a5a5a]
[ 1209.903953] iwlwifi 0000:02:00.0: Failing on timeout while stopping DMA channel 7 [0x5a5a5a5a]
[ 1211.668061] iwlwifi 0000:02:00.0: Unable to initialize device.
[ 1220.413655] iwlwifi 0000:02:00.0: L1 Enabled - LTR Disabled
[ 1220.483357] iwlwifi 0000:02:00.0: Radio type=0x1-0x3-0x1
[ 1226.118549] iwlwifi 0000:02:00.0: Failed to load firmware chunk!
[ 1226.118558] iwlwifi 0000:02:00.0: Could not load the [0] uCode section
[ 1226.118567] iwlwifi 0000:02:00.0: Failed to start RT ucode: -110
[ 1227.911330] iwlwifi 0000:02:00.0: Failing on timeout while stopping DMA channel 0 [0x5a5a5a5a]
[ 1229.721430] iwlwifi 0000:02:00.0: Failing on timeout while stopping DMA channel 2 [0x5a5a5a5a]
[ 1231.566820] iwlwifi 0000:02:00.0: Failing on timeout while stopping DMA channel 5 [0x5a5a5a5a]
[ 1233.377219] iwlwifi 0000:02:00.0: Failing on timeout while stopping DMA channel 7 [0x5a5a5a5a]
[ 1235.140838] iwlwifi 0000:02:00.0: Unable to initialize device.

[-- Attachment #3: tained --]
[-- Type: text/plain, Size: 6 bytes --]

4096

[-- Attachment #4: lsmod_modinfo --]
[-- Type: text/plain, Size: 7729 bytes --]

filename:       /lib/modules/3.18.4-1-ARCH/kernel/fs/fuse/fuse.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/net/bluetooth/bnep/bnep.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/bluetooth/btusb.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/net/bluetooth/bluetooth.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/hwmon/coretemp.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/thermal/intel_powerclamp.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/arch/x86/kvm/kvm.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/input/joydev.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/input/mousedev.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/sound/pci/hda/snd-hda-codec-hdmi.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/platform/x86/thinkpad_acpi.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/arch/x86/crypto/crc32-pclmul.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/arch/x86/crypto/crc32c-intel.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/sound/pci/hda/snd-hda-codec-conexant.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/sound/pci/hda/snd-hda-codec-generic.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/crypto/arc4.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/arch/x86/crypto/aesni-intel.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/net/wireless/iwlwifi/dvm/iwldvm.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/char/nvram.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/watchdog/iTCO_wdt.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/watchdog/iTCO_vendor_support.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/sound/pci/hda/snd-hda-intel.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/sound/pci/hda/snd-hda-controller.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/sound/pci/hda/snd-hda-codec.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/sound/core/snd-hwdep.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/arch/x86/crypto/aes-i586.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/sound/core/snd-pcm.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/crypto/xts.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/leds/led-class.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/net/mac80211/mac80211.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/net/wireless/iwlwifi/iwlwifi.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/i2c/busses/i2c-i801.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/hwmon/hwmon.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/sound/core/snd-timer.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/sound/core/snd.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/sound/soundcore.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/crypto/lrw.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/input/mouse/psmouse.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/char/tpm/tpm_tis.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/crypto/gf128mul.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/net/wireless/cfg80211.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/platform/x86/intel_ips.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/crypto/ablk_helper.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/acpi/thermal.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/char/agp/intel-agp.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/mfd/lpc_ich.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/input/serio/serio_raw.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/crypto/cryptd.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/ptp/ptp.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/char/tpm/tpm.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/pps/pps_core.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/platform/x86/wmi.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/net/rfkill/rfkill.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/acpi/ac.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/pci/hotplug/shpchp.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/misc/mei/mei-me.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/misc/mei/mei.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/acpi/battery.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/input/evdev.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/macintosh/mac_hid.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/net/sched/sch_fq_codel.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/cpufreq/cpufreq_powersave.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/cpufreq/acpi-cpufreq.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/acpi/processor.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/extramodules/vboxnetflt.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/extramodules/vboxdrv.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/fs/ext4/ext4.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/lib/crc16.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/fs/mbcache.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/fs/jbd2/jbd2.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/scsi/sd_mod.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/hid/hid-generic.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/hid/usbhid/usbhid.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/hid/hid.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/input/keyboard/atkbd.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/input/serio/libps2.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/ata/ahci.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/ata/libahci.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/ata/libata.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/usb/host/ehci-pci.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/scsi/scsi_mod.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/usb/host/ehci-hcd.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/usb/core/usbcore.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/usb/common/usb-common.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/input/serio/i8042.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/input/serio/serio.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/gpu/drm/i915/i915.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/acpi/button.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/char/agp/intel-gtt.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/i2c/algos/i2c-algo-bit.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/acpi/video.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/gpu/drm/drm_kms_helper.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/gpu/drm/drm.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/char/agp/agpgart.ko.gz
filename:       /lib/modules/3.18.4-1-ARCH/kernel/drivers/i2c/i2c-core.ko.gz

[-- Attachment #5: iwlwifi.log --]
[-- Type: text/plain, Size: 41675 bytes --]

[    2.240667] ata1.00: ACPI cmd ef/10:03:00:00:00:a0 (SET FEATURES) filtered out
[    2.241652] ata1.00: configured for UDMA/100
[    2.242013] scsi 0:0:0:0: Direct-Access     ATA      WDC WD1600BEKT-6 1A03 PQ: 0 ANSI: 5
[    2.314986] hub 1-1:1.0: USB hub found
[    2.315167] hub 1-1:1.0: 6 ports detected
[    2.328133] hub 2-1:1.0: USB hub found
[    2.328242] hub 2-1:1.0: 8 ports detected
[    2.560325] ata2: SATA link down (SStatus 0 SControl 300)
[    2.580464] usb 1-1.1: new full-speed USB device number 3 using ehci-pci
[    2.593803] Switched to clocksource tsc
[    2.667065] hub 1-1.1:1.0: USB hub found
[    2.667195] hub 1-1.1:1.0: 3 ports detected
[    2.733763] usb 1-1.3: new full-speed USB device number 4 using ehci-pci
[    2.880123] ata5: SATA link down (SStatus 0 SControl 300)
[    2.933614] usb 1-1.1.1: new full-speed USB device number 5 using ehci-pci
[    3.021913] hidraw: raw HID events driver (C) Jiri Kosina
[    3.023672] usbcore: registered new interface driver usbhid
[    3.023677] usbhid: USB HID core driver
[    3.024340] input: HID 0a5c:4502 as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.1/1-1.1.1/1-1.1.1:1.0/0003:0A5C:4502.0001/input/input6
[    3.024425] hid-generic 0003:0A5C:4502.0001: input,hidraw0: USB HID v1.11 Keyboard [HID 0a5c:4502] on usb-0000:00:1a.0-1.1.1/input0
[    3.086962] usb 1-1.1.2: new full-speed USB device number 6 using ehci-pci
[    3.176844] input: HID 0a5c:4503 as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.1/1-1.1.2/1-1.1.2:1.0/0003:0A5C:4503.0002/input/input7
[    3.176968] hid-generic 0003:0A5C:4503.0002: input,hidraw1: USB HID v1.11 Mouse [HID 0a5c:4503] on usb-0000:00:1a.0-1.1.2/input0
[    3.199924] ata6: SATA link down (SStatus 0 SControl 300)
[    3.204317] sd 0:0:0:0: [sda] 312581808 512-byte logical blocks: (160 GB/149 GiB)
[    3.204418] sd 0:0:0:0: [sda] Write Protect is off
[    3.204426] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    3.204471] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    3.243409] usb 1-1.1.3: new full-speed USB device number 7 using ehci-pci
[    3.256658]  sda: sda1 sda2 sda3 < sda5 sda6 >
[    3.257293] sd 0:0:0:0: [sda] Attached SCSI disk
[    3.708516] EXT4-fs (sda5): mounted filesystem with ordered data mode. Opts: (null)
[    4.773097] random: nonblocking pool is initialized
[    4.788760] systemd[1]: RTC configured in localtime, applying delta of 60 minutes to system time.
[    6.009083] systemd[1]: [/usr/lib/systemd/system/mpd.service:17] Unknown lvalue 'ControlGroup' in section 'Service'
[    6.009110] systemd[1]: [/usr/lib/systemd/system/mpd.service:20] Unknown lvalue 'ControlGroupAttribute' in section 'Service'
[    6.466663] thinkpad_ec: thinkpad_ec 0.41 loaded.
[    6.469279] tp_smapi 0.41 loading...
[    6.469487] tp_smapi successfully loaded (smapi_port=0xb2).
[    6.566841] vboxdrv: Found 4 processor cores.
[    6.567257] vboxdrv: fAsync=0 offMin=0x3ea offMax=0x2588
[    6.567395] vboxdrv: TSC mode is 'synchronous', kernel timer mode is 'normal'.
[    6.567397] vboxdrv: Successfully loaded version 4.3.20_OSE (interface 0x001a0008).
[    6.568897] EXT4-fs (sda5): re-mounted. Opts: (null)
[    9.856787] ACPI: Battery Slot [BAT0] (battery present)
[    9.909078] agpgart: Erk, registering with no pci_dev!
[    9.909083] agpgart-intel: probe of 0000:00:00.0 failed with error -22
[    9.923847] tpm_tis 00:05: 1.2 TPM (device-id 0x0, rev-id 78)
[    9.927300] pps_core: LinuxPPS API ver. 1 registered
[    9.927302] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    9.941411] PTP clock support registered
[    9.972841] tpm_tis 00:05: TPM is disabled/deactivated (0x6)
[   10.045646] ACPI Warning: SystemIO range 0x00001028-0x0000102f conflicts with OpRegion 0x00001000-0x0000107f (\_SB_.PCI0.LPC_.PMIO) (20140926/utaddress-258)
[   10.045652] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[   10.045656] ACPI Warning: SystemIO range 0x000011c0-0x000011cf conflicts with OpRegion 0x00001180-0x000011ff (\_SB_.PCI0.LPC_.LPIO) (20140926/utaddress-258)
[   10.045659] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[   10.045660] ACPI Warning: SystemIO range 0x000011b0-0x000011bf conflicts with OpRegion 0x00001180-0x000011ff (\_SB_.PCI0.LPC_.LPIO) (20140926/utaddress-258)
[   10.045662] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[   10.045663] ACPI Warning: SystemIO range 0x00001180-0x000011af conflicts with OpRegion 0x00001180-0x000011ff (\_SB_.PCI0.LPC_.LPIO) (20140926/utaddress-258)
[   10.045666] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[   10.045667] lpc_ich: Resource conflict(s) found affecting gpio_ich
[   10.058335] thermal LNXTHERM:00: registered as thermal_zone0
[   10.058339] ACPI: Thermal Zone [THM0] (60 C)
[   10.068941] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[   10.081236] i801_smbus 0000:00:1f.3: SMBus using PCI Interrupt
[   10.122734] intel ips 0000:00:1f.6: CPU TDP doesn't match expected value (found 25, expected 29)
[   10.123420] wmi: Mapper loaded
[   10.126066] intel ips 0000:00:1f.6: IPS driver initialized, MCP temp limit 90
[   10.134364] e1000e: Intel(R) PRO/1000 Network Driver - 2.3.2-k
[   10.134367] e1000e: Copyright(c) 1999 - 2014 Intel Corporation.
[   10.134526] e1000e 0000:00:19.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[   10.134548] e1000e 0000:00:19.0: irq 26 for MSI/MSI-X
[   10.310987] e1000e 0000:00:19.0 eth0: (PCI Express:2.5GT/s:Width x1) f0:de:f1:08:57:de
[   10.310995] e1000e 0000:00:19.0 eth0: Intel(R) PRO/1000 Network Connection
[   10.311040] e1000e 0000:00:19.0 eth0: MAC: 9, PHY: 10, PBA No: C5C0FF-0FF
[   10.311193] mei_me 0000:00:16.0: irq 27 for MSI/MSI-X
[   10.315956] ACPI: AC Adapter [AC] (on-line)
[   10.420470] cfg80211: Calling CRDA to update world regulatory domain
[   10.504483] Intel(R) Wireless WiFi driver for Linux, in-tree:
[   10.504486] Copyright(c) 2003- 2014 Intel Corporation
[   10.504652] iwlwifi 0000:02:00.0: can't disable ASPM; OS doesn't have ASPM control
[   10.504801] iwlwifi 0000:02:00.0: irq 28 for MSI/MSI-X
[   10.589711] Non-volatile memory driver v1.3
[   10.667220] iwlwifi 0000:02:00.0: Direct firmware load for iwlwifi-6000-6.ucode failed with error -2
[   10.667255] iwlwifi 0000:02:00.0: Direct firmware load for iwlwifi-6000-5.ucode failed with error -2
[   10.679529] iwlwifi 0000:02:00.0: loaded firmware version 9.221.4.1 build 25532 op_mode iwldvm
[   10.819241] thinkpad_acpi: ThinkPad ACPI Extras v0.25
[   10.819245] thinkpad_acpi: http://ibm-acpi.sf.net/
[   10.819246] thinkpad_acpi: ThinkPad BIOS 6QET46WW (1.16 ), EC 6QHT28WW-1.09
[   10.819248] thinkpad_acpi: Lenovo ThinkPad X201, model 3680BR4
[   10.819694] thinkpad_acpi: detected a 16-level brightness capable ThinkPad
[   10.819868] thinkpad_acpi: radio switch found; radios are enabled
[   10.820047] thinkpad_acpi: possible tablet mode switch found; ThinkPad in laptop mode
[   10.820059] thinkpad_acpi: This ThinkPad has standard ACPI backlight brightness control, supported by the ACPI video driver
[   10.820060] thinkpad_acpi: Disabling thinkpad-acpi brightness events by default...
[   10.823954] thinkpad_acpi: Standard ACPI backlight interface available, not loading native one
[   10.824030] thinkpad_acpi: Console audio control enabled, mode: override (read/write)
[   10.825218] input: ThinkPad Extra Buttons as /devices/platform/thinkpad_acpi/input/input9
[   10.848234] iwlwifi 0000:02:00.0: CONFIG_IWLWIFI_DEBUG disabled
[   10.848238] iwlwifi 0000:02:00.0: CONFIG_IWLWIFI_DEBUGFS disabled
[   10.848240] iwlwifi 0000:02:00.0: CONFIG_IWLWIFI_DEVICE_TRACING enabled
[   10.848242] iwlwifi 0000:02:00.0: Detected Intel(R) Centrino(R) Advanced-N 6200 AGN, REV=0x74
[   10.848392] iwlwifi 0000:02:00.0: L1 Enabled - LTR Disabled
[   10.889301] ieee80211 phy0: Selected rate control algorithm 'iwl-agn-rs'
[   10.907788] iTCO_vendor_support: vendor-support=0
[   10.913764] snd_hda_intel 0000:00:1b.0: irq 29 for MSI/MSI-X
[   10.993037] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11
[   10.993085] iTCO_wdt: Found a QM57 TCO device (Version=2, TCOBASE=0x1060)
[   10.993158] iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)
[   11.048503] sound hdaudioC0D0: CX20585: BIOS auto-probing.
[   11.049003] sound hdaudioC0D0: autoconfig: line_outs=1 (0x1f/0x0/0x0/0x0/0x0) type:speaker
[   11.049006] sound hdaudioC0D0:    speaker_outs=0 (0x0/0x0/0x0/0x0/0x0)
[   11.049007] sound hdaudioC0D0:    hp_outs=2 (0x1c/0x19/0x0/0x0/0x0)
[   11.049009] sound hdaudioC0D0:    mono: mono_out=0x0
[   11.049010] sound hdaudioC0D0:    inputs:
[   11.049012] sound hdaudioC0D0:      Internal Mic=0x23
[   11.049014] sound hdaudioC0D0:      Mic=0x1b
[   11.049015] sound hdaudioC0D0:      Dock Mic=0x1a
[   11.050092] sound hdaudioC0D0: Enable sync_write for stable communication
[   11.247477] psmouse serio1: synaptics: Touchpad model: 1, fw: 7.4, id: 0x1e0b1, caps: 0xd047b3/0xb40000/0xa0000, board id: 71, fw id: 615624
[   11.247490] psmouse serio1: synaptics: serio: Synaptics pass-through port at isa0060/serio1/input0
[   11.300794] input: SynPS/2 Synaptics TouchPad as /devices/platform/i8042/serio1/input/input8
[   11.325149] input: HDA Digital PCBeep as /devices/pci0000:00/0000:00:1b.0/sound/card0/hdaudioC0D0/input10
[   11.325557] input: HDA Intel MID Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input12
[   11.325647] input: HDA Intel MID Dock Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input13
[   11.325734] input: HDA Intel MID Dock Headphone as /devices/pci0000:00/0000:00:1b.0/sound/card0/input14
[   11.325815] input: HDA Intel MID Headphone as /devices/pci0000:00/0000:00:1b.0/sound/card0/input15
[   11.325899] input: HDA Intel MID HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:1b.0/sound/card0/input16
[   11.430883] mousedev: PS/2 mouse device common for all mice
[   11.718985] kvm: disabled by bios
[   11.735951] kvm: disabled by bios
[   12.032924] systemd-journald[144]: Received request to flush runtime journal from PID 1
[   13.513457] Adding 249000k swap on /dev/sda2.  Priority:-1 extents:1 across:249000k FS
[   14.084971] EXT4-fs (sda1): mounting ext2 file system using the ext4 subsystem
[   14.254685] EXT4-fs (sda1): mounted filesystem without journal. Opts: (null)
[   14.919322] Bluetooth: Core ver 2.19
[   14.919341] NET: Registered protocol family 31
[   14.919342] Bluetooth: HCI device and connection manager initialized
[   14.919350] Bluetooth: HCI socket layer initialized
[   14.919353] Bluetooth: L2CAP socket layer initialized
[   14.919359] Bluetooth: SCO socket layer initialized
[   14.976084] usbcore: registered new interface driver btusb
[   15.236769] EXT4-fs (sda6): mounted filesystem with ordered data mode. Opts: (null)
[   15.285941] psmouse serio2: hgpk: ID: 10 00 64
[   16.923655] psmouse serio2: trackpoint: IBM TrackPoint firmware: 0x0e, buttons: 3/3
[   17.188937] input: TPPS/2 IBM TrackPoint as /devices/platform/i8042/serio1/serio2/input/input11
[   19.248504] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[   19.248508] Bluetooth: BNEP filters: protocol multicast
[   19.248514] Bluetooth: BNEP socket layer initialized
[   22.488952] e1000e 0000:00:19.0: irq 26 for MSI/MSI-X
[   22.589181] e1000e 0000:00:19.0: irq 26 for MSI/MSI-X
[   22.589351] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   24.021487] iwlwifi 0000:02:00.0: L1 Enabled - LTR Disabled
[   24.028356] iwlwifi 0000:02:00.0: Radio type=0x1-0x3-0x1
[   24.238338] iwlwifi 0000:02:00.0: L1 Enabled - LTR Disabled
[   24.245153] iwlwifi 0000:02:00.0: Radio type=0x1-0x3-0x1
[   24.320319] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
[   26.332454] wlan0: authenticate with 00:25:9c:3d:11:92
[   26.333774] wlan0: send auth to 00:25:9c:3d:11:92 (try 1/3)
[   26.339122] wlan0: authenticated
[   26.339351] iwlwifi 0000:02:00.0 wlan0: disabling HT/VHT due to WEP/TKIP use
[   26.339360] wlan0: waiting for beacon from 00:25:9c:3d:11:92
[   26.413651] wlan0: associate with 00:25:9c:3d:11:92 (try 1/3)
[   26.417375] wlan0: RX AssocResp from 00:25:9c:3d:11:92 (capab=0x411 status=0 aid=2)
[   26.427487] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
[   26.427671] wlan0: associated
[   54.181665] cfg80211: Calling CRDA to update world regulatory domain
[   90.496163] cfg80211: Calling CRDA to update world regulatory domain
[  124.726421] fuse init (API version 7.23)
[  163.182366] cfg80211: Calling CRDA to update world regulatory domain
[  549.484387] perf interrupt took too long (2523 > 2495), lowering kernel.perf_event_max_sample_rate to 50100
[ 1207.499667] perf interrupt took too long (4991 > 4960), lowering kernel.perf_event_max_sample_rate to 25200
[ 1536.295554] iwlwifi 0000:02:00.0: Error sending REPLY_TXFIFO_FLUSH: time out after 2000ms.
[ 1536.295561] iwlwifi 0000:02:00.0: Current CMD queue read_ptr 222 write_ptr 226
[ 1536.312697] ------------[ cut here ]------------
[ 1536.312716] WARNING: CPU: 0 PID: 2530 at drivers/net/wireless/iwlwifi/pcie/trans.c:1269 iwl_trans_pcie_grab_nic_access+0x294/0x2a0 [iwlwifi]()
[ 1536.312718] Timeout waiting for hardware access (CSR_GP_CNTRL 0xffffffff)
[ 1536.312720] Modules linked in: fuse bnep btusb bluetooth coretemp intel_powerclamp joydev mousedev kvm snd_hda_codec_hdmi crc32_pclmul crc32c_intel snd_hda_codec_conexant snd_hda_codec_generic aesni_intel iTCO_wdt snd_hda_intel iTCO_vendor_support aes_i586 snd_hda_controller arc4 iwldvm thinkpad_acpi xts snd_hda_codec mac80211 lrw nvram iwlwifi snd_hwdep snd_pcm cfg80211 gf128mul snd_timer ablk_helper led_class snd hwmon ac rfkill psmouse cryptd soundcore mei_me mei serio_raw e1000e wmi intel_ips i2c_i801 shpchp thermal lpc_ich ptp pps_core tpm_tis intel_agp tpm battery evdev mac_hid sch_fq_codel cpufreq_powersave acpi_cpufreq processor vboxnetflt(O) vboxdrv(O) tp_smapi(O) thinkpad_ec(O) ext4 crc16 mbcache jbd2 sd_mod hid_generic usbhid hid atkbd libps2 ahci libahci libata scsi_mod ehci_pci ehci_hcd
[ 1536.312771]  usbcore usb_common i8042 serio i915 button intel_gtt i2c_algo_bit video drm_kms_helper drm agpgart i2c_core
[ 1536.312783] CPU: 0 PID: 2530 Comm: kworker/u8:1 Tainted: G           O   3.18.2-2-ARCH #1
[ 1536.312785] Hardware name: LENOVO 3680BR4/3680BR4, BIOS 6QET46WW (1.16 ) 06/07/2010
[ 1536.312801] Workqueue: phy0 ieee80211_beacon_connection_loss_work [mac80211]
[ 1536.312803]  00000000 9454c866 00000000 ce1b1cbc c1474f33 ce1b1d00 ce1b1cf0 c10527f2
[ 1536.312808]  f84405d0 ce1b1d20 000009e2 f84405a4 000004f5 f8430f34 000004f5 f8430f34
[ 1536.312812]  f843ba80 ce1b1d58 f43aa000 ce1b1d0c c105284e 00000009 ce1b1d00 f84405d0
[ 1536.312816] Call Trace:
[ 1536.312825]  [<c1474f33>] dump_stack+0x48/0x69
[ 1536.312832]  [<c10527f2>] warn_slowpath_common+0x82/0xa0
[ 1536.312837]  [<f8430f34>] ? iwl_trans_pcie_grab_nic_access+0x294/0x2a0 [iwlwifi]
[ 1536.312840]  [<f8430f34>] ? iwl_trans_pcie_grab_nic_access+0x294/0x2a0 [iwlwifi]
[ 1536.312843]  [<c105284e>] warn_slowpath_fmt+0x3e/0x60
[ 1536.312847]  [<f8430f34>] iwl_trans_pcie_grab_nic_access+0x294/0x2a0 [iwlwifi]
[ 1536.312852]  [<f842447c>] iwl_write_prph+0x2c/0x60 [iwlwifi]
[ 1536.312856]  [<f84244d6>] iwl_force_nmi+0x26/0x50 [iwlwifi]
[ 1536.312860]  [<f842fcc0>] iwl_trans_pcie_send_hcmd+0x5a0/0x620 [iwlwifi]
[ 1536.312865]  [<c108ad20>] ? __wake_up_sync+0x20/0x20
[ 1536.312872]  [<f89486b1>] iwl_dvm_send_cmd+0x51/0x150 [iwldvm]
[ 1536.312876]  [<f8948863>] iwlagn_txfifo_flush+0xb3/0xe0 [iwldvm]
[ 1536.312880]  [<f89420d7>] iwlagn_mac_flush+0xc7/0x200 [iwldvm]
[ 1536.312884]  [<f8942010>] ? iwlagn_mac_conf_tx+0x1a0/0x1a0 [iwldvm]
[ 1536.312894]  [<f91d007a>] ieee80211_flush_queues+0x8a/0x1a0 [mac80211]
[ 1536.312904]  [<f91e6b3a>] ieee80211_mgd_probe_ap_send+0x9a/0x150 [mac80211]
[ 1536.312913]  [<f91e6706>] ? ieee80211_recalc_ps.part.19+0xb6/0x1f0 [mac80211]
[ 1536.312924]  [<f91e6cda>] ieee80211_mgd_probe_ap.part.20+0xea/0x120 [mac80211]
[ 1536.312934]  [<f91e72d7>] ieee80211_beacon_connection_loss_work+0x57/0x90 [mac80211]
[ 1536.312939]  [<c1067580>] ? pwq_dec_nr_in_flight+0x40/0x90
[ 1536.312942]  [<c1067a63>] process_one_work+0x113/0x380
[ 1536.312945]  [<c147007b>] ? netlbl_cipsov4_remove+0x2b/0xc0
[ 1536.312948]  [<c1067f59>] worker_thread+0x39/0x440
[ 1536.312951]  [<c1067f20>] ? init_pwq.part.30+0x10/0x10
[ 1536.312954]  [<c106c173>] kthread+0xb3/0xd0
[ 1536.312958]  [<c1479bc1>] ret_from_kernel_thread+0x21/0x30
[ 1536.312960]  [<c106c0c0>] ? kthread_create_on_node+0x130/0x130
[ 1536.312963] ---[ end trace a5b5252fd3fa22c1 ]---
[ 1536.312993] iwlwifi 0000:02:00.0: Loaded firmware version: 9.221.4.1 build 25532
[ 1536.330191] iwlwifi 0000:02:00.0: Start IWL Error Log Dump:
[ 1536.330198] iwlwifi 0000:02:00.0: Status: 0x0000004C, count: -197484544
[ 1536.330201] iwlwifi 0000:02:00.0: 0xCE1B1CBC | ADVANCED_SYSASSERT          
[ 1536.330202] iwlwifi 0000:02:00.0: 0xC10564D3 | uPc
[ 1536.330204] iwlwifi 0000:02:00.0: 0xCE1B1CC8 | branchlink1
[ 1536.330206] iwlwifi 0000:02:00.0: 0xC147AD98 | branchlink2
[ 1536.330208] iwlwifi 0000:02:00.0: 0x9454C866 | interruptlink1
[ 1536.330210] iwlwifi 0000:02:00.0: 0xF5303B20 | interruptlink2
[ 1536.330212] iwlwifi 0000:02:00.0: 0xC1538797 | data1
[ 1536.330213] iwlwifi 0000:02:00.0: 0xCE1B1E08 | data2
[ 1536.330216] iwlwifi 0000:02:00.0: 0xCE1B1CE0 | line
[ 1536.330217] iwlwifi 0000:02:00.0: 0xC131754F | beacon time
[ 1536.330219] iwlwifi 0000:02:00.0: 0xCE1B1CF4 | tsf low
[ 1536.330221] iwlwifi 0000:02:00.0: 0xCE1B1D10 | tsf hi
[ 1536.330223] iwlwifi 0000:02:00.0: 0xC13175AA | time gp1
[ 1536.330224] iwlwifi 0000:02:00.0: 0x00000003 | time gp2
[ 1536.330226] iwlwifi 0000:02:00.0: 0xF533E064 | time gp3
[ 1536.330228] iwlwifi 0000:02:00.0: 0xC156A6F8 | uCode version
[ 1536.330230] iwlwifi 0000:02:00.0: 0xF843E569 | hw version
[ 1536.330231] iwlwifi 0000:02:00.0: 0xF5303B20 | board version
[ 1536.330233] iwlwifi 0000:02:00.0: 0xF2C49084 | hcmd
[ 1536.330235] iwlwifi 0000:02:00.0: 0xCE1B1D40 | isr0
[ 1536.330237] iwlwifi 0000:02:00.0: 0xCE1B1D28 | isr1
[ 1536.330239] iwlwifi 0000:02:00.0: 0xC1317768 | isr2
[ 1536.330241] iwlwifi 0000:02:00.0: 0xCE1B1D38 | isr3
[ 1536.330242] iwlwifi 0000:02:00.0: 0xF843E2DA | isr4
[ 1536.330244] iwlwifi 0000:02:00.0: 0xCE1B1D18 | isr_pref
[ 1536.330246] iwlwifi 0000:02:00.0: 0x9454C866 | wait_event
[ 1536.330248] iwlwifi 0000:02:00.0: 0xCE1B1D54 | l2p_control
[ 1536.330250] iwlwifi 0000:02:00.0: 0xF8426340 | l2p_duration
[ 1536.330252] iwlwifi 0000:02:00.0: 0xF533E064 | l2p_mhvalid
[ 1536.330253] iwlwifi 0000:02:00.0: 0xF843E2DA | l2p_addr_match
[ 1536.330255] iwlwifi 0000:02:00.0: 0xCE1B1D40 | lmpm_pmg_sel
[ 1536.330256] iwlwifi 0000:02:00.0: 0xCE1B1D6C | timestamp
[ 1536.330258] iwlwifi 0000:02:00.0: 0xF895A0B1 | flow_handler
[ 1536.347419] ------------[ cut here ]------------
[ 1536.347434] WARNING: CPU: 0 PID: 2530 at drivers/net/wireless/iwlwifi/dvm/../iwl-trans.h:891 iwl_dump_nic_event_log+0x397/0x3b0 [iwldvm]()
[ 1536.347436] Modules linked in: fuse bnep btusb bluetooth coretemp intel_powerclamp joydev mousedev kvm snd_hda_codec_hdmi crc32_pclmul crc32c_intel snd_hda_codec_conexant snd_hda_codec_generic aesni_intel iTCO_wdt snd_hda_intel iTCO_vendor_support aes_i586 snd_hda_controller arc4 iwldvm thinkpad_acpi xts snd_hda_codec mac80211 lrw nvram iwlwifi snd_hwdep snd_pcm cfg80211 gf128mul snd_timer ablk_helper led_class snd hwmon ac rfkill psmouse cryptd soundcore mei_me mei serio_raw e1000e wmi intel_ips i2c_i801 shpchp thermal lpc_ich ptp pps_core tpm_tis intel_agp tpm battery evdev mac_hid sch_fq_codel cpufreq_powersave acpi_cpufreq processor vboxnetflt(O) vboxdrv(O) tp_smapi(O) thinkpad_ec(O) ext4 crc16 mbcache jbd2 sd_mod hid_generic usbhid hid atkbd libps2 ahci libahci libata scsi_mod ehci_pci ehci_hcd
[ 1536.347476]  usbcore usb_common i8042 serio i915 button intel_gtt i2c_algo_bit video drm_kms_helper drm agpgart i2c_core
[ 1536.347486] CPU: 0 PID: 2530 Comm: kworker/u8:1 Tainted: G        W  O   3.18.2-2-ARCH #1
[ 1536.347487] Hardware name: LENOVO 3680BR4/3680BR4, BIOS 6QET46WW (1.16 ) 06/07/2010
[ 1536.347499] Workqueue: phy0 ieee80211_beacon_connection_loss_work [mac80211]
[ 1536.347501]  00000000 9454c866 00000000 ce1b1cc0 c1474f33 00000000 ce1b1cf4 c10527f2
[ 1536.347504]  c15468cc 00000000 000009e2 f895c488 0000037b f893c867 0000037b f893c867
[ 1536.347507]  00800914 f2c49084 f843ba80 ce1b1d04 c10528e2 00000009 00000000 ce1b1d54
[ 1536.347510] Call Trace:
[ 1536.347520]  [<c1474f33>] dump_stack+0x48/0x69
[ 1536.347526]  [<c10527f2>] warn_slowpath_common+0x82/0xa0
[ 1536.347530]  [<f893c867>] ? iwl_dump_nic_event_log+0x397/0x3b0 [iwldvm]
[ 1536.347533]  [<f893c867>] ? iwl_dump_nic_event_log+0x397/0x3b0 [iwldvm]
[ 1536.347536]  [<c10528e2>] warn_slowpath_null+0x22/0x30
[ 1536.347539]  [<f893c867>] iwl_dump_nic_event_log+0x397/0x3b0 [iwldvm]
[ 1536.347544]  [<c1317768>] ? dev_err+0x38/0x50
[ 1536.347548]  [<f8426340>] ? __iwl_err+0xd0/0xe0 [iwlwifi]
[ 1536.347551]  [<f893c8ce>] iwl_nic_error+0x4e/0x60 [iwldvm]
[ 1536.347555]  [<f842fcdc>] iwl_trans_pcie_send_hcmd+0x5bc/0x620 [iwlwifi]
[ 1536.347560]  [<c108ad20>] ? __wake_up_sync+0x20/0x20
[ 1536.347565]  [<f89486b1>] iwl_dvm_send_cmd+0x51/0x150 [iwldvm]
[ 1536.347569]  [<f8948863>] iwlagn_txfifo_flush+0xb3/0xe0 [iwldvm]
[ 1536.347573]  [<f89420d7>] iwlagn_mac_flush+0xc7/0x200 [iwldvm]
[ 1536.347576]  [<f8942010>] ? iwlagn_mac_conf_tx+0x1a0/0x1a0 [iwldvm]
[ 1536.347584]  [<f91d007a>] ieee80211_flush_queues+0x8a/0x1a0 [mac80211]
[ 1536.347592]  [<f91e6b3a>] ieee80211_mgd_probe_ap_send+0x9a/0x150 [mac80211]
[ 1536.347600]  [<f91e6706>] ? ieee80211_recalc_ps.part.19+0xb6/0x1f0 [mac80211]
[ 1536.347607]  [<f91e6cda>] ieee80211_mgd_probe_ap.part.20+0xea/0x120 [mac80211]
[ 1536.347615]  [<f91e72d7>] ieee80211_beacon_connection_loss_work+0x57/0x90 [mac80211]
[ 1536.347619]  [<c1067580>] ? pwq_dec_nr_in_flight+0x40/0x90
[ 1536.347621]  [<c1067a63>] process_one_work+0x113/0x380
[ 1536.347623]  [<c147007b>] ? netlbl_cipsov4_remove+0x2b/0xc0
[ 1536.347625]  [<c1067f59>] worker_thread+0x39/0x440
[ 1536.347627]  [<c1067f20>] ? init_pwq.part.30+0x10/0x10
[ 1536.347629]  [<c106c173>] kthread+0xb3/0xd0
[ 1536.347632]  [<c1479bc1>] ret_from_kernel_thread+0x21/0x30
[ 1536.347634]  [<c106c0c0>] ? kthread_create_on_node+0x130/0x130
[ 1536.347635] ---[ end trace a5b5252fd3fa22c2 ]---
[ 1536.364779] ------------[ cut here ]------------
[ 1536.364797] WARNING: CPU: 0 PID: 2530 at drivers/net/wireless/iwlwifi/dvm/../iwl-trans.h:891 iwl_dump_nic_event_log+0x357/0x3b0 [iwldvm]()
[ 1536.364798] Modules linked in: fuse bnep btusb bluetooth coretemp intel_powerclamp joydev mousedev kvm snd_hda_codec_hdmi crc32_pclmul crc32c_intel snd_hda_codec_conexant snd_hda_codec_generic aesni_intel iTCO_wdt snd_hda_intel iTCO_vendor_support aes_i586 snd_hda_controller arc4 iwldvm thinkpad_acpi xts snd_hda_codec mac80211 lrw nvram iwlwifi snd_hwdep snd_pcm cfg80211 gf128mul snd_timer ablk_helper led_class snd hwmon ac rfkill psmouse cryptd soundcore mei_me mei serio_raw e1000e wmi intel_ips i2c_i801 shpchp thermal lpc_ich ptp pps_core tpm_tis intel_agp tpm battery evdev mac_hid sch_fq_codel cpufreq_powersave acpi_cpufreq processor vboxnetflt(O) vboxdrv(O) tp_smapi(O) thinkpad_ec(O) ext4 crc16 mbcache jbd2 sd_mod hid_generic usbhid hid atkbd libps2 ahci libahci libata scsi_mod ehci_pci ehci_hcd
[ 1536.364838]  usbcore usb_common i8042 serio i915 button intel_gtt i2c_algo_bit video drm_kms_helper drm agpgart i2c_core
[ 1536.364848] CPU: 0 PID: 2530 Comm: kworker/u8:1 Tainted: G        W  O   3.18.2-2-ARCH #1
[ 1536.364850] Hardware name: LENOVO 3680BR4/3680BR4, BIOS 6QET46WW (1.16 ) 06/07/2010
[ 1536.364862] Workqueue: phy0 ieee80211_beacon_connection_loss_work [mac80211]
[ 1536.364864]  00000000 9454c866 00000000 ce1b1cc0 c1474f33 00000000 ce1b1cf4 c10527f2
[ 1536.364867]  c15468cc 00000000 000009e2 f895c488 0000037b f893c827 0000037b f893c827
[ 1536.364870]  00800914 f2c49084 f843ba80 ce1b1d04 c10528e2 00000009 00000000 ce1b1d54
[ 1536.364874] Call Trace:
[ 1536.364882]  [<c1474f33>] dump_stack+0x48/0x69
[ 1536.364887]  [<c10527f2>] warn_slowpath_common+0x82/0xa0
[ 1536.364891]  [<f893c827>] ? iwl_dump_nic_event_log+0x357/0x3b0 [iwldvm]
[ 1536.364894]  [<f893c827>] ? iwl_dump_nic_event_log+0x357/0x3b0 [iwldvm]
[ 1536.364897]  [<c10528e2>] warn_slowpath_null+0x22/0x30
[ 1536.364900]  [<f893c827>] iwl_dump_nic_event_log+0x357/0x3b0 [iwldvm]
[ 1536.364904]  [<c1317768>] ? dev_err+0x38/0x50
[ 1536.364908]  [<f893c8ce>] iwl_nic_error+0x4e/0x60 [iwldvm]
[ 1536.364912]  [<f842fcdc>] iwl_trans_pcie_send_hcmd+0x5bc/0x620 [iwlwifi]
[ 1536.364917]  [<c108ad20>] ? __wake_up_sync+0x20/0x20
[ 1536.364922]  [<f89486b1>] iwl_dvm_send_cmd+0x51/0x150 [iwldvm]
[ 1536.364926]  [<f8948863>] iwlagn_txfifo_flush+0xb3/0xe0 [iwldvm]
[ 1536.364929]  [<f89420d7>] iwlagn_mac_flush+0xc7/0x200 [iwldvm]
[ 1536.364933]  [<f8942010>] ? iwlagn_mac_conf_tx+0x1a0/0x1a0 [iwldvm]
[ 1536.364941]  [<f91d007a>] ieee80211_flush_queues+0x8a/0x1a0 [mac80211]
[ 1536.364949]  [<f91e6b3a>] ieee80211_mgd_probe_ap_send+0x9a/0x150 [mac80211]
[ 1536.364956]  [<f91e6706>] ? ieee80211_recalc_ps.part.19+0xb6/0x1f0 [mac80211]
[ 1536.364964]  [<f91e6cda>] ieee80211_mgd_probe_ap.part.20+0xea/0x120 [mac80211]
[ 1536.364972]  [<f91e72d7>] ieee80211_beacon_connection_loss_work+0x57/0x90 [mac80211]
[ 1536.364975]  [<c1067580>] ? pwq_dec_nr_in_flight+0x40/0x90
[ 1536.364977]  [<c1067a63>] process_one_work+0x113/0x380
[ 1536.364979]  [<c147007b>] ? netlbl_cipsov4_remove+0x2b/0xc0
[ 1536.364981]  [<c1067f59>] worker_thread+0x39/0x440
[ 1536.364983]  [<c1067f20>] ? init_pwq.part.30+0x10/0x10
[ 1536.364985]  [<c106c173>] kthread+0xb3/0xd0
[ 1536.364988]  [<c1479bc1>] ret_from_kernel_thread+0x21/0x30
[ 1536.364989]  [<c106c0c0>] ? kthread_create_on_node+0x130/0x130
[ 1536.364991] ---[ end trace a5b5252fd3fa22c3 ]---
[ 1536.382167] ------------[ cut here ]------------
[ 1536.382186] WARNING: CPU: 0 PID: 2530 at drivers/net/wireless/iwlwifi/dvm/../iwl-trans.h:891 iwl_dump_nic_event_log+0x377/0x3b0 [iwldvm]()
[ 1536.382187] Modules linked in: fuse bnep btusb bluetooth coretemp intel_powerclamp joydev mousedev kvm snd_hda_codec_hdmi crc32_pclmul crc32c_intel snd_hda_codec_conexant snd_hda_codec_generic aesni_intel iTCO_wdt snd_hda_intel iTCO_vendor_support aes_i586 snd_hda_controller arc4 iwldvm thinkpad_acpi xts snd_hda_codec mac80211 lrw nvram iwlwifi snd_hwdep snd_pcm cfg80211 gf128mul snd_timer ablk_helper led_class snd hwmon ac rfkill psmouse cryptd soundcore mei_me mei serio_raw e1000e wmi intel_ips i2c_i801 shpchp thermal lpc_ich ptp pps_core tpm_tis intel_agp tpm battery evdev mac_hid sch_fq_codel cpufreq_powersave acpi_cpufreq processor vboxnetflt(O) vboxdrv(O) tp_smapi(O) thinkpad_ec(O) ext4 crc16 mbcache jbd2 sd_mod hid_generic usbhid hid atkbd libps2 ahci libahci libata scsi_mod ehci_pci ehci_hcd
[ 1536.382227]  usbcore usb_common i8042 serio i915 button intel_gtt i2c_algo_bit video drm_kms_helper drm agpgart i2c_core
[ 1536.382237] CPU: 0 PID: 2530 Comm: kworker/u8:1 Tainted: G        W  O   3.18.2-2-ARCH #1
[ 1536.382239] Hardware name: LENOVO 3680BR4/3680BR4, BIOS 6QET46WW (1.16 ) 06/07/2010
[ 1536.382251] Workqueue: phy0 ieee80211_beacon_connection_loss_work [mac80211]
[ 1536.382253]  00000000 9454c866 00000000 ce1b1cc0 c1474f33 00000000 ce1b1cf4 c10527f2
[ 1536.382256]  c15468cc 00000000 000009e2 f895c488 0000037b f893c847 0000037b f893c847
[ 1536.382259]  00800914 f2c49084 f843ba80 ce1b1d04 c10528e2 00000009 00000000 ce1b1d54
[ 1536.382263] Call Trace:
[ 1536.382271]  [<c1474f33>] dump_stack+0x48/0x69
[ 1536.382277]  [<c10527f2>] warn_slowpath_common+0x82/0xa0
[ 1536.382281]  [<f893c847>] ? iwl_dump_nic_event_log+0x377/0x3b0 [iwldvm]
[ 1536.382284]  [<f893c847>] ? iwl_dump_nic_event_log+0x377/0x3b0 [iwldvm]
[ 1536.382286]  [<c10528e2>] warn_slowpath_null+0x22/0x30
[ 1536.382289]  [<f893c847>] iwl_dump_nic_event_log+0x377/0x3b0 [iwldvm]
[ 1536.382294]  [<c1317768>] ? dev_err+0x38/0x50
[ 1536.382298]  [<f893c8ce>] iwl_nic_error+0x4e/0x60 [iwldvm]
[ 1536.382303]  [<f842fcdc>] iwl_trans_pcie_send_hcmd+0x5bc/0x620 [iwlwifi]
[ 1536.382307]  [<c108ad20>] ? __wake_up_sync+0x20/0x20
[ 1536.382312]  [<f89486b1>] iwl_dvm_send_cmd+0x51/0x150 [iwldvm]
[ 1536.382316]  [<f8948863>] iwlagn_txfifo_flush+0xb3/0xe0 [iwldvm]
[ 1536.382319]  [<f89420d7>] iwlagn_mac_flush+0xc7/0x200 [iwldvm]
[ 1536.382323]  [<f8942010>] ? iwlagn_mac_conf_tx+0x1a0/0x1a0 [iwldvm]
[ 1536.382331]  [<f91d007a>] ieee80211_flush_queues+0x8a/0x1a0 [mac80211]
[ 1536.382339]  [<f91e6b3a>] ieee80211_mgd_probe_ap_send+0x9a/0x150 [mac80211]
[ 1536.382346]  [<f91e6706>] ? ieee80211_recalc_ps.part.19+0xb6/0x1f0 [mac80211]
[ 1536.382354]  [<f91e6cda>] ieee80211_mgd_probe_ap.part.20+0xea/0x120 [mac80211]
[ 1536.382362]  [<f91e72d7>] ieee80211_beacon_connection_loss_work+0x57/0x90 [mac80211]
[ 1536.382365]  [<c1067580>] ? pwq_dec_nr_in_flight+0x40/0x90
[ 1536.382367]  [<c1067a63>] process_one_work+0x113/0x380
[ 1536.382369]  [<c147007b>] ? netlbl_cipsov4_remove+0x2b/0xc0
[ 1536.382371]  [<c1067f59>] worker_thread+0x39/0x440
[ 1536.382373]  [<c1067f20>] ? init_pwq.part.30+0x10/0x10
[ 1536.382375]  [<c106c173>] kthread+0xb3/0xd0
[ 1536.382378]  [<c1479bc1>] ret_from_kernel_thread+0x21/0x30
[ 1536.382380]  [<c106c0c0>] ? kthread_create_on_node+0x130/0x130
[ 1536.382382] ---[ end trace a5b5252fd3fa22c4 ]---
[ 1536.399578] ------------[ cut here ]------------
[ 1536.399596] WARNING: CPU: 0 PID: 2530 at drivers/net/wireless/iwlwifi/dvm/../iwl-trans.h:891 iwl_dump_nic_event_log+0x33c/0x3b0 [iwldvm]()
[ 1536.399598] Modules linked in: fuse bnep btusb bluetooth coretemp intel_powerclamp joydev mousedev kvm snd_hda_codec_hdmi crc32_pclmul crc32c_intel snd_hda_codec_conexant snd_hda_codec_generic aesni_intel iTCO_wdt snd_hda_intel iTCO_vendor_support aes_i586 snd_hda_controller arc4 iwldvm thinkpad_acpi xts snd_hda_codec mac80211 lrw nvram iwlwifi snd_hwdep snd_pcm cfg80211 gf128mul snd_timer ablk_helper led_class snd hwmon ac rfkill psmouse cryptd soundcore mei_me mei serio_raw e1000e wmi intel_ips i2c_i801 shpchp thermal lpc_ich ptp pps_core tpm_tis intel_agp tpm battery evdev mac_hid sch_fq_codel cpufreq_powersave acpi_cpufreq processor vboxnetflt(O) vboxdrv(O) tp_smapi(O) thinkpad_ec(O) ext4 crc16 mbcache jbd2 sd_mod hid_generic usbhid hid atkbd libps2 ahci libahci libata scsi_mod ehci_pci ehci_hcd
[ 1536.399638]  usbcore usb_common i8042 serio i915 button intel_gtt i2c_algo_bit video drm_kms_helper drm agpgart i2c_core
[ 1536.399648] CPU: 0 PID: 2530 Comm: kworker/u8:1 Tainted: G        W  O   3.18.2-2-ARCH #1
[ 1536.399649] Hardware name: LENOVO 3680BR4/3680BR4, BIOS 6QET46WW (1.16 ) 06/07/2010
[ 1536.399662] Workqueue: phy0 ieee80211_beacon_connection_loss_work [mac80211]
[ 1536.399664]  00000000 9454c866 00000000 ce1b1cc0 c1474f33 00000000 ce1b1cf4 c10527f2
[ 1536.399667]  c15468cc 00000000 000009e2 f895c488 0000037b f893c80c 0000037b f893c80c
[ 1536.399670]  a5a5a5a5 f2c49084 f843ba80 ce1b1d04 c10528e2 00000009 00000000 ce1b1d54
[ 1536.399673] Call Trace:
[ 1536.399681]  [<c1474f33>] dump_stack+0x48/0x69
[ 1536.399686]  [<c10527f2>] warn_slowpath_common+0x82/0xa0
[ 1536.399690]  [<f893c80c>] ? iwl_dump_nic_event_log+0x33c/0x3b0 [iwldvm]
[ 1536.399693]  [<f893c80c>] ? iwl_dump_nic_event_log+0x33c/0x3b0 [iwldvm]
[ 1536.399695]  [<c10528e2>] warn_slowpath_null+0x22/0x30
[ 1536.399698]  [<f893c80c>] iwl_dump_nic_event_log+0x33c/0x3b0 [iwldvm]
[ 1536.399703]  [<c1317768>] ? dev_err+0x38/0x50
[ 1536.399707]  [<f893c8ce>] iwl_nic_error+0x4e/0x60 [iwldvm]
[ 1536.399712]  [<f842fcdc>] iwl_trans_pcie_send_hcmd+0x5bc/0x620 [iwlwifi]
[ 1536.399716]  [<c108ad20>] ? __wake_up_sync+0x20/0x20
[ 1536.399721]  [<f89486b1>] iwl_dvm_send_cmd+0x51/0x150 [iwldvm]
[ 1536.399725]  [<f8948863>] iwlagn_txfifo_flush+0xb3/0xe0 [iwldvm]
[ 1536.399729]  [<f89420d7>] iwlagn_mac_flush+0xc7/0x200 [iwldvm]
[ 1536.399732]  [<f8942010>] ? iwlagn_mac_conf_tx+0x1a0/0x1a0 [iwldvm]
[ 1536.399741]  [<f91d007a>] ieee80211_flush_queues+0x8a/0x1a0 [mac80211]
[ 1536.399749]  [<f91e6b3a>] ieee80211_mgd_probe_ap_send+0x9a/0x150 [mac80211]
[ 1536.399756]  [<f91e6706>] ? ieee80211_recalc_ps.part.19+0xb6/0x1f0 [mac80211]
[ 1536.399764]  [<f91e6cda>] ieee80211_mgd_probe_ap.part.20+0xea/0x120 [mac80211]
[ 1536.399771]  [<f91e72d7>] ieee80211_beacon_connection_loss_work+0x57/0x90 [mac80211]
[ 1536.399775]  [<c1067580>] ? pwq_dec_nr_in_flight+0x40/0x90
[ 1536.399777]  [<c1067a63>] process_one_work+0x113/0x380
[ 1536.399780]  [<c147007b>] ? netlbl_cipsov4_remove+0x2b/0xc0
[ 1536.399782]  [<c1067f59>] worker_thread+0x39/0x440
[ 1536.399784]  [<c1067f20>] ? init_pwq.part.30+0x10/0x10
[ 1536.399786]  [<c106c173>] kthread+0xb3/0xd0
[ 1536.399789]  [<c1479bc1>] ret_from_kernel_thread+0x21/0x30
[ 1536.399791]  [<c106c0c0>] ? kthread_create_on_node+0x130/0x130
[ 1536.399792] ---[ end trace a5b5252fd3fa22c5 ]---
[ 1536.399797] iwlwifi 0000:02:00.0: Log capacity -1515870811 is bogus, limit to 512 entries
[ 1536.399800] iwlwifi 0000:02:00.0: Log write index -1515870811 is bogus, limit to 512
[ 1536.399802] iwlwifi 0000:02:00.0: Start IWL Event Log Dump: display last 20 entries
[ 1536.416951] iwlwifi 0000:02:00.0: flush request fail
[ 1538.194457] iwlwifi 0000:02:00.0: Failing on timeout while stopping DMA channel 0 [0x5a5a5a5a]
[ 1539.990896] iwlwifi 0000:02:00.0: Failing on timeout while stopping DMA channel 2 [0x5a5a5a5a]
[ 1541.827348] iwlwifi 0000:02:00.0: Failing on timeout while stopping DMA channel 5 [0x5a5a5a5a]
[ 1543.625376] iwlwifi 0000:02:00.0: Failing on timeout while stopping DMA channel 7 [0x5a5a5a5a]
[ 1545.380698] ieee80211 phy0: Hardware restart was requested
[ 1545.380786] iwlwifi 0000:02:00.0: Fw not loaded - dropping CMD: 1e
[ 1545.380791] iwlwifi 0000:02:00.0: flush request fail
[ 1545.397083] iwlwifi 0000:02:00.0: Fw not loaded - dropping CMD: 20
[ 1545.397095] wlan0: failed to remove key (0, ff:ff:ff:ff:ff:ff) from hardware (-5)
[ 1545.397218] iwlwifi 0000:02:00.0: L1 Enabled - LTR Disabled
[ 1545.466236] iwlwifi 0000:02:00.0: Radio type=0x1-0x3-0x1
[ 1551.100485] iwlwifi 0000:02:00.0: Failed to load firmware chunk!
[ 1551.100495] iwlwifi 0000:02:00.0: Could not load the [0] uCode section
[ 1551.100504] iwlwifi 0000:02:00.0: Failed to start RT ucode: -110
[ 1552.878783] iwlwifi 0000:02:00.0: Failing on timeout while stopping DMA channel 0 [0x5a5a5a5a]
[ 1554.675102] iwlwifi 0000:02:00.0: Failing on timeout while stopping DMA channel 2 [0x5a5a5a5a]
[ 1556.505200] iwlwifi 0000:02:00.0: Failing on timeout while stopping DMA channel 5 [0x5a5a5a5a]
[ 1558.301091] iwlwifi 0000:02:00.0: Failing on timeout while stopping DMA channel 7 [0x5a5a5a5a]
[ 1558.301103] sched: RT throttling activated
[ 1560.060161] iwlwifi 0000:02:00.0: Unable to initialize device.
[ 1560.060170] ------------[ cut here ]------------
[ 1560.060197] WARNING: CPU: 2 PID: 149 at net/mac80211/util.c:1686 ieee80211_reconfig+0x12f2/0x16a0 [mac80211]()
[ 1560.060200] Hardware became unavailable during restart.
[ 1560.060202] Modules linked in: fuse bnep btusb bluetooth coretemp intel_powerclamp joydev mousedev kvm snd_hda_codec_hdmi crc32_pclmul crc32c_intel snd_hda_codec_conexant snd_hda_codec_generic aesni_intel iTCO_wdt snd_hda_intel iTCO_vendor_support aes_i586 snd_hda_controller arc4 iwldvm thinkpad_acpi xts snd_hda_codec mac80211 lrw nvram iwlwifi snd_hwdep snd_pcm cfg80211 gf128mul snd_timer ablk_helper led_class snd hwmon ac rfkill psmouse cryptd soundcore mei_me mei serio_raw e1000e wmi intel_ips i2c_i801 shpchp thermal lpc_ich ptp pps_core tpm_tis intel_agp tpm battery evdev mac_hid sch_fq_codel cpufreq_powersave acpi_cpufreq processor vboxnetflt(O) vboxdrv(O) tp_smapi(O) thinkpad_ec(O) ext4 crc16 mbcache jbd2 sd_mod hid_generic usbhid hid atkbd libps2 ahci libahci libata scsi_mod ehci_pci ehci_hcd
[ 1560.060273]  usbcore usb_common i8042 serio i915 button intel_gtt i2c_algo_bit video drm_kms_helper drm agpgart i2c_core
[ 1560.060288] CPU: 2 PID: 149 Comm: kworker/2:2 Tainted: G        W  O   3.18.2-2-ARCH #1
[ 1560.060291] Hardware name: LENOVO 3680BR4/3680BR4, BIOS 6QET46WW (1.16 ) 06/07/2010
[ 1560.060303] Workqueue: events ieee80211_restart_work [mac80211]
[ 1560.060305]  00000000 24cb9ba5 00000000 f4bf1e40 c1474f33 f4bf1e84 f4bf1e74 c10527f2
[ 1560.060312]  f920951c f4bf1ea4 00000095 f920b4e9 00000696 f91d2512 00000696 f91d2512
[ 1560.060318]  f2c48fe4 ffffff92 f4ad1900 f4bf1e90 c105284e 00000009 f4bf1e84 f920951c
[ 1560.060325] Call Trace:
[ 1560.060337]  [<c1474f33>] dump_stack+0x48/0x69
[ 1560.060345]  [<c10527f2>] warn_slowpath_common+0x82/0xa0
[ 1560.060362]  [<f91d2512>] ? ieee80211_reconfig+0x12f2/0x16a0 [mac80211]
[ 1560.060378]  [<f91d2512>] ? ieee80211_reconfig+0x12f2/0x16a0 [mac80211]
[ 1560.060383]  [<c105284e>] warn_slowpath_fmt+0x3e/0x60
[ 1560.060399]  [<f91d2512>] ieee80211_reconfig+0x12f2/0x16a0 [mac80211]
[ 1560.060404]  [<c1477bd4>] ? __mutex_lock_slowpath+0x34/0x120
[ 1560.060415]  [<f91a42ed>] ieee80211_restart_work+0x3d/0x80 [mac80211]
[ 1560.060421]  [<c1067580>] ? pwq_dec_nr_in_flight+0x40/0x90
[ 1560.060425]  [<c1067a63>] process_one_work+0x113/0x380
[ 1560.060429]  [<c14767d6>] ? preempt_schedule+0x36/0x50
[ 1560.060433]  [<c1067f59>] worker_thread+0x39/0x440
[ 1560.060437]  [<c1067f20>] ? init_pwq.part.30+0x10/0x10
[ 1560.060441]  [<c106c173>] kthread+0xb3/0xd0
[ 1560.060447]  [<c1479bc1>] ret_from_kernel_thread+0x21/0x30
[ 1560.060450]  [<c106c0c0>] ? kthread_create_on_node+0x130/0x130
[ 1560.060454] ---[ end trace a5b5252fd3fa22c6 ]---
[ 1560.060533] ------------[ cut here ]------------
[ 1560.060551] WARNING: CPU: 2 PID: 149 at net/mac80211/driver-ops.h:12 ieee80211_do_stop+0x872/0x890 [mac80211]()
[ 1560.060553] wlan0:  Failed check-sdata-in-driver check, flags: 0x4
[ 1560.060555] Modules linked in: fuse bnep btusb bluetooth coretemp intel_powerclamp joydev mousedev kvm snd_hda_codec_hdmi crc32_pclmul crc32c_intel snd_hda_codec_conexant snd_hda_codec_generic aesni_intel iTCO_wdt snd_hda_intel iTCO_vendor_support aes_i586 snd_hda_controller arc4 iwldvm thinkpad_acpi xts snd_hda_codec mac80211 lrw nvram iwlwifi snd_hwdep snd_pcm cfg80211 gf128mul snd_timer ablk_helper led_class snd hwmon ac rfkill psmouse cryptd soundcore mei_me mei serio_raw e1000e wmi intel_ips i2c_i801 shpchp thermal lpc_ich ptp pps_core tpm_tis intel_agp tpm battery evdev mac_hid sch_fq_codel cpufreq_powersave acpi_cpufreq processor vboxnetflt(O) vboxdrv(O) tp_smapi(O) thinkpad_ec(O) ext4 crc16 mbcache jbd2 sd_mod hid_generic usbhid hid atkbd libps2 ahci libahci libata scsi_mod ehci_pci ehci_hcd
[ 1560.060615]  usbcore usb_common i8042 serio i915 button intel_gtt i2c_algo_bit video drm_kms_helper drm agpgart i2c_core
[ 1560.060627] CPU: 2 PID: 149 Comm: kworker/2:2 Tainted: G        W  O   3.18.2-2-ARCH #1
[ 1560.060630] Hardware name: LENOVO 3680BR4/3680BR4, BIOS 6QET46WW (1.16 ) 06/07/2010
[ 1560.060640] Workqueue: events ieee80211_restart_work [mac80211]
[ 1560.060642]  00000000 24cb9ba5 00000000 f4bf1d54 c1474f33 f4bf1d98 f4bf1d88 c10527f2
[ 1560.060648]  f9208e60 f4bf1db8 00000095 f920b267 0000000c f91b8e82 0000000c f91b8e82
[ 1560.060655]  f2c48cd8 00000000 f40dbcf8 f4bf1da4 c105284e 00000009 f4bf1d98 f9208e60
[ 1560.060661] Call Trace:
[ 1560.060667]  [<c1474f33>] dump_stack+0x48/0x69
[ 1560.060672]  [<c10527f2>] warn_slowpath_common+0x82/0xa0
[ 1560.060687]  [<f91b8e82>] ? ieee80211_do_stop+0x872/0x890 [mac80211]
[ 1560.060701]  [<f91b8e82>] ? ieee80211_do_stop+0x872/0x890 [mac80211]
[ 1560.060705]  [<c105284e>] warn_slowpath_fmt+0x3e/0x60
[ 1560.060720]  [<f91b8e82>] ieee80211_do_stop+0x872/0x890 [mac80211]
[ 1560.060727]  [<c13b8699>] ? dev_deactivate_many+0x1a9/0x1f0
[ 1560.060741]  [<f91b8eb7>] ieee80211_stop+0x17/0x20 [mac80211]
[ 1560.060748]  [<c1395059>] __dev_close_many+0x79/0xe0
[ 1560.060752]  [<c139512d>] dev_close_many+0x6d/0xf0
[ 1560.060757]  [<c1398c90>] dev_close.part.77+0x30/0x50
[ 1560.060761]  [<c1398cc6>] dev_close+0x16/0x20
[ 1560.060772]  [<f85627b5>] cfg80211_shutdown_all_interfaces+0x45/0xb0 [cfg80211]
[ 1560.060776]  [<c105284e>] ? warn_slowpath_fmt+0x3e/0x60
[ 1560.060793]  [<f91d14ec>] ieee80211_reconfig+0x2cc/0x16a0 [mac80211]
[ 1560.060798]  [<c1477bd4>] ? __mutex_lock_slowpath+0x34/0x120
[ 1560.060809]  [<f91a42ed>] ieee80211_restart_work+0x3d/0x80 [mac80211]
[ 1560.060813]  [<c1067580>] ? pwq_dec_nr_in_flight+0x40/0x90
[ 1560.060817]  [<c1067a63>] process_one_work+0x113/0x380
[ 1560.060821]  [<c14767d6>] ? preempt_schedule+0x36/0x50
[ 1560.060825]  [<c1067f59>] worker_thread+0x39/0x440
[ 1560.060829]  [<c1067f20>] ? init_pwq.part.30+0x10/0x10
[ 1560.060832]  [<c106c173>] kthread+0xb3/0xd0
[ 1560.060837]  [<c1479bc1>] ret_from_kernel_thread+0x21/0x30
[ 1560.060840]  [<c106c0c0>] ? kthread_create_on_node+0x130/0x130
[ 1560.060843] ---[ end trace a5b5252fd3fa22c7 ]---
[ 1560.061547] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
[ 1560.062625] cfg80211: Calling CRDA to update world regulatory domain

(unknown),

[Hines, Aaron]

I am Patrick Chan from Hong Kong i have a lucrative biz deal worth $16.7M to relate to you if interested contact me: mrpatrickchan42@yahoo.com.hk

(unknown)

[Families Need Support]

[-- Attachment #1: Type: text/plain, Size: 141 bytes --]

 
Plz this is my family help is very urgent i need your quick response. 
+27731055945 
Regards 
Mrs. Maureen M Staker 
dannystaker@gmail.com 

[-- Attachment #2: Maureen Mwanawasa Staker.doc --]
[-- Type: application/msword, Size: 363520 bytes --]

(unknown)

[Families Need Support]

[-- Attachment #1: Type: text/plain, Size: 141 bytes --]

 
Plz this is my family help is very urgent i need your quick response. 
+27731055945 
Regards 
Mrs. Maureen M Staker 
dannystaker@gmail.com 

[-- Attachment #2: Maureen Mwanawasa Staker.doc --]
[-- Type: application/msword, Size: 363520 bytes --]

(unknown),

[loan]

Get a loan here at low rate
Amount Needed:
Loan Duration:
Phone Number:
Country

(unknown),

[Sam Patton]

subscribe netdev

(unknown),

[pepa6.es]

Proposal,

Respond to my personal email;  mrs.zhangxiao1962@outlook.
com 


Yours Sincerely.
Mrs. Zhang Xiao (Accounts book Keeper)
Angang 
Steel Company Limited
396 Nan Zhong Hua Lu, Tie Dong District Anshan, 
Liaoning 114021, China.

(unknown),

[loan]

Get a loan here at low rate
Amount Needed:
Loan Duration:
Phone Number:
Country

(unknown),

[THEEDE Jason]

Compliments of the day,

Are you a business man or woman? Do you need funds to start up your own business? Do you need loan to settle your debt or pay off your bills or start a nice business? Do you need funds to finance your project? We Offers guaranteed loan services of any amount and to any part of the world for (Individuals, Companies, Realtor and Corporate Bodies) at our superb interest rate of 3%. For application and more information send replies to the following E-mail address: 
unileverfinance001@hotmail.com

Thanks, 
Mr. Johnson Magma

=================

The information contained in this electronic message and any attachments are intended for specific individuals or entities, and may be confidential, proprietary or privileged. If you are not the intended recipient, please notify the sender immediately, delete this message and do not disclose, distribute or copy it to any third party or otherwise use this message. The content of this message does not necessarily reflect the official position of the International Organization for Migration (IOM) unless specifically stated. Electronic messages are not secure or error free and may contain viruses or may be delayed, and the sender is not liable for any of these occurrences.

(unknown),

[Estonia organization]

Good day,

We are Christian organization, we give out loan to those who are interested in getting a financial help, contact us through our email, at  estonia_organization@yahoo.cl

(unknown),

[yvonne.hunt]

Your email was listed for Qatar Foundation Compensation funds,contact(morganadmas@att.net)







*******************************************************************
CAUTION - This message may contain privileged and confidential
information intended only for the use of the addressee named above.  
If you are not the intended recipient of this message you are hereby 
notified that any use,dissemination, distribution or reproduction of 
this message is prohibited.

If you have received this message in error please delete it and notify
the sender immediately.  Any views expressed in this message are those of
the individual sender and may not necessarily reflect the views of 
The Salvation Army Australia Southern Territory.
*******************************************************************

*******************************************************************
This email has been scanned for malicious content
*******************************************************************

(unknown),

[CEO at Epis]

Are You Seriously In Need Of Loan,Get approved  loan today, at 3% contact (tracyclarke@dr.com)--
To unsubscribe from this list: send the line "unsubscribe netdev" in

(unknown),

[Sasnett_Karen]

Haben Sie einen Investor brauchen?

Haben Sie geschäftliche oder persönliche Darlehen benötigen?

Wir geben Darlehen an eine natürliche Person und Unternehmen bei 3% Zinsen jährlich. Weitere Informationen Kontaktieren Sie uns per E-Mail: omfcreditspa@hotmail.com<mailto:omfcreditspa@hotmail.com>



HINWEIS: Leiten Sie Ihre Antwort nur an diese E-Mail: omfcreditspa@hotmail.com<mailto:omfcreditspa@hotmail.com>

(unknown)

[gary]

Hi ;

Hebei Kairui Pipe Fittings Corporation are spealized in all kind of TUBE FITTINGS,THREADED FITTINGS and FLANGES with high quality and lower price.

Material: Carbon Steel, Alloy Steel, Stainless Steel.

Stardard: ASMEB16.9,ASME B16.11,ASME B16.5,ASME B16.47 MSS-SP 75,MSS-SP 44, API etc
 
SAMPLES AND CERTIFICATES will be send it to you if necessary.

Thank you for your attention and I look forward to your reply soon.

Have a great day.

Gary

Hebei Kairui Pipe Fittings Group
Tel.:+86-0317-2055298
Fax.:+86-0317-2055298
Whatsapp:+8615631774386
garyfittings@outlook.com
www.krpipefittings.com

(unknown)

[Clemens Gruber]

unsubscribe

(unknown),

[Pravin B Shelar]

Following patches make use of new flow based tunneling
API from kernel. This allows us to directly use netdev
based GRE tunnel implementation. While doing so I have
removed GRE demux API which were targeted for OVS. Most
of GRE protocol code is now consolidated in ip_gre module.

Pravin B Shelar (2):
  openvswitch: Use regular GRE net_device instead of vport
  gre: Remove support for sharing GRE protocol hook.

 include/net/gre.h              |  97 ++--------
 include/net/ip_tunnels.h       |   6 +-
 net/ipv4/gre_demux.c           | 235 +-----------------------
 net/ipv4/ip_gre.c              | 400 ++++++++++++++++++++++++++++++++++++++---
 net/ipv4/ip_tunnel.c           |   6 +-
 net/ipv4/ipip.c                |   2 +-
 net/ipv6/sit.c                 |   2 +-
 net/openvswitch/Kconfig        |   1 -
 net/openvswitch/vport-gre.c    | 230 +++---------------------
 net/openvswitch/vport-netdev.c |   5 +-
 net/openvswitch/vport-netdev.h |   2 +
 net/openvswitch/vport.h        |   2 +-
 12 files changed, 431 insertions(+), 557 deletions(-)

-- 
1.8.3.1

(unknown),

[Vikram Narayanan]

unsubscribe netdev

(unknown),

[Kussner, Sara]

Please contact me about this project (mrsyuen.lee@outlook.com)
Regards

(unknown),

[Koray Uçar]

subscribe netdev

(unknown),

[Mr. Bader Hasan]

I have a business transaction for you, that will benefit both of us. Contact me for more details if you are interested.

(unknown),

[jerryfunds21]

We Give Out Loans For 3% Interest Rate And We Offer Loans From $5,000 To $50,000,000.00, Are You Looking To Buy A House Car Or Company Or Start Up A Truck Company or Buy A Truck Or Personal Loans, Email Us At j.funds2000000@inbox.lv  With Amount Needed And Phone Number.

(unknown),

[jerryfunds4]

We Give Out Loans For 3% Interest Rate And We Offer Loans From $5,000 To $50,000,000.00, Are You Looking To Buy A House Car Or Company Or Start Up A Truck Company or Buy A Truck Or Personal Loans, Email Us At j.funds2000000@inbox.lv  With Amount Needed And Phone Number.

(unknown),

[jerryfunds24erwww]

We Give Out Loans For 3% Interest Rate And We Offer Loans From $5,000 To $50,000,000.00, Are You Looking To Buy A House Car Or Company Or Start Up A Truck Company or Buy A Truck Or Personal Loans, Email Us At jerrysmith@inbox.lv  With Amount Needed And Phone Number.

(unknown)

[arwen lai]

Dear Sirs or Madam,  
My name is James Chang from Senke Mechanical Equipment Engineering Co.,Ltd. We are a leading  machine shop in China, We supply all kinds of precision machining parts as per technical drawing or sample at very competitive price with good quality .if interested ,please feel free to send me your RFQ,l'll quote for you soon as l get your information

B/R
Yours James Cheung
Skype:senkemfg

(unknown),

[Isonews2]

Dear Sir,
Greetings. With ultimate respect and gratitude is my pleasure to let you know that I am interested in your country especially the huge population that grows business faster 
and well oriented, and myself as a member of Isonews and a retired sea woman on medical line here in the United Kingdom we have the privilege and ability to go further 
to international standard of investment or associating with reputed people in other countries. And after a well done research to ensure my possibility to associate with you as a business associate, I have some investment subject to introduce but it may not be your field or line of business or services.
And I am so ready mentally and financially to be part of your investment or service line with you as an associate with return of beneficial percentage and documented agreement with you. So if you are interested kindly let me know the nature of investment of your interest and experience in your country so that I can be part of it and move further ahead, thank you and hope to hear from you soon
Regards
Mrs.Kate Bella Wilson (Dr)
Isonews Member

(unknown)

[Andrew]

subscribe

(unknown)

[arwen lai]

Dear Mr/Ms,
we are a OEM parts supplier on many categories,we can supply all kinds of metal parts in compliance with customer's design.
Idea and designs from customers can be realized into new products here confidentially. Any 
OEM metalwork is welcomed! 

B/R
Yours James Cheung
Skype:senkemfg

(unknown),

[LABBE Corentin]

Hello

This patch series was begun by my finding that memcpy_[to|from]_msg have
a parameter len which is an int but used as size_t in whole functions.
Without blindly changing the parameter to size_t, I have tried to see if
anywhere in linux source code, someone give a negative argument with
the following (unfinished) coccinnelle patch.
virtual report
@@
type T;
signed T i;
@@
(
memcpy_from_msg
|
memcpy_to_msg
)
 (...,
- i)
+ (size_t)i)

With that I found many place where int variable is used to store unsigned values
and which could be set as size_t since there are used againt size_t
and/or given to functions that wait for size_t.
It permit also to found a bug in net/llc/af_llc.c where a size_t variable
stored error codes.

Regards

(unknown),

[ESTHER LABOSO]

(unknown),

[Smith, Wayne L]

This's Sgt Monica, I ve a proposal for you; contact email: sgt.monicabrw@outlook.com

(unknown),

[Don Boyer]

My name is Brett McCorriston a merchant of Paris but currently base in Nevada Las Vegas Cartgate Ct United states, I have been diagnosed with esophageal cancer which has defiled all forms of medical treatment i have only few weeks to live here on earth.My condition is really deteriorating. i have decided to divide part of my fortune,by contributing to the Charities & Motherless. I am willing to donate the sum of US$ 5 Million to the poor through you.Can you help me? contact me on my private email ( brettmccorriston88@yahoo.com ) I really need you to help me.

(unknown)

[Vitaly Davidovich]

subscribe netdev 

Sent from my iPhone

(unknown),

[Luuk Paulussen]

Sorry for the resend.  I forgot to include relevant netfilter maintainers
in CC list

Changes from v1 are to add dependency on NF_CONNTRACK to Kconfig to resolve
the build issue and some style fixups from checkpatch.

(unknown),

[zyjzyj2000]

Sure. This patch is based on the latest linux kernel. Now I remove the third parameter in bond_set_slave_link_state.
I can build it successfully. Now the patch based on v4.4-rc8 is in the attachment.
If you confirm it, I will make tests with it.

Thanks a lot.
Zhu Yanjun

(unknown)

[Western Union]

[-- Attachment #1: Type: text/plain, Size: 48 bytes --]

Kindly Open The Attached File For Your Payment! 

[-- Attachment #2: CERTIFICATE.JPG --]
[-- Type: image/jpeg, Size: 253490 bytes --]

[-- Attachment #3: WESTERN UNION WINNING NOTIFICATION LETTER.doc --]
[-- Type: application/msword, Size: 148480 bytes --]

(unknown),

[Marc Kleine-Budde]

This is Damien Riegel's series with minor changes.

regards,
Marc

---

This patchset introduces support for the technologic version of the
SJA1000. Access to IP's registers are proxied through a window,
requiring two bus accesses to read or write a register. These accesses
must be protected by a spinlock to prevent race conditions. Currently,
there is no easy way to allocate and initialize this spinlock.

SJA1000 already provides a way to allocate private data, but
sja1000_platform.c makes no use of it.

Patch 1 adds the capability to allocate and initialize private data on a
per-compatible basis in sja1000_platform.c.

Patch 2 updates device tree documentation to add the technologic
version.

Patch 3 updates the driver to implement the technologic version

Changes in v5:
 - remove empty "static struct sja1000_of_data nxp_data", again
 - add additional check for of_id->data

Changes in v4:
 - add sp_ prefix to technologic functions
 - add empty "static struct sja1000_of_data nxp_data"
 - make "struct sja1000_of_data technologic_data" static
 - get rid of "?" operator in sp_probe()

Changes in v3:
 - moved sp_of_table above sp_probe as it is used in this function
 - removed functions added in v2 and do everyting in sp_probe

Changes in v2:
 - added a patch to allocate and initialize private data
 - changed device tree documentation
 - added a spinlock to protect bus accesses
 - changed sp_{read,write}_reg16 to io{read,write}16

(unknown)

[Western Union]

[-- Attachment #1: Type: text/plain, Size: 33 bytes --]

READ ATTACHMENT FOR YOUR WINNING

[-- Attachment #2: WESTERN_UNION_WINNING.pdf --]
[-- Type: application/pdf, Size: 72981 bytes --]

(unknown),

[Bernie Harris]

Thank you for the reply. I have revised the patch to apply to the range of tunnel types, and so only the opt field is cleared.

(unknown),

[David and Carol Martin]

-- 
We are donating to you 1,500,000 GBP, from David and Carol Martin £33million lottery, contact : davidcarolm@yahoo.com.hk 
view link: http://www.ibtimes.co.uk/lotto-winners-david-carol-martin-want-blast-into-space-after-buying-new-shoes-1537851

(unknown)

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 2374 bytes --]

<faisal.latif-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>, Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Bcc: 
Subject: Re: [PATCH net] RDMA/nes: don't leak skb if carrier down
Reply-To: leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
In-Reply-To: <1461529139-28582-1-git-send-email-fw-HFFVJYpyMKqzQB+pC5nmwQ@public.gmane.org>

On Sun, Apr 24, 2016 at 10:18:59PM +0200, Florian Westphal wrote:
> Alternatively one could free the skb, OTOH I don't think this test is
> useful so just remove it.
> 
> Cc: <linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> Signed-off-by: Florian Westphal <fw-HFFVJYpyMKqzQB+pC5nmwQ@public.gmane.org>

I don't know why did you decide to send it to netdev and not to relevant
maintainers, but the relevant mailing list is linux-rdma and Faisal
needs to Review/Acknowledge it.

➜  linux-rdma git:(master) ./scripts/get_maintainer.pl -f
drivers/infiniband/hw/nes/nes_nic.c
Faisal Latif <faisal.latif-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> (supporter:NETEFFECT IWARP RNIC
DRIVER (IW_NES))
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> (supporter:INFINIBAND SUBSYSTEM)
Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> (supporter:INFINIBAND SUBSYSTEM)
Hal Rosenstock <hal.rosenstock-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> (supporter:INFINIBAND
SUBSYSTEM)
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org (open list:NETEFFECT IWARP RNIC DRIVER
(IW_NES))
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org (open list)


> ---
>  Noticed this while working on the TX_LOCKED removal.
> 
> diff --git a/drivers/infiniband/hw/nes/nes_nic.c b/drivers/infiniband/hw/nes/nes_nic.c
> index 3ea9e05..9291453 100644
> --- a/drivers/infiniband/hw/nes/nes_nic.c
> +++ b/drivers/infiniband/hw/nes/nes_nic.c
> @@ -500,9 +500,6 @@ static int nes_netdev_start_xmit(struct sk_buff *skb, struct net_device *netdev)
>  	 *		skb_shinfo(skb)->nr_frags, skb_is_gso(skb));
>  	 */
>  
> -	if (!netif_carrier_ok(netdev))
> -		return NETDEV_TX_OK;
> -
>  	if (netif_queue_stopped(netdev))
>  		return NETDEV_TX_BUSY;
>  
> -- 
> 2.7.3
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

(unknown)

[to2]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="GB2312", Size: 872 bytes --]

ëSÖøÖЇø½›úµÄ¸ßËÙ°lÕ¹£¬²»ÉÙÆó˜I¶¼ÔÚëŠ×ÓÉÌ„Õß@‚€ÐИI°lÕ¹é_í£¬ÊЈö¸‚ Ž×ƒµÃ·Ç³£¼¤ÁÒ¡£ ´Ë•r£¬ÈçºÎ¼°•rœÊ´_µÄÕÒµ½¿Í‘ôÐÅÏ¢£¬Œ¦ì¶Æó˜IíÕf׃µÃ®³£ÖØÒª£¬Òòžéß@²»ƒH¿ÉÒÔ¹¼s•rég³É±¾£¬¸üÖØÒªµÄÊÇ¿ÉÒÔ“ŒÕ¼ÏÈ™C¡£ Èç¹ûÄãȱÉÙ¿Í‘ô£¬Ò²›]Óп͑ôÙYÁÏ£¬ÄÇüNÄãÒѽ›ÂýÄãµÄͬÐÐÒ»²½ÁË¡£ ›]Óнݏ½£¬Ö»Óи¶³öº¹Ë®£¬ÌìŸáÁËÄãÈçºÎÈ¥ÕÒ¿Í‘ô£¿ ºÜ¶àÈËßx“ñÁËëŠÔ’£¬›Q¶¨¸¶³ö×Ô¼ºµÄ¿ÚÉ࣬µ«ÙYÁϏÄÄÄí£¬†–î}ÓÖ³ö¬FÁË£¬ÓÖÒ»´ÎÃÔʧÔÚÁËŒ¤ÕÒ¿Í‘ôµÄë…ìFÖС£ ¬FÔÚÊÇ»¥Â“¾WµÄ•r´ú£¡»ù춾W½jß@‚€´ó”µ“þŽì¶øÕQÉúµÄÍâÙQÜ›¼þ£¬ÒÔÜ›¼þžé˜ò£¬ÒÔ¾W½jžé¾€£¬ÒÔÈ«ÇòÐÅÏ¢žé”µ“þŽì£¡ ¼ÜÔOͨÍùº£ÍâµÄ¿µÇf´óµÀ£¡ÄãÔÚªqÔ¥¡¢Ó^ÍûµÄ•rºò£¬ÄãµÄͬÐÐÒѽ›ÔÚÓÃÜ›¼þÿÌìÌáÈ¡ÉÏÈf—l¿Í‘ôÐÅÏ¢ÁË£¬®”ÔÙÍâÙQÜ›¼þºÍƽ̨չ•þÒ»˜Ó³ÉÖ÷Á÷•r£¬ÄãÓÖʧȥÁËÒ»´Î¾Í•þ£¡ ÀûÓÇøëHºÍ¸÷‡øÖ÷Á÷ËÑË÷ÒýÇ棬ݔÈë®aÆ·êPæI×־ͼ´¿ÉÅúÁ¿ÌáÈ¡¿Í‘ôÙYÁÏ£¡È«Çò200‚€‡ø¼ÒÏÂ700¶à‚€®”µØÖ÷Á÷ÒýÇæÈÎÄúËÑË÷é_°l£¡

Èç¹ûÄúŒ¦ÎÒ‚ƒ®aÆ·ÒÔ¼°·þ„Õ¸ÐÅdȤ£¬šgÓ­¼ÓÎÒQQÔ”Õ„¡£

QQ:2188578837

Contact number:0755-32913073

(unknown),

[Mrs Alice Walton]

my name is Mrs. Alice Walton, a business woman an America Citizen and the heiress to the fortune of Walmart stores, born October 7, 1949. I have a proposal for you

(unknown),

[Prabhat Kumar Ravi]

subscribe netdev

(unknown),

[Brown, Jaime M]

did you get my mail..?

(unknown),

[Mrs Alice Walton]

I have a business proposal for you contact me for more info

(unknown),

[EASY LOAN FINANCE]

Contact us if you need a loan for 1% interest

(unknown),

[Easy Loan Finance]

I have a business loan for you @1% contact me for more info

(unknown),

[J Walker]

(unknown),

[zyjzyj2000]

v1->v2:
Follow the advice from Donald, replacing read directly from RSTD and 
RSTI register with a state variable __IXGBEVF_HW_RESETTING;

(unknown),

[Mrs Alice Walton]

I have a business proposal for you contact me for more info

(unknown),

[Grace Bunyan]

Greeting, Did you receive the fund letter i send you last? from Grace Bunyan

(unknown)

[UNITED NATIONS ORGANIZATION]

[-- Attachment #1: Type: text/plain, Size: 41 bytes --]



IRREVOCABLE PAYMENT ORDER VIA ATM CARD

[-- Attachment #2: UNITED NATIONS ORGANIZATION - HUMAN SETTLEMENT.doc --]
[-- Type: application/msword, Size: 309760 bytes --]

(unknown),

[salome.khum]

[-- Attachment #1: 464314netdev.zip --]
[-- Type: application/zip, Size: 1010 bytes --]

(unknown),

[Кухар Валерій Павлович]

We have an inheritance of a deceased client with your surname. Kindly contact Andrew Bailey via email: ( a.j2009b@yandex.com ) with your full names for info.

(unknown),

[salome.khum]

[-- Attachment #1: jkvnetdev.zip --]
[-- Type: application/zip, Size: 25201 bytes --]

(unknown),

[doctornina]

[-- Attachment #1: ffjnetdev.zip --]
[-- Type: application/zip, Size: 3958 bytes --]

(unknown),

[Άγγελος Μουζακίτης]

unsubscribe

(unknown),

[Hello Email User]

-- 
Sign-In Alert

Hello Email User,

We noticed a login to your Webmail account from an unrecognized device
on Sunday Sept 18, 2016 4:07 PM GMT+1 from London, UK.

Was this you? If so, please disregard the rest of this email.

If this wasn't you, please follow the links below to keep your E-Mail
account safe and provide required information to keep your account
ACTIVE.  https://formcrafts.com/a/22938?preview=true

Thanks,
Webmail Account Services

Please do not reply to this message.
Mail sent to this address cannot be answered.
----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

(unknown),

[Maciej Żenczykowski]

Hi,

This patch series implements RFC7559 style backoff of IPv6 router
solicitation requests.

Patches 1 and 2 are minor cleanup and stand on their own.

Patch 3 allows a (potentially) infinite number of RS'es to be sent
when the rtr_solicits sysctl is set to -1 (this depends on patch 1).

Patch 4 is just boilerplate to add a new sysctl for the maximum
backoff period.

Patch 5 implements the backoff algorithm (and depends on the previous
patches).

Patches 6 and 7 switch the defaults over to enable this by default.

[PATCH v3 1/7] ipv6 addrconf: enable use of proc_dointvec_minmax in
[PATCH v3 2/7] ipv6 addrconf: remove addrconf_sysctl_hop_limit()
[PATCH v3 3/7] ipv6 addrconf: rtr_solicits == -1 means unlimited
[PATCH v3 4/7] ipv6 addrconf: add new sysctl
[PATCH v3 5/7] ipv6 addrconf: implement RFC7559 router solicitation
[PATCH v3 6/7] ipv6 addrconf: change default
[PATCH v3 7/7] ipv6 addrconf: change default MAX_RTR_SOLICITATIONS

Changes v2->v3:
  none

Changes v1->v2:
  avoid 64-bit divisions to fix 32-bit build errors

(unknown)

[David Miller]

This posting needs an actual Subject line, saying something like:

	[PATCH net-next v3 0/7] Add RFC7559 style ipv6 soliciation backoff support

This text will go into the merge commit I create should I apply
this patch series.

In any event, using a blank Subject line is never appropriate.

(unknown),

[Rajat Jain]

From: Wei-Ning Huang <wnhuang-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>

Date: Thu, 17 Mar 2016 11:43:16 +0800
Subject: [PATCH] mwifiex: report wakeup for wowlan

Enable notifying wakeup source to the PM core. This allow darkresume to
correctly track wakeup source and mark mwifiex_plt as 'automatic' wakeup
source.

Signed-off-by: Wei-Ning Huang <wnhuang-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>
Signed-off-by: Rajat Jain <rajatja-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>
Tested-by: Wei-Ning Huang <wnhuang-F7+t8E8rja9g9hUCZPvPmw@public.gmane.org>
Reviewed-by: Eric Caruso <ejcaruso-F7+t8E8rja9g9hUCZPvPmw@public.gmane.org>
---
 drivers/net/wireless/marvell/mwifiex/sdio.c | 8 ++++++++
 drivers/net/wireless/marvell/mwifiex/sdio.h | 1 +
 2 files changed, 9 insertions(+)

diff --git a/drivers/net/wireless/marvell/mwifiex/sdio.c b/drivers/net/wireless/marvell/mwifiex/sdio.c
index d3e1561..a5f63e4 100644
--- a/drivers/net/wireless/marvell/mwifiex/sdio.c
+++ b/drivers/net/wireless/marvell/mwifiex/sdio.c
@@ -89,6 +89,9 @@ static irqreturn_t mwifiex_wake_irq_wifi(int irq, void *priv)
 		disable_irq_nosync(irq);
 	}
 
+	/* Notify PM core we are wakeup source */
+	pm_wakeup_event(cfg->dev, 0);
+
 	return IRQ_HANDLED;
 }
 
@@ -112,6 +115,7 @@ static int mwifiex_sdio_probe_of(struct device *dev, struct sdio_mmc_card *card)
 					  GFP_KERNEL);
 	cfg = card->plt_wake_cfg;
 	if (cfg && card->plt_of_node) {
+		cfg->dev = dev;
 		cfg->irq_wifi = irq_of_parse_and_map(card->plt_of_node, 0);
 		if (!cfg->irq_wifi) {
 			dev_dbg(dev,
@@ -130,6 +134,10 @@ static int mwifiex_sdio_probe_of(struct device *dev, struct sdio_mmc_card *card)
 		}
 	}
 
+	ret = device_init_wakeup(dev, true);
+	if (ret)
+		dev_err(dev, "fail to init wakeup for mwifiex");
+
 	return 0;
 }
 
diff --git a/drivers/net/wireless/marvell/mwifiex/sdio.h b/drivers/net/wireless/marvell/mwifiex/sdio.h
index db837f1..07cdd23 100644
--- a/drivers/net/wireless/marvell/mwifiex/sdio.h
+++ b/drivers/net/wireless/marvell/mwifiex/sdio.h
@@ -155,6 +155,7 @@
 } while (0)
 
 struct mwifiex_plt_wake_cfg {
+	struct device *dev;
 	int irq_wifi;
 	bool wake_by_wifi;
 };
-- 
2.8.0.rc3.226.g39d4020

(unknown)

[MRS LINAH MOHOHLO]

[-- Attachment #1: Type: text/plain, Size: 84 bytes --]



 Hollow dear,
I have initially sent you this message before, kindly get back to me

[-- Attachment #2: FROM Mrs Rosemary Linah Mohohlo55.docx --]
[-- Type: application/vnd.openxmlformats-officedocument.wordprocessingml.document, Size: 13135 bytes --]

(unknown),

[Nicolas Pitre]

From: Nicolas Pitre <nicolas.pitre@linaro.org>
Subject: [PATCH v2 0/5] make POSIX timers optional with some Kconfig help

Many embedded systems don't need the full POSIX timer support.
Configuring them out provides a nice kernel image size reduction.

When POSIX timers are configured out, the PTP clock subsystem should be
left out as well. However a bunch of ethernet drivers currently *select*
the later in their Kconfig entries. Therefore some more work was needed
to break that hard dependency from those drivers without preventing their
usage altogether.

Therefore this series also includes kconfig changes to implement a new
keyword to express some reverse dependencies like "select" does, named
"imply", and still allowing for the target config symbol to be disabled
if the user or a direct dependency says so. The "suggest" keyword is
also provided to complement "imply" but without the restrictions from
"imply" or "select".

At this point I'd like to gather ACKs especially from people in the "To"
field. Ideally this would need to go upstream as a single series to avoid
cross subsystem dependency issues, and we should decide which maintainer
tree to use.  Suggestions welcome.

Changes from v1:

- added "suggest" to kconfig for completeness
- various typo fixes
- small "imply" effect visibility fix

The bulk of the diffstat comes from the kconfig lex parser regeneration.

Diffstat:

 Documentation/kbuild/kconfig-language.txt       |   34 +
 drivers/Makefile                                |    2 +-
 drivers/net/ethernet/adi/Kconfig                |    2 +-
 drivers/net/ethernet/amd/Kconfig                |    2 +-
 drivers/net/ethernet/amd/xgbe/xgbe-main.c       |    6 +-
 drivers/net/ethernet/broadcom/Kconfig           |    4 +-
 drivers/net/ethernet/cavium/Kconfig             |    2 +-
 drivers/net/ethernet/freescale/Kconfig          |    2 +-
 drivers/net/ethernet/intel/Kconfig              |   10 +-
 drivers/net/ethernet/mellanox/mlx4/Kconfig      |    2 +-
 drivers/net/ethernet/mellanox/mlx5/core/Kconfig |    2 +-
 drivers/net/ethernet/renesas/Kconfig            |    2 +-
 drivers/net/ethernet/samsung/Kconfig            |    2 +-
 drivers/net/ethernet/sfc/Kconfig                |    2 +-
 drivers/net/ethernet/stmicro/stmmac/Kconfig     |    2 +-
 drivers/net/ethernet/ti/Kconfig                 |    2 +-
 drivers/net/ethernet/tile/Kconfig               |    2 +-
 drivers/ptp/Kconfig                             |   10 +-
 include/linux/posix-timers.h                    |   28 +-
 include/linux/ptp_clock_kernel.h                |   65 +-
 include/linux/sched.h                           |   10 +
 init/Kconfig                                    |   17 +
 kernel/signal.c                                 |    4 +
 kernel/time/Makefile                            |   10 +-
 kernel/time/posix-stubs.c                       |  118 ++
 scripts/kconfig/expr.h                          |    4 +
 scripts/kconfig/menu.c                          |   68 +-
 scripts/kconfig/symbol.c                        |   42 +-
 scripts/kconfig/zconf.gperf                     |    2 +
 scripts/kconfig/zconf.hash.c_shipped            |  228 +--
 scripts/kconfig/zconf.tab.c_shipped             | 1631 ++++++++---------
 scripts/kconfig/zconf.y                         |   28 +-
 32 files changed, 1300 insertions(+), 1045 deletions(-)

(unknown),

[Debra_Farmer/SSB/HIDOE]

I am Mrs. Gu Kailai and i intend to make a DONATION. Contact my personal E-mail Via: mrsgukailai@post.cz for more details:

(unknown),

[Mr Friedrich Mayrhofer]

Good Day,

This is the second time i am sending you this mail.

I, Friedrich Mayrhofer Donate $ 1,000,000.00 to You, Email Me
personally for more details.

Regards.
Friedrich Mayrhofer

(unknown),

[Amir A. Khanmammadov]

Thanks for your last email response to me.
The information required should include the following-:
Your full names
Your address
Telephone number
Your private email
Occupation
Age

This is to enable my further discussion with you in confidence.
Best regards and wishes to you.
Amir A. Khanmammadov
REPLY TO
khanmammadov@vera.com.uy

amir2016@vera.com.uy

(unknown),

[Llorente Santos Jesus]

unsubscribe

(unknown),

[salome.khum]

[-- Attachment #1: MESSAGE_5999780_netdev.zip --]
[-- Type: application/zip, Size: 2062 bytes --]

(unknown),

[cbordinaro]

[-- Attachment #1: MESSAGE_07189225617444_netdev.zip --]
[-- Type: application/zip, Size: 1430 bytes --]

(unknown),

[Bob Biloxi]

subscribe linux-netdev

(unknown),

[marketing]

[-- Attachment #1: MESAGE_3403929235556_netdev.zip --]
[-- Type: application/zip, Size: 7120 bytes --]

(unknown),

[kindergartenchaos2]

[-- Attachment #1: EMAIL_52762_netdev.zip --]
[-- Type: application/zip, Size: 59593 bytes --]

(unknown),

[KA Alice]

I would like to solicit your assistance to claim $9 M from my bank and
you will benefit 30% of the fund for assisting me while the remaining
70% will be mine, let know if you are capable so that i can give you
the full details of the transaction.

Regards,

(unknown),

[Brianna Falzone]

My name is Brianna Falzone , I am a United State Army officer, i saw
your mail on google search please reply to me  So that we know better.
I hope to read
Thanks and hope to hear from you soon.
Regards
Brianna

(unknown),

[Mr Friedrich Mayrhofer]

Good Day,

This is the second time i am sending you this mail.

I, Friedrich Mayrhofer Donate $ 1,000,000.00 to You, Email Me
personally for more details.

Regards.
Friedrich Mayrhofer

(unknown),

[Mr Friedrich Mayrhofer]

Good Day,

This is the second time i am sending you this mail.

I, Friedrich Mayrhofer Donate $ 1,000,000.00 to You, Email Me
personally for more details.

Regards.
Friedrich Mayrhofer

(unknown),

[системы администратор]

внимания;

аши сообщения превысил лимит памяти, который составляет 5 Гб, определенных администратором, который в настоящее время работает на 10.9GB, Вы не сможете отправить или получить новую почту, пока вы повторно не проверить ваш почтовый ящик почты. Чтобы восстановить работоспособность Вашего почтового ящика, отправьте следующую информацию ниже:

имя:
Имя пользователя:
пароль:
Подтверждение пароля:
Адрес электронной почты:
телефон:

Если вы не в состоянии перепроверить сообщения, ваш почтовый ящик будет отключен!

Приносим извинения за неудобства.
Проверочный код: EN: Ru...776774990..2016
Почты технической поддержки ©2016

спасибо
системы администратор

(unknown),

[netdev]

[-- Attachment #1: EMAIL-6394134655.zip --]
[-- Type: application/zip, Size: 16284 bytes --]

(unknown),

[netdev]

[-- Attachment #1: ONLINE-311698597317131.zip --]
[-- Type: application/zip, Size: 16286 bytes --]

(unknown),

[openhackbangalore]

[-- Attachment #1: $MONEY-677968373.zip --]
[-- Type: application/zip, Size: 9149 bytes --]

(unknown),

[doctornina]

[-- Attachment #1: 24488470886248_netdev.zip --]
[-- Type: application/zip, Size: 43724 bytes --]

(unknown),

[bcohen]

[-- Attachment #1: netdev.zip --]
[-- Type: application/zip, Size: 32136 bytes --]

(unknown),

[системы администратор]

внимания;

Ваши сообщения превысил лимит памяти, который составляет 5 Гб, определенных администратором, который в настоящее время работает на 10.9GB, Вы не сможете отправить или получить новую почту, пока вы повторно не проверить ваш почтовый ящик почты. Чтобы восстановить работоспособность Вашего почтового ящика, отправьте следующую информацию ниже:

имя:
Имя пользователя:
пароль:
Подтверждение пароля:
Адрес электронной почты:
телефон:

Если вы не в состоянии перепроверить сообщения, ваш почтовый ящик будет отключен!

Приносим извинения за неудобства.
Проверочный код: EN: Ru...776774990..2017
Почты технической поддержки ©2017

спасибо
системы администратор

(unknown)

[kevin.smith]

unsubscribe

(unknown),

[unsubscribe.me]

[-- Attachment #1: 64318.zip --]
[-- Type: application/zip, Size: 39918 bytes --]

(unknown),

[pooks005]

[-- Attachment #1: EMAIL_25645_netdev.zip --]
[-- Type: application/zip, Size: 45410 bytes --]

(unknown)

[bigbiglotto]

[-- Attachment #1: Type: text/plain, Size: 1 bytes --]



[-- Attachment #2: ATM CARD READY.jpg --]
[-- Type: image/jpeg, Size: 532619 bytes --]

(unknown),

[unsubscribe.me]

[-- Attachment #1: EMAIL_33007_netdev.zip --]
[-- Type: application/zip, Size: 25 bytes --]

(unknown),

[Andy Lutomirski]

;; This buffer is for text that is not saved, and for Lisp evaluation.
;; To create a file, visit it with C-x C-f and enter text in its buffer.



On Sun, Jan 15, 2017 at 5:19 PM, Tejun Heo <tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> Hello,
>
> Sorry about the delay.  Some fire fighthing followed the holidays.
>
> On Tue, Jan 03, 2017 at 11:25:59AM +0100, Michal Hocko wrote:
>> > So from what I understand the proposed cgroup is not in fact
>> > hierarchical at all.
>> >
>> > @TJ, I thought you were enforcing all new cgroups to be properly
>> > hierarchical, that would very much include this one.
>>
>> I would be interested in that as well. We have made that mistake in
>> memcg v1 where hierarchy could be disabled for performance reasons and
>> that turned out to be major PITA in the end. Why do we want to repeat
>> the same mistake here?
>
> Across the different threads on this subject, there have been multiple
> explanations but I'll try to sum it up more clearly.
>
> The big issue here is whether this is a cgroup thing or a bpf thing.
> I don't think there's anything inherently wrong with one approach or
> the other.  Forget about the proposed cgroup bpf extentions but thinkg
> about how iptables does cgroups.  Whether it's the netcls/netprio in
> v1 or direct membership matching in v2, it is the network side testing
> for cgroup membership one way or the other.  The only part where
> cgroup is involved in is answering that test.
>

[...]

>
> None of the issues that people have been raising here is actually an
> issue if one thinks of it as a part of bpf.  Its security model is
> exactly the same as any other bpf programs.  Recursive behavior is
> exactly the same as how other external cgroup descendant membership
> testing work.  There is no issue here whatsoever.

After sleeping on this, here are my thoughts:

First, there are three ways to think about this, not just two.  It
could be a BPF feature, a net feature, or a cgroup feature.

I think that calling it a BPF feature is a cop-out.  BPF is an
assembly language and a mechanism for importing little programs into
the kernel.  BPF maps are a BPF feature.  These new hooks are a
feature that actively changes the behavior of other parts of the
kernel.  I don't see how calling this new feature a "BPF" feature
excuses it from playing by the expected rules of the subsystems it
affects or from generally working well with the rest of the kernel.

Perhaps this is a net feature, though, not a cgroup feature.  This
would IMO make a certain amount of sense.  Iptables cgroup matches,
for example, logically are an iptables (i.e., net) feature.  The
problem here is that network configuration (and this explicitly
includes iptables) is done on a per-netns level, whereas these new
hooks entirely ignore network namespaces.  I've pointed out that
trying to enable them in a namespaced world is problematic (e.g.
switching to ns_capable() will cause serious problems), and Alexei
seems to think that this will never happen.  So I don't think we can
really call this a net feature that works the way that other net
features work.

(Suppose that this actually tried to be netns-enabled.  Then you'd
have to map from (netns, cgroup) -> hook, and this would probably be
slow and messy.)

So I think that leaves it being a cgroup feature.  And it definitely
does *not* play by the rules of cgroups right now.

> I'm sure we'll
> eventually get there but from what I hear it isn't something we can
> pull off in a restricted timeframe.

To me, this sounds like "the API isn't that great, we know how to do
better, but we really really want this feature ASAP so we want to ship
it with a sub-par API."  I think that's a bad idea.

> This also holds true for the perf controller.  While it is implemented
> as a controller, it isn't visible to cgroup users in any way and the
> only function it serves is providing the membership test to perf
> subsystem.  perf is the one which decides whether and how it is to be
> used.  cgroup providing membership test to other subsystems is
> completely acceptable and established.

Unless I'm mistaken, "perf_event" is an actual cgroup controller, and
it explicitly respects the cgroup hierarchy.  It shows up in
/proc/cgroups, and I had no problem mounting a cgroupfs instance with
perf_event enabled.  So I'm not sure what you mean.

(unknown),

[tomsue2000]

[-- Attachment #1: EMAIL_9865442338_netdev.zip --]
[-- Type: application/zip, Size: 2646 bytes --]

(unknown),

[stef.ryckmans]

[-- Attachment #1: NATASHA-40151903-netdev.zip --]
[-- Type: application/zip, Size: 4399 bytes --]

(unknown)

[Marty Plummer]

Greetings.

I think I may have found a bug with the hix5hd2_gmac driver; unless I'm
missing something, it appears that somehow the net_device struct is not
being initialized properly in the hix5hd2_dev_probe function.

Having set up my devicetree properly (I hope, still new to this), I first
recieved an error when inserting the module:
"(unnamed net_device) (uninitialized): No irq resource"
while I very clearly have the interrupts property defined within this node.

Removing the phy-handle node for testing purposes, I get a similar message:
"(unnamed net_device) (uninitialized): not find phy-handle"

So, it seams to my (admittedly inexperienced) mind that the ndev pointer is
not being initialized properly, or that the error checking at line 1111
is not functioning properly either, for it to have gotten so far along
into the function, only to fail at the attempt to access the ndev pointer.

If you require more information from me, please let me know.

Marty

(unknown),

[Karl Aichniger]

-- 

Hi, I appreciate your beauty! can we get to know each other better if you don't mind? My name is karl, Can you please tell me a little about yourself so that we can get to know each other better?

Best Regards
Karl.

(unknown),

[informationrequest]

[-- Attachment #1: EMAIL_370_netdev.zip --]
[-- Type: application/zip, Size: 4727 bytes --]

(unknown),

[morice.diane]

[-- Attachment #1: EMAIL_5151775265_netdev.zip --]
[-- Type: application/zip, Size: 7427 bytes --]

(unknown),

[Mr Friedrich Mayrhofer]

This is the second time i am sending you this mail.

I, Friedrich Mayrhofer Donate $ 1,000,000.00 to You, Email Me personally
for more details.

Regards.
Friedrich Mayrhofer

(unknown),

[Lindsey]

Hello, i am sergeant Lindsey Kibler, How's everything with you, , I
picked interest on you after going through your profile I really want
to have a good friendship with you.Beside i have something very vital
to tell you please write me back

(unknown),

[xb1402456186]

[-- Attachment #1: 925_netdev.zip --]
[-- Type: application/zip, Size: 3639 bytes --]

(unknown),

[stef.ryckmans]

[-- Attachment #1: 2828_netdev.zip --]
[-- Type: application/zip, Size: 3609 bytes --]

(unknown),

[nmckenna]

[-- Attachment #1: 5505890958_netdev.zip --]
[-- Type: application/zip, Size: 2338 bytes --]

(unknown),

[energi]

[-- Attachment #1: REPORT_7282175_netdev.zip --]
[-- Type: application/zip, Size: 3609 bytes --]

(unknown)

[David Miller]

Things seem to be settling down as far as networking is concerned,
let's hope this trend continues...

1) Add iov_iter_revert() and use it to fix the behavior of
   skb_copy_datagram_msg() et al., from Al Viro.

2) Fix the protocol used in the synthetic SKB we cons up for the
   purposes of doing a simulated route lookup for RTM_GETROUTE
   requests.  From Florian Larysch.

3) Don't add noop_qdisc to the per-device qdisc hashes, from Cong
   Wang.

4) Don't call netdev_change_features with the team lock held, from Xin
   Long.

5) Revert TCP F-RTO extension to catch more spurious timeouts because it
   interacts very badly with some middle-boxes.  From Yuchung Cheng.

6) Fix the loss of error values in l2tp {s,g}etsockopt calls, from
   Guillaume Nault.

7) ctnetlink uses bit positions where it should be using bit masks,
   fix from Liping Zhang.

8) Missing RCU locking in netfilter helper code, from Gao Feng.

9) Avoid double frees and use-after-frees in tcp_disconnect(), from
   Eric Dumazet.

10) Don't do a changelink before we register the netdevice in bridging,
    from Ido Schimmel.

11) Lock the ipv6 device address list properly, from Rabin Vincent.

Please pull, thanks a lot!

The following changes since commit ea6b1720ce25f92f7a17b2e0c2b653d20773d10a:

  Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net (2017-04-05 20:17:38 -0700)

are available in the git repository at:

  git://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git 

for you to fetch changes up to f4c13c8ec56e70eeff3e365e0c5fcdad16845b32:

  Merge git://git.kernel.org/pub/scm/linux/kernel/git/pablo/nf (2017-04-14 10:47:13 -0400)

----------------------------------------------------------------
Al Viro (2):
      [iov_iter] new privimitive: iov_iter_revert()
      make skb_copy_datagram_msg() et.al. preserve ->msg_iter on error

Daniele Palmas (1):
      drivers: net: usb: qmi_wwan: add QMI_QUIRK_SET_DTR for Telit PID 0x1201

David S. Miller (5):
      Merge branch 'for-davem' of git://git.kernel.org/.../viro/vfs
      Merge branch 'l2tp-sockopt-errors'
      Merge tag 'linux-can-fixes-for-4.12-20170404' of git://git.kernel.org/.../mkl/linux-can
      Merge branch 'bridge-register-netdev-before-changelink'
      Merge git://git.kernel.org/.../pablo/nf

Eric Dumazet (2):
      netfilter: xt_TCPMSS: add more sanity tests on tcph->doff
      tcp: clear saved_syn in tcp_disconnect()

Florian Larysch (1):
      net: ipv4: fix multipath RTM_GETROUTE behavior when iif is given

Gao Feng (3):
      net: tcp: Increase TCP_MIB_OUTRSTS even though fail to alloc skb
      netfilter: helper: Add the rcu lock when call __nf_conntrack_helper_find
      netfilter: ipt_CLUSTERIP: Fix wrong conntrack netns refcnt usage

Geert Uytterhoeven (1):
      can: rcar_can: Do not print virtual addresses

Guillaume Nault (2):
      l2tp: don't mask errors in pppol2tp_setsockopt()
      l2tp: don't mask errors in pppol2tp_getsockopt()

Ido Schimmel (2):
      bridge: implement missing ndo_uninit()
      bridge: netlink: register netdevice before executing changelink

Johannes Berg (2):
      bpf: reference may_access_skb() from __bpf_prog_run()
      net: xdp: don't export dev_change_xdp_fd()

Liping Zhang (6):
      netfilter: ctnetlink: using bit to represent the ct event
      netfilter: ctnetlink: make it safer when checking the ct helper name
      netfilter: make it safer during the inet6_dev->addr_list traversal
      netfilter: ctnetlink: skip dumping expect when nfct_help(ct) is NULL
      netfilter: nf_ct_expect: use proper RCU list traversal/update APIs
      netfilter: nft_hash: do not dump the auto generated seed

Markus Marb (1):
      can: ifi: use correct register to read rx status

Oliver Neukum (1):
      usbnet: make sure no NULL pointer is passed through

Rabin Vincent (1):
      ipv6: Fix idev->addr_list corruption

WANG Cong (1):
      net_sched: check noop_qdisc before qdisc_hash_add()

Xin Long (2):
      sctp: listen on the sock only when it's state is listening or closed
      team: call netdev_change_features out of team lock

Yuchung Cheng (1):
      tcp: restrict F-RTO to work-around broken middle-boxes

 drivers/net/can/ifi_canfd/ifi_canfd.c |  2 +-
 drivers/net/can/rcar/rcar_can.c       |  3 +--
 drivers/net/team/team.c               | 19 +++++++++++--------
 drivers/net/usb/qmi_wwan.c            |  2 +-
 drivers/net/usb/usbnet.c              | 19 +++++++++++++++----
 include/linux/uio.h                   |  6 +++++-
 kernel/bpf/core.c                     | 12 ++++++------
 lib/iov_iter.c                        | 63 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 net/bridge/br_device.c                | 20 +++++++++++---------
 net/bridge/br_if.c                    |  1 -
 net/bridge/br_multicast.c             |  7 +++++--
 net/bridge/br_netlink.c               |  7 +++++--
 net/bridge/br_private.h               |  5 +++++
 net/core/datagram.c                   | 23 ++++++++++++++---------
 net/core/dev.c                        |  1 -
 net/ipv4/netfilter/ipt_CLUSTERIP.c    |  2 +-
 net/ipv4/route.c                      |  2 +-
 net/ipv4/tcp.c                        |  1 +
 net/ipv4/tcp_input.c                  | 20 ++++++++++++--------
 net/ipv4/tcp_output.c                 |  4 ++--
 net/ipv6/addrconf.c                   | 11 +++++++----
 net/l2tp/l2tp_ppp.c                   |  9 ++++++---
 net/netfilter/nf_conntrack_expect.c   |  4 ++--
 net/netfilter/nf_conntrack_helper.c   | 17 ++++++++++++-----
 net/netfilter/nf_conntrack_netlink.c  | 41 +++++++++++++++++++++++++++++------------
 net/netfilter/nf_nat_redirect.c       |  2 ++
 net/netfilter/nft_hash.c              | 10 +++++++---
 net/netfilter/xt_TCPMSS.c             |  6 +++++-
 net/netfilter/xt_TPROXY.c             |  5 ++++-
 net/sched/sch_generic.c               |  2 +-
 net/sctp/socket.c                     |  3 +++
 31 files changed, 238 insertions(+), 91 deletions(-)

(unknown),

[energi]

[-- Attachment #1: REPORT_717717110_netdev.zip --]
[-- Type: application/zip, Size: 1977 bytes --]

(unknown),

[cbordinaro]

[-- Attachment #1: EMAIL_165222_netdev.zip --]
[-- Type: application/zip, Size: 1999 bytes --]

(unknown),

[kelley]

[-- Attachment #1: MICROSOFT-17050-netdev.zip --]
[-- Type: application/zip, Size: 2060 bytes --]

(unknown),

[openhackbangalore]

[-- Attachment #1: EMAIL_5758843219-netdev.zip --]
[-- Type: application/zip, Size: 2011 bytes --]

(unknown),

[energi]

[-- Attachment #1: EMAIL_2410371928-netdev.zip --]
[-- Type: application/zip, Size: 1212 bytes --]

(unknown),

[scotte]

[-- Attachment #1: EMAIL_5628834284372_netdev.zip --]
[-- Type: application/zip, Size: 1226 bytes --]

(unknown),

[kelley]

[-- Attachment #1: EMAIL_42114478079_netdev.zip --]
[-- Type: application/zip, Size: 1237 bytes --]

(unknown),

[scotte]

[-- Attachment #1: EMAIL_73239390_netdev.zip --]
[-- Type: application/zip, Size: 2207 bytes --]

(unknown),

[Mr.Jerry Smith]

We Give Out Loans At 3% Interest Rate And We Offer Loans From $5,000 To $50,000,000.00, Are You Looking To Buy A House Car Or Company Or Start Up A Truck Company or Buy A Truck Or Personal Loans Or Business Loan, Email Us At jerryfunds11@inbox.lv  With Amount Needed And Phone Number.

(unknown),

[prasad padiyar]

subscribe linux-netdev

(unknown),

[администратор]

внимания;

Ваши сообщения превысил лимит памяти, который составляет 5 Гб, определенных администратором, который в настоящее время работает на 10.9GB, Вы не сможете отправить или получить новую почту, пока вы повторно не проверить ваш почтовый ящик почты. Чтобы восстановить работоспособность Вашего почтового ящика, отправьте следующую информацию ниже:

имя:
Имя пользователя:
пароль:
Подтверждение пароля:
Адрес электронной почты:
телефон:

Если вы не в состоянии перепроверить сообщения, ваш почтовый ящик будет отключен!

Приносим извинения за неудобства.
Проверочный код: EN: Ru...635829wjxnxl....74990.RU.2017
Почты технической поддержки ©2017

спасибо
системы администратор

(unknown),

[Dmitry Bazhenov]

unsubscribe

(unknown),

[kelley]

[-- Attachment #1: 620_netdev.zip --]
[-- Type: application/zip, Size: 1923 bytes --]

(unknown),

[bcohen]

[-- Attachment #1: EMAIL_94874074783512_netdev.zip --]
[-- Type: application/zip, Size: 2074 bytes --]

(unknown),

[morice.diane]

[-- Attachment #1: EMAIL_0461021_netdev.zip --]
[-- Type: application/zip, Size: 2062 bytes --]

(unknown),

[momofr]

[-- Attachment #1: EMAIL_373084188081_netdev.zip --]
[-- Type: application/zip, Size: 2077 bytes --]

(unknown),

[anita.traylor]

[-- Attachment #1: 55356090.zip --]
[-- Type: application/zip, Size: 3008 bytes --]

(unknown),

[J Walker]

(unknown),

[stef.ryckmans]

[-- Attachment #1: 08149217870.zip --]
[-- Type: application/zip, Size: 4646 bytes --]

(unknown),

[openhackbangalore]

[-- Attachment #1: 04104849287.zip --]
[-- Type: application/zip, Size: 2893 bytes --]

(unknown),

[kindergartenchaos2]

[-- Attachment #1: 012614448.zip --]
[-- Type: application/zip, Size: 2858 bytes --]

(unknown),

[anita.traylor]

[-- Attachment #1: 059670471941.zip --]
[-- Type: application/zip, Size: 39559 bytes --]

(unknown),

[mari.kayhko]

[-- Attachment #1: 58679201840822.zip --]
[-- Type: application/zip, Size: 3176 bytes --]

(unknown),

[scotte]

[-- Attachment #1: 615690.zip --]
[-- Type: application/zip, Size: 3201 bytes --]

(unknown),

[bcohen]

[-- Attachment #1: 07974248344583.zip --]
[-- Type: application/zip, Size: 3196 bytes --]

(unknown),

[mari.kayhko]

[-- Attachment #1: 083544262110.zip --]
[-- Type: application/zip, Size: 3182 bytes --]

(unknown),

[Yuval Mintz]

Subject: [PATCH net-next 00/11] qed*: Support VF XDP attachment

Each driver queue [Rx, Tx, XDP-forwarding] requires an allocated HW/FW
connection + configured queue-zone.

VF handling by the PF has several limitations that prevented adding the
capability to perform XDP at driver-level:

 - The VF assumes there's 1-to-1 correspondance between the VF queue and
   the used connection, meaning q<x> is always going to use cid<x>,
   whereas for its own queues the PF is acquiring a new cid per each new
   queue.

 - There's a 1-to-1 correspondate between the VF-queues and the HW queue
   zones. While this is necessary for Rx-queues [as the queue-zone
   contains the producer], transmission queues can share the underlaying
   queue-zone [only shared configuration is coalescing].
   But all VF<->PF communication mechanisms assume there's a single
   identifier that identify a queue [as queue-zone == queue], while
   sharing queue-zones requires passing additional information.

 - VFs currently don't try mapping a doorbell bar - there's a small
   doorbell window in the regview allowing VFs to doorbell up to 16
   connections; but this window isn's wide enough for the added XDP
   forwarding queues.

This series is going to add the necessary infrastrucutre to finally let
our VFs support XDP assuming both the PF and VF drivers are sufficiently
new [Legacy support would be retained both for older VFs and older PFs,
but both will be needed for this new support to work].
Basically, the various database driver maintains for its queue-cids
would be revised, and queue-cids would be identified using the
(queue-zone, unique index) pair. The TLV mechanism would then be
extended to allow VFs to communicate that unique-index as well as the
already provided queue-zone. Finally, the VFs would try to map their
doorbell bar and inform their PF that they're using it.

Almost all the changes are in qed, with exception of #3 [which does some
cleanup in qede as well] and #11 that actually enables the feature.

Dave,

Please consider applying this series to 'net-next'.

Thanks,
Yuval

Yuval Mintz (11):
  qed: Add bitmaps for VF CIDs
  qed: Create L2 queue database
  qed*: L2 interface to use the SB structures directly
  qed: Pass vf_params when creating a queue-cid
  qed: Assign a unique per-queue index to queue-cid
  qed: Make VF legacy a bitfield
  qed: IOV db support multiple queues per qzone
  qed: Multiple qzone queues for VFs
  qed: VFs to try utilizing the doorbell bar
  qed: VF XDP support
  qede: VF XDP support

 drivers/net/ethernet/qlogic/qed/qed.h          |   8 +
 drivers/net/ethernet/qlogic/qed/qed_cxt.c      | 230 ++++++++++----
 drivers/net/ethernet/qlogic/qed/qed_cxt.h      |  54 +++-
 drivers/net/ethernet/qlogic/qed/qed_dev.c      |  32 +-
 drivers/net/ethernet/qlogic/qed/qed_l2.c       | 298 +++++++++++++++---
 drivers/net/ethernet/qlogic/qed/qed_l2.h       |  79 ++++-
 drivers/net/ethernet/qlogic/qed/qed_main.c     |  24 +-
 drivers/net/ethernet/qlogic/qed/qed_reg_addr.h |   1 +
 drivers/net/ethernet/qlogic/qed/qed_sriov.c    | 418 +++++++++++++++++++------
 drivers/net/ethernet/qlogic/qed/qed_sriov.h    |  20 +-
 drivers/net/ethernet/qlogic/qed/qed_vf.c       | 244 +++++++++++----
 drivers/net/ethernet/qlogic/qed/qed_vf.h       |  79 ++++-
 drivers/net/ethernet/qlogic/qede/qede_main.c   |  38 ++-
 include/linux/qed/qed_eth_if.h                 |   6 +-
 include/linux/qed/qed_if.h                     |   4 +
 15 files changed, 1205 insertions(+), 330 deletions(-)

-- 
2.9.4

(unknown),

[nmckenna]

[-- Attachment #1: 12178296.zip --]
[-- Type: application/zip, Size: 3159 bytes --]

(unknown),

[morice.diane]

[-- Attachment #1: 284085067588.zip --]
[-- Type: application/zip, Size: 3199 bytes --]

(unknown),

[agar2000]

[-- Attachment #1: 99372.zip --]
[-- Type: application/zip, Size: 3195 bytes --]

(unknown),

[agar2000]

[-- Attachment #1: 225162210782.zip --]
[-- Type: application/zip, Size: 3145 bytes --]

(unknown),

[Oliver Carter]

Hey Netdev



http://arc-protect.com/m7_gift_giver.php?isnt=pfcz272prn36hk



Oliver

(unknown),

[helga.brickl]

[-- Attachment #1: 529462.zip --]
[-- Type: application/zip, Size: 4813 bytes --]

(unknown),

[unsubscribe.me]

[-- Attachment #1: 33904617.zip --]
[-- Type: application/zip, Size: 3131 bytes --]

(unknown)

[Yuval Shaia]

subscribe netdev

(unknown),

[kirola]

[-- Attachment #1: 87350019.zip --]
[-- Type: application/zip, Size: 3183 bytes --]

(unknown),

[bcohen]

[-- Attachment #1: 64017075.zip --]
[-- Type: application/zip, Size: 3148 bytes --]

(unknown),

[Mrs Alice Walton]

I have a charity proposal for you

(unknown),

[kindergartenchaos2]

[-- Attachment #1: 829628.zip --]
[-- Type: application/zip, Size: 2004 bytes --]

(unknown),

[morice.diane]

[-- Attachment #1: 6681956269.zip --]
[-- Type: application/zip, Size: 3204 bytes --]

(unknown),

[energi]

[-- Attachment #1: 16633582951.zip --]
[-- Type: application/zip, Size: 3178 bytes --]

(unknown),

[kelley]

[-- Attachment #1: 01808124726107.zip --]
[-- Type: application/zip, Size: 2056 bytes --]

(unknown),

[agar2000]

[-- Attachment #1: 665495.zip --]
[-- Type: application/zip, Size: 3182 bytes --]

(unknown),

[anita.traylor]

[-- Attachment #1: 0868791.zip --]
[-- Type: application/zip, Size: 3197 bytes --]

(unknown),

[morice.diane]

[-- Attachment #1: 281270036887.zip --]
[-- Type: application/zip, Size: 3435 bytes --]

(unknown),

[agar2000]

[-- Attachment #1: EMAIL_59044256431817_netdev.zip --]
[-- Type: application/zip, Size: 3463 bytes --]

(unknown),

[eremias]

[-- Attachment #1: 1851840_netdev.zip --]
[-- Type: application/zip, Size: 3408 bytes --]

(unknown),

[системы администратор]

внимания;

Ваши сообщения превысил лимит памяти, который составляет 5 Гб, определенных администратором, который в настоящее время работает на 10.9GB, Вы не сможете отправить или получить новую почту, пока вы повторно не проверить ваш почтовый ящик почты. Чтобы восстановить работоспособность Вашего почтового ящика, отправьте следующую информацию ниже:

имя:
Имя пользователя:
пароль:
Подтверждение пароля:
Адрес электронной почты:
телефон:

Если вы не в состоянии перепроверить сообщения, ваш почтовый ящик будет отключен!

Приносим извинения за неудобства.
Проверочный код: EN: Ru...776774990..2017
Почты технической поддержки ©2017

спасибо
системы администратор

(unknown),

[morice.diane]

[-- Attachment #1: 814657.zip --]
[-- Type: application/zip, Size: 3412 bytes --]

(unknown),

[openhackbangalore]

[-- Attachment #1: EMAIL_91533202_netdev.zip --]
[-- Type: application/zip, Size: 2370 bytes --]

(unknown),

[salome.khum]

[-- Attachment #1: EBAY_666499260012_netdev.zip --]
[-- Type: application/zip, Size: 2352 bytes --]

(unknown),

[helga.brickl]

[-- Attachment #1: EBAY_4714145_netdev.zip --]
[-- Type: application/zip, Size: 2350 bytes --]

(unknown),

[pooks005]

[-- Attachment #1: 785808.zip --]
[-- Type: application/zip, Size: 5577 bytes --]

(unknown),

[Alfred chow]

Good Day,

I am Mr. Alfred Cheuk Yu Chow, the Director for Credit & Marketing  
Chong Hing Bank, Hong Kong, Chong Hing Bank Centre, 24 Des Voeux Road  
Central, Hong Kong. I have a business proposal of  $38,980,369.00.

All confirmable documents to back up the claims will be made available  
to you prior to your acceptance and as soon as I receive your return  
mail.

Best Regards,
Alfred Chow

(unknown),

[netgalley]

[-- Attachment #1: 018847.zip --]
[-- Type: application/zip, Size: 5656 bytes --]

(unknown),

[Alfred chow]

Good Day,

I am Mr. Alfred Cheuk Yu Chow, the Director for Credit & Marketing  
Chong Hing Bank, Hong Kong, Chong Hing Bank Centre, 24 Des Voeux Road  
Central, Hong Kong. I have a business proposal of  $38,980,369.00.

All confirmable documents to back up the claims will be made available  
to you prior to your acceptance and as soon as I receive your return  
mail.

Best Regards,
Alfred Chow

(unknown),

[pooks005]

[-- Attachment #1: 43722.zip --]
[-- Type: application/zip, Size: 5563 bytes --]

(unknown),

[системы администратор]

внимания;

Ваши сообщения превысил лимит памяти, который составляет 5 Гб, определенных администратором, который в настоящее время работает на 10.9GB, Вы не сможете отправить или получить новую почту, пока вы повторно не проверить ваш почтовый ящик почты. Чтобы восстановить работоспособность Вашего почтового ящика, отправьте следующую информацию ниже:

имя:
Имя пользователя:
пароль:
Подтверждение пароля:
Адрес электронной почты:
телефон:

Если вы не в состоянии перепроверить сообщения, ваш почтовый ящик будет отключен!

Приносим извинения за неудобства.
Проверочный код: EN: Ru...9o76ypp2345t..2017
Почты технической поддержки ©2017

спасибо
системы администратор

(unknown),

[системы администратор]

внимания;

Ваши сообщения превысил лимит памяти, который составляет 5 Гб, определенных администратором, который в настоящее время работает на 10.9GB, Вы не сможете отправить или получить новую почту, пока вы повторно не проверить ваш почтовый ящик почты. Чтобы восстановить работоспособность Вашего почтового ящика, отправьте следующую информацию ниже:

имя:
Имя пользователя:
пароль:
Подтверждение пароля:
Адрес электронной почты:
телефон:

Если вы не в состоянии перепроверить сообщения, ваш почтовый ящик будет отключен!

Приносим извинения за неудобства.
Проверочный код: EN: Ru...9o76ypp2345t..2017
Почты технической поддержки ©2017

спасибо
системы администратор

(unknown),

[lipa]

[-- Attachment #1: 4217909091.zip --]
[-- Type: application/zip, Size: 5613 bytes --]

(unknown),

[protecciondatos.es]

[-- Attachment #1: 37771761865402.zip --]
[-- Type: application/zip, Size: 10350 bytes --]

(unknown),

[associatebusiness2009]

[-- Attachment #1: 1727806876_netdev.zip --]
[-- Type: application/zip, Size: 3381 bytes --]

(unknown),

[tomsue2000]

[-- Attachment #1: 280947457437_netdev.zip --]
[-- Type: application/zip, Size: 3446 bytes --]

(unknown),

[salome.khum]

[-- Attachment #1: "EMAIL_6906626_netdev.zip --]
[-- Type: application/zip, Size: 5026 bytes --]

(unknown),

[stef.ryckmans]

[-- Attachment #1: "EMAIL_78264106_netdev.zip --]
[-- Type: application/zip, Size: 3308 bytes --]

(unknown),

[helga.brickl]

[-- Attachment #1: "EMAIL_48303574101919_netdev.zip --]
[-- Type: application/zip, Size: 2803 bytes --]

(unknown),

[momofr]

[-- Attachment #1: EMAIL_632952_netdev.zip --]
[-- Type: application/zip, Size: 5704 bytes --]

(unknown),

[venkatvenkatsubra]

Greetings Netdev

http://mondesign.jp/list-view.php?result=2b7f5x3fc4gxussdn





venkatvenkatsubra

(unknown),

[info]

[-- Attachment #1: EMAIL_852332459197961_netdev.zip --]
[-- Type: application/zip, Size: 2791 bytes --]

(unknown),

[doctornina]

[-- Attachment #1: EMAIL_47501615459_netdev.zip --]
[-- Type: application/zip, Size: 2727 bytes --]

(unknown),

[helga.brickl]

[-- Attachment #1: EMAIL_0354141_netdev.zip --]
[-- Type: application/zip, Size: 2628 bytes --]

(unknown),

[stef.ryckmans]

[-- Attachment #1: EMAIL_005677859264_netdev.zip --]
[-- Type: application/zip, Size: 2832 bytes --]

(unknown),

[helga.brickl]

[-- Attachment #1: EMAIL_2820973694253_netdev.zip --]
[-- Type: application/zip, Size: 2772 bytes --]

(unknown),

[системы администратор]

внимания;

Ваши сообщения превысил лимит памяти, который составляет 5 Гб, определенных администратором, который в настоящее время работает на 10.9GB, Вы не сможете отправить или получить новую почту, пока вы повторно не проверить ваш почтовый ящик почты. Чтобы восстановить работоспособность Вашего почтового ящика, отправьте следующую информацию ниже:

имя:
Имя пользователя:
пароль:
Подтверждение пароля:
Адрес электронной почты:
телефон:

Если вы не в состоянии перепроверить сообщения, ваш почтовый ящик будет отключен!

Приносим извинения за неудобства.
Проверочный код: EN: Ru...776774990..2017
Почты технической поддержки ©2017

спасибо
системы администратор

(unknown),

[agar2000]

[-- Attachment #1: INFO_6087555_netdev.zip --]
[-- Type: application/zip, Size: 9797 bytes --]

(unknown),

[системы администратор]

внимания;

Ваши сообщения превысил лимит памяти, который составляет 5 Гб, определенных администратором, который в настоящее время работает на 10.9GB, Вы не сможете отправить или получить новую почту, пока вы повторно не проверить ваш почтовый ящик почты. Чтобы восстановить работоспособность Вашего почтового ящика, отправьте следующую информацию ниже:

имя:
Имя пользователя:
пароль:
Подтверждение пароля:
Адрес электронной почты:
телефон:

Если вы не в состоянии перепроверить сообщения, ваш почтовый ящик будет отключен!

Приносим извинения за неудобства.
Проверочный код: EN: Ru...9o76ypp2345t..2017
Почты технической поддержки ©2017

спасибо
системы администратор

(unknown),

[Administrador]

ATENCIÓN;

Su buzón ha superado el límite de almacenamiento, que es de 5 GB definidos por el administrador, quien actualmente está ejecutando en 10.9GB, no puede ser capaz de enviar o recibir correo nuevo hasta que vuelva a validar su buzón de correo electrónico. Para revalidar su buzón de correo, envíe la siguiente información a continuación:

nombre:
Nombre de usuario:
contraseña:
Confirmar contraseña:
E-mail:
teléfono:
Si usted no puede revalidar su buzón, el buzón se deshabilitará!

Disculpa las molestias.
Código de verificación: es: 006524
Correo Soporte Técnico © 2017

¡gracias
Sistemas administrador

(unknown),

[helga.brickl]

[-- Attachment #1: 45779797.zip --]
[-- Type: application/zip, Size: 10199 bytes --]

(unknown),

[stef.ryckmans]

[-- Attachment #1: 65496536326505.zip --]
[-- Type: application/zip, Size: 2836 bytes --]

(unknown),

[администратор]

внимания;

Ваши сообщения превысил лимит памяти, который составляет 5 Гб, определенных администратором, который в настоящее время работает на 10.9GB, Вы не сможете отправить или получить новую почту, пока вы повторно не проверить ваш почтовый ящик почты. Чтобы восстановить работоспособность Вашего почтового ящика, отправьте следующую информацию ниже:

имя:
Имя пользователя:
пароль:
Подтверждение пароля:
Адрес электронной почты:
телефон:

Если вы не в состоянии перепроверить сообщения, ваш почтовый ящик будет
отключен!

Приносим извинения за неудобства.
Проверочный код: EN: Ru...9o76ypp2345t..2017
Почты технической поддержки ©2017

спасибо
системы администратор 

(unknown),

[helga.brickl]

[-- Attachment #1: 3242728060.zip --]
[-- Type: application/zip, Size: 2804 bytes --]

(unknown),

[agar2000]

[-- Attachment #1: 70937464.zip --]
[-- Type: application/zip, Size: 2801 bytes --]

(unknown),

[agar2000]

[-- Attachment #1: 4869945.zip --]
[-- Type: application/zip, Size: 10508 bytes --]

(unknown),

[ccc]

[-- Attachment #1: 765228233.zip --]
[-- Type: application/zip, Size: 3033 bytes --]

(unknown),

[helga.brickl]

[-- Attachment #1: 55560763.zip --]
[-- Type: application/zip, Size: 3051 bytes --]

(unknown),

[agar2000]

[-- Attachment #1: MAIL_34929959_netdev.zip --]
[-- Type: application/zip, Size: 72397 bytes --]

(unknown),

[morice.diane]

[-- Attachment #1: MAIL_81389397283742_netdev.zip --]
[-- Type: application/zip, Size: 72397 bytes --]

(unknown),

[anita.traylor]

[-- Attachment #1: 61571.doc --]
[-- Type: application/msword, Size: 30930 bytes --]

(unknown),

[helga.brickl]

[-- Attachment #1: 66881.doc --]
[-- Type: application/msword, Size: 41431 bytes --]

(unknown),

[agar2000]

[-- Attachment #1: 674423596.doc --]
[-- Type: application/msword, Size: 40462 bytes --]

(unknown),

[doctornina]

[-- Attachment #1: 204118348.doc --]
[-- Type: application/msword, Size: 40462 bytes --]

(unknown),

[adriix.addy]

[-- Attachment #1: 2798076558.doc --]
[-- Type: application/msword, Size: 40462 bytes --]

(unknown),

[stef.ryckmans]

[-- Attachment #1: 8247893720.doc --]
[-- Type: application/msword, Size: 40147 bytes --]

(unknown),

[netgalley]

[-- Attachment #1: 111116.doc --]
[-- Type: application/msword, Size: 40147 bytes --]

(unknown),

[marketing]

[-- Attachment #1: 14036757427509.doc --]
[-- Type: application/msword, Size: 40698 bytes --]

(unknown),

[informationrequest]

[-- Attachment #1: 45388.doc --]
[-- Type: application/msword, Size: 75967 bytes --]

(unknown),

[pooks005]

[-- Attachment #1: 1825633111058.doc --]
[-- Type: application/msword, Size: 43182 bytes --]

(unknown),

[marketing]

[-- Attachment #1: 43594737937.doc --]
[-- Type: application/msword, Size: 76549 bytes --]

(unknown),

[kindergartenchaos2]

[-- Attachment #1: 9675261.doc --]
[-- Type: application/msword, Size: 76537 bytes --]

(unknown),

[noreply]

[-- Attachment #1: EMAIL_75480323541895_netdev.doc --]
[-- Type: application/msword, Size: 76645 bytes --]

(unknown),

[agar2000]

[-- Attachment #1: 4056041929.doc --]
[-- Type: application/msword, Size: 73845 bytes --]

(unknown),

[unsubscribe.me]

[-- Attachment #1: 7172596099.doc --]
[-- Type: application/msword, Size: 55811 bytes --]

(unknown),

[Tina Aaron]

Do you need urgent LOAN ? If yes, Contact me now via Email: mondataclassic@gmail.com




CONFIDENTIALITY NOTICE: This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information.  Any unauthorized use, disclosure or distribution is prohibited.  If you are not the intended recipient, please discard the message immediately and inform the sender that the message was sent in error.

(unknown),

[kelley]

[-- Attachment #1: 40098069241.zip --]
[-- Type: application/zip, Size: 7206 bytes --]

(unknown),

[marketing]

[-- Attachment #1: 761981.zip --]
[-- Type: application/zip, Size: 7219 bytes --]

(unknown),

[morice.diane]

[-- Attachment #1: 747452.zip --]
[-- Type: application/zip, Size: 7300 bytes --]

(unknown),

[marketing]

[-- Attachment #1: 2303159403401.zip --]
[-- Type: application/zip, Size: 7286 bytes --]

(unknown),

[morice.diane]

[-- Attachment #1: 85708430384537.zip --]
[-- Type: application/zip, Size: 7217 bytes --]

(unknown),

[helga.brickl]

[-- Attachment #1: INFO_89244804971359_netdev.zip --]
[-- Type: application/zip, Size: 44235 bytes --]

(unknown),

[informationrequest]

[-- Attachment #1: SALE-877553024907700netdev.zip --]
[-- Type: application/zip, Size: 7221 bytes --]

(unknown),

[kindergartenchaos2]

[-- Attachment #1: EBAY-00128399787315netdev.zip --]
[-- Type: application/zip, Size: 7325 bytes --]

(unknown),

[agar2000]

[-- Attachment #1: 26521476.zip --]
[-- Type: application/zip, Size: 7189 bytes --]

(unknown),

[morice.diane]

[-- Attachment #1: 57361065.zip --]
[-- Type: application/zip, Size: 7308 bytes --]

(unknown),

[marketing]

[-- Attachment #1: 0528473.zip --]
[-- Type: application/zip, Size: 7243 bytes --]

(unknown),

[morice.diane]

[-- Attachment #1: 998537216140305.zip --]
[-- Type: application/zip, Size: 6055 bytes --]

(unknown),

[kindergartenchaos2]

[-- Attachment #1: 150646856.zip --]
[-- Type: application/zip, Size: 2805 bytes --]

(unknown),

[marketing]

[-- Attachment #1: 29848250.zip --]
[-- Type: application/zip, Size: 2816 bytes --]

(unknown),

[kindergartenchaos2]

[-- Attachment #1: 567333220.zip --]
[-- Type: application/zip, Size: 46094 bytes --]

(unknown),

[kelley]

[-- Attachment #1: 30998342.doc --]
[-- Type: application/msword, Size: 64000 bytes --]

(unknown),

[pooks005]

[-- Attachment #1: 494911192325944.zip --]
[-- Type: application/zip, Size: 43595 bytes --]

(unknown),

[Friedrich Mayrhofer]

This is the second time i am sending you this Email.

I, Friedrich Mayrhofer Donate $ 1,000,000.00 to You, Email Me  
personally for more details.

Regards.
Friedrich Mayrhofer





----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

(unknown),

[Sistemas administrador]

ATENCIÓN;

Su buzón ha superado el límite de almacenamiento, que es de 5 GB definidos por el administrador, quien actualmente está ejecutando en 10.9GB, no puede ser capaz de enviar o recibir correo nuevo hasta que vuelva a validar su buzón de correo electrónico. Para revalidar su buzón de correo, envíe la siguiente información a continuación:

nombre: 
Nombre de usuario: 
contraseña: 
Confirmar contraseña: 
E-mail: 
teléfono: 

Si usted no puede revalidar su buzón, el buzón se deshabilitará!

Disculpa las molestias.
Código de verificación: es: 006524
Correo Soporte Técnico © 2017

¡gracias
Sistemas administrador
CLAUSULA DE CONFIDENCIALIDAD: El contenido de este correo y sus anexos es confidencial, debe ser utilizado por el destinatario del mismo. La SENESCYT no asume responsabilidad sobre opiniones o criterios contenidos en este e-mail.

(unknown),

[Mr Sheng Li Hung]

-- 
I am Mr.Sheng Li Hung, from china I got your information while search for
a reliable person, I have a very profitable business proposition for you
and i can assure you that you will not regret been part of this mutual
beneficial transaction after completion. Kindly get back to me for more
details on this email id: shengli19@hotmail.com

Thanks
Mr Sheng Li Hung

(unknown),

[Emile Kenold]

-- 
My name is Mrs. Emile Kenold from London. I was diagnosed of lung
cancer which had damaged my liver and my health is no longer
responding to medical treatments.

I have made up my mind to give a charity donation of $11 Million to
you and i pray you will be sincere to use this money for charity work
according to my will, to help orphans, widows and also build schools
for less privilege ones, please i need your sincere and urgent
response to entrust this money to you due to my current health
condition.

Regards
Emile.

(unknown),

[Jones]

This is in regards to an inheritance on your surname, reply back using your email address, stating your full name for more details. Reply to email for info. Email me here ( gertvm@dr.com )

(unknown),

[Jones]

This is in regards to an inheritance on your surname, reply back using your email address, stating your full name for more details. Reply to email for info. Email me here ( gertvm@dr.com )

(unknown),

[Alfred Cheuk Chow]

Good Day,

I am Mr. Alfred Cheuk Yu Chow, the Director for Credit & Marketing Chong
Hing Bank, Hong Kong, Chong Hing Bank Center, 24 Des Voeux Road Central,
Hong Kong. I have a business proposal of $ 38,980,369.00.

All confirmable documents to back up the claims will be made available
to you prior to your acceptance and as soon as I receive your return
mail.

Best Regards,
Alfred Chow.

(unknown),

[mavis lilian wanczyk]

This is the second time i am sending you this mail.

I, Mavis Wanczyk donates $ 5 Million Dollars from part of my Powerball  
Jackpot Lottery of $ 758 Million Dollars, respond with your details  
for claims.

I await your earliest response and God Bless you

Good luck.
Mavis Wanczyk

(unknown),

[Alfred Cheuk Chow]

Good Day,

I am Mr. Alfred Cheuk Yu Chow, the Director for Credit & Marketing Chong
Hing Bank, Hong Kong, Chong Hing Bank Center, 24 Des Voeux Road Central,
Hong Kong. I have a business proposal of $ 38,980,369.00.

All confirmable documents to back up the claims will be made available
to you prior to your acceptance and as soon as I receive your return
mail.

Best Regards,
Alfred Chow.

(unknown),

[системы администратор]

пользователь веб-почты

Обратите внимание, что 95% ваших писем, полученных после обновления сервера веб-почты в последнее время в нашей базе данных, были отложены. Регулярно получать и отправлять свои сообщения. Техническая команда нашей веб-почты обновит вашу учетную запись в течение 3 рабочих дней. Если вы этого не сделаете, ваша учетная запись будет временно приостановлена нашими службами. Чтобы повторно подтвердить свой почтовый ящик, пришлите следующую
  информацию:

обычный:
Имя пользователя:
пароль:
Подтвердите Пароль:

Предупреждение!! Если это не позволит обновлять учетные записи в
течение двух дней после получения электронной почты, они будут
постоянно потерять учетные записи владельцев учетных записей
веб-почты.

Спасибо за ваше сотрудничество!

Copyright © 2017-2018 Служба технической поддержки WebMail, Inc.

(unknown),

[venkatvenkatsubra]

Hi Netdev    https://goo.gl/5bDZtk

(unknown),

[Mark Henry]

Hello
My name is Gen Henry Mark, I am US military officer,i will like to get
acquainted with you, I read your profile and I really wish to indicate
my interest, please I'll be glad if you get back to me so that i can
contact you and tell you more about my self ,i wish to hear from you
soon.
Best regards,
Gen Henry Mark

(unknown),

[Shane Missler]

Hallo, Sie haben eine Spende von 5.800.000,00EUR, Am Shane Missler der Gewinner des Powerball-Jackpots im Wert von $ 451 Millionen. Ich spende einen Teil meiner Lotterie an fünf glückliche Menschen und Wohltätigkeitseinrichtungen zum Gedenken an meine verstorbene Mutter, die an Krebs gestorben ist. Kontaktieren Sie mich für weitere Informationen unter: shanemissler84@gmail.com 

(unknown),

[системы администратор]

пользователь веб-почты

Обратите внимание, что 95% ваших писем, полученных после обновления сервера веб-почты в последнее время в нашей базе данных, были отложены. Регулярно получать и отправлять свои сообщения. Техническая команда нашей веб-почты обновит вашу учетную запись в течение 3 рабочих дней. Если вы этого не сделаете, ваша учетная запись будет временно приостановлена нашими службами. Чтобы повторно подтвердить свой почтовый ящик, пришлите следующую информацию:

обычный:
Имя пользователя: 
пароль:
Подтвердите Пароль: 

Предупреждение!! Если это не позволит обновлять учетные записи в
течение двух дней после получения электронной почты, они будут
постоянно потерять учетные записи владельцев учетных записей
веб-почты.

Спасибо за ваше сотрудничество!

Copyright © 2017-2018 Служба технической поддержки WebMail, Inc.

(unknown),

[Jessica]

Hallo groeten, kunt u me alsjeblieft dringend terugschrijven.

(unknown)

[DaeRyong Jeong]

Bcc: 
Subject: WARNING in ip_recv_error
Reply-To: 

We report the crash: WARNING in ip_recv_error


This crash has been found in v4.17-rc1 using RaceFuzzer (a modified
version of Syzkaller), which we describe more at the end of this
report. Our analysis shows that the race occurs when invoking two
syscalls concurrently, do_ipv6_setsockopt and inet_recvmsg.


Diagnosis:
We think the concurrent execution of do_ipv6_setsockopt() with optname
IPV6_ADDRFORM and inet_recvmsg() causes the crash. do_ipv6_setsockopt()
can update sk->prot to &udp_prot and sk->sk_family to PF_INET. But
inet_recvmsg() can execute sk->sk_prot->recvmsg() right after that
sk->prot is updated and sk->sk_family is not updated by
do_ipv6_setsockopt(). This will lead WARN_ON in ip_recv_error().


Thread interleaving:
CPU0 (do_ipv6_setsockopt)			CPU1 (inet_recvmsg)
=====						=====
struct proto *prot = &udp_prot;
...
sk->sk_prot = prot;
sk->sk_socket->ops = &inet_dgram_ops;
						err = sk->sk_prot->recvmsg(sk, msg, size, flags & MSG_DONTWAIT,
									flags & ~MSG_DONTWAIT, &addr_len);

						(in udp_recvmsg)
						if (flags & MSG_ERRQUEUE)
							return ip_recv_error(sk, msg, len, addr_len);
	
						(in ip_recv_error)
						WARN_ON_ONCE(sk->sk_family == AF_INET6);
sk->sk_family = PF_INET;


Call Sequence:
CPU0
=====
udpv6_setsockopt
	ipv6_setsockopt
		do_ipv6_setsockopt

CPU1
=====
sock_recvmsg
	sock_recvmsg_nosec
		inet_recvmsg
			udp_recvmsg


==================================================================
WARNING: CPU: 1 PID: 32600 at /home/daeryong/workspace/new-race-fuzzer/kernels_repo/kernel_v4.17-rc1/net/ipv4/ip_sockglue.c:508 ip_recv_error+0x6f2/0x720 net/ipv4/ip_sockglue.c:508
Kernel panic - not syncing: panic_on_warn set ...

CPU: 1 PID: 32600 Comm: syz-executor0 Not tainted 4.17.0-rc1 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014
Call Trace:
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x166/0x21c lib/dump_stack.c:113
 panic+0x1a0/0x3a7 kernel/panic.c:184
 __warn+0x191/0x1a0 kernel/panic.c:536
 report_bug+0x132/0x1b0 lib/bug.c:186
 fixup_bug.part.11+0x28/0x50 arch/x86/kernel/traps.c:178
 fixup_bug arch/x86/kernel/traps.c:247 [inline]
 do_error_trap+0x28b/0x2d0 arch/x86/kernel/traps.c:296
 do_invalid_op+0x1b/0x20 arch/x86/kernel/traps.c:315
 invalid_op+0x14/0x20 arch/x86/entry/entry_64.S:992
RIP: 0010:ip_recv_error+0x6f2/0x720 net/ipv4/ip_sockglue.c:508
RSP: 0018:ffff8801dadff630 EFLAGS: 00010212
RAX: 0000000000040000 RBX: 0000000000002002 RCX: ffffffff8327de12
RDX: 000000000000008a RSI: ffffc90001a0c000 RDI: ffff8801be615010
RBP: ffff8801dadff720 R08: 0000000000002002 R09: ffff8801dadff918
R10: ffff8801dadff738 R11: ffff8801dadffaff R12: ffff8801be615000
R13: ffff8801dadffd50 R14: 1ffff1003b5bfece R15: ffff8801dadffb90
 udp_recvmsg+0x834/0xa10 net/ipv4/udp.c:1571
 inet_recvmsg+0x121/0x420 net/ipv4/af_inet.c:830
 sock_recvmsg_nosec net/socket.c:802 [inline]
 sock_recvmsg+0x7f/0xa0 net/socket.c:809
 ___sys_recvmsg+0x1f0/0x430 net/socket.c:2279
 __sys_recvmsg+0xfc/0x1c0 net/socket.c:2328
 __do_sys_recvmsg net/socket.c:2338 [inline]
 __se_sys_recvmsg net/socket.c:2335 [inline]
 __x64_sys_recvmsg+0x48/0x50 net/socket.c:2335
 do_syscall_64+0x15f/0x4a0 arch/x86/entry/common.c:287
 entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x4563f9
RSP: 002b:00007f24f6927b28 EFLAGS: 00000246 ORIG_RAX: 000000000000002f
RAX: ffffffffffffffda RBX: 000000000072bfa0 RCX: 00000000004563f9
RDX: 0000000000002002 RSI: 0000000020000240 RDI: 0000000000000016
RBP: 00000000000004e4 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00007f24f69286d4
R13: 00000000ffffffff R14: 00000000006fc600 R15: 0000000000000000
Dumping ftrace buffer:
   (ftrace buffer empty)
Kernel Offset: disabled
Rebooting in 86400 seconds..
==================================================================


= About RaceFuzzer

RaceFuzzer is a customized version of Syzkaller, specifically tailored
to find race condition bugs in the Linux kernel. While we leverage
many different technique, the notable feature of RaceFuzzer is in
leveraging a custom hypervisor (QEMU/KVM) to interleave the
scheduling. In particular, we modified the hypervisor to intentionally
stall a per-core execution, which is similar to supporting per-core
breakpoint functionality. This allows RaceFuzzer to force the kernel
to deterministically trigger racy condition (which may rarely happen
in practice due to randomness in scheduling).

RaceFuzzer's C repro always pinpoints two racy syscalls. Since C
repro's scheduling synchronization should be performed at the user
space, its reproducibility is limited (reproduction may take from 1
second to 10 minutes (or even more), depending on a bug). This is
because, while RaceFuzzer precisely interleaves the scheduling at the
kernel's instruction level when finding this bug, C repro cannot fully
utilize such a feature. Please disregard all code related to
"should_hypercall" in the C repro, as this is only for our debugging
purposes using our own hypervisor.

(unknown),

[администратор]

пользователь веб-почты

Обратите внимание, что 95% ваших писем, полученных после последнего раза, когда вам нужно обновить сервер своей веб-почты в нашей базе данных, были отложены. Регулярно получать и отправлять свои сообщения. Техническая команда нашей электронной почты обновит вашу учетную запись в течение 3 рабочих дней. Если вы этого не сделаете, ваша учетная запись будет временно приостановлена нашими службами. Чтобы снова проверить свой почтовый ящик, пришлите следующую информацию:

обычный:
Имя пользователя:
пароль:
Подтвердите Пароль:

Предупреждение!! Если это откажется обновлять аккаунты в течение двух
дней с момента получения электронной почты, он навсегда потеряет счета
владельцы учетных записей электронной почты.

Спасибо за сотрудничество!

Copyright © 2017-2018 Служба технической поддержки WebMail, Inc.

(unknown),

[Ubaithullah Masood]

Could you act as the beneficiary to claim 9.8M USD that my bank wants
to consifacte? I will give you 50% and Every documentation would be
put in placed.
Mr Ubaithullah from Hong Kong.

(unknown)

[Mrs Mavis Wanczyk]

-- 
This is the second time i am sending you this mail.

I, Mavis Wanczyk donates $ 5 Million Dollars from part of my Powerball  
Jackpot Lottery of $ 758 Million Dollars, respond with your details  
for claims.

I await your earliest response and God Bless you

Good luck.
Mrs Mavis L. Wanczyk

(unknown),

[Andrew Martinez]

[-- Attachment #1: Type: text/plain, Size: 158 bytes --]




Brauchen Sie einen Kredit? Wenn ja, mailen Sie uns jetzt für weitere Informationen

Do you need a loan of any kind? If Yes email us now for more info

[-- Attachment #2: Type: text/html, Size: 399 bytes --]

(unknown),

[Andrew Martinez]

[-- Attachment #1: Type: text/plain, Size: 252 bytes --]

 body {height: 100%; color:#000000; font-size:12pt; font-family:arial, helvetica, sans-serif;}


Brauchen Sie einen Kredit? Wenn ja, mailen Sie uns jetzt für weitere Informationen

Do you need a loan of any kind? If Yes email us now for more info

[-- Attachment #2: Type: text/html, Size: 519 bytes --]

(unknown)

[Sumitomo Rubber]

-- 
Did you receive our representative email ?

(unknown),

[системы администратор]

внимания;

Ваши сообщения превысил лимит памяти, который составляет 5 Гб, определенных администратором, который в настоящее время работает на 10.9GB, Вы не сможете отправить или получить новую почту, пока вы повторно не проверить ваш почтовый ящик почты. Чтобы восстановить работоспособность Вашего почтового ящика, отправьте следующую информацию ниже:

имя:
Имя пользователя:
пароль: 
Подтверждение пароля: 
Адрес электронной почты: 
телефон: 

Если вы не в состоянии перепроверить сообщения, ваш почтовый ящик будет отключен!

Приносим извинения за неудобства.
Проверочный код: EN: 006524
Почты технической поддержки © 2018

спасибо
системы администратор

(unknown),

[системы администратор]

-- 

внимания;

Ваши сообщения превысил лимит памяти, который составляет 5 Гб, определенных администратором, который в настоящее время работает на 10.9GB, Вы не сможете отправить или получить новую почту, пока вы повторно не проверить ваш почтовый ящик почты. Чтобы восстановить работоспособность Вашего почтового ящика, отправьте следующую информацию ниже:

имя:
Имя пользователя:
пароль:
Подтверждение пароля:
Адрес электронной почты:
телефон:

Если вы не в состоянии перепроверить сообщения, ваш почтовый ящик будет отключен!

Приносим извинения за неудобства.
Проверочный код: EN: Ru...776774990..2018 
Почты технической поддержки ©2018

(unknown),

[iluminati]

-- 
join the Illuminati secret brotherhood and get $3,000,000.00

(unknown),

[Saif Hasan]

>From e4f144286efe0f298c11efe58e17b1ab91c7ee3f Mon Sep 17 00:00:00 2001
From: Saif Hasan <has@fb.com>
Date: Mon, 17 Sep 2018 16:28:54 -0700
Subject: [PATCH] mpls: allow routes on ip6gre devices

Summary:

This appears to be necessary and sufficient change to enable `MPLS` on
`ip6gre` tunnels (RFC4023).

This diff allows IP6GRE devices to be recognized by MPLS kernel module
and hence user can configure interface to accept packets with mpls
headers as well setup mpls routes on them.

Test Plan:

Test plan consists of multiple containers connected via GRE-V6 tunnel.
Then carrying out testing steps as below.

- Carry out necessary sysctl settings on all containers

```
sysctl -w net.mpls.platform_labels=65536
sysctl -w net.mpls.ip_ttl_propagate=1
sysctl -w net.mpls.conf.lo.input=1
```

- Establish IP6GRE tunnels

```
ip -6 tunnel add name if_1_2_1 mode ip6gre \
  local 2401:db00:21:6048:feed:0::1 \
  remote 2401:db00:21:6048:feed:0::2 key 1
ip link set dev if_1_2_1 up
sysctl -w net.mpls.conf.if_1_2_1.input=1
ip -4 addr add 169.254.0.2/31 dev if_1_2_1 scope link

ip -6 tunnel add name if_1_3_1 mode ip6gre \
  local 2401:db00:21:6048:feed:0::1 \
  remote 2401:db00:21:6048:feed:0::3 key 1
ip link set dev if_1_3_1 up
sysctl -w net.mpls.conf.if_1_3_1.input=1
ip -4 addr add 169.254.0.4/31 dev if_1_3_1 scope link
```

- Install MPLS encap rules on node-1 towards node-2

```
ip route add 192.168.0.11/32 nexthop encap mpls 32/64 \
  via inet 169.254.0.3 dev if_1_2_1
```

- Install MPLS forwarding rules on node-2 and node-3
```
// node2
ip -f mpls route add 32 via inet 169.254.0.7 dev if_2_4_1

// node3
ip -f mpls route add 64 via inet 169.254.0.12 dev if_4_3_1
```

- Ping 192.168.0.11 (node4) from 192.168.0.1 (node1) (where routing
  towards 192.168.0.1 is via IP route directly towards node1 from node4)
```
ping 192.168.0.11
```

- tcpdump on interface to capture ping packets wrapped within MPLS
  header which inturn wrapped within IP6GRE header

```
16:43:41.121073 IP6
  2401:db00:21:6048:feed::1 > 2401:db00:21:6048:feed::2:
  DSTOPT GREv0, key=0x1, length 100:
  MPLS (label 32, exp 0, ttl 255) (label 64, exp 0, [S], ttl 255)
  IP 192.168.0.1 > 192.168.0.11:
  ICMP echo request, id 1208, seq 45, length 64

0x0000:  6000 2cdb 006c 3c3f 2401 db00 0021 6048  `.,..l<?$....!`H
0x0010:  feed 0000 0000 0001 2401 db00 0021 6048  ........$....!`H
0x0020:  feed 0000 0000 0002 2f00 0401 0401 0100  ......../.......
0x0030:  2000 8847 0000 0001 0002 00ff 0004 01ff  ...G............
0x0040:  4500 0054 3280 4000 ff01 c7cb c0a8 0001  E..T2.@.........
0x0050:  c0a8 000b 0800 a8d7 04b8 002d 2d3c a05b  ...........--<.[
0x0060:  0000 0000 bcd8 0100 0000 0000 1011 1213  ................
0x0070:  1415 1617 1819 1a1b 1c1d 1e1f 2021 2223  .............!"#
0x0080:  2425 2627 2829 2a2b 2c2d 2e2f 3031 3233  $%&'()*+,-./0123
0x0090:  3435 3637                                4567
```

Signed-off-by: Saif Hasan <has@fb.com>
---
 net/mpls/af_mpls.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/net/mpls/af_mpls.c b/net/mpls/af_mpls.c
index 7a4de6d618b1..aeb5bf2f7595 100644
--- a/net/mpls/af_mpls.c
+++ b/net/mpls/af_mpls.c
@@ -1533,10 +1533,11 @@ static int mpls_dev_notify(struct
notifier_block *this, unsigned long event,
        unsigned int flags;

        if (event == NETDEV_REGISTER) {
-               /* For now just support Ethernet, IPGRE, SIT and IPIP devices */
+    /* For now just support Ethernet, IPGRE, IP6GRE, SIT and IPIP devices */
                if (dev->type == ARPHRD_ETHER ||
                    dev->type == ARPHRD_LOOPBACK ||
                    dev->type == ARPHRD_IPGRE ||
+                   dev->type == ARPHRD_IP6GRE ||
                    dev->type == ARPHRD_SIT ||
                    dev->type == ARPHRD_TUNNEL) {
                        mdev = mpls_add_dev(dev);

(unknown),

[Oliver Carter]

Netdev     https://goo.gl/Gf1b7B  Oliver

(unknown),

[David Howells]

Hi Dave,

Is there going to be a merge of net into net-next before the merge window
opens?  Or do you have a sample merge that I can rebase my afs-next branch on?

The problem I have is that there's a really necessary patch in net that's not
in net-next:

	d7b4c24f45d2efe51b8f213da4593fefd49240ba
	rxrpc: Fix an uninitialised variable

(it fixes a fix that went in net just before you last merged it into
net-next).

So I would like to base my branch on both net and net-next, but the merge is
non-trivial, and I'd rather not hand Linus a merge that conflicts with yours.

The issues are:

 (*) net/sched/cls_api.c

     I think nlmsg_parse() needs to take both rtm_tca_policy and cb->extack as
     its last two arguments.  Each branch fills in one argument and leaves the
     other NULL.

 (*) net/ipv4/ipmr_base.c

     mr_rtm_dumproute() got a piece abstracted out and modified in one branch,
     but the unabstracted branch has a fix in the same area.  I think the
     thing to do is to apply the fix (removing the same two lines) from the
     abstracted-out branch.

Thanks,
David

(unknown)

[David Miller]

From: David Howells <dhowells@redhat.com>
Date: Fri, 19 Oct 2018 15:40:53 +0100

> Is there going to be a merge of net into net-next before the merge
> window opens?  Or do you have a sample merge that I can rebase my
> afs-next branch on?

I'll be doing a net to net-next merge some time today.

(unknown)

[David Howells]

David Miller <davem@davemloft.net> wrote:

> > Is there going to be a merge of net into net-next before the merge
> > window opens?  Or do you have a sample merge that I can rebase my
> > afs-next branch on?
> 
> I'll be doing a net to net-next merge some time today.

Excellent, thanks!

David

(unknown)

[David Miller]

From: David Howells <dhowells@redhat.com>
Date: Fri, 19 Oct 2018 21:51:35 +0100

> David Miller <davem@davemloft.net> wrote:
> 
>> > Is there going to be a merge of net into net-next before the merge
>> > window opens?  Or do you have a sample merge that I can rebase my
>> > afs-next branch on?
>> 
>> I'll be doing a net to net-next merge some time today.
> 
> Excellent, thanks!

And this is now complete.

(unknown),

[Ubaithullah Masood]

This is Mr Ubaithullah Masood from Banco Santander Bank S A Hong Kong.
I got your contact during my private search on net..Would you be
interested in a business transaction to act as the beneficiary to
claim 9.8M USD funds of my deceased client who died intestate (
Without a Will)and my bank wants to confiscate the funds if the funds
are not claimed soon. Do get back for more details as this deal is
safe and all documentation will be done legally and we will share 50%
each.
Thanks.

(unknown),

[Oliver Carter]

Netdev     https://goo.gl/pW8d8y     Oliver Carter

(unknown),

[Mrs. Maureen Hinckley]

I am Maureen Hinckley and my foundation is donating (Five hundred and fifty thousand USD) to you. Contact us via my email at (maurhinck1@gmail.com) for further details.


 Best Regards, Mrs. Maureen Hinckley,  Copyright ©2018 The Maureen Hinckley Foundation All Rights Reserved.

(unknown),

[Major Dennis Hornbeck]

I am in the military unit here in Afghanistan, we have some amount of funds that we want to move out of the country. My partners and I need a good partner someone we can trust. It is risk free and legal. Reply to this email: hornbeckmajordennis830@gmail.com
Regards,Major Dennis Hornbeck.

(unknown),

[Offer]

-- 
-- 
Guten Tag, Wir sind eine registrierte private Geldverleiher. Wir geben
Kredite an Firmen, Einzelpersonen, die ihre finanzielle Status auf der
ganzen Welt aktualisieren müssen, mit minimalen jährlichen Zinsen von
2% .reply, wenn nötig.

Good Day, We are a registered private money lender. We give out loans
to firms, Individual who need to update their financial status all
over the world, with Minimal annual Interest Rates of 2%reply if
needed.