* Re: tproxy related crash in inet_hashtables
From: Eric Dumazet @ 2010-08-13 10:54 UTC (permalink / raw)
To: Stephen Buck
Cc: netdev, Netfilter Development Mailinglist, Patrick McHardy,
KOVACS Krisztian
In-Reply-To: <4C64FF2E.6060509@exinda.com>
Le vendredi 13 août 2010 à 18:15 +1000, Stephen Buck a écrit :
> Recently I encountered a number of crashes related to tproxy on the
> 2.6.34.1 (x86_64 SMP) kernel. These usually manifested as a bug like the
> following (Although the bug was confirmed to be present on a vanilla
> kernel, this particular trace is from a kernel with some customisations):
>
> [ 1504.765077] BUG: unable to handle kernel NULL pointer dereference at
> (null)
> [ 1504.848183] IP: [<ffffffff8135a79b>] inet_bind_bucket_destroy+0x1b/0x30
> [ 1504.927126] PGD 1a9933067 PUD 1ad909067 PMD 0
> [ 1504.980125] Thread overran stack, or stack corrupted
> [ 1505.039325] Oops: 0002 #1 SMP
> [ 1505.077775] last sysfs file:
> /sys/devices/system/cpu/cpu15/topology/thread_siblings
> [ 1505.169166] CPU 0
> [ 1505.193070] Modules linked in: sch_sfq cls_fw sch_htb xt_physdev
> 8021q bridge stp llc
> [ 1505.923769] Pid: 0, comm: swapper Not tainted 2.6.31-12EXINDAsmp #0
> PowerEdge R710
> [ 1506.014118] RIP: 0010:[<ffffffff8135a79b>] [<ffffffff8135a79b>]
> inet_bind_bucket_destroy+0x1b/0x30
> [ 1506.122242] RSP: 0018:ffffc90000003e10 EFLAGS: 00010246
> [ 1506.185655] RAX: 0000000000000000 RBX: ffffc900164a02a0 RCX:
> ffffea00098e24b0
> [ 1506.270863] RDX: 0000000000000000 RSI: ffff8802e1186280 RDI:
> ffffffff815b4600
> [ 1506.356077] RBP: ffffc90000003e10 R08: 0000000000000016 R09:
> 0000000000000001
> [ 1506.441284] R10: 0000000000000000 R11: 0000000000000000 R12:
> ffff88011b894fc0
> [ 1506.526500] R13: ffffffff81744c80 R14: ffffffff815fe6c0 R15:
> 0000000000000003
> [ 1506.611722] FS: 0000000000000000(0000) GS:ffffc90000000000(0000)
> knlGS:0000000000000000
> [ 1506.708410] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> [ 1506.777011] CR2: 0000000000000000 CR3: 00000001ae17a000 CR4:
> 00000000000006f0
> [ 1506.862225] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [ 1506.947442] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [ 1507.032650] Process swapper (pid: 0, threadinfo ffffffff815b0000,
> task ffffffff815b6bc0)
> [ 1507.129345] Stack:
> [ 1507.153299] ffffc90000003e40 ffffffff8135b834 0000000000000001
> ffff88011b894fc0
> [ 1507.239581] <0> ffffffff815fe560 0000000000000002 ffffc90000003e80
> ffffffff8135bbf6
> [ 1507.331133] <0> ffffffff815fe560 ffffffff815fe560 ffffffff815fe6c0
> ffffc90000003eb0
> [ 1507.424833] Call Trace:
> [ 1507.453976] <IRQ>
> [ 1507.478971] [<ffffffff8135b834>] __inet_twsk_kill+0xb4/0xf0
> [ 1507.546538] [<ffffffff8135bbf6>] inet_twdr_do_twkill_work+0x66/0xd0
> [ 1507.622408] [<ffffffff8135bd40>] ? inet_twdr_hangman+0x0/0xd0
> [ 1507.692041] [<ffffffff8135bd95>] inet_twdr_hangman+0x55/0xd0
> [ 1507.760650] [<ffffffff810515ec>] run_timer_softirq+0x18c/0x220
> [ 1507.831330] [<ffffffff8104c3b8>] __do_softirq+0xc8/0x1f0
> [ 1507.895799] [<ffffffff8100cf5c>] call_softirq+0x1c/0x30
> [ 1507.959209] [<ffffffff8100e5f5>] do_softirq+0x45/0x80
> [ 1508.020538] [<ffffffff8104c2e7>] irq_exit+0x87/0x90
> [ 1508.079797] [<ffffffff813bd0c1>] smp_apic_timer_interrupt+0x71/0x9d
> [ 1508.155667] [<ffffffff8100c933>] apic_timer_interrupt+0x13/0x20
> [ 1508.227385] <EOI>
> [ 1508.252383] [<ffffffff8101317e>] ? mwait_idle+0x7e/0x110
> [ 1508.316836] [<ffffffff813bb0bd>] ? __atomic_notifier_call_chain+0xd/0x10
> [ 1508.397903] [<ffffffff813bb0d1>] ? atomic_notifier_call_chain+0x11/0x20
> [ 1508.477928] [<ffffffff8100ac31>] ? cpu_idle+0x51/0x90
> [ 1508.539265] [<ffffffff813aa92b>] ? rest_init+0x6b/0x80
> [ 1508.601639] [<ffffffff816230f5>] ? start_kernel+0x2c5/0x370
> [ 1508.669200] [<ffffffff81622611>] ? x86_64_start_reservations+0x81/0xc0
> [ 1508.748192] [<ffffffff81622726>] ? x86_64_start_kernel+0xd6/0x100
> [ 1508.821979] Code: 64 10 40 eb 94 0f 1f 44 00 00 66 0f 1f 44 00 00 55
> 48 89 e5 0f 1f 44 00 00 48 83 7e 18 00 75 19 48 8b 46 08 48 8b 56 10
> [ 1509.048123] RIP [<ffffffff8135a79b>] inet_bind_bucket_destroy+0x1b/0x30
> [ 1509.128157] RSP <ffffc90000003e10>
> [ 1509.169759] CR2: 0000000000000000
>
>
> After spending a while tracking it down, I discovered that the wrong
> locks get held when operating on the bind hash table's chains.
>
> This is due to the listen socket and the child socket having different
> local ports when __inet_inherit_port() is called. The lock is held based
> on the child socket's port, but the list operated on is the one the
> listen socket belongs to.
>
> e.g.
> There is a transparent proxy listening on port 9999.
> A new http connection (with port 80) is redirected to the proxy.
>
> The inet_bind_hashbucket locked in this case is table->bhash[80].lock,
> but the inet_bind_bucket the child socket is added to is in the chain of
> the table->bhash[9999] inet_bind_hashbucket. This means that if another
> connection with a different local port arrived and was redirected to the
> proxy, they could both be operating on the list at the same time.
>
>
> Attached is a patch that should fix this by looking up the correct
> inet_bind_bucket based on the child's local port when the
> inet_bind_bucket from the listen socket has a different port to the
> child's inet_num. It was built against 2.6.34.1, but should apply to any
> mainline kernel.
>
> It is also possible the same bug exists in the IPv6 code as well. As I
> have not had to deal with IPv6 yet, I have not had a look.
Hi Stephen
CC netfilter-devel & Patrick & Krisztian
I cannot convince myself this patch is a right fix.
This probably should be fixed in netfilter tree, not in
net/ipv4/inet_hashtables.c ?
Once tproxy is involved, the original port (80) should be changed to
9999 by tproxy (for SYN packet) and conntrack for following ones.
So listening socket and its children all use source port 9999 ?
(inet_sk(child)->inet_num == inet_sk(parent)->inet_num)
You claim wrong lock is taken at insert time, but are you sure the right
lock is taken at deletion time ?
Hmm...
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [MeeGo-Dev][PATCH] Topcliff: Update PCH_CAN driver to 2.6.35
From: Wolfgang Grandegger @ 2010-08-13 10:58 UTC (permalink / raw)
To: Wang, Qi
Cc: Khor, Andrew Chih Howe, Masayuki Ohtak,
netdev-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
gregkh-l3A5Bk7waGM@public.gmane.org, Wang, Yong Y,
socketcan-core-0fE9KPoRgkgATYTw5x5z8w@public.gmane.org,
meego-dev-WXzIur8shnEAvxtiuMwx3w@public.gmane.org,
arjan-VuQAYsv1563Yd54FQh9/CA@public.gmane.org
In-Reply-To: <D5AB6E638E5A3E4B8F4406B113A5A19A28EA2AB1-QQHDSDV1ERZpB2pF5aRoyrfspsVTdybXVpNB7YpNyf8@public.gmane.org>
On 08/13/2010 02:23 AM, Wang, Qi wrote:
>> -----Original Message-----
>> From: Wolfgang Grandegger [mailto:wg-5Yr1BZd7O62+XT7JhA+gdA@public.gmane.org]
>> Sent: Thursday, August 12, 2010 5:04 PM
>> To: Wang, Qi
>> Cc: Daniel Baluta; Masayuki Ohtak; meego-dev-WXzIur8shnEAvxtiuMwx3w@public.gmane.org;
>> socketcan-core-0fE9KPoRgkgATYTw5x5z8w@public.gmane.org; netdev-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; Khor, Andrew Chih
>> Howe; gregkh-l3A5Bk7waGM@public.gmane.org; arjan-VuQAYsv1563Yd54FQh9/CA@public.gmane.org; Wang, Yong Y
>> Subject: Re: [MeeGo-Dev][PATCH] Topcliff: Update PCH_CAN driver to 2.6.35
>>
>> On 08/12/2010 03:42 AM, Wang, Qi wrote:
>>>> -----Original Message-----
>>>> From: Daniel Baluta [mailto:daniel.baluta-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org]
>>>> Sent: Wednesday, August 11, 2010 6:37 PM
>>>> To: Masayuki Ohtak
>>>> Cc: meego-dev-WXzIur8shnEAvxtiuMwx3w@public.gmane.org; Wolfgang Grandegger;
>>>> socketcan-core-0fE9KPoRgkgATYTw5x5z8w@public.gmane.org; netdev-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; Khor, Andrew
>> Chih
>>>> Howe; gregkh-l3A5Bk7waGM@public.gmane.org; arjan-VuQAYsv1563Yd54FQh9/CA@public.gmane.org; Wang, Qi; Wang, Yong Y
>>>> Subject: Re: [MeeGo-Dev][PATCH] Topcliff: Update PCH_CAN driver to 2.6.35
>>>>
>>>> Hi,
>>>>
>>>> 2010/8/11 Masayuki Ohtak <masa-korg-ECg8zkTtlr0C6LszWs/t0g@public.gmane.org>:
>>>>> CAN driver of Topcliff PCH
>>>>>
>>>>> Topcliff PCH is the platform controller hub that is going to be used in
>>>>> Intel's upcoming general embedded platform. All IO peripherals in
>>>>> Topcliff PCH are actually devices sitting on AMBA bus.
>>>>> Topcliff PCH has CAN I/F. This driver enables CAN function.
>>>>>
>>>>> Signed-off-by: Masayuki Ohtake <masa-korg-ECg8zkTtlr0C6LszWs/t0g@public.gmane.org>
>>>>
>>>> I have a few questions:
>>>>
>>>> 1. Is your code based on Intel's CAN EP80579 ([1]) ?
>>> No.
>>
>> For curiosity, is the controller similar to the OKI MSM9225 or ML9620?
> The Topcliff IOH is developed by OKI actually.
>
>>
>>>> 2. Why don't you use kernel existing kfifo infrastructure? ([2]).
>>> Just take a look at kfifo.h. This structure has been changed. I remembered
>> there was a spin_lock from kfifo previously. Currently it's been removed, good.
>>> OKI-sans, would you please take a look at ./include/linux/kfifo.h, and try to
>> use this structure and APIs?
>>
>> As I see it, the code related to that fifo is not used (== dead code)?
> I'm not familiar with kfifo structure, and I didn't like it because there need a spin_lock to use it.
>>
>>> Daniel,
>>>
>>> We're anxious to integrate those codes now. Perhaps it'll take us quite a long
>> time to use kfifo. How about implementing it with the next version?
>>
>> See above. What do you mean with the next version. The driver posted by
>> Masayuki is far away from being accepted as it does not yet comply with
>> the Socket-CAN driver API, to say the least.
> I've few experience on CAN driver and it's also the first time for OKI-san to write Can driver. Would you please give us a reference and we'll follow it up. We only read 'can.txt' from kernel document. Thank you for your help in advance.
You are welcome. I think I/we already gave useful hints on what is
missing and what examples to follow.
Wolfgang.
^ permalink raw reply
* Re: Problem with non aligned DMA in usbnet on ARM
From: Oliver Neukum @ 2010-08-13 10:58 UTC (permalink / raw)
To: mfuzzey-Re5JQEeQqe8AvxtiuMwx3w
Cc: Matthieu CASTET, Russell King - ARM Linux, Gary King,
linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org,
linux-usb, netdev-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Greg KH
In-Reply-To: <4C65193E.4090807-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Am Freitag, 13. August 2010, 12:06:54 schrieb Martin Fuzzey:
> So I think a policy needs to be defined to ensure this and enforced in
> the code. I can see two possible methods:
>
> 1) Require that usb drivers submit buffers obtained from kmalloc() and
> friends with no extra offsets. If they want some other alignment later
> they can use memmove in the completion handler. Enforce this in the core
> by checking the buffer pointers are aligned to ARCH_KMALLOC_MINALIGN
>
> or
>
> 2) Require that usb_submit_urb() accept byte aligned buffers. Enforce
> this by a new test in usbtest (which all HCDs are expected to pass).
> Implement it either in individual HCDs that require it or by letting
> HCDs inform the core of their requirements and have the core do the
> alignment (as it already does the dma mapping). Of course HCDs that can
> implement byte aligned transfers (either natively or using tricks such
> as the one Russell suggested) should do so.
>
> I think 2) is the better solution because:
> a) Solution 1 will impose a runtime overhead even on platforms / HCDs
> that don't need it (including the most common cases)
> b) There are more drivers than HCDs
Yes. But it doesn't prevent you from publishing this information so
drivers can help the lower levels.
Regards
Oliver
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: 2.6.35, netfilter/iptables not working with 32bit userspace + 64bit kernel?
From: Eric Dumazet @ 2010-08-13 12:18 UTC (permalink / raw)
To: Jussi Kivilinna; +Cc: netdev, Netfilter Development Mailinglist
In-Reply-To: <20100813134706.509472n222yo71d6@hayate.sektori.org>
CC netfilter-devel to get more people in touch
Thanks !
Le vendredi 13 août 2010 à 13:47 +0300, Jussi Kivilinna a écrit :
> Hello!
>
> I have server/firewall running Debian lenny with 32bit userspace and
> 64bit kernel. After upgrading from 2.6.34 to 2.6.35, I couldn't get
> any new chains work. Simply doing (with 32bit iptables)
>
> iptables -N new_chain
> iptables -A OUTPUT -j new_chain
> iptables -A OUTPUT -j ACCEPT
>
> cause output to freeze. "iptables -L -vn" shows:
>
> Chain OUTPUT (policy ACCEPT 3397 packets, 637K bytes)
> pkts bytes target prot opt in out source destination
> 3 252 new_chain all -- * * 0.0.0.0/0 0.0.0.0/0
> 0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
>
> Ping shows "ping: sendmsg: Operation not permitted".
>
> With recompiled 64bit iptables, example above works fine.
>
> -Jussi
>
> --
^ permalink raw reply
* Re: 2.6.35, netfilter/iptables not working with 32bit userspace + 64bit kernel?
From: Florian Westphal @ 2010-08-13 12:19 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Jussi Kivilinna, netdev, Netfilter Development Mailinglist
In-Reply-To: <1281701885.4470.22.camel@edumazet-laptop>
Eric Dumazet <eric.dumazet@gmail.com> wrote:
> CC netfilter-devel to get more people in touch
>
> Thanks !
>
> Le vendredi 13 août 2010 à 13:47 +0300, Jussi Kivilinna a écrit :
> > Hello!
> >
> > I have server/firewall running Debian lenny with 32bit userspace and
> > 64bit kernel. After upgrading from 2.6.34 to 2.6.35, I couldn't get
> > any new chains work. Simply doing (with 32bit iptables)
> >
> > iptables -N new_chain
> > iptables -A OUTPUT -j new_chain
> > iptables -A OUTPUT -j ACCEPT
> >
> > cause output to freeze. "iptables -L -vn" shows:
> >
> > Chain OUTPUT (policy ACCEPT 3397 packets, 637K bytes)
> > pkts bytes target prot opt in out source destination
> > 3 252 new_chain all -- * * 0.0.0.0/0 0.0.0.0/0
> > 0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
> >
> > Ping shows "ping: sendmsg: Operation not permitted".
> >
> > With recompiled 64bit iptables, example above works fine.
You need this patch:
commit f3c5c1bfd430858d3a05436f82c51e53104feb6b
(netfilter: xtables: make ip_tables reentrant) forgot to
also compute the jumpstack size in the compat handlers.
Result is that "iptables -I INPUT -j userchain" turns into -j DROP.
Reported by Sebastian Roesner on #netfilter, closes
http://bugzilla.netfilter.org/show_bug.cgi?id=669.
Note: arptables change is compile-tested only.
Signed-off-by: Florian Westphal <fw@strlen.de>
---
net/ipv4/netfilter/arp_tables.c | 3 +++
net/ipv4/netfilter/ip_tables.c | 3 +++
net/ipv6/netfilter/ip6_tables.c | 3 +++
3 files changed, 9 insertions(+), 0 deletions(-)
diff --git a/net/ipv4/netfilter/arp_tables.c b/net/ipv4/netfilter/arp_tables.c
index 6bccba3..4829766 100644
--- a/net/ipv4/netfilter/arp_tables.c
+++ b/net/ipv4/netfilter/arp_tables.c
@@ -1418,6 +1418,9 @@ static int translate_compat_table(const char *name,
if (ret != 0)
break;
++i;
+ if (strcmp(arpt_get_target(iter1)->u.user.name,
+ XT_ERROR_TARGET) == 0)
+ ++newinfo->stacksize;
}
if (ret) {
/*
diff --git a/net/ipv4/netfilter/ip_tables.c b/net/ipv4/netfilter/ip_tables.c
index c439721..3d3b695 100644
--- a/net/ipv4/netfilter/ip_tables.c
+++ b/net/ipv4/netfilter/ip_tables.c
@@ -1749,6 +1749,9 @@ translate_compat_table(struct net *net,
if (ret != 0)
break;
++i;
+ if (strcmp(ipt_get_target(iter1)->u.user.name,
+ XT_ERROR_TARGET) == 0)
+ ++newinfo->stacksize;
}
if (ret) {
/*
diff --git a/net/ipv6/netfilter/ip6_tables.c b/net/ipv6/netfilter/ip6_tables.c
index 5359ef4..bad3c7f 100644
--- a/net/ipv6/netfilter/ip6_tables.c
+++ b/net/ipv6/netfilter/ip6_tables.c
@@ -1764,6 +1764,9 @@ translate_compat_table(struct net *net,
if (ret != 0)
break;
++i;
+ if (strcmp(ip6t_get_target(iter1)->u.user.name,
+ XT_ERROR_TARGET) == 0)
+ ++newinfo->stacksize;
}
if (ret) {
/*
--
1.7.1
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* RE: [PATCH] Add firmware label support to iproute2
From: Narendra_K @ 2010-08-13 12:36 UTC (permalink / raw)
To: shemminger; +Cc: netdev, Matt_Domsch, Charles_Rose, Jordan_Hargrave
In-Reply-To: <20100812141240.417949e2@s6510>
> -----Original Message-----
> From: Stephen Hemminger [mailto:shemminger@vyatta.com]
> Sent: Thursday, August 12, 2010 11:43 PM
> To: K, Narendra
> Cc: netdev@vger.kernel.org; Domsch, Matt; Rose, Charles; Hargrave,
> Jordan
> Subject: Re: [PATCH] Add firmware label support to iproute2
>
> On Thu, 12 Aug 2010 12:35:37 -0500
> Narendra K <Narendra_K@dell.com> wrote:
>
> > +#ifndef LIBNETDEVNAME_PRESENT
> > filter_dev = *argv;
> > +#else
> > + if (netdev_alias_to_kernelname(*argv,
kernel_name) <
> 0)
> > + show_firmware_alias_usage(*argv);
> > + filter_dev = kernel_name;
> > +#endif
> > }
>
> like the kernel, I don't like ifdef cases in main code.
> You should put in stub inline that returns appropriate error.
>
Stephen,
Thanks for the suggestions. I will rework the patch and send a revised
version soon.
> What happens if alias matches existing interface name?
If the user inputs an existing 'ethN' name, the function
'netdev_alias_to_kernelaname' will return it as it is. If the user
inputs 'Embedded NIC N', then the function would do a sysfs lookup and
return the corresponding 'ethN' name.
With regards,
Narendra K
^ permalink raw reply
* Re: tproxy related crash in inet_hashtables
From: Stephen Buck @ 2010-08-13 13:05 UTC (permalink / raw)
To: Eric Dumazet
Cc: netdev@vger.kernel.org, Netfilter Development Mailinglist,
Patrick McHardy, KOVACS Krisztian
In-Reply-To: <1281696852.4470.20.camel@edumazet-laptop>
On 13/08/10 20:54, Eric Dumazet wrote:
> Le vendredi 13 août 2010 à 18:15 +1000, Stephen Buck a écrit :
>> Recently I encountered a number of crashes related to tproxy on the
>> 2.6.34.1 (x86_64 SMP) kernel. These usually manifested as a bug like the
>> following (Although the bug was confirmed to be present on a vanilla
>> kernel, this particular trace is from a kernel with some customisations):
>>
>> [ 1504.765077] BUG: unable to handle kernel NULL pointer dereference at
>> (null)
>> [ 1504.848183] IP: [<ffffffff8135a79b>] inet_bind_bucket_destroy+0x1b/0x30
>> [ 1504.927126] PGD 1a9933067 PUD 1ad909067 PMD 0
>> [ 1504.980125] Thread overran stack, or stack corrupted
>> [ 1505.039325] Oops: 0002 #1 SMP
>> [ 1505.077775] last sysfs file:
>> /sys/devices/system/cpu/cpu15/topology/thread_siblings
>> [ 1505.169166] CPU 0
>> [ 1505.193070] Modules linked in: sch_sfq cls_fw sch_htb xt_physdev
>> 8021q bridge stp llc
>> [ 1505.923769] Pid: 0, comm: swapper Not tainted 2.6.31-12EXINDAsmp #0
>> PowerEdge R710
>> [ 1506.014118] RIP: 0010:[<ffffffff8135a79b>] [<ffffffff8135a79b>]
>> inet_bind_bucket_destroy+0x1b/0x30
>> [ 1506.122242] RSP: 0018:ffffc90000003e10 EFLAGS: 00010246
>> [ 1506.185655] RAX: 0000000000000000 RBX: ffffc900164a02a0 RCX:
>> ffffea00098e24b0
>> [ 1506.270863] RDX: 0000000000000000 RSI: ffff8802e1186280 RDI:
>> ffffffff815b4600
>> [ 1506.356077] RBP: ffffc90000003e10 R08: 0000000000000016 R09:
>> 0000000000000001
>> [ 1506.441284] R10: 0000000000000000 R11: 0000000000000000 R12:
>> ffff88011b894fc0
>> [ 1506.526500] R13: ffffffff81744c80 R14: ffffffff815fe6c0 R15:
>> 0000000000000003
>> [ 1506.611722] FS: 0000000000000000(0000) GS:ffffc90000000000(0000)
>> knlGS:0000000000000000
>> [ 1506.708410] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
>> [ 1506.777011] CR2: 0000000000000000 CR3: 00000001ae17a000 CR4:
>> 00000000000006f0
>> [ 1506.862225] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>> 0000000000000000
>> [ 1506.947442] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
>> 0000000000000400
>> [ 1507.032650] Process swapper (pid: 0, threadinfo ffffffff815b0000,
>> task ffffffff815b6bc0)
>> [ 1507.129345] Stack:
>> [ 1507.153299] ffffc90000003e40 ffffffff8135b834 0000000000000001
>> ffff88011b894fc0
>> [ 1507.239581]<0> ffffffff815fe560 0000000000000002 ffffc90000003e80
>> ffffffff8135bbf6
>> [ 1507.331133]<0> ffffffff815fe560 ffffffff815fe560 ffffffff815fe6c0
>> ffffc90000003eb0
>> [ 1507.424833] Call Trace:
>> [ 1507.453976]<IRQ>
>> [ 1507.478971] [<ffffffff8135b834>] __inet_twsk_kill+0xb4/0xf0
>> [ 1507.546538] [<ffffffff8135bbf6>] inet_twdr_do_twkill_work+0x66/0xd0
>> [ 1507.622408] [<ffffffff8135bd40>] ? inet_twdr_hangman+0x0/0xd0
>> [ 1507.692041] [<ffffffff8135bd95>] inet_twdr_hangman+0x55/0xd0
>> [ 1507.760650] [<ffffffff810515ec>] run_timer_softirq+0x18c/0x220
>> [ 1507.831330] [<ffffffff8104c3b8>] __do_softirq+0xc8/0x1f0
>> [ 1507.895799] [<ffffffff8100cf5c>] call_softirq+0x1c/0x30
>> [ 1507.959209] [<ffffffff8100e5f5>] do_softirq+0x45/0x80
>> [ 1508.020538] [<ffffffff8104c2e7>] irq_exit+0x87/0x90
>> [ 1508.079797] [<ffffffff813bd0c1>] smp_apic_timer_interrupt+0x71/0x9d
>> [ 1508.155667] [<ffffffff8100c933>] apic_timer_interrupt+0x13/0x20
>> [ 1508.227385]<EOI>
>> [ 1508.252383] [<ffffffff8101317e>] ? mwait_idle+0x7e/0x110
>> [ 1508.316836] [<ffffffff813bb0bd>] ? __atomic_notifier_call_chain+0xd/0x10
>> [ 1508.397903] [<ffffffff813bb0d1>] ? atomic_notifier_call_chain+0x11/0x20
>> [ 1508.477928] [<ffffffff8100ac31>] ? cpu_idle+0x51/0x90
>> [ 1508.539265] [<ffffffff813aa92b>] ? rest_init+0x6b/0x80
>> [ 1508.601639] [<ffffffff816230f5>] ? start_kernel+0x2c5/0x370
>> [ 1508.669200] [<ffffffff81622611>] ? x86_64_start_reservations+0x81/0xc0
>> [ 1508.748192] [<ffffffff81622726>] ? x86_64_start_kernel+0xd6/0x100
>> [ 1508.821979] Code: 64 10 40 eb 94 0f 1f 44 00 00 66 0f 1f 44 00 00 55
>> 48 89 e5 0f 1f 44 00 00 48 83 7e 18 00 75 19 48 8b 46 08 48 8b 56 10
>> [ 1509.048123] RIP [<ffffffff8135a79b>] inet_bind_bucket_destroy+0x1b/0x30
>> [ 1509.128157] RSP<ffffc90000003e10>
>> [ 1509.169759] CR2: 0000000000000000
>>
>>
>> After spending a while tracking it down, I discovered that the wrong
>> locks get held when operating on the bind hash table's chains.
>>
>> This is due to the listen socket and the child socket having different
>> local ports when __inet_inherit_port() is called. The lock is held based
>> on the child socket's port, but the list operated on is the one the
>> listen socket belongs to.
>>
>> e.g.
>> There is a transparent proxy listening on port 9999.
>> A new http connection (with port 80) is redirected to the proxy.
>>
>> The inet_bind_hashbucket locked in this case is table->bhash[80].lock,
>> but the inet_bind_bucket the child socket is added to is in the chain of
>> the table->bhash[9999] inet_bind_hashbucket. This means that if another
>> connection with a different local port arrived and was redirected to the
>> proxy, they could both be operating on the list at the same time.
>>
>>
>> Attached is a patch that should fix this by looking up the correct
>> inet_bind_bucket based on the child's local port when the
>> inet_bind_bucket from the listen socket has a different port to the
>> child's inet_num. It was built against 2.6.34.1, but should apply to any
>> mainline kernel.
>>
>> It is also possible the same bug exists in the IPv6 code as well. As I
>> have not had to deal with IPv6 yet, I have not had a look.
> Hi Stephen
>
> CC netfilter-devel& Patrick& Krisztian
>
> I cannot convince myself this patch is a right fix.
>
> This probably should be fixed in netfilter tree, not in
> net/ipv4/inet_hashtables.c ?
>
> Once tproxy is involved, the original port (80) should be changed to
> 9999 by tproxy (for SYN packet) and conntrack for following ones.
>
> So listening socket and its children all use source port 9999 ?
>
> (inet_sk(child)->inet_num == inet_sk(parent)->inet_num)
I think you are referring to the older method of transparent proxying.
Tproxy4 does not rely on conntrack. The socket is created with the
connection's original ports and IPs.
>
> You claim wrong lock is taken at insert time, but are you sure the right
> lock is taken at deletion time ?
>
> Hmm...
>
Without the patch, you have the same problem in __inet_put_port(). The
lock is taken based on the child's inet_num, but the icsk_bind_hash of
the socket was inherited from the parent, so it belongs to a different
inet_bind_hashbucket.
With the patch, inet_bind_hash refers to the inet_bind_bucket that was
found by searching the hash table, rather than directly inherited from
the parent. This means that the correct lock is chosen for the list
being manipulated.
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: 2.6.35, netfilter/iptables not working with 32bit userspace + 64bit kernel?
From: Jussi Kivilinna @ 2010-08-13 13:40 UTC (permalink / raw)
To: Florian Westphal; +Cc: Eric Dumazet, netdev, Netfilter Development Mailinglist
In-Reply-To: <20100813121955.GI15614@Chamillionaire.breakpoint.cc>
Thanks!
Is this patch going for stable?
-Jussi
Quoting "Florian Westphal" <fw@strlen.de>:
> Eric Dumazet <eric.dumazet@gmail.com> wrote:
>> CC netfilter-devel to get more people in touch
>>
>> Thanks !
>>
>> Le vendredi 13 août 2010 à 13:47 +0300, Jussi Kivilinna a écrit :
>> > Hello!
>> >
>> > I have server/firewall running Debian lenny with 32bit userspace and
>> > 64bit kernel. After upgrading from 2.6.34 to 2.6.35, I couldn't get
>> > any new chains work. Simply doing (with 32bit iptables)
>> >
>> > iptables -N new_chain
>> > iptables -A OUTPUT -j new_chain
>> > iptables -A OUTPUT -j ACCEPT
>> >
>> > cause output to freeze. "iptables -L -vn" shows:
>> >
>> > Chain OUTPUT (policy ACCEPT 3397 packets, 637K bytes)
>> > pkts bytes target prot opt in out source destination
>> > 3 252 new_chain all -- * * 0.0.0.0/0 0.0.0.0/0
>> > 0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
>> >
>> > Ping shows "ping: sendmsg: Operation not permitted".
>> >
>> > With recompiled 64bit iptables, example above works fine.
>
> You need this patch:
>
> commit f3c5c1bfd430858d3a05436f82c51e53104feb6b
> (netfilter: xtables: make ip_tables reentrant) forgot to
> also compute the jumpstack size in the compat handlers.
>
> Result is that "iptables -I INPUT -j userchain" turns into -j DROP.
>
> Reported by Sebastian Roesner on #netfilter, closes
> http://bugzilla.netfilter.org/show_bug.cgi?id=669.
>
> Note: arptables change is compile-tested only.
>
> Signed-off-by: Florian Westphal <fw@strlen.de>
> ---
> net/ipv4/netfilter/arp_tables.c | 3 +++
> net/ipv4/netfilter/ip_tables.c | 3 +++
> net/ipv6/netfilter/ip6_tables.c | 3 +++
> 3 files changed, 9 insertions(+), 0 deletions(-)
>
> diff --git a/net/ipv4/netfilter/arp_tables.c
> b/net/ipv4/netfilter/arp_tables.c
> index 6bccba3..4829766 100644
> --- a/net/ipv4/netfilter/arp_tables.c
> +++ b/net/ipv4/netfilter/arp_tables.c
> @@ -1418,6 +1418,9 @@ static int translate_compat_table(const char *name,
> if (ret != 0)
> break;
> ++i;
> + if (strcmp(arpt_get_target(iter1)->u.user.name,
> + XT_ERROR_TARGET) == 0)
> + ++newinfo->stacksize;
> }
> if (ret) {
> /*
> diff --git a/net/ipv4/netfilter/ip_tables.c b/net/ipv4/netfilter/ip_tables.c
> index c439721..3d3b695 100644
> --- a/net/ipv4/netfilter/ip_tables.c
> +++ b/net/ipv4/netfilter/ip_tables.c
> @@ -1749,6 +1749,9 @@ translate_compat_table(struct net *net,
> if (ret != 0)
> break;
> ++i;
> + if (strcmp(ipt_get_target(iter1)->u.user.name,
> + XT_ERROR_TARGET) == 0)
> + ++newinfo->stacksize;
> }
> if (ret) {
> /*
> diff --git a/net/ipv6/netfilter/ip6_tables.c
> b/net/ipv6/netfilter/ip6_tables.c
> index 5359ef4..bad3c7f 100644
> --- a/net/ipv6/netfilter/ip6_tables.c
> +++ b/net/ipv6/netfilter/ip6_tables.c
> @@ -1764,6 +1764,9 @@ translate_compat_table(struct net *net,
> if (ret != 0)
> break;
> ++i;
> + if (strcmp(ip6t_get_target(iter1)->u.user.name,
> + XT_ERROR_TARGET) == 0)
> + ++newinfo->stacksize;
> }
> if (ret) {
> /*
> --
> 1.7.1
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Problem with non aligned DMA in usbnet on ARM
From: David Brownell @ 2010-08-13 13:42 UTC (permalink / raw)
To: mfuzzey-Re5JQEeQqe8AvxtiuMwx3w, Oliver Neukum
Cc: Matthieu CASTET, Russell King - ARM Linux, Gary King,
linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org,
linux-usb, netdev-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Greg KH
In-Reply-To: <201008131258.55016.oneukum-l3A5Bk7waGM@public.gmane.org>
> Subject: Re: Problem with non aligned DMA in usbnet on ARM
I remain unconvinced the bug is anywhere except
in whatever host controller is rejecting the
buffers it's given.
Does anyone have proof the bug is elsewhere?
Think for a moment what chaos the kernel would be
if arbitrary subsystems were allowed to introduce
random DMA alignment "requirements". It would not
be possible to pass buffers between subsystems with
any success at all...
> > 1) Require that usb drivers submit buffers obtained
> from kmalloc() and > friends
Requirements are already documented with the
Description of what memory is DMA-able. It's
not kmalloc() specifically that's required.
with no extra offsets.
Offsets have nothing to do with being DMA-able.
They might have to do with working around hardware
flaws though. (ISTR PXA255 also had a 32-bit goof,
making its DMA engine unusable for most purposes.)
> > 2) Require that usb_submit_urb() accept byte aligned buffers.
I believe we already require that. Of course,
the requirement is on HCDs, not on that call.
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Problem with non aligned DMA in usbnet on ARM
From: Oliver Neukum @ 2010-08-13 13:53 UTC (permalink / raw)
To: David Brownell
Cc: mfuzzey-Re5JQEeQqe8AvxtiuMwx3w, Matthieu CASTET,
Russell King - ARM Linux, Gary King,
linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org,
linux-usb, netdev-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Greg KH
In-Reply-To: <732137.54230.qm-g47maUHHHF/6X00i2u5GFvu2YVrzzGjVVpNB7YpNyf8@public.gmane.org>
Am Freitag, 13. August 2010, 15:42:32 schrieb David Brownell:
> > > 2) Require that usb_submit_urb() accept byte aligned buffers.
>
> I believe we already require that. Of course,
> the requirement is on HCDs, not on th
Divers fail if HCDs can't do it. But we've never explicitly said that HCDs
need to be able to do that. We should probably document that requirement.
Regards
Oliver
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: tproxy related crash in inet_hashtables
From: Eric Dumazet @ 2010-08-13 13:55 UTC (permalink / raw)
To: Stephen Buck
Cc: netdev@vger.kernel.org, Netfilter Development Mailinglist,
Patrick McHardy, KOVACS Krisztian
In-Reply-To: <4C65432F.4030809@exinda.com>
Le vendredi 13 août 2010 à 23:05 +1000, Stephen Buck a écrit :
> On 13/08/10 20:54, Eric Dumazet wrote:
> > Le vendredi 13 août 2010 à 18:15 +1000, Stephen Buck a écrit :
> >> Recently I encountered a number of crashes related to tproxy on the
> >> 2.6.34.1 (x86_64 SMP) kernel. These usually manifested as a bug like the
> >> following (Although the bug was confirmed to be present on a vanilla
> >> kernel, this particular trace is from a kernel with some customisations):
> >>
> >> [ 1504.765077] BUG: unable to handle kernel NULL pointer dereference at
> >> (null)
> >> [ 1504.848183] IP: [<ffffffff8135a79b>] inet_bind_bucket_destroy+0x1b/0x30
> >> [ 1504.927126] PGD 1a9933067 PUD 1ad909067 PMD 0
> >> [ 1504.980125] Thread overran stack, or stack corrupted
> >> [ 1505.039325] Oops: 0002 #1 SMP
> >> [ 1505.077775] last sysfs file:
> >> /sys/devices/system/cpu/cpu15/topology/thread_siblings
> >> [ 1505.169166] CPU 0
> >> [ 1505.193070] Modules linked in: sch_sfq cls_fw sch_htb xt_physdev
> >> 8021q bridge stp llc
> >> [ 1505.923769] Pid: 0, comm: swapper Not tainted 2.6.31-12EXINDAsmp #0
> >> PowerEdge R710
> >> [ 1506.014118] RIP: 0010:[<ffffffff8135a79b>] [<ffffffff8135a79b>]
> >> inet_bind_bucket_destroy+0x1b/0x30
> >> [ 1506.122242] RSP: 0018:ffffc90000003e10 EFLAGS: 00010246
> >> [ 1506.185655] RAX: 0000000000000000 RBX: ffffc900164a02a0 RCX:
> >> ffffea00098e24b0
> >> [ 1506.270863] RDX: 0000000000000000 RSI: ffff8802e1186280 RDI:
> >> ffffffff815b4600
> >> [ 1506.356077] RBP: ffffc90000003e10 R08: 0000000000000016 R09:
> >> 0000000000000001
> >> [ 1506.441284] R10: 0000000000000000 R11: 0000000000000000 R12:
> >> ffff88011b894fc0
> >> [ 1506.526500] R13: ffffffff81744c80 R14: ffffffff815fe6c0 R15:
> >> 0000000000000003
> >> [ 1506.611722] FS: 0000000000000000(0000) GS:ffffc90000000000(0000)
> >> knlGS:0000000000000000
> >> [ 1506.708410] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> >> [ 1506.777011] CR2: 0000000000000000 CR3: 00000001ae17a000 CR4:
> >> 00000000000006f0
> >> [ 1506.862225] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> >> 0000000000000000
> >> [ 1506.947442] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> >> 0000000000000400
> >> [ 1507.032650] Process swapper (pid: 0, threadinfo ffffffff815b0000,
> >> task ffffffff815b6bc0)
> >> [ 1507.129345] Stack:
> >> [ 1507.153299] ffffc90000003e40 ffffffff8135b834 0000000000000001
> >> ffff88011b894fc0
> >> [ 1507.239581]<0> ffffffff815fe560 0000000000000002 ffffc90000003e80
> >> ffffffff8135bbf6
> >> [ 1507.331133]<0> ffffffff815fe560 ffffffff815fe560 ffffffff815fe6c0
> >> ffffc90000003eb0
> >> [ 1507.424833] Call Trace:
> >> [ 1507.453976]<IRQ>
> >> [ 1507.478971] [<ffffffff8135b834>] __inet_twsk_kill+0xb4/0xf0
> >> [ 1507.546538] [<ffffffff8135bbf6>] inet_twdr_do_twkill_work+0x66/0xd0
> >> [ 1507.622408] [<ffffffff8135bd40>] ? inet_twdr_hangman+0x0/0xd0
> >> [ 1507.692041] [<ffffffff8135bd95>] inet_twdr_hangman+0x55/0xd0
> >> [ 1507.760650] [<ffffffff810515ec>] run_timer_softirq+0x18c/0x220
> >> [ 1507.831330] [<ffffffff8104c3b8>] __do_softirq+0xc8/0x1f0
> >> [ 1507.895799] [<ffffffff8100cf5c>] call_softirq+0x1c/0x30
> >> [ 1507.959209] [<ffffffff8100e5f5>] do_softirq+0x45/0x80
> >> [ 1508.020538] [<ffffffff8104c2e7>] irq_exit+0x87/0x90
> >> [ 1508.079797] [<ffffffff813bd0c1>] smp_apic_timer_interrupt+0x71/0x9d
> >> [ 1508.155667] [<ffffffff8100c933>] apic_timer_interrupt+0x13/0x20
> >> [ 1508.227385]<EOI>
> >> [ 1508.252383] [<ffffffff8101317e>] ? mwait_idle+0x7e/0x110
> >> [ 1508.316836] [<ffffffff813bb0bd>] ? __atomic_notifier_call_chain+0xd/0x10
> >> [ 1508.397903] [<ffffffff813bb0d1>] ? atomic_notifier_call_chain+0x11/0x20
> >> [ 1508.477928] [<ffffffff8100ac31>] ? cpu_idle+0x51/0x90
> >> [ 1508.539265] [<ffffffff813aa92b>] ? rest_init+0x6b/0x80
> >> [ 1508.601639] [<ffffffff816230f5>] ? start_kernel+0x2c5/0x370
> >> [ 1508.669200] [<ffffffff81622611>] ? x86_64_start_reservations+0x81/0xc0
> >> [ 1508.748192] [<ffffffff81622726>] ? x86_64_start_kernel+0xd6/0x100
> >> [ 1508.821979] Code: 64 10 40 eb 94 0f 1f 44 00 00 66 0f 1f 44 00 00 55
> >> 48 89 e5 0f 1f 44 00 00 48 83 7e 18 00 75 19 48 8b 46 08 48 8b 56 10
> >> [ 1509.048123] RIP [<ffffffff8135a79b>] inet_bind_bucket_destroy+0x1b/0x30
> >> [ 1509.128157] RSP<ffffc90000003e10>
> >> [ 1509.169759] CR2: 0000000000000000
> >>
> >>
> >> After spending a while tracking it down, I discovered that the wrong
> >> locks get held when operating on the bind hash table's chains.
> >>
> >> This is due to the listen socket and the child socket having different
> >> local ports when __inet_inherit_port() is called. The lock is held based
> >> on the child socket's port, but the list operated on is the one the
> >> listen socket belongs to.
> >>
> >> e.g.
> >> There is a transparent proxy listening on port 9999.
> >> A new http connection (with port 80) is redirected to the proxy.
> >>
> >> The inet_bind_hashbucket locked in this case is table->bhash[80].lock,
> >> but the inet_bind_bucket the child socket is added to is in the chain of
> >> the table->bhash[9999] inet_bind_hashbucket. This means that if another
> >> connection with a different local port arrived and was redirected to the
> >> proxy, they could both be operating on the list at the same time.
> >>
> >>
> >> Attached is a patch that should fix this by looking up the correct
> >> inet_bind_bucket based on the child's local port when the
> >> inet_bind_bucket from the listen socket has a different port to the
> >> child's inet_num. It was built against 2.6.34.1, but should apply to any
> >> mainline kernel.
> >>
> >> It is also possible the same bug exists in the IPv6 code as well. As I
> >> have not had to deal with IPv6 yet, I have not had a look.
> > Hi Stephen
> >
> > CC netfilter-devel& Patrick& Krisztian
> >
> > I cannot convince myself this patch is a right fix.
> >
> > This probably should be fixed in netfilter tree, not in
> > net/ipv4/inet_hashtables.c ?
> >
> > Once tproxy is involved, the original port (80) should be changed to
> > 9999 by tproxy (for SYN packet) and conntrack for following ones.
> >
> > So listening socket and its children all use source port 9999 ?
> >
> > (inet_sk(child)->inet_num == inet_sk(parent)->inet_num)
> I think you are referring to the older method of transparent proxying.
> Tproxy4 does not rely on conntrack. The socket is created with the
> connection's original ports and IPs.
So in your case "netstat -atn" displays port 80 , not port 9999 for
children, and 9999 for the listener ?
> >
> > You claim wrong lock is taken at insert time, but are you sure the right
> > lock is taken at deletion time ?
> >
> > Hmm...
> >
> Without the patch, you have the same problem in __inet_put_port(). The
> lock is taken based on the child's inet_num, but the icsk_bind_hash of
> the socket was inherited from the parent, so it belongs to a different
> inet_bind_hashbucket.
>
> With the patch, inet_bind_hash refers to the inet_bind_bucket that was
> found by searching the hash table, rather than directly inherited from
> the parent. This means that the correct lock is chosen for the list
> being manipulated.
Really ?
So lock[80] is taken at insert time (as your patch doesnt change
locking), and lock[9999] taken at delete time ? How comes it doesnt
race ?
Look I am _very_ confused by this stuff.
How can you explain lock is taken on table->bhash[80].lock but your
patch adds a chain to table->bhash[9999] ? This is not what your patch
is trying to do. Or you should lock table->bhash[9999].lock
Please check the comment in front of inet_bind_bucket_create() :
/*
* Allocate and initialize a new local port bind bucket.
* The bindhash mutex for snum's hash chain must be held here.
*/
Maybe change this comment, because clearly "snum's hash" meaning is very
different after tproxy inclusion (and your patch) ?
Your patch Changelog is not clear enough, or very misleading ...
commit f517f61e37907ea66e48f224c135a9e1c257bb04
Author: Stephen Buck <stephen.buck@exinda.com>
Date: Thu Aug 12 16:19:19 2010 +1000
Fix locking issue in inet_hashtables when using tproxy
When __inet_inherit_port() is called on a tproxy connection the wrong locks
are held for the inet_bind_bucket it is added to.
Since your patch does nothing about the lock itself, but adds a
new struct inet_bind_bucket in case the parent one is not appropriate, and
another sibling is not yet present.
I suggest you include in ChangeLog _all_ your email, because I am
pretty sure nobody will understand the intent of your patch in 2011.
I am sure the full explanation is better than a very short log entry.
Thanks
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH net-next-2.6] netlink: netlink_recvmsg() fix
From: Johannes Berg @ 2010-08-13 14:00 UTC (permalink / raw)
To: David Miller; +Cc: eric.dumazet, netdev
In-Reply-To: <20100725.215548.112589000.davem@davemloft.net>
On Sun, 2010-07-25 at 21:55 -0700, David Miller wrote:
> From: Eric Dumazet <eric.dumazet@gmail.com>
> Date: Wed, 21 Jul 2010 10:43:55 +0200
>
> > [PATCH net-next-2.6 v3] netlink: netlink_recvmsg() fix
> >
> > commit 1dacc76d0014
> > (net/compat/wext: send different messages to compat tasks)
> > introduced a race condition on netlink, in case MSG_PEEK is used.
> >
> > An skb given by skb_recv_datagram() might be shared, we must copy it
> > before any modification, or risk fatal corruption.
> >
> > Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
>
> Applied, thanks Eric.
I keep getting errors like below in 2.6.35+wireless-testing. Not saying
that it's this patch's fault, but it is the only thing I remember
touching that area.
[10548.768754] =============================================================================
[10548.773474] BUG kmalloc-4096: Object already free
[10548.775342] -----------------------------------------------------------------------------
[10548.775344]
[10548.778505] INFO: Allocated in wireless_send_event+0x1f2/0x410 age=1 cpu=0 pid=28521
[10548.778505] INFO: Freed in skb_release_data+0xd0/0xe0 age=2 cpu=2 pid=27730
[10548.778505] INFO: Slab 0xffffea0003fa4b80 objects=7 used=1 fp=0xffff880122f12090 flags=0x80000000000040c3
[10548.778505] INFO: Object 0xffff880122f12090 @offset=8336 fp=0xffff880122f11048
[10548.778505]
[10548.778505] Bytes b4 0xffff880122f12080: 6c a3 0f 00 01 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a l�......ZZZZZZZZ
...
[10548.778505] Object 0xffff880122f13080: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk�
[10548.778505] Redzone 0xffff880122f13090: bb bb bb bb bb bb bb bb ��������
[10548.778505] Padding 0xffff880122f130d0: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
[10548.778505] Pid: 916, comm: avahi-daemon Tainted: G W 2.6.35-wl-67184-g1595c70-dirty #12
[10548.778505] Call Trace:
[10548.778505] [<ffffffff81133993>] print_trailer+0x103/0x160
[10548.778505] [<ffffffff81133a31>] object_err+0x41/0x50
[10548.778505] [<ffffffff81135b0f>] __slab_free+0x24f/0x3b0
[10548.778505] [<ffffffff81135da2>] kfree+0x132/0x1a0
[10548.778505] [<ffffffff81383710>] skb_release_data+0xd0/0xe0
[10548.778505] [<ffffffff813831be>] __kfree_skb+0x1e/0xa0
[10548.778505] [<ffffffff813832c2>] kfree_skb+0x42/0xb0
[10548.778505] [<ffffffff813ac3f2>] netlink_recvmsg+0x422/0x480
[10548.778505] [<ffffffff8137afdd>] sock_recvmsg+0xfd/0x130
[10548.778505] [<ffffffff8137c1e4>] __sys_recvmsg+0x144/0x2e0
[10548.778505] [<ffffffff8137c629>] sys_recvmsg+0x49/0x80
[10548.778505] [<ffffffff8100b132>] system_call_fastpath+0x16/0x1b
[10548.778505] FIX kmalloc-4096: Object at 0xffff880122f12090 not freed
johannes
^ permalink raw reply
* Re: [PATCH net-next-2.6] netlink: netlink_recvmsg() fix
From: Johannes Berg @ 2010-08-13 14:35 UTC (permalink / raw)
To: David Miller; +Cc: eric.dumazet, netdev
In-Reply-To: <1281708024.3760.12.camel@jlt3.sipsolutions.net>
On Fri, 2010-08-13 at 16:00 +0200, Johannes Berg wrote:
> On Sun, 2010-07-25 at 21:55 -0700, David Miller wrote:
> > From: Eric Dumazet <eric.dumazet@gmail.com>
> > Date: Wed, 21 Jul 2010 10:43:55 +0200
> >
> > > [PATCH net-next-2.6 v3] netlink: netlink_recvmsg() fix
> > >
> > > commit 1dacc76d0014
> > > (net/compat/wext: send different messages to compat tasks)
> > > introduced a race condition on netlink, in case MSG_PEEK is used.
> > >
> > > An skb given by skb_recv_datagram() might be shared, we must copy it
> > > before any modification, or risk fatal corruption.
> > >
> > > Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
> >
> > Applied, thanks Eric.
>
> I keep getting errors like below in 2.6.35+wireless-testing. Not saying
> that it's this patch's fault, but it is the only thing I remember
> touching that area.
In fact, I think something's wrong with this patch, since my comment
(that right now unfortunately I no longer fully understand) says:
* If this skb has a frag_list, then here that means that
* we will have to use the frag_list skb for compat tasks
* and the regular skb for non-compat tasks.
*
* The skb might (and likely will) be cloned, so we can't
* just reset frag_list and go on with things -- we need to
* keep that. For the compat case that's easy -- simply get
* a reference to the compat skb and free the regular one
* including the frag. For the non-compat case, we need to
* avoid sending the frag to the user -- so assign NULL but
* restore it below before freeing the skb.
and that's no longer true, afaict.
johannes
^ permalink raw reply
* Re: [PATCH] netpoll: use non-BH variant of RCU
From: Herbert Xu @ 2010-08-13 14:39 UTC (permalink / raw)
To: Paul E. McKenney; +Cc: David Miller, linville, netdev
In-Reply-To: <20100812154213.GB2524@linux.vnet.ibm.com>
On Thu, Aug 12, 2010 at 08:42:13AM -0700, Paul E. McKenney wrote:
>
> +/**
> + * rcu_read_lock_bh_irqsoff() - mark the beginning of an RCU-bh critical section
> + *
> + * This is equivalent of rcu_read_lock_bh(), but to be used where the
> + * caller either is in an irq handler or has irqs disabled. Note that
> + * this function assumes that PREEMPT_RT kernels run irq handlers at
> + * higher priority than softirq handlers!
> + */
> +static inline void rcu_read_lock_bh_irqsoff(void)
> +{
> + rcu_read_unlock_bh_irqsoff_check();
> + __acquire(RCU_BH);
> + rcu_read_acquire_bh();
> +}
Thanks for the patch Paul!
But this doesn't really solve the problem for netif_rx. The reason
is that netif_rx can either be called with IRQs on OR off. So we
need to take the right precautions in the case where IRQs are
enabled along with BH.
Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH net-next-2.6] netlink: netlink_recvmsg() fix
From: Eric Dumazet @ 2010-08-13 14:48 UTC (permalink / raw)
To: Johannes Berg; +Cc: David Miller, netdev
In-Reply-To: <1281710115.3760.27.camel@jlt3.sipsolutions.net>
Le vendredi 13 août 2010 à 16:35 +0200, Johannes Berg a écrit :
> On Fri, 2010-08-13 at 16:00 +0200, Johannes Berg wrote:
> > On Sun, 2010-07-25 at 21:55 -0700, David Miller wrote:
> > > From: Eric Dumazet <eric.dumazet@gmail.com>
> > > Date: Wed, 21 Jul 2010 10:43:55 +0200
> > >
> > > > [PATCH net-next-2.6 v3] netlink: netlink_recvmsg() fix
> > > >
> > > > commit 1dacc76d0014
> > > > (net/compat/wext: send different messages to compat tasks)
> > > > introduced a race condition on netlink, in case MSG_PEEK is used.
> > > >
> > > > An skb given by skb_recv_datagram() might be shared, we must copy it
> > > > before any modification, or risk fatal corruption.
> > > >
> > > > Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
> > >
> > > Applied, thanks Eric.
> >
> > I keep getting errors like below in 2.6.35+wireless-testing. Not saying
> > that it's this patch's fault, but it is the only thing I remember
> > touching that area.
>
> In fact, I think something's wrong with this patch, since my comment
> (that right now unfortunately I no longer fully understand) says:
>
> * If this skb has a frag_list, then here that means that
> * we will have to use the frag_list skb for compat tasks
> * and the regular skb for non-compat tasks.
> *
> * The skb might (and likely will) be cloned, so we can't
> * just reset frag_list and go on with things -- we need to
> * keep that. For the compat case that's easy -- simply get
> * a reference to the compat skb and free the regular one
> * including the frag. For the non-compat case, we need to
> * avoid sending the frag to the user -- so assign NULL but
> * restore it below before freeing the skb.
>
> and that's no longer true, afaict.
>
Comment was not updated by the patch.
But do you agree temporarly setting frag_list to NULL was a bug ?
Unfortunatly I cannot test this path...
I assume reverting 1235f504aaba removes these errors ?
Its strange we have a double-free on a data part and not a skb_head.
maybe pskb_copy() has a problem with frag_list...
Maybe we can revert the patch and find another way to make sure two
process can not manipulate this skb in // (adding a mutex, or using
RTNL ?)
^ permalink raw reply
* Re: [PATCH net-next-2.6] netlink: netlink_recvmsg() fix
From: Johannes Berg @ 2010-08-13 15:13 UTC (permalink / raw)
To: Eric Dumazet; +Cc: David Miller, netdev
In-Reply-To: <1281710887.4470.78.camel@edumazet-laptop>
On Fri, 2010-08-13 at 16:48 +0200, Eric Dumazet wrote:
> > * If this skb has a frag_list, then here that means that
> > * we will have to use the frag_list skb for compat tasks
> > * and the regular skb for non-compat tasks.
> > *
> > * The skb might (and likely will) be cloned, so we can't
> > * just reset frag_list and go on with things -- we need to
> > * keep that. For the compat case that's easy -- simply get
> > * a reference to the compat skb and free the regular one
> > * including the frag. For the non-compat case, we need to
> > * avoid sending the frag to the user -- so assign NULL but
> > * restore it below before freeing the skb.
> >
> > and that's no longer true, afaict.
> >
>
> Comment was not updated by the patch.
>
> But do you agree temporarly setting frag_list to NULL was a bug ?
No, that was actually intentional, as the comment describes?
> Unfortunatly I cannot test this path...
No wireless devices? :)
> I assume reverting 1235f504aaba removes these errors ?
I haven't tried yet, but it only happened very recently and I didn't
find any other candidate -- the error always points to
wireless_send_event too.
> Its strange we have a double-free on a data part and not a skb_head.
There also was an apparent use-after-free error.
> maybe pskb_copy() has a problem with frag_list...
>
>
> Maybe we can revert the patch and find another way to make sure two
> process can not manipulate this skb in // (adding a mutex, or using
> RTNL ?)
I'm getting confused. Why did I think the NULLing and then restoring was
not racy to start with?
johannes
^ permalink raw reply
* Re: [PATCH] Fix deadlock between boomerang_interrupt and boomerang_start_tx in 3c59x
From: Herbert Xu @ 2010-08-13 15:15 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: nhorman, netdev
In-Reply-To: <20100811120900.3629706c@s6510>
Stephen Hemminger <shemminger@vyatta.com> wrote:
>
> I thought we agreed that any device supporting netconsole agrees
> to not call printk in the transmit path. Just kill the pr_* call.
Not calling printk from the driver doesn't help since you can
still call printk from an IRQ handler.
Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* [PATCH v2 2/3] cxgb4i v5: iscsi driver
From: kxie @ 2010-08-13 15:19 UTC (permalink / raw)
To: netdev, linux-scsi, open-iscsi
Cc: rranjan, kxie, James.Bottomley, michaelc, davem
[PATCH v2 2/3] cxgb4i v5: iscsi driver
From: Karen Xie <kxie@chelsio.com>
Added cxgb4i iSCSI driver.
This patch implements the cxgb4i iscsi connection acceleration for the
open-iscsi initiator.
The cxgb4i driver offers the iscsi PDU based offload:
- digest insertion and verification
- payload direct-placement into host memory buffer.
Signed-off-by: Karen Xie <kxie@chelsio.com>
---
drivers/scsi/Kconfig | 1
drivers/scsi/Makefile | 1
drivers/scsi/cxgbi/Kconfig | 1
drivers/scsi/cxgbi/Makefile | 1
drivers/scsi/cxgbi/cxgb4i/Kbuild | 3
drivers/scsi/cxgbi/cxgb4i/Kconfig | 7
drivers/scsi/cxgbi/cxgb4i/cxgb4i.c | 1622 ++++++++++++++++++++++++++++++++++++
drivers/scsi/cxgbi/cxgb4i/cxgb4i.h | 38 +
8 files changed, 1674 insertions(+), 0 deletions(-)
create mode 100644 drivers/scsi/cxgbi/Kconfig
create mode 100644 drivers/scsi/cxgbi/Makefile
create mode 100644 drivers/scsi/cxgbi/cxgb4i/Kbuild
create mode 100644 drivers/scsi/cxgbi/cxgb4i/Kconfig
create mode 100644 drivers/scsi/cxgbi/cxgb4i/cxgb4i.c
create mode 100644 drivers/scsi/cxgbi/cxgb4i/cxgb4i.h
diff --git a/drivers/scsi/Kconfig b/drivers/scsi/Kconfig
index a479b3b..efe66bb 100644
--- a/drivers/scsi/Kconfig
+++ b/drivers/scsi/Kconfig
@@ -379,6 +379,7 @@ config ISCSI_BOOT_SYSFS
say Y. Otherwise, say N.
source "drivers/scsi/cxgb3i/Kconfig"
+source "drivers/scsi/cxgbi/Kconfig"
source "drivers/scsi/bnx2i/Kconfig"
source "drivers/scsi/be2iscsi/Kconfig"
diff --git a/drivers/scsi/Makefile b/drivers/scsi/Makefile
index 2703c6e..cb31f8c 100644
--- a/drivers/scsi/Makefile
+++ b/drivers/scsi/Makefile
@@ -134,6 +134,7 @@ obj-$(CONFIG_SCSI_STEX) += stex.o
obj-$(CONFIG_SCSI_MVSAS) += mvsas/
obj-$(CONFIG_PS3_ROM) += ps3rom.o
obj-$(CONFIG_SCSI_CXGB3_ISCSI) += libiscsi.o libiscsi_tcp.o cxgb3i/
+obj-$(CONFIG_SCSI_CXGB4_ISCSI) += libiscsi.o libiscsi_tcp.o cxgbi/
obj-$(CONFIG_SCSI_BNX2_ISCSI) += libiscsi.o bnx2i/
obj-$(CONFIG_BE2ISCSI) += libiscsi.o be2iscsi/
obj-$(CONFIG_SCSI_PMCRAID) += pmcraid.o
diff --git a/drivers/scsi/cxgbi/Kconfig b/drivers/scsi/cxgbi/Kconfig
new file mode 100644
index 0000000..a470e38
--- /dev/null
+++ b/drivers/scsi/cxgbi/Kconfig
@@ -0,0 +1 @@
+source "drivers/scsi/cxgbi/cxgb4i/Kconfig"
diff --git a/drivers/scsi/cxgbi/Makefile b/drivers/scsi/cxgbi/Makefile
new file mode 100644
index 0000000..9e8f604
--- /dev/null
+++ b/drivers/scsi/cxgbi/Makefile
@@ -0,0 +1 @@
+obj-$(CONFIG_SCSI_CXGB4_ISCSI) += libcxgbi.o cxgb4i/
diff --git a/drivers/scsi/cxgbi/cxgb4i/Kbuild b/drivers/scsi/cxgbi/cxgb4i/Kbuild
new file mode 100644
index 0000000..b9f4af7
--- /dev/null
+++ b/drivers/scsi/cxgbi/cxgb4i/Kbuild
@@ -0,0 +1,3 @@
+EXTRA_CFLAGS += -I$(srctree)/drivers/net/cxgb4
+
+obj-$(CONFIG_SCSI_CXGB4_ISCSI) += cxgb4i.o
diff --git a/drivers/scsi/cxgbi/cxgb4i/Kconfig b/drivers/scsi/cxgbi/cxgb4i/Kconfig
new file mode 100644
index 0000000..bb94b39
--- /dev/null
+++ b/drivers/scsi/cxgbi/cxgb4i/Kconfig
@@ -0,0 +1,7 @@
+config SCSI_CXGB4_ISCSI
+ tristate "Chelsio T4 iSCSI support"
+ depends on CHELSIO_T4_DEPENDS
+ select CHELSIO_T4
+ select SCSI_ISCSI_ATTRS
+ ---help---
+ This driver supports iSCSI offload for the Chelsio T4 devices.
diff --git a/drivers/scsi/cxgbi/cxgb4i/cxgb4i.c b/drivers/scsi/cxgbi/cxgb4i/cxgb4i.c
new file mode 100644
index 0000000..b97ff13
--- /dev/null
+++ b/drivers/scsi/cxgbi/cxgb4i/cxgb4i.c
@@ -0,0 +1,1622 @@
+/*
+ * cxgb4i.c: Chelsio T4 iSCSI driver.
+ *
+ * Copyright (c) 2010 Chelsio Communications, Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation.
+ *
+ * Written by: Karen Xie (kxie@chelsio.com)
+ * Rakesh Ranjan (rranjan@chelsio.com)
+ */
+
+#include <linux/version.h>
+#include <linux/module.h>
+#include <linux/moduleparam.h>
+#include <scsi/scsi_host.h>
+#include <net/tcp.h>
+#include <net/dst.h>
+#include <linux/netdevice.h>
+
+#include "t4_msg.h"
+#include "cxgb4.h"
+#include "cxgb4_uld.h"
+#include "t4fw_api.h"
+#include "l2t.h"
+#include "cxgb4i.h"
+
+static unsigned int dbg_level;
+#include "../libcxgbi.h"
+
+#define DRV_MODULE_NAME "cxgb4i"
+#define DRV_MODULE_DESC "Chelsio T4 iSCSI Driver"
+#define DRV_MODULE_VERSION "0.9.0"
+#define DRV_MODULE_RELDATE "May 2010"
+
+static char version[] =
+ DRV_MODULE_DESC " " DRV_MODULE_NAME
+ " v" DRV_MODULE_VERSION " (" DRV_MODULE_RELDATE ")\n";
+
+MODULE_AUTHOR("Chelsio Communications, Inc.");
+MODULE_DESCRIPTION(DRV_MODULE_DESC);
+MODULE_VERSION(DRV_MODULE_VERSION);
+MODULE_LICENSE("GPL");
+
+module_param(dbg_level, uint, 0644);
+MODULE_PARM_DESC(dbg_level, "Debug flag (default=0)");
+
+static int cxgb4i_rcv_win = 256 * 1024;
+module_param(cxgb4i_rcv_win, int, 0644);
+MODULE_PARM_DESC(cxgb4i_rcv_win, "TCP reveive window in bytes");
+
+static int cxgb4i_snd_win = 128 * 1024;
+module_param(cxgb4i_snd_win, int, 0644);
+MODULE_PARM_DESC(cxgb4i_snd_win, "TCP send window in bytes");
+
+static int cxgb4i_rx_credit_thres = 10 * 1024;
+module_param(cxgb4i_rx_credit_thres, int, 0644);
+MODULE_PARM_DESC(cxgb4i_rx_credit_thres,
+ "RX credits return threshold in bytes (default=10KB)");
+
+static unsigned int cxgb4i_max_connect = (8 * 1024);
+module_param(cxgb4i_max_connect, uint, 0644);
+MODULE_PARM_DESC(cxgb4i_max_connect, "Maximum number of connections");
+
+static unsigned short cxgb4i_sport_base = 20000;
+module_param(cxgb4i_sport_base, ushort, 0644);
+MODULE_PARM_DESC(cxgb4i_sport_base, "Starting port number (default 20000)");
+
+#define cxgb4i_log_error cxgbi_dbg_error
+#define cxgb4i_log_warn cxgbi_dbg_warn
+#define cxgb4i_log_info cxgbi_dbg_info
+#define cxgb4i_log_debug(level, fmt, args...) \
+ cxgbi_dbg_print(level, fmt, ## args)
+
+typedef void (*cxgb4i_cplhandler_func)(struct cxgbi_device *, struct sk_buff *);
+
+static void *t4_uld_add(const struct cxgb4_lld_info *);
+static int t4_uld_rx_handler(void *, const __be64 *, const struct pkt_gl *);
+static int t4_uld_state_change(void *, enum cxgb4_state state);
+
+static const struct cxgb4_uld_info cxgb4i_uld_info = {
+ .name = DRV_MODULE_NAME,
+ .add = t4_uld_add,
+ .rx_handler = t4_uld_rx_handler,
+ .state_change = t4_uld_state_change,
+};
+
+static struct scsi_host_template cxgb4i_host_template = {
+ .module = THIS_MODULE,
+ .name = DRV_MODULE_NAME,
+ .proc_name = DRV_MODULE_NAME,
+ .can_queue = CXGB4I_SCSI_HOST_QDEPTH,
+ .queuecommand = iscsi_queuecommand,
+ .change_queue_depth = iscsi_change_queue_depth,
+ .sg_tablesize = SG_ALL,
+ .max_sectors = 0xFFFF,
+ .cmd_per_lun = ISCSI_DEF_CMD_PER_LUN,
+ .eh_abort_handler = iscsi_eh_abort,
+ .eh_device_reset_handler = iscsi_eh_device_reset,
+ .eh_target_reset_handler = iscsi_eh_recover_target,
+ .target_alloc = iscsi_target_alloc,
+ .use_clustering = DISABLE_CLUSTERING,
+ .this_id = -1,
+};
+
+static struct iscsi_transport cxgb4i_iscsi_transport = {
+ .owner = THIS_MODULE,
+ .name = DRV_MODULE_NAME,
+ .caps = CAP_RECOVERY_L0 | CAP_MULTI_R2T | CAP_HDRDGST |
+ CAP_DATADGST | CAP_DIGEST_OFFLOAD |
+ CAP_PADDING_OFFLOAD,
+ .param_mask = ISCSI_MAX_RECV_DLENGTH | ISCSI_MAX_XMIT_DLENGTH |
+ ISCSI_HDRDGST_EN | ISCSI_DATADGST_EN |
+ ISCSI_INITIAL_R2T_EN | ISCSI_MAX_R2T |
+ ISCSI_IMM_DATA_EN | ISCSI_FIRST_BURST |
+ ISCSI_MAX_BURST | ISCSI_PDU_INORDER_EN |
+ ISCSI_DATASEQ_INORDER_EN | ISCSI_ERL |
+ ISCSI_CONN_PORT | ISCSI_CONN_ADDRESS |
+ ISCSI_EXP_STATSN | ISCSI_PERSISTENT_PORT |
+ ISCSI_PERSISTENT_ADDRESS |
+ ISCSI_TARGET_NAME | ISCSI_TPGT |
+ ISCSI_USERNAME | ISCSI_PASSWORD |
+ ISCSI_USERNAME_IN | ISCSI_PASSWORD_IN |
+ ISCSI_FAST_ABORT | ISCSI_ABORT_TMO |
+ ISCSI_LU_RESET_TMO | ISCSI_TGT_RESET_TMO |
+ ISCSI_PING_TMO | ISCSI_RECV_TMO |
+ ISCSI_IFACE_NAME | ISCSI_INITIATOR_NAME,
+ .host_param_mask = ISCSI_HOST_HWADDRESS | ISCSI_HOST_IPADDRESS |
+ ISCSI_HOST_INITIATOR_NAME |
+ ISCSI_HOST_NETDEV_NAME,
+ .get_host_param = cxgbi_get_host_param,
+ .set_host_param = cxgbi_set_host_param,
+ /* session management */
+ .create_session = cxgbi_create_session,
+ .destroy_session = cxgbi_destroy_session,
+ .get_session_param = iscsi_session_get_param,
+ /* connection management */
+ .create_conn = cxgbi_create_conn,
+ .bind_conn = cxgbi_bind_conn,
+ .destroy_conn = iscsi_tcp_conn_teardown,
+ .start_conn = iscsi_conn_start,
+ .stop_conn = iscsi_conn_stop,
+ .get_conn_param = cxgbi_get_conn_param,
+ .set_param = cxgbi_set_conn_param,
+ .get_stats = cxgbi_get_conn_stats,
+ /* pdu xmit req from user space */
+ .send_pdu = iscsi_conn_send_pdu,
+ /* task */
+ .init_task = iscsi_tcp_task_init,
+ .xmit_task = iscsi_tcp_task_xmit,
+ .cleanup_task = cxgbi_cleanup_task,
+ /* pdu */
+ .alloc_pdu = cxgbi_conn_alloc_pdu,
+ .init_pdu = cxgbi_conn_init_pdu,
+ .xmit_pdu = cxgbi_conn_xmit_pdu,
+ .parse_pdu_itt = cxgbi_parse_pdu_itt,
+ /* TCP connect/disconnect */
+ .ep_connect = cxgbi_ep_connect,
+ .ep_poll = cxgbi_ep_poll,
+ .ep_disconnect = cxgbi_ep_disconnect,
+ /* Error recovery timeout call */
+ .session_recovery_timedout = iscsi_session_recovery_timedout,
+};
+
+static struct scsi_transport_template *cxgb4i_stt;
+
+/*
+ * CPL (Chelsio Protocol Language) defines a message passing interface between
+ * the host driver and Chelsio asic.
+ * The section below implments CPLs that related to iscsi tcp connection
+ * open/close/abort and data send/receive.
+ */
+#define DIV_ROUND_UP(n, d) (((n) + (d) - 1) / (d))
+#define RCV_BUFSIZ_MASK 0x3FFU
+#define MAX_IMM_TX_PKT_LEN 128
+
+static inline void set_queue(struct sk_buff *skb, unsigned int queue,
+ const struct cxgbi_sock *csk)
+{
+ skb->queue_mapping = queue;
+}
+
+static int push_tx_frames(struct cxgbi_sock *, int);
+
+/*
+ * is_ofld_imm - check whether a packet can be sent as immediate data
+ * @skb: the packet
+ *
+ * Returns true if a packet can be sent as an offload WR with immediate
+ * data. We currently use the same limit as for Ethernet packets.
+ */
+static inline int is_ofld_imm(const struct sk_buff *skb)
+{
+ return skb->len <= (MAX_IMM_TX_PKT_LEN -
+ sizeof(struct fw_ofld_tx_data_wr));
+}
+
+static void send_act_open_req(struct cxgbi_sock *csk, struct sk_buff *skb,
+ struct l2t_entry *e)
+{
+ struct cpl_act_open_req *req;
+ int wscale = cxgbi_sock_compute_wscale(csk->mss_idx);
+ unsigned long long opt0;
+ unsigned int opt2;
+ unsigned int qid_atid = ((unsigned int)csk->atid) |
+ (((unsigned int)csk->rss_qid) << 14);
+
+ opt0 = KEEP_ALIVE(1) |
+ WND_SCALE(wscale) |
+ MSS_IDX(csk->mss_idx) |
+ L2T_IDX(((struct l2t_entry *)csk->l2t)->idx) |
+ TX_CHAN(csk->tx_chan) |
+ SMAC_SEL(csk->smac_idx) |
+ ULP_MODE(ULP_MODE_ISCSI) |
+ RCV_BUFSIZ(cxgb4i_rcv_win >> 10);
+ opt2 = RX_CHANNEL(0) |
+ RSS_QUEUE_VALID |
+ (1 << 20) | (1 << 22) |
+ RSS_QUEUE(csk->rss_qid);
+
+ set_wr_txq(skb, CPL_PRIORITY_SETUP, csk->port_id);
+ req = (struct cpl_act_open_req *)skb->head;
+
+ INIT_TP_WR(req, 0);
+ OPCODE_TID(req) = cpu_to_be32(MK_OPCODE_TID(CPL_ACT_OPEN_REQ,
+ qid_atid));
+ req->local_port = csk->saddr.sin_port;
+ req->peer_port = csk->daddr.sin_port;
+ req->local_ip = csk->saddr.sin_addr.s_addr;
+ req->peer_ip = csk->daddr.sin_addr.s_addr;
+ req->opt0 = cpu_to_be64(opt0);
+ req->params = 0;
+ req->opt2 = cpu_to_be32(opt2);
+
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p, %pI4:%u-%pI4:%u, atid %d, qid %u.\n",
+ csk, &req->local_ip, ntohs(req->local_port),
+ &req->peer_ip, ntohs(req->peer_port),
+ csk->atid, csk->rss_qid);
+
+ cxgb4_l2t_send(csk->cdev->ports[csk->port_id], skb, csk->l2t);
+}
+
+static void send_close_req(struct cxgbi_sock *csk)
+{
+ struct sk_buff *skb = csk->cpl_close;
+ struct cpl_close_con_req *req = (struct cpl_close_con_req *)skb->head;
+ unsigned int tid = csk->tid;
+
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx,%u.\n",
+ csk, csk->state, csk->flags, csk->tid);
+
+ csk->cpl_close = NULL;
+ set_wr_txq(skb, CPL_PRIORITY_DATA, csk->port_id);
+ INIT_TP_WR(req, tid);
+ OPCODE_TID(req) = cpu_to_be32(MK_OPCODE_TID(CPL_CLOSE_CON_REQ, tid));
+ req->rsvd = 0;
+
+ cxgbi_sock_skb_entail(csk, skb);
+ if (csk->state >= CTP_ESTABLISHED)
+ push_tx_frames(csk, 1);
+}
+
+static void abort_arp_failure(void *handle, struct sk_buff *skb)
+{
+ struct cxgbi_sock *csk = (struct cxgbi_sock *)handle;
+ struct cpl_abort_req *req;
+
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx, tid %u, abort.\n",
+ csk, csk->state, csk->flags, csk->tid);
+ req = (struct cpl_abort_req *)skb->data;
+ req->cmd = CPL_ABORT_NO_RST;
+ cxgb4_ofld_send(csk->cdev->ports[csk->port_id], skb);
+}
+
+static void send_abort_req(struct cxgbi_sock *csk)
+{
+ struct cpl_abort_req *req;
+ struct sk_buff *skb = csk->cpl_abort_req;
+
+ if (unlikely(csk->state == CTP_ABORTING) || !skb || !csk->cdev)
+ return;
+ cxgbi_sock_set_state(csk, CTP_ABORTING);
+ cxgbi_sock_set_flag(csk, CTPF_ABORT_RPL_PENDING);
+ cxgbi_sock_purge_write_queue(csk);
+
+ csk->cpl_abort_req = NULL;
+ req = (struct cpl_abort_req *)skb->head;
+ set_queue(skb, CPL_PRIORITY_DATA, csk);
+ req->cmd = CPL_ABORT_SEND_RST;
+ t4_set_arp_err_handler(skb, csk, abort_arp_failure);
+ INIT_TP_WR(req, csk->tid);
+ OPCODE_TID(req) = cpu_to_be32(MK_OPCODE_TID(CPL_ABORT_REQ, csk->tid));
+ req->rsvd0 = htonl(csk->snd_nxt);
+ req->rsvd1 = !cxgbi_sock_flag(csk, CTPF_TX_DATA_SENT);
+
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx,%u, snd_nxt %u, 0x%x.\n",
+ csk, csk->state, csk->flags, csk->tid, csk->snd_nxt,
+ req->rsvd1);
+
+ cxgb4_l2t_send(csk->cdev->ports[csk->port_id], skb, csk->l2t);
+}
+
+static void send_abort_rpl(struct cxgbi_sock *csk, int rst_status)
+{
+ struct sk_buff *skb = csk->cpl_abort_rpl;
+ struct cpl_abort_rpl *rpl = (struct cpl_abort_rpl *)skb->head;
+
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx,%u, status %d.\n",
+ csk, csk->state, csk->flags, csk->tid, rst_status);
+
+ csk->cpl_abort_rpl = NULL;
+ set_queue(skb, CPL_PRIORITY_DATA, csk);
+ INIT_TP_WR(rpl, csk->tid);
+ OPCODE_TID(rpl) = cpu_to_be32(MK_OPCODE_TID(CPL_ABORT_RPL, csk->tid));
+ rpl->cmd = rst_status;
+ cxgb4_ofld_send(csk->cdev->ports[csk->port_id], skb);
+}
+
+/*
+ * CPL connection rx data ack: host ->
+ * Send RX credits through an RX_DATA_ACK CPL message. Returns the number of
+ * credits sent.
+ */
+static u32 send_rx_credits(struct cxgbi_sock *csk, u32 credits)
+{
+ struct sk_buff *skb;
+ struct cpl_rx_data_ack *req;
+
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_PDU_RX,
+ "csk 0x%p,%u,0x%lx,%u, credit %u.\n",
+ csk, csk->state, csk->flags, csk->tid, credits);
+
+ skb = alloc_cpl(sizeof(*req), 0, GFP_ATOMIC);
+ if (!skb) {
+ cxgb4i_log_info("csk 0x%p, credit %u, OOM.\n", csk, credits);
+ return 0;
+ }
+ req = (struct cpl_rx_data_ack *)skb->head;
+
+ set_wr_txq(skb, CPL_PRIORITY_ACK, csk->port_id);
+ INIT_TP_WR(req, csk->tid);
+ OPCODE_TID(req) = cpu_to_be32(MK_OPCODE_TID(CPL_RX_DATA_ACK,
+ csk->tid));
+ req->credit_dack = cpu_to_be32(RX_CREDITS(credits) | RX_FORCE_ACK(1));
+ cxgb4_ofld_send(csk->cdev->ports[csk->port_id], skb);
+ return credits;
+}
+
+/*
+ * sgl_len - calculates the size of an SGL of the given capacity
+ * @n: the number of SGL entries
+ * Calculates the number of flits needed for a scatter/gather list that
+ * can hold the given number of entries.
+ */
+static inline unsigned int sgl_len(unsigned int n)
+{
+ n--;
+ return (3 * n) / 2 + (n & 1) + 2;
+}
+
+/*
+ * calc_tx_flits_ofld - calculate # of flits for an offload packet
+ * @skb: the packet
+ *
+ * Returns the number of flits needed for the given offload packet.
+ * These packets are already fully constructed and no additional headers
+ * will be added.
+ */
+static inline unsigned int calc_tx_flits_ofld(const struct sk_buff *skb)
+{
+ unsigned int flits, cnt;
+
+ if (is_ofld_imm(skb))
+ return DIV_ROUND_UP(skb->len, 8);
+ flits = skb_transport_offset(skb) / 8;
+ cnt = skb_shinfo(skb)->nr_frags;
+ if (skb->tail != skb->transport_header)
+ cnt++;
+ return flits + sgl_len(cnt);
+}
+
+static inline void send_tx_flowc_wr(struct cxgbi_sock *csk)
+{
+ struct sk_buff *skb;
+ struct fw_flowc_wr *flowc;
+ int flowclen, i;
+
+ flowclen = 80;
+ skb = alloc_cpl(flowclen, 0, GFP_ATOMIC);
+ flowc = (struct fw_flowc_wr *)skb->head;
+ flowc->op_to_nparams =
+ htonl(FW_WR_OP(FW_FLOWC_WR) | FW_FLOWC_WR_NPARAMS(8));
+ flowc->flowid_len16 =
+ htonl(FW_WR_LEN16(DIV_ROUND_UP(flowclen, 16)) |
+ FW_WR_FLOWID(csk->tid));
+ flowc->mnemval[0].mnemonic = FW_FLOWC_MNEM_PFNVFN;
+ flowc->mnemval[0].val = htonl(0);
+ flowc->mnemval[1].mnemonic = FW_FLOWC_MNEM_CH;
+ flowc->mnemval[1].val = htonl(csk->tx_chan);
+ flowc->mnemval[2].mnemonic = FW_FLOWC_MNEM_PORT;
+ flowc->mnemval[2].val = htonl(csk->tx_chan);
+ flowc->mnemval[3].mnemonic = FW_FLOWC_MNEM_IQID;
+ flowc->mnemval[3].val = htonl(csk->rss_qid);
+ flowc->mnemval[4].mnemonic = FW_FLOWC_MNEM_SNDNXT;
+ flowc->mnemval[4].val = htonl(csk->snd_nxt);
+ flowc->mnemval[5].mnemonic = FW_FLOWC_MNEM_RCVNXT;
+ flowc->mnemval[5].val = htonl(csk->rcv_nxt);
+ flowc->mnemval[6].mnemonic = FW_FLOWC_MNEM_SNDBUF;
+ flowc->mnemval[6].val = htonl(cxgb4i_snd_win);
+ flowc->mnemval[7].mnemonic = FW_FLOWC_MNEM_MSS;
+ flowc->mnemval[7].val = htonl(csk->advmss);
+ flowc->mnemval[8].mnemonic = 0;
+ flowc->mnemval[8].val = 0;
+ for (i = 0; i < 9; i++) {
+ flowc->mnemval[i].r4[0] = 0;
+ flowc->mnemval[i].r4[1] = 0;
+ flowc->mnemval[i].r4[2] = 0;
+ }
+ set_queue(skb, CPL_PRIORITY_DATA, csk);
+
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p, tid 0x%x, %u,%u,%u,%u,%u,%u,%u.\n",
+ csk, csk->tid, 0, csk->tx_chan, csk->rss_qid,
+ csk->snd_nxt, csk->rcv_nxt, cxgb4i_snd_win,
+ csk->advmss);
+
+ cxgb4_ofld_send(csk->cdev->ports[csk->port_id], skb);
+}
+
+static inline void make_tx_data_wr(struct cxgbi_sock *csk, struct sk_buff *skb,
+ int dlen, int len, u32 credits, int compl)
+{
+ struct fw_ofld_tx_data_wr *req;
+ unsigned int submode = cxgbi_skcb_ulp_mode(skb) & 3;
+ unsigned int wr_ulp_mode = 0;
+
+ req = (struct fw_ofld_tx_data_wr *)__skb_push(skb, sizeof(*req));
+
+ if (is_ofld_imm(skb)) {
+ req->op_to_immdlen = htonl(FW_WR_OP(FW_OFLD_TX_DATA_WR) |
+ FW_WR_COMPL(1) |
+ FW_WR_IMMDLEN(dlen));
+ req->flowid_len16 = htonl(FW_WR_FLOWID(csk->tid) |
+ FW_WR_LEN16(credits));
+ } else {
+ req->op_to_immdlen =
+ cpu_to_be32(FW_WR_OP(FW_OFLD_TX_DATA_WR) |
+ FW_WR_COMPL(1) |
+ FW_WR_IMMDLEN(0));
+ req->flowid_len16 =
+ cpu_to_be32(FW_WR_FLOWID(csk->tid) |
+ FW_WR_LEN16(credits));
+ }
+ if (submode)
+ wr_ulp_mode = FW_OFLD_TX_DATA_WR_ULPMODE(ULP2_MODE_ISCSI) |
+ FW_OFLD_TX_DATA_WR_ULPSUBMODE(submode);
+ req->tunnel_to_proxy = htonl(wr_ulp_mode) |
+ FW_OFLD_TX_DATA_WR_SHOVE(skb_peek(&csk->write_queue) ? 0 : 1);
+ req->plen = htonl(len);
+ if (!cxgbi_sock_flag(csk, CTPF_TX_DATA_SENT))
+ cxgbi_sock_set_flag(csk, CTPF_TX_DATA_SENT);
+}
+
+static void arp_failure_skb_discard(void *handle, struct sk_buff *skb)
+{
+ kfree_skb(skb);
+}
+
+static int push_tx_frames(struct cxgbi_sock *csk, int req_completion)
+{
+ int total_size = 0;
+ struct sk_buff *skb;
+
+ if (unlikely(csk->state < CTP_ESTABLISHED ||
+ csk->state == CTP_CLOSE_WAIT_1 || csk->state >= CTP_ABORTING)) {
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK |
+ 1 << CXGBI_DBG_PDU_TX,
+ "csk 0x%p,%u,0x%lx,%u, in closing state.\n",
+ csk, csk->state, csk->flags, csk->tid);
+ return 0;
+ }
+
+ while (csk->wr_cred && (skb = skb_peek(&csk->write_queue)) != NULL) {
+ int dlen = skb->len;
+ int len = skb->len;
+ unsigned int credits_needed;
+
+ skb_reset_transport_header(skb);
+ if (is_ofld_imm(skb))
+ credits_needed = DIV_ROUND_UP(dlen +
+ sizeof(struct fw_ofld_tx_data_wr), 16);
+ else
+ credits_needed = DIV_ROUND_UP(8*calc_tx_flits_ofld(skb)
+ + sizeof(struct fw_ofld_tx_data_wr),
+ 16);
+
+ if (csk->wr_cred < credits_needed) {
+ cxgb4i_log_debug(1 << CXGBI_DBG_PDU_TX,
+ "csk 0x%p, skb %u/%u, wr %d < %u.\n",
+ csk, skb->len, skb->data_len,
+ credits_needed, csk->wr_cred);
+ break;
+ }
+ __skb_unlink(skb, &csk->write_queue);
+ set_queue(skb, CPL_PRIORITY_DATA, csk);
+ skb->csum = credits_needed;
+ csk->wr_cred -= credits_needed;
+ csk->wr_una_cred += credits_needed;
+ cxgbi_sock_enqueue_wr(csk, skb);
+
+ cxgb4i_log_debug(1 << CXGBI_DBG_PDU_TX,
+ "csk 0x%p, enqueue, skb len %u/%u, "
+ "wr %d, left %u, unack %u.\n",
+ csk, skb->len, skb->data_len,
+ credits_needed, csk->wr_cred,
+ csk->wr_una_cred);
+
+ if (likely(cxgbi_skcb_test_flag(skb, SKCBF_TX_NEED_HDR))) {
+ if (!cxgbi_sock_flag(csk, CTPF_TX_DATA_SENT)) {
+ send_tx_flowc_wr(csk);
+ skb->csum += 5;
+ csk->wr_cred -= 5;
+ csk->wr_una_cred += 5;
+ }
+ len += cxgbi_ulp_extra_len(cxgbi_skcb_ulp_mode(skb));
+ make_tx_data_wr(csk, skb, dlen, len, credits_needed,
+ req_completion);
+ csk->snd_nxt += len;
+ cxgbi_skcb_clear_flag(skb, SKCBF_TX_NEED_HDR);
+ }
+ total_size += skb->truesize;
+ t4_set_arp_err_handler(skb, csk, arp_failure_skb_discard);
+
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_PDU_TX,
+ "csk 0x%p,%u,0x%lx,%u, skb 0x%p, %u.\n",
+ csk, csk->state, csk->flags, csk->tid, skb, len);
+
+ cxgb4_l2t_send(csk->cdev->ports[csk->port_id], skb, csk->l2t);
+ }
+ return total_size;
+}
+
+static inline void free_atid(struct cxgbi_sock *csk)
+{
+ struct cxgb4_lld_info *lldi = cxgbi_cdev_priv(csk->cdev);
+
+ if (cxgbi_sock_flag(csk, CTPF_HAS_ATID)) {
+ cxgb4_free_atid(lldi->tids, csk->atid);
+ cxgbi_sock_clear_flag(csk, CTPF_HAS_ATID);
+ cxgbi_sock_put(csk);
+ }
+}
+
+static void do_act_establish(struct cxgbi_device *cdev, struct sk_buff *skb)
+{
+ struct cxgbi_sock *csk;
+ struct cpl_act_establish *req = (struct cpl_act_establish *)skb->data;
+ unsigned short tcp_opt = ntohs(req->tcp_opt);
+ unsigned int tid = GET_TID(req);
+ unsigned int atid = GET_TID_TID(ntohl(req->tos_atid));
+ struct cxgb4_lld_info *lldi = cxgbi_cdev_priv(cdev);
+ struct tid_info *t = lldi->tids;
+ u32 rcv_isn = be32_to_cpu(req->rcv_isn);
+
+ csk = lookup_atid(t, atid);
+ if (unlikely(!csk)) {
+ cxgb4i_log_error("NO connection for atid %u, cdev 0x%p\n",
+ atid, cdev);
+ goto rel_skb;
+ }
+
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx, tid %u, atid %u, rseq %u.\n",
+ csk, csk->state, csk->flags, tid, atid, rcv_isn);
+
+ cxgbi_sock_get(csk);
+ csk->tid = tid;
+ cxgb4_insert_tid(lldi->tids, csk, tid);
+ cxgbi_sock_set_flag(csk, CTPF_HAS_TID);
+
+ free_atid(csk);
+
+ spin_lock_bh(&csk->lock);
+ if (unlikely(csk->state != CTP_ACTIVE_OPEN))
+ cxgb4i_log_info("csk 0x%p,%u,0x%lx,%u, got EST.\n",
+ csk, csk->state, csk->flags, csk->tid);
+
+ if (csk->retry_timer.function) {
+ del_timer(&csk->retry_timer);
+ csk->retry_timer.function = NULL;
+ }
+
+ csk->copied_seq = csk->rcv_wup = csk->rcv_nxt = rcv_isn;
+ /*
+ * Causes the first RX_DATA_ACK to supply any Rx credits we couldn't
+ * pass through opt0.
+ */
+ if (cxgb4i_rcv_win > (RCV_BUFSIZ_MASK << 10))
+ csk->rcv_wup -= cxgb4i_rcv_win - (RCV_BUFSIZ_MASK << 10);
+
+ csk->advmss = lldi->mtus[GET_TCPOPT_MSS(tcp_opt)] - 40;
+ if (GET_TCPOPT_TSTAMP(tcp_opt))
+ csk->advmss -= 12;
+ if (csk->advmss < 128)
+ csk->advmss = 128;
+
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p, mss_idx %u, advmss %u.\n",
+ csk, GET_TCPOPT_MSS(tcp_opt), csk->advmss);
+
+ cxgbi_sock_established(csk, ntohl(req->snd_isn), ntohs(req->tcp_opt));
+
+ if (unlikely(cxgbi_sock_flag(csk, CTPF_ACTIVE_CLOSE_NEEDED)))
+ send_abort_req(csk);
+ else {
+ if (skb_queue_len(&csk->write_queue))
+ push_tx_frames(csk, 0);
+ cxgbi_conn_tx_open(csk);
+ }
+ spin_unlock_bh(&csk->lock);
+
+rel_skb:
+ __kfree_skb(skb);
+}
+
+static int act_open_rpl_status_to_errno(int status)
+{
+ switch (status) {
+ case CPL_ERR_CONN_RESET:
+ return -ECONNREFUSED;
+ case CPL_ERR_ARP_MISS:
+ return -EHOSTUNREACH;
+ case CPL_ERR_CONN_TIMEDOUT:
+ return -ETIMEDOUT;
+ case CPL_ERR_TCAM_FULL:
+ return -ENOMEM;
+ case CPL_ERR_CONN_EXIST:
+ return -EADDRINUSE;
+ default:
+ return -EIO;
+ }
+}
+
+static void csk_act_open_retry_timer(unsigned long data)
+{
+ struct sk_buff *skb;
+ struct cxgbi_sock *csk = (struct cxgbi_sock *)data;
+
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx,%u.\n",
+ csk, csk->state, csk->flags, csk->tid);
+
+ cxgbi_sock_get(csk);
+ spin_lock_bh(&csk->lock);
+ skb = alloc_cpl(sizeof(struct cpl_act_open_req), 0, GFP_ATOMIC);
+ if (!skb)
+ cxgbi_sock_fail_act_open(csk, -ENOMEM);
+ else {
+ skb->sk = (struct sock *)csk;
+ t4_set_arp_err_handler(skb, csk,
+ cxgbi_sock_act_open_req_arp_failure);
+ send_act_open_req(csk, skb, csk->l2t);
+ }
+ spin_unlock_bh(&csk->lock);
+ cxgbi_sock_put(csk);
+}
+
+static void do_act_open_rpl(struct cxgbi_device *cdev, struct sk_buff *skb)
+{
+ struct cxgbi_sock *csk;
+ struct cpl_act_open_rpl *rpl = (struct cpl_act_open_rpl *)skb->data;
+ unsigned int tid = GET_TID(rpl);
+ unsigned int atid =
+ GET_TID_TID(GET_AOPEN_ATID(be32_to_cpu(rpl->atid_status)));
+ unsigned int status = GET_AOPEN_STATUS(be32_to_cpu(rpl->atid_status));
+ struct cxgb4_lld_info *lldi = cxgbi_cdev_priv(cdev);
+ struct tid_info *t = lldi->tids;
+
+ csk = lookup_atid(t, atid);
+ if (unlikely(!csk)) {
+ cxgb4i_log_error("NO matching conn. atid %u, tid %u.\n",
+ atid, tid);
+ goto rel_skb;
+ }
+
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx, status %u, atid %u, tid %u.\n",
+ csk, csk->state, csk->flags, status, atid, tid);
+
+ if (status && status != CPL_ERR_TCAM_FULL &&
+ status != CPL_ERR_CONN_EXIST &&
+ status != CPL_ERR_ARP_MISS)
+ cxgb4_remove_tid(lldi->tids, csk->port_id, GET_TID(rpl));
+
+ cxgbi_sock_get(csk);
+ spin_lock_bh(&csk->lock);
+
+ if (status == CPL_ERR_CONN_EXIST &&
+ csk->retry_timer.function != csk_act_open_retry_timer) {
+ csk->retry_timer.function = csk_act_open_retry_timer;
+ mod_timer(&csk->retry_timer, jiffies + HZ / 2);
+ } else
+ cxgbi_sock_fail_act_open(csk,
+ act_open_rpl_status_to_errno(status));
+
+ spin_unlock_bh(&csk->lock);
+ cxgbi_sock_put(csk);
+rel_skb:
+ __kfree_skb(skb);
+}
+
+static void do_peer_close(struct cxgbi_device *cdev, struct sk_buff *skb)
+{
+ struct cxgbi_sock *csk;
+ struct cpl_peer_close *req = (struct cpl_peer_close *)skb->data;
+ unsigned int tid = GET_TID(req);
+ struct cxgb4_lld_info *lldi = cxgbi_cdev_priv(cdev);
+ struct tid_info *t = lldi->tids;
+
+ csk = lookup_tid(t, tid);
+ if (unlikely(!csk)) {
+ cxgb4i_log_error("can't find connection for tid %u\n", tid);
+ goto rel_skb;
+ }
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx,%u.\n",
+ csk, csk->state, csk->flags, csk->tid);
+ cxgbi_sock_rcv_peer_close(csk);
+rel_skb:
+ __kfree_skb(skb);
+}
+
+static void do_close_con_rpl(struct cxgbi_device *cdev, struct sk_buff *skb)
+{
+ struct cxgbi_sock *csk;
+ struct cpl_close_con_rpl *rpl = (struct cpl_close_con_rpl *)skb->data;
+ unsigned int tid = GET_TID(rpl);
+ struct cxgb4_lld_info *lldi = cxgbi_cdev_priv(cdev);
+ struct tid_info *t = lldi->tids;
+
+ csk = lookup_tid(t, tid);
+ if (unlikely(!csk)) {
+ cxgb4i_log_error("can't find connection for tid %u\n", tid);
+ goto rel_skb;
+ }
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx,%u.\n",
+ csk, csk->state, csk->flags, csk->tid);
+ cxgbi_sock_rcv_close_conn_rpl(csk, ntohl(rpl->snd_nxt));
+rel_skb:
+ __kfree_skb(skb);
+}
+
+static int abort_status_to_errno(struct cxgbi_sock *csk, int abort_reason,
+ int *need_rst)
+{
+ switch (abort_reason) {
+ case CPL_ERR_BAD_SYN: /* fall through */
+ case CPL_ERR_CONN_RESET:
+ return csk->state > CTP_ESTABLISHED ?
+ -EPIPE : -ECONNRESET;
+ case CPL_ERR_XMIT_TIMEDOUT:
+ case CPL_ERR_PERSIST_TIMEDOUT:
+ case CPL_ERR_FINWAIT2_TIMEDOUT:
+ case CPL_ERR_KEEPALIVE_TIMEDOUT:
+ return -ETIMEDOUT;
+ default:
+ return -EIO;
+ }
+}
+
+static void do_abort_req_rss(struct cxgbi_device *cdev, struct sk_buff *skb)
+{
+ struct cxgbi_sock *csk;
+ struct cpl_abort_req_rss *req = (struct cpl_abort_req_rss *)skb->data;
+ unsigned int tid = GET_TID(req);
+ struct cxgb4_lld_info *lldi = cxgbi_cdev_priv(cdev);
+ struct tid_info *t = lldi->tids;
+ int rst_status = CPL_ABORT_NO_RST;
+
+ csk = lookup_tid(t, tid);
+ if (unlikely(!csk)) {
+ cxgb4i_log_error("can't find connection for tid %u\n", tid);
+ goto rel_skb;
+ }
+
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx, tid %u, status 0x%x.\n",
+ csk, csk->state, csk->flags, csk->tid, req->status);
+
+ if (req->status == CPL_ERR_RTX_NEG_ADVICE ||
+ req->status == CPL_ERR_PERSIST_NEG_ADVICE)
+ goto rel_skb;
+
+ cxgbi_sock_get(csk);
+ spin_lock_bh(&csk->lock);
+
+ if (!cxgbi_sock_flag(csk, CTPF_ABORT_REQ_RCVD)) {
+ cxgbi_sock_set_flag(csk, CTPF_ABORT_REQ_RCVD);
+ cxgbi_sock_set_state(csk, CTP_ABORTING);
+ goto done;
+ }
+
+ cxgbi_sock_clear_flag(csk, CTPF_ABORT_REQ_RCVD);
+ send_abort_rpl(csk, rst_status);
+
+ if (!cxgbi_sock_flag(csk, CTPF_ABORT_RPL_PENDING)) {
+ csk->err = abort_status_to_errno(csk, req->status, &rst_status);
+ cxgbi_sock_closed(csk);
+ }
+done:
+ spin_unlock_bh(&csk->lock);
+ cxgbi_sock_put(csk);
+rel_skb:
+ __kfree_skb(skb);
+}
+
+static void do_abort_rpl_rss(struct cxgbi_device *cdev, struct sk_buff *skb)
+{
+ struct cxgbi_sock *csk;
+ struct cpl_abort_rpl_rss *rpl = (struct cpl_abort_rpl_rss *)skb->data;
+ unsigned int tid = GET_TID(rpl);
+ struct cxgb4_lld_info *lldi = cxgbi_cdev_priv(cdev);
+ struct tid_info *t = lldi->tids;
+
+ csk = lookup_tid(t, tid);
+ if (!csk)
+ goto rel_skb;
+
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "status 0x%x, csk 0x%p, s %u, 0x%lx.\n",
+ rpl->status, csk, csk ? csk->state : 0,
+ csk ? csk->flags : 0UL);
+
+ if (rpl->status == CPL_ERR_ABORT_FAILED)
+ goto rel_skb;
+
+ cxgbi_sock_rcv_abort_rpl(csk);
+rel_skb:
+ __kfree_skb(skb);
+}
+
+static void do_rx_iscsi_hdr(struct cxgbi_device *cdev, struct sk_buff *skb)
+{
+ struct cxgbi_sock *csk;
+ struct cpl_iscsi_hdr *cpl = (struct cpl_iscsi_hdr *)skb->data;
+ unsigned short pdu_len_ddp = be16_to_cpu(cpl->pdu_len_ddp);
+ unsigned int tid = GET_TID(cpl);
+ struct cxgb4_lld_info *lldi = cxgbi_cdev_priv(cdev);
+ struct tid_info *t = lldi->tids;
+ struct sk_buff *lskb;
+
+ csk = lookup_tid(t, tid);
+ if (unlikely(!csk)) {
+ cxgb4i_log_error("can't find conn. for tid %u\n", tid);
+ goto rel_skb;
+ }
+
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_PDU_RX,
+ "csk 0x%p,%u,0x%lx, tid %u, skb 0x%p,%u, 0x%x.\n",
+ csk, csk->state, csk->flags, csk->tid, skb, skb->len,
+ pdu_len_ddp);
+
+ spin_lock_bh(&csk->lock);
+
+ if (unlikely(csk->state >= CTP_PASSIVE_CLOSE)) {
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx,%u, bad state.\n",
+ csk, csk->state, csk->flags, csk->tid);
+ if (csk->state != CTP_ABORTING)
+ goto abort_conn;
+ else
+ goto discard;
+ }
+
+ cxgbi_skcb_tcp_seq(skb) = ntohl(cpl->seq);
+ skb_reset_transport_header(skb);
+ __skb_pull(skb, sizeof(*cpl));
+ __pskb_trim(skb, ntohs(cpl->len));
+
+ if (!csk->skb_ulp_lhdr) {
+ unsigned char *bhs;
+ unsigned int hlen, dlen;
+
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_PDU_RX,
+ "csk 0x%p,%u,0x%lx, tid %u, skb 0x%p header.\n",
+ csk, csk->state, csk->flags, csk->tid, skb);
+ csk->skb_ulp_lhdr = skb;
+ lskb = csk->skb_ulp_lhdr;
+ cxgbi_skcb_set_flag(lskb, SKCBF_RX_HDR);
+
+ if (cxgbi_skcb_tcp_seq(lskb) != csk->rcv_nxt) {
+ cxgb4i_log_info("tid 0x%x, CPL_ISCSI_HDR, bad seq got "
+ "0x%x, exp 0x%x\n",
+ csk->tid, cxgbi_skcb_tcp_seq(lskb),
+ csk->rcv_nxt);
+ goto abort_conn;
+ }
+
+ bhs = lskb->data;
+ hlen = ntohs(cpl->len);
+ dlen = ntohl(*(unsigned int *)(bhs + 4)) & 0xFFFFFF;
+
+ if ((hlen + dlen) != ISCSI_PDU_LEN(pdu_len_ddp) - 40) {
+ cxgb4i_log_info("tid 0x%x, CPL_ISCSI_HDR, pdu len "
+ "mismatch %u != %u + %u, seq 0x%x\n",
+ csk->tid,
+ ISCSI_PDU_LEN(pdu_len_ddp) - 40,
+ hlen, dlen, cxgbi_skcb_tcp_seq(skb));
+ goto abort_conn;
+ }
+
+ cxgbi_skcb_rx_pdulen(skb) = (hlen + dlen + 3) & (~0x3);
+ if (dlen)
+ cxgbi_skcb_rx_pdulen(skb) += csk->dcrc_len;
+ csk->rcv_nxt += cxgbi_skcb_rx_pdulen(skb);
+
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_PDU_RX,
+ "csk 0x%p, skb 0x%p, 0x%x,%u+%u,0x%x,0x%x.\n",
+ csk, skb, *bhs, hlen, dlen,
+ ntohl(*((unsigned int *)(bhs + 16))),
+ ntohl(*((unsigned int *)(bhs + 24))));
+
+ } else {
+ lskb = csk->skb_ulp_lhdr;
+ cxgbi_skcb_set_flag(lskb, SKCBF_RX_DATA);
+
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_PDU_RX,
+ "csk 0x%p,%u,0x%lx, skb 0x%p data, 0x%p.\n",
+ csk, csk->state, csk->flags, skb, lskb);
+ }
+
+ __skb_queue_tail(&csk->receive_queue, skb);
+ spin_unlock_bh(&csk->lock);
+ return;
+
+abort_conn:
+ send_abort_req(csk);
+discard:
+ spin_unlock_bh(&csk->lock);
+rel_skb:
+ __kfree_skb(skb);
+}
+
+static void do_rx_data_ddp(struct cxgbi_device *cdev,
+ struct sk_buff *skb)
+{
+ struct cxgbi_sock *csk;
+ struct sk_buff *lskb;
+ struct cpl_rx_data_ddp *rpl = (struct cpl_rx_data_ddp *)skb->data;
+ unsigned int tid = GET_TID(rpl);
+ struct cxgb4_lld_info *lldi = cxgbi_cdev_priv(cdev);
+ struct tid_info *t = lldi->tids;
+ unsigned int status = ntohl(rpl->ddpvld);
+
+ csk = lookup_tid(t, tid);
+ if (unlikely(!csk)) {
+ cxgb4i_log_error("can't find connection for tid %u\n", tid);
+ goto rel_skb;
+ }
+
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_PDU_RX,
+ "csk 0x%p,%u,0x%lx, skb 0x%p,0x%x, lhdr 0x%p.\n",
+ csk, csk->state, csk->flags, skb, status,
+ csk->skb_ulp_lhdr);
+
+ spin_lock_bh(&csk->lock);
+
+ if (unlikely(csk->state >= CTP_PASSIVE_CLOSE)) {
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx,%u, bad state.\n",
+ csk, csk->state, csk->flags, csk->tid);
+ if (csk->state != CTP_ABORTING)
+ goto abort_conn;
+ else
+ goto discard;
+ }
+
+ if (!csk->skb_ulp_lhdr) {
+ cxgb4i_log_error("tid 0x%x, rcv RX_DATA_DDP w/o pdu header\n",
+ csk->tid);
+ goto abort_conn;
+ }
+
+ lskb = csk->skb_ulp_lhdr;
+ csk->skb_ulp_lhdr = NULL;
+
+ cxgbi_skcb_set_flag(lskb, SKCBF_RX_STATUS);
+ cxgbi_skcb_rx_ddigest(lskb) = ntohl(rpl->ulp_crc);
+
+ if (ntohs(rpl->len) != cxgbi_skcb_rx_pdulen(lskb)) {
+ cxgb4i_log_info("tid 0x%x, RX_DATA_DDP pdulen %u != %u.\n",
+ csk->tid, ntohs(rpl->len),
+ cxgbi_skcb_rx_pdulen(lskb));
+ }
+
+ if (status & (1 << CPL_RX_DDP_STATUS_HCRC_SHIFT)) {
+ cxgb4i_log_debug(1 << CXGBI_DBG_PDU_RX,
+ "csk 0x%p, lhdr 0x%p, status 0x%x, hcrc bad.\n",
+ csk, lskb, status);
+ cxgbi_skcb_set_flag(lskb, SKCBF_RX_HCRC_ERR);
+ }
+ if (status & (1 << CPL_RX_DDP_STATUS_DCRC_SHIFT)) {
+ cxgb4i_log_debug(1 << CXGBI_DBG_PDU_RX,
+ "csk 0x%p, lhdr 0x%p, status 0x%x, dcrc bad.\n",
+ csk, lskb, status);
+ cxgbi_skcb_set_flag(lskb, SKCBF_RX_DCRC_ERR);
+ }
+ if (status & (1 << CPL_RX_DDP_STATUS_PAD_SHIFT)) {
+ cxgb4i_log_debug(1 << CXGBI_DBG_PDU_RX,
+ "csk 0x%p, lhdr 0x%p, status 0x%x, pad bad.\n",
+ csk, lskb, status);
+ cxgbi_skcb_set_flag(lskb, SKCBF_RX_PAD_ERR);
+ }
+ if ((status & (1 << CPL_RX_DDP_STATUS_DDP_SHIFT)) &&
+ !cxgbi_skcb_test_flag(lskb, SKCBF_RX_DATA)) {
+ cxgb4i_log_debug(1 << CXGBI_DBG_PDU_RX,
+ "csk 0x%p, lhdr 0x%p, 0x%x, data ddp'ed.\n",
+ csk, lskb, status);
+ cxgbi_skcb_set_flag(lskb, SKCBF_RX_DATA_DDPD);
+ }
+ cxgb4i_log_debug(1 << CXGBI_DBG_PDU_RX,
+ "csk 0x%p, lskb 0x%p, f 0x%lx.\n",
+ csk, lskb, cxgbi_skcb_flags(lskb));
+
+ cxgbi_conn_pdu_ready(csk);
+ spin_unlock_bh(&csk->lock);
+ goto rel_skb;
+
+abort_conn:
+ send_abort_req(csk);
+discard:
+ spin_unlock_bh(&csk->lock);
+rel_skb:
+ __kfree_skb(skb);
+}
+
+static void do_fw4_ack(struct cxgbi_device *cdev, struct sk_buff *skb)
+{
+ struct cxgbi_sock *csk;
+ struct cpl_fw4_ack *rpl = (struct cpl_fw4_ack *)skb->data;
+ unsigned int tid = GET_TID(rpl);
+ struct cxgb4_lld_info *lldi = cxgbi_cdev_priv(cdev);
+ struct tid_info *t = lldi->tids;
+
+ csk = lookup_tid(t, tid);
+ if (unlikely(!csk))
+ cxgb4i_log_error("can't find connection for tid %u\n", tid);
+ else {
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx,%u.\n",
+ csk, csk->state, csk->flags, csk->tid);
+ cxgbi_sock_rcv_wr_ack(csk, rpl->credits, ntohl(rpl->snd_una),
+ rpl->seq_vld);
+ }
+ __kfree_skb(skb);
+}
+
+static void do_set_tcb_rpl(struct cxgbi_device *cdev, struct sk_buff *skb)
+{
+ struct cpl_set_tcb_rpl *rpl = (struct cpl_set_tcb_rpl *)skb->data;
+ unsigned int tid = GET_TID(rpl);
+ struct cxgb4_lld_info *lldi = cxgbi_cdev_priv(cdev);
+ struct tid_info *t = lldi->tids;
+ struct cxgbi_sock *csk;
+
+ csk = lookup_tid(t, tid);
+ if (!csk)
+ cxgb4i_log_error("can't find conn. for tid %u\n", tid);
+
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,%lx,%u, status 0x%x.\n",
+ csk, csk->state, csk->flags, csk->tid, rpl->status);
+
+ if (rpl->status != CPL_ERR_NONE)
+ cxgb4i_log_error("csk 0x%p,%u, SET_TCB_RPL status %u.\n",
+ csk, tid, rpl->status);
+
+ __kfree_skb(skb);
+}
+
+static int alloc_cpls(struct cxgbi_sock *csk)
+{
+ csk->cpl_close = alloc_cpl(sizeof(struct cpl_close_con_req),
+ 0, GFP_NOIO);
+ if (!csk->cpl_close)
+ return -ENOMEM;
+
+ csk->cpl_abort_req = alloc_cpl(sizeof(struct cpl_abort_req),
+ 0, GFP_NOIO);
+ if (!csk->cpl_abort_req)
+ goto free_cpls;
+
+ csk->cpl_abort_rpl = alloc_cpl(sizeof(struct cpl_abort_rpl),
+ 0, GFP_NOIO);
+ if (!csk->cpl_abort_rpl)
+ goto free_cpls;
+ return 0;
+
+free_cpls:
+ cxgbi_sock_free_cpl_skbs(csk);
+ return -ENOMEM;
+}
+
+static inline void l2t_put(struct cxgbi_sock *csk)
+{
+ if (csk->l2t) {
+ cxgb4_l2t_release(csk->l2t);
+ csk->l2t = NULL;
+ cxgbi_sock_put(csk);
+ }
+}
+
+static void release_offload_resources(struct cxgbi_sock *csk)
+{
+ struct cxgb4_lld_info *lldi;
+
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx,%u.\n",
+ csk, csk->state, csk->flags, csk->tid);
+
+ cxgbi_sock_free_cpl_skbs(csk);
+ if (csk->wr_cred != csk->wr_max_cred) {
+ cxgbi_sock_purge_wr_queue(csk);
+ cxgbi_sock_reset_wr_list(csk);
+ }
+
+ l2t_put(csk);
+ if (cxgbi_sock_flag(csk, CTPF_HAS_ATID))
+ free_atid(csk);
+ else if (cxgbi_sock_flag(csk, CTPF_HAS_TID)) {
+ lldi = cxgbi_cdev_priv(csk->cdev);
+ cxgb4_remove_tid(lldi->tids, 0, csk->tid);
+ cxgbi_sock_clear_flag(csk, CTPF_HAS_TID);
+ cxgbi_sock_put(csk);
+ }
+ csk->dst = NULL;
+ csk->cdev = NULL;
+}
+
+static int init_act_open(struct cxgbi_sock *csk)
+{
+ struct cxgbi_device *cdev = csk->cdev;
+ struct cxgb4_lld_info *lldi = cxgbi_cdev_priv(cdev);
+ struct net_device *ndev = cdev->ports[csk->port_id];
+ struct port_info *pi = netdev_priv(ndev);
+ struct sk_buff *skb = NULL;
+ unsigned int step;
+
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx,%u.\n",
+ csk, csk->state, csk->flags, csk->tid);
+
+ csk->atid = cxgb4_alloc_atid(lldi->tids, csk);
+ if (csk->atid < 0) {
+ cxgb4i_log_error("%s, NO atid available.\n", ndev->name);
+ return -EINVAL;
+ }
+ cxgbi_sock_set_flag(csk, CTPF_HAS_ATID);
+ cxgbi_sock_get(csk);
+
+ csk->l2t = cxgb4_l2t_get(lldi->l2t, csk->dst->neighbour, ndev, 0);
+ if (!csk->l2t) {
+ cxgb4i_log_error("%s, cannot alloc l2t.\n", ndev->name);
+ goto rel_resource;
+ }
+ cxgbi_sock_get(csk);
+
+ skb = alloc_cpl(sizeof(struct cpl_act_open_req), 0, GFP_NOIO);
+ if (!skb)
+ goto rel_resource;
+ skb->sk = (struct sock *)csk;
+ t4_set_arp_err_handler(skb, csk, cxgbi_sock_act_open_req_arp_failure);
+
+ if (!csk->mtu)
+ csk->mtu = dst_mtu(csk->dst);
+ cxgb4_best_mtu(lldi->mtus, csk->mtu, &csk->mss_idx);
+ csk->tx_chan = cxgb4_port_chan(ndev);
+ /* SMT two entries per row */
+ csk->smac_idx = ((cxgb4_port_viid(ndev) & 0x7F)) << 1;
+ step = lldi->ntxq / lldi->nchan;
+ csk->txq_idx = cxgb4_port_idx(ndev) * step;
+ step = lldi->nrxq / lldi->nchan;
+ csk->rss_qid = lldi->rxq_ids[cxgb4_port_idx(ndev) * step];
+ csk->wr_max_cred = csk->wr_cred = lldi->wr_cred;
+ csk->wr_una_cred = 0;
+ cxgbi_sock_reset_wr_list(csk);
+ csk->err = 0;
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,p%d,%s, %u,%u,%u, mss %u,%u, smac %u.\n",
+ csk, pi->port_id, ndev->name, csk->tx_chan,
+ csk->txq_idx, csk->rss_qid, csk->mtu, csk->mss_idx,
+ csk->smac_idx);
+
+ cxgbi_sock_set_state(csk, CTP_ACTIVE_OPEN);
+ send_act_open_req(csk, skb, csk->l2t);
+ return 0;
+
+rel_resource:
+ if (skb)
+ __kfree_skb(skb);
+ return -EINVAL;
+}
+
+cxgb4i_cplhandler_func cxgb4i_cplhandlers[NUM_CPL_CMDS] = {
+ [CPL_ACT_ESTABLISH] = do_act_establish,
+ [CPL_ACT_OPEN_RPL] = do_act_open_rpl,
+ [CPL_PEER_CLOSE] = do_peer_close,
+ [CPL_ABORT_REQ_RSS] = do_abort_req_rss,
+ [CPL_ABORT_RPL_RSS] = do_abort_rpl_rss,
+ [CPL_CLOSE_CON_RPL] = do_close_con_rpl,
+ [CPL_FW4_ACK] = do_fw4_ack,
+ [CPL_ISCSI_HDR] = do_rx_iscsi_hdr,
+ [CPL_SET_TCB_RPL] = do_set_tcb_rpl,
+ [CPL_RX_DATA_DDP] = do_rx_data_ddp,
+};
+
+int cxgb4i_ofld_init(struct cxgbi_device *cdev)
+{
+ int rc;
+
+ if (cxgb4i_max_connect > CXGB4I_MAX_CONN)
+ cxgb4i_max_connect = CXGB4I_MAX_CONN;
+
+ rc = cxgbi_device_portmap_create(cdev, cxgb4i_sport_base,
+ cxgb4i_max_connect);
+ if (rc < 0)
+ return rc;
+
+ cdev->csk_release_offload_resources = release_offload_resources;
+ cdev->csk_push_tx_frames = push_tx_frames;
+ cdev->csk_send_abort_req = send_abort_req;
+ cdev->csk_send_close_req = send_close_req;
+ cdev->csk_send_rx_credits = send_rx_credits;
+ cdev->csk_alloc_cpls = alloc_cpls;
+ cdev->csk_init_act_open = init_act_open;
+
+ cxgb4i_log_info("cdev 0x%p, offload up, added.\n", cdev);
+ return 0;
+}
+
+/*
+ * functions to program the pagepod in h/w
+ */
+static inline void ulp_mem_io_set_hdr(struct ulp_mem_io *req,
+ unsigned int dlen, unsigned int pm_addr)
+{
+ struct ulptx_sgl *sgl;
+ unsigned int wr_len = roundup(sizeof(struct ulp_mem_io) +
+ sizeof(struct ulptx_sgl), 16);
+
+ INIT_ULPTX_WR(req, wr_len, 0, 0);
+ req->cmd = htonl(ULPTX_CMD(ULP_TX_MEM_WRITE));
+ req->dlen = htonl(ULP_MEMIO_DATA_LEN(dlen >> 5));
+ req->lock_addr = htonl(ULP_MEMIO_ADDR(pm_addr >> 5));
+ req->len16 = htonl(DIV_ROUND_UP(wr_len - sizeof(req->wr), 16));
+ sgl = (struct ulptx_sgl *)(req + 1);
+ sgl->cmd_nsge = htonl(ULPTX_CMD(ULP_TX_SC_DSGL) | ULPTX_NSGE(1));
+ sgl->len0 = htonl(dlen);
+}
+
+static int ddp_ppod_write_sgl(struct cxgbi_device *cdev, unsigned int port_id,
+ struct cxgbi_pagepod_hdr *hdr, unsigned int idx,
+ unsigned int npods,
+ struct cxgbi_gather_list *gl,
+ unsigned int gl_pidx)
+{
+ struct cxgbi_ddp_info *ddp = cdev->ddp;
+ unsigned int dlen, pm_addr;
+ struct sk_buff *skb;
+ struct ulp_mem_io *req;
+ struct ulptx_sgl *sgl;
+ struct cxgbi_pagepod *ppod;
+ unsigned int i;
+
+ dlen = PPOD_SIZE * npods;
+ pm_addr = idx * PPOD_SIZE + ddp->llimit;
+
+ skb = alloc_cpl(sizeof(*req) + sizeof(*sgl), dlen, GFP_ATOMIC);
+ if (!skb) {
+ cxgb4i_log_error("cdev 0x%p, idx %u, npods %u, OOM\n",
+ cdev, idx, npods);
+ return -ENOMEM;
+ }
+ req = (struct ulp_mem_io *)skb->head;
+ set_queue(skb, CPL_PRIORITY_CONTROL, NULL);
+
+ ulp_mem_io_set_hdr(req, dlen, pm_addr);
+ sgl = (struct ulptx_sgl *)(req + 1);
+ ppod = (struct cxgbi_pagepod *)(sgl + 1);
+ sgl->addr0 = cpu_to_be64(virt_to_phys(ppod));
+
+ for (i = 0; i < npods; i++, ppod++, gl_pidx += PPOD_PAGES_MAX) {
+ if (!hdr && !gl)
+ cxgbi_ddp_ppod_clear(ppod);
+ else
+ cxgbi_ddp_ppod_set(ppod, hdr, gl, gl_pidx);
+ }
+
+ cxgb4_ofld_send(cdev->ports[port_id], skb);
+ return 0;
+}
+
+static int ddp_set_map(struct cxgbi_sock *csk, struct cxgbi_pagepod_hdr *hdr,
+ unsigned int idx, unsigned int npods,
+ struct cxgbi_gather_list *gl)
+{
+ unsigned int i, cnt;
+ int err = 0;
+
+ for (i = 0; i < npods; i += cnt, idx += cnt) {
+ cnt = npods - i;
+ if (cnt > ULPMEM_DSGL_MAX_NPPODS)
+ cnt = ULPMEM_DSGL_MAX_NPPODS;
+ err = ddp_ppod_write_sgl(csk->cdev, csk->port_id, hdr,
+ idx, cnt, gl, 4 * i);
+ if (err < 0)
+ break;
+ }
+ return err;
+}
+
+static void ddp_clear_map(struct cxgbi_hba *chba, unsigned int tag,
+ unsigned int idx, unsigned int npods)
+{
+ unsigned int i, cnt;
+ int err;
+
+ for (i = 0; i < npods; i += cnt, idx += cnt) {
+ cnt = npods - i;
+ if (cnt > ULPMEM_DSGL_MAX_NPPODS)
+ cnt = ULPMEM_DSGL_MAX_NPPODS;
+ err = ddp_ppod_write_sgl(chba->cdev, chba->port_id, NULL,
+ idx, cnt, NULL, 0);
+ if (err < 0)
+ break;
+ }
+}
+
+static int ddp_setup_conn_pgidx(struct cxgbi_sock *csk, unsigned int tid,
+ int pg_idx, bool reply)
+{
+ struct sk_buff *skb;
+ struct cpl_set_tcb_field *req;
+ u64 val = pg_idx < DDP_PGIDX_MAX ? pg_idx : 0;
+
+ if (!pg_idx)
+ return 0;
+
+ skb = alloc_cpl(sizeof(*req), 0, GFP_KERNEL);
+ if (!skb)
+ return -ENOMEM;
+
+ /* set up ulp submode and page size */
+ val = (val & 0x03) << 2;
+ val |= TCB_ULP_TYPE(ULP2_MODE_ISCSI);
+
+ req = (struct cpl_set_tcb_field *)skb->head;
+ INIT_TP_WR(req, csk->tid);
+ OPCODE_TID(req) = htonl(MK_OPCODE_TID(CPL_SET_TCB_FIELD, csk->tid));
+ req->reply_ctrl = htons(NO_REPLY(reply) | QUEUENO(csk->rss_qid));
+ req->word_cookie = htons(TCB_WORD(W_TCB_ULP_RAW));
+ req->mask = cpu_to_be64(TCB_ULP_TYPE(TCB_ULP_TYPE_MASK));
+ req->val = cpu_to_be64(val);
+ set_wr_txq(skb, CPL_PRIORITY_CONTROL, csk->port_id);
+
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p, tid 0x%x, pg_idx %u.\n",
+ csk, csk->tid, pg_idx);
+
+ cxgb4_ofld_send(csk->cdev->ports[csk->port_id], skb);
+ return 0;
+}
+
+static int ddp_setup_conn_digest(struct cxgbi_sock *csk, unsigned int tid,
+ int hcrc, int dcrc, int reply)
+{
+ struct sk_buff *skb;
+ struct cpl_set_tcb_field *req;
+ u64 val = (hcrc ? ULP_CRC_HEADER : 0) | (dcrc ? ULP_CRC_DATA : 0);
+
+ val = TCB_ULP_RAW(val);
+ val |= TCB_ULP_TYPE(ULP2_MODE_ISCSI);
+
+ skb = alloc_cpl(sizeof(*req), 0, GFP_KERNEL);
+ if (!skb)
+ return -ENOMEM;
+
+ csk->hcrc_len = (hcrc ? 4 : 0);
+ csk->dcrc_len = (dcrc ? 4 : 0);
+ /* set up ulp submode and page size */
+ req = (struct cpl_set_tcb_field *)skb->head;
+ INIT_TP_WR(req, tid);
+ OPCODE_TID(req) = htonl(MK_OPCODE_TID(CPL_SET_TCB_FIELD, tid));
+ req->reply_ctrl = htons(NO_REPLY(reply) | QUEUENO(csk->rss_qid));
+ req->word_cookie = htons(TCB_WORD(W_TCB_ULP_RAW));
+ req->mask = cpu_to_be64(TCB_ULP_RAW(TCB_ULP_RAW_MASK));
+ req->val = cpu_to_be64(val);
+ set_wr_txq(skb, CPL_PRIORITY_CONTROL, csk->port_id);
+
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p, tid 0x%x, crc %d,%d.\n",
+ csk, csk->tid, hcrc, dcrc);
+
+ cxgb4_ofld_send(csk->cdev->ports[csk->port_id], skb);
+ return 0;
+}
+
+static int cxgb4i_ddp_init(struct cxgbi_device *cdev)
+{
+ struct cxgb4_lld_info *lldi = cxgbi_cdev_priv(cdev);
+ struct cxgbi_ddp_info *ddp = cdev->ddp;
+ unsigned int tagmask, pgsz_factor[4];
+ int err;
+
+ if (ddp) {
+ kref_get(&ddp->refcnt);
+ cxgb4i_log_warn("cdev 0x%p, ddp 0x%p already set up\n",
+ cdev, cdev->ddp);
+ return -EALREADY;
+ }
+
+ err = cxgbi_ddp_init(cdev, lldi->vr->iscsi.start,
+ lldi->vr->iscsi.start + lldi->vr->iscsi.size - 1,
+ lldi->iscsi_iolen, lldi->iscsi_iolen);
+ if (err < 0)
+ return err;
+
+ ddp = cdev->ddp;
+
+ tagmask = ddp->idx_mask << PPOD_IDX_SHIFT;
+ cxgbi_ddp_page_size_factor(pgsz_factor);
+ cxgb4_iscsi_init(lldi->ports[0], tagmask, pgsz_factor);
+
+ cdev->csk_ddp_free_gl_skb = NULL;
+ cdev->csk_ddp_alloc_gl_skb = NULL;
+ cdev->csk_ddp_setup_digest = ddp_setup_conn_digest;
+ cdev->csk_ddp_setup_pgidx = ddp_setup_conn_pgidx;
+ cdev->csk_ddp_set = ddp_set_map;
+ cdev->csk_ddp_clear = ddp_clear_map;
+
+ cxgb4i_log_info("cxgb4i 0x%p tag: sw %u, rsvd %u,%u, mask 0x%x.\n",
+ cdev, cdev->tag_format.sw_bits,
+ cdev->tag_format.rsvd_bits,
+ cdev->tag_format.rsvd_shift,
+ cdev->tag_format.rsvd_mask);
+ cxgb4i_log_info("cxgb4i 0x%p, nppods %u, bits %u, mask 0x%x,0x%x "
+ "pkt %u/%u, %u/%u\n",
+ cdev, ddp->nppods, ddp->idx_bits, ddp->idx_mask,
+ ddp->rsvd_tag_mask, ddp->max_txsz, lldi->iscsi_iolen,
+ ddp->max_rxsz, lldi->iscsi_iolen);
+ cxgb4i_log_info("cxgb4i 0x%p max payload size: %u/%u, %u/%u.\n",
+ cdev, cdev->tx_max_size, ddp->max_txsz,
+ cdev->rx_max_size, ddp->max_rxsz);
+ return 0;
+}
+
+static void *t4_uld_add(const struct cxgb4_lld_info *lldi)
+{
+ struct cxgbi_device *cdev;
+ struct port_info *pi;
+ int i, rc;
+
+ cxgb4i_log_info("%s", version);
+
+ cdev = cxgbi_device_register(sizeof(*lldi), lldi->nports);
+ if (!cdev) {
+ cxgb4i_log_info("t4 device 0x%p, register failed.\n", lldi);
+ return NULL;
+ }
+ cxgb4i_log_debug(1 << CXGBI_DBG_DEV,
+ "t4 0x%p,0x%x, ports %u,%s, chan %u, q %u,%u, wr %u.\n",
+ cdev, lldi->adapter_type, lldi->nports,
+ lldi->ports[0]->name, lldi->nchan, lldi->ntxq,
+ lldi->nrxq, lldi->wr_cred);
+ for (i = 0; i < lldi->nrxq; i++)
+ cxgb4i_log_debug(1 << CXGBI_DBG_DEV,
+ "t4 0x%p, rxq id #%d: %u.\n",
+ cdev, i, lldi->rxq_ids[i]);
+
+ memcpy(cxgbi_cdev_priv(cdev), lldi, sizeof(*lldi));
+ cdev->flags = CXGBI_FLAG_DEV_T4;
+ cdev->pdev = lldi->pdev;
+ cdev->ports = lldi->ports;
+ cdev->nports = lldi->nports;
+ cdev->mtus = lldi->mtus;
+ cdev->nmtus = NMTUS;
+ cdev->snd_win = cxgb4i_snd_win;
+ cdev->rcv_win = cxgb4i_rcv_win;
+ cdev->rx_credit_thres = cxgb4i_rx_credit_thres;
+ cdev->skb_tx_rsvd = CXGB4I_TX_HEADER_LEN;
+ cdev->skb_rx_extra = sizeof(struct cpl_iscsi_hdr);
+ cdev->itp = &cxgb4i_iscsi_transport;
+
+ rc = cxgb4i_ddp_init(cdev);
+ if (rc) {
+ cxgb4i_log_info("t4 0x%p ddp init failed\n", cdev);
+ goto err_out;
+ }
+ rc = cxgb4i_ofld_init(cdev);
+ if (rc) {
+ cxgb4i_log_info("t4 0x%p ofld init failed\n", cdev);
+ goto err_out;
+ }
+
+ rc = cxgbi_hbas_add(cdev, CXGB4I_MAX_LUN, CXGBI_MAX_CONN,
+ &cxgb4i_host_template, cxgb4i_stt);
+ if (rc)
+ goto err_out;
+
+ for (i = 0; i < cdev->nports; i++) {
+ pi = netdev_priv(lldi->ports[i]);
+ cdev->hbas[i]->port_id = pi->port_id;
+ }
+ cxgb4i_log_info("cdev 0x%p, f 0x%x, open, err %d.\n",
+ cdev, cdev->flags, rc);
+ return cdev;
+
+err_out:
+ cxgbi_device_unregister(cdev);
+ return ERR_PTR(-ENOMEM);
+}
+
+#define RX_PULL_LEN 128
+static int t4_uld_rx_handler(void *handle, const __be64 *rsp,
+ const struct pkt_gl *pgl)
+{
+ const struct cpl_act_establish *rpl;
+ struct sk_buff *skb;
+ unsigned int opc;
+ struct cxgbi_device *cdev = handle;
+
+ if (pgl == NULL) {
+ unsigned int len = 64 - sizeof(struct rsp_ctrl) - 8;
+
+ skb = alloc_cpl(len, 0, GFP_ATOMIC);
+ if (!skb)
+ goto nomem;
+ skb_copy_to_linear_data(skb, &rsp[1], len);
+ } else {
+ if (unlikely(*(u8 *)rsp != *(u8 *)pgl->va)) {
+ cxgb4i_log_info("? FL 0x%p,RSS%#llx,FL %#llx,len %u.\n",
+ pgl->va, be64_to_cpu(*rsp),
+ be64_to_cpu(*(u64 *)pgl->va),
+ pgl->tot_len);
+ return 0;
+ }
+ skb = cxgb4_pktgl_to_skb(pgl, RX_PULL_LEN, RX_PULL_LEN);
+ if (unlikely(!skb))
+ goto nomem;
+ }
+
+ rpl = (struct cpl_act_establish *)skb->data;
+ opc = rpl->ot.opcode;
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE,
+ "cdev %p, opcode 0x%x(0x%x,0x%x), skb %p\n",
+ cdev, opc, rpl->ot.opcode_tid,
+ ntohl(rpl->ot.opcode_tid), skb);
+ if (cxgb4i_cplhandlers[opc])
+ cxgb4i_cplhandlers[opc](cdev, skb);
+ else {
+ cxgb4i_log_error("No handler for opcode 0x%x\n", opc);
+ __kfree_skb(skb);
+ }
+ return 0;
+nomem:
+ cxgb4i_log_debug(1 << CXGBI_DBG_TOE, "OOM bailing out\n");
+ return 1;
+}
+
+static int t4_uld_state_change(void *handle, enum cxgb4_state state)
+{
+ struct cxgbi_device *cdev = handle;
+
+ switch (state) {
+ case CXGB4_STATE_UP:
+ cxgb4i_log_info("cdev 0x%p, UP.\n", cdev);
+ /* re-initialize */
+ break;
+ case CXGB4_STATE_START_RECOVERY:
+ cxgb4i_log_info("cdev 0x%p, RECOVERY.\n", cdev);
+ /* close all connections */
+ break;
+ case CXGB4_STATE_DOWN:
+ cxgb4i_log_info("cdev 0x%p, DOWN.\n", cdev);
+ break;
+ case CXGB4_STATE_DETACH:
+ cxgb4i_log_info("cdev 0x%p, DETACH.\n", cdev);
+ break;
+ default:
+ cxgb4i_log_info("cdev 0x%p, unknown state %d.\n", cdev, state);
+ break;
+ }
+ return 0;
+}
+
+static int __init cxgb4i_init_module(void)
+{
+ int rc;
+
+ rc = cxgbi_iscsi_init(&cxgb4i_iscsi_transport, &cxgb4i_stt);
+ if (rc < 0)
+ return rc;
+ cxgb4_register_uld(CXGB4_ULD_ISCSI, &cxgb4i_uld_info);
+ return 0;
+}
+
+static void __exit cxgb4i_exit_module(void)
+{
+ cxgb4_unregister_uld(CXGB4_ULD_ISCSI);
+ cxgbi_device_unregister_all(CXGBI_FLAG_DEV_T4);
+ cxgbi_iscsi_cleanup(&cxgb4i_iscsi_transport, &cxgb4i_stt);
+}
+
+module_init(cxgb4i_init_module);
+module_exit(cxgb4i_exit_module);
diff --git a/drivers/scsi/cxgbi/cxgb4i/cxgb4i.h b/drivers/scsi/cxgbi/cxgb4i/cxgb4i.h
new file mode 100644
index 0000000..342263b
--- /dev/null
+++ b/drivers/scsi/cxgbi/cxgb4i/cxgb4i.h
@@ -0,0 +1,38 @@
+/*
+ * cxgb4i.h: Chelsio T4 iSCSI driver.
+ *
+ * Copyright (c) 2010 Chelsio Communications, Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation.
+ *
+ * Written by: Karen Xie (kxie@chelsio.com)
+ * Written by: Rakesh Ranjan (rranjan@chelsio.com)
+ */
+
+#ifndef __CXGB4I_H__
+#define __CXGB4I_H__
+
+#define CXGB4I_SCSI_HOST_QDEPTH 1024
+#define CXGB4I_MAX_CONN 16384
+#define CXGB4I_MAX_TARGET CXGB4I_MAX_CONN
+#define CXGB4I_MAX_LUN 0x1000
+
+/* for TX: a skb must have a headroom of at least TX_HEADER_LEN bytes */
+#define CXGB4I_TX_HEADER_LEN \
+ (sizeof(struct fw_ofld_tx_data_wr) + sizeof(struct sge_opaque_hdr))
+
+struct cpl_rx_data_ddp {
+ union opcode_tid ot;
+ __be16 urg;
+ __be16 len;
+ __be32 seq;
+ union {
+ __be32 nxt_seq;
+ __be32 ddp_report;
+ };
+ __be32 ulp_crc;
+ __be32 ddpvld;
+};
+#endif /* __CXGB4I_H__ */
^ permalink raw reply related
* [PATCH v2 3/3] cxgb3i: change cxgb3i to use libcxgbi
From: kxie @ 2010-08-13 15:19 UTC (permalink / raw)
To: netdev, linux-scsi, open-iscsi
Cc: rranjan, kxie, James.Bottomley, michaelc, davem
[PATCH v2 3/3] cxgb3i: change cxgb3i to use libcxgbi
From: Karen Xie <kxie@chelsio.com>
Changed cxgb3i to make use of libcxgbi.
Signed-off-by: Karen Xie <kxie@chelsio.com>
---
drivers/scsi/Kconfig | 1
drivers/scsi/Makefile | 2
drivers/scsi/cxgbi/Kconfig | 1
drivers/scsi/cxgbi/Makefile | 1
drivers/scsi/cxgbi/cxgb3i/Kbuild | 3
drivers/scsi/cxgbi/cxgb3i/Kconfig | 7
drivers/scsi/cxgbi/cxgb3i/cxgb3i.c | 1454 ++++++++++++++++++++++++++++++++++++
drivers/scsi/cxgbi/cxgb3i/cxgb3i.h | 51 +
8 files changed, 1518 insertions(+), 2 deletions(-)
create mode 100644 drivers/scsi/cxgbi/cxgb3i/Kbuild
create mode 100644 drivers/scsi/cxgbi/cxgb3i/Kconfig
create mode 100644 drivers/scsi/cxgbi/cxgb3i/cxgb3i.c
create mode 100644 drivers/scsi/cxgbi/cxgb3i/cxgb3i.h
diff --git a/drivers/scsi/Kconfig b/drivers/scsi/Kconfig
index efe66bb..0c85653 100644
--- a/drivers/scsi/Kconfig
+++ b/drivers/scsi/Kconfig
@@ -378,7 +378,6 @@ config ISCSI_BOOT_SYSFS
via sysfs to userspace. If you wish to export this information,
say Y. Otherwise, say N.
-source "drivers/scsi/cxgb3i/Kconfig"
source "drivers/scsi/cxgbi/Kconfig"
source "drivers/scsi/bnx2i/Kconfig"
source "drivers/scsi/be2iscsi/Kconfig"
diff --git a/drivers/scsi/Makefile b/drivers/scsi/Makefile
index cb31f8c..2e9a87e 100644
--- a/drivers/scsi/Makefile
+++ b/drivers/scsi/Makefile
@@ -133,7 +133,7 @@ obj-$(CONFIG_SCSI_HPTIOP) += hptiop.o
obj-$(CONFIG_SCSI_STEX) += stex.o
obj-$(CONFIG_SCSI_MVSAS) += mvsas/
obj-$(CONFIG_PS3_ROM) += ps3rom.o
-obj-$(CONFIG_SCSI_CXGB3_ISCSI) += libiscsi.o libiscsi_tcp.o cxgb3i/
+obj-$(CONFIG_SCSI_CXGB3_ISCSI) += libiscsi.o libiscsi_tcp.o cxgbi/
obj-$(CONFIG_SCSI_CXGB4_ISCSI) += libiscsi.o libiscsi_tcp.o cxgbi/
obj-$(CONFIG_SCSI_BNX2_ISCSI) += libiscsi.o bnx2i/
obj-$(CONFIG_BE2ISCSI) += libiscsi.o be2iscsi/
diff --git a/drivers/scsi/cxgbi/Kconfig b/drivers/scsi/cxgbi/Kconfig
index a470e38..17eb5d5 100644
--- a/drivers/scsi/cxgbi/Kconfig
+++ b/drivers/scsi/cxgbi/Kconfig
@@ -1 +1,2 @@
+source "drivers/scsi/cxgbi/cxgb3i/Kconfig"
source "drivers/scsi/cxgbi/cxgb4i/Kconfig"
diff --git a/drivers/scsi/cxgbi/Makefile b/drivers/scsi/cxgbi/Makefile
index 9e8f604..86007e3 100644
--- a/drivers/scsi/cxgbi/Makefile
+++ b/drivers/scsi/cxgbi/Makefile
@@ -1 +1,2 @@
+obj-$(CONFIG_SCSI_CXGB3_ISCSI) += libcxgbi.o cxgb3i/
obj-$(CONFIG_SCSI_CXGB4_ISCSI) += libcxgbi.o cxgb4i/
diff --git a/drivers/scsi/cxgbi/cxgb3i/Kbuild b/drivers/scsi/cxgbi/cxgb3i/Kbuild
new file mode 100644
index 0000000..09dbf9e
--- /dev/null
+++ b/drivers/scsi/cxgbi/cxgb3i/Kbuild
@@ -0,0 +1,3 @@
+EXTRA_CFLAGS += -I$(srctree)/drivers/net/cxgb3
+
+obj-$(CONFIG_SCSI_CXGB3_ISCSI) += cxgb3i.o
diff --git a/drivers/scsi/cxgbi/cxgb3i/Kconfig b/drivers/scsi/cxgbi/cxgb3i/Kconfig
new file mode 100644
index 0000000..5cf4e98
--- /dev/null
+++ b/drivers/scsi/cxgbi/cxgb3i/Kconfig
@@ -0,0 +1,7 @@
+config SCSI_CXGB3_ISCSI
+ tristate "Chelsio T3 iSCSI support"
+ depends on CHELSIO_T3_DEPENDS
+ select CHELSIO_T3
+ select SCSI_ISCSI_ATTRS
+ ---help---
+ This driver supports iSCSI offload for the Chelsio T3 devices.
diff --git a/drivers/scsi/cxgbi/cxgb3i/cxgb3i.c b/drivers/scsi/cxgbi/cxgb3i/cxgb3i.c
new file mode 100644
index 0000000..6cd9bd6
--- /dev/null
+++ b/drivers/scsi/cxgbi/cxgb3i/cxgb3i.c
@@ -0,0 +1,1454 @@
+/*
+ * cxgb3i.c: Chelsio S3xx iscsi offload driver
+ *
+ * Copyright (C) 2003-2008 Chelsio Communications. All rights reserved.
+ *
+ * This program is distributed in the hope that it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the LICENSE file included in this
+ * release for licensing terms and conditions.
+ *
+ * Written by: Dimitris Michailidis (dm@chelsio.com)
+ * Karen Xie (kxie@chelsio.com)
+ */
+
+#include <linux/version.h>
+#include <linux/module.h>
+#include <linux/moduleparam.h>
+#include <scsi/scsi_host.h>
+
+#include "common.h"
+#include "t3_cpl.h"
+#include "t3cdev.h"
+#include "cxgb3_defs.h"
+#include "cxgb3_ctl_defs.h"
+#include "cxgb3_offload.h"
+#include "firmware_exports.h"
+#include "cxgb3i.h"
+
+static unsigned int dbg_level;
+#include "../libcxgbi.h"
+
+#define DRV_MODULE_NAME "cxgb3i"
+#define DRV_MODULE_DESC "Chelsio T3 iSCSI Driver"
+#define DRV_MODULE_VERSION "2.0.0"
+#define DRV_MODULE_RELDATE "Jun. 2010"
+
+static char version[] =
+ DRV_MODULE_DESC " " DRV_MODULE_NAME
+ " v" DRV_MODULE_VERSION " (" DRV_MODULE_RELDATE ")\n";
+
+MODULE_AUTHOR("Chelsio Communications, Inc.");
+MODULE_DESCRIPTION(DRV_MODULE_DESC);
+MODULE_VERSION(DRV_MODULE_VERSION);
+MODULE_LICENSE("GPL");
+
+module_param(dbg_level, uint, 0644);
+MODULE_PARM_DESC(dbg_level, "debug flag (default=0)");
+
+static int cxgb3i_rcv_win = 256 * 1024;
+module_param(cxgb3i_rcv_win, int, 0644);
+MODULE_PARM_DESC(cxgb3i_rcv_win, "TCP receive window in bytes (default=256KB)");
+
+static int cxgb3i_snd_win = 128 * 1024;
+module_param(cxgb3i_snd_win, int, 0644);
+MODULE_PARM_DESC(cxgb3i_snd_win, "TCP send window in bytes (default=128KB)");
+
+static int cxgb3i_rx_credit_thres = 10 * 1024;
+module_param(cxgb3i_rx_credit_thres, int, 0644);
+MODULE_PARM_DESC(rx_credit_thres,
+ "RX credits return threshold in bytes (default=10KB)");
+
+static unsigned int cxgb3i_max_connect = 8 * 1024;
+module_param(cxgb3i_max_connect, uint, 0644);
+MODULE_PARM_DESC(cxgb3i_max_connect, "Max. # of connections (default=8092)");
+
+static unsigned int cxgb3i_sport_base = 20000;
+module_param(cxgb3i_sport_base, uint, 0644);
+MODULE_PARM_DESC(cxgb3i_sport_base, "starting port number (default=20000)");
+
+#define cxgb3i_log_error cxgbi_dbg_error
+#define cxgb3i_log_warn cxgbi_dbg_warn
+#define cxgb3i_log_info cxgbi_dbg_info
+#define cxgb3i_log_debug(flag, fmt, args...) \
+ cxgbi_dbg_print(flag, fmt, ## args)
+
+static void cxgb3i_dev_open(struct t3cdev *);
+static void cxgb3i_dev_close(struct t3cdev *);
+static void cxgb3i_dev_event_handler(struct t3cdev *, u32, u32);
+
+static struct cxgb3_client t3_client = {
+ .name = DRV_MODULE_NAME,
+ .handlers = cxgb3i_cpl_handlers,
+ .add = cxgb3i_dev_open,
+ .remove = cxgb3i_dev_close,
+ .event_handler = cxgb3i_dev_event_handler,
+};
+
+static struct scsi_host_template cxgb3i_host_template = {
+ .module = THIS_MODULE,
+ .name = DRV_MODULE_NAME,
+ .proc_name = DRV_MODULE_NAME,
+ .can_queue = CXGB3I_SCSI_HOST_QDEPTH,
+ .queuecommand = iscsi_queuecommand,
+ .change_queue_depth = iscsi_change_queue_depth,
+ .sg_tablesize = SG_ALL,
+ .max_sectors = 0xFFFF,
+ .cmd_per_lun = ISCSI_DEF_CMD_PER_LUN,
+ .eh_abort_handler = iscsi_eh_abort,
+ .eh_device_reset_handler = iscsi_eh_device_reset,
+ .eh_target_reset_handler = iscsi_eh_recover_target,
+ .target_alloc = iscsi_target_alloc,
+ .use_clustering = DISABLE_CLUSTERING,
+ .this_id = -1,
+};
+
+static struct iscsi_transport cxgb3i_iscsi_transport = {
+ .owner = THIS_MODULE,
+ .name = DRV_MODULE_NAME,
+ /* owner and name should be set already */
+ .caps = CAP_RECOVERY_L0 | CAP_MULTI_R2T | CAP_HDRDGST
+ | CAP_DATADGST | CAP_DIGEST_OFFLOAD |
+ CAP_PADDING_OFFLOAD,
+ .param_mask = ISCSI_MAX_RECV_DLENGTH | ISCSI_MAX_XMIT_DLENGTH |
+ ISCSI_HDRDGST_EN | ISCSI_DATADGST_EN |
+ ISCSI_INITIAL_R2T_EN | ISCSI_MAX_R2T |
+ ISCSI_IMM_DATA_EN | ISCSI_FIRST_BURST |
+ ISCSI_MAX_BURST | ISCSI_PDU_INORDER_EN |
+ ISCSI_DATASEQ_INORDER_EN | ISCSI_ERL |
+ ISCSI_CONN_PORT | ISCSI_CONN_ADDRESS |
+ ISCSI_EXP_STATSN | ISCSI_PERSISTENT_PORT |
+ ISCSI_PERSISTENT_ADDRESS |
+ ISCSI_TARGET_NAME | ISCSI_TPGT |
+ ISCSI_USERNAME | ISCSI_PASSWORD |
+ ISCSI_USERNAME_IN | ISCSI_PASSWORD_IN |
+ ISCSI_FAST_ABORT | ISCSI_ABORT_TMO |
+ ISCSI_LU_RESET_TMO | ISCSI_TGT_RESET_TMO |
+ ISCSI_PING_TMO | ISCSI_RECV_TMO |
+ ISCSI_IFACE_NAME | ISCSI_INITIATOR_NAME,
+ .host_param_mask = ISCSI_HOST_HWADDRESS | ISCSI_HOST_IPADDRESS |
+ ISCSI_HOST_INITIATOR_NAME |
+ ISCSI_HOST_NETDEV_NAME,
+ .get_host_param = cxgbi_get_host_param,
+ .set_host_param = cxgbi_set_host_param,
+ /* session management */
+ .create_session = cxgbi_create_session,
+ .destroy_session = cxgbi_destroy_session,
+ .get_session_param = iscsi_session_get_param,
+ /* connection management */
+ .create_conn = cxgbi_create_conn,
+ .bind_conn = cxgbi_bind_conn,
+ .destroy_conn = iscsi_tcp_conn_teardown,
+ .start_conn = iscsi_conn_start,
+ .stop_conn = iscsi_conn_stop,
+ .get_conn_param = cxgbi_get_conn_param,
+ .set_param = cxgbi_set_conn_param,
+ .get_stats = cxgbi_get_conn_stats,
+ /* pdu xmit req from user space */
+ .send_pdu = iscsi_conn_send_pdu,
+ /* task */
+ .init_task = iscsi_tcp_task_init,
+ .xmit_task = iscsi_tcp_task_xmit,
+ .cleanup_task = cxgbi_cleanup_task,
+ /* pdu */
+ .alloc_pdu = cxgbi_conn_alloc_pdu,
+ .init_pdu = cxgbi_conn_init_pdu,
+ .xmit_pdu = cxgbi_conn_xmit_pdu,
+ .parse_pdu_itt = cxgbi_parse_pdu_itt,
+ /* TCP connect/disconnect */
+ .ep_connect = cxgbi_ep_connect,
+ .ep_poll = cxgbi_ep_poll,
+ .ep_disconnect = cxgbi_ep_disconnect,
+ /* Error recovery timeout call */
+ .session_recovery_timedout = iscsi_session_recovery_timedout,
+};
+
+static struct scsi_transport_template *cxgb3i_stt;
+
+/*
+ * CPL (Chelsio Protocol Language) defines a message passing interface between
+ * the host driver and Chelsio asic.
+ * The section below implments CPLs that related to iscsi tcp connection
+ * open/close/abort and data send/receive.
+ */
+
+static int push_tx_frames(struct cxgbi_sock *csk, int req_completion);
+
+static void send_act_open_req(struct cxgbi_sock *csk, struct sk_buff *skb,
+ const struct l2t_entry *e)
+{
+ unsigned int wscale = cxgbi_sock_compute_wscale(cxgb3i_rcv_win);
+ struct cpl_act_open_req *req = (struct cpl_act_open_req *)skb->head;
+
+ skb->priority = CPL_PRIORITY_SETUP;
+
+ req->wr.wr_hi = htonl(V_WR_OP(FW_WROPCODE_FORWARD));
+ OPCODE_TID(req) = htonl(MK_OPCODE_TID(CPL_ACT_OPEN_REQ, csk->atid));
+ req->local_port = csk->saddr.sin_port;
+ req->peer_port = csk->daddr.sin_port;
+ req->local_ip = csk->saddr.sin_addr.s_addr;
+ req->peer_ip = csk->daddr.sin_addr.s_addr;
+
+ req->opt0h = htonl(V_KEEP_ALIVE(1) | F_TCAM_BYPASS |
+ V_WND_SCALE(wscale) | V_MSS_IDX(csk->mss_idx) |
+ V_L2T_IDX(e->idx) | V_TX_CHANNEL(e->smt_idx));
+ req->opt0l = htonl(V_ULP_MODE(ULP2_MODE_ISCSI) |
+ V_RCV_BUFSIZ(cxgb3i_rcv_win>>10));
+
+ cxgb3i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx,%u, %pI4:%u-%pI4:%u, %u,%u,%u.\n",
+ csk, csk->state, csk->flags, csk->atid,
+ &req->local_ip, ntohs(req->local_port),
+ &req->peer_ip, ntohs(req->peer_port),
+ csk->mss_idx, e->idx, e->smt_idx);
+
+ l2t_send(csk->cdev->lldev, skb, csk->l2t);
+}
+
+static inline void act_open_arp_failure(struct t3cdev *dev, struct sk_buff *skb)
+{
+ cxgbi_sock_act_open_req_arp_failure(NULL, skb);
+}
+
+/*
+ * CPL connection close request: host ->
+ *
+ * Close a connection by sending a CPL_CLOSE_CON_REQ message and queue it to
+ * the write queue (i.e., after any unsent txt data).
+ */
+static void send_close_req(struct cxgbi_sock *csk)
+{
+ struct sk_buff *skb = csk->cpl_close;
+ struct cpl_close_con_req *req = (struct cpl_close_con_req *)skb->head;
+ unsigned int tid = csk->tid;
+
+ cxgb3i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx,%u.\n",
+ csk, csk->state, csk->flags, csk->tid);
+
+ csk->cpl_close = NULL;
+ req->wr.wr_hi = htonl(V_WR_OP(FW_WROPCODE_OFLD_CLOSE_CON));
+ req->wr.wr_lo = htonl(V_WR_TID(tid));
+ OPCODE_TID(req) = htonl(MK_OPCODE_TID(CPL_CLOSE_CON_REQ, tid));
+ req->rsvd = htonl(csk->write_seq);
+
+ cxgbi_sock_skb_entail(csk, skb);
+ if (csk->state >= CTP_ESTABLISHED)
+ push_tx_frames(csk, 1);
+}
+
+/*
+ * CPL connection abort request: host ->
+ *
+ * Send an ABORT_REQ message. Makes sure we do not send multiple ABORT_REQs
+ * for the same connection and also that we do not try to send a message
+ * after the connection has closed.
+ */
+static void abort_arp_failure(struct t3cdev *tdev, struct sk_buff *skb)
+{
+ struct cpl_abort_req *req = cplhdr(skb);
+
+ cxgb3i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "t3dev 0x%p, tid %u, skb 0x%p.\n",
+ tdev, GET_TID(req), skb);
+ req->cmd = CPL_ABORT_NO_RST;
+ cxgb3_ofld_send(tdev, skb);
+}
+
+static void send_abort_req(struct cxgbi_sock *csk)
+{
+ struct sk_buff *skb = csk->cpl_abort_req;
+ struct cpl_abort_req *req;
+
+ if (unlikely(csk->state == CTP_ABORTING || !skb))
+ return;
+ cxgbi_sock_set_state(csk, CTP_ABORTING);
+ cxgbi_sock_set_flag(csk, CTPF_ABORT_RPL_PENDING);
+ /* Purge the send queue so we don't send anything after an abort. */
+ cxgbi_sock_purge_write_queue(csk);
+
+ csk->cpl_abort_req = NULL;
+ req = (struct cpl_abort_req *)skb->head;
+ skb->priority = CPL_PRIORITY_DATA;
+ set_arp_failure_handler(skb, abort_arp_failure);
+ req->wr.wr_hi = htonl(V_WR_OP(FW_WROPCODE_OFLD_HOST_ABORT_CON_REQ));
+ req->wr.wr_lo = htonl(V_WR_TID(csk->tid));
+ OPCODE_TID(req) = htonl(MK_OPCODE_TID(CPL_ABORT_REQ, csk->tid));
+ req->rsvd0 = htonl(csk->snd_nxt);
+ req->rsvd1 = !cxgbi_sock_flag(csk, CTPF_TX_DATA_SENT);
+ req->cmd = CPL_ABORT_SEND_RST;
+
+ cxgb3i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx,%u, snd_nxt %u, 0x%x.\n",
+ csk, csk->state, csk->flags, csk->tid, csk->snd_nxt,
+ req->rsvd1);
+
+ l2t_send(csk->cdev->lldev, skb, csk->l2t);
+}
+
+/*
+ * CPL connection abort reply: host ->
+ *
+ * Send an ABORT_RPL message in response of the ABORT_REQ received.
+ */
+static void send_abort_rpl(struct cxgbi_sock *csk, int rst_status)
+{
+ struct sk_buff *skb = csk->cpl_abort_rpl;
+ struct cpl_abort_rpl *rpl = (struct cpl_abort_rpl *)skb->head;
+
+ cxgb3i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx,%u, status %d.\n",
+ csk, csk->state, csk->flags, csk->tid, rst_status);
+
+ csk->cpl_abort_rpl = NULL;
+ skb->priority = CPL_PRIORITY_DATA;
+ rpl->wr.wr_hi = htonl(V_WR_OP(FW_WROPCODE_OFLD_HOST_ABORT_CON_RPL));
+ rpl->wr.wr_lo = htonl(V_WR_TID(csk->tid));
+ OPCODE_TID(rpl) = htonl(MK_OPCODE_TID(CPL_ABORT_RPL, csk->tid));
+ rpl->cmd = rst_status;
+ cxgb3_ofld_send(csk->cdev->lldev, skb);
+}
+
+/*
+ * CPL connection rx data ack: host ->
+ * Send RX credits through an RX_DATA_ACK CPL message. Returns the number of
+ * credits sent.
+ */
+static u32 send_rx_credits(struct cxgbi_sock *csk, u32 credits)
+{
+ struct sk_buff *skb;
+ struct cpl_rx_data_ack *req;
+ u32 dack = F_RX_DACK_CHANGE | V_RX_DACK_MODE(1);
+
+ cxgb3i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_PDU_RX,
+ "csk 0x%p,%u,0x%lx,%u, credit %u, dack %u\n",
+ csk, csk->state, csk->flags, csk->tid, credits, dack);
+
+ skb = alloc_cpl(sizeof(*req), 0, GFP_ATOMIC);
+ if (!skb) {
+ cxgb3i_log_info("csk 0x%p, credit %u, OOM.\n", csk, credits);
+ return 0;
+ }
+ req = (struct cpl_rx_data_ack *)skb->head;
+ req->wr.wr_hi = htonl(V_WR_OP(FW_WROPCODE_FORWARD));
+ OPCODE_TID(req) = htonl(MK_OPCODE_TID(CPL_RX_DATA_ACK, csk->tid));
+ req->credit_dack = htonl(F_RX_DACK_CHANGE | V_RX_DACK_MODE(1) |
+ V_RX_CREDITS(credits));
+ skb->priority = CPL_PRIORITY_ACK;
+ cxgb3_ofld_send(csk->cdev->lldev, skb);
+ return credits;
+}
+
+/*
+ * CPL connection tx data: host ->
+ *
+ * Send iscsi PDU via TX_DATA CPL message. Returns the number of
+ * credits sent.
+ * Each TX_DATA consumes work request credit (wrs), so we need to keep track of
+ * how many we've used so far and how many are pending (i.e., yet ack'ed by T3).
+ */
+
+static unsigned int wrlen __read_mostly;
+static unsigned int skb_wrs[SKB_WR_LIST_SIZE] __read_mostly;
+
+static void init_wr_tab(unsigned int wr_len)
+{
+ int i;
+
+ if (skb_wrs[1]) /* already initialized */
+ return;
+ for (i = 1; i < SKB_WR_LIST_SIZE; i++) {
+ int sgl_len = (3 * i) / 2 + (i & 1);
+
+ sgl_len += 3;
+ skb_wrs[i] = (sgl_len <= wr_len
+ ? 1 : 1 + (sgl_len - 2) / (wr_len - 1));
+ }
+ wrlen = wr_len * 8;
+}
+
+static inline void make_tx_data_wr(struct cxgbi_sock *csk, struct sk_buff *skb,
+ int len, int req_completion)
+{
+ struct tx_data_wr *req;
+ struct l2t_entry *l2t = csk->l2t;
+
+ skb_reset_transport_header(skb);
+ req = (struct tx_data_wr *)__skb_push(skb, sizeof(*req));
+ req->wr_hi = htonl(V_WR_OP(FW_WROPCODE_OFLD_TX_DATA) |
+ (req_completion ? F_WR_COMPL : 0));
+ req->wr_lo = htonl(V_WR_TID(csk->tid));
+ /* len includes the length of any HW ULP additions */
+ req->len = htonl(len);
+ /* V_TX_ULP_SUBMODE sets both the mode and submode */
+ req->flags = htonl(V_TX_ULP_SUBMODE(cxgbi_skcb_ulp_mode(skb)) |
+ V_TX_SHOVE((skb_peek(&csk->write_queue) ? 0 : 1)));
+ req->sndseq = htonl(csk->snd_nxt);
+ req->param = htonl(V_TX_PORT(l2t->smt_idx));
+
+ if (!cxgbi_sock_flag(csk, CTPF_TX_DATA_SENT)) {
+ req->flags |= htonl(V_TX_ACK_PAGES(2) | F_TX_INIT |
+ V_TX_CPU_IDX(csk->rss_qid));
+ /* sendbuffer is in units of 32KB. */
+ req->param |= htonl(V_TX_SNDBUF(cxgb3i_snd_win >> 15));
+ cxgbi_sock_set_flag(csk, CTPF_TX_DATA_SENT);
+ }
+}
+
+/**
+ * push_tx_frames -- start transmit
+ * @c3cn: the offloaded connection
+ * @req_completion: request wr_ack or not
+ *
+ * Prepends TX_DATA_WR or CPL_CLOSE_CON_REQ headers to buffers waiting in a
+ * connection's send queue and sends them on to T3. Must be called with the
+ * connection's lock held. Returns the amount of send buffer space that was
+ * freed as a result of sending queued data to T3.
+ */
+
+static void arp_failure_skb_discard(struct t3cdev *dev, struct sk_buff *skb)
+{
+ kfree_skb(skb);
+}
+
+static int push_tx_frames(struct cxgbi_sock *csk, int req_completion)
+{
+ int total_size = 0;
+ struct sk_buff *skb;
+
+ if (unlikely(csk->state < CTP_ESTABLISHED ||
+ csk->state == CTP_CLOSE_WAIT_1 || csk->state >= CTP_ABORTING)) {
+ cxgb3i_log_debug(1 << CXGBI_DBG_TOE |
+ 1 << CXGBI_DBG_PDU_TX,
+ "csk 0x%p,%u,0x%lx,%u, in closing state.\n",
+ csk, csk->state, csk->flags, csk->tid);
+ return 0;
+ }
+
+ while (csk->wr_cred && (skb = skb_peek(&csk->write_queue)) != NULL) {
+ int len = skb->len; /* length before skb_push */
+ int frags = skb_shinfo(skb)->nr_frags + (len != skb->data_len);
+ int wrs_needed = skb_wrs[frags];
+
+ if (wrs_needed > 1 && len + sizeof(struct tx_data_wr) <= wrlen)
+ wrs_needed = 1;
+
+ WARN_ON(frags >= SKB_WR_LIST_SIZE || wrs_needed < 1);
+
+ if (csk->wr_cred < wrs_needed) {
+ cxgb3i_log_debug(1 << CXGBI_DBG_PDU_TX,
+ "csk 0x%p, skb len %u/%u, frag %u, "
+ "wr %d < %u.\n",
+ csk, skb->len, skb->data_len, frags,
+ wrs_needed, csk->wr_cred);
+ break;
+ }
+
+ __skb_unlink(skb, &csk->write_queue);
+ skb->priority = CPL_PRIORITY_DATA;
+ skb->csum = wrs_needed; /* remember this until the WR_ACK */
+ csk->wr_cred -= wrs_needed;
+ csk->wr_una_cred += wrs_needed;
+ cxgbi_sock_enqueue_wr(csk, skb);
+
+ cxgb3i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_PDU_TX,
+ "csk 0x%p, enqueue, skb len %u/%u, frag %u, "
+ "wr %d, left %u, unack %u.\n",
+ csk, skb->len, skb->data_len, frags,
+ skb->csum, csk->wr_cred, csk->wr_una_cred);
+
+ if (likely(cxgbi_skcb_test_flag(skb, SKCBF_TX_NEED_HDR))) {
+ if ((req_completion &&
+ csk->wr_una_cred == wrs_needed) ||
+ csk->wr_una_cred >= csk->wr_max_cred / 2) {
+ req_completion = 1;
+ csk->wr_una_cred = 0;
+ }
+ len += cxgbi_ulp_extra_len(cxgbi_skcb_ulp_mode(skb));
+ make_tx_data_wr(csk, skb, len, req_completion);
+ csk->snd_nxt += len;
+ cxgbi_skcb_clear_flag(skb, SKCBF_TX_NEED_HDR);
+ }
+ total_size += skb->truesize;
+ cxgb3i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_PDU_TX,
+ "csk 0x%p, tid 0x%x, send skb 0x%p.\n",
+ csk, csk->tid, skb);
+ set_arp_failure_handler(skb, arp_failure_skb_discard);
+ l2t_send(csk->cdev->lldev, skb, csk->l2t);
+ }
+ return total_size;
+}
+
+/*
+ * Process a CPL_ACT_ESTABLISH message: -> host
+ * Updates connection state from an active establish CPL message. Runs with
+ * the connection lock held.
+ */
+
+static inline void free_atid(struct cxgbi_sock *csk)
+{
+ if (cxgbi_sock_flag(csk, CTPF_HAS_ATID)) {
+ cxgb3_free_atid(csk->cdev->lldev, csk->atid);
+ cxgbi_sock_clear_flag(csk, CTPF_HAS_ATID);
+ cxgbi_sock_put(csk);
+ }
+}
+
+static int do_act_establish(struct t3cdev *tdev, struct sk_buff *skb, void *ctx)
+{
+ struct cxgbi_sock *csk = ctx;
+ struct cpl_act_establish *req = cplhdr(skb);
+ unsigned int tid = GET_TID(req);
+ unsigned int atid = G_PASS_OPEN_TID(ntohl(req->tos_tid));
+ u32 rcv_isn = ntohl(req->rcv_isn); /* real RCV_ISN + 1 */
+
+ cxgb3i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "atid 0x%x,tid 0x%x, csk 0x%p,%u,0x%lx, isn %u.\n",
+ atid, atid, csk, csk->state, csk->flags, rcv_isn);
+
+ cxgbi_sock_get(csk);
+ cxgbi_sock_set_flag(csk, CTPF_HAS_TID);
+ csk->tid = tid;
+ cxgb3_insert_tid(csk->cdev->lldev, &t3_client, csk, tid);
+
+ free_atid(csk);
+
+ csk->rss_qid = G_QNUM(ntohs(skb->csum));
+
+ spin_lock_bh(&csk->lock);
+ if (csk->retry_timer.function) {
+ del_timer(&csk->retry_timer);
+ csk->retry_timer.function = NULL;
+ }
+
+ if (unlikely(csk->state != CTP_ACTIVE_OPEN))
+ cxgb3i_log_info("csk 0x%p,%u,0x%lx,%u, got EST.\n",
+ csk, csk->state, csk->flags, csk->tid);
+
+ csk->copied_seq = csk->rcv_wup = csk->rcv_nxt = rcv_isn;
+ if (cxgb3i_rcv_win > (M_RCV_BUFSIZ << 10))
+ csk->rcv_wup -= cxgb3i_rcv_win - (M_RCV_BUFSIZ << 10);
+
+ cxgbi_sock_established(csk, ntohl(req->snd_isn), ntohs(req->tcp_opt));
+
+ if (unlikely(cxgbi_sock_flag(csk, CTPF_ACTIVE_CLOSE_NEEDED)))
+ /* upper layer has requested closing */
+ send_abort_req(csk);
+ else {
+ if (skb_queue_len(&csk->write_queue))
+ push_tx_frames(csk, 1);
+ cxgbi_conn_tx_open(csk);
+ }
+
+ spin_unlock_bh(&csk->lock);
+ __kfree_skb(skb);
+ return 0;
+}
+
+/*
+ * Process a CPL_ACT_OPEN_RPL message: -> host
+ * Handle active open failures.
+ */
+static int act_open_rpl_status_to_errno(int status)
+{
+ switch (status) {
+ case CPL_ERR_CONN_RESET:
+ return -ECONNREFUSED;
+ case CPL_ERR_ARP_MISS:
+ return -EHOSTUNREACH;
+ case CPL_ERR_CONN_TIMEDOUT:
+ return -ETIMEDOUT;
+ case CPL_ERR_TCAM_FULL:
+ return -ENOMEM;
+ case CPL_ERR_CONN_EXIST:
+ return -EADDRINUSE;
+ default:
+ return -EIO;
+ }
+}
+
+static void act_open_retry_timer(unsigned long data)
+{
+ struct sk_buff *skb;
+ struct cxgbi_sock *csk = (struct cxgbi_sock *)data;
+
+ cxgb3i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx,%u.\n",
+ csk, csk->state, csk->flags, csk->tid);
+
+ cxgbi_sock_get(csk);
+ spin_lock_bh(&csk->lock);
+ skb = alloc_cpl(sizeof(struct cpl_act_open_req), 0, GFP_ATOMIC);
+ if (!skb)
+ cxgbi_sock_fail_act_open(csk, -ENOMEM);
+ else {
+ skb->sk = (struct sock *)csk;
+ set_arp_failure_handler(skb, act_open_arp_failure);
+ send_act_open_req(csk, skb, csk->l2t);
+ }
+ spin_unlock_bh(&csk->lock);
+ cxgbi_sock_put(csk);
+}
+
+static int do_act_open_rpl(struct t3cdev *tdev, struct sk_buff *skb, void *ctx)
+{
+ struct cxgbi_sock *csk = ctx;
+ struct cpl_act_open_rpl *rpl = cplhdr(skb);
+
+ cxgb3i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx,%u, status %u.\n",
+ csk, csk->state, csk->flags, csk->atid, rpl->status);
+
+ if (rpl->status != CPL_ERR_TCAM_FULL &&
+ rpl->status != CPL_ERR_CONN_EXIST &&
+ rpl->status != CPL_ERR_ARP_MISS)
+ cxgb3_queue_tid_release(tdev, GET_TID(rpl));
+
+ cxgbi_sock_get(csk);
+ spin_lock_bh(&csk->lock);
+ if (rpl->status == CPL_ERR_CONN_EXIST &&
+ csk->retry_timer.function != act_open_retry_timer) {
+ csk->retry_timer.function = act_open_retry_timer;
+ mod_timer(&csk->retry_timer, jiffies + HZ / 2);
+ } else
+ cxgbi_sock_fail_act_open(csk,
+ act_open_rpl_status_to_errno(rpl->status));
+
+ spin_unlock_bh(&csk->lock);
+ cxgbi_sock_put(csk);
+ __kfree_skb(skb);
+ return 0;
+}
+
+/*
+ * Process PEER_CLOSE CPL messages: -> host
+ * Handle peer FIN.
+ */
+static int do_peer_close(struct t3cdev *cdev, struct sk_buff *skb, void *ctx)
+{
+ struct cxgbi_sock *csk = ctx;
+
+ cxgb3i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx,%u.\n",
+ csk, csk->state, csk->flags, csk->tid);
+
+ cxgbi_sock_rcv_peer_close(csk);
+ __kfree_skb(skb);
+ return 0;
+}
+
+/*
+ * Process CLOSE_CONN_RPL CPL message: -> host
+ * Process a peer ACK to our FIN.
+ */
+static int do_close_con_rpl(struct t3cdev *cdev, struct sk_buff *skb,
+ void *ctx)
+{
+ struct cxgbi_sock *csk = ctx;
+ struct cpl_close_con_rpl *rpl = cplhdr(skb);
+
+ cxgb3i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx,%u, snxt %u.\n",
+ csk, csk->state, csk->flags, csk->tid,
+ ntohl(rpl->snd_nxt));
+
+ cxgbi_sock_rcv_close_conn_rpl(csk, ntohl(rpl->snd_nxt));
+ __kfree_skb(skb);
+ return 0;
+}
+
+/*
+ * Process ABORT_REQ_RSS CPL message: -> host
+ * Process abort requests. If we are waiting for an ABORT_RPL we ignore this
+ * request except that we need to reply to it.
+ */
+
+static int abort_status_to_errno(struct cxgbi_sock *csk, int abort_reason,
+ int *need_rst)
+{
+ switch (abort_reason) {
+ case CPL_ERR_BAD_SYN: /* fall through */
+ case CPL_ERR_CONN_RESET:
+ return csk->state > CTP_ESTABLISHED ?
+ -EPIPE : -ECONNRESET;
+ case CPL_ERR_XMIT_TIMEDOUT:
+ case CPL_ERR_PERSIST_TIMEDOUT:
+ case CPL_ERR_FINWAIT2_TIMEDOUT:
+ case CPL_ERR_KEEPALIVE_TIMEDOUT:
+ return -ETIMEDOUT;
+ default:
+ return -EIO;
+ }
+}
+
+static int do_abort_req(struct t3cdev *cdev, struct sk_buff *skb, void *ctx)
+{
+ const struct cpl_abort_req_rss *req = cplhdr(skb);
+ struct cxgbi_sock *csk = ctx;
+ int rst_status = CPL_ABORT_NO_RST;
+
+ cxgb3i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx,%u.\n",
+ csk, csk->state, csk->flags, csk->tid);
+
+ if (req->status == CPL_ERR_RTX_NEG_ADVICE ||
+ req->status == CPL_ERR_PERSIST_NEG_ADVICE) {
+ goto done;
+ }
+
+ cxgbi_sock_get(csk);
+ spin_lock_bh(&csk->lock);
+
+ if (!cxgbi_sock_flag(csk, CTPF_ABORT_REQ_RCVD)) {
+ cxgbi_sock_set_flag(csk, CTPF_ABORT_REQ_RCVD);
+ cxgbi_sock_set_state(csk, CTP_ABORTING);
+ goto out;
+ }
+
+ cxgbi_sock_clear_flag(csk, CTPF_ABORT_REQ_RCVD);
+ send_abort_rpl(csk, rst_status);
+
+ if (!cxgbi_sock_flag(csk, CTPF_ABORT_RPL_PENDING)) {
+ csk->err = abort_status_to_errno(csk, req->status, &rst_status);
+ cxgbi_sock_closed(csk);
+ }
+
+out:
+ spin_unlock_bh(&csk->lock);
+ cxgbi_sock_put(csk);
+done:
+ __kfree_skb(skb);
+ return 0;
+}
+
+/*
+ * Process ABORT_RPL_RSS CPL message: -> host
+ * Process abort replies. We only process these messages if we anticipate
+ * them as the coordination between SW and HW in this area is somewhat lacking
+ * and sometimes we get ABORT_RPLs after we are done with the connection that
+ * originated the ABORT_REQ.
+ */
+static int do_abort_rpl(struct t3cdev *cdev, struct sk_buff *skb, void *ctx)
+{
+ struct cpl_abort_rpl_rss *rpl = cplhdr(skb);
+ struct cxgbi_sock *csk = ctx;
+
+ cxgb3i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "status 0x%x, csk 0x%p, s %u, 0x%lx.\n",
+ rpl->status, csk, csk ? csk->state : 0,
+ csk ? csk->flags : 0UL);
+ /*
+ * Ignore replies to post-close aborts indicating that the abort was
+ * requested too late. These connections are terminated when we get
+ * PEER_CLOSE or CLOSE_CON_RPL and by the time the abort_rpl_rss
+ * arrives the TID is either no longer used or it has been recycled.
+ */
+ if (rpl->status == CPL_ERR_ABORT_FAILED)
+ goto rel_skb;
+ /*
+ * Sometimes we've already closed the connection, e.g., a post-close
+ * abort races with ABORT_REQ_RSS, the latter frees the connection
+ * expecting the ABORT_REQ will fail with CPL_ERR_ABORT_FAILED,
+ * but FW turns the ABORT_REQ into a regular one and so we get
+ * ABORT_RPL_RSS with status 0 and no connection.
+ */
+ if (csk)
+ cxgbi_sock_rcv_abort_rpl(csk);
+rel_skb:
+ __kfree_skb(skb);
+ return 0;
+}
+
+/*
+ * Process RX_ISCSI_HDR CPL message: -> host
+ * Handle received PDUs, the payload could be DDP'ed. If not, the payload
+ * follow after the bhs.
+ */
+static int do_iscsi_hdr(struct t3cdev *t3dev, struct sk_buff *skb, void *ctx)
+{
+ struct cxgbi_sock *csk = ctx;
+ struct cpl_iscsi_hdr *hdr_cpl = cplhdr(skb);
+ struct cpl_iscsi_hdr_norss data_cpl;
+ struct cpl_rx_data_ddp_norss ddp_cpl;
+ unsigned int hdr_len, data_len, status;
+ unsigned int len;
+ int err;
+
+ cxgb3i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_PDU_RX,
+ "csk 0x%p,%u,0x%lx,%u, skb 0x%p,%u.\n",
+ csk, csk->state, csk->flags, csk->tid, skb, skb->len);
+
+ spin_lock_bh(&csk->lock);
+
+ if (unlikely(csk->state >= CTP_PASSIVE_CLOSE)) {
+ cxgb3i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx,%u, bad state.\n",
+ csk, csk->state, csk->flags, csk->tid);
+ if (csk->state != CTP_ABORTING)
+ goto abort_conn;
+ else
+ goto discard;
+ }
+
+ cxgbi_skcb_tcp_seq(skb) = ntohl(hdr_cpl->seq);
+ cxgbi_skcb_flags(skb) = 0;
+
+ skb_reset_transport_header(skb);
+ __skb_pull(skb, sizeof(struct cpl_iscsi_hdr));
+
+ len = hdr_len = ntohs(hdr_cpl->len);
+ /* msg coalesce is off or not enough data received */
+ if (skb->len <= hdr_len) {
+ cxgb3i_log_error("%s: TID %u, ISCSI_HDR, skb len %u < %u.\n",
+ csk->cdev->ports[csk->port_id]->name,
+ csk->tid, skb->len, hdr_len);
+ goto abort_conn;
+ }
+ cxgbi_skcb_set_flag(skb, SKCBF_RX_COALESCED);
+
+ err = skb_copy_bits(skb, skb->len - sizeof(ddp_cpl), &ddp_cpl,
+ sizeof(ddp_cpl));
+ if (err < 0) {
+ cxgb3i_log_error("%s: TID %u, copy cpl_ddp %u-%lu failed %d.\n",
+ csk->cdev->ports[csk->port_id]->name,
+ csk->tid, skb->len, sizeof(ddp_cpl), err);
+ goto abort_conn;
+ }
+
+ cxgbi_skcb_set_flag(skb, SKCBF_RX_STATUS);
+ cxgbi_skcb_rx_pdulen(skb) = ntohs(ddp_cpl.len);
+ cxgbi_skcb_rx_ddigest(skb) = ntohl(ddp_cpl.ulp_crc);
+ status = ntohl(ddp_cpl.ddp_status);
+
+ cxgb3i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_PDU_RX,
+ "csk 0x%p, skb 0x%p,%u, pdulen %u, status 0x%x.\n",
+ csk, skb, skb->len, cxgbi_skcb_rx_pdulen(skb), status);
+
+ if (status & (1 << CPL_RX_DDP_STATUS_HCRC_SHIFT))
+ cxgbi_skcb_set_flag(skb, SKCBF_RX_HCRC_ERR);
+ if (status & (1 << CPL_RX_DDP_STATUS_DCRC_SHIFT))
+ cxgbi_skcb_set_flag(skb, SKCBF_RX_DCRC_ERR);
+ if (status & (1 << CPL_RX_DDP_STATUS_PAD_SHIFT))
+ cxgbi_skcb_set_flag(skb, SKCBF_RX_PAD_ERR);
+
+ if (skb->len > (hdr_len + sizeof(ddp_cpl))) {
+ err = skb_copy_bits(skb, hdr_len, &data_cpl, sizeof(data_cpl));
+ if (err < 0) {
+ cxgb3i_log_error("%s: tid %u, cp %lu/%u failed %d.\n",
+ csk->cdev->ports[csk->port_id]->name,
+ csk->tid, sizeof(data_cpl), skb->len, err);
+ goto abort_conn;
+ }
+ data_len = ntohs(data_cpl.len);
+ cxgb3i_log_debug(1 << CXGBI_DBG_DDP | 1 << CXGBI_DBG_PDU_RX,
+ "skb 0x%p, pdu not ddp'ed %u/%u, status 0x%x.\n",
+ skb, data_len, cxgbi_skcb_rx_pdulen(skb), status);
+ len += sizeof(data_cpl) + data_len;
+ } else if (status & (1 << CPL_RX_DDP_STATUS_DDP_SHIFT))
+ cxgbi_skcb_set_flag(skb, SKCBF_RX_DATA_DDPD);
+
+ csk->rcv_nxt = ntohl(ddp_cpl.seq) + cxgbi_skcb_rx_pdulen(skb);
+ __pskb_trim(skb, len);
+ __skb_queue_tail(&csk->receive_queue, skb);
+ cxgbi_conn_pdu_ready(csk);
+
+ spin_unlock_bh(&csk->lock);
+ return 0;
+
+abort_conn:
+ send_abort_req(csk);
+discard:
+ spin_unlock_bh(&csk->lock);
+ __kfree_skb(skb);
+ return 0;
+}
+
+/*
+ * Process TX_DATA_ACK CPL messages: -> host
+ * Process an acknowledgment of WR completion. Advance snd_una and send the
+ * next batch of work requests from the write queue.
+ */
+static int do_wr_ack(struct t3cdev *cdev, struct sk_buff *skb, void *ctx)
+{
+ struct cxgbi_sock *csk = ctx;
+ struct cpl_wr_ack *hdr = cplhdr(skb);
+
+ cxgb3i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_PDU_RX,
+ "csk 0x%p,%u,0x%lx,%u, cr %u.\n",
+ csk, csk->state, csk->flags, csk->tid,
+ ntohs(hdr->credits));
+
+ cxgbi_sock_rcv_wr_ack(csk, ntohs(hdr->credits), ntohl(hdr->snd_una), 1);
+ __kfree_skb(skb);
+ return 0;
+}
+
+/*
+ * for each connection, pre-allocate skbs needed for close/abort requests. So
+ * that we can service the request right away.
+ */
+static int alloc_cpls(struct cxgbi_sock *csk)
+{
+ csk->cpl_close = alloc_cpl(sizeof(struct cpl_close_con_req), 0,
+ GFP_KERNEL);
+ if (!csk->cpl_close)
+ return -ENOMEM;
+ csk->cpl_abort_req = alloc_cpl(sizeof(struct cpl_abort_req), 0,
+ GFP_KERNEL);
+ if (!csk->cpl_abort_req)
+ goto free_cpl_skbs;
+
+ csk->cpl_abort_rpl = alloc_cpl(sizeof(struct cpl_abort_rpl), 0,
+ GFP_KERNEL);
+ if (!csk->cpl_abort_rpl)
+ goto free_cpl_skbs;
+
+ return 0;
+
+free_cpl_skbs:
+ cxgbi_sock_free_cpl_skbs(csk);
+ return -ENOMEM;
+}
+
+/**
+ * release_offload_resources - release offload resource
+ * @c3cn: the offloaded iscsi tcp connection.
+ * Release resources held by an offload connection (TID, L2T entry, etc.)
+ */
+static void l2t_put(struct cxgbi_sock *csk)
+{
+ struct t3cdev *t3dev = (struct t3cdev *)csk->cdev->lldev;
+
+ if (csk->l2t) {
+ l2t_release(L2DATA(t3dev), csk->l2t);
+ csk->l2t = NULL;
+ cxgbi_sock_put(csk);
+ }
+}
+
+static void release_offload_resources(struct cxgbi_sock *csk)
+{
+ struct t3cdev *t3dev = (struct t3cdev *)csk->cdev->lldev;
+
+ cxgb3i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx,%u.\n",
+ csk, csk->state, csk->flags, csk->tid);
+
+ csk->rss_qid = 0;
+ cxgbi_sock_free_cpl_skbs(csk);
+
+ if (csk->wr_cred != csk->wr_max_cred) {
+ cxgbi_sock_purge_wr_queue(csk);
+ cxgbi_sock_reset_wr_list(csk);
+ }
+ l2t_put(csk);
+ if (cxgbi_sock_flag(csk, CTPF_HAS_ATID))
+ free_atid(csk);
+ else if (cxgbi_sock_flag(csk, CTPF_HAS_TID)) {
+ cxgb3_remove_tid(t3dev, (void *)csk, csk->tid);
+ cxgbi_sock_clear_flag(csk, CTPF_HAS_TID);
+ cxgbi_sock_put(csk);
+ }
+ csk->dst = NULL;
+ csk->cdev = NULL;
+}
+
+static int init_act_open(struct cxgbi_sock *csk)
+{
+ struct dst_entry *dst = csk->dst;
+ struct cxgbi_device *cdev = csk->cdev;
+ struct t3cdev *t3dev = (struct t3cdev *)cdev->lldev;
+ struct net_device *ndev = cdev->ports[csk->port_id];
+ struct sk_buff *skb = NULL;
+
+ cxgb3i_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx.\n",
+ csk, csk->state, csk->flags);
+
+ csk->rss_qid = 0;
+ csk->l2t = t3_l2t_get(t3dev, dst->neighbour, ndev);
+ if (!csk->l2t) {
+ cxgb3i_log_error("NO l2t available.\n");
+ return -EINVAL;
+ }
+ cxgbi_sock_get(csk);
+
+ csk->atid = cxgb3_alloc_atid(t3dev, &t3_client, csk);
+ if (csk->atid < 0) {
+ cxgb3i_log_error("NO atid available.\n");
+ goto rel_resource;
+ }
+ cxgbi_sock_set_flag(csk, CTPF_HAS_ATID);
+ cxgbi_sock_get(csk);
+
+ skb = alloc_cpl(sizeof(struct cpl_act_open_req), 0, GFP_KERNEL);
+ if (!skb)
+ goto rel_resource;
+ skb->sk = (struct sock *)csk;
+ set_arp_failure_handler(skb, act_open_arp_failure);
+
+ csk->wr_max_cred = csk->wr_cred = T3C_DATA(t3dev)->max_wrs - 1;
+ csk->wr_una_cred = 0;
+ csk->mss_idx = cxgbi_sock_select_mss(csk, dst_mtu(dst));
+ cxgbi_sock_reset_wr_list(csk);
+ csk->err = 0;
+
+ cxgbi_sock_set_state(csk, CTP_ACTIVE_OPEN);
+ send_act_open_req(csk, skb, csk->l2t);
+ return 0;
+
+rel_resource:
+ if (skb)
+ __kfree_skb(skb);
+ return -EINVAL;
+}
+
+cxgb3_cpl_handler_func cxgb3i_cpl_handlers[NUM_CPL_CMDS] = {
+ [CPL_ACT_ESTABLISH] = do_act_establish,
+ [CPL_ACT_OPEN_RPL] = do_act_open_rpl,
+ [CPL_PEER_CLOSE] = do_peer_close,
+ [CPL_ABORT_REQ_RSS] = do_abort_req,
+ [CPL_ABORT_RPL_RSS] = do_abort_rpl,
+ [CPL_CLOSE_CON_RPL] = do_close_con_rpl,
+ [CPL_TX_DMA_ACK] = do_wr_ack,
+ [CPL_ISCSI_HDR] = do_iscsi_hdr,
+};
+
+/**
+ * cxgb3i_ofld_init - allocate and initialize resources for each adapter found
+ * @cdev: cxgbi adapter
+ */
+int cxgb3i_ofld_init(struct cxgbi_device *cdev)
+{
+ struct t3cdev *t3dev = (struct t3cdev *)cdev->lldev;
+ struct adap_ports port;
+ struct ofld_page_info rx_page_info;
+ unsigned int wr_len;
+ int rc;
+
+ if (t3dev->ctl(t3dev, GET_WR_LEN, &wr_len) < 0 ||
+ t3dev->ctl(t3dev, GET_PORTS, &port) < 0 ||
+ t3dev->ctl(t3dev, GET_RX_PAGE_INFO, &rx_page_info) < 0) {
+ cxgb3i_log_warn("t3 0x%p, offload up, ioctl failed.\n",
+ t3dev);
+ return -EINVAL;
+ }
+
+ if (cxgb3i_max_connect > CXGBI_MAX_CONN)
+ cxgb3i_max_connect = CXGBI_MAX_CONN;
+
+ rc = cxgbi_device_portmap_create(cdev, cxgb3i_sport_base,
+ cxgb3i_max_connect);
+ if (rc < 0)
+ return rc;
+
+ init_wr_tab(wr_len);
+ cdev->csk_release_offload_resources = release_offload_resources;
+ cdev->csk_push_tx_frames = push_tx_frames;
+ cdev->csk_send_abort_req = send_abort_req;
+ cdev->csk_send_close_req = send_close_req;
+ cdev->csk_send_rx_credits = send_rx_credits;
+ cdev->csk_alloc_cpls = alloc_cpls;
+ cdev->csk_init_act_open = init_act_open;
+
+ cxgb3i_log_info("cdev 0x%p, offload up, added.\n", cdev);
+ return 0;
+}
+
+/*
+ * functions to program the pagepod in h/w
+ */
+static inline void ulp_mem_io_set_hdr(struct sk_buff *skb, unsigned int addr)
+{
+ struct ulp_mem_io *req = (struct ulp_mem_io *)skb->head;
+
+ memset(req, 0, sizeof(*req));
+
+ req->wr.wr_hi = htonl(V_WR_OP(FW_WROPCODE_BYPASS));
+ req->cmd_lock_addr = htonl(V_ULP_MEMIO_ADDR(addr >> 5) |
+ V_ULPTX_CMD(ULP_MEM_WRITE));
+ req->len = htonl(V_ULP_MEMIO_DATA_LEN(PPOD_SIZE >> 5) |
+ V_ULPTX_NFLITS((PPOD_SIZE >> 3) + 1));
+}
+
+static int ddp_set_map(struct cxgbi_sock *csk, struct cxgbi_pagepod_hdr *hdr,
+ unsigned int idx, unsigned int npods,
+ struct cxgbi_gather_list *gl)
+{
+ struct cxgbi_device *cdev = csk->cdev;
+ struct cxgbi_ddp_info *ddp = cdev->ddp;
+ unsigned int pm_addr = (idx << PPOD_SIZE_SHIFT) + ddp->llimit;
+ int i;
+
+ cxgb3i_log_debug(1 << CXGBI_DBG_DDP,
+ "csk 0x%p, idx %u, npods %u, gl 0x%p.\n",
+ csk, idx, npods, gl);
+
+ for (i = 0; i < npods; i++, idx++, pm_addr += PPOD_SIZE) {
+ struct sk_buff *skb = ddp->gl_skb[idx];
+
+ /* hold on to the skb until we clear the ddp mapping */
+ skb_get(skb);
+
+ ulp_mem_io_set_hdr(skb, pm_addr);
+ cxgbi_ddp_ppod_set((struct cxgbi_pagepod *)(skb->head +
+ sizeof(struct ulp_mem_io)),
+ hdr, gl, i * PPOD_PAGES_MAX);
+ skb->priority = CPL_PRIORITY_CONTROL;
+ cxgb3_ofld_send(cdev->lldev, skb);
+ }
+ return 0;
+}
+
+static void ddp_clear_map(struct cxgbi_hba *chba, unsigned int tag,
+ unsigned int idx, unsigned int npods)
+{
+ struct cxgbi_device *cdev = chba->cdev;
+ struct cxgbi_ddp_info *ddp = cdev->ddp;
+ unsigned int pm_addr = (idx << PPOD_SIZE_SHIFT) + ddp->llimit;
+ int i;
+
+ cxgb3i_log_debug(1 << CXGBI_DBG_DDP,
+ "cdev 0x%p, idx %u, npods %u, tag 0x%x.\n",
+ cdev, idx, npods, tag);
+
+ for (i = 0; i < npods; i++, idx++, pm_addr += PPOD_SIZE) {
+ struct sk_buff *skb = ddp->gl_skb[idx];
+
+ if (!skb) {
+ cxgb3i_log_error("tag 0x%x, 0x%x, %d/%u, skb NULL.\n",
+ tag, idx, i, npods);
+ continue;
+ }
+ ddp->gl_skb[idx] = NULL;
+ memset(skb->head + sizeof(struct ulp_mem_io), 0, PPOD_SIZE);
+ ulp_mem_io_set_hdr(skb, pm_addr);
+ skb->priority = CPL_PRIORITY_CONTROL;
+ cxgb3_ofld_send(cdev->lldev, skb);
+ }
+}
+
+static void ddp_free_gl_skb(struct cxgbi_ddp_info *ddp, int idx, int cnt)
+{
+ int i;
+
+ cxgb3i_log_debug(1 << CXGBI_DBG_DDP,
+ "ddp 0x%p, idx %d, cnt %d.\n",
+ ddp, idx, cnt);
+
+ for (i = 0; i < cnt; i++, idx++)
+ if (ddp->gl_skb[idx]) {
+ kfree_skb(ddp->gl_skb[idx]);
+ ddp->gl_skb[idx] = NULL;
+ }
+}
+
+static int ddp_alloc_gl_skb(struct cxgbi_ddp_info *ddp, int idx,
+ int cnt, gfp_t gfp)
+{
+ int i;
+
+ cxgb3i_log_debug(1 << CXGBI_DBG_DDP,
+ "ddp 0x%p, idx %d, cnt %d.\n",
+ ddp, idx, cnt);
+
+ for (i = 0; i < cnt; i++) {
+ struct sk_buff *skb = alloc_cpl(sizeof(struct ulp_mem_io) +
+ PPOD_SIZE, 0, gfp);
+ if (skb) {
+ ddp->gl_skb[idx + i] = skb;
+ } else {
+ ddp_free_gl_skb(ddp, idx, i);
+ return -ENOMEM;
+ }
+ }
+ return 0;
+}
+
+static int ddp_setup_conn_pgidx(struct cxgbi_sock *csk,
+ unsigned int tid, int pg_idx, bool reply)
+{
+ struct sk_buff *skb = alloc_cpl(sizeof(struct cpl_set_tcb_field), 0,
+ GFP_KERNEL);
+ struct cpl_set_tcb_field *req;
+ u64 val = pg_idx < DDP_PGIDX_MAX ? pg_idx : 0;
+
+ cxgb3i_log_debug(1 << CXGBI_DBG_DDP,
+ "csk 0x%p, tid %u, pg_idx %d.\n",
+ csk, tid, pg_idx);
+
+ if (!skb)
+ return -ENOMEM;
+
+ /* set up ulp submode and page size */
+ req = (struct cpl_set_tcb_field *)skb->head;
+ req->wr.wr_hi = htonl(V_WR_OP(FW_WROPCODE_FORWARD));
+ OPCODE_TID(req) = htonl(MK_OPCODE_TID(CPL_SET_TCB_FIELD, tid));
+ req->reply = V_NO_REPLY(reply ? 0 : 1);
+ req->cpu_idx = 0;
+ req->word = htons(31);
+ req->mask = cpu_to_be64(0xF0000000);
+ req->val = cpu_to_be64(val << 28);
+ skb->priority = CPL_PRIORITY_CONTROL;
+
+ cxgb3_ofld_send(csk->cdev->lldev, skb);
+ return 0;
+}
+
+/**
+ * cxgb3i_setup_conn_digest - setup conn. digest setting
+ * @csk: cxgb tcp socket
+ * @tid: connection id
+ * @hcrc: header digest enabled
+ * @dcrc: data digest enabled
+ * @reply: request reply from h/w
+ * set up the iscsi digest settings for a connection identified by tid
+ */
+static int ddp_setup_conn_digest(struct cxgbi_sock *csk, unsigned int tid,
+ int hcrc, int dcrc, int reply)
+{
+ struct sk_buff *skb = alloc_cpl(sizeof(struct cpl_set_tcb_field), 0,
+ GFP_KERNEL);
+ struct cpl_set_tcb_field *req;
+ u64 val = (hcrc ? 1 : 0) | (dcrc ? 2 : 0);
+
+ cxgb3i_log_debug(1 << CXGBI_DBG_DDP,
+ "csk 0x%p, tid %u, crc %d,%d.\n",
+ csk, tid, hcrc, dcrc);
+ if (!skb)
+ return -ENOMEM;
+
+ /* set up ulp submode and page size */
+ req = (struct cpl_set_tcb_field *)skb->head;
+ req->wr.wr_hi = htonl(V_WR_OP(FW_WROPCODE_FORWARD));
+ OPCODE_TID(req) = htonl(MK_OPCODE_TID(CPL_SET_TCB_FIELD, tid));
+ req->reply = V_NO_REPLY(reply ? 0 : 1);
+ req->cpu_idx = 0;
+ req->word = htons(31);
+ req->mask = cpu_to_be64(0x0F000000);
+ req->val = cpu_to_be64(val << 24);
+ skb->priority = CPL_PRIORITY_CONTROL;
+
+ cxgb3_ofld_send(csk->cdev->lldev, skb);
+ return 0;
+}
+
+/**
+ * t3_ddp_cleanup - release the cxgb3 adapter's ddp resource
+ * @cdev: cxgb3i adapter
+ * release all the resource held by the ddp pagepod manager for a given
+ * adapter if needed
+ */
+
+static void t3_ddp_cleanup(struct cxgbi_device *cdev)
+{
+ struct t3cdev *tdev = (struct t3cdev *)cdev->lldev;
+
+ if (cxgbi_ddp_cleanup(cdev)) {
+ cxgb3i_log_info("t3dev 0x%p, ulp_iscsi no more user.\n",
+ tdev);
+ tdev->ulp_iscsi = NULL;
+ }
+}
+
+/**
+ * ddp_init - initialize the cxgb3 adapter's ddp resource
+ * @cdev: cxgb3i adapter
+ * initialize the ddp pagepod manager for a given adapter
+ */
+static int cxgb3i_ddp_init(struct cxgbi_device *cdev)
+{
+ struct t3cdev *tdev = (struct t3cdev *)cdev->lldev;
+ struct cxgbi_ddp_info *ddp = tdev->ulp_iscsi;
+ struct ulp_iscsi_info uinfo;
+ unsigned int pgsz_factor[4];
+ int err;
+
+ if (ddp) {
+ kref_get(&ddp->refcnt);
+ cxgb3i_log_warn("t3dev 0x%p, ddp 0x%p already set up.\n",
+ tdev, tdev->ulp_iscsi);
+ cdev->ddp = ddp;
+ return -EALREADY;
+ }
+
+ err = tdev->ctl(tdev, ULP_ISCSI_GET_PARAMS, &uinfo);
+ if (err < 0) {
+ cxgb3i_log_error("%s, failed to get iscsi param err=%d.\n",
+ tdev->name, err);
+ return err;
+ }
+
+ err = cxgbi_ddp_init(cdev, uinfo.llimit, uinfo.ulimit,
+ uinfo.max_txsz, uinfo.max_rxsz);
+ if (err < 0)
+ return err;
+
+ ddp = cdev->ddp;
+
+ uinfo.tagmask = ddp->idx_mask << PPOD_IDX_SHIFT;
+ cxgbi_ddp_page_size_factor(pgsz_factor);
+ uinfo.ulimit = uinfo.llimit + (ddp->nppods << PPOD_SIZE_SHIFT);
+
+ err = tdev->ctl(tdev, ULP_ISCSI_SET_PARAMS, &uinfo);
+ if (err < 0) {
+ cxgb3i_log_warn("%s unable to set iscsi param err=%d, "
+ "ddp disabled.\n", tdev->name, err);
+ cxgbi_ddp_cleanup(cdev);
+ return err;
+ }
+ tdev->ulp_iscsi = ddp;
+
+ cdev->csk_ddp_free_gl_skb = ddp_free_gl_skb;
+ cdev->csk_ddp_alloc_gl_skb = ddp_alloc_gl_skb;
+ cdev->csk_ddp_setup_digest = ddp_setup_conn_digest;
+ cdev->csk_ddp_setup_pgidx = ddp_setup_conn_pgidx;
+ cdev->csk_ddp_set = ddp_set_map;
+ cdev->csk_ddp_clear = ddp_clear_map;
+
+ cxgb3i_log_info("tdev 0x%p, nppods %u, bits %u, mask 0x%x,0x%x "
+ "pkt %u/%u, %u/%u.\n",
+ tdev, ddp->nppods, ddp->idx_bits, ddp->idx_mask,
+ ddp->rsvd_tag_mask, ddp->max_txsz, uinfo.max_txsz,
+ ddp->max_rxsz, uinfo.max_rxsz);
+ return 0;
+}
+
+static void cxgb3i_dev_close(struct t3cdev *t3dev)
+{
+ struct cxgbi_device *cdev = cxgbi_device_find_by_lldev(t3dev);
+
+ if (!cdev || cdev->flags & CXGBI_FLAG_ADAPTER_RESET) {
+ cxgb3i_log_info("t3 0x%p close, f 0x%x.\n",
+ cdev, cdev ? cdev->flags : 0);
+ return;
+ }
+
+ cxgbi_device_unregister(cdev);
+}
+
+/**
+ * cxgb3i_dev_open - init a t3 adapter structure and any h/w settings
+ * @t3dev: t3cdev adapter
+ */
+static void cxgb3i_dev_open(struct t3cdev *t3dev)
+{
+ static int vers_printed;
+
+ struct cxgbi_device *cdev = cxgbi_device_find_by_lldev(t3dev);
+ struct adapter *adapter = tdev2adap(t3dev);
+ int i, err;
+
+ if (!vers_printed) {
+ printk(KERN_INFO "%s", version);
+ vers_printed = 1;
+ }
+
+ if (cdev) {
+ cxgb3i_log_info("t3 0x%p, updating.\n", cdev);
+ return;
+ }
+
+ cdev = cxgbi_device_register(0, adapter->params.nports);
+ if (!cdev) {
+ cxgb3i_log_warn("t3 device 0x%p register failed.\n", t3dev);
+ return;
+ }
+
+ cdev->flags = CXGBI_FLAG_DEV_T3;
+ cdev->lldev = t3dev;
+ cdev->pdev = adapter->pdev;
+ cdev->ports = adapter->port;
+ cdev->nports = adapter->params.nports;
+ cdev->mtus = adapter->params.mtus;
+ cdev->nmtus = NMTUS;
+ cdev->snd_win = cxgb3i_snd_win;
+ cdev->rcv_win = cxgb3i_rcv_win;
+ cdev->rx_credit_thres = cxgb3i_rx_credit_thres;
+ cdev->skb_tx_rsvd = CXGB3I_TX_HEADER_LEN;
+ cdev->skb_rx_extra = sizeof(struct cpl_iscsi_hdr_norss);
+ cdev->dev_ddp_cleanup = t3_ddp_cleanup;
+ cdev->itp = &cxgb3i_iscsi_transport;
+
+ err = cxgb3i_ddp_init(cdev);
+ if (err) {
+ cxgb3i_log_info("t3 0x%p ddp init failed\n", cdev);
+ goto err_out;
+ }
+
+ err = cxgb3i_ofld_init(cdev);
+ if (err) {
+ cxgb3i_log_info("t3 0x%p offload init failed\n", cdev);
+ goto err_out;
+ }
+
+ err = cxgbi_hbas_add(cdev, CXGB3I_MAX_LUN, CXGBI_MAX_CONN,
+ &cxgb3i_host_template, cxgb3i_stt);
+ if (err)
+ goto err_out;
+
+ for (i = 0; i < cdev->nports; i++)
+ cdev->hbas[i]->ipv4addr =
+ cxgb3i_get_private_ipv4addr(cdev->ports[i]);
+
+ cxgb3i_log_info("cdev 0x%p, f 0x%x, t3dev 0x%p open, err %d.\n",
+ cdev, cdev ? cdev->flags : 0, t3dev, err);
+ return;
+
+err_out:
+ cxgbi_device_unregister(cdev);
+}
+
+static void cxgb3i_dev_event_handler(struct t3cdev *t3dev, u32 event, u32 port)
+{
+ struct cxgbi_device *cdev = cxgbi_device_find_by_lldev(t3dev);
+
+ cxgb3i_log_debug(CXGBI_DBG_TOE,
+ "t3dev 0x%p, cdev 0x%p, event 0x%x, port 0x%x.\n",
+ t3dev, cdev, event, port);
+ if (!cdev)
+ return;
+
+ switch (event) {
+ case OFFLOAD_STATUS_DOWN:
+ cdev->flags |= CXGBI_FLAG_ADAPTER_RESET;
+ break;
+ case OFFLOAD_STATUS_UP:
+ cdev->flags &= ~CXGBI_FLAG_ADAPTER_RESET;
+ break;
+ }
+}
+
+/**
+ * cxgb3i_init_module - module init entry point
+ *
+ * initialize any driver wide global data structures and register itself
+ * with the cxgb3 module
+ */
+static int __init cxgb3i_init_module(void)
+{
+ int rc;
+
+ rc = cxgbi_iscsi_init(&cxgb3i_iscsi_transport, &cxgb3i_stt);
+ if (rc < 0)
+ return rc;
+
+ cxgb3_register_client(&t3_client);
+ return 0;
+}
+
+/**
+ * cxgb3i_exit_module - module cleanup/exit entry point
+ *
+ * go through the driver hba list and for each hba, release any resource held.
+ * and unregisters iscsi transport and the cxgb3 module
+ */
+static void __exit cxgb3i_exit_module(void)
+{
+ cxgb3_unregister_client(&t3_client);
+ cxgbi_device_unregister_all(CXGBI_FLAG_DEV_T3);
+ cxgbi_iscsi_cleanup(&cxgb3i_iscsi_transport, &cxgb3i_stt);
+}
+
+module_init(cxgb3i_init_module);
+module_exit(cxgb3i_exit_module);
diff --git a/drivers/scsi/cxgbi/cxgb3i/cxgb3i.h b/drivers/scsi/cxgbi/cxgb3i/cxgb3i.h
new file mode 100644
index 0000000..5f5e339
--- /dev/null
+++ b/drivers/scsi/cxgbi/cxgb3i/cxgb3i.h
@@ -0,0 +1,51 @@
+/*
+ * cxgb3i.h: Chelsio S3xx iSCSI driver.
+ *
+ * Copyright (c) 2008 Chelsio Communications, Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation.
+ *
+ * Written by: Karen Xie (kxie@chelsio.com)
+ */
+
+#ifndef __CXGB3I_H__
+#define __CXGB3I_H__
+
+#define CXGB3I_SCSI_HOST_QDEPTH 1024
+#define CXGB3I_MAX_LUN 512
+#define ISCSI_PDU_NONPAYLOAD_MAX \
+ (sizeof(struct iscsi_hdr) + ISCSI_MAX_AHS_SIZE + 2*ISCSI_DIGEST_SIZE)
+
+/*for TX: a skb must have a headroom of at least TX_HEADER_LEN bytes */
+#define CXGB3I_TX_HEADER_LEN \
+ (sizeof(struct tx_data_wr) + sizeof(struct sge_opaque_hdr))
+
+extern cxgb3_cpl_handler_func cxgb3i_cpl_handlers[NUM_CPL_CMDS];
+
+#define cxgb3i_get_private_ipv4addr(ndev) \
+ (((struct port_info *)(netdev_priv(ndev)))->iscsi_ipv4addr)
+#define cxgb3i_set_private_ipv4addr(ndev, addr) \
+ (((struct port_info *)(netdev_priv(ndev)))->iscsi_ipv4addr) = addr
+
+struct cpl_iscsi_hdr_norss {
+ union opcode_tid ot;
+ u16 pdu_len_ddp;
+ u16 len;
+ u32 seq;
+ u16 urg;
+ u8 rsvd;
+ u8 status;
+};
+
+struct cpl_rx_data_ddp_norss {
+ union opcode_tid ot;
+ u16 urg;
+ u16 len;
+ u32 seq;
+ u32 nxt_seq;
+ u32 ulp_crc;
+ u32 ddp_status;
+};
+#endif
^ permalink raw reply related
* [PATCH v2 1/3] libcxgbi: common library for cxgb3i and cxgb4i
From: kxie @ 2010-08-13 15:19 UTC (permalink / raw)
To: netdev, linux-scsi, open-iscsi
Cc: rranjan, kxie, James.Bottomley, michaelc, davem
[PATCH v2 1/3] libcxgbi: common library for cxgb3i and cxgb4i
From: Karen Xie <kxie@chelsio.com>
Extracted common functions to libcxgbi.
Signed-off-by: Karen Xie <kxie@chelsio.com>
---
drivers/scsi/cxgbi/libcxgbi.c | 2658 +++++++++++++++++++++++++++++++++++++++++
drivers/scsi/cxgbi/libcxgbi.h | 754 ++++++++++++
2 files changed, 3412 insertions(+), 0 deletions(-)
create mode 100644 drivers/scsi/cxgbi/libcxgbi.c
create mode 100644 drivers/scsi/cxgbi/libcxgbi.h
diff --git a/drivers/scsi/cxgbi/libcxgbi.c b/drivers/scsi/cxgbi/libcxgbi.c
new file mode 100644
index 0000000..0087ff7
--- /dev/null
+++ b/drivers/scsi/cxgbi/libcxgbi.c
@@ -0,0 +1,2658 @@
+/*
+ * libcxgbi.c: Chelsio common library for T3/T4 iSCSI driver.
+ *
+ * Copyright (c) 2010 Chelsio Communications, Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation.
+ *
+ * Written by: Karen Xie (kxie@chelsio.com)
+ * Written by: Rakesh Ranjan (rranjan@chelsio.com)
+ */
+
+#include <linux/skbuff.h>
+#include <linux/crypto.h>
+#include <linux/scatterlist.h>
+#include <linux/pci.h>
+#include <scsi/scsi.h>
+#include <scsi/scsi_cmnd.h>
+#include <scsi/scsi_host.h>
+#include <linux/if_vlan.h>
+#include <linux/inet.h>
+#include <net/dst.h>
+#include <net/route.h>
+#include <linux/inetdevice.h> /* ip_dev_find */
+#include <net/tcp.h>
+
+static unsigned int dbg_level;
+#include "libcxgbi.h"
+
+#define DRV_MODULE_NAME "libcxgbi"
+#define DRV_MODULE_DESC "Chelsio iSCSI driver library"
+#define DRV_MODULE_VERSION "0.9.0"
+#define DRV_MODULE_RELDATE "Jun. 2010"
+
+MODULE_AUTHOR("Chelsio Communications, Inc.");
+MODULE_DESCRIPTION(DRV_MODULE_DESC);
+MODULE_VERSION(DRV_MODULE_VERSION);
+MODULE_LICENSE("GPL");
+
+module_param(dbg_level, uint, 0644);
+MODULE_PARM_DESC(dbg_level, "libiscsi debug level (default=0)");
+
+#define libcxgbi_log_error cxgbi_dbg_error
+#define libcxgbi_log_warn cxgbi_dbg_warn
+#define libcxgbi_log_info cxgbi_dbg_info
+#define libcxgbi_log_debug(flag, fmt, args...) \
+ cxgbi_dbg_print(flag, fmt, ## args)
+
+/*
+ * cxgbi device management
+ * maintains a list of the cxgbi devices
+ */
+static LIST_HEAD(cdev_list);
+static DEFINE_MUTEX(cdev_mutex);
+
+int cxgbi_device_portmap_create(struct cxgbi_device *cdev, unsigned int base,
+ unsigned int max_conn)
+{
+ struct cxgbi_ports_map *pmap = &cdev->pmap;
+
+ pmap->port_csk = cxgbi_alloc_big_mem(max_conn *
+ sizeof(struct cxgbi_sock *),
+ GFP_KERNEL);
+ if (!pmap->port_csk) {
+ libcxgbi_log_warn("cdev 0x%p, portmap OOM %u.\n",
+ cdev, max_conn);
+ return -ENOMEM;
+ }
+
+ pmap->max_connect = max_conn;
+ pmap->sport_base = base;
+ spin_lock_init(&pmap->lock);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(cxgbi_device_portmap_create);
+
+void cxgbi_device_portmap_cleanup(struct cxgbi_device *cdev)
+{
+ struct cxgbi_ports_map *pmap = &cdev->pmap;
+ struct cxgbi_sock *csk;
+ int i;
+
+ for (i = 0; i < pmap->max_connect; i++) {
+ if (pmap->port_csk[i]) {
+ csk = pmap->port_csk[i];
+ pmap->port_csk[i] = NULL;
+ libcxgbi_log_debug(1 << CXGBI_DBG_SOCK,
+ "csk 0x%p, cdev 0x%p, offload down.\n",
+ csk, cdev);
+ spin_lock_bh(&csk->lock);
+ cxgbi_sock_set_flag(csk, CTPF_OFFLOAD_DOWN);
+ cxgbi_sock_closed(csk);
+ spin_unlock_bh(&csk->lock);
+ cxgbi_sock_put(csk);
+ }
+ }
+}
+EXPORT_SYMBOL_GPL(cxgbi_device_portmap_cleanup);
+
+static inline void cxgbi_device_destroy(struct cxgbi_device *cdev)
+{
+ libcxgbi_log_debug(1 << CXGBI_DBG_DEV,
+ "cdev 0x%p, p# %u.\n", cdev, cdev->nports);
+ cxgbi_hbas_remove(cdev);
+ cxgbi_device_portmap_cleanup(cdev);
+ if (cdev->dev_ddp_cleanup)
+ cdev->dev_ddp_cleanup(cdev);
+ else
+ cxgbi_ddp_cleanup(cdev);
+ if (cdev->ddp)
+ cxgbi_ddp_cleanup(cdev);
+ if (cdev->pmap.max_connect)
+ cxgbi_free_big_mem(cdev->pmap.port_csk);
+ kfree(cdev);
+}
+
+struct cxgbi_device *cxgbi_device_register(unsigned int extra,
+ unsigned int nports)
+{
+ struct cxgbi_device *cdev;
+
+ cdev = kzalloc(sizeof(*cdev) + extra + nports *
+ (sizeof(struct cxgbi_hba *) +
+ sizeof(struct net_device *)),
+ GFP_KERNEL);
+ if (!cdev) {
+ libcxgbi_log_warn("nport %d, OOM.\n", nports);
+ return NULL;
+ }
+ cdev->ports = (struct net_device **)(cdev + 1);
+ cdev->hbas = (struct cxgbi_hba **)(((char*)cdev->ports) + nports *
+ sizeof(struct net_device *));
+ if (extra)
+ cdev->dd_data = ((char *)cdev->hbas) +
+ nports * sizeof(struct cxgbi_hba *);
+ spin_lock_init(&cdev->pmap.lock);
+
+ mutex_lock(&cdev_mutex);
+ list_add_tail(&cdev->list_head, &cdev_list);
+ mutex_unlock(&cdev_mutex);
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_DEV,
+ "cdev 0x%p, p# %u.\n", cdev, nports);
+ return cdev;
+}
+EXPORT_SYMBOL_GPL(cxgbi_device_register);
+
+void cxgbi_device_unregister(struct cxgbi_device *cdev)
+{
+ libcxgbi_log_debug(1 << CXGBI_DBG_DEV,
+ "cdev 0x%p, p# %u,%s.\n", cdev,
+ cdev->nports, cdev->nports ? cdev->ports[0]->name : "");
+ mutex_lock(&cdev_mutex);
+ list_del(&cdev->list_head);
+ mutex_unlock(&cdev_mutex);
+ cxgbi_device_destroy(cdev);
+}
+EXPORT_SYMBOL_GPL(cxgbi_device_unregister);
+
+void cxgbi_device_unregister_all(unsigned int flag)
+{
+ struct cxgbi_device *cdev, *tmp;
+
+ mutex_lock(&cdev_mutex);
+ list_for_each_entry_safe(cdev, tmp, &cdev_list, list_head) {
+ if ((cdev->flags & flag) == flag) {
+ libcxgbi_log_debug(1 << CXGBI_DBG_DEV,
+ "cdev 0x%p, p# %u,%s.\n",
+ cdev, cdev->nports, cdev->nports ?
+ cdev->ports[0]->name : "");
+ list_del(&cdev->list_head);
+ cxgbi_device_destroy(cdev);
+ }
+ }
+ mutex_unlock(&cdev_mutex);
+}
+EXPORT_SYMBOL_GPL(cxgbi_device_unregister_all);
+
+struct cxgbi_device *cxgbi_device_find_by_lldev(void *lldev)
+{
+ struct cxgbi_device *cdev, *tmp;
+
+ mutex_lock(&cdev_mutex);
+ list_for_each_entry_safe(cdev, tmp, &cdev_list, list_head) {
+ if (cdev->lldev == lldev) {
+ mutex_unlock(&cdev_mutex);
+ return cdev;
+ }
+ }
+ mutex_unlock(&cdev_mutex);
+ libcxgbi_log_debug(1 << CXGBI_DBG_DEV,
+ "lldev 0x%p, NO match found.\n", lldev);
+ return NULL;
+}
+EXPORT_SYMBOL_GPL(cxgbi_device_find_by_lldev);
+
+static struct cxgbi_device *cxgbi_device_find_by_netdev(struct net_device *ndev,
+ int *port)
+{
+ struct cxgbi_device *cdev, *tmp;
+ int i;
+
+ if (ndev->priv_flags & IFF_802_1Q_VLAN)
+ ndev = vlan_dev_real_dev(ndev);
+
+ mutex_lock(&cdev_mutex);
+ list_for_each_entry_safe(cdev, tmp, &cdev_list, list_head) {
+ for (i = 0; i < cdev->nports; i++) {
+ if (ndev == cdev->ports[i]) {
+ mutex_unlock(&cdev_mutex);
+ if (port)
+ *port = i;
+ return cdev;
+ }
+ }
+ }
+ mutex_unlock(&cdev_mutex);
+ libcxgbi_log_debug(1 << CXGBI_DBG_DEV,
+ "ndev 0x%p, %s, NO match found.\n", ndev, ndev->name);
+ return NULL;
+}
+
+struct cxgbi_hba *cxgbi_hba_find_by_netdev(struct net_device *dev,
+ struct cxgbi_device *cdev)
+{
+ int i;
+
+ if (dev->priv_flags & IFF_802_1Q_VLAN)
+ dev = vlan_dev_real_dev(dev);
+
+ for (i = 0; i < cdev->nports; i++) {
+ if (cdev->hbas[i]->ndev == dev)
+ return cdev->hbas[i];
+ }
+ libcxgbi_log_debug(1 << CXGBI_DBG_DEV,
+ "ndev 0x%p, %s, cdev 0x%p, NO match found.\n",
+ dev, dev->name, cdev);
+ return NULL;
+}
+
+void cxgbi_hbas_remove(struct cxgbi_device *cdev)
+{
+ int i;
+ struct cxgbi_hba *chba;
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_DEV,
+ "cdev 0x%p, p#%u.\n", cdev, cdev->nports);
+
+ for (i = 0; i < cdev->nports; i++) {
+ chba = cdev->hbas[i];
+ if (chba) {
+ cdev->hbas[i] = NULL;
+ iscsi_host_remove(chba->shost);
+ pci_dev_put(cdev->pdev);
+ iscsi_host_free(chba->shost);
+ }
+ }
+}
+EXPORT_SYMBOL_GPL(cxgbi_hbas_remove);
+
+int cxgbi_hbas_add(struct cxgbi_device *cdev, unsigned int max_lun,
+ unsigned int max_id, struct scsi_host_template *sht,
+ struct scsi_transport_template *stt)
+{
+ struct cxgbi_hba *chba;
+ struct Scsi_Host *shost;
+ int i, err;
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_DEV, "cdev 0x%p, p#%u.\n",
+ cdev, cdev->nports);
+
+ for (i = 0; i < cdev->nports; i++) {
+ shost = iscsi_host_alloc(sht, sizeof(*chba), 1);
+ if (!shost) {
+ libcxgbi_log_info("0x%p, p%d, %s, host alloc failed\n",
+ cdev, i, cdev->ports[i]->name);
+ err = -ENOMEM;
+ goto err_out;
+ }
+
+ shost->transportt = stt;
+ shost->max_lun = max_lun;
+ shost->max_id = max_id;
+ shost->max_channel = 0;
+ shost->max_cmd_len = 16;
+
+ chba = iscsi_host_priv(shost);
+ chba->cdev = cdev;
+ chba->ndev = cdev->ports[i];
+ chba->shost = shost;
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_DEV,
+ "cdev 0x%p, p#%d %s: chba 0x%p\n",
+ cdev, i, cdev->ports[i]->name, chba);
+
+ pci_dev_get(cdev->pdev);
+ err = iscsi_host_add(shost, &cdev->pdev->dev);
+ if (err) {
+ libcxgbi_log_info("cdev 0x%p, p#%d %s, "
+ "host add failed.\n",
+ cdev, i, cdev->ports[i]->name);
+ pci_dev_put(cdev->pdev);
+ scsi_host_put(shost);
+ goto err_out;
+ }
+
+ cdev->hbas[i] = chba;
+ }
+
+ return 0;
+
+err_out:
+ cxgbi_hbas_remove(cdev);
+ return err;
+}
+EXPORT_SYMBOL_GPL(cxgbi_hbas_add);
+
+/*
+ * iSCSI offload
+ *
+ * - source port management
+ * To find a free source port in the port allocation map we use a very simple
+ * rotor scheme to look for the next free port.
+ *
+ * If a source port has been specified make sure that it doesn't collide with
+ * our normal source port allocation map. If it's outside the range of our
+ * allocation/deallocation scheme just let them use it.
+ *
+ * If the source port is outside our allocation range, the caller is
+ * responsible for keeping track of their port usage.
+ */
+static int sock_get_port(struct cxgbi_sock *csk)
+{
+ struct cxgbi_device *cdev = csk->cdev;
+ struct cxgbi_ports_map *pmap = &cdev->pmap;
+ unsigned int start;
+ int idx;
+
+ if (!pmap->max_connect) {
+ libcxgbi_log_error("cdev 0x%p, p#%u %s, NO port map.\n",
+ cdev, csk->port_id,
+ cdev->ports[csk->port_id]->name);
+ return -EADDRNOTAVAIL;
+ }
+
+ if (csk->saddr.sin_port) {
+ libcxgbi_log_error("source port NON-ZERO %u.\n",
+ ntohs(csk->saddr.sin_port));
+ return -EADDRINUSE;
+ }
+
+ spin_lock_bh(&pmap->lock);
+ if (pmap->used >= pmap->max_connect) {
+ spin_unlock_bh(&pmap->lock);
+ libcxgbi_log_info("cdev 0x%p, p#%u %s, ALL ports used.\n",
+ cdev, csk->port_id, cdev->ports[csk->port_id]->name);
+ return -EADDRNOTAVAIL;
+ }
+
+ start = idx = pmap->next;
+ do {
+ if (++idx >= pmap->max_connect)
+ idx = 0;
+ if (!pmap->port_csk[idx]) {
+ pmap->used++;
+ csk->saddr.sin_port =
+ htons(pmap->sport_base + idx);
+ pmap->next = idx;
+ pmap->port_csk[idx] = csk;
+ spin_unlock_bh(&pmap->lock);
+ cxgbi_sock_get(csk);
+ libcxgbi_log_debug(1 << CXGBI_DBG_SOCK,
+ "cdev 0x%p, p#%u %s, p %u, %u.\n",
+ cdev, csk->port_id,
+ cdev->ports[csk->port_id]->name,
+ pmap->sport_base + idx,
+ pmap->next);
+ return 0;
+ }
+ } while (idx != start);
+ spin_unlock_bh(&pmap->lock);
+
+ /* should not happen */
+ libcxgbi_log_warn("cdev 0x%p, p#%u %s, next %u?\n",
+ cdev, csk->port_id, cdev->ports[csk->port_id]->name,
+ pmap->next);
+ return -EADDRNOTAVAIL;
+}
+
+static void sock_put_port(struct cxgbi_sock *csk)
+{
+ struct cxgbi_device *cdev = csk->cdev;
+ struct cxgbi_ports_map *pmap = &cdev->pmap;
+
+ if (csk->saddr.sin_port) {
+ int idx = ntohs(csk->saddr.sin_port) - pmap->sport_base;
+
+ csk->saddr.sin_port = 0;
+ if (idx < 0 || idx >= pmap->max_connect) {
+ libcxgbi_log_error("cdev 0x%p, p#%u %s, port %u OOR.\n",
+ cdev, csk->port_id,
+ cdev->ports[csk->port_id]->name,
+ ntohs(csk->saddr.sin_port));
+ return;
+ }
+
+ spin_lock_bh(&pmap->lock);
+ pmap->port_csk[idx] = NULL;
+ pmap->used--;
+ spin_unlock_bh(&pmap->lock);
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_SOCK,
+ "cdev 0x%p, p#%u %s, release %u.\n",
+ cdev, csk->port_id,
+ cdev->ports[csk->port_id]->name,
+ pmap->sport_base + idx);
+
+ cxgbi_sock_put(csk);
+ }
+}
+
+/*
+ * iscsi tcp connection
+ */
+void cxgbi_sock_free_cpl_skbs(struct cxgbi_sock *csk)
+{
+ if (csk->cpl_close) {
+ kfree_skb(csk->cpl_close);
+ csk->cpl_close = NULL;
+ }
+ if (csk->cpl_abort_req) {
+ kfree_skb(csk->cpl_abort_req);
+ csk->cpl_abort_req = NULL;
+ }
+ if (csk->cpl_abort_rpl) {
+ kfree_skb(csk->cpl_abort_rpl);
+ csk->cpl_abort_rpl = NULL;
+ }
+}
+EXPORT_SYMBOL_GPL(cxgbi_sock_free_cpl_skbs);
+
+static struct cxgbi_sock *cxgbi_sock_create(struct cxgbi_device *cdev)
+{
+ struct cxgbi_sock *csk = kzalloc(sizeof(*csk), GFP_NOIO);
+
+ if (!csk) {
+ libcxgbi_log_info("alloc csk %lu failed.\n", sizeof(*csk));
+ return NULL;
+ }
+
+ if (cdev->csk_alloc_cpls(csk) < 0) {
+ libcxgbi_log_info("csk 0x%p, alloc cpls failed.\n", csk);
+ kfree(csk);
+ return NULL;
+ }
+
+ spin_lock_init(&csk->lock);
+ kref_init(&csk->refcnt);
+ skb_queue_head_init(&csk->receive_queue);
+ skb_queue_head_init(&csk->write_queue);
+ setup_timer(&csk->retry_timer, NULL, (unsigned long)csk);
+ rwlock_init(&csk->callback_lock);
+ csk->cdev = cdev;
+ csk->flags = 0;
+ cxgbi_sock_set_state(csk, CTP_CLOSED);
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_SOCK, "cdev 0x%p, new csk 0x%p.\n",
+ cdev, csk);
+
+ return csk;
+}
+
+static struct rtable *find_route_ipv4(__be32 saddr, __be32 daddr,
+ __be16 sport, __be16 dport, u8 tos)
+{
+ struct rtable *rt;
+ struct flowi fl = {
+ .oif = 0,
+ .nl_u = {
+ .ip4_u = {
+ .daddr = daddr,
+ .saddr = saddr,
+ .tos = tos }
+ },
+ .proto = IPPROTO_TCP,
+ .uli_u = {
+ .ports = {
+ .sport = sport,
+ .dport = dport }
+ }
+ };
+
+ if (ip_route_output_flow(&init_net, &rt, &fl, NULL, 0))
+ return NULL;
+
+ return rt;
+}
+
+static struct cxgbi_sock *cxgbi_check_route(struct sockaddr *dst_addr)
+{
+ struct sockaddr_in *daddr = (struct sockaddr_in *)dst_addr;
+ struct dst_entry *dst;
+ struct net_device *ndev;
+ struct cxgbi_device *cdev;
+ struct rtable *rt = NULL;
+ struct cxgbi_sock *csk = NULL;
+ unsigned int mtu = 0;
+ int port = 0xFFFF;
+ int err = 0;
+
+ if (daddr->sin_family != AF_INET) {
+ libcxgbi_log_info("address family 0x%x NOT supported.\n",
+ daddr->sin_family);
+ err = -EAFNOSUPPORT;
+ goto err_out;
+ }
+
+ rt = find_route_ipv4(0, daddr->sin_addr.s_addr, 0, daddr->sin_port, 0);
+ if (!rt) {
+ libcxgbi_log_info("no route to ipv4 0x%x, port %u.\n",
+ daddr->sin_addr.s_addr, daddr->sin_port);
+ err = -ENETUNREACH;
+ goto err_out;
+ }
+ dst = &rt->dst;
+ ndev = dst->neighbour->dev;
+
+ if (rt->rt_flags & (RTCF_MULTICAST | RTCF_BROADCAST)) {
+ libcxgbi_log_info("multi-cast route %pI4, port %u, dev %s.\n",
+ &daddr->sin_addr.s_addr,
+ ntohs(daddr->sin_port),
+ ndev->name);
+ err = -ENETUNREACH;
+ goto rel_rt;
+ }
+
+ if (ndev->flags & IFF_LOOPBACK) {
+ ndev = ip_dev_find(&init_net, daddr->sin_addr.s_addr);
+ mtu = ndev->mtu;
+ libcxgbi_log_info("rt dev %s, loopback -> %s, mtu %u.\n",
+ dst->neighbour->dev->name, ndev->name, mtu);
+ }
+
+ if (ndev->priv_flags & IFF_802_1Q_VLAN) {
+ ndev = vlan_dev_real_dev(ndev);
+ libcxgbi_log_info("rt dev %s, vlan -> %s.\n",
+ dst->neighbour->dev->name, ndev->name);
+ }
+
+ cdev = cxgbi_device_find_by_netdev(ndev, &port);
+ if (!cdev) {
+ libcxgbi_log_info("dst %pI4, %s, NOT cxgbi device.\n",
+ &daddr->sin_addr.s_addr, ndev->name);
+ err = -ENETUNREACH;
+ goto rel_rt;
+ }
+ libcxgbi_log_debug(1 << CXGBI_DBG_SOCK,
+ "route to %pI4 :%u, ndev p#%d,%s, cdev 0x%p.\n",
+ &daddr->sin_addr.s_addr, ntohs(daddr->sin_port),
+ port, ndev->name, cdev);
+
+ csk = cxgbi_sock_create(cdev);
+ if (!csk) {
+ err = -ENOMEM;
+ goto rel_rt;
+ }
+ csk->cdev = cdev;
+ csk->port_id = port;
+ csk->mtu = mtu;
+ csk->dst = dst;
+ csk->daddr.sin_addr.s_addr = daddr->sin_addr.s_addr;
+ csk->daddr.sin_port = daddr->sin_port;
+ if (cdev->hbas[port]->ipv4addr)
+ csk->saddr.sin_addr.s_addr = cdev->hbas[port]->ipv4addr;
+ else
+ csk->saddr.sin_addr.s_addr = rt->rt_src;
+
+ return csk;
+
+rel_rt:
+ ip_rt_put(rt);
+ if (csk)
+ cxgbi_sock_closed(csk);
+err_out:
+ return ERR_PTR(err);
+}
+
+void cxgbi_sock_established(struct cxgbi_sock *csk, unsigned int snd_isn,
+ unsigned int opt)
+{
+ csk->write_seq = csk->snd_nxt = csk->snd_una = snd_isn;
+ dst_confirm(csk->dst);
+ smp_mb();
+ cxgbi_sock_set_state(csk, CTP_ESTABLISHED);
+}
+EXPORT_SYMBOL_GPL(cxgbi_sock_established);
+
+static void cxgbi_inform_iscsi_conn_closing(struct cxgbi_sock *csk)
+{
+ libcxgbi_log_debug(1 << CXGBI_DBG_SOCK,
+ "csk 0x%p, state %u, flags 0x%lx, conn 0x%p.\n",
+ csk, csk->state, csk->flags, csk->user_data);
+
+ if (csk->state != CTP_ESTABLISHED) {
+ read_lock(&csk->callback_lock);
+ if (csk->user_data)
+ iscsi_conn_failure(csk->user_data,
+ ISCSI_ERR_CONN_FAILED);
+ read_unlock(&csk->callback_lock);
+ }
+}
+
+void cxgbi_sock_closed(struct cxgbi_sock *csk)
+{
+ libcxgbi_log_debug(1 << CXGBI_DBG_SOCK, "csk 0x%p,%u,0x%lx.\n",
+ csk, csk->state, csk->flags);
+
+ cxgbi_sock_set_flag(csk, CTPF_ACTIVE_CLOSE_NEEDED);
+ if (csk->state == CTP_ACTIVE_OPEN || csk->state == CTP_CLOSED)
+ return;
+ if (csk->saddr.sin_port)
+ sock_put_port(csk);
+ if (csk->dst)
+ dst_release(csk->dst);
+ csk->cdev->csk_release_offload_resources(csk);
+ cxgbi_sock_set_state(csk, CTP_CLOSED);
+ cxgbi_inform_iscsi_conn_closing(csk);
+ cxgbi_sock_put(csk);
+}
+EXPORT_SYMBOL_GPL(cxgbi_sock_closed);
+
+static void need_active_close(struct cxgbi_sock *csk)
+{
+ int data_lost;
+ int close_req = 0;
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_SOCK,
+ "csk 0x%p, state %u, flags 0x%lx\n",
+ csk, csk->state, csk->flags);
+
+ spin_lock_bh(&csk->lock);
+
+ dst_confirm(csk->dst);
+ data_lost = skb_queue_len(&csk->receive_queue);
+ __skb_queue_purge(&csk->receive_queue);
+
+ if (csk->state == CTP_ACTIVE_OPEN)
+ cxgbi_sock_set_flag(csk, CTPF_ACTIVE_CLOSE_NEEDED);
+ else if (csk->state == CTP_ESTABLISHED) {
+ close_req = 1;
+ cxgbi_sock_set_state(csk, CTP_ACTIVE_CLOSE);
+ } else if (csk->state == CTP_PASSIVE_CLOSE) {
+ close_req = 1;
+ cxgbi_sock_set_state(csk, CTP_CLOSE_WAIT_2);
+ }
+
+ if (close_req) {
+ if (data_lost)
+ csk->cdev->csk_send_abort_req(csk);
+ else
+ csk->cdev->csk_send_close_req(csk);
+ }
+
+ spin_unlock_bh(&csk->lock);
+}
+
+void cxgbi_sock_fail_act_open(struct cxgbi_sock *csk, int errno)
+{
+ libcxgbi_log_info("csk 0x%p,%u,%lx, %pI4:%u-%pI4:%u, err %d.\n",
+ csk, csk->state, csk->flags,
+ &csk->saddr.sin_addr.s_addr, csk->saddr.sin_port,
+ &csk->daddr.sin_addr.s_addr, csk->daddr.sin_port,
+ errno);
+
+ cxgbi_sock_set_state(csk, CTP_CONNECTING);
+ csk->err = errno;
+ cxgbi_sock_closed(csk);
+}
+EXPORT_SYMBOL_GPL(cxgbi_sock_fail_act_open);
+
+void cxgbi_sock_act_open_req_arp_failure(void *handle, struct sk_buff *skb)
+{
+ struct cxgbi_sock *csk = (struct cxgbi_sock *)skb->sk;
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_SOCK,
+ "csk 0x%p, state %u, flags %lu\n",
+ csk, csk->state, csk->flags);
+
+ cxgbi_sock_get(csk);
+ spin_lock_bh(&csk->lock);
+ if (csk->state == CTP_ACTIVE_OPEN)
+ cxgbi_sock_fail_act_open(csk, -EHOSTUNREACH);
+ spin_unlock_bh(&csk->lock);
+ cxgbi_sock_put(csk);
+ __kfree_skb(skb);
+}
+EXPORT_SYMBOL_GPL(cxgbi_sock_act_open_req_arp_failure);
+
+void cxgbi_sock_rcv_abort_rpl(struct cxgbi_sock *csk)
+{
+ cxgbi_sock_get(csk);
+ spin_lock_bh(&csk->lock);
+ if (cxgbi_sock_flag(csk, CTPF_ABORT_RPL_PENDING)) {
+ if (!cxgbi_sock_flag(csk, CTPF_ABORT_RPL_RCVD))
+ cxgbi_sock_set_flag(csk, CTPF_ABORT_RPL_RCVD);
+ else {
+ cxgbi_sock_clear_flag(csk, CTPF_ABORT_RPL_RCVD);
+ cxgbi_sock_clear_flag(csk, CTPF_ABORT_RPL_PENDING);
+ if (cxgbi_sock_flag(csk, CTPF_ABORT_REQ_RCVD))
+ libcxgbi_log_error("csk 0x%p,%u,0x%lx,%u, "
+ "ABORT_RPL_RSS.\n",
+ csk, csk->state, csk->flags,
+ csk->tid);
+ cxgbi_sock_closed(csk);
+ }
+ }
+ spin_unlock_bh(&csk->lock);
+ cxgbi_sock_put(csk);
+}
+EXPORT_SYMBOL_GPL(cxgbi_sock_rcv_abort_rpl);
+
+void cxgbi_sock_rcv_peer_close(struct cxgbi_sock *csk)
+{
+ libcxgbi_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx, tid %u.\n",
+ csk, csk->state, csk->flags, csk->tid);
+
+ cxgbi_sock_get(csk);
+ spin_lock_bh(&csk->lock);
+
+ if (cxgbi_sock_flag(csk, CTPF_ABORT_RPL_PENDING))
+ goto done;
+
+ switch (csk->state) {
+ case CTP_ESTABLISHED:
+ cxgbi_sock_set_state(csk, CTP_PASSIVE_CLOSE);
+ break;
+ case CTP_ACTIVE_CLOSE:
+ cxgbi_sock_set_state(csk, CTP_CLOSE_WAIT_2);
+ break;
+ case CTP_CLOSE_WAIT_1:
+ cxgbi_sock_closed(csk);
+ break;
+ case CTP_ABORTING:
+ break;
+ default:
+ libcxgbi_log_error("csk 0x%p,%u,0x%lx,%u, bad state.\n",
+ csk, csk->state, csk->flags, csk->tid);
+ }
+ cxgbi_inform_iscsi_conn_closing(csk);
+done:
+ spin_unlock_bh(&csk->lock);
+ cxgbi_sock_put(csk);
+}
+EXPORT_SYMBOL_GPL(cxgbi_sock_rcv_peer_close);
+
+void cxgbi_sock_rcv_close_conn_rpl(struct cxgbi_sock *csk, u32 snd_nxt)
+{
+ libcxgbi_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx, tid %u.\n",
+ csk, csk->state, csk->flags, csk->tid);
+
+ cxgbi_sock_get(csk);
+ spin_lock_bh(&csk->lock);
+
+ csk->snd_una = snd_nxt - 1;
+ if (cxgbi_sock_flag(csk, CTPF_ABORT_RPL_PENDING))
+ goto done;
+
+ switch (csk->state) {
+ case CTP_ACTIVE_CLOSE:
+ cxgbi_sock_set_state(csk, CTP_CLOSE_WAIT_1);
+ break;
+ case CTP_CLOSE_WAIT_1:
+ case CTP_CLOSE_WAIT_2:
+ cxgbi_sock_closed(csk);
+ break;
+ case CTP_ABORTING:
+ break;
+ default:
+ libcxgbi_log_error("csk 0x%p,%u,0x%lx,%u, bad state.\n",
+ csk, csk->state, csk->flags, csk->tid);
+ }
+done:
+ spin_unlock_bh(&csk->lock);
+ cxgbi_sock_put(csk);
+}
+EXPORT_SYMBOL_GPL(cxgbi_sock_rcv_close_conn_rpl);
+
+void cxgbi_sock_rcv_wr_ack(struct cxgbi_sock *csk, unsigned int credits,
+ unsigned int snd_una, int seq_chk)
+{
+ libcxgbi_log_debug(1 << CXGBI_DBG_TOE | 1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx,%u, cr %u, %u+%u, snd_una %u,%d.\n",
+ csk, csk->state, csk->flags, csk->tid, credits,
+ csk->wr_cred, csk->wr_una_cred, snd_una, seq_chk);
+
+ spin_lock_bh(&csk->lock);
+
+ csk->wr_cred += credits;
+ if (csk->wr_una_cred > csk->wr_max_cred - csk->wr_cred)
+ csk->wr_una_cred = csk->wr_max_cred - csk->wr_cred;
+
+ while (credits) {
+ struct sk_buff *p = cxgbi_sock_peek_wr(csk);
+
+ if (unlikely(!p)) {
+ libcxgbi_log_error("csk 0x%p,%u,0x%lx,%u, cr %u,%u+%u,"
+ " NO pending.\n",
+ csk, csk->state, csk->flags, csk->tid,
+ credits, csk->wr_cred,
+ csk->wr_una_cred);
+ break;
+ }
+
+ if (unlikely(credits < p->csum)) {
+ libcxgbi_log_warn("csk 0x%p,%u,0x%lx,%u, cr %u, %u+%u, "
+ "partial %u.\n",
+ csk, csk->state, csk->flags, csk->tid,
+ credits, csk->wr_cred,
+ csk->wr_una_cred, p->csum);
+ p->csum -= credits;
+ break;
+ } else {
+ cxgbi_sock_dequeue_wr(csk);
+ credits -= p->csum;
+ kfree_skb(p);
+ }
+ }
+
+ cxgbi_sock_check_wr_invariants(csk);
+
+ if (seq_chk) {
+ if (unlikely(before(snd_una, csk->snd_una))) {
+ libcxgbi_log_warn("csk 0x%p,%u,0x%lx,%u, suna %u/%u.",
+ csk, csk->state, csk->flags, csk->tid,
+ snd_una, csk->snd_una);
+ goto done;
+ }
+
+ if (csk->snd_una != snd_una) {
+ csk->snd_una = snd_una;
+ dst_confirm(csk->dst);
+ }
+ }
+
+ if (skb_queue_len(&csk->write_queue)) {
+ if (csk->cdev->csk_push_tx_frames(csk, 0))
+ cxgbi_conn_tx_open(csk);
+ } else
+ cxgbi_conn_tx_open(csk);
+done:
+ spin_unlock_bh(&csk->lock);
+}
+EXPORT_SYMBOL_GPL(cxgbi_sock_rcv_wr_ack);
+
+static unsigned int cxgbi_sock_find_best_mtu(struct cxgbi_sock *csk,
+ unsigned short mtu)
+{
+ int i = 0;
+
+ while (i < csk->cdev->nmtus - 1 && csk->cdev->mtus[i + 1] <= mtu)
+ ++i;
+
+ return i;
+}
+
+unsigned int cxgbi_sock_select_mss(struct cxgbi_sock *csk, unsigned int pmtu)
+{
+ unsigned int idx;
+ struct dst_entry *dst = csk->dst;
+
+ csk->advmss = dst_metric(dst, RTAX_ADVMSS);
+
+ if (csk->advmss > pmtu - 40)
+ csk->advmss = pmtu - 40;
+ if (csk->advmss < csk->cdev->mtus[0] - 40)
+ csk->advmss = csk->cdev->mtus[0] - 40;
+ idx = cxgbi_sock_find_best_mtu(csk, csk->advmss + 40);
+
+ return idx;
+}
+EXPORT_SYMBOL_GPL(cxgbi_sock_select_mss);
+
+void cxgbi_sock_skb_entail(struct cxgbi_sock *csk, struct sk_buff *skb)
+{
+ cxgbi_skcb_tcp_seq(skb) = csk->write_seq;
+ __skb_queue_tail(&csk->write_queue, skb);
+}
+EXPORT_SYMBOL_GPL(cxgbi_sock_skb_entail);
+
+void cxgbi_sock_purge_wr_queue(struct cxgbi_sock *csk)
+{
+ struct sk_buff *skb;
+
+ while ((skb = cxgbi_sock_dequeue_wr(csk)) != NULL)
+ kfree_skb(skb);
+}
+EXPORT_SYMBOL_GPL(cxgbi_sock_purge_wr_queue);
+
+void cxgbi_sock_check_wr_invariants(const struct cxgbi_sock *csk)
+{
+ int pending = cxgbi_sock_count_pending_wrs(csk);
+
+ if (unlikely(csk->wr_cred + pending != csk->wr_max_cred))
+ libcxgbi_log_error("csk 0x%p, tid %u, credit %u + %u != %u.\n",
+ csk, csk->tid, csk->wr_cred, pending,
+ csk->wr_max_cred);
+}
+EXPORT_SYMBOL_GPL(cxgbi_sock_check_wr_invariants);
+
+static int cxgbi_sock_send_pdus(struct cxgbi_sock *csk, struct sk_buff *skb)
+{
+ struct cxgbi_device *cdev = csk->cdev;
+ struct sk_buff *next;
+ int err, copied = 0;
+
+ spin_lock_bh(&csk->lock);
+
+ if (csk->state != CTP_ESTABLISHED) {
+ libcxgbi_log_debug(1 << CXGBI_DBG_PDU_TX,
+ "csk 0x%p,%u,0x%lx,%u, EAGAIN.\n",
+ csk, csk->state, csk->flags, csk->tid);
+ err = -EAGAIN;
+ goto out_err;
+ }
+
+ if (csk->err) {
+ libcxgbi_log_debug(1 << CXGBI_DBG_PDU_TX,
+ "csk 0x%p,%u,0x%lx,%u, EPIPE %d.\n",
+ csk, csk->state, csk->flags, csk->tid,
+ csk->err);
+ err = -EPIPE;
+ goto out_err;
+ }
+
+ if (csk->write_seq - csk->snd_una >= cdev->snd_win) {
+ libcxgbi_log_debug(1 << CXGBI_DBG_PDU_TX,
+ "csk 0x%p,%u,0x%lx,%u, FULL %u-%u >= %u.\n",
+ csk, csk->state, csk->flags, csk->tid,
+ csk->write_seq, csk->snd_una, cdev->snd_win);
+ err = -ENOBUFS;
+ goto out_err;
+ }
+
+ while (skb) {
+ int frags = skb_shinfo(skb)->nr_frags +
+ (skb->len != skb->data_len);
+
+ if (unlikely(skb_headroom(skb) < cdev->skb_tx_rsvd)) {
+ libcxgbi_log_error("csk 0x%p, skb head %u < %u.\n",
+ csk, skb_headroom(skb),
+ cdev->skb_tx_rsvd);
+ err = -EINVAL;
+ goto out_err;
+ }
+
+ if (frags >= SKB_WR_LIST_SIZE) {
+ libcxgbi_log_error("csk 0x%p, frags %d, %u,%u >%lu.\n",
+ csk, skb_shinfo(skb)->nr_frags,
+ skb->len, skb->data_len,
+ SKB_WR_LIST_SIZE);
+ err = -EINVAL;
+ goto out_err;
+ }
+
+ next = skb->next;
+ skb->next = NULL;
+ cxgbi_skcb_set_flag(skb, SKCBF_TX_NEED_HDR);
+ cxgbi_sock_skb_entail(csk, skb);
+ copied += skb->len;
+ csk->write_seq += skb->len +
+ cxgbi_ulp_extra_len(cxgbi_skcb_ulp_mode(skb));
+ skb = next;
+ }
+done:
+ if (likely(skb_queue_len(&csk->write_queue)))
+ cdev->csk_push_tx_frames(csk, 1);
+ spin_unlock_bh(&csk->lock);
+ return copied;
+
+out_err:
+ if (copied == 0 && err == -EPIPE)
+ copied = csk->err ? csk->err : -EPIPE;
+ else
+ copied = err;
+ goto done;
+}
+
+/*
+ * Direct Data Placement -
+ * Directly place the iSCSI Data-In or Data-Out PDU's payload into pre-posted
+ * final destination host-memory buffers based on the Initiator Task Tag (ITT)
+ * in Data-In or Target Task Tag (TTT) in Data-Out PDUs.
+ * The host memory address is programmed into h/w in the format of pagepod
+ * entries.
+ * The location of the pagepod entry is encoded into ddp tag which is used as
+ * the base for ITT/TTT.
+ */
+
+static unsigned char ddp_page_order[DDP_PGIDX_MAX] = {0, 1, 2, 4};
+static unsigned char ddp_page_shift[DDP_PGIDX_MAX] = {12, 13, 14, 16};
+static unsigned char page_idx = DDP_PGIDX_MAX;
+
+static unsigned char sw_tag_idx_bits;
+static unsigned char sw_tag_age_bits;
+
+/*
+ * Direct-Data Placement page size adjustment
+ */
+static int ddp_adjust_page_table(void)
+{
+ int i;
+ unsigned int base_order, order;
+
+ if (PAGE_SIZE < (1UL << ddp_page_shift[0])) {
+ libcxgbi_log_info("PAGE_SIZE 0x%lx too small, min 0x%lx\n",
+ PAGE_SIZE, 1UL << ddp_page_shift[0]);
+ return -EINVAL;
+ }
+
+ base_order = get_order(1UL << ddp_page_shift[0]);
+ order = get_order(1UL << PAGE_SHIFT);
+
+ for (i = 0; i < DDP_PGIDX_MAX; i++) {
+ /* first is the kernel page size, then just doubling */
+ ddp_page_order[i] = order - base_order + i;
+ ddp_page_shift[i] = PAGE_SHIFT + i;
+ }
+ return 0;
+}
+
+static int ddp_find_page_index(unsigned long pgsz)
+{
+ int i;
+
+ for (i = 0; i < DDP_PGIDX_MAX; i++) {
+ if (pgsz == (1UL << ddp_page_shift[i]))
+ return i;
+ }
+ libcxgbi_log_info("ddp page size %lu not supported\n", pgsz);
+ return DDP_PGIDX_MAX;
+}
+
+static void ddp_setup_host_page_size(void)
+{
+ if (page_idx == DDP_PGIDX_MAX) {
+ page_idx = ddp_find_page_index(PAGE_SIZE);
+
+ if (page_idx == DDP_PGIDX_MAX) {
+ libcxgbi_log_info("system PAGE_SIZE %lu, update hw\n",
+ PAGE_SIZE);
+ if (ddp_adjust_page_table() < 0) {
+ libcxgbi_log_info("PAGE %lu, disable ddp.\n",
+ PAGE_SIZE);
+ return;
+ }
+ page_idx = ddp_find_page_index(PAGE_SIZE);
+ }
+ libcxgbi_log_info("system PAGE_SIZE %lu, ddp idx %u\n",
+ PAGE_SIZE, page_idx);
+ }
+}
+
+void cxgbi_ddp_page_size_factor(int *pgsz_factor)
+{
+ int i;
+
+ for (i = 0; i < DDP_PGIDX_MAX; i++)
+ pgsz_factor[i] = ddp_page_order[i];
+}
+EXPORT_SYMBOL_GPL(cxgbi_ddp_page_size_factor);
+
+/*
+ * DDP setup & teardown
+ */
+
+void cxgbi_ddp_ppod_set(struct cxgbi_pagepod *ppod,
+ struct cxgbi_pagepod_hdr *hdr,
+ struct cxgbi_gather_list *gl, unsigned int gidx)
+{
+ int i;
+
+ memcpy(ppod, hdr, sizeof(*hdr));
+ for (i = 0; i < (PPOD_PAGES_MAX + 1); i++, gidx++) {
+ ppod->addr[i] = gidx < gl->nelem ?
+ cpu_to_be64(gl->phys_addr[gidx]) : 0ULL;
+ }
+}
+EXPORT_SYMBOL_GPL(cxgbi_ddp_ppod_set);
+
+void cxgbi_ddp_ppod_clear(struct cxgbi_pagepod *ppod)
+{
+ memset(ppod, 0, sizeof(*ppod));
+}
+EXPORT_SYMBOL_GPL(cxgbi_ddp_ppod_clear);
+
+static inline int ddp_find_unused_entries(struct cxgbi_ddp_info *ddp,
+ unsigned int start, unsigned int max,
+ unsigned int count,
+ struct cxgbi_gather_list *gl)
+{
+ unsigned int i, j, k;
+
+ /* not enough entries */
+ if ((max - start) < count) {
+ libcxgbi_log_debug(1 << CXGBI_DBG_DDP,
+ "NOT enough entries %u + %u < %u.\n",
+ start, count, max);
+ return -EBUSY;
+ }
+
+ max -= count;
+ spin_lock(&ddp->map_lock);
+ for (i = start; i < max;) {
+ for (j = 0, k = i; j < count; j++, k++) {
+ if (ddp->gl_map[k])
+ break;
+ }
+ if (j == count) {
+ for (j = 0, k = i; j < count; j++, k++)
+ ddp->gl_map[k] = gl;
+ spin_unlock(&ddp->map_lock);
+ return i;
+ }
+ i += j + 1;
+ }
+ spin_unlock(&ddp->map_lock);
+ libcxgbi_log_debug(1 << CXGBI_DBG_DDP,
+ "NO suitable entries %u available.\n", count);
+ return -EBUSY;
+}
+
+static inline void ddp_unmark_entries(struct cxgbi_ddp_info *ddp,
+ int start, int count)
+{
+ spin_lock(&ddp->map_lock);
+ memset(&ddp->gl_map[start], 0,
+ count * sizeof(struct cxgbi_gather_list *));
+ spin_unlock(&ddp->map_lock);
+}
+
+static inline void ddp_gl_unmap(struct pci_dev *pdev,
+ struct cxgbi_gather_list *gl)
+{
+ int i;
+
+ for (i = 0; i < gl->nelem; i++)
+ dma_unmap_page(&pdev->dev, gl->phys_addr[i], PAGE_SIZE,
+ PCI_DMA_FROMDEVICE);
+}
+
+static inline int ddp_gl_map(struct pci_dev *pdev,
+ struct cxgbi_gather_list *gl)
+{
+ int i;
+
+ for (i = 0; i < gl->nelem; i++) {
+ gl->phys_addr[i] = dma_map_page(&pdev->dev, gl->pages[i], 0,
+ PAGE_SIZE,
+ PCI_DMA_FROMDEVICE);
+ if (unlikely(dma_mapping_error(&pdev->dev, gl->phys_addr[i]))) {
+ libcxgbi_log_debug(1 << CXGBI_DBG_DDP,
+ "page %d 0x%p, 0x%p dma mapping err.\n",
+ i, gl->pages[i], pdev);
+ goto unmap;
+ }
+ }
+ return i;
+unmap:
+ if (i) {
+ unsigned int nelem = gl->nelem;
+
+ gl->nelem = i;
+ ddp_gl_unmap(pdev, gl);
+ gl->nelem = nelem;
+ }
+ return -EINVAL;
+}
+
+static void ddp_release_gl(struct cxgbi_gather_list *gl,
+ struct pci_dev *pdev)
+{
+ ddp_gl_unmap(pdev, gl);
+ kfree(gl);
+}
+
+static struct cxgbi_gather_list *ddp_make_gl(unsigned int xferlen,
+ struct scatterlist *sgl,
+ unsigned int sgcnt,
+ struct pci_dev *pdev,
+ gfp_t gfp)
+{
+ struct cxgbi_gather_list *gl;
+ struct scatterlist *sg = sgl;
+ struct page *sgpage = sg_page(sg);
+ unsigned int sglen = sg->length;
+ unsigned int sgoffset = sg->offset;
+ unsigned int npages = (xferlen + sgoffset + PAGE_SIZE - 1) >>
+ PAGE_SHIFT;
+ int i = 1, j = 0;
+
+ if (xferlen < DDP_THRESHOLD) {
+ libcxgbi_log_debug(1 << CXGBI_DBG_DDP,
+ "xfer %u < threshold %u, no ddp.\n",
+ xferlen, DDP_THRESHOLD);
+ return NULL;
+ }
+
+ gl = kzalloc(sizeof(struct cxgbi_gather_list) +
+ npages * (sizeof(dma_addr_t) +
+ sizeof(struct page *)), gfp);
+ if (!gl) {
+ libcxgbi_log_debug(1 << CXGBI_DBG_DDP,
+ "xfer %u, %u pages, OOM.\n", xferlen, npages);
+ return NULL;
+ }
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_DDP,
+ "xfer %u, sgl %u, gl max %u.\n",
+ xferlen, sgcnt, npages);
+
+ gl->pages = (struct page **)&gl->phys_addr[npages];
+ gl->nelem = npages;
+ gl->length = xferlen;
+ gl->offset = sgoffset;
+ gl->pages[0] = sgpage;
+
+ for (i = 1, sg = sg_next(sgl), j = 0; i < sgcnt;
+ i++, sg = sg_next(sg)) {
+ struct page *page = sg_page(sg);
+
+ if (sgpage == page && sg->offset == sgoffset + sglen)
+ sglen += sg->length;
+ else {
+ /* make sure the sgl is fit for ddp:
+ * each has the same page size, and
+ * all of the middle pages are used completely
+ */
+ if ((j && sgoffset) || ((i != sgcnt - 1) &&
+ ((sglen + sgoffset) & ~PAGE_MASK))) {
+ libcxgbi_log_debug(1 << CXGBI_DBG_DDP,
+ "page %d/%u, %u + %u.\n",
+ i, sgcnt, sgoffset, sglen);
+ goto error_out;
+ }
+
+ j++;
+ if (j == gl->nelem || sg->offset) {
+ libcxgbi_log_debug(1 << CXGBI_DBG_DDP,
+ "page %d/%u, offset %u.\n",
+ j, gl->nelem, sg->offset);
+ goto error_out;
+ }
+ gl->pages[j] = page;
+ sglen = sg->length;
+ sgoffset = sg->offset;
+ sgpage = page;
+ }
+ }
+ gl->nelem = ++j;
+
+ if (ddp_gl_map(pdev, gl) < 0)
+ goto error_out;
+
+ return gl;
+
+error_out:
+ kfree(gl);
+ return NULL;
+}
+
+static void ddp_tag_release(struct cxgbi_hba *chba, u32 tag)
+{
+ struct cxgbi_device *cdev = chba->cdev;
+ struct cxgbi_ddp_info *ddp = cdev->ddp;
+ u32 idx;
+
+ idx = (tag >> PPOD_IDX_SHIFT) & ddp->idx_mask;
+ if (idx < ddp->nppods) {
+ struct cxgbi_gather_list *gl = ddp->gl_map[idx];
+ unsigned int npods;
+
+ if (!gl || !gl->nelem) {
+ libcxgbi_log_warn("tag 0x%x, idx %u, gl 0x%p, %u\n",
+ tag, idx, gl, gl ? gl->nelem : 0);
+ return;
+ }
+ npods = (gl->nelem + PPOD_PAGES_MAX - 1) >> PPOD_PAGES_SHIFT;
+ libcxgbi_log_debug(1 << CXGBI_DBG_DDP,
+ "tag 0x%x, release idx %u, npods %u.\n",
+ tag, idx, npods);
+ cdev->csk_ddp_clear(chba, tag, idx, npods);
+ ddp_unmark_entries(ddp, idx, npods);
+ ddp_release_gl(gl, ddp->pdev);
+ } else
+ libcxgbi_log_warn("tag 0x%x, idx %u > max %u.\n",
+ tag, idx, ddp->nppods);
+}
+
+static int ddp_tag_reserve(struct cxgbi_sock *csk, unsigned int tid,
+ u32 sw_tag, u32 *tagp, struct cxgbi_gather_list *gl,
+ gfp_t gfp)
+{
+ struct cxgbi_device *cdev = csk->cdev;
+ struct cxgbi_ddp_info *ddp = cdev->ddp;
+ struct cxgbi_tag_format *tformat = &cdev->tag_format;
+ struct cxgbi_pagepod_hdr hdr;
+ unsigned int npods;
+ int idx = -1;
+ int err = -ENOMEM;
+ u32 tag;
+
+ npods = (gl->nelem + PPOD_PAGES_MAX - 1) >> PPOD_PAGES_SHIFT;
+ if (ddp->idx_last == ddp->nppods)
+ idx = ddp_find_unused_entries(ddp, 0, ddp->nppods,
+ npods, gl);
+ else {
+ idx = ddp_find_unused_entries(ddp, ddp->idx_last + 1,
+ ddp->nppods, npods,
+ gl);
+ if (idx < 0 && ddp->idx_last >= npods) {
+ idx = ddp_find_unused_entries(ddp, 0,
+ min(ddp->idx_last + npods, ddp->nppods),
+ npods, gl);
+ }
+ }
+ if (idx < 0) {
+ libcxgbi_log_debug(1 << CXGBI_DBG_DDP,
+ "xferlen %u, gl %u, npods %u NO DDP.\n",
+ gl->length, gl->nelem, npods);
+ return idx;
+ }
+
+ if (cdev->csk_ddp_alloc_gl_skb) {
+ err = cdev->csk_ddp_alloc_gl_skb(ddp, idx, npods, gfp);
+ if (err < 0)
+ goto unmark_entries;
+ }
+
+ tag = cxgbi_ddp_tag_base(tformat, sw_tag);
+ tag |= idx << PPOD_IDX_SHIFT;
+
+ hdr.rsvd = 0;
+ hdr.vld_tid = htonl(PPOD_VALID_FLAG | PPOD_TID(tid));
+ hdr.pgsz_tag_clr = htonl(tag & ddp->rsvd_tag_mask);
+ hdr.max_offset = htonl(gl->length);
+ hdr.page_offset = htonl(gl->offset);
+
+ err = cdev->csk_ddp_set(csk, &hdr, idx, npods, gl);
+ if (err < 0) {
+ if (cdev->csk_ddp_free_gl_skb)
+ cdev->csk_ddp_free_gl_skb(ddp, idx, npods);
+ goto unmark_entries;
+ }
+
+ ddp->idx_last = idx;
+ libcxgbi_log_debug(1 << CXGBI_DBG_DDP,
+ "xfer %u, gl %u,%u, tid 0x%x, tag 0x%x->0x%x(%u,%u).\n",
+ gl->length, gl->nelem, gl->offset, tid, sw_tag, tag,
+ idx, npods);
+ *tagp = tag;
+ return 0;
+
+unmark_entries:
+ ddp_unmark_entries(ddp, idx, npods);
+ return err;
+}
+
+int cxgbi_ddp_reserve(struct cxgbi_sock *csk, unsigned int *tagp,
+ unsigned int sw_tag, unsigned int xferlen,
+ struct scatterlist *sgl, unsigned int sgcnt, gfp_t gfp)
+{
+ struct cxgbi_device *cdev = csk->cdev;
+ struct cxgbi_tag_format *tformat = &cdev->tag_format;
+ struct cxgbi_gather_list *gl;
+ int err;
+
+ if (page_idx >= DDP_PGIDX_MAX || !cdev->ddp ||
+ xferlen < DDP_THRESHOLD) {
+ libcxgbi_log_debug(1 << CXGBI_DBG_DDP,
+ "pgidx %u, xfer %u, NO ddp.\n",
+ page_idx, xferlen);
+ return -EINVAL;
+ }
+
+ if (!cxgbi_sw_tag_usable(tformat, sw_tag)) {
+ libcxgbi_log_debug(1 << CXGBI_DBG_DDP,
+ "sw_tag 0x%x NOT usable.\n", sw_tag);
+ return -EINVAL;
+ }
+
+ gl = ddp_make_gl(xferlen, sgl, sgcnt, cdev->pdev, gfp);
+ if (!gl)
+ return -ENOMEM;
+
+ err = ddp_tag_reserve(csk, csk->tid, sw_tag, tagp, gl, gfp);
+ if (err < 0)
+ ddp_release_gl(gl, cdev->pdev);
+
+ return err;
+}
+
+static void ddp_destroy(struct kref *kref)
+{
+ struct cxgbi_ddp_info *ddp = container_of(kref,
+ struct cxgbi_ddp_info,
+ refcnt);
+ struct cxgbi_device *cdev = ddp->cdev;
+ int i = 0;
+
+ libcxgbi_log_info("kref 0, destroy ddp 0x%p, cdev 0x%p.\n", ddp, cdev);
+
+ while (i < ddp->nppods) {
+ struct cxgbi_gather_list *gl = ddp->gl_map[i];
+
+ if (gl) {
+ int npods = (gl->nelem + PPOD_PAGES_MAX - 1)
+ >> PPOD_PAGES_SHIFT;
+ libcxgbi_log_info("cdev 0x%p, ddp %d + %d.\n",
+ cdev, i, npods);
+ kfree(gl);
+ if (cdev->csk_ddp_free_gl_skb)
+ cdev->csk_ddp_free_gl_skb(ddp, i, npods);
+ i += npods;
+ } else
+ i++;
+ }
+ cxgbi_free_big_mem(ddp);
+}
+
+int cxgbi_ddp_cleanup(struct cxgbi_device *cdev)
+{
+ struct cxgbi_ddp_info *ddp = cdev->ddp;
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_DDP,
+ "cdev 0x%p, release ddp 0x%p.\n", cdev, ddp);
+ cdev->ddp = NULL;
+ if (ddp)
+ return kref_put(&ddp->refcnt, ddp_destroy);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(cxgbi_ddp_cleanup);
+
+int cxgbi_ddp_init(struct cxgbi_device *cdev,
+ unsigned int llimit, unsigned int ulimit,
+ unsigned int max_txsz, unsigned int max_rxsz)
+{
+ struct cxgbi_ddp_info *ddp;
+ unsigned int ppmax, bits;
+
+ ppmax = (ulimit - llimit + 1) >> PPOD_SIZE_SHIFT;
+ bits = __ilog2_u32(ppmax) + 1;
+ if (bits > PPOD_IDX_MAX_SIZE)
+ bits = PPOD_IDX_MAX_SIZE;
+ ppmax = (1 << (bits - 1)) - 1;
+
+ ddp = cxgbi_alloc_big_mem(sizeof(struct cxgbi_ddp_info) +
+ ppmax * (sizeof(struct cxgbi_gather_list *) +
+ sizeof(struct sk_buff *)),
+ GFP_KERNEL);
+ if (!ddp) {
+ libcxgbi_log_warn("cdev 0x%p, ddp ppmax %u OOM.\n",
+ cdev, ppmax);
+ return -ENOMEM;
+ }
+ ddp->gl_map = (struct cxgbi_gather_list **)(ddp + 1);
+ ddp->gl_skb = (struct sk_buff **)(((char *)ddp->gl_map) +
+ ppmax * sizeof(struct cxgbi_gather_list *));
+ cdev->ddp = ddp;
+
+ spin_lock_init(&ddp->map_lock);
+ kref_init(&ddp->refcnt);
+
+ ddp->cdev = cdev;
+ ddp->pdev = cdev->pdev;
+ ddp->llimit = llimit;
+ ddp->ulimit = ulimit;
+ ddp->max_txsz = min_t(unsigned int, max_txsz, ULP2_MAX_PKT_SIZE);
+ ddp->max_rxsz = min_t(unsigned int, max_rxsz, ULP2_MAX_PKT_SIZE);
+ ddp->nppods = ppmax;
+ ddp->idx_last = ppmax;
+ ddp->idx_bits = bits;
+ ddp->idx_mask = (1 << bits) - 1;
+ ddp->rsvd_tag_mask = (1 << (bits + PPOD_IDX_SHIFT)) - 1;
+
+ cdev->tag_format.sw_bits = sw_tag_idx_bits + sw_tag_age_bits;
+ cdev->tag_format.rsvd_bits = ddp->idx_bits;
+ cdev->tag_format.rsvd_shift = PPOD_IDX_SHIFT;
+ cdev->tag_format.rsvd_mask = (1 << cdev->tag_format.rsvd_bits) - 1;
+
+ libcxgbi_log_info("%s tag format, sw %u, rsvd %u,%u, mask 0x%x.\n",
+ cdev->ports[0]->name, cdev->tag_format.sw_bits,
+ cdev->tag_format.rsvd_bits, cdev->tag_format.rsvd_shift,
+ cdev->tag_format.rsvd_mask);
+
+ cdev->tx_max_size = min_t(unsigned int, ULP2_MAX_PDU_PAYLOAD,
+ ddp->max_txsz - ISCSI_PDU_NONPAYLOAD_LEN);
+ cdev->rx_max_size = min_t(unsigned int, ULP2_MAX_PDU_PAYLOAD,
+ ddp->max_rxsz - ISCSI_PDU_NONPAYLOAD_LEN);
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_DDP,
+ "%s max payload size: %u/%u, %u/%u.\n",
+ cdev->ports[0]->name, cdev->tx_max_size, ddp->max_txsz,
+ cdev->rx_max_size, ddp->max_rxsz);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(cxgbi_ddp_init);
+
+/*
+ * APIs interacting with open-iscsi libraries
+ */
+
+static unsigned char padding[4];
+
+static void task_release_itt(struct iscsi_task *task, itt_t hdr_itt)
+{
+ struct scsi_cmnd *sc = task->sc;
+ struct iscsi_tcp_conn *tcp_conn = task->conn->dd_data;
+ struct cxgbi_conn *cconn = tcp_conn->dd_data;
+ struct cxgbi_hba *chba = cconn->chba;
+ struct cxgbi_tag_format *tformat = &chba->cdev->tag_format;
+ u32 tag = ntohl((__force u32)hdr_itt);
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_DDP,
+ "cdev 0x%p, release tag 0x%x.\n", chba->cdev, tag);
+ if (sc &&
+ (scsi_bidi_cmnd(sc) || sc->sc_data_direction == DMA_FROM_DEVICE) &&
+ cxgbi_is_ddp_tag(tformat, tag))
+ ddp_tag_release(chba, tag);
+}
+
+static int task_reserve_itt(struct iscsi_task *task, itt_t *hdr_itt)
+{
+ struct scsi_cmnd *sc = task->sc;
+ struct iscsi_conn *conn = task->conn;
+ struct iscsi_session *sess = conn->session;
+ struct iscsi_tcp_conn *tcp_conn = conn->dd_data;
+ struct cxgbi_conn *cconn = tcp_conn->dd_data;
+ struct cxgbi_hba *chba = cconn->chba;
+ struct cxgbi_tag_format *tformat = &chba->cdev->tag_format;
+ u32 sw_tag = (sess->age << cconn->task_idx_bits) | task->itt;
+ u32 tag = 0;
+ int err = -EINVAL;
+
+ if (sc &&
+ (scsi_bidi_cmnd(sc) || sc->sc_data_direction == DMA_FROM_DEVICE)) {
+ err = cxgbi_ddp_reserve(cconn->cep->csk, &tag, sw_tag,
+ scsi_in(sc)->length,
+ scsi_in(sc)->table.sgl,
+ scsi_in(sc)->table.nents,
+ GFP_ATOMIC);
+ if (err < 0)
+ libcxgbi_log_debug(1 << CXGBI_DBG_DDP,
+ "csk 0x%p, R task 0x%p, %u,%u, no ddp.\n",
+ cconn->cep->csk, task, scsi_in(sc)->length,
+ scsi_in(sc)->table.nents);
+ }
+
+ if (err < 0)
+ tag = cxgbi_set_non_ddp_tag(tformat, sw_tag);
+ /* the itt need to sent in big-endian order */
+ *hdr_itt = (__force itt_t)htonl(tag);
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_DDP,
+ "cdev 0x%p, task 0x%p, 0x%x(0x%x,0x%x)->0x%x/0x%x.\n",
+ chba->cdev, task, sw_tag, task->itt,
+ sess->age, tag, *hdr_itt);
+ return 0;
+}
+
+void cxgbi_parse_pdu_itt(struct iscsi_conn *conn, itt_t itt, int *idx, int *age)
+{
+ struct iscsi_tcp_conn *tcp_conn = conn->dd_data;
+ struct cxgbi_conn *cconn = tcp_conn->dd_data;
+ struct cxgbi_device *cdev = cconn->chba->cdev;
+ u32 tag = ntohl((__force u32) itt);
+ u32 sw_bits;
+
+ sw_bits = cxgbi_tag_nonrsvd_bits(&cdev->tag_format, tag);
+ if (idx)
+ *idx = sw_bits & ((1 << cconn->task_idx_bits) - 1);
+ if (age)
+ *age = (sw_bits >> cconn->task_idx_bits) & ISCSI_AGE_MASK;
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_DDP,
+ "cdev 0x%p, tag 0x%x/0x%x, -> 0x%x(0x%x,0x%x).\n",
+ cdev, tag, itt, sw_bits, idx ? *idx : 0xFFFFF,
+ age ? *age : 0xFF);
+}
+EXPORT_SYMBOL_GPL(cxgbi_parse_pdu_itt);
+
+void cxgbi_conn_tx_open(struct cxgbi_sock *csk)
+{
+ struct iscsi_conn *conn = csk->user_data;
+
+ if (conn) {
+ libcxgbi_log_debug(1 << CXGBI_DBG_SOCK,
+ "csk 0x%p, cid %d.\n", csk, conn->id);
+ iscsi_conn_queue_work(conn);
+ }
+}
+EXPORT_SYMBOL_GPL(cxgbi_conn_tx_open);
+
+/*
+ * pdu receive, interact with libiscsi_tcp
+ */
+static inline int read_pdu_skb(struct iscsi_conn *conn,
+ struct sk_buff *skb,
+ unsigned int offset,
+ int offloaded)
+{
+ int status = 0;
+ int bytes_read;
+
+ bytes_read = iscsi_tcp_recv_skb(conn, skb, offset, offloaded, &status);
+ switch (status) {
+ case ISCSI_TCP_CONN_ERR:
+ libcxgbi_log_info("skb 0x%p, off %u, %d, TCP_ERR.\n",
+ skb, offset, offloaded);
+ return -EIO;
+ case ISCSI_TCP_SUSPENDED:
+ libcxgbi_log_debug(1 << CXGBI_DBG_PDU_RX,
+ "skb 0x%p, off %u, %d, TCP_SUSPEND, rc %d.\n",
+ skb, offset, offloaded, bytes_read);
+ /* no transfer - just have caller flush queue */
+ return bytes_read;
+ case ISCSI_TCP_SKB_DONE:
+ libcxgbi_log_info("skb 0x%p, off %u, %d, TCP_SKB_DONE.\n",
+ skb, offset, offloaded);
+ /*
+ * pdus should always fit in the skb and we should get
+ * segment done notifcation.
+ */
+ iscsi_conn_printk(KERN_ERR, conn, "Invalid pdu or skb.");
+ return -EFAULT;
+ case ISCSI_TCP_SEGMENT_DONE:
+ libcxgbi_log_debug(1 << CXGBI_DBG_PDU_RX,
+ "skb 0x%p, off %u, %d, TCP_SEG_DONE, rc %d.\n",
+ skb, offset, offloaded, bytes_read);
+ return bytes_read;
+ default:
+ libcxgbi_log_info("skb 0x%p, off %u, %d, invalid status %d.\n",
+ skb, offset, offloaded, status);
+ return -EINVAL;
+ }
+}
+
+static int skb_read_pdu_bhs(struct iscsi_conn *conn, struct sk_buff *skb)
+{
+ struct iscsi_tcp_conn *tcp_conn = conn->dd_data;
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_PDU_RX,
+ "conn 0x%p, skb 0x%p, len %u, flag 0x%lx.\n",
+ conn, skb, skb->len, cxgbi_skcb_flags(skb));
+
+ if (!iscsi_tcp_recv_segment_is_hdr(tcp_conn)) {
+ libcxgbi_log_info("conn 0x%p, skb 0x%p, not hdr.\n", conn, skb);
+ iscsi_conn_failure(conn, ISCSI_ERR_PROTO);
+ return -EIO;
+ }
+
+ if (conn->hdrdgst_en &&
+ cxgbi_skcb_test_flag(skb, SKCBF_RX_HCRC_ERR)) {
+ libcxgbi_log_info("conn 0x%p, skb 0x%p, hcrc.\n", conn, skb);
+ iscsi_conn_failure(conn, ISCSI_ERR_HDR_DGST);
+ return -EIO;
+ }
+
+ return read_pdu_skb(conn, skb, 0, 0);
+}
+
+static int skb_read_pdu_data(struct iscsi_conn *conn, struct sk_buff *lskb,
+ struct sk_buff *skb, unsigned int offset)
+{
+ struct iscsi_tcp_conn *tcp_conn = conn->dd_data;
+ bool offloaded = 0;
+ int opcode = tcp_conn->in.hdr->opcode & ISCSI_OPCODE_MASK;
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_PDU_RX,
+ "conn 0x%p, skb 0x%p, len %u, flag 0x%lx.\n",
+ conn, skb, skb->len, cxgbi_skcb_flags(skb));
+
+ if (conn->datadgst_en &&
+ cxgbi_skcb_test_flag(lskb, SKCBF_RX_DCRC_ERR)) {
+ libcxgbi_log_info("conn 0x%p, skb 0x%p, dcrc 0x%lx.\n",
+ conn, lskb, cxgbi_skcb_flags(lskb));
+ iscsi_conn_failure(conn, ISCSI_ERR_DATA_DGST);
+ return -EIO;
+ }
+
+ if (iscsi_tcp_recv_segment_is_hdr(tcp_conn))
+ return 0;
+
+ /* coalesced, add header digest length */
+ if (lskb == skb && conn->hdrdgst_en)
+ offset += ISCSI_DIGEST_SIZE;
+
+ if (cxgbi_skcb_test_flag(lskb, SKCBF_RX_DATA_DDPD))
+ offloaded = 1;
+
+ if (opcode == ISCSI_OP_SCSI_DATA_IN)
+ libcxgbi_log_debug(1 << CXGBI_DBG_PDU_RX,
+ "skb 0x%p, op 0x%x, itt 0x%x, %u %s ddp'ed.\n",
+ skb, opcode, ntohl(tcp_conn->in.hdr->itt),
+ tcp_conn->in.datalen, offloaded ? "is" : "not");
+
+ return read_pdu_skb(conn, skb, offset, offloaded);
+}
+
+static void csk_return_rx_credits(struct cxgbi_sock *csk, int copied)
+{
+ struct cxgbi_device *cdev = csk->cdev;
+ int must_send;
+ u32 credits;
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_PDU_RX,
+ "csk 0x%p,%u,0x%lu,%u, seq %u, wup %u, thre %u, %u.\n",
+ csk, csk->state, csk->flags, csk->tid,
+ csk->copied_seq, csk->rcv_wup, cdev->rx_credit_thres,
+ cdev->rcv_win);
+
+ if (csk->state != CTP_ESTABLISHED)
+ return;
+
+ credits = csk->copied_seq - csk->rcv_wup;
+ if (unlikely(!credits))
+ return;
+ if (unlikely(cdev->rx_credit_thres == 0))
+ return;
+
+ must_send = credits + 16384 >= cdev->rcv_win;
+ if (must_send || credits >= cdev->rx_credit_thres)
+ csk->rcv_wup += cdev->csk_send_rx_credits(csk, credits);
+}
+
+void cxgbi_conn_pdu_ready(struct cxgbi_sock *csk)
+{
+ struct cxgbi_device *cdev = csk->cdev;
+ struct iscsi_conn *conn = csk->user_data;
+ struct sk_buff *skb;
+ unsigned int read = 0;
+ int err = 0;
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_PDU_RX,
+ "csk 0x%p, conn 0x%p.\n", csk, conn);
+
+ if (unlikely(!conn || conn->suspend_rx)) {
+ libcxgbi_log_debug(1 << CXGBI_DBG_PDU_RX,
+ "csk 0x%p, conn 0x%p, id %d, suspend_rx %lu!\n",
+ csk, conn, conn ? conn->id : 0xFF,
+ conn ? conn->suspend_rx : 0xFF);
+ read_unlock(&csk->callback_lock);
+ return;
+ }
+
+ while (!err) {
+ read_lock(&csk->callback_lock);
+ skb = skb_peek(&csk->receive_queue);
+ if (!skb ||
+ !(cxgbi_skcb_test_flag(skb, SKCBF_RX_STATUS))) {
+ if (skb)
+ libcxgbi_log_debug(1 << CXGBI_DBG_PDU_RX,
+ "skb 0x%p, NOT ready 0x%lx.\n",
+ skb, cxgbi_skcb_flags(skb));
+ read_unlock(&csk->callback_lock);
+ break;
+ }
+ __skb_unlink(skb, &csk->receive_queue);
+ read_unlock(&csk->callback_lock);
+
+ read += cxgbi_skcb_rx_pdulen(skb);
+ libcxgbi_log_debug(1 << CXGBI_DBG_PDU_RX,
+ "csk 0x%p, skb 0x%p,%u,f 0x%lx, pdu len %u.\n",
+ csk, skb, skb->len, cxgbi_skcb_flags(skb),
+ cxgbi_skcb_rx_pdulen(skb));
+
+ if (cxgbi_skcb_test_flag(skb, SKCBF_RX_COALESCED)) {
+ err = skb_read_pdu_bhs(conn, skb);
+ if (err < 0)
+ break;
+ err = skb_read_pdu_data(conn, skb, skb,
+ err + cdev->skb_rx_extra);
+ } else {
+ err = skb_read_pdu_bhs(conn, skb);
+ if (err < 0)
+ break;
+ if (cxgbi_skcb_test_flag(skb, SKCBF_RX_DATA)) {
+ struct sk_buff *dskb;
+
+ read_lock(&csk->callback_lock);
+ dskb = skb_peek(&csk->receive_queue);
+ if (!dskb) {
+ read_unlock(&csk->callback_lock);
+ libcxgbi_log_error(
+ "csk 0x%p, data NULL.\n", csk);
+ err = -EAGAIN;
+ break;
+ }
+ __skb_unlink(dskb, &csk->receive_queue);
+ read_unlock(&csk->callback_lock);
+
+ err = skb_read_pdu_data(conn, skb, dskb, 0);
+ __kfree_skb(dskb);
+ } else
+ err = skb_read_pdu_data(conn, skb, skb, 0);
+ }
+ if (err < 0)
+ break;
+
+ __kfree_skb(skb);
+ }
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_PDU_RX,
+ "csk 0x%p, read %u.\n", csk, read);
+ if (read) {
+ csk->copied_seq += read;
+ csk_return_rx_credits(csk, read);
+ conn->rxdata_octets += read;
+ }
+
+ if (err < 0) {
+ libcxgbi_log_info("csk 0x%p, conn 0x%p, rx failed err %d.\n",
+ csk, conn, err);
+ iscsi_conn_failure(conn, ISCSI_ERR_CONN_FAILED);
+ }
+}
+EXPORT_SYMBOL_GPL(cxgbi_conn_pdu_ready);
+
+static int sgl_seek_offset(struct scatterlist *sgl, unsigned int sgcnt,
+ unsigned int offset, unsigned int *off,
+ struct scatterlist **sgp)
+{
+ int i;
+ struct scatterlist *sg;
+
+ for_each_sg(sgl, sg, sgcnt, i) {
+ if (offset < sg->length) {
+ *off = offset;
+ *sgp = sg;
+ return 0;
+ }
+ offset -= sg->length;
+ }
+ return -EFAULT;
+}
+
+static int sgl_read_to_frags(struct scatterlist *sg, unsigned int sgoffset,
+ unsigned int dlen, skb_frag_t *frags,
+ int frag_max)
+{
+ unsigned int datalen = dlen;
+ unsigned int sglen = sg->length - sgoffset;
+ struct page *page = sg_page(sg);
+ int i;
+
+ i = 0;
+ do {
+ unsigned int copy;
+
+ if (!sglen) {
+ sg = sg_next(sg);
+ if (!sg) {
+ libcxgbi_log_warn("sg %d NULL, len %u/%u.\n",
+ i, datalen, dlen);
+ return -EINVAL;
+ }
+ sgoffset = 0;
+ sglen = sg->length;
+ page = sg_page(sg);
+
+ }
+ copy = min(datalen, sglen);
+ if (i && page == frags[i - 1].page &&
+ sgoffset + sg->offset ==
+ frags[i - 1].page_offset + frags[i - 1].size) {
+ frags[i - 1].size += copy;
+ } else {
+ if (i >= frag_max) {
+ libcxgbi_log_warn("too many pages %u, "
+ "dlen %u.\n", frag_max, dlen);
+ return -EINVAL;
+ }
+
+ frags[i].page = page;
+ frags[i].page_offset = sg->offset + sgoffset;
+ frags[i].size = copy;
+ i++;
+ }
+ datalen -= copy;
+ sgoffset += copy;
+ sglen -= copy;
+ } while (datalen);
+
+ return i;
+}
+
+int cxgbi_conn_alloc_pdu(struct iscsi_task *task, u8 opcode)
+{
+ struct iscsi_tcp_conn *tcp_conn = task->conn->dd_data;
+ struct cxgbi_conn *cconn = tcp_conn->dd_data;
+ struct cxgbi_device *cdev = cconn->chba->cdev;
+ struct iscsi_conn *conn = task->conn;
+ struct iscsi_tcp_task *tcp_task = task->dd_data;
+ struct cxgbi_task_data *tdata = task->dd_data + sizeof(*tcp_task);
+ struct scsi_cmnd *sc = task->sc;
+ int headroom = SKB_TX_ISCSI_PDU_HEADER_MAX;
+
+ tcp_task->dd_data = tdata;
+ task->hdr = NULL;
+
+ if (SKB_MAX_HEAD(cdev->skb_tx_rsvd) > (512 * MAX_SKB_FRAGS) &&
+ (opcode == ISCSI_OP_SCSI_DATA_OUT ||
+ (opcode == ISCSI_OP_SCSI_CMD &&
+ (scsi_bidi_cmnd(sc) || sc->sc_data_direction == DMA_TO_DEVICE))))
+ /* data could goes into skb head */
+ headroom += min_t(unsigned int,
+ SKB_MAX_HEAD(cdev->skb_tx_rsvd),
+ conn->max_xmit_dlength);
+
+ tdata->skb = alloc_skb(cdev->skb_tx_rsvd + headroom, GFP_ATOMIC);
+ if (!tdata->skb) {
+ libcxgbi_log_warn("alloc skb %u+%u, opcode 0x%x failed.\n",
+ cdev->skb_tx_rsvd, headroom, opcode);
+ return -ENOMEM;
+ }
+
+ skb_reserve(tdata->skb, cdev->skb_tx_rsvd);
+ task->hdr = (struct iscsi_hdr *)tdata->skb->data;
+ task->hdr_max = SKB_TX_ISCSI_PDU_HEADER_MAX; /* BHS + AHS */
+
+ /* data_out uses scsi_cmd's itt */
+ if (opcode != ISCSI_OP_SCSI_DATA_OUT)
+ task_reserve_itt(task, &task->hdr->itt);
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_ISCSI | 1 << CXGBI_DBG_PDU_TX,
+ "task 0x%p, op 0x%x, skb 0x%p,%u+%u/%u, itt 0x%x.\n",
+ task, opcode, tdata->skb, cdev->skb_tx_rsvd, headroom,
+ conn->max_xmit_dlength, ntohl(task->hdr->itt));
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(cxgbi_conn_alloc_pdu);
+
+static inline void tx_skb_setmode(struct sk_buff *skb, int hcrc, int dcrc)
+{
+ u8 submode = 0;
+
+ if (hcrc)
+ submode |= 1;
+ if (dcrc)
+ submode |= 2;
+ cxgbi_skcb_ulp_mode(skb) = (ULP2_MODE_ISCSI << 4) | submode;
+}
+
+int cxgbi_conn_init_pdu(struct iscsi_task *task, unsigned int offset,
+ unsigned int count)
+{
+ struct iscsi_conn *conn = task->conn;
+ struct iscsi_tcp_task *tcp_task = task->dd_data;
+ struct cxgbi_task_data *tdata = tcp_task->dd_data;
+ struct sk_buff *skb = tdata->skb;
+ unsigned int datalen = count;
+ int i, padlen = iscsi_padding(count);
+ struct page *pg;
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_ISCSI | 1 << CXGBI_DBG_PDU_TX,
+ "task 0x%p,0x%p, skb 0x%p, 0x%x,0x%x,0x%x, %u+%u.\n",
+ task, task->sc, skb, (*skb->data) & ISCSI_OPCODE_MASK,
+ ntohl(task->cmdsn), ntohl(task->hdr->itt),
+ offset, count);
+
+ skb_put(skb, task->hdr_len);
+ tx_skb_setmode(skb, conn->hdrdgst_en, datalen ? conn->datadgst_en : 0);
+ if (!count)
+ return 0;
+
+ if (task->sc) {
+ struct scsi_data_buffer *sdb = scsi_out(task->sc);
+ struct scatterlist *sg = NULL;
+ int err;
+
+ tdata->offset = offset;
+ tdata->count = count;
+ err = sgl_seek_offset(
+ sdb->table.sgl, sdb->table.nents,
+ tdata->offset, &tdata->sgoffset, &sg);
+ if (err < 0) {
+ libcxgbi_log_warn("tpdu, sgl %u, bad offset %u/%u.\n",
+ sdb->table.nents, tdata->offset,
+ sdb->length);
+ return err;
+ }
+ err = sgl_read_to_frags(sg, tdata->sgoffset, tdata->count,
+ tdata->frags, MAX_PDU_FRAGS);
+ if (err < 0) {
+ libcxgbi_log_warn("tpdu, sgl %u, bad offset %u + %u.\n",
+ sdb->table.nents,
+ tdata->offset,
+ tdata->count);
+ return err;
+ }
+ tdata->nr_frags = err;
+
+ if (tdata->nr_frags > MAX_SKB_FRAGS ||
+ (padlen && tdata->nr_frags == MAX_SKB_FRAGS)) {
+ char *dst = skb->data + task->hdr_len;
+ skb_frag_t *frag = tdata->frags;
+
+ /* data fits in the skb's headroom */
+ for (i = 0; i < tdata->nr_frags; i++, frag++) {
+ char *src = kmap_atomic(frag->page,
+ KM_SOFTIRQ0);
+
+ memcpy(dst, src+frag->page_offset, frag->size);
+ dst += frag->size;
+ kunmap_atomic(src, KM_SOFTIRQ0);
+ }
+ if (padlen) {
+ memset(dst, 0, padlen);
+ padlen = 0;
+ }
+ skb_put(skb, count + padlen);
+ } else {
+ /* data fit into frag_list */
+ for (i = 0; i < tdata->nr_frags; i++)
+ get_page(tdata->frags[i].page);
+
+ memcpy(skb_shinfo(skb)->frags, tdata->frags,
+ sizeof(skb_frag_t) * tdata->nr_frags);
+ skb_shinfo(skb)->nr_frags = tdata->nr_frags;
+ skb->len += count;
+ skb->data_len += count;
+ skb->truesize += count;
+ }
+
+ } else {
+ pg = virt_to_page(task->data);
+
+ get_page(pg);
+ skb_fill_page_desc(skb, 0, pg, offset_in_page(task->data),
+ count);
+ skb->len += count;
+ skb->data_len += count;
+ skb->truesize += count;
+ }
+
+ if (padlen) {
+ i = skb_shinfo(skb)->nr_frags;
+ skb_fill_page_desc(skb, skb_shinfo(skb)->nr_frags,
+ virt_to_page(padding), offset_in_page(padding),
+ padlen);
+
+ skb->data_len += padlen;
+ skb->truesize += padlen;
+ skb->len += padlen;
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(cxgbi_conn_init_pdu);
+
+int cxgbi_conn_xmit_pdu(struct iscsi_task *task)
+{
+ struct iscsi_tcp_conn *tcp_conn = task->conn->dd_data;
+ struct cxgbi_conn *cconn = tcp_conn->dd_data;
+ struct iscsi_tcp_task *tcp_task = task->dd_data;
+ struct cxgbi_task_data *tdata = tcp_task->dd_data;
+ struct sk_buff *skb = tdata->skb;
+ unsigned int datalen;
+ int err;
+
+ if (!skb) {
+ libcxgbi_log_debug(1 << CXGBI_DBG_ISCSI | 1 << CXGBI_DBG_PDU_TX,
+ "task 0x%p, skb NULL.\n", task);
+ return 0;
+ }
+
+ datalen = skb->data_len;
+ tdata->skb = NULL;
+ err = cxgbi_sock_send_pdus(cconn->cep->csk, skb);
+ if (err > 0) {
+ int pdulen = err;
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_PDU_TX,
+ "task 0x%p,0x%p, skb 0x%p, len %u/%u, rv %d.\n",
+ task, task->sc, skb, skb->len, skb->data_len,
+ err);
+
+ if (task->conn->hdrdgst_en)
+ pdulen += ISCSI_DIGEST_SIZE;
+
+ if (datalen && task->conn->datadgst_en)
+ pdulen += ISCSI_DIGEST_SIZE;
+
+ task->conn->txdata_octets += pdulen;
+ return 0;
+ }
+
+ if (err == -EAGAIN || err == -ENOBUFS) {
+ libcxgbi_log_debug(1 << CXGBI_DBG_PDU_TX,
+ "task 0x%p, skb 0x%p, len %u/%u, %d EAGAIN.\n",
+ task, skb, skb->len, skb->data_len, err);
+ /* reset skb to send when we are called again */
+ tdata->skb = skb;
+ return err;
+ }
+
+ kfree_skb(skb);
+ libcxgbi_log_debug(1 << CXGBI_DBG_ISCSI | 1 << CXGBI_DBG_PDU_TX,
+ "itt 0x%x, skb 0x%p, len %u/%u, xmit err %d.\n",
+ task->itt, skb, skb->len, skb->data_len, err);
+ iscsi_conn_printk(KERN_ERR, task->conn, "xmit err %d.\n", err);
+ iscsi_conn_failure(task->conn, ISCSI_ERR_XMIT_FAILED);
+ return err;
+}
+EXPORT_SYMBOL_GPL(cxgbi_conn_xmit_pdu);
+
+void cxgbi_cleanup_task(struct iscsi_task *task)
+{
+ struct cxgbi_task_data *tdata = task->dd_data +
+ sizeof(struct iscsi_tcp_task);
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_ISCSI,
+ "task 0x%p, skb 0x%p, itt 0x%x.\n",
+ task, tdata->skb, task->hdr_itt);
+
+ /* never reached the xmit task callout */
+ if (tdata->skb)
+ __kfree_skb(tdata->skb);
+ memset(tdata, 0, sizeof(*tdata));
+
+ task_release_itt(task, task->hdr_itt);
+ iscsi_tcp_cleanup_task(task);
+}
+EXPORT_SYMBOL_GPL(cxgbi_cleanup_task);
+
+void cxgbi_get_conn_stats(struct iscsi_cls_conn *cls_conn,
+ struct iscsi_stats *stats)
+{
+ struct iscsi_conn *conn = cls_conn->dd_data;
+
+ stats->txdata_octets = conn->txdata_octets;
+ stats->rxdata_octets = conn->rxdata_octets;
+ stats->scsicmd_pdus = conn->scsicmd_pdus_cnt;
+ stats->dataout_pdus = conn->dataout_pdus_cnt;
+ stats->scsirsp_pdus = conn->scsirsp_pdus_cnt;
+ stats->datain_pdus = conn->datain_pdus_cnt;
+ stats->r2t_pdus = conn->r2t_pdus_cnt;
+ stats->tmfcmd_pdus = conn->tmfcmd_pdus_cnt;
+ stats->tmfrsp_pdus = conn->tmfrsp_pdus_cnt;
+ stats->digest_err = 0;
+ stats->timeout_err = 0;
+ stats->custom_length = 1;
+ strcpy(stats->custom[0].desc, "eh_abort_cnt");
+ stats->custom[0].value = conn->eh_abort_cnt;
+}
+EXPORT_SYMBOL_GPL(cxgbi_get_conn_stats);
+
+static int cxgbi_conn_max_xmit_dlength(struct iscsi_conn *conn)
+{
+ struct iscsi_tcp_conn *tcp_conn = conn->dd_data;
+ struct cxgbi_conn *cconn = tcp_conn->dd_data;
+ struct cxgbi_device *cdev = cconn->chba->cdev;
+ unsigned int headroom = SKB_MAX_HEAD(cdev->skb_tx_rsvd);
+ unsigned int max_def = 512 * MAX_SKB_FRAGS;
+ unsigned int max = max(max_def, headroom);
+
+ max = min(cconn->chba->cdev->tx_max_size, max);
+ if (conn->max_xmit_dlength)
+ conn->max_xmit_dlength = min(conn->max_xmit_dlength, max);
+ else
+ conn->max_xmit_dlength = max;
+ cxgbi_align_pdu_size(conn->max_xmit_dlength);
+
+ return 0;
+}
+
+static int cxgbi_conn_max_recv_dlength(struct iscsi_conn *conn)
+{
+ struct iscsi_tcp_conn *tcp_conn = conn->dd_data;
+ struct cxgbi_conn *cconn = tcp_conn->dd_data;
+ unsigned int max = cconn->chba->cdev->rx_max_size;
+
+ cxgbi_align_pdu_size(max);
+
+ if (conn->max_recv_dlength) {
+ if (conn->max_recv_dlength > max) {
+ libcxgbi_log_error("MaxRecvDataSegmentLength %u needs "
+ "to be <= %u.\n",
+ conn->max_recv_dlength, max);
+ return -EINVAL;
+ }
+ conn->max_recv_dlength = min(conn->max_recv_dlength, max);
+ cxgbi_align_pdu_size(conn->max_recv_dlength);
+ } else
+ conn->max_recv_dlength = max;
+
+ return 0;
+}
+
+int cxgbi_set_conn_param(struct iscsi_cls_conn *cls_conn,
+ enum iscsi_param param, char *buf, int buflen)
+{
+ struct iscsi_conn *conn = cls_conn->dd_data;
+ struct iscsi_session *session = conn->session;
+ struct iscsi_tcp_conn *tcp_conn = conn->dd_data;
+ struct cxgbi_conn *cconn = tcp_conn->dd_data;
+ struct cxgbi_sock *csk = cconn->cep->csk;
+ int value, err = 0;
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_ISCSI,
+ "cls_conn 0x%p, param %d, buf(%d) %s.\n",
+ cls_conn, param, buflen, buf);
+
+ switch (param) {
+ case ISCSI_PARAM_HDRDGST_EN:
+ err = iscsi_set_param(cls_conn, param, buf, buflen);
+ if (!err && conn->hdrdgst_en)
+ err = csk->cdev->csk_ddp_setup_digest(csk, csk->tid,
+ conn->hdrdgst_en,
+ conn->datadgst_en, 0);
+ break;
+ case ISCSI_PARAM_DATADGST_EN:
+ err = iscsi_set_param(cls_conn, param, buf, buflen);
+ if (!err && conn->datadgst_en)
+ err = csk->cdev->csk_ddp_setup_digest(csk, csk->tid,
+ conn->hdrdgst_en,
+ conn->datadgst_en, 0);
+ break;
+ case ISCSI_PARAM_MAX_R2T:
+ sscanf(buf, "%d", &value);
+ if (value <= 0 || !is_power_of_2(value))
+ return -EINVAL;
+ if (session->max_r2t == value)
+ break;
+ iscsi_tcp_r2tpool_free(session);
+ err = iscsi_set_param(cls_conn, param, buf, buflen);
+ if (!err && iscsi_tcp_r2tpool_alloc(session))
+ return -ENOMEM;
+ case ISCSI_PARAM_MAX_RECV_DLENGTH:
+ err = iscsi_set_param(cls_conn, param, buf, buflen);
+ if (!err)
+ err = cxgbi_conn_max_recv_dlength(conn);
+ break;
+ case ISCSI_PARAM_MAX_XMIT_DLENGTH:
+ err = iscsi_set_param(cls_conn, param, buf, buflen);
+ if (!err)
+ err = cxgbi_conn_max_xmit_dlength(conn);
+ break;
+ default:
+ return iscsi_set_param(cls_conn, param, buf, buflen);
+ }
+ return err;
+}
+EXPORT_SYMBOL_GPL(cxgbi_set_conn_param);
+
+int cxgbi_get_conn_param(struct iscsi_cls_conn *cls_conn,
+ enum iscsi_param param, char *buf)
+{
+ struct iscsi_conn *iconn = cls_conn->dd_data;
+ int len;
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_ISCSI,
+ "cls_conn 0x%p, param %d.\n", cls_conn, param);
+
+ switch (param) {
+ case ISCSI_PARAM_CONN_PORT:
+ spin_lock_bh(&iconn->session->lock);
+ len = sprintf(buf, "%hu\n", iconn->portal_port);
+ spin_unlock_bh(&iconn->session->lock);
+ break;
+ case ISCSI_PARAM_CONN_ADDRESS:
+ spin_lock_bh(&iconn->session->lock);
+ len = sprintf(buf, "%s\n", iconn->portal_address);
+ spin_unlock_bh(&iconn->session->lock);
+ break;
+ default:
+ return iscsi_conn_get_param(cls_conn, param, buf);
+ }
+ return len;
+}
+EXPORT_SYMBOL_GPL(cxgbi_get_conn_param);
+
+struct iscsi_cls_conn *
+cxgbi_create_conn(struct iscsi_cls_session *cls_session, u32 cid)
+{
+ struct iscsi_cls_conn *cls_conn;
+ struct iscsi_conn *conn;
+ struct iscsi_tcp_conn *tcp_conn;
+ struct cxgbi_conn *cconn;
+
+ cls_conn = iscsi_tcp_conn_setup(cls_session, sizeof(*cconn), cid);
+ if (!cls_conn)
+ return NULL;
+
+ conn = cls_conn->dd_data;
+ tcp_conn = conn->dd_data;
+ cconn = tcp_conn->dd_data;
+ cconn->iconn = conn;
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_ISCSI,
+ "cid %u(0x%x), cls 0x%p,0x%p, conn 0x%p,0x%p,0x%p.\n",
+ cid, cid, cls_session, cls_conn, conn, tcp_conn, cconn);
+
+ return cls_conn;
+}
+EXPORT_SYMBOL_GPL(cxgbi_create_conn);
+
+int cxgbi_bind_conn(struct iscsi_cls_session *cls_session,
+ struct iscsi_cls_conn *cls_conn,
+ u64 transport_eph, int is_leading)
+{
+ struct iscsi_conn *conn = cls_conn->dd_data;
+ struct iscsi_tcp_conn *tcp_conn = conn->dd_data;
+ struct cxgbi_conn *cconn = tcp_conn->dd_data;
+ struct iscsi_endpoint *ep;
+ struct cxgbi_endpoint *cep;
+ struct cxgbi_sock *csk;
+ int err;
+
+ ep = iscsi_lookup_endpoint(transport_eph);
+ if (!ep)
+ return -EINVAL;
+
+ /* setup ddp pagesize */
+ cep = ep->dd_data;
+ csk = cep->csk;
+ err = csk->cdev->csk_ddp_setup_pgidx(csk, csk->tid, page_idx, 0);
+ if (err < 0)
+ return err;
+
+ err = iscsi_conn_bind(cls_session, cls_conn, is_leading);
+ if (err)
+ return -EINVAL;
+
+ /* calculate the tag idx bits needed for this conn based on cmds_max */
+ cconn->task_idx_bits = (__ilog2_u32(conn->session->cmds_max - 1)) + 1;
+
+ write_lock(&csk->callback_lock);
+ csk->user_data = conn;
+ cconn->chba = cep->chba;
+ cconn->cep = cep;
+ cep->cconn = cconn;
+ write_unlock(&csk->callback_lock);
+
+ cxgbi_conn_max_xmit_dlength(conn);
+ cxgbi_conn_max_recv_dlength(conn);
+
+ spin_lock_bh(&conn->session->lock);
+ sprintf(conn->portal_address, "%pI4", &csk->daddr.sin_addr.s_addr);
+ conn->portal_port = ntohs(csk->daddr.sin_port);
+ spin_unlock_bh(&conn->session->lock);
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_ISCSI,
+ "cls 0x%p,0x%p, ep 0x%p, cconn 0x%p, csk 0x%p.\n",
+ cls_session, cls_conn, ep, cconn, csk);
+ /* init recv engine */
+ iscsi_tcp_hdr_recv_prep(tcp_conn);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(cxgbi_bind_conn);
+
+struct iscsi_cls_session *cxgbi_create_session(struct iscsi_endpoint *ep,
+ u16 cmds_max, u16 qdepth,
+ u32 initial_cmdsn)
+{
+ struct cxgbi_endpoint *cep;
+ struct cxgbi_hba *chba;
+ struct Scsi_Host *shost;
+ struct iscsi_cls_session *cls_session;
+ struct iscsi_session *session;
+
+ if (!ep) {
+ libcxgbi_log_error("missing endpoint\n");
+ return NULL;
+ }
+
+ cep = ep->dd_data;
+ chba = cep->chba;
+ shost = chba->shost;
+
+ BUG_ON(chba != iscsi_host_priv(shost));
+
+ cls_session = iscsi_session_setup(chba->cdev->itp, shost,
+ cmds_max, 0,
+ sizeof(struct iscsi_tcp_task) +
+ sizeof(struct cxgbi_task_data),
+ initial_cmdsn, ISCSI_MAX_TARGET);
+ if (!cls_session)
+ return NULL;
+
+ session = cls_session->dd_data;
+ if (iscsi_tcp_r2tpool_alloc(session))
+ goto remove_session;
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_ISCSI,
+ "ep 0x%p, cls sess 0x%p.\n", ep, cls_session);
+ return cls_session;
+
+remove_session:
+ iscsi_session_teardown(cls_session);
+ return NULL;
+}
+EXPORT_SYMBOL_GPL(cxgbi_create_session);
+
+void cxgbi_destroy_session(struct iscsi_cls_session *cls_session)
+{
+ libcxgbi_log_debug(1 << CXGBI_DBG_ISCSI,
+ "cls sess 0x%p.\n", cls_session);
+
+ iscsi_tcp_r2tpool_free(cls_session->dd_data);
+ iscsi_session_teardown(cls_session);
+}
+EXPORT_SYMBOL_GPL(cxgbi_destroy_session);
+
+int cxgbi_set_host_param(struct Scsi_Host *shost, enum iscsi_host_param param,
+ char *buf, int buflen)
+{
+ struct cxgbi_hba *chba = iscsi_host_priv(shost);
+
+ if (!chba->ndev) {
+ shost_printk(KERN_ERR, shost, "Could not get host param. "
+ "netdev for host not set\n");
+ return -ENODEV;
+ }
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_ISCSI,
+ "shost 0x%p, hba 0x%p,%s, param %d, buf(%d) %s.\n",
+ shost, chba, chba->ndev->name, param, buflen, buf);
+
+ switch (param) {
+ case ISCSI_HOST_PARAM_IPADDRESS:
+ {
+ __be32 addr = in_aton(buf);
+ cxgbi_set_iscsi_ipv4(chba, addr);
+ libcxgbi_log_debug(1 << CXGBI_DBG_ISCSI,
+ "hba %s, set ipv4 %pI4.\n", chba->ndev->name, &addr);
+ return 0;
+ }
+ case ISCSI_HOST_PARAM_HWADDRESS:
+ case ISCSI_HOST_PARAM_NETDEV_NAME:
+ return 0;
+ default:
+ return iscsi_host_set_param(shost, param, buf, buflen);
+ }
+}
+EXPORT_SYMBOL_GPL(cxgbi_set_host_param);
+
+int cxgbi_get_host_param(struct Scsi_Host *shost, enum iscsi_host_param param,
+ char *buf)
+{
+ struct cxgbi_hba *chba = iscsi_host_priv(shost);
+ int len = 0;
+
+ if (!chba->ndev) {
+ shost_printk(KERN_ERR, shost, "Could not get host param. "
+ "netdev for host not set.\n");
+ return -ENODEV;
+ }
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_ISCSI,
+ "shost 0x%p, hba 0x%p,%s, param %d.\n",
+ shost, chba, chba->ndev->name, param);
+
+ switch (param) {
+ case ISCSI_HOST_PARAM_HWADDRESS:
+ len = sysfs_format_mac(buf, chba->ndev->dev_addr, 6);
+ break;
+ case ISCSI_HOST_PARAM_NETDEV_NAME:
+ len = sprintf(buf, "%s\n", chba->ndev->name);
+ break;
+ case ISCSI_HOST_PARAM_IPADDRESS:
+ {
+ __be32 addr;
+
+ addr = cxgbi_get_iscsi_ipv4(chba);
+ len = sprintf(buf, "%pI4", &addr);
+ libcxgbi_log_debug(1 << CXGBI_DBG_ISCSI,
+ "hba %s, ipv4 %pI4.\n", chba->ndev->name, &addr);
+ break;
+ }
+ default:
+ return iscsi_host_get_param(shost, param, buf);
+ }
+
+ return len;
+}
+EXPORT_SYMBOL_GPL(cxgbi_get_host_param);
+
+struct iscsi_endpoint *cxgbi_ep_connect(struct Scsi_Host *shost,
+ struct sockaddr *dst_addr,
+ int non_blocking)
+{
+ struct iscsi_endpoint *ep;
+ struct cxgbi_endpoint *cep;
+ struct cxgbi_hba *hba = NULL;
+ struct cxgbi_sock *csk;
+ int err = -EINVAL;
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_ISCSI | 1 << CXGBI_DBG_SOCK,
+ "shost 0x%p, non_blocking %d, dst_addr 0x%p.\n",
+ shost, non_blocking, dst_addr);
+
+ if (shost) {
+ hba = iscsi_host_priv(shost);
+ if (!hba) {
+ libcxgbi_log_info("shost 0x%p, priv NULL.\n", shost);
+ goto err_out;
+ }
+ }
+
+ csk = cxgbi_check_route(dst_addr);
+ if (IS_ERR(csk))
+ return (struct iscsi_endpoint *)csk;
+ cxgbi_sock_get(csk);
+
+ if (!hba)
+ hba = csk->cdev->hbas[csk->port_id];
+ else if (hba != csk->cdev->hbas[csk->port_id]) {
+ libcxgbi_log_info("Could not connect through requested host %u"
+ "hba 0x%p != 0x%p (%u).\n",
+ shost->host_no, hba,
+ csk->cdev->hbas[csk->port_id], csk->port_id);
+ err = -ENOSPC;
+ goto release_conn;
+ }
+
+ err = sock_get_port(csk);
+ if (err)
+ goto release_conn;
+
+ cxgbi_sock_set_state(csk, CTP_CONNECTING);
+ err = csk->cdev->csk_init_act_open(csk);
+ if (err)
+ goto release_conn;
+
+ if (cxgbi_sock_is_closing(csk)) {
+ err = -ENOSPC;
+ libcxgbi_log_info("csk 0x%p is closing.\n", csk);
+ goto release_conn;
+ }
+
+ ep = iscsi_create_endpoint(sizeof(*cep));
+ if (!ep) {
+ err = -ENOMEM;
+ libcxgbi_log_info("iscsi alloc ep, OOM\n");
+ goto release_conn;
+ }
+
+ cep = ep->dd_data;
+ cep->csk = csk;
+ cep->chba = hba;
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_ISCSI | 1 << CXGBI_DBG_SOCK,
+ "ep 0x%p, cep 0x%p, csk 0x%p, hba 0x%p,%s.\n",
+ ep, cep, csk, hba, hba->ndev->name);
+ return ep;
+
+release_conn:
+ cxgbi_sock_put(csk);
+ cxgbi_sock_closed(csk);
+err_out:
+ return ERR_PTR(err);
+}
+EXPORT_SYMBOL_GPL(cxgbi_ep_connect);
+
+int cxgbi_ep_poll(struct iscsi_endpoint *ep, int timeout_ms)
+{
+ struct cxgbi_endpoint *cep = ep->dd_data;
+ struct cxgbi_sock *csk = cep->csk;
+
+ if (!cxgbi_sock_is_established(csk))
+ return 0;
+ return 1;
+}
+EXPORT_SYMBOL_GPL(cxgbi_ep_poll);
+
+void cxgbi_ep_disconnect(struct iscsi_endpoint *ep)
+{
+ struct cxgbi_endpoint *cep = ep->dd_data;
+ struct cxgbi_conn *cconn = cep->cconn;
+ struct cxgbi_sock *csk = cep->csk;
+
+ libcxgbi_log_debug(1 << CXGBI_DBG_ISCSI | 1 << CXGBI_DBG_SOCK,
+ "ep 0x%p, cep 0x%p, cconn 0x%p, csk 0x%p,%u,0x%lx.\n",
+ ep, cep, cconn, csk, csk->state, csk->flags);
+
+ if (cconn && cconn->iconn) {
+ iscsi_suspend_tx(cconn->iconn);
+ write_lock_bh(&csk->callback_lock);
+ cep->csk->user_data = NULL;
+ cconn->cep = NULL;
+ write_unlock_bh(&csk->callback_lock);
+ }
+ iscsi_destroy_endpoint(ep);
+
+ if (likely(csk->state >= CTP_ESTABLISHED))
+ need_active_close(csk);
+ else
+ cxgbi_sock_closed(csk);
+
+ cxgbi_sock_put(csk);
+}
+EXPORT_SYMBOL_GPL(cxgbi_ep_disconnect);
+
+int cxgbi_iscsi_init(struct iscsi_transport *itp,
+ struct scsi_transport_template **stt)
+{
+ *stt = iscsi_register_transport(itp);
+ if (*stt == NULL) {
+ libcxgbi_log_error("unable to register %s transport 0x%p.\n",
+ itp->name, itp);
+ return -ENODEV;
+ }
+ libcxgbi_log_debug(1 << CXGBI_DBG_ISCSI,
+ "%s, registered iscsi transport 0x%p.\n",
+ itp->name, stt);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(cxgbi_iscsi_init);
+
+void cxgbi_iscsi_cleanup(struct iscsi_transport *itp,
+ struct scsi_transport_template **stt)
+{
+ if (*stt) {
+ libcxgbi_log_debug(1 << CXGBI_DBG_ISCSI,
+ "de-register transport 0x%p, %s, stt 0x%p.\n",
+ itp, itp->name, *stt);
+ *stt = NULL;
+ iscsi_unregister_transport(itp);
+ }
+}
+EXPORT_SYMBOL_GPL(cxgbi_iscsi_cleanup);
+
+static int __init libcxgbi_init_module(void)
+{
+ sw_tag_idx_bits = (__ilog2_u32(ISCSI_ITT_MASK)) + 1;
+ sw_tag_age_bits = (__ilog2_u32(ISCSI_AGE_MASK)) + 1;
+
+ libcxgbi_log_info("tag itt 0x%x, %u bits, age 0x%x, %u bits\n",
+ ISCSI_ITT_MASK, sw_tag_idx_bits,
+ ISCSI_AGE_MASK, sw_tag_age_bits);
+
+ ddp_setup_host_page_size();
+ return 0;
+}
+
+static void __exit libcxgbi_exit_module(void)
+{
+ cxgbi_device_unregister_all(0xFF);
+ return;
+}
+
+module_init(libcxgbi_init_module);
+module_exit(libcxgbi_exit_module);
diff --git a/drivers/scsi/cxgbi/libcxgbi.h b/drivers/scsi/cxgbi/libcxgbi.h
new file mode 100644
index 0000000..f8a9590
--- /dev/null
+++ b/drivers/scsi/cxgbi/libcxgbi.h
@@ -0,0 +1,754 @@
+/*
+ * libcxgbi.h: Chelsio common library for T3/T4 iSCSI driver.
+ *
+ * Copyright (c) 2010 Chelsio Communications, Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation.
+ *
+ * Written by: Karen Xie (kxie@chelsio.com)
+ * Written by: Rakesh Ranjan (rranjan@chelsio.com)
+ */
+
+#ifndef __LIBCXGBI_H__
+#define __LIBCXGBI_H__
+
+#include <linux/errno.h>
+#include <linux/types.h>
+#include <linux/debugfs.h>
+#include <linux/list.h>
+#include <linux/netdevice.h>
+#include <linux/if_vlan.h>
+#include <linux/scatterlist.h>
+#include <linux/skbuff.h>
+#include <linux/vmalloc.h>
+#include <scsi/scsi_device.h>
+#include <scsi/libiscsi_tcp.h>
+
+enum cxgbi_dbg_flag {
+ CXGBI_DBG_ISCSI,
+ CXGBI_DBG_DDP,
+ CXGBI_DBG_TOE,
+ CXGBI_DBG_SOCK,
+
+ CXGBI_DBG_PDU_TX,
+ CXGBI_DBG_PDU_RX,
+ CXGBI_DBG_DEV,
+};
+
+#define cxgbi_dbg_print(level, fmt, args...) \
+ do { \
+ if (dbg_level & (level)) \
+ printk(KERN_INFO "cxgbi: %s - " fmt, \
+ __func__ , ## args); \
+ } while (0)
+#define cxgbi_dbg_error(fmt, args...) \
+ printk(KERN_ERR "cxgbi: ERR! %s, " fmt, __func__ , ## args)
+#define cxgbi_dbg_warn(fmt, args...) \
+ printk(KERN_WARNING "cxgbi: WARN! %s, " fmt, __func__ , ## args)
+#define cxgbi_dbg_info(fmt, args...) \
+ printk(KERN_INFO "cxgbi: %s, " fmt, __func__ , ## args)
+
+/* max. connections per adapter */
+#define CXGBI_MAX_CONN 16384
+
+/* always allocate rooms for AHS */
+#define SKB_TX_ISCSI_PDU_HEADER_MAX \
+ (sizeof(struct iscsi_hdr) + ISCSI_MAX_AHS_SIZE)
+
+#define ISCSI_PDU_NONPAYLOAD_LEN 312 /* bhs(48) + ahs(256) + digest(8)*/
+
+/*
+ * align pdu size to multiple of 512 for better performance
+ */
+#define cxgbi_align_pdu_size(n) do { n = (n) & (~511); } while (0)
+
+#define ULP2_MODE_ISCSI 2
+
+#define ULP2_MAX_PKT_SIZE 16224
+#define ULP2_MAX_PDU_PAYLOAD \
+ (ULP2_MAX_PKT_SIZE - ISCSI_PDU_NONPAYLOAD_LEN)
+
+/*
+ * For iscsi connections HW may inserts digest bytes into the pdu. Those digest
+ * bytes are not sent by the host but are part of the TCP payload and therefore
+ * consume TCP sequence space.
+ */
+static const unsigned int ulp2_extra_len[] = { 0, 4, 4, 8 };
+static inline unsigned int cxgbi_ulp_extra_len(int submode)
+{
+ return ulp2_extra_len[submode & 3];
+}
+
+/*
+ * struct pagepod_hdr, pagepod - pagepod format
+ */
+
+#define CPL_RX_DDP_STATUS_DDP_SHIFT 16 /* ddp'able */
+#define CPL_RX_DDP_STATUS_PAD_SHIFT 19 /* pad error */
+#define CPL_RX_DDP_STATUS_HCRC_SHIFT 20 /* hcrc error */
+#define CPL_RX_DDP_STATUS_DCRC_SHIFT 21 /* dcrc error */
+
+struct cxgbi_pagepod_hdr {
+ u32 vld_tid;
+ u32 pgsz_tag_clr;
+ u32 max_offset;
+ u32 page_offset;
+ u64 rsvd;
+};
+
+#define PPOD_PAGES_MAX 4
+struct cxgbi_pagepod {
+ struct cxgbi_pagepod_hdr hdr;
+ u64 addr[PPOD_PAGES_MAX + 1];
+};
+
+struct cxgbi_tag_format {
+ unsigned char sw_bits;
+ unsigned char rsvd_bits;
+ unsigned char rsvd_shift;
+ unsigned char filler[1];
+ u32 rsvd_mask;
+};
+
+struct cxgbi_gather_list {
+ unsigned int tag;
+ unsigned int length;
+ unsigned int offset;
+ unsigned int nelem;
+ struct page **pages;
+ dma_addr_t phys_addr[0];
+};
+
+struct cxgbi_ddp_info {
+ struct kref refcnt;
+ struct cxgbi_device *cdev;
+ struct pci_dev *pdev;
+ unsigned int max_txsz;
+ unsigned int max_rxsz;
+ unsigned int llimit;
+ unsigned int ulimit;
+ unsigned int nppods;
+ unsigned int idx_last;
+ unsigned char idx_bits;
+ unsigned char filler[3];
+ unsigned int idx_mask;
+ unsigned int rsvd_tag_mask;
+ spinlock_t map_lock;
+ struct cxgbi_gather_list **gl_map;
+ struct sk_buff **gl_skb;
+};
+
+#define DDP_PGIDX_MAX 4
+#define DDP_THRESHOLD 2048
+
+#define PPOD_PAGES_SHIFT 2 /* 4 pages per pod */
+
+#define PPOD_SIZE sizeof(struct cxgbi_pagepod) /* 64 */
+#define PPOD_SIZE_SHIFT 6
+
+#define ULPMEM_DSGL_MAX_NPPODS 16 /* 1024/PPOD_SIZE */
+#define ULPMEM_IDATA_MAX_NPPODS 4 /* 256/PPOD_SIZE */
+#define PCIE_MEMWIN_MAX_NPPODS 16 /* 1024/PPOD_SIZE */
+
+#define PPOD_COLOR_SHIFT 0
+#define PPOD_COLOR(x) ((x) << PPOD_COLOR_SHIFT)
+
+#define PPOD_IDX_SHIFT 6
+#define PPOD_IDX_MAX_SIZE 24
+
+#define PPOD_TID_SHIFT 0
+#define PPOD_TID(x) ((x) << PPOD_TID_SHIFT)
+
+#define PPOD_TAG_SHIFT 6
+#define PPOD_TAG(x) ((x) << PPOD_TAG_SHIFT)
+
+#define PPOD_VALID_SHIFT 24
+#define PPOD_VALID(x) ((x) << PPOD_VALID_SHIFT)
+#define PPOD_VALID_FLAG PPOD_VALID(1U)
+
+#define W_TCB_ULP_TYPE 0
+#define TCB_ULP_TYPE_SHIFT 0
+#define TCB_ULP_TYPE_MASK 0xfULL
+#define TCB_ULP_TYPE(x) ((x) << TCB_ULP_TYPE_SHIFT)
+
+#define W_TCB_ULP_RAW 0
+#define TCB_ULP_RAW_SHIFT 4
+#define TCB_ULP_RAW_MASK 0xffULL
+#define TCB_ULP_RAW(x) ((x) << TCB_ULP_RAW_SHIFT)
+
+/*
+ * sge_opaque_hdr -
+ * Opaque version of structure the SGE stores at skb->head of TX_DATA packets
+ * and for which we must reserve space.
+ */
+struct sge_opaque_hdr {
+ void *dev;
+ dma_addr_t addr[MAX_SKB_FRAGS + 1];
+};
+
+struct cxgbi_sock {
+ struct cxgbi_device *cdev;
+
+ int tid;
+ int atid;
+ unsigned long flags;
+ unsigned int mtu;
+ unsigned short rss_qid;
+ unsigned short txq_idx;
+ unsigned short advmss;
+ unsigned int tx_chan;
+ unsigned int rx_chan;
+ unsigned int mss_idx;
+ unsigned int smac_idx;
+ unsigned char port_id;
+ int wr_max_cred;
+ int wr_cred;
+ int wr_una_cred;
+ unsigned char hcrc_len;
+ unsigned char dcrc_len;
+
+ void *l2t;
+ struct sk_buff *wr_pending_head;
+ struct sk_buff *wr_pending_tail;
+ struct sk_buff *cpl_close;
+ struct sk_buff *cpl_abort_req;
+ struct sk_buff *cpl_abort_rpl;
+ struct sk_buff *skb_ulp_lhdr;
+ spinlock_t lock;
+ struct kref refcnt;
+ unsigned int state;
+ struct sockaddr_in saddr;
+ struct sockaddr_in daddr;
+ struct dst_entry *dst;
+ struct sk_buff_head receive_queue;
+ struct sk_buff_head write_queue;
+ struct timer_list retry_timer;
+ int err;
+ rwlock_t callback_lock;
+ void *user_data;
+
+ u32 rcv_nxt;
+ u32 copied_seq;
+ u32 rcv_wup;
+ u32 snd_nxt;
+ u32 snd_una;
+ u32 write_seq;
+};
+
+/*
+ * connection states
+ */
+enum cxgbi_sock_states{
+ CTP_CLOSED,
+ CTP_CONNECTING,
+ CTP_ACTIVE_OPEN,
+ CTP_ESTABLISHED,
+ CTP_ACTIVE_CLOSE,
+ CTP_PASSIVE_CLOSE,
+ CTP_CLOSE_WAIT_1,
+ CTP_CLOSE_WAIT_2,
+ CTP_ABORTING,
+};
+
+/*
+ * Connection flags -- many to track some close related events.
+ */
+enum cxgbi_sock_flags {
+ CTPF_ABORT_RPL_RCVD, /*received one ABORT_RPL_RSS message */
+ CTPF_ABORT_REQ_RCVD, /*received one ABORT_REQ_RSS message */
+ CTPF_ABORT_RPL_PENDING, /* expecting an abort reply */
+ CTPF_TX_DATA_SENT, /* already sent a TX_DATA WR */
+ CTPF_ACTIVE_CLOSE_NEEDED,/* need to be closed */
+ CTPF_HAS_ATID, /* reserved atid */
+ CTPF_HAS_TID, /* reserved hw tid */
+ CTPF_OFFLOAD_DOWN, /* offload function off */
+};
+
+struct cxgbi_skb_rx_cb {
+ __u32 ddigest;
+ __u32 pdulen;
+};
+
+struct cxgbi_skb_tx_cb {
+ void *l2t;
+ struct sk_buff *wr_next;
+};
+
+enum cxgbi_skcb_flags {
+ SKCBF_TX_NEED_HDR, /* packet needs a header */
+ SKCBF_RX_COALESCED, /* received whole pdu */
+ SKCBF_RX_HDR, /* recieved pdu header */
+ SKCBF_RX_DATA, /* recieved pdu payload */
+ SKCBF_RX_STATUS, /* recieved ddp status */
+ SKCBF_RX_DATA_DDPD, /* pdu payload ddp'd */
+ SKCBF_RX_HCRC_ERR, /* header digest error */
+ SKCBF_RX_DCRC_ERR, /* data digest error */
+ SKCBF_RX_PAD_ERR, /* padding byte error */
+};
+
+struct cxgbi_skb_cb {
+ unsigned char ulp_mode;
+ unsigned long flags;
+ unsigned int seq;
+ union {
+ struct cxgbi_skb_rx_cb rx;
+ struct cxgbi_skb_tx_cb tx;
+ };
+};
+
+#define CXGBI_SKB_CB(skb) ((struct cxgbi_skb_cb *)&((skb)->cb[0]))
+#define cxgbi_skcb_flags(skb) (CXGBI_SKB_CB(skb)->flags)
+#define cxgbi_skcb_ulp_mode(skb) (CXGBI_SKB_CB(skb)->ulp_mode)
+#define cxgbi_skcb_tcp_seq(skb) (CXGBI_SKB_CB(skb)->seq)
+#define cxgbi_skcb_rx_ddigest(skb) (CXGBI_SKB_CB(skb)->rx.ddigest)
+#define cxgbi_skcb_rx_pdulen(skb) (CXGBI_SKB_CB(skb)->rx.pdulen)
+#define cxgbi_skcb_tx_wr_next(skb) (CXGBI_SKB_CB(skb)->tx.wr_next)
+
+static inline void cxgbi_skcb_set_flag(struct sk_buff *skb,
+ enum cxgbi_skcb_flags flag)
+{
+ __set_bit(flag, &(cxgbi_skcb_flags(skb)));
+}
+
+static inline void cxgbi_skcb_clear_flag(struct sk_buff *skb,
+ enum cxgbi_skcb_flags flag)
+{
+ __clear_bit(flag, &(cxgbi_skcb_flags(skb)));
+}
+
+static inline int cxgbi_skcb_test_flag(struct sk_buff *skb,
+ enum cxgbi_skcb_flags flag)
+{
+ return test_bit(flag, &(cxgbi_skcb_flags(skb)));
+}
+
+static inline void cxgbi_sock_set_flag(struct cxgbi_sock *csk,
+ enum cxgbi_sock_flags flag)
+{
+ __set_bit(flag, &csk->flags);
+ cxgbi_dbg_print(1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx, bit %d.\n",
+ csk, csk->state, csk->flags, flag);
+}
+
+static inline void cxgbi_sock_clear_flag(struct cxgbi_sock *csk,
+ enum cxgbi_sock_flags flag)
+{
+ __clear_bit(flag, &csk->flags);
+ cxgbi_dbg_print(1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx, bit %d.\n",
+ csk, csk->state, csk->flags, flag);
+}
+
+static inline int cxgbi_sock_flag(struct cxgbi_sock *csk,
+ enum cxgbi_sock_flags flag)
+{
+ if (csk == NULL)
+ return 0;
+ return test_bit(flag, &csk->flags);
+}
+
+static inline void cxgbi_sock_set_state(struct cxgbi_sock *csk, int state)
+{
+ cxgbi_dbg_print(1 << CXGBI_DBG_SOCK,
+ "csk 0x%p,%u,0x%lx, state -> %u.\n",
+ csk, csk->state, csk->flags, state);
+ csk->state = state;
+}
+
+static inline void cxgbi_sock_free(struct kref *kref)
+{
+ struct cxgbi_sock *csk = container_of(kref,
+ struct cxgbi_sock,
+ refcnt);
+ if (csk) {
+ cxgbi_dbg_print(1 << CXGBI_DBG_SOCK,
+ "free csk 0x%p, state %u, flags 0x%lx\n",
+ csk, csk->state, csk->flags);
+ kfree(csk);
+ }
+}
+
+static inline void __cxgbi_sock_put(const char *fn, struct cxgbi_sock *csk)
+{
+ cxgbi_dbg_print(1 << CXGBI_DBG_SOCK,
+ "%s, put csk 0x%p, ref %u-1.\n",
+ fn, csk, atomic_read(&csk->refcnt.refcount));
+ kref_put(&csk->refcnt, cxgbi_sock_free);
+}
+#define cxgbi_sock_put(csk) __cxgbi_sock_put(__func__, csk)
+
+static inline void __cxgbi_sock_get(const char *fn, struct cxgbi_sock *csk)
+{
+ cxgbi_dbg_print(1 << CXGBI_DBG_SOCK,
+ "%s, get csk 0x%p, ref %u+1.\n",
+ fn, csk, atomic_read(&csk->refcnt.refcount));
+ kref_get(&csk->refcnt);
+}
+#define cxgbi_sock_get(csk) __cxgbi_sock_get(__func__, csk)
+
+static inline int cxgbi_sock_is_closing(struct cxgbi_sock *csk)
+{
+ return csk->state >= CTP_ACTIVE_CLOSE;
+}
+
+static inline int cxgbi_sock_is_established(struct cxgbi_sock *csk)
+{
+ return csk->state == CTP_ESTABLISHED;
+}
+
+static inline void cxgbi_sock_purge_write_queue(struct cxgbi_sock *csk)
+{
+ struct sk_buff *skb;
+
+ while ((skb = __skb_dequeue(&csk->write_queue)))
+ __kfree_skb(skb);
+}
+
+static inline unsigned int cxgbi_sock_compute_wscale(unsigned int win)
+{
+ unsigned int wscale = 0;
+
+ while (wscale < 14 && (65535 << wscale) < win)
+ wscale++;
+ return wscale;
+}
+
+static inline struct sk_buff *alloc_cpl(int cpl_len, int dlen, gfp_t gfp)
+{
+ int wrlen = roundup(cpl_len, 16);
+ struct sk_buff *skb = alloc_skb(wrlen + dlen, gfp);
+
+ if (skb) {
+ __skb_put(skb, wrlen);
+ memset(skb->head, 0, wrlen + dlen);
+ } else
+ cxgbi_dbg_info("alloc cpl skb %u+%u, OOM.\n", cpl_len, dlen);
+ return skb;
+}
+
+
+/*
+ * The number of WRs needed for an skb depends on the number of fragments
+ * in the skb and whether it has any payload in its main body. This maps the
+ * length of the gather list represented by an skb into the # of necessary WRs.
+ * The extra two fragments are for iscsi bhs and payload padding.
+ */
+#define SKB_WR_LIST_SIZE (MAX_SKB_FRAGS + 2)
+
+static inline void cxgbi_sock_reset_wr_list(struct cxgbi_sock *csk)
+{
+ csk->wr_pending_head = csk->wr_pending_tail = NULL;
+}
+
+static inline void cxgbi_sock_enqueue_wr(struct cxgbi_sock *csk,
+ struct sk_buff *skb)
+{
+ cxgbi_skcb_tx_wr_next(skb) = NULL;
+ /*
+ * We want to take an extra reference since both us and the driver
+ * need to free the packet before it's really freed. We know there's
+ * just one user currently so we use atomic_set rather than skb_get
+ * to avoid the atomic op.
+ */
+ atomic_set(&skb->users, 2);
+
+ if (!csk->wr_pending_head)
+ csk->wr_pending_head = skb;
+ else
+ cxgbi_skcb_tx_wr_next(csk->wr_pending_tail) = skb;
+ csk->wr_pending_tail = skb;
+}
+
+static inline int cxgbi_sock_count_pending_wrs(const struct cxgbi_sock *csk)
+{
+ int n = 0;
+ const struct sk_buff *skb = csk->wr_pending_head;
+
+ while (skb) {
+ n += skb->csum;
+ skb = cxgbi_skcb_tx_wr_next(skb);
+ }
+ return n;
+}
+
+static inline struct sk_buff *cxgbi_sock_peek_wr(const struct cxgbi_sock *csk)
+{
+ return csk->wr_pending_head;
+}
+
+static inline struct sk_buff *cxgbi_sock_dequeue_wr(struct cxgbi_sock *csk)
+{
+ struct sk_buff *skb = csk->wr_pending_head;
+
+ if (likely(skb)) {
+ csk->wr_pending_head = cxgbi_skcb_tx_wr_next(skb);
+ cxgbi_skcb_tx_wr_next(skb) = NULL;
+ }
+ return skb;
+}
+
+void cxgbi_sock_check_wr_invariants(const struct cxgbi_sock *);
+void cxgbi_sock_purge_wr_queue(struct cxgbi_sock *);
+void cxgbi_sock_skb_entail(struct cxgbi_sock *, struct sk_buff *);
+void cxgbi_sock_fail_act_open(struct cxgbi_sock *, int);
+void cxgbi_sock_act_open_req_arp_failure(void *, struct sk_buff *);
+void cxgbi_sock_closed(struct cxgbi_sock *);
+void cxgbi_sock_established(struct cxgbi_sock *, unsigned int, unsigned int);
+void cxgbi_sock_rcv_abort_rpl(struct cxgbi_sock *);
+void cxgbi_sock_rcv_peer_close(struct cxgbi_sock *);
+void cxgbi_sock_rcv_close_conn_rpl(struct cxgbi_sock *, u32);
+void cxgbi_sock_rcv_wr_ack(struct cxgbi_sock *, unsigned int, unsigned int,
+ int);
+unsigned int cxgbi_sock_select_mss(struct cxgbi_sock *, unsigned int);
+void cxgbi_sock_free_cpl_skbs(struct cxgbi_sock *);
+
+struct cxgbi_hba {
+ struct net_device *ndev;
+ struct Scsi_Host *shost;
+ struct cxgbi_device *cdev;
+ __be32 ipv4addr;
+ unsigned char port_id;
+};
+
+struct cxgbi_ports_map {
+ unsigned int max_connect;
+ unsigned int used;
+ unsigned short sport_base;
+ spinlock_t lock;
+ unsigned int next;
+ struct cxgbi_sock **port_csk;
+};
+
+#define CXGBI_FLAG_DEV_T3 0x1
+#define CXGBI_FLAG_DEV_T4 0x2
+#define CXGBI_FLAG_ADAPTER_RESET 0x4
+struct cxgbi_device {
+ struct list_head list_head;
+ unsigned int flags;
+ struct net_device **ports;
+ void *lldev;
+ struct cxgbi_hba **hbas;
+ const unsigned short *mtus;
+ unsigned char nmtus;
+ unsigned char nports;
+ struct pci_dev *pdev;
+ struct dentry *debugfs_root;
+ struct iscsi_transport *itp;
+
+ unsigned int pfvf;
+ unsigned int snd_win;
+ unsigned int rcv_win;
+ unsigned int rx_credit_thres;
+ unsigned int skb_tx_rsvd;
+ unsigned int skb_rx_extra; /* for msg coalesced mode */
+ unsigned int tx_max_size;
+ unsigned int rx_max_size;
+ struct cxgbi_ports_map pmap;
+ struct cxgbi_tag_format tag_format;
+ struct cxgbi_ddp_info *ddp;
+
+ void (*dev_ddp_cleanup)(struct cxgbi_device *);
+ void (*csk_ddp_free_gl_skb)(struct cxgbi_ddp_info *, int, int);
+ int (*csk_ddp_alloc_gl_skb)(struct cxgbi_ddp_info *, int, int, gfp_t);
+ int (*csk_ddp_set)(struct cxgbi_sock *, struct cxgbi_pagepod_hdr *,
+ unsigned int, unsigned int,
+ struct cxgbi_gather_list *);
+ void (*csk_ddp_clear)(struct cxgbi_hba *,
+ unsigned int, unsigned int, unsigned int);
+ int (*csk_ddp_setup_digest)(struct cxgbi_sock *,
+ unsigned int, int, int, int);
+ int (*csk_ddp_setup_pgidx)(struct cxgbi_sock *,
+ unsigned int, int, bool);
+
+ void (*csk_release_offload_resources)(struct cxgbi_sock *);
+ int (*csk_rx_pdu_ready)(struct cxgbi_sock *, struct sk_buff *);
+ u32 (*csk_send_rx_credits)(struct cxgbi_sock *, u32);
+ int (*csk_push_tx_frames)(struct cxgbi_sock *, int);
+ void (*csk_send_abort_req)(struct cxgbi_sock *);
+ void (*csk_send_close_req)(struct cxgbi_sock *);
+ int (*csk_alloc_cpls)(struct cxgbi_sock *);
+ int (*csk_init_act_open)(struct cxgbi_sock *);
+
+ void *dd_data;
+};
+#define cxgbi_cdev_priv(cdev) ((cdev)->dd_data)
+
+struct cxgbi_conn {
+ struct cxgbi_endpoint *cep;
+ struct iscsi_conn *iconn;
+ struct cxgbi_hba *chba;
+ u32 task_idx_bits;
+};
+
+struct cxgbi_endpoint {
+ struct cxgbi_conn *cconn;
+ struct cxgbi_hba *chba;
+ struct cxgbi_sock *csk;
+};
+
+#define MAX_PDU_FRAGS ((ULP2_MAX_PDU_PAYLOAD + 512 - 1) / 512)
+struct cxgbi_task_data {
+ unsigned short nr_frags;
+ skb_frag_t frags[MAX_PDU_FRAGS];
+ struct sk_buff *skb;
+ unsigned int offset;
+ unsigned int count;
+ unsigned int sgoffset;
+};
+
+static inline int cxgbi_is_ddp_tag(struct cxgbi_tag_format *tformat, u32 tag)
+{
+ return !(tag & (1 << (tformat->rsvd_bits + tformat->rsvd_shift - 1)));
+}
+
+static inline int cxgbi_sw_tag_usable(struct cxgbi_tag_format *tformat,
+ u32 sw_tag)
+{
+ sw_tag >>= (32 - tformat->rsvd_bits);
+ return !sw_tag;
+}
+
+static inline u32 cxgbi_set_non_ddp_tag(struct cxgbi_tag_format *tformat,
+ u32 sw_tag)
+{
+ unsigned char shift = tformat->rsvd_bits + tformat->rsvd_shift - 1;
+ u32 mask = (1 << shift) - 1;
+
+ if (sw_tag && (sw_tag & ~mask)) {
+ u32 v1 = sw_tag & ((1 << shift) - 1);
+ u32 v2 = (sw_tag >> (shift - 1)) << shift;
+
+ return v2 | v1 | 1 << shift;
+ }
+
+ return sw_tag | 1 << shift;
+}
+
+static inline u32 cxgbi_ddp_tag_base(struct cxgbi_tag_format *tformat,
+ u32 sw_tag)
+{
+ u32 mask = (1 << tformat->rsvd_shift) - 1;
+
+ if (sw_tag && (sw_tag & ~mask)) {
+ u32 v1 = sw_tag & mask;
+ u32 v2 = sw_tag >> tformat->rsvd_shift;
+
+ v2 <<= tformat->rsvd_bits + tformat->rsvd_shift;
+
+ return v2 | v1;
+ }
+
+ return sw_tag;
+}
+
+static inline u32 cxgbi_tag_rsvd_bits(struct cxgbi_tag_format *tformat,
+ u32 tag)
+{
+ if (cxgbi_is_ddp_tag(tformat, tag))
+ return (tag >> tformat->rsvd_shift) & tformat->rsvd_mask;
+
+ return 0;
+}
+
+static inline u32 cxgbi_tag_nonrsvd_bits(struct cxgbi_tag_format *tformat,
+ u32 tag)
+{
+ unsigned char shift = tformat->rsvd_bits + tformat->rsvd_shift - 1;
+ u32 v1, v2;
+
+ if (cxgbi_is_ddp_tag(tformat, tag)) {
+ v1 = tag & ((1 << tformat->rsvd_shift) - 1);
+ v2 = (tag >> (shift + 1)) << tformat->rsvd_shift;
+ } else {
+ u32 mask = (1 << shift) - 1;
+ tag &= ~(1 << shift);
+ v1 = tag & mask;
+ v2 = (tag >> 1) & ~mask;
+ }
+ return v1 | v2;
+}
+
+static inline void *cxgbi_alloc_big_mem(unsigned int size,
+ gfp_t gfp)
+{
+ void *p = kmalloc(size, gfp);
+ if (!p)
+ p = vmalloc(size);
+ if (p)
+ memset(p, 0, size);
+ return p;
+}
+
+static inline void cxgbi_free_big_mem(void *addr)
+{
+ if (is_vmalloc_addr(addr))
+ vfree(addr);
+ else
+ kfree(addr);
+}
+
+static inline void cxgbi_set_iscsi_ipv4(struct cxgbi_hba *chba, __be32 ipaddr)
+{
+ chba->ipv4addr = ipaddr;
+}
+
+static inline __be32 cxgbi_get_iscsi_ipv4(struct cxgbi_hba *chba)
+{
+ return chba->ipv4addr;
+}
+
+struct cxgbi_device *cxgbi_device_register(unsigned int, unsigned int);
+void cxgbi_device_unregister(struct cxgbi_device *);
+void cxgbi_device_unregister_all(unsigned int flag);
+struct cxgbi_device *cxgbi_device_find_by_lldev(void *);
+int cxgbi_hbas_add(struct cxgbi_device *, unsigned int, unsigned int,
+ struct scsi_host_template *,
+ struct scsi_transport_template *);
+void cxgbi_hbas_remove(struct cxgbi_device *);
+
+int cxgbi_device_portmap_create(struct cxgbi_device *cdev, unsigned int base,
+ unsigned int max_conn);
+void cxgbi_device_portmap_cleanup(struct cxgbi_device *cdev);
+
+void cxgbi_conn_tx_open(struct cxgbi_sock *);
+void cxgbi_conn_pdu_ready(struct cxgbi_sock *);
+int cxgbi_conn_alloc_pdu(struct iscsi_task *, u8);
+int cxgbi_conn_init_pdu(struct iscsi_task *, unsigned int , unsigned int);
+int cxgbi_conn_xmit_pdu(struct iscsi_task *);
+
+void cxgbi_cleanup_task(struct iscsi_task *task);
+
+void cxgbi_get_conn_stats(struct iscsi_cls_conn *, struct iscsi_stats *);
+int cxgbi_set_conn_param(struct iscsi_cls_conn *,
+ enum iscsi_param, char *, int);
+int cxgbi_get_conn_param(struct iscsi_cls_conn *, enum iscsi_param, char *);
+struct iscsi_cls_conn *cxgbi_create_conn(struct iscsi_cls_session *, u32);
+int cxgbi_bind_conn(struct iscsi_cls_session *,
+ struct iscsi_cls_conn *, u64, int);
+void cxgbi_destroy_session(struct iscsi_cls_session *);
+struct iscsi_cls_session *cxgbi_create_session(struct iscsi_endpoint *,
+ u16, u16, u32);
+int cxgbi_set_host_param(struct Scsi_Host *,
+ enum iscsi_host_param, char *, int);
+int cxgbi_get_host_param(struct Scsi_Host *, enum iscsi_host_param, char *);
+struct iscsi_endpoint *cxgbi_ep_connect(struct Scsi_Host *,
+ struct sockaddr *, int);
+int cxgbi_ep_poll(struct iscsi_endpoint *, int);
+void cxgbi_ep_disconnect(struct iscsi_endpoint *);
+
+int cxgbi_iscsi_init(struct iscsi_transport *,
+ struct scsi_transport_template **);
+void cxgbi_iscsi_cleanup(struct iscsi_transport *,
+ struct scsi_transport_template **);
+void cxgbi_parse_pdu_itt(struct iscsi_conn *, itt_t, int *, int *);
+int cxgbi_ddp_init(struct cxgbi_device *, unsigned int, unsigned int,
+ unsigned int, unsigned int);
+int cxgbi_ddp_cleanup(struct cxgbi_device *);
+void cxgbi_ddp_page_size_factor(int *);
+void cxgbi_ddp_ppod_clear(struct cxgbi_pagepod *);
+void cxgbi_ddp_ppod_set(struct cxgbi_pagepod *, struct cxgbi_pagepod_hdr *,
+ struct cxgbi_gather_list *, unsigned int);
+#endif /*__LIBCXGBI_H__*/
^ permalink raw reply related
* [PATCH v2 0/3] cxgbi iscsi driver
From: kxie @ 2010-08-13 15:19 UTC (permalink / raw)
To: netdev, linux-scsi, open-iscsi
Cc: rranjan, kxie, James.Bottomley, michaelc, davem
[PATCH v2 0/3] cxgbi iscsi driver
From: Karen Xie <kxie@chelsio.com>
Re-submission, missed netdev list last time.
This patchset includes:
1. added libcxgbi as the common library for both cxgb4i and cxgb3i driver
2. added cxgb4i iscsi initiator v5.
3. changed cxgb3i to use libcxgbi.
The version 2 includes the following changes:
- Made dbg_level static to avoid compiling errors when both cxgb3i and cxgb4i
are built into the kernel.
- Deleted cxgbi_fill_iscsi_transport() and cxgbi_fill_scsi_host_template().
The iscsi_transport and scsi_host_template are initialized in the drivers
instead of being filled in by the library at runtime..
- Only register with iscsi class transport once per driver type.
Thanks.
Karen
^ permalink raw reply
* Re: [xfrm_user] BUG: sleeping function called from invalid context
From: Herbert Xu @ 2010-08-13 15:36 UTC (permalink / raw)
To: Luca Tettamanti; +Cc: netdev, linux-kernel
In-Reply-To: <20100812125027.GA12982@nb-core2.darkstar.lan>
Luca Tettamanti <kronos.it@gmail.com> wrote:
>
> xfrm_user_policy takes read_lock(&xfrm_km_lock) before calling
> xfrm_compile_policy (via km->compile_policy), which in turn calls
> xfrm_policy_alloc with GFP_KERNEL.
Thanks for discovering this bug, it only took 8 years :)
xfrm: Use GFP_ATOMIC in xfrm_compile_policy
As xfrm_compile_policy runs within a read_lock, we cannot use
GFP_KERNEL for memory allocations.
Reported-by: Luca Tettamanti <kronos.it@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
diff --git a/net/xfrm/xfrm_user.c b/net/xfrm/xfrm_user.c
index ba59983..b14ed4b 100644
--- a/net/xfrm/xfrm_user.c
+++ b/net/xfrm/xfrm_user.c
@@ -2504,7 +2504,7 @@ static struct xfrm_policy *xfrm_compile_policy(struct sock *sk, int opt,
if (p->dir > XFRM_POLICY_OUT)
return NULL;
- xp = xfrm_policy_alloc(net, GFP_KERNEL);
+ xp = xfrm_policy_alloc(net, GFP_ATOMIC);
if (xp == NULL) {
*dir = -ENOBUFS;
return NULL;
Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply related
* Re: [PATCH v2 0/3] cxgbi iscsi driver
From: Joe Perches @ 2010-08-13 15:49 UTC (permalink / raw)
To: kxie
Cc: netdev, linux-scsi, open-iscsi, rranjan, James.Bottomley,
michaelc, davem
In-Reply-To: <201008131519.o7DFJsaH002618@localhost.localdomain>
On Fri, 2010-08-13 at 08:19 -0700, kxie@chelsio.com wrote:
> [PATCH v2 0/3] cxgbi iscsi driver
> From: Karen Xie <kxie@chelsio.com>
> Re-submission, missed netdev list last time.
Hi Karen.
I presume this is not to be applied as you've
already received some feedback on it and that
you are sending it to netdev for any additional
comments.
^ permalink raw reply
* Re: i386 allmodconfig, current mainline
From: Randy Dunlap @ 2010-08-13 16:23 UTC (permalink / raw)
To: Geert Uytterhoeven
Cc: Stephen Rothwell, Andrew Morton, Greg KH, David Woodhouse,
Mike Frysinger, Jesse Barnes, Kevin Curtis, Phillip Lougher,
Faisal Latif, Chien Tung, Dan Williams, Samuel Ortiz,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
netdev-u79uwXL29TY76Z2rM5mHXA,
linux-wireless-u79uwXL29TY76Z2rM5mHXA, x86-DgEjT+Ai2ygdnm+yROfE0A,
linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <AANLkTimtZ7H=rT68V6UGC2GBky8z9VJ5m6-pwFHfm7yw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
On Tue, 10 Aug 2010 18:53:20 +0200 Geert Uytterhoeven wrote:
> On Tue, Aug 10, 2010 at 02:15, Stephen Rothwell <sfr-3FnU+UHB4dNDw9hX6IcOSA@public.gmane.org> wrote:
> > On Mon, 9 Aug 2010 16:43:46 -0700 Andrew Morton <akpm@linux-foundation.org> wrote:
> >>
> >> Guys. What's goin' on out there?
> >
> > I guess we are all so up to date that noone does 32 bit builds any
> > more ... Also noone is bothering to look at the build logs:
> >
> > linus tree: http://kisskb.ellerman.id.au/kisskb/branch/3/
> > linux-next: http://kisskb.ellerman.id.au/kisskb/branch/9/
>
> Yeah, we have build failures migrating from -next to -linus :-(
>
> BTW, still no one has commented on http://lkml.org/lkml/2010/7/13/378?
OK.
The diff (regression) script output looks useful to me.
The log summary is still quite (too) verbose, or maybe that's partly due
to the number of warnings & errors. :(
Maybe someone could look at http://www.xenotime.net/linux/scripts/buildsummary.pl also.
Here is today's linux-next *i386* build summary, short mode (totals only):
> buildsummary.pl -t build*.out | more
build-r5372.out: totals: error/warning files: 27, errors: 3, warnings: 44, Section mismatches: 36
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
build-r5373.out: totals: error/warning files: 10, errors: 3, warnings: 14, Section mismatches: 36
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
build-r5374.out: totals: error/warning files: 29, errors: 0, warnings: 45, Section mismatches: 38
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
build-r5375.out: totals: error/warning files: 18, errors: 3, warnings: 36, Section mismatches: 38
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
build-r5377.out: totals: error/warning files: 21, errors: 0, warnings: 43, Section mismatches: 67
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
build-r5380.out: totals: error/warning files: 16, errors: 4, warnings: 22, Section mismatches: 81
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
build-r5381.out: totals: error/warning files: 17, errors: 3, warnings: 75, Section mismatches: 88
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
build-r5383.out: totals: error/warning files: 11, errors: 0, warnings: 32, Section mismatches: 130
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
build-r5384.out: totals: error/warning files: 23, errors: 3, warnings: 47, Section mismatches: 150
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
build-r5385.out: totals: error/warning files: 6, errors: 12, warnings: 9, Section mismatches: 160
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
build-r5386.out: totals: error/warning files: 10, errors: 1, warnings: 53, Section mismatches: 163
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
build-r5387.out: totals: error/warning files: 18, errors: 0, warnings: 45, Section mismatches: 164
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
build-r5388.out: totals: error/warning files: 7, errors: 1, warnings: 8, Section mismatches: 164
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
build-r5389.out: totals: error/warning files: 43, errors: 0, warnings: 80, Section mismatches: 167
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
build-r5390.out: totals: error/warning files: 12, errors: 0, warnings: 18, Section mismatches: 167
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
build-r5391.out: totals: error/warning files: 46, errors: 0, warnings: 77, Section mismatches: 170
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
build-r5392.out: totals: error/warning files: 15, errors: 12, warnings: 31, Section mismatches: 184
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
build-r5393.out: totals: error/warning files: 15, errors: 0, warnings: 23, Section mismatches: 187
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
build-r5394.out: totals: error/warning files: 76, errors: 0, warnings: 141, Section mismatches: 190
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
build-r5395.out: totals: error/warning files: 11, errors: 0, warnings: 18, Section mismatches: 192
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
build-r5396.out: totals: error/warning files: 22, errors: 0, warnings: 28, Section mismatches: 193
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***
--
To unsubscribe from this list: send the line "unsubscribe linux-wireless" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] netpoll: use non-BH variant of RCU
From: Paul E. McKenney @ 2010-08-13 16:29 UTC (permalink / raw)
To: Herbert Xu; +Cc: David Miller, linville, netdev
In-Reply-To: <20100813143904.GA27261@gondor.apana.org.au>
On Fri, Aug 13, 2010 at 10:39:04AM -0400, Herbert Xu wrote:
> On Thu, Aug 12, 2010 at 08:42:13AM -0700, Paul E. McKenney wrote:
> >
> > +/**
> > + * rcu_read_lock_bh_irqsoff() - mark the beginning of an RCU-bh critical section
> > + *
> > + * This is equivalent of rcu_read_lock_bh(), but to be used where the
> > + * caller either is in an irq handler or has irqs disabled. Note that
> > + * this function assumes that PREEMPT_RT kernels run irq handlers at
> > + * higher priority than softirq handlers!
> > + */
> > +static inline void rcu_read_lock_bh_irqsoff(void)
> > +{
> > + rcu_read_unlock_bh_irqsoff_check();
> > + __acquire(RCU_BH);
> > + rcu_read_acquire_bh();
> > +}
>
> Thanks for the patch Paul!
>
> But this doesn't really solve the problem for netif_rx. The reason
> is that netif_rx can either be called with IRQs on OR off. So we
> need to take the right precautions in the case where IRQs are
> enabled along with BH.
Interesting...
Is it possible that IRQs are off at rcu_read_lock_bh_irqsoff() time, but
enabled by the time we get to rcu_read_unlock_bh_irqsoff()? I hope not,
but have to ask. If I am guaranteed of the same state in both cases,
I can do something like the following:
static inline void rcu_read_lock_bh_irqsoff(void)
{
if (!in_irq() && !irqs_disabled())
local_bh_disable();
__acquire(RCU_BH);
rcu_read_acquire_bh();
}
static inline void rcu_read_unlock_bh_irqsoff(void)
{
rcu_read_release_bh();
__release(RCU_BH);
if (!in_irq() && !irqs_disabled())
local_bh_enable();
}
If the state can change in the RCU-bh read-side critical section, then
I would have to record the state in the task structure or some such.
But all in all, mightn't it be easier to remove the checks from
_local_bh_enable(), and then just use rcu_read_lock_bh()? Have those
checks really been that helpful in finding bugs? ;-)
Thanx, Paul
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox