* Re: [PATCH] slub: fix slab_pad_check()
From: Paul E. McKenney @ 2009-09-03 23:17 UTC (permalink / raw)
To: Christoph Lameter
Cc: Eric Dumazet, Pekka Enberg, Zdenek Kabelac, Patrick McHardy,
Robin Holt, Linux Kernel Mailing List, Jesper Dangaard Brouer,
Linux Netdev List, Netfilter Developers
In-Reply-To: <alpine.DEB.1.10.0909031731540.24199@V090114053VZO-1>
On Thu, Sep 03, 2009 at 05:44:54PM -0500, Christoph Lameter wrote:
> On Thu, 3 Sep 2009, Eric Dumazet wrote:
>
> > It seems very smart, but needs review of all callers to make sure no slabs
> > are waiting for final freeing in call_rcu queue on some cpu.
>
> Yes. Again this is the first time we encounter a situation where a
> DESTROY_BY_RCU slab has to be destroyed. So the review is quite short.
>
> > I suspect most of them will then have to use rcu_barrier() before calling
> > kmem_cache_destroy(), so why not factorizing code in one place ?
>
> There are different forms of RCU which require different forms of
> barriers. Its best to leave that up to the user. Again the user must make
> sure that no objects are in use before a slab is destroyed. For
> SLAB_DESTROY_BY_RCU this means that there are no potential outstanding
> reads on the structure. You may need an rcu_barrier() to accomplish that.
>
> Slight variations in the use of RCU could require different method. Better
> reduce the entanglement of slabs to RCU to a mininum possible.
If it were the user of the slab who was invoking some variant of
call_rcu(), then I would agree with you.
However, call_rcu() is instead being invoked by the slab itself in the
case of SLAB_DESTROY_BY_RCU, so that there is no variation in usage.
Requiring that the user call rcu_barrier() is asking for subtle bugs.
Therefore, the best approach is to have kmem_cache_destroy() handle
the RCU cleanup, given that this cleanup is for actions taken by
kmem_cache_free(), not by the user.
Thanx, Paul
^ permalink raw reply
* Re: [NET] Add proc file to display the state of all qdiscs.
From: David Miller @ 2009-09-03 23:22 UTC (permalink / raw)
To: hawk; +Cc: cl, eric.dumazet, jarkao2, kaber, netdev
In-Reply-To: <Pine.LNX.4.64.0909031617060.28935@ask.diku.dk>
From: Jesper Dangaard Brouer <hawk@diku.dk>
Date: Thu, 3 Sep 2009 16:19:44 +0200 (CEST)
>
> On Wed, 2 Sep 2009, Christoph Lameter wrote:
>> On Wed, 2 Sep 2009, Eric Dumazet wrote:
>>
>>> Same name "eth0" is displayed, that might confuse parsers...
>>>
>>> What naming convention should we choose for multiqueue devices ?
>>
>> eth0/tx<number> ?
>
> Remember that we already have a naming convention in /proc/interrupts
>
> eth0-tx-<number>
>
> Lets not introduce too many new once ;-)
Yes, please :-)
^ permalink raw reply
* Re: skb header allocation
From: Stephen Hemminger @ 2009-09-03 23:23 UTC (permalink / raw)
To: Frank Blaschka; +Cc: Chris Ross, netdev
In-Reply-To: <4A94CFB3.4020706@linux.vnet.ibm.com>
On Wed, 26 Aug 2009 08:01:23 +0200
Frank Blaschka <blaschka@linux.vnet.ibm.com> wrote:
> Is it save to change dev->hard_header_len to a value other than ETH_HLEN for
> an ethernet device?
>
> I rememeber we run in trouble with some kernel components (netfilter?, I'm
> not sure can't remember the details ...) when we did this is the past.
>
You can add extra data but need a minimum with ETH_HLEN of space above
packet when receiving.
--
^ permalink raw reply
* Re: [NET] Add proc file to display the state of all qdiscs.
From: David Miller @ 2009-09-03 23:28 UTC (permalink / raw)
To: hawk; +Cc: kaber, cl, eric.dumazet, jarkao2, netdev
In-Reply-To: <Pine.LNX.4.64.0909031911400.3490@ask.diku.dk>
From: Jesper Dangaard Brouer <hawk@diku.dk>
Date: Thu, 3 Sep 2009 19:30:00 +0200 (CEST)
> I especially like the possibility to access each qdisc seperately.
> Does it then support having seperate qdisc per TX queue? (I'm toying
> with the idea of transmitting our multicast traffic into/via a
> seperate TX hardware queue, and making a special qdisc for IPTV
> MPEG2-TS shaping)
Indeed it sounds interesting.
But I want to warn everyone against adding support for anything
that adds a way to do operations "on every device qdisc" at the
kernel level.
I tried to do that and the error unwinding complexity is unacceptable.
We can make the tools support things like this. Keep the kernel
simple and allow it to only operate on one queue at a time for
a given individual config request.
^ permalink raw reply
* Re: [NET] Add proc file to display the state of all qdiscs.
From: David Miller @ 2009-09-03 23:31 UTC (permalink / raw)
To: kaber; +Cc: hawk, cl, eric.dumazet, jarkao2, netdev
In-Reply-To: <4AA00351.4090601@trash.net>
From: Patrick McHardy <kaber@trash.net>
Date: Thu, 03 Sep 2009 19:56:33 +0200
> It would probably also be possible to use TC classifiers for queue
> selection.
You mean just like Intel's multiqueue scheduler and multiqueue
classifiers which we already have in the tree :-)
^ permalink raw reply
* [net-next PATCH 1/3] igb: add support for set_rx_mode netdevice operation
From: Jeff Kirsher @ 2009-09-04 0:48 UTC (permalink / raw)
To: davem; +Cc: netdev, gospo, Alexander Duyck, Jeff Kirsher
From: Alexander Duyck <alexander.h.duyck@intel.com>
This patch adds support for the set_rx_mode netdevice operation so that igb
can better support multiple unicast addresses.
Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
---
drivers/net/igb/igb_main.c | 73 +++++++++++++++++++++++++++++---------------
1 files changed, 48 insertions(+), 25 deletions(-)
diff --git a/drivers/net/igb/igb_main.c b/drivers/net/igb/igb_main.c
index 7a054d9..95089a8 100644
--- a/drivers/net/igb/igb_main.c
+++ b/drivers/net/igb/igb_main.c
@@ -94,7 +94,7 @@ static void igb_clean_all_tx_rings(struct igb_adapter *);
static void igb_clean_all_rx_rings(struct igb_adapter *);
static void igb_clean_tx_ring(struct igb_ring *);
static void igb_clean_rx_ring(struct igb_ring *);
-static void igb_set_multi(struct net_device *);
+static void igb_set_rx_mode(struct net_device *);
static void igb_update_phy_info(unsigned long);
static void igb_watchdog(unsigned long);
static void igb_watchdog_task(struct work_struct *);
@@ -928,7 +928,7 @@ static void igb_configure(struct igb_adapter *adapter)
int i;
igb_get_hw_control(adapter);
- igb_set_multi(netdev);
+ igb_set_rx_mode(netdev);
igb_restore_vlan(adapter);
@@ -1169,7 +1169,8 @@ static const struct net_device_ops igb_netdev_ops = {
.ndo_stop = igb_close,
.ndo_start_xmit = igb_xmit_frame_adv,
.ndo_get_stats = igb_get_stats,
- .ndo_set_multicast_list = igb_set_multi,
+ .ndo_set_rx_mode = igb_set_rx_mode,
+ .ndo_set_multicast_list = igb_set_rx_mode,
.ndo_set_mac_address = igb_set_mac,
.ndo_change_mtu = igb_change_mtu,
.ndo_do_ioctl = igb_ioctl,
@@ -2519,48 +2520,70 @@ static int igb_set_mac(struct net_device *netdev, void *p)
memcpy(netdev->dev_addr, addr->sa_data, netdev->addr_len);
memcpy(hw->mac.addr, addr->sa_data, netdev->addr_len);
- hw->mac.ops.rar_set(hw, hw->mac.addr, 0);
-
+ igb_rar_set(hw, hw->mac.addr, 0);
igb_set_rah_pool(hw, adapter->vfs_allocated_count, 0);
return 0;
}
/**
- * igb_set_multi - Multicast and Promiscuous mode set
+ * igb_set_rx_mode - Secondary Unicast, Multicast and Promiscuous mode set
* @netdev: network interface device structure
*
- * The set_multi entry point is called whenever the multicast address
- * list or the network interface flags are updated. This routine is
- * responsible for configuring the hardware for proper multicast,
+ * The set_rx_mode entry point is called whenever the unicast or multicast
+ * address lists or the network interface flags are updated. This routine is
+ * responsible for configuring the hardware for proper unicast, multicast,
* promiscuous mode, and all-multi behavior.
**/
-static void igb_set_multi(struct net_device *netdev)
+static void igb_set_rx_mode(struct net_device *netdev)
{
struct igb_adapter *adapter = netdev_priv(netdev);
struct e1000_hw *hw = &adapter->hw;
- struct dev_mc_list *mc_ptr;
+ unsigned int rar_entries = hw->mac.rar_entry_count -
+ (adapter->vfs_allocated_count + 1);
+ struct dev_mc_list *mc_ptr = netdev->mc_list;
u8 *mta_list = NULL;
u32 rctl;
int i;
/* Check for Promiscuous and All Multicast modes */
-
rctl = rd32(E1000_RCTL);
if (netdev->flags & IFF_PROMISC) {
rctl |= (E1000_RCTL_UPE | E1000_RCTL_MPE);
rctl &= ~E1000_RCTL_VFE;
} else {
- if (netdev->flags & IFF_ALLMULTI) {
+ if (netdev->flags & IFF_ALLMULTI)
rctl |= E1000_RCTL_MPE;
+ else
+ rctl &= ~E1000_RCTL_MPE;
+
+ if (netdev->uc.count > rar_entries)
+ rctl |= E1000_RCTL_UPE;
+ else
rctl &= ~E1000_RCTL_UPE;
- } else
- rctl &= ~(E1000_RCTL_UPE | E1000_RCTL_MPE);
rctl |= E1000_RCTL_VFE;
}
wr32(E1000_RCTL, rctl);
+ if (netdev->uc.count && rar_entries) {
+ struct netdev_hw_addr *ha;
+ list_for_each_entry(ha, &netdev->uc.list, list) {
+ if (!rar_entries)
+ break;
+ igb_rar_set(hw, ha->addr, rar_entries);
+ igb_set_rah_pool(hw, adapter->vfs_allocated_count,
+ rar_entries);
+ rar_entries--;
+ }
+ }
+ /* write the addresses in reverse order to avoid write combining */
+ for (; rar_entries > 0 ; rar_entries--) {
+ wr32(E1000_RAH(rar_entries), 0);
+ wr32(E1000_RAL(rar_entries), 0);
+ }
+ wrfl();
+
if (!netdev->mc_count) {
/* nothing to program, so clear mc list */
igb_update_mc_addr_list(hw, NULL, 0);
@@ -2576,8 +2599,6 @@ static void igb_set_multi(struct net_device *netdev)
}
/* The shared function expects a packed array of only addresses. */
- mc_ptr = netdev->mc_list;
-
for (i = 0; i < netdev->mc_count; i++) {
if (!mc_ptr)
break;
@@ -3938,7 +3959,7 @@ static int igb_set_vf_multicasts(struct igb_adapter *adapter,
vf_data->vf_mc_hashes[i] = hash_list[i];;
/* Flush and reset the mta with the new values */
- igb_set_multi(adapter->netdev);
+ igb_set_rx_mode(adapter->netdev);
return 0;
}
@@ -4072,13 +4093,14 @@ static inline void igb_vf_reset_event(struct igb_adapter *adapter, u32 vf)
adapter->vf_data[vf].num_vf_mc_hashes = 0;
/* Flush and reset the mta with the new values */
- igb_set_multi(adapter->netdev);
+ igb_set_rx_mode(adapter->netdev);
}
static inline void igb_vf_reset_msg(struct igb_adapter *adapter, u32 vf)
{
struct e1000_hw *hw = &adapter->hw;
unsigned char *vf_mac = adapter->vf_data[vf].vf_mac_addresses;
+ int rar_entry = hw->mac.rar_entry_count - (vf + 1);
u32 reg, msgbuf[3];
u8 *addr = (u8 *)(&msgbuf[1]);
@@ -4086,8 +4108,8 @@ static inline void igb_vf_reset_msg(struct igb_adapter *adapter, u32 vf)
igb_vf_reset_event(adapter, vf);
/* set vf mac address */
- igb_rar_set(hw, vf_mac, vf + 1);
- igb_set_rah_pool(hw, vf, vf + 1);
+ igb_rar_set(hw, vf_mac, rar_entry);
+ igb_set_rah_pool(hw, vf, rar_entry);
/* enable transmit and receive for vf */
reg = rd32(E1000_VFTE);
@@ -5228,7 +5250,7 @@ static int __igb_shutdown(struct pci_dev *pdev, bool *enable_wake)
if (wufc) {
igb_setup_rctl(adapter);
- igb_set_multi(netdev);
+ igb_set_rx_mode(netdev);
/* turn on all-multi mode if wake on multicast is enabled */
if (wufc & E1000_WUFC_MC) {
@@ -5482,12 +5504,13 @@ static int igb_set_vf_mac(struct igb_adapter *adapter,
int vf, unsigned char *mac_addr)
{
struct e1000_hw *hw = &adapter->hw;
- int rar_entry = vf + 1; /* VF MAC addresses start at entry 1 */
-
- igb_rar_set(hw, mac_addr, rar_entry);
+ /* VF MAC addresses start at end of receive addresses and moves
+ * torwards the first, as a result a collision should not be possible */
+ int rar_entry = hw->mac.rar_entry_count - (vf + 1);
memcpy(adapter->vf_data[vf].vf_mac_addresses, mac_addr, ETH_ALEN);
+ igb_rar_set(hw, mac_addr, rar_entry);
igb_set_rah_pool(hw, vf, rar_entry);
return 0;
^ permalink raw reply related
* [net-next PATCH 2/3] igb: only disable/enable interrupt bits for igb physical function
From: Jeff Kirsher @ 2009-09-04 0:49 UTC (permalink / raw)
To: davem; +Cc: netdev, gospo, Alexander Duyck, Jeff Kirsher
In-Reply-To: <20090904004828.14537.72187.stgit@localhost.localdomain>
From: Alexander Duyck <alexander.h.duyck@intel.com>
The igb_irq_disable/enable calls were causing virtual functions associated
with the igb physical function to have their interrupts disabled. In order
to prevent this from occuring we should only clear/set the bits related to
the physical function.
Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
---
drivers/net/igb/igb_main.c | 14 +++++++++-----
1 files changed, 9 insertions(+), 5 deletions(-)
diff --git a/drivers/net/igb/igb_main.c b/drivers/net/igb/igb_main.c
index 95089a8..1d03fcb 100644
--- a/drivers/net/igb/igb_main.c
+++ b/drivers/net/igb/igb_main.c
@@ -821,9 +821,11 @@ static void igb_irq_disable(struct igb_adapter *adapter)
struct e1000_hw *hw = &adapter->hw;
if (adapter->msix_entries) {
- wr32(E1000_EIAM, 0);
- wr32(E1000_EIMC, ~0);
- wr32(E1000_EIAC, 0);
+ u32 regval = rd32(E1000_EIAM);
+ wr32(E1000_EIAM, regval & ~adapter->eims_enable_mask);
+ wr32(E1000_EIMC, adapter->eims_enable_mask);
+ regval = rd32(E1000_EIAC);
+ wr32(E1000_EIAC, regval & ~adapter->eims_enable_mask);
}
wr32(E1000_IAM, 0);
@@ -841,8 +843,10 @@ static void igb_irq_enable(struct igb_adapter *adapter)
struct e1000_hw *hw = &adapter->hw;
if (adapter->msix_entries) {
- wr32(E1000_EIAC, adapter->eims_enable_mask);
- wr32(E1000_EIAM, adapter->eims_enable_mask);
+ u32 regval = rd32(E1000_EIAC);
+ wr32(E1000_EIAC, regval | adapter->eims_enable_mask);
+ regval = rd32(E1000_EIAM);
+ wr32(E1000_EIAM, regval | adapter->eims_enable_mask);
wr32(E1000_EIMS, adapter->eims_enable_mask);
if (adapter->vfs_allocated_count)
wr32(E1000_MBVFIMR, 0xFF);
^ permalink raw reply related
* [net-next PATCH 3/3] igb: set vf rlpml wasn't taking vlan tag into account
From: Jeff Kirsher @ 2009-09-04 0:49 UTC (permalink / raw)
To: davem; +Cc: netdev, gospo, Alexander Duyck, Jeff Kirsher
In-Reply-To: <20090904004828.14537.72187.stgit@localhost.localdomain>
From: Alexander Duyck <alexander.h.duyck@intel.com>
This patch updates things so that vlan tags are taken into account when
setting the receive large packet maximum length. This allows the VF driver
to correctly receive full sized frames when vlans are enabled.
Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
---
drivers/net/igb/igb.h | 1 +
drivers/net/igb/igb_main.c | 39 +++++++++++++++++++++++++++++++++++++++
2 files changed, 40 insertions(+), 0 deletions(-)
diff --git a/drivers/net/igb/igb.h b/drivers/net/igb/igb.h
index b2c98de..7126fea 100644
--- a/drivers/net/igb/igb.h
+++ b/drivers/net/igb/igb.h
@@ -70,6 +70,7 @@ struct vf_data_storage {
unsigned char vf_mac_addresses[ETH_ALEN];
u16 vf_mc_hashes[IGB_MAX_VF_MC_ENTRIES];
u16 num_vf_mc_hashes;
+ u16 vlans_enabled;
bool clear_to_send;
};
diff --git a/drivers/net/igb/igb_main.c b/drivers/net/igb/igb_main.c
index 1d03fcb..90b0b1b 100644
--- a/drivers/net/igb/igb_main.c
+++ b/drivers/net/igb/igb_main.c
@@ -154,6 +154,12 @@ static inline int igb_set_vf_rlpml(struct igb_adapter *adapter, int size,
struct e1000_hw *hw = &adapter->hw;
u32 vmolr;
+ /* if it isn't the PF check to see if VFs are enabled and
+ * increase the size to support vlan tags */
+ if (vfn < adapter->vfs_allocated_count &&
+ adapter->vf_data[vfn].vlans_enabled)
+ size += VLAN_TAG_SIZE;
+
vmolr = rd32(E1000_VMOLR(vfn));
vmolr &= ~E1000_VMOLR_RLPML_MASK;
vmolr |= size | E1000_VMOLR_LPE;
@@ -4006,6 +4012,8 @@ static void igb_clear_vf_vfta(struct igb_adapter *adapter, u32 vf)
wr32(E1000_VLVF(i), reg);
}
+
+ adapter->vf_data[vf].vlans_enabled = 0;
}
static s32 igb_vlvf_set(struct igb_adapter *adapter, u32 vid, bool add, u32 vf)
@@ -4054,6 +4062,22 @@ static s32 igb_vlvf_set(struct igb_adapter *adapter, u32 vid, bool add, u32 vf)
reg |= vid;
wr32(E1000_VLVF(i), reg);
+
+ /* do not modify RLPML for PF devices */
+ if (vf >= adapter->vfs_allocated_count)
+ return 0;
+
+ if (!adapter->vf_data[vf].vlans_enabled) {
+ u32 size;
+ reg = rd32(E1000_VMOLR(vf));
+ size = reg & E1000_VMOLR_RLPML_MASK;
+ size += 4;
+ reg &= ~E1000_VMOLR_RLPML_MASK;
+ reg |= size;
+ wr32(E1000_VMOLR(vf), reg);
+ }
+ adapter->vf_data[vf].vlans_enabled++;
+
return 0;
}
} else {
@@ -4066,6 +4090,21 @@ static s32 igb_vlvf_set(struct igb_adapter *adapter, u32 vid, bool add, u32 vf)
igb_vfta_set(hw, vid, false);
}
wr32(E1000_VLVF(i), reg);
+
+ /* do not modify RLPML for PF devices */
+ if (vf >= adapter->vfs_allocated_count)
+ return 0;
+
+ adapter->vf_data[vf].vlans_enabled--;
+ if (!adapter->vf_data[vf].vlans_enabled) {
+ u32 size;
+ reg = rd32(E1000_VMOLR(vf));
+ size = reg & E1000_VMOLR_RLPML_MASK;
+ size -= 4;
+ reg &= ~E1000_VMOLR_RLPML_MASK;
+ reg |= size;
+ wr32(E1000_VMOLR(vf), reg);
+ }
return 0;
}
}
^ permalink raw reply related
* [net-next PATCH 1/3] ixgbe: Add support for multiple Tx queues for FCoE in 82599
From: Jeff Kirsher @ 2009-09-04 0:55 UTC (permalink / raw)
To: davem; +Cc: netdev, gospo, Yi Zou, Jeff Kirsher
From: Yi Zou <yi.zou@intel.com>
This patch adds support for multiple transmit queues to the Fiber Channel
over Ethernet (FCoE) feature found in 82599. Currently, FCoE has multiple
Rx queues available, along with a redirection table, that helps distribute
the I/O load across multiple CPUs based on the FC exchange ID. To make
this the most effective, we need to provide the same layout of transmit
queues to match receive.
Particularly, when Data Center Bridging (DCB) is enabled, the designated
traffic class for FCoE can have dedicated queues for just FCoE traffic,
while not affecting any other type of traffic flow.
Signed-off-by: Yi Zou <yi.zou@intel.com>
Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
---
drivers/net/ixgbe/ixgbe_dcb_nl.c | 37 +++++++++++++++++++++++++++------
drivers/net/ixgbe/ixgbe_main.c | 43 ++++++++++++++++++++++++++++++--------
2 files changed, 65 insertions(+), 15 deletions(-)
diff --git a/drivers/net/ixgbe/ixgbe_dcb_nl.c b/drivers/net/ixgbe/ixgbe_dcb_nl.c
index 47d9dde..a6bc1ef 100644
--- a/drivers/net/ixgbe/ixgbe_dcb_nl.c
+++ b/drivers/net/ixgbe/ixgbe_dcb_nl.c
@@ -36,6 +36,7 @@
#define BIT_PFC 0x02
#define BIT_PG_RX 0x04
#define BIT_PG_TX 0x08
+#define BIT_APP_UPCHG 0x10
#define BIT_RESETLINK 0x40
#define BIT_LINKSPEED 0x80
@@ -348,8 +349,14 @@ static u8 ixgbe_dcbnl_set_all(struct net_device *netdev)
while (test_and_set_bit(__IXGBE_RESETTING, &adapter->state))
msleep(1);
- if (netif_running(netdev))
- ixgbe_down(adapter);
+ if (adapter->dcb_set_bitmap & BIT_APP_UPCHG) {
+ if (netif_running(netdev))
+ netdev->netdev_ops->ndo_stop(netdev);
+ ixgbe_clear_interrupt_scheme(adapter);
+ } else {
+ if (netif_running(netdev))
+ ixgbe_down(adapter);
+ }
}
ret = ixgbe_copy_dcb_cfg(&adapter->temp_dcb_cfg, &adapter->dcb_cfg,
@@ -373,8 +380,14 @@ static u8 ixgbe_dcbnl_set_all(struct net_device *netdev)
}
if (adapter->dcb_set_bitmap & BIT_RESETLINK) {
- if (netif_running(netdev))
- ixgbe_up(adapter);
+ if (adapter->dcb_set_bitmap & BIT_APP_UPCHG) {
+ ixgbe_init_interrupt_scheme(adapter);
+ if (netif_running(netdev))
+ netdev->netdev_ops->ndo_open(netdev);
+ } else {
+ if (netif_running(netdev))
+ ixgbe_up(adapter);
+ }
ret = DCB_HW_CHG_RST;
} else if (adapter->dcb_set_bitmap & BIT_PFC) {
if (adapter->hw.mac.type == ixgbe_mac_82598EB)
@@ -526,8 +539,20 @@ static u8 ixgbe_dcbnl_setapp(struct net_device *netdev,
switch (idtype) {
case DCB_APP_IDTYPE_ETHTYPE:
#ifdef IXGBE_FCOE
- if (id == ETH_P_FCOE)
- rval = ixgbe_fcoe_setapp(netdev_priv(netdev), up);
+ if (id == ETH_P_FCOE) {
+ u8 tc;
+ struct ixgbe_adapter *adapter;
+
+ adapter = netdev_priv(netdev);
+ tc = adapter->fcoe.tc;
+ rval = ixgbe_fcoe_setapp(adapter, up);
+ if ((!rval) && (tc != adapter->fcoe.tc) &&
+ (adapter->flags & IXGBE_FLAG_DCB_ENABLED) &&
+ (adapter->flags & IXGBE_FLAG_FCOE_ENABLED)) {
+ adapter->dcb_set_bitmap |= BIT_APP_UPCHG;
+ adapter->dcb_set_bitmap |= BIT_RESETLINK;
+ }
+ }
#endif
break;
case DCB_APP_IDTYPE_PORTNUM:
diff --git a/drivers/net/ixgbe/ixgbe_main.c b/drivers/net/ixgbe/ixgbe_main.c
index 4042d87..724754a 100644
--- a/drivers/net/ixgbe/ixgbe_main.c
+++ b/drivers/net/ixgbe/ixgbe_main.c
@@ -3113,14 +3113,16 @@ static inline bool ixgbe_set_fcoe_queues(struct ixgbe_adapter *adapter)
f->indices = min((int)num_online_cpus(), f->indices);
if (adapter->flags & IXGBE_FLAG_FCOE_ENABLED) {
+ adapter->num_rx_queues = 1;
+ adapter->num_tx_queues = 1;
#ifdef CONFIG_IXGBE_DCB
if (adapter->flags & IXGBE_FLAG_DCB_ENABLED) {
- DPRINTK(PROBE, INFO, "FCOE enabled with DCB \n");
+ DPRINTK(PROBE, INFO, "FCoE enabled with DCB \n");
ixgbe_set_dcb_queues(adapter);
}
#endif
if (adapter->flags & IXGBE_FLAG_RSS_ENABLED) {
- DPRINTK(PROBE, INFO, "FCOE enabled with RSS \n");
+ DPRINTK(PROBE, INFO, "FCoE enabled with RSS \n");
if ((adapter->flags & IXGBE_FLAG_FDIR_HASH_CAPABLE) ||
(adapter->flags & IXGBE_FLAG_FDIR_PERFECT_CAPABLE))
ixgbe_set_fdir_queues(adapter);
@@ -3130,8 +3132,7 @@ static inline bool ixgbe_set_fcoe_queues(struct ixgbe_adapter *adapter)
/* adding FCoE rx rings to the end */
f->mask = adapter->num_rx_queues;
adapter->num_rx_queues += f->indices;
- if (adapter->num_tx_queues == 0)
- adapter->num_tx_queues = f->indices;
+ adapter->num_tx_queues += f->indices;
ret = true;
}
@@ -3371,15 +3372,36 @@ static bool inline ixgbe_cache_ring_fdir(struct ixgbe_adapter *adapter)
*/
static inline bool ixgbe_cache_ring_fcoe(struct ixgbe_adapter *adapter)
{
- int i, fcoe_i = 0;
+ int i, fcoe_rx_i = 0, fcoe_tx_i = 0;
bool ret = false;
struct ixgbe_ring_feature *f = &adapter->ring_feature[RING_F_FCOE];
if (adapter->flags & IXGBE_FLAG_FCOE_ENABLED) {
#ifdef CONFIG_IXGBE_DCB
if (adapter->flags & IXGBE_FLAG_DCB_ENABLED) {
+ struct ixgbe_fcoe *fcoe = &adapter->fcoe;
+
ixgbe_cache_ring_dcb(adapter);
- fcoe_i = adapter->rx_ring[0].reg_idx + 1;
+ /* find out queues in TC for FCoE */
+ fcoe_rx_i = adapter->rx_ring[fcoe->tc].reg_idx + 1;
+ fcoe_tx_i = adapter->tx_ring[fcoe->tc].reg_idx + 1;
+ /*
+ * In 82599, the number of Tx queues for each traffic
+ * class for both 8-TC and 4-TC modes are:
+ * TCs : TC0 TC1 TC2 TC3 TC4 TC5 TC6 TC7
+ * 8 TCs: 32 32 16 16 8 8 8 8
+ * 4 TCs: 64 64 32 32
+ * We have max 8 queues for FCoE, where 8 the is
+ * FCoE redirection table size. If TC for FCoE is
+ * less than or equal to TC3, we have enough queues
+ * to add max of 8 queues for FCoE, so we start FCoE
+ * tx descriptor from the next one, i.e., reg_idx + 1.
+ * If TC for FCoE is above TC3, implying 8 TC mode,
+ * and we need 8 for FCoE, we have to take all queues
+ * in that traffic class for FCoE.
+ */
+ if ((f->indices == IXGBE_FCRETA_SIZE) && (fcoe->tc > 3))
+ fcoe_tx_i--;
}
#endif /* CONFIG_IXGBE_DCB */
if (adapter->flags & IXGBE_FLAG_RSS_ENABLED) {
@@ -3389,10 +3411,13 @@ static inline bool ixgbe_cache_ring_fcoe(struct ixgbe_adapter *adapter)
else
ixgbe_cache_ring_rss(adapter);
- fcoe_i = f->mask;
+ fcoe_rx_i = f->mask;
+ fcoe_tx_i = f->mask;
+ }
+ for (i = 0; i < f->indices; i++, fcoe_rx_i++, fcoe_tx_i++) {
+ adapter->rx_ring[f->mask + i].reg_idx = fcoe_rx_i;
+ adapter->tx_ring[f->mask + i].reg_idx = fcoe_tx_i;
}
- for (i = 0; i < f->indices; i++, fcoe_i++)
- adapter->rx_ring[f->mask + i].reg_idx = fcoe_i;
ret = true;
}
return ret;
^ permalink raw reply related
* [net-next PATCH 2/3] ixgbe: Distribute transmission of FCoE traffic in 82599
From: Jeff Kirsher @ 2009-09-04 0:56 UTC (permalink / raw)
To: davem; +Cc: netdev, gospo, Yi Zou, Jeff Kirsher
In-Reply-To: <20090904005549.14869.66878.stgit@localhost.localdomain>
From: Yi Zou <yi.zou@intel.com>
This adds a simple selection of a FCoE tx queue based on the current cpu id to
distribute transmission of FCoE traffic evenly among multiple FCoE transmit
queues.
Signed-off-by: Yi Zou <yi.zou@intel.com>
Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
---
drivers/net/ixgbe/ixgbe_main.c | 10 ++++++++--
1 files changed, 8 insertions(+), 2 deletions(-)
diff --git a/drivers/net/ixgbe/ixgbe_main.c b/drivers/net/ixgbe/ixgbe_main.c
index 724754a..45bf8b9 100644
--- a/drivers/net/ixgbe/ixgbe_main.c
+++ b/drivers/net/ixgbe/ixgbe_main.c
@@ -5168,9 +5168,15 @@ static netdev_tx_t ixgbe_xmit_frame(struct sk_buff *skb,
tx_ring = &adapter->tx_ring[r_idx];
if ((adapter->flags & IXGBE_FLAG_FCOE_ENABLED) &&
- (skb->protocol == htons(ETH_P_FCOE)))
+ (skb->protocol == htons(ETH_P_FCOE))) {
tx_flags |= IXGBE_TX_FLAGS_FCOE;
-
+#ifdef IXGBE_FCOE
+ r_idx = smp_processor_id();
+ r_idx &= (adapter->ring_feature[RING_F_FCOE].indices - 1);
+ r_idx += adapter->ring_feature[RING_F_FCOE].mask;
+ tx_ring = &adapter->tx_ring[r_idx];
+#endif
+ }
/* four things can cause us to need a context descriptor */
if (skb_is_gso(skb) ||
(skb->ip_summed == CHECKSUM_PARTIAL) ||
^ permalink raw reply related
* [net-next PATCH 3/3] ixgbe: Add support for using FCoE DDP in 82599 as FCoE targets
From: Jeff Kirsher @ 2009-09-04 0:56 UTC (permalink / raw)
To: davem; +Cc: netdev, gospo, Yi Zou, Jeff Kirsher
In-Reply-To: <20090904005549.14869.66878.stgit@localhost.localdomain>
From: Yi Zou <yi.zou@intel.com>
The FCoE DDP in 82599 can be used for both FCoE initiator as well as FCoE
target, depending on the indication of the exchange being the responder or
originator in the F_CTL (frame control) field in the encapsulated Fiber
Channel frame header (T10 Spec., FC-FS). For the initiator, OX_ID is used
for FCoE DDP, where for the target RX_ID is used for FCoE DDP.
Signed-off-by: Yi Zou <yi.zou@intel.com>
Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
---
drivers/net/ixgbe/ixgbe_fcoe.c | 8 +++++++-
1 files changed, 7 insertions(+), 1 deletions(-)
diff --git a/drivers/net/ixgbe/ixgbe_fcoe.c b/drivers/net/ixgbe/ixgbe_fcoe.c
index 0607cff..a3c9f99 100644
--- a/drivers/net/ixgbe/ixgbe_fcoe.c
+++ b/drivers/net/ixgbe/ixgbe_fcoe.c
@@ -292,6 +292,7 @@ int ixgbe_fcoe_ddp(struct ixgbe_adapter *adapter,
struct sk_buff *skb)
{
u16 xid;
+ u32 fctl;
u32 sterr, fceofe, fcerr, fcstat;
int rc = -EINVAL;
struct ixgbe_fcoe *fcoe;
@@ -312,7 +313,12 @@ int ixgbe_fcoe_ddp(struct ixgbe_adapter *adapter,
skb_set_transport_header(skb, skb_network_offset(skb) +
sizeof(struct fcoe_hdr));
fh = (struct fc_frame_header *)skb_transport_header(skb);
- xid = be16_to_cpu(fh->fh_ox_id);
+ fctl = ntoh24(fh->fh_f_ctl);
+ if (fctl & FC_FC_EX_CTX)
+ xid = be16_to_cpu(fh->fh_ox_id);
+ else
+ xid = be16_to_cpu(fh->fh_rx_id);
+
if (xid >= IXGBE_FCOE_DDP_MAX)
goto ddp_out;
^ permalink raw reply related
* Re: [NET] Add proc file to display the state of all qdiscs.
From: Patrick McHardy @ 2009-09-04 1:36 UTC (permalink / raw)
To: David Miller; +Cc: hawk, cl, eric.dumazet, jarkao2, netdev
In-Reply-To: <20090903.163101.154177782.davem@davemloft.net>
David Miller wrote:
> From: Patrick McHardy <kaber@trash.net>
> Date: Thu, 03 Sep 2009 19:56:33 +0200
>
>
>> It would probably also be possible to use TC classifiers for queue
>> selection.
>>
>
> You mean just like Intel's multiqueue scheduler and multiqueue
> classifiers which we already have in the tree :-
No, actually integrated in the current infrastructure :) The patches
add a virtual root qdisc, so we still have a single root from userspace's
POV. This qdisc is only used for sch_api purposes and doesn't necessarily
match the real root qdiscs. When a "mq" qdisc is attached (done by
default for multiqueue devices), the queues are exposed as child
classes of this qdisc and can be individually grafted with different
qdiscs.
Since the virtual root is global per device, what we could do is allow
to attach classifiers and execute them in dev_pick_tx() to select the
queue. But currently this would require global locking, so this would
have to be fixed previously. Probably not an easy thing to do.
^ permalink raw reply
* Re: [NET] Add proc file to display the state of all qdiscs.
From: Patrick McHardy @ 2009-09-04 1:43 UTC (permalink / raw)
To: David Miller; +Cc: hawk, cl, eric.dumazet, jarkao2, netdev
In-Reply-To: <4AA06F3B.2050500@trash.net>
[-- Attachment #1: Type: text/plain, Size: 908 bytes --]
Patrick McHardy wrote:
> David Miller wrote:
>
>> You mean just like Intel's multiqueue scheduler and multiqueue
>> classifiers which we already have in the tree :-
>>
>
> No, actually integrated in the current infrastructure :) The patches
> add a virtual root qdisc, so we still have a single root from userspace's
> POV. This qdisc is only used for sch_api purposes and doesn't necessarily
> match the real root qdiscs. When a "mq" qdisc is attached (done by
> default for multiqueue devices), the queues are exposed as child
> classes of this qdisc and can be individually grafted with different
> qdiscs.
>
Since its getting late and I won't finish this tonight, here are the
patches I'm currently working on in case someone already wants to have
a look. Its working as intended and should be testable. Patch 1 is good
shape IMO, patch 3 still needs some work. Userspace needs no changes.
[-- Attachment #2: 01.diff --]
[-- Type: text/x-diff, Size: 9550 bytes --]
commit 6ab8d4a8e3df7976fc2db1e1181216a213df89d5
Author: Patrick McHardy <kaber@trash.net>
Date: Fri Sep 4 03:07:39 2009 +0200
net_sched: reintroduce dev->qdisc for use by sch_api
Currently the multiqueue integration with the qdisc API suffers from
a few problems:
- with multiple queues, all root qdiscs use the same handle. This means
they can't be exposed to userspace in a backwards compatible fashion.
- all API operations always refer to queue number 0. Newly created
qdiscs are automatically shared between all queues, its not possible
to address individual queues or restore multiqueue behaviour once a
shared qdisc has been attached.
- Dumps only contain the root qdisc of queue 0, in case of non-shared
qdiscs this means the statistics are incomplete.
This patch reintroduces dev->qdisc, which points to the (single) root qdisc
from userspace's point of view. Currently it either points to the first
(non-shared) default qdisc, or a qdisc shared between all queues. The
following patches will introduce a classful dummy qdisc, which will be used
as root qdisc and contain the per-queue qdiscs as children.
Signed-off-by: Patrick McHardy <kaber@trash.net>
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 121cbad..a44118b 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -832,6 +832,9 @@ struct net_device
/* Number of TX queues currently active in device */
unsigned int real_num_tx_queues;
+ /* root qdisc from userspace point of view */
+ struct Qdisc *qdisc;
+
unsigned long tx_queue_len; /* Max frames per queue allowed */
spinlock_t tx_global_lock;
/*
diff --git a/net/core/rtnetlink.c b/net/core/rtnetlink.c
index 5c1fe53..1aebdc1 100644
--- a/net/core/rtnetlink.c
+++ b/net/core/rtnetlink.c
@@ -606,7 +606,6 @@ static int rtnl_fill_ifinfo(struct sk_buff *skb, struct net_device *dev,
int type, u32 pid, u32 seq, u32 change,
unsigned int flags)
{
- struct netdev_queue *txq;
struct ifinfomsg *ifm;
struct nlmsghdr *nlh;
const struct net_device_stats *stats;
@@ -637,9 +636,8 @@ static int rtnl_fill_ifinfo(struct sk_buff *skb, struct net_device *dev,
if (dev->master)
NLA_PUT_U32(skb, IFLA_MASTER, dev->master->ifindex);
- txq = netdev_get_tx_queue(dev, 0);
- if (txq->qdisc_sleeping)
- NLA_PUT_STRING(skb, IFLA_QDISC, txq->qdisc_sleeping->ops->id);
+ if (dev->qdisc)
+ NLA_PUT_STRING(skb, IFLA_QDISC, dev->qdisc->ops->id);
if (dev->ifalias)
NLA_PUT_STRING(skb, IFLA_IFALIAS, dev->ifalias);
diff --git a/net/sched/cls_api.c b/net/sched/cls_api.c
index 09cdcdf..e857441 100644
--- a/net/sched/cls_api.c
+++ b/net/sched/cls_api.c
@@ -168,8 +168,7 @@ replay:
/* Find qdisc */
if (!parent) {
- struct netdev_queue *dev_queue = netdev_get_tx_queue(dev, 0);
- q = dev_queue->qdisc_sleeping;
+ q = dev->qdisc;
parent = q->handle;
} else {
q = qdisc_lookup(dev, TC_H_MAJ(t->tcm_parent));
@@ -405,7 +404,6 @@ static int tcf_node_dump(struct tcf_proto *tp, unsigned long n,
static int tc_dump_tfilter(struct sk_buff *skb, struct netlink_callback *cb)
{
struct net *net = sock_net(skb->sk);
- struct netdev_queue *dev_queue;
int t;
int s_t;
struct net_device *dev;
@@ -424,9 +422,8 @@ static int tc_dump_tfilter(struct sk_buff *skb, struct netlink_callback *cb)
if ((dev = dev_get_by_index(&init_net, tcm->tcm_ifindex)) == NULL)
return skb->len;
- dev_queue = netdev_get_tx_queue(dev, 0);
if (!tcm->tcm_parent)
- q = dev_queue->qdisc_sleeping;
+ q = dev->qdisc;
else
q = qdisc_lookup(dev, TC_H_MAJ(tcm->tcm_parent));
if (!q)
diff --git a/net/sched/sch_api.c b/net/sched/sch_api.c
index 24d17ce..285ca25 100644
--- a/net/sched/sch_api.c
+++ b/net/sched/sch_api.c
@@ -207,7 +207,7 @@ static struct Qdisc *qdisc_match_from_root(struct Qdisc *root, u32 handle)
static void qdisc_list_add(struct Qdisc *q)
{
if ((q->parent != TC_H_ROOT) && !(q->flags & TCQ_F_INGRESS))
- list_add_tail(&q->list, &qdisc_root_sleeping(q)->list);
+ list_add_tail(&q->list, &qdisc_dev(q)->qdisc->list);
}
void qdisc_list_del(struct Qdisc *q)
@@ -219,17 +219,11 @@ EXPORT_SYMBOL(qdisc_list_del);
struct Qdisc *qdisc_lookup(struct net_device *dev, u32 handle)
{
- unsigned int i;
struct Qdisc *q;
- for (i = 0; i < dev->num_tx_queues; i++) {
- struct netdev_queue *txq = netdev_get_tx_queue(dev, i);
- struct Qdisc *txq_root = txq->qdisc_sleeping;
-
- q = qdisc_match_from_root(txq_root, handle);
- if (q)
- goto out;
- }
+ q = qdisc_match_from_root(dev->qdisc, handle);
+ if (q)
+ goto out;
q = qdisc_match_from_root(dev->rx_queue.qdisc_sleeping, handle);
out:
@@ -720,9 +714,14 @@ static int qdisc_graft(struct net_device *dev, struct Qdisc *parent,
if (new && i > 0)
atomic_inc(&new->refcnt);
- notify_and_destroy(skb, n, classid, old, new);
+ qdisc_destroy(old);
}
+ notify_and_destroy(skb, n, classid, dev->qdisc, new);
+ if (new)
+ atomic_inc(&new->refcnt);
+ dev->qdisc = new ? : &noop_qdisc;
+
if (dev->flags & IFF_UP)
dev_activate(dev);
} else {
@@ -974,9 +973,7 @@ static int tc_get_qdisc(struct sk_buff *skb, struct nlmsghdr *n, void *arg)
q = dev->rx_queue.qdisc_sleeping;
}
} else {
- struct netdev_queue *dev_queue;
- dev_queue = netdev_get_tx_queue(dev, 0);
- q = dev_queue->qdisc_sleeping;
+ q = dev->qdisc;
}
if (!q)
return -ENOENT;
@@ -1044,9 +1041,7 @@ replay:
q = dev->rx_queue.qdisc_sleeping;
}
} else {
- struct netdev_queue *dev_queue;
- dev_queue = netdev_get_tx_queue(dev, 0);
- q = dev_queue->qdisc_sleeping;
+ q = dev->qdisc;
}
/* It may be default qdisc, ignore it */
@@ -1291,8 +1286,7 @@ static int tc_dump_qdisc(struct sk_buff *skb, struct netlink_callback *cb)
s_q_idx = 0;
q_idx = 0;
- dev_queue = netdev_get_tx_queue(dev, 0);
- if (tc_dump_qdisc_root(dev_queue->qdisc_sleeping, skb, cb, &q_idx, s_q_idx) < 0)
+ if (tc_dump_qdisc_root(dev->qdisc, skb, cb, &q_idx, s_q_idx) < 0)
goto done;
dev_queue = &dev->rx_queue;
@@ -1323,7 +1317,6 @@ done:
static int tc_ctl_tclass(struct sk_buff *skb, struct nlmsghdr *n, void *arg)
{
struct net *net = sock_net(skb->sk);
- struct netdev_queue *dev_queue;
struct tcmsg *tcm = NLMSG_DATA(n);
struct nlattr *tca[TCA_MAX + 1];
struct net_device *dev;
@@ -1361,7 +1354,6 @@ static int tc_ctl_tclass(struct sk_buff *skb, struct nlmsghdr *n, void *arg)
/* Step 1. Determine qdisc handle X:0 */
- dev_queue = netdev_get_tx_queue(dev, 0);
if (pid != TC_H_ROOT) {
u32 qid1 = TC_H_MAJ(pid);
@@ -1372,7 +1364,7 @@ static int tc_ctl_tclass(struct sk_buff *skb, struct nlmsghdr *n, void *arg)
} else if (qid1) {
qid = qid1;
} else if (qid == 0)
- qid = dev_queue->qdisc_sleeping->handle;
+ qid = dev->qdisc->handle;
/* Now qid is genuine qdisc handle consistent
both with parent and child.
@@ -1383,7 +1375,7 @@ static int tc_ctl_tclass(struct sk_buff *skb, struct nlmsghdr *n, void *arg)
pid = TC_H_MAKE(qid, pid);
} else {
if (qid == 0)
- qid = dev_queue->qdisc_sleeping->handle;
+ qid = dev->qdisc->handle;
}
/* OK. Locate qdisc */
@@ -1584,8 +1576,7 @@ static int tc_dump_tclass(struct sk_buff *skb, struct netlink_callback *cb)
s_t = cb->args[0];
t = 0;
- dev_queue = netdev_get_tx_queue(dev, 0);
- if (tc_dump_tclass_root(dev_queue->qdisc_sleeping, skb, tcm, cb, &t, s_t) < 0)
+ if (tc_dump_tclass_root(dev->qdisc, skb, tcm, cb, &t, s_t) < 0)
goto done;
dev_queue = &dev->rx_queue;
diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
index 6128e6f..a91f079 100644
--- a/net/sched/sch_generic.c
+++ b/net/sched/sch_generic.c
@@ -623,19 +623,6 @@ void qdisc_destroy(struct Qdisc *qdisc)
}
EXPORT_SYMBOL(qdisc_destroy);
-static bool dev_all_qdisc_sleeping_noop(struct net_device *dev)
-{
- unsigned int i;
-
- for (i = 0; i < dev->num_tx_queues; i++) {
- struct netdev_queue *txq = netdev_get_tx_queue(dev, i);
-
- if (txq->qdisc_sleeping != &noop_qdisc)
- return false;
- }
- return true;
-}
-
static void attach_one_default_qdisc(struct net_device *dev,
struct netdev_queue *dev_queue,
void *_unused)
@@ -677,6 +664,7 @@ static void transition_one_qdisc(struct net_device *dev,
void dev_activate(struct net_device *dev)
{
+ struct netdev_queue *txq;
int need_watchdog;
/* No queueing discipline is attached to device;
@@ -685,9 +673,14 @@ void dev_activate(struct net_device *dev)
virtual interfaces
*/
- if (dev_all_qdisc_sleeping_noop(dev))
+ if (dev->qdisc == &noop_qdisc) {
netdev_for_each_tx_queue(dev, attach_one_default_qdisc, NULL);
+ txq = netdev_get_tx_queue(dev, 0);
+ dev->qdisc = txq->qdisc_sleeping;
+ atomic_inc(&dev->qdisc->refcnt);
+ }
+
if (!netif_carrier_ok(dev))
/* Delay activation until next carrier-on event */
return;
@@ -777,6 +770,7 @@ static void dev_init_scheduler_queue(struct net_device *dev,
void dev_init_scheduler(struct net_device *dev)
{
+ dev->qdisc = &noop_qdisc;
netdev_for_each_tx_queue(dev, dev_init_scheduler_queue, &noop_qdisc);
dev_init_scheduler_queue(dev, &dev->rx_queue, &noop_qdisc);
@@ -802,5 +796,8 @@ void dev_shutdown(struct net_device *dev)
{
netdev_for_each_tx_queue(dev, shutdown_scheduler_queue, &noop_qdisc);
shutdown_scheduler_queue(dev, &dev->rx_queue, &noop_qdisc);
+ qdisc_destroy(dev->qdisc);
+ dev->qdisc = &noop_qdisc;
+
WARN_ON(timer_pending(&dev->watchdog_timer));
}
[-- Attachment #3: 02.diff --]
[-- Type: text/x-diff, Size: 2943 bytes --]
commit 9b1d1d62936b6ec6e51956d77b83743bea67e1ac
Author: Patrick McHardy <kaber@trash.net>
Date: Fri Sep 4 03:13:54 2009 +0200
net_sched: move dev_graft_qdisc() to sch_generic.c
It will be used in a following patch by the multiqueue qdisc.
Signed-off-by: Patrick McHardy <kaber@trash.net>
diff --git a/include/net/sch_generic.h b/include/net/sch_generic.h
index 84b3fc2..a1a3f3f 100644
--- a/include/net/sch_generic.h
+++ b/include/net/sch_generic.h
@@ -302,6 +302,8 @@ extern void dev_init_scheduler(struct net_device *dev);
extern void dev_shutdown(struct net_device *dev);
extern void dev_activate(struct net_device *dev);
extern void dev_deactivate(struct net_device *dev);
+extern struct Qdisc *dev_graft_qdisc(struct netdev_queue *dev_queue,
+ struct Qdisc *qdisc);
extern void qdisc_reset(struct Qdisc *qdisc);
extern void qdisc_destroy(struct Qdisc *qdisc);
extern void qdisc_tree_decrease_qlen(struct Qdisc *qdisc, unsigned int n);
diff --git a/net/sched/sch_api.c b/net/sched/sch_api.c
index 285ca25..c0d2704 100644
--- a/net/sched/sch_api.c
+++ b/net/sched/sch_api.c
@@ -610,32 +610,6 @@ static u32 qdisc_alloc_handle(struct net_device *dev)
return i>0 ? autohandle : 0;
}
-/* Attach toplevel qdisc to device queue. */
-
-static struct Qdisc *dev_graft_qdisc(struct netdev_queue *dev_queue,
- struct Qdisc *qdisc)
-{
- struct Qdisc *oqdisc = dev_queue->qdisc_sleeping;
- spinlock_t *root_lock;
-
- root_lock = qdisc_lock(oqdisc);
- spin_lock_bh(root_lock);
-
- /* Prune old scheduler */
- if (oqdisc && atomic_read(&oqdisc->refcnt) <= 1)
- qdisc_reset(oqdisc);
-
- /* ... and graft new one */
- if (qdisc == NULL)
- qdisc = &noop_qdisc;
- dev_queue->qdisc_sleeping = qdisc;
- rcu_assign_pointer(dev_queue->qdisc, &noop_qdisc);
-
- spin_unlock_bh(root_lock);
-
- return oqdisc;
-}
-
void qdisc_tree_decrease_qlen(struct Qdisc *sch, unsigned int n)
{
const struct Qdisc_class_ops *cops;
diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
index a91f079..e7c47ce 100644
--- a/net/sched/sch_generic.c
+++ b/net/sched/sch_generic.c
@@ -623,6 +623,31 @@ void qdisc_destroy(struct Qdisc *qdisc)
}
EXPORT_SYMBOL(qdisc_destroy);
+/* Attach toplevel qdisc to device queue. */
+struct Qdisc *dev_graft_qdisc(struct netdev_queue *dev_queue,
+ struct Qdisc *qdisc)
+{
+ struct Qdisc *oqdisc = dev_queue->qdisc_sleeping;
+ spinlock_t *root_lock;
+
+ root_lock = qdisc_lock(oqdisc);
+ spin_lock_bh(root_lock);
+
+ /* Prune old scheduler */
+ if (oqdisc && atomic_read(&oqdisc->refcnt) <= 1)
+ qdisc_reset(oqdisc);
+
+ /* ... and graft new one */
+ if (qdisc == NULL)
+ qdisc = &noop_qdisc;
+ dev_queue->qdisc_sleeping = qdisc;
+ rcu_assign_pointer(dev_queue->qdisc, &noop_qdisc);
+
+ spin_unlock_bh(root_lock);
+
+ return oqdisc;
+}
+
static void attach_one_default_qdisc(struct net_device *dev,
struct netdev_queue *dev_queue,
void *_unused)
[-- Attachment #4: 03.diff --]
[-- Type: text/x-diff, Size: 10612 bytes --]
commit 7f8e95c412644cc23173c9c5673df794fb9b65f7
Author: Patrick McHardy <kaber@trash.net>
Date: Fri Sep 4 03:24:28 2009 +0200
net_sched: add classful multiqueue dummy scheduler
This patch adds a classful dummy scheduler which can be used as root qdisc
for multiqueue devices and exposes each device queue as a child class.
This allows to address queues individually and graft them similar to regular
classes. Additionally it presents an accumulated view of the statistics of
all real root qdiscs in the dummy root.
Signed-off-by: Patrick McHardy <kaber@trash.net>
diff --git a/include/net/sch_generic.h b/include/net/sch_generic.h
index a1a3f3f..b76e4fa 100644
--- a/include/net/sch_generic.h
+++ b/include/net/sch_generic.h
@@ -122,6 +122,7 @@ struct Qdisc_ops
void (*reset)(struct Qdisc *);
void (*destroy)(struct Qdisc *);
int (*change)(struct Qdisc *, struct nlattr *arg);
+ void (*attach)(struct Qdisc *);
int (*dump)(struct Qdisc *, struct sk_buff *);
int (*dump_stats)(struct Qdisc *, struct gnet_dump *);
@@ -255,6 +256,8 @@ static inline void sch_tree_unlock(struct Qdisc *q)
extern struct Qdisc noop_qdisc;
extern struct Qdisc_ops noop_qdisc_ops;
+extern struct Qdisc_ops pfifo_fast_ops;
+extern struct Qdisc_ops mq_qdisc_ops;
struct Qdisc_class_common
{
diff --git a/net/sched/Makefile b/net/sched/Makefile
index 54d950c..f14e71b 100644
--- a/net/sched/Makefile
+++ b/net/sched/Makefile
@@ -2,7 +2,7 @@
# Makefile for the Linux Traffic Control Unit.
#
-obj-y := sch_generic.o
+obj-y := sch_generic.o sch_mq.o
obj-$(CONFIG_NET_SCHED) += sch_api.o sch_blackhole.o
obj-$(CONFIG_NET_CLS) += cls_api.o
diff --git a/net/sched/sch_api.c b/net/sched/sch_api.c
index c0d2704..dc41cb1 100644
--- a/net/sched/sch_api.c
+++ b/net/sched/sch_api.c
@@ -678,6 +678,11 @@ static int qdisc_graft(struct net_device *dev, struct Qdisc *parent,
if (dev->flags & IFF_UP)
dev_deactivate(dev);
+ if (new && new->ops->attach) {
+ new->ops->attach(new);
+ num_q = 0;
+ }
+
for (i = 0; i < num_q; i++) {
struct netdev_queue *dev_queue = &dev->rx_queue;
@@ -692,7 +697,7 @@ static int qdisc_graft(struct net_device *dev, struct Qdisc *parent,
}
notify_and_destroy(skb, n, classid, dev->qdisc, new);
- if (new)
+ if (new && !new->ops->attach)
atomic_inc(&new->refcnt);
dev->qdisc = new ? : &noop_qdisc;
@@ -1670,6 +1675,7 @@ static int __init pktsched_init(void)
{
register_qdisc(&pfifo_qdisc_ops);
register_qdisc(&bfifo_qdisc_ops);
+ register_qdisc(&mq_qdisc_ops);
proc_net_fops_create(&init_net, "psched", 0, &psched_fops);
rtnl_register(PF_UNSPEC, RTM_NEWQDISC, tc_modify_qdisc, NULL);
diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
index e7c47ce..4ae6aa5 100644
--- a/net/sched/sch_generic.c
+++ b/net/sched/sch_generic.c
@@ -514,7 +514,7 @@ static int pfifo_fast_init(struct Qdisc *qdisc, struct nlattr *opt)
return 0;
}
-static struct Qdisc_ops pfifo_fast_ops __read_mostly = {
+struct Qdisc_ops pfifo_fast_ops __read_mostly = {
.id = "pfifo_fast",
.priv_size = sizeof(struct pfifo_fast_priv),
.enqueue = pfifo_fast_enqueue,
@@ -670,6 +670,26 @@ static void attach_one_default_qdisc(struct net_device *dev,
dev_queue->qdisc_sleeping = qdisc;
}
+static void attach_default_qdiscs(struct net_device *dev)
+{
+ struct netdev_queue *txq;
+ struct Qdisc *qdisc;
+
+ txq = netdev_get_tx_queue(dev, 0);
+
+ if (!netif_is_multiqueue(dev) || dev->tx_queue_len == 0) {
+ netdev_for_each_tx_queue(dev, attach_one_default_qdisc, NULL);
+ dev->qdisc = txq->qdisc_sleeping;
+ atomic_inc(&dev->qdisc->refcnt);
+ } else {
+ qdisc = qdisc_create_dflt(dev, txq, &mq_qdisc_ops, TC_H_ROOT);
+ if (qdisc) {
+ qdisc->ops->attach(qdisc);
+ dev->qdisc = qdisc;
+ }
+ }
+}
+
static void transition_one_qdisc(struct net_device *dev,
struct netdev_queue *dev_queue,
void *_need_watchdog)
@@ -689,7 +709,6 @@ static void transition_one_qdisc(struct net_device *dev,
void dev_activate(struct net_device *dev)
{
- struct netdev_queue *txq;
int need_watchdog;
/* No queueing discipline is attached to device;
@@ -698,13 +717,8 @@ void dev_activate(struct net_device *dev)
virtual interfaces
*/
- if (dev->qdisc == &noop_qdisc) {
- netdev_for_each_tx_queue(dev, attach_one_default_qdisc, NULL);
-
- txq = netdev_get_tx_queue(dev, 0);
- dev->qdisc = txq->qdisc_sleeping;
- atomic_inc(&dev->qdisc->refcnt);
- }
+ if (dev->qdisc == &noop_qdisc)
+ attach_default_qdiscs(dev);
if (!netif_carrier_ok(dev))
/* Delay activation until next carrier-on event */
diff --git a/net/sched/sch_mq.c b/net/sched/sch_mq.c
new file mode 100644
index 0000000..fadaa6d
--- /dev/null
+++ b/net/sched/sch_mq.c
@@ -0,0 +1,244 @@
+/*
+ * net/sched/sch_mq.c Classful multiqueue dummy scheduler
+ *
+ * Copyright (c) 2009 Patrick McHardy <kaber@trash.net>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * version 2 as published by the Free Software Foundation.
+ */
+
+#include <linux/types.h>
+#include <linux/kernel.h>
+#include <linux/string.h>
+#include <linux/errno.h>
+#include <linux/skbuff.h>
+#include <net/netlink.h>
+#include <net/pkt_sched.h>
+
+struct mq_sched {
+ struct Qdisc **qdiscs;
+};
+
+static int mq_init(struct Qdisc *sch, struct nlattr *opt)
+{
+ struct net_device *dev = qdisc_dev(sch);
+ struct mq_sched *priv = qdisc_priv(sch);
+ struct netdev_queue *dev_queue;
+ struct Qdisc *qdisc;
+ unsigned int ntx;
+
+ if (sch->parent != TC_H_ROOT)
+ return -EOPNOTSUPP;
+
+ if (!netif_is_multiqueue(dev))
+ return -EOPNOTSUPP;
+
+ /* pre-allocate qdiscs, attachment can't fail */
+ priv->qdiscs = kcalloc(dev->num_tx_queues, sizeof(priv->qdiscs[0]),
+ GFP_KERNEL);
+ if (priv->qdiscs == NULL)
+ return -ENOMEM;
+
+ for (ntx = 0; ntx < dev->num_tx_queues; ntx++) {
+ dev_queue = netdev_get_tx_queue(dev, ntx);
+ qdisc = qdisc_create_dflt(dev, dev_queue, &pfifo_fast_ops,
+ TC_H_MAKE(TC_H_MAJ(sch->handle),
+ TC_H_MIN(ntx + 1)));
+ if (qdisc == NULL)
+ goto err;
+ qdisc->flags |= TCQ_F_CAN_BYPASS;
+ priv->qdiscs[ntx] = qdisc;
+ }
+
+ return 0;
+
+err:
+ while (ntx > 0)
+ qdisc_destroy(priv->qdiscs[--ntx]);
+ kfree(priv->qdiscs);
+ return -ENOMEM;
+}
+
+static void mq_attach(struct Qdisc *sch)
+{
+ struct net_device *dev = qdisc_dev(sch);
+ struct mq_sched *priv = qdisc_priv(sch);
+ struct Qdisc *qdisc;
+ unsigned int ntx;
+
+ for (ntx = 0; ntx < dev->num_tx_queues; ntx++) {
+ qdisc = priv->qdiscs[ntx];
+ qdisc = dev_graft_qdisc(qdisc->dev_queue, qdisc);
+ if (qdisc)
+ qdisc_destroy(qdisc);
+ }
+ kfree(priv->qdiscs);
+}
+
+static int mq_dump(struct Qdisc *sch, struct sk_buff *skb)
+{
+ struct net_device *dev = qdisc_dev(sch);
+ struct Qdisc *qdisc;
+ unsigned int ntx;
+
+ sch->q.qlen = 0;
+ memset(&sch->bstats, 0, sizeof(sch->bstats));
+ memset(&sch->qstats, 0, sizeof(sch->qstats));
+
+ for (ntx = 0; ntx < dev->num_tx_queues; ntx++) {
+ qdisc = netdev_get_tx_queue(dev, ntx)->qdisc;
+ sch->q.qlen += qdisc->q.qlen;
+ sch->bstats.bytes += qdisc->bstats.bytes;
+ sch->bstats.packets += qdisc->bstats.packets;
+ sch->qstats.qlen += qdisc->qstats.qlen;
+ sch->qstats.backlog += qdisc->qstats.backlog;
+ sch->qstats.drops += qdisc->qstats.drops;
+ sch->qstats.requeues += qdisc->qstats.requeues;
+ sch->qstats.overlimits += qdisc->qstats.overlimits;
+ }
+ return 0;
+}
+
+static struct netdev_queue *mq_queue_get(struct Qdisc *sch, unsigned long arg)
+{
+ struct net_device *dev = qdisc_dev(sch);
+ unsigned long ntx = arg - 1;
+
+ if (ntx >= dev->num_tx_queues)
+ return NULL;
+ return netdev_get_tx_queue(dev, ntx);
+}
+
+static int mq_graft(struct Qdisc *sch, unsigned long arg, struct Qdisc *new,
+ struct Qdisc **old)
+{
+ struct net_device *dev = qdisc_dev(sch);
+ struct netdev_queue *dev_queue;
+
+ dev_queue = mq_queue_get(sch, arg);
+ if (dev_queue == NULL)
+ return -ENOENT;
+
+ if (dev->flags & IFF_UP)
+ dev_deactivate(dev);
+
+ *old = dev_graft_qdisc(dev_queue, new);
+
+ if (dev->flags & IFF_UP)
+ dev_activate(dev);
+ return 0;
+}
+
+static struct Qdisc *mq_leaf(struct Qdisc *sch, unsigned long arg)
+{
+ struct netdev_queue *dev_queue;
+
+ dev_queue = mq_queue_get(sch, arg);
+ if (dev_queue == NULL)
+ return NULL;
+
+ return dev_queue->qdisc;
+}
+
+static unsigned long mq_get(struct Qdisc *sch, u32 classid)
+{
+ unsigned int ntx = TC_H_MIN(classid);
+
+ if (!mq_queue_get(sch, ntx))
+ return 0;
+ return ntx;
+}
+
+static void mq_put(struct Qdisc *sch, unsigned long cl)
+{
+ return;
+}
+
+static int mq_change(struct Qdisc *sch, u32 handle, u32 parent,
+ struct nlattr **tca, unsigned long *arg)
+{
+ if (!mq_queue_get(sch, *arg))
+ return -ENOENT;
+ return 0;
+}
+
+static int mq_delete(struct Qdisc *sch, unsigned long cl)
+{
+ return -EOPNOTSUPP;
+}
+
+static int mq_dump_class(struct Qdisc *sch, unsigned long cl,
+ struct sk_buff *skb, struct tcmsg *tcm)
+{
+ struct netdev_queue *dev_queue;
+
+ dev_queue = mq_queue_get(sch, cl);
+ if (dev_queue == NULL)
+ return -ENOENT;
+
+ tcm->tcm_parent = TC_H_ROOT;
+ tcm->tcm_handle |= TC_H_MIN(cl);
+ if (dev_queue->qdisc)
+ tcm->tcm_info = dev_queue->qdisc->handle;
+ return 0;
+}
+
+static int mq_dump_class_stats(struct Qdisc *sch, unsigned long cl,
+ struct gnet_dump *d)
+{
+ struct netdev_queue *dev_queue;
+
+ dev_queue = mq_queue_get(sch, cl);
+ sch = dev_queue->qdisc_sleeping;
+
+ if (gnet_stats_copy_basic(d, &sch->bstats) < 0 ||
+ gnet_stats_copy_queue(d, &sch->qstats) < 0)
+ return -1;
+ return 0;
+}
+
+static void mq_walk(struct Qdisc *sch, struct qdisc_walker *arg)
+{
+ struct net_device *dev = qdisc_dev(sch);
+ unsigned int ntx;
+
+ if (arg->stop)
+ return;
+
+ arg->count = arg->skip;
+ for (ntx = arg->skip; ntx < dev->num_tx_queues; ntx++) {
+ if (arg->fn(sch, ntx + 1, arg) < 0) {
+ arg->stop = 1;
+ break;
+ }
+ arg->count++;
+ }
+}
+
+static const struct Qdisc_class_ops mq_class_ops = {
+ .graft = mq_graft,
+ .leaf = mq_leaf,
+ .get = mq_get,
+ .put = mq_put,
+ .change = mq_change,
+ .delete = mq_delete,
+ .walk = mq_walk,
+ .tcf_chain = NULL,
+ .bind_tcf = NULL,
+ .unbind_tcf = NULL,
+ .dump = mq_dump_class,
+ .dump_stats = mq_dump_class_stats,
+};
+
+struct Qdisc_ops mq_qdisc_ops __read_mostly = {
+ .cl_ops = &mq_class_ops,
+ .id = "mq",
+ .priv_size = sizeof(struct mq_sched),
+ .init = mq_init,
+ //.reset = mq_reset,
+ //.destroy = mq_destroy,
+ .attach = mq_attach,
+ .dump = mq_dump,
+ .owner = THIS_MODULE,
+};
^ permalink raw reply related
* [PATCH 00/29] ioatdma: towards raid support
From: Dan Williams @ 2009-09-04 2:30 UTC (permalink / raw)
To: linux-kernel; +Cc: linux-raid, netdev, maciej.sosnowski
The v3.2 version of the Intel(R) QuickData Technology specification adds
support for raid5 and raid6 offloads. The v3.2 implementation, like v3,
reuses the same basic dma descriptor ring mechanism that was introduced
for v2 devices. At the same time v3.2 breaks some assumptions of the
existing driver with the addition of mechanisms like extended
descriptors and interrupt-driven completion callbacks as specified by
the async_tx/raid-offload implementation.
The current driver blurs the lines between v1, v2, and v3 making it
difficult to append raid functionality without modifying legacy code
paths. The primary goal of this patchset is to refactor and isolate the
legacy (v1/v2) code paths from feature additions going forward by
migrating hardware version specific code to hardware version specific
files. It also takes the opportunity to perform some cleanups and
optimizations. For example, the conversion from a linked list to a
ring-buffer for the v2/v3 descriptor ring makes the code more readable
and reduces the size of the software descriptor from 136-bytes to
88-bytes with all the fields needed for NET_DMA fitting into one
cacheline.
One robustness fix required by the raid code is the ability to poll for
a descriptor while the channel is experiencing an "out of descriptors"
condition. This has been added for the v2/v3 ring implementation.
This cleanup is available via git at:
git://git.kernel.org/pub/scm/linux/kernel/git/djbw/async_tx.git ioat-refactor
It will hit -next shortly.
The driver is used by net/ when CONFIG_NET_DMA=y, and will be used by
drivers/md/ (after the raid extensions) when CONFIG_ASYNC_TX_DMA=y.
Hence the Cc: of netdev and linux-raid.
Regards,
Dan
---
Dan Williams (29):
ioat: move to drivers/dma/ioat/
ioat: move definitions to dma.h
ioat: convert ioat_probe to pcim/devm
ioat: cleanup some long deref chains and 80 column collisions
ioat: kill function prototype ifdef guards
ioat: split ioat_dma_probe into core/version-specific routines
ioat: fix type mismatch for ->dmacount
ioat: define descriptor control bit-field
ioat1: move descriptor allocation from submit to prep
ioat: fix self test interrupts
ioat: prepare the code for ioat[12]_dma_chan split
ioat2,3: convert to a true ring buffer
ioat1: kill unused unmap parameters
ioat: add some dev_dbg() calls
ioat: cleanup completion status reads
ioat: ignore reserved bits for chancnt and xfercap
ioat: preserve chanctrl bits when re-arming interrupts
ioat: ___devinit annotate the initialization paths
ioat1: trim ioat_dma_desc_sw
ioat: switch watchdog and reset handler from workqueue to timer
ioat2,3: dynamically resize descriptor ring
net_dma: poll for a descriptor after allocation failure
dw_dmac: implement a private tx_list
fsldma: implement a private tx_list
iop-adma: implement a private tx_list
ioat: implement a private tx_list
mv_xor: implement a private tx_list
dmaengine: kill tx_list
ioat2,3: cacheline align software descriptor allocations
arch/arm/include/asm/hardware/iop_adma.h | 2 +
drivers/dma/Makefile | 3 +-
drivers/dma/dmaengine.c | 1 -
drivers/dma/dw_dmac.c | 19 +-
drivers/dma/dw_dmac_regs.h | 1 +
drivers/dma/fsldma.c | 7 +-
drivers/dma/fsldma.h | 1 +
drivers/dma/ioat/Makefile | 2 +
drivers/dma/{ioat_dca.c => ioat/dca.c} | 13 +-
drivers/dma/ioat/dma.c | 1140 +++++++++++++
drivers/dma/ioat/dma.h | 307 ++++
drivers/dma/ioat/dma_v2.c | 883 ++++++++++
drivers/dma/ioat/dma_v2.h | 147 ++
drivers/dma/{ioatdma_hw.h => ioat/hw.h} | 45 +-
drivers/dma/{ioat.c => ioat/pci.c} | 163 +-
.../dma/{ioatdma_registers.h => ioat/registers.h} | 34 +-
drivers/dma/ioat_dma.c | 1741 --------------------
drivers/dma/ioatdma.h | 165 --
drivers/dma/iop-adma.c | 9 +-
drivers/dma/iovlock.c | 10 +
drivers/dma/mv_xor.c | 7 +-
drivers/dma/mv_xor.h | 4 +-
include/linux/dmaengine.h | 3 -
23 files changed, 2647 insertions(+), 2060 deletions(-)
create mode 100644 drivers/dma/ioat/Makefile
rename drivers/dma/{ioat_dca.c => ioat/dca.c} (98%)
create mode 100644 drivers/dma/ioat/dma.c
create mode 100644 drivers/dma/ioat/dma.h
create mode 100644 drivers/dma/ioat/dma_v2.c
create mode 100644 drivers/dma/ioat/dma_v2.h
rename drivers/dma/{ioatdma_hw.h => ioat/hw.h} (67%)
rename drivers/dma/{ioat.c => ioat/pci.c} (62%)
rename drivers/dma/{ioatdma_registers.h => ioat/registers.h} (91%)
delete mode 100644 drivers/dma/ioat_dma.c
delete mode 100644 drivers/dma/ioatdma.h
^ permalink raw reply
* [PATCH 01/29] ioat: move to drivers/dma/ioat/
From: Dan Williams @ 2009-09-04 2:30 UTC (permalink / raw)
To: linux-kernel; +Cc: linux-raid, netdev, Maciej Sosnowski
In-Reply-To: <20090904022733.32667.77626.stgit@dwillia2-linux.ch.intel.com>
When first created the ioat driver was the only inhabitant of
drivers/dma/. Now, it is the only multi-file (more than a .c and a .h)
driver in the directory. Moving it to an ioat/ subdirectory allows the
naming convention to be cleaned up, and allows for future splitting of
the source files by hardware version (v1, v2, and v3).
Signed-off-by: Maciej Sosnowski <maciej.sosnowski@intel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
drivers/dma/Makefile | 3 +--
drivers/dma/ioat/Makefile | 2 ++
drivers/dma/ioat/dca.c | 4 ++--
drivers/dma/ioat/dma.c | 6 +++---
drivers/dma/ioat/dma.h | 2 +-
drivers/dma/ioat/hw.h | 0
drivers/dma/ioat/pci.c | 6 +++---
drivers/dma/ioat/registers.h | 0
8 files changed, 12 insertions(+), 11 deletions(-)
create mode 100644 drivers/dma/ioat/Makefile
rename drivers/dma/{ioat_dca.c => ioat/dca.c} (99%)
rename drivers/dma/{ioat_dma.c => ioat/dma.c} (99%)
rename drivers/dma/{ioatdma.h => ioat/dma.h} (99%)
rename drivers/dma/{ioatdma_hw.h => ioat/hw.h} (100%)
rename drivers/dma/{ioat.c => ioat/pci.c} (98%)
rename drivers/dma/{ioatdma_registers.h => ioat/registers.h} (100%)
diff --git a/drivers/dma/Makefile b/drivers/dma/Makefile
index 2e5dc96..a1cb285 100644
--- a/drivers/dma/Makefile
+++ b/drivers/dma/Makefile
@@ -1,8 +1,7 @@
obj-$(CONFIG_DMA_ENGINE) += dmaengine.o
obj-$(CONFIG_NET_DMA) += iovlock.o
obj-$(CONFIG_DMATEST) += dmatest.o
-obj-$(CONFIG_INTEL_IOATDMA) += ioatdma.o
-ioatdma-objs := ioat.o ioat_dma.o ioat_dca.o
+obj-$(CONFIG_INTEL_IOATDMA) += ioat/
obj-$(CONFIG_INTEL_IOP_ADMA) += iop-adma.o
obj-$(CONFIG_FSL_DMA) += fsldma.o
obj-$(CONFIG_MV_XOR) += mv_xor.o
diff --git a/drivers/dma/ioat/Makefile b/drivers/dma/ioat/Makefile
new file mode 100644
index 0000000..2ce3d3a
--- /dev/null
+++ b/drivers/dma/ioat/Makefile
@@ -0,0 +1,2 @@
+obj-$(CONFIG_INTEL_IOATDMA) += ioatdma.o
+ioatdma-objs := pci.o dma.o dca.o
diff --git a/drivers/dma/ioat_dca.c b/drivers/dma/ioat/dca.c
similarity index 99%
rename from drivers/dma/ioat_dca.c
rename to drivers/dma/ioat/dca.c
index c012a1e..af1c762 100644
--- a/drivers/dma/ioat_dca.c
+++ b/drivers/dma/ioat/dca.c
@@ -33,8 +33,8 @@
#define cpu_physical_id(cpu) (cpuid_ebx(1) >> 24)
#endif
-#include "ioatdma.h"
-#include "ioatdma_registers.h"
+#include "dma.h"
+#include "registers.h"
/*
* Bit 7 of a tag map entry is the "valid" bit, if it is set then bits 0:6
diff --git a/drivers/dma/ioat_dma.c b/drivers/dma/ioat/dma.c
similarity index 99%
rename from drivers/dma/ioat_dma.c
rename to drivers/dma/ioat/dma.c
index a600fc0..648797e 100644
--- a/drivers/dma/ioat_dma.c
+++ b/drivers/dma/ioat/dma.c
@@ -34,9 +34,9 @@
#include <linux/dma-mapping.h>
#include <linux/workqueue.h>
#include <linux/i7300_idle.h>
-#include "ioatdma.h"
-#include "ioatdma_registers.h"
-#include "ioatdma_hw.h"
+#include "dma.h"
+#include "registers.h"
+#include "hw.h"
#define to_ioat_chan(chan) container_of(chan, struct ioat_dma_chan, common)
#define to_ioatdma_device(dev) container_of(dev, struct ioatdma_device, common)
diff --git a/drivers/dma/ioatdma.h b/drivers/dma/ioat/dma.h
similarity index 99%
rename from drivers/dma/ioatdma.h
rename to drivers/dma/ioat/dma.h
index a52ff4b..e80e787 100644
--- a/drivers/dma/ioatdma.h
+++ b/drivers/dma/ioat/dma.h
@@ -22,7 +22,7 @@
#define IOATDMA_H
#include <linux/dmaengine.h>
-#include "ioatdma_hw.h"
+#include "hw.h"
#include <linux/init.h>
#include <linux/dmapool.h>
#include <linux/cache.h>
diff --git a/drivers/dma/ioatdma_hw.h b/drivers/dma/ioat/hw.h
similarity index 100%
rename from drivers/dma/ioatdma_hw.h
rename to drivers/dma/ioat/hw.h
diff --git a/drivers/dma/ioat.c b/drivers/dma/ioat/pci.c
similarity index 98%
rename from drivers/dma/ioat.c
rename to drivers/dma/ioat/pci.c
index 2225bb6..d7948bf 100644
--- a/drivers/dma/ioat.c
+++ b/drivers/dma/ioat/pci.c
@@ -30,9 +30,9 @@
#include <linux/pci.h>
#include <linux/interrupt.h>
#include <linux/dca.h>
-#include "ioatdma.h"
-#include "ioatdma_registers.h"
-#include "ioatdma_hw.h"
+#include "dma.h"
+#include "registers.h"
+#include "hw.h"
MODULE_VERSION(IOAT_DMA_VERSION);
MODULE_LICENSE("GPL");
diff --git a/drivers/dma/ioatdma_registers.h b/drivers/dma/ioat/registers.h
similarity index 100%
rename from drivers/dma/ioatdma_registers.h
rename to drivers/dma/ioat/registers.h
^ permalink raw reply related
* [PATCH 02/29] ioat: move definitions to dma.h
From: Dan Williams @ 2009-09-04 2:30 UTC (permalink / raw)
To: linux-kernel; +Cc: linux-raid, netdev, Maciej Sosnowski
In-Reply-To: <20090904022733.32667.77626.stgit@dwillia2-linux.ch.intel.com>
Some of these defines may be useful outside of dma.c and the header is
private so there are no namespace pollution concerns.
Signed-off-by: Maciej Sosnowski <maciej.sosnowski@intel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
drivers/dma/ioat/dma.c | 14 --------------
drivers/dma/ioat/dma.h | 16 ++++++++++++++++
2 files changed, 16 insertions(+), 14 deletions(-)
diff --git a/drivers/dma/ioat/dma.c b/drivers/dma/ioat/dma.c
index 648797e..16c0807 100644
--- a/drivers/dma/ioat/dma.c
+++ b/drivers/dma/ioat/dma.c
@@ -38,28 +38,14 @@
#include "registers.h"
#include "hw.h"
-#define to_ioat_chan(chan) container_of(chan, struct ioat_dma_chan, common)
-#define to_ioatdma_device(dev) container_of(dev, struct ioatdma_device, common)
-#define to_ioat_desc(lh) container_of(lh, struct ioat_desc_sw, node)
-#define tx_to_ioat_desc(tx) container_of(tx, struct ioat_desc_sw, async_tx)
-
-#define chan_num(ch) ((int)((ch)->reg_base - (ch)->device->reg_base) / 0x80)
static int ioat_pending_level = 4;
module_param(ioat_pending_level, int, 0644);
MODULE_PARM_DESC(ioat_pending_level,
"high-water mark for pushing ioat descriptors (default: 4)");
-#define RESET_DELAY msecs_to_jiffies(100)
-#define WATCHDOG_DELAY round_jiffies(msecs_to_jiffies(2000))
static void ioat_dma_chan_reset_part2(struct work_struct *work);
static void ioat_dma_chan_watchdog(struct work_struct *work);
-/*
- * workaround for IOAT ver.3.0 null descriptor issue
- * (channel returns error when size is 0)
- */
-#define NULL_DESC_BUFFER_SIZE 1
-
/* internal functions */
static void ioat_dma_start_null_desc(struct ioat_dma_chan *ioat_chan);
static void ioat_dma_memcpy_cleanup(struct ioat_dma_chan *ioat_chan);
diff --git a/drivers/dma/ioat/dma.h b/drivers/dma/ioat/dma.h
index e80e787..ccb400f 100644
--- a/drivers/dma/ioat/dma.h
+++ b/drivers/dma/ioat/dma.h
@@ -43,6 +43,22 @@ enum ioat_interrupt {
#define IOAT_DMA_DCA_ANY_CPU ~0
#define IOAT_WATCHDOG_PERIOD (2 * HZ)
+#define to_ioat_chan(chan) container_of(chan, struct ioat_dma_chan, common)
+#define to_ioatdma_device(dev) container_of(dev, struct ioatdma_device, common)
+#define to_ioat_desc(lh) container_of(lh, struct ioat_desc_sw, node)
+#define tx_to_ioat_desc(tx) container_of(tx, struct ioat_desc_sw, async_tx)
+
+#define chan_num(ch) ((int)((ch)->reg_base - (ch)->device->reg_base) / 0x80)
+
+#define RESET_DELAY msecs_to_jiffies(100)
+#define WATCHDOG_DELAY round_jiffies(msecs_to_jiffies(2000))
+
+/*
+ * workaround for IOAT ver.3.0 null descriptor issue
+ * (channel returns error when size is 0)
+ */
+#define NULL_DESC_BUFFER_SIZE 1
+
/**
* struct ioatdma_device - internal representation of a IOAT device
^ permalink raw reply related
* [PATCH 03/29] ioat: convert ioat_probe to pcim/devm
From: Dan Williams @ 2009-09-04 2:30 UTC (permalink / raw)
To: linux-kernel; +Cc: linux-raid, netdev, Maciej Sosnowski
In-Reply-To: <20090904022733.32667.77626.stgit@dwillia2-linux.ch.intel.com>
The driver currently duplicates much of what these routines offer, so
just use the common code. For example ->irq_mode tracks what interrupt
mode was initialized, which duplicates the ->msix_enabled and
->msi_enabled handling in pcim_release.
This also adds a check to the return value of dma_async_device_register,
which can fail.
Signed-off-by: Maciej Sosnowski <maciej.sosnowski@intel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
drivers/dma/ioat/dma.c | 130 ++++++++++++++++--------------------------------
drivers/dma/ioat/dma.h | 11 ----
drivers/dma/ioat/hw.h | 1
drivers/dma/ioat/pci.c | 67 +++++++++----------------
4 files changed, 68 insertions(+), 141 deletions(-)
diff --git a/drivers/dma/ioat/dma.c b/drivers/dma/ioat/dma.c
index 16c0807..65f8b74 100644
--- a/drivers/dma/ioat/dma.c
+++ b/drivers/dma/ioat/dma.c
@@ -121,6 +121,7 @@ static int ioat_dma_enumerate_channels(struct ioatdma_device *device)
u32 xfercap;
int i;
struct ioat_dma_chan *ioat_chan;
+ struct device *dev = &device->pdev->dev;
/*
* IOAT ver.3 workarounds
@@ -164,7 +165,7 @@ static int ioat_dma_enumerate_channels(struct ioatdma_device *device)
}
#endif
for (i = 0; i < device->common.chancnt; i++) {
- ioat_chan = kzalloc(sizeof(*ioat_chan), GFP_KERNEL);
+ ioat_chan = devm_kzalloc(dev, sizeof(*ioat_chan), GFP_KERNEL);
if (!ioat_chan) {
device->common.chancnt = i;
break;
@@ -1450,7 +1451,11 @@ MODULE_PARM_DESC(ioat_interrupt_style,
static int ioat_dma_setup_interrupts(struct ioatdma_device *device)
{
struct ioat_dma_chan *ioat_chan;
- int err, i, j, msixcnt;
+ struct pci_dev *pdev = device->pdev;
+ struct device *dev = &pdev->dev;
+ struct msix_entry *msix;
+ int i, j, msixcnt;
+ int err = -EINVAL;
u8 intrctrl = 0;
if (!strcmp(ioat_interrupt_style, "msix"))
@@ -1461,8 +1466,7 @@ static int ioat_dma_setup_interrupts(struct ioatdma_device *device)
goto msi;
if (!strcmp(ioat_interrupt_style, "intx"))
goto intx;
- dev_err(&device->pdev->dev, "invalid ioat_interrupt_style %s\n",
- ioat_interrupt_style);
+ dev_err(dev, "invalid ioat_interrupt_style %s\n", ioat_interrupt_style);
goto err_no_irq;
msix:
@@ -1471,55 +1475,55 @@ msix:
for (i = 0; i < msixcnt; i++)
device->msix_entries[i].entry = i;
- err = pci_enable_msix(device->pdev, device->msix_entries, msixcnt);
+ err = pci_enable_msix(pdev, device->msix_entries, msixcnt);
if (err < 0)
goto msi;
if (err > 0)
goto msix_single_vector;
for (i = 0; i < msixcnt; i++) {
+ msix = &device->msix_entries[i];
ioat_chan = ioat_lookup_chan_by_index(device, i);
- err = request_irq(device->msix_entries[i].vector,
- ioat_dma_do_interrupt_msix,
- 0, "ioat-msix", ioat_chan);
+ err = devm_request_irq(dev, msix->vector,
+ ioat_dma_do_interrupt_msix, 0,
+ "ioat-msix", ioat_chan);
if (err) {
for (j = 0; j < i; j++) {
+ msix = &device->msix_entries[j];
ioat_chan =
ioat_lookup_chan_by_index(device, j);
- free_irq(device->msix_entries[j].vector,
- ioat_chan);
+ devm_free_irq(dev, msix->vector, ioat_chan);
}
goto msix_single_vector;
}
}
intrctrl |= IOAT_INTRCTRL_MSIX_VECTOR_CONTROL;
- device->irq_mode = msix_multi_vector;
goto done;
msix_single_vector:
- device->msix_entries[0].entry = 0;
- err = pci_enable_msix(device->pdev, device->msix_entries, 1);
+ msix = &device->msix_entries[0];
+ msix->entry = 0;
+ err = pci_enable_msix(pdev, device->msix_entries, 1);
if (err)
goto msi;
- err = request_irq(device->msix_entries[0].vector, ioat_dma_do_interrupt,
- 0, "ioat-msix", device);
+ err = devm_request_irq(dev, msix->vector, ioat_dma_do_interrupt, 0,
+ "ioat-msix", device);
if (err) {
- pci_disable_msix(device->pdev);
+ pci_disable_msix(pdev);
goto msi;
}
- device->irq_mode = msix_single_vector;
goto done;
msi:
- err = pci_enable_msi(device->pdev);
+ err = pci_enable_msi(pdev);
if (err)
goto intx;
- err = request_irq(device->pdev->irq, ioat_dma_do_interrupt,
- 0, "ioat-msi", device);
+ err = devm_request_irq(dev, pdev->irq, ioat_dma_do_interrupt, 0,
+ "ioat-msi", device);
if (err) {
- pci_disable_msi(device->pdev);
+ pci_disable_msi(pdev);
goto intx;
}
/*
@@ -1527,21 +1531,17 @@ msi:
*/
if (device->version == IOAT_VER_1_2) {
u32 dmactrl;
- pci_read_config_dword(device->pdev,
- IOAT_PCI_DMACTRL_OFFSET, &dmactrl);
+ pci_read_config_dword(pdev, IOAT_PCI_DMACTRL_OFFSET, &dmactrl);
dmactrl |= IOAT_PCI_DMACTRL_MSI_EN;
- pci_write_config_dword(device->pdev,
- IOAT_PCI_DMACTRL_OFFSET, dmactrl);
+ pci_write_config_dword(pdev, IOAT_PCI_DMACTRL_OFFSET, dmactrl);
}
- device->irq_mode = msi;
goto done;
intx:
- err = request_irq(device->pdev->irq, ioat_dma_do_interrupt,
- IRQF_SHARED, "ioat-intx", device);
+ err = devm_request_irq(dev, pdev->irq, ioat_dma_do_interrupt,
+ IRQF_SHARED, "ioat-intx", device);
if (err)
goto err_no_irq;
- device->irq_mode = intx;
done:
intrctrl |= IOAT_INTRCTRL_MASTER_INT_EN;
@@ -1551,60 +1551,26 @@ done:
err_no_irq:
/* Disable all interrupt generation */
writeb(0, device->reg_base + IOAT_INTRCTRL_OFFSET);
- dev_err(&device->pdev->dev, "no usable interrupts\n");
- device->irq_mode = none;
- return -1;
+ dev_err(dev, "no usable interrupts\n");
+ return err;
}
-/**
- * ioat_dma_remove_interrupts - remove whatever interrupts were set
- * @device: ioat device
- */
-static void ioat_dma_remove_interrupts(struct ioatdma_device *device)
+static void ioat_disable_interrupts(struct ioatdma_device *device)
{
- struct ioat_dma_chan *ioat_chan;
- int i;
-
/* Disable all interrupt generation */
writeb(0, device->reg_base + IOAT_INTRCTRL_OFFSET);
-
- switch (device->irq_mode) {
- case msix_multi_vector:
- for (i = 0; i < device->common.chancnt; i++) {
- ioat_chan = ioat_lookup_chan_by_index(device, i);
- free_irq(device->msix_entries[i].vector, ioat_chan);
- }
- pci_disable_msix(device->pdev);
- break;
- case msix_single_vector:
- free_irq(device->msix_entries[0].vector, device);
- pci_disable_msix(device->pdev);
- break;
- case msi:
- free_irq(device->pdev->irq, device);
- pci_disable_msi(device->pdev);
- break;
- case intx:
- free_irq(device->pdev->irq, device);
- break;
- case none:
- dev_warn(&device->pdev->dev,
- "call to %s without interrupts setup\n", __func__);
- }
- device->irq_mode = none;
}
struct ioatdma_device *ioat_dma_probe(struct pci_dev *pdev,
void __iomem *iobase)
{
int err;
+ struct device *dev = &pdev->dev;
struct ioatdma_device *device;
- device = kzalloc(sizeof(*device), GFP_KERNEL);
- if (!device) {
+ device = devm_kzalloc(dev, sizeof(*device), GFP_KERNEL);
+ if (!device)
err = -ENOMEM;
- goto err_kzalloc;
- }
device->pdev = pdev;
device->reg_base = iobase;
device->version = readb(device->reg_base + IOAT_VER_OFFSET);
@@ -1651,14 +1617,12 @@ struct ioatdma_device *ioat_dma_probe(struct pci_dev *pdev,
break;
}
- dev_err(&device->pdev->dev,
- "Intel(R) I/OAT DMA Engine found,"
+ dev_err(dev, "Intel(R) I/OAT DMA Engine found,"
" %d channels, device version 0x%02x, driver version %s\n",
device->common.chancnt, device->version, IOAT_DMA_VERSION);
if (!device->common.chancnt) {
- dev_err(&device->pdev->dev,
- "Intel(R) I/OAT DMA Engine problem found: "
+ dev_err(dev, "Intel(R) I/OAT DMA Engine problem found: "
"zero channels detected\n");
goto err_setup_interrupts;
}
@@ -1671,9 +1635,11 @@ struct ioatdma_device *ioat_dma_probe(struct pci_dev *pdev,
if (err)
goto err_self_test;
- ioat_set_tcp_copy_break(device);
+ err = dma_async_device_register(&device->common);
+ if (err)
+ goto err_self_test;
- dma_async_device_register(&device->common);
+ ioat_set_tcp_copy_break(device);
if (device->version != IOAT_VER_3_0) {
INIT_DELAYED_WORK(&device->work, ioat_dma_chan_watchdog);
@@ -1684,16 +1650,12 @@ struct ioatdma_device *ioat_dma_probe(struct pci_dev *pdev,
return device;
err_self_test:
- ioat_dma_remove_interrupts(device);
+ ioat_disable_interrupts(device);
err_setup_interrupts:
pci_pool_destroy(device->completion_pool);
err_completion_pool:
pci_pool_destroy(device->dma_pool);
err_dma_pool:
- kfree(device);
-err_kzalloc:
- dev_err(&pdev->dev,
- "Intel(R) I/OAT DMA Engine initialization failed\n");
return NULL;
}
@@ -1705,23 +1667,17 @@ void ioat_dma_remove(struct ioatdma_device *device)
if (device->version != IOAT_VER_3_0)
cancel_delayed_work(&device->work);
- ioat_dma_remove_interrupts(device);
+ ioat_disable_interrupts(device);
dma_async_device_unregister(&device->common);
pci_pool_destroy(device->dma_pool);
pci_pool_destroy(device->completion_pool);
- iounmap(device->reg_base);
- pci_release_regions(device->pdev);
- pci_disable_device(device->pdev);
-
list_for_each_entry_safe(chan, _chan,
&device->common.channels, device_node) {
ioat_chan = to_ioat_chan(chan);
list_del(&chan->device_node);
- kfree(ioat_chan);
}
- kfree(device);
}
diff --git a/drivers/dma/ioat/dma.h b/drivers/dma/ioat/dma.h
index ccb400f..5e8d7cf 100644
--- a/drivers/dma/ioat/dma.h
+++ b/drivers/dma/ioat/dma.h
@@ -31,14 +31,6 @@
#define IOAT_DMA_VERSION "3.64"
-enum ioat_interrupt {
- none = 0,
- msix_multi_vector = 1,
- msix_single_vector = 2,
- msi = 3,
- intx = 4,
-};
-
#define IOAT_LOW_COMPLETION_MASK 0xffffffc0
#define IOAT_DMA_DCA_ANY_CPU ~0
#define IOAT_WATCHDOG_PERIOD (2 * HZ)
@@ -59,7 +51,6 @@ enum ioat_interrupt {
*/
#define NULL_DESC_BUFFER_SIZE 1
-
/**
* struct ioatdma_device - internal representation of a IOAT device
* @pdev: PCI-Express device
@@ -67,7 +58,6 @@ enum ioat_interrupt {
* @dma_pool: for allocating DMA descriptors
* @common: embedded struct dma_device
* @version: version of ioatdma device
- * @irq_mode: which style irq to use
* @msix_entries: irq handlers
* @idx: per channel data
*/
@@ -79,7 +69,6 @@ struct ioatdma_device {
struct pci_pool *completion_pool;
struct dma_device common;
u8 version;
- enum ioat_interrupt irq_mode;
struct delayed_work work;
struct msix_entry msix_entries[4];
struct ioat_dma_chan *idx[4];
diff --git a/drivers/dma/ioat/hw.h b/drivers/dma/ioat/hw.h
index afa57ee..1438fa5 100644
--- a/drivers/dma/ioat/hw.h
+++ b/drivers/dma/ioat/hw.h
@@ -23,6 +23,7 @@
/* PCI Configuration Space Values */
#define IOAT_PCI_VID 0x8086
+#define IOAT_MMIO_BAR 0
/* CB device ID's */
#define IOAT_PCI_DID_5000 0x1A38
diff --git a/drivers/dma/ioat/pci.c b/drivers/dma/ioat/pci.c
index d7948bf..982e38f 100644
--- a/drivers/dma/ioat/pci.c
+++ b/drivers/dma/ioat/pci.c
@@ -62,7 +62,6 @@ static struct pci_device_id ioat_pci_tbl[] = {
struct ioat_device {
struct pci_dev *pdev;
- void __iomem *iobase;
struct ioatdma_device *dma;
struct dca_provider *dca;
};
@@ -75,8 +74,10 @@ static int ioat_dca_enabled = 1;
module_param(ioat_dca_enabled, int, 0644);
MODULE_PARM_DESC(ioat_dca_enabled, "control support of dca service (default: 1)");
+#define DRV_NAME "ioatdma"
+
static struct pci_driver ioat_pci_driver = {
- .name = "ioatdma",
+ .name = DRV_NAME,
.id_table = ioat_pci_tbl,
.probe = ioat_probe,
.remove = __devexit_p(ioat_remove),
@@ -85,47 +86,42 @@ static struct pci_driver ioat_pci_driver = {
static int __devinit ioat_probe(struct pci_dev *pdev,
const struct pci_device_id *id)
{
+ void __iomem * const *iomap;
void __iomem *iobase;
+ struct device *dev = &pdev->dev;
struct ioat_device *device;
- unsigned long mmio_start, mmio_len;
int err;
- err = pci_enable_device(pdev);
+ err = pcim_enable_device(pdev);
if (err)
- goto err_enable_device;
+ return err;
- err = pci_request_regions(pdev, ioat_pci_driver.name);
+ err = pcim_iomap_regions(pdev, 1 << IOAT_MMIO_BAR, DRV_NAME);
if (err)
- goto err_request_regions;
+ return err;
+ iomap = pcim_iomap_table(pdev);
+ if (!iomap)
+ return -ENOMEM;
err = pci_set_dma_mask(pdev, DMA_BIT_MASK(64));
if (err)
err = pci_set_dma_mask(pdev, DMA_BIT_MASK(32));
if (err)
- goto err_set_dma_mask;
+ return err;
err = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64));
if (err)
err = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(32));
if (err)
- goto err_set_dma_mask;
-
- mmio_start = pci_resource_start(pdev, 0);
- mmio_len = pci_resource_len(pdev, 0);
- iobase = ioremap(mmio_start, mmio_len);
- if (!iobase) {
- err = -ENOMEM;
- goto err_ioremap;
- }
+ return err;
+
+ device = devm_kzalloc(dev, sizeof(*device), GFP_KERNEL);
+ if (!device)
+ return -ENOMEM;
- device = kzalloc(sizeof(*device), GFP_KERNEL);
- if (!device) {
- err = -ENOMEM;
- goto err_kzalloc;
- }
device->pdev = pdev;
pci_set_drvdata(pdev, device);
- device->iobase = iobase;
+ iobase = iomap[IOAT_MMIO_BAR];
pci_set_master(pdev);
@@ -146,28 +142,15 @@ static int __devinit ioat_probe(struct pci_dev *pdev,
device->dca = ioat3_dca_init(pdev, iobase);
break;
default:
- err = -ENODEV;
- break;
+ return -ENODEV;
}
- if (!device->dma)
- err = -ENODEV;
- if (err)
- goto err_version;
+ if (!device->dma) {
+ dev_err(dev, "Intel(R) I/OAT DMA Engine init failed\n");
+ return -ENODEV;
+ }
return 0;
-
-err_version:
- kfree(device);
-err_kzalloc:
- iounmap(iobase);
-err_ioremap:
-err_set_dma_mask:
- pci_release_regions(pdev);
- pci_disable_device(pdev);
-err_request_regions:
-err_enable_device:
- return err;
}
static void __devexit ioat_remove(struct pci_dev *pdev)
@@ -185,8 +168,6 @@ static void __devexit ioat_remove(struct pci_dev *pdev)
ioat_dma_remove(device->dma);
device->dma = NULL;
}
-
- kfree(device);
}
static int __init ioat_init_module(void)
^ permalink raw reply related
* [PATCH 04/29] ioat: cleanup some long deref chains and 80 column collisions
From: Dan Williams @ 2009-09-04 2:30 UTC (permalink / raw)
To: linux-kernel; +Cc: linux-raid, netdev, Maciej Sosnowski
In-Reply-To: <20090904022733.32667.77626.stgit@dwillia2-linux.ch.intel.com>
* reduce device->common. to dma-> in ioat_dma_{probe,remove,selftest}
* ioat_lookup_chan_by_index to ioat_chan_by_index
* multi-line function definitions
* ioat_desc_sw.async_tx to ioat_desc_sw.txd
* desc->txd. to tx-> in cleanup routine
Signed-off-by: Maciej Sosnowski <maciej.sosnowski@intel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
drivers/dma/ioat/dma.c | 304 ++++++++++++++++++++++--------------------------
drivers/dma/ioat/dma.h | 7 +
2 files changed, 144 insertions(+), 167 deletions(-)
diff --git a/drivers/dma/ioat/dma.c b/drivers/dma/ioat/dma.c
index 65f8b74..462dae6 100644
--- a/drivers/dma/ioat/dma.c
+++ b/drivers/dma/ioat/dma.c
@@ -55,9 +55,8 @@ ioat1_dma_get_next_descriptor(struct ioat_dma_chan *ioat_chan);
static struct ioat_desc_sw *
ioat2_dma_get_next_descriptor(struct ioat_dma_chan *ioat_chan);
-static inline struct ioat_dma_chan *ioat_lookup_chan_by_index(
- struct ioatdma_device *device,
- int index)
+static inline struct ioat_dma_chan *
+ioat_chan_by_index(struct ioatdma_device *device, int index)
{
return device->idx[index];
}
@@ -87,7 +86,7 @@ static irqreturn_t ioat_dma_do_interrupt(int irq, void *data)
attnstatus = readl(instance->reg_base + IOAT_ATTNSTATUS_OFFSET);
for_each_bit(bit, &attnstatus, BITS_PER_LONG) {
- ioat_chan = ioat_lookup_chan_by_index(instance, bit);
+ ioat_chan = ioat_chan_by_index(instance, bit);
tasklet_schedule(&ioat_chan->cleanup_task);
}
@@ -205,8 +204,8 @@ static int ioat_dma_enumerate_channels(struct ioatdma_device *device)
* descriptors to hw
* @chan: DMA channel handle
*/
-static inline void __ioat1_dma_memcpy_issue_pending(
- struct ioat_dma_chan *ioat_chan)
+static inline void
+__ioat1_dma_memcpy_issue_pending(struct ioat_dma_chan *ioat_chan)
{
ioat_chan->pending = 0;
writeb(IOAT_CHANCMD_APPEND, ioat_chan->reg_base + IOAT1_CHANCMD_OFFSET);
@@ -223,8 +222,8 @@ static void ioat1_dma_memcpy_issue_pending(struct dma_chan *chan)
}
}
-static inline void __ioat2_dma_memcpy_issue_pending(
- struct ioat_dma_chan *ioat_chan)
+static inline void
+__ioat2_dma_memcpy_issue_pending(struct ioat_dma_chan *ioat_chan)
{
ioat_chan->pending = 0;
writew(ioat_chan->dmacount,
@@ -279,18 +278,18 @@ static void ioat_dma_chan_reset_part2(struct work_struct *work)
desc = to_ioat_desc(ioat_chan->used_desc.prev);
switch (ioat_chan->device->version) {
case IOAT_VER_1_2:
- writel(((u64) desc->async_tx.phys) & 0x00000000FFFFFFFF,
+ writel(((u64) desc->txd.phys) & 0x00000000FFFFFFFF,
ioat_chan->reg_base + IOAT1_CHAINADDR_OFFSET_LOW);
- writel(((u64) desc->async_tx.phys) >> 32,
+ writel(((u64) desc->txd.phys) >> 32,
ioat_chan->reg_base + IOAT1_CHAINADDR_OFFSET_HIGH);
writeb(IOAT_CHANCMD_START, ioat_chan->reg_base
+ IOAT_CHANCMD_OFFSET(ioat_chan->device->version));
break;
case IOAT_VER_2_0:
- writel(((u64) desc->async_tx.phys) & 0x00000000FFFFFFFF,
+ writel(((u64) desc->txd.phys) & 0x00000000FFFFFFFF,
ioat_chan->reg_base + IOAT2_CHAINADDR_OFFSET_LOW);
- writel(((u64) desc->async_tx.phys) >> 32,
+ writel(((u64) desc->txd.phys) >> 32,
ioat_chan->reg_base + IOAT2_CHAINADDR_OFFSET_HIGH);
/* tell the engine to go with what's left to be done */
@@ -299,7 +298,7 @@ static void ioat_dma_chan_reset_part2(struct work_struct *work)
break;
}
- dev_err(&ioat_chan->device->pdev->dev,
+ dev_err(to_dev(ioat_chan),
"chan%d reset - %d descs waiting, %d total desc\n",
chan_num(ioat_chan), ioat_chan->dmacount, ioat_chan->desccount);
@@ -322,7 +321,7 @@ static void ioat_dma_reset_channel(struct ioat_dma_chan *ioat_chan)
chansts = (ioat_chan->completion_virt->low
& IOAT_CHANSTS_DMA_TRANSFER_STATUS);
if (chanerr) {
- dev_err(&ioat_chan->device->pdev->dev,
+ dev_err(to_dev(ioat_chan),
"chan%d, CHANSTS = 0x%08x CHANERR = 0x%04x, clearing\n",
chan_num(ioat_chan), chansts, chanerr);
writel(chanerr, ioat_chan->reg_base + IOAT_CHANERR_OFFSET);
@@ -367,7 +366,7 @@ static void ioat_dma_chan_watchdog(struct work_struct *work)
unsigned long compl_desc_addr_hw;
for (i = 0; i < device->common.chancnt; i++) {
- ioat_chan = ioat_lookup_chan_by_index(device, i);
+ ioat_chan = ioat_chan_by_index(device, i);
if (ioat_chan->device->version == IOAT_VER_1_2
/* have we started processing anything yet */
@@ -475,7 +474,7 @@ static dma_cookie_t ioat1_tx_submit(struct dma_async_tx_descriptor *tx)
len = first->len;
src = first->src;
dst = first->dst;
- orig_flags = first->async_tx.flags;
+ orig_flags = first->txd.flags;
new = first;
spin_lock_bh(&ioat_chan->desc_lock);
@@ -484,7 +483,7 @@ static dma_cookie_t ioat1_tx_submit(struct dma_async_tx_descriptor *tx)
do {
copy = min_t(size_t, len, ioat_chan->xfercap);
- async_tx_ack(&new->async_tx);
+ async_tx_ack(&new->txd);
hw = new->hw;
hw->size = copy;
@@ -495,7 +494,7 @@ static dma_cookie_t ioat1_tx_submit(struct dma_async_tx_descriptor *tx)
/* chain together the physical address list for the HW */
wmb();
- prev->hw->next = (u64) new->async_tx.phys;
+ prev->hw->next = (u64) new->txd.phys;
len -= copy;
dst += copy;
@@ -507,27 +506,26 @@ static dma_cookie_t ioat1_tx_submit(struct dma_async_tx_descriptor *tx)
} while (len && (new = ioat1_dma_get_next_descriptor(ioat_chan)));
if (!new) {
- dev_err(&ioat_chan->device->pdev->dev,
- "tx submit failed\n");
+ dev_err(to_dev(ioat_chan), "tx submit failed\n");
spin_unlock_bh(&ioat_chan->desc_lock);
return -ENOMEM;
}
hw->ctl = IOAT_DMA_DESCRIPTOR_CTL_CP_STS;
- if (first->async_tx.callback) {
+ if (first->txd.callback) {
hw->ctl |= IOAT_DMA_DESCRIPTOR_CTL_INT_GN;
if (first != new) {
/* move callback into to last desc */
- new->async_tx.callback = first->async_tx.callback;
- new->async_tx.callback_param
- = first->async_tx.callback_param;
- first->async_tx.callback = NULL;
- first->async_tx.callback_param = NULL;
+ new->txd.callback = first->txd.callback;
+ new->txd.callback_param
+ = first->txd.callback_param;
+ first->txd.callback = NULL;
+ first->txd.callback_param = NULL;
}
}
new->tx_cnt = desc_count;
- new->async_tx.flags = orig_flags; /* client is in control of this ack */
+ new->txd.flags = orig_flags; /* client is in control of this ack */
/* store the original values for use in later cleanup */
if (new != first) {
@@ -541,11 +539,11 @@ static dma_cookie_t ioat1_tx_submit(struct dma_async_tx_descriptor *tx)
cookie++;
if (cookie < 0)
cookie = 1;
- ioat_chan->common.cookie = new->async_tx.cookie = cookie;
+ ioat_chan->common.cookie = new->txd.cookie = cookie;
/* write address into NextDescriptor field of last desc in chain */
to_ioat_desc(ioat_chan->used_desc.prev)->hw->next =
- first->async_tx.phys;
+ first->txd.phys;
list_splice_tail(&new_chain, &ioat_chan->used_desc);
ioat_chan->dmacount += desc_count;
@@ -574,7 +572,7 @@ static dma_cookie_t ioat2_tx_submit(struct dma_async_tx_descriptor *tx)
len = first->len;
src = first->src;
dst = first->dst;
- orig_flags = first->async_tx.flags;
+ orig_flags = first->txd.flags;
new = first;
/*
@@ -584,7 +582,7 @@ static dma_cookie_t ioat2_tx_submit(struct dma_async_tx_descriptor *tx)
do {
copy = min_t(size_t, len, ioat_chan->xfercap);
- async_tx_ack(&new->async_tx);
+ async_tx_ack(&new->txd);
hw = new->hw;
hw->size = copy;
@@ -599,27 +597,26 @@ static dma_cookie_t ioat2_tx_submit(struct dma_async_tx_descriptor *tx)
} while (len && (new = ioat2_dma_get_next_descriptor(ioat_chan)));
if (!new) {
- dev_err(&ioat_chan->device->pdev->dev,
- "tx submit failed\n");
+ dev_err(to_dev(ioat_chan), "tx submit failed\n");
spin_unlock_bh(&ioat_chan->desc_lock);
return -ENOMEM;
}
hw->ctl |= IOAT_DMA_DESCRIPTOR_CTL_CP_STS;
- if (first->async_tx.callback) {
+ if (first->txd.callback) {
hw->ctl |= IOAT_DMA_DESCRIPTOR_CTL_INT_GN;
if (first != new) {
/* move callback into to last desc */
- new->async_tx.callback = first->async_tx.callback;
- new->async_tx.callback_param
- = first->async_tx.callback_param;
- first->async_tx.callback = NULL;
- first->async_tx.callback_param = NULL;
+ new->txd.callback = first->txd.callback;
+ new->txd.callback_param
+ = first->txd.callback_param;
+ first->txd.callback = NULL;
+ first->txd.callback_param = NULL;
}
}
new->tx_cnt = desc_count;
- new->async_tx.flags = orig_flags; /* client is in control of this ack */
+ new->txd.flags = orig_flags; /* client is in control of this ack */
/* store the original values for use in later cleanup */
if (new != first) {
@@ -633,7 +630,7 @@ static dma_cookie_t ioat2_tx_submit(struct dma_async_tx_descriptor *tx)
cookie++;
if (cookie < 0)
cookie = 1;
- ioat_chan->common.cookie = new->async_tx.cookie = cookie;
+ ioat_chan->common.cookie = new->txd.cookie = cookie;
ioat_chan->dmacount += desc_count;
ioat_chan->pending += desc_count;
@@ -649,9 +646,8 @@ static dma_cookie_t ioat2_tx_submit(struct dma_async_tx_descriptor *tx)
* @ioat_chan: the channel supplying the memory pool for the descriptors
* @flags: allocation flags
*/
-static struct ioat_desc_sw *ioat_dma_alloc_descriptor(
- struct ioat_dma_chan *ioat_chan,
- gfp_t flags)
+static struct ioat_desc_sw *
+ioat_dma_alloc_descriptor(struct ioat_dma_chan *ioat_chan, gfp_t flags)
{
struct ioat_dma_descriptor *desc;
struct ioat_desc_sw *desc_sw;
@@ -670,19 +666,19 @@ static struct ioat_desc_sw *ioat_dma_alloc_descriptor(
}
memset(desc, 0, sizeof(*desc));
- dma_async_tx_descriptor_init(&desc_sw->async_tx, &ioat_chan->common);
+ dma_async_tx_descriptor_init(&desc_sw->txd, &ioat_chan->common);
switch (ioat_chan->device->version) {
case IOAT_VER_1_2:
- desc_sw->async_tx.tx_submit = ioat1_tx_submit;
+ desc_sw->txd.tx_submit = ioat1_tx_submit;
break;
case IOAT_VER_2_0:
case IOAT_VER_3_0:
- desc_sw->async_tx.tx_submit = ioat2_tx_submit;
+ desc_sw->txd.tx_submit = ioat2_tx_submit;
break;
}
desc_sw->hw = desc;
- desc_sw->async_tx.phys = phys;
+ desc_sw->txd.phys = phys;
return desc_sw;
}
@@ -712,9 +708,9 @@ static void ioat2_dma_massage_chan_desc(struct ioat_dma_chan *ioat_chan)
/* circle link the hw descriptors */
desc = to_ioat_desc(ioat_chan->free_desc.next);
- desc->hw->next = to_ioat_desc(desc->node.next)->async_tx.phys;
+ desc->hw->next = to_ioat_desc(desc->node.next)->txd.phys;
list_for_each_entry_safe(desc, _desc, ioat_chan->free_desc.next, node) {
- desc->hw->next = to_ioat_desc(desc->node.next)->async_tx.phys;
+ desc->hw->next = to_ioat_desc(desc->node.next)->txd.phys;
}
}
@@ -743,8 +739,7 @@ static int ioat_dma_alloc_chan_resources(struct dma_chan *chan)
chanerr = readl(ioat_chan->reg_base + IOAT_CHANERR_OFFSET);
if (chanerr) {
- dev_err(&ioat_chan->device->pdev->dev,
- "CHANERR = %x, clearing\n", chanerr);
+ dev_err(to_dev(ioat_chan), "CHANERR = %x, clearing\n", chanerr);
writel(chanerr, ioat_chan->reg_base + IOAT_CHANERR_OFFSET);
}
@@ -752,7 +747,7 @@ static int ioat_dma_alloc_chan_resources(struct dma_chan *chan)
for (i = 0; i < ioat_initial_desc_count; i++) {
desc = ioat_dma_alloc_descriptor(ioat_chan, GFP_KERNEL);
if (!desc) {
- dev_err(&ioat_chan->device->pdev->dev,
+ dev_err(to_dev(ioat_chan),
"Only %d initial descriptors\n", i);
break;
}
@@ -819,14 +814,14 @@ static void ioat_dma_free_chan_resources(struct dma_chan *chan)
in_use_descs++;
list_del(&desc->node);
pci_pool_free(ioatdma_device->dma_pool, desc->hw,
- desc->async_tx.phys);
+ desc->txd.phys);
kfree(desc);
}
list_for_each_entry_safe(desc, _desc,
&ioat_chan->free_desc, node) {
list_del(&desc->node);
pci_pool_free(ioatdma_device->dma_pool, desc->hw,
- desc->async_tx.phys);
+ desc->txd.phys);
kfree(desc);
}
break;
@@ -836,12 +831,12 @@ static void ioat_dma_free_chan_resources(struct dma_chan *chan)
ioat_chan->free_desc.next, node) {
list_del(&desc->node);
pci_pool_free(ioatdma_device->dma_pool, desc->hw,
- desc->async_tx.phys);
+ desc->txd.phys);
kfree(desc);
}
desc = to_ioat_desc(ioat_chan->free_desc.next);
pci_pool_free(ioatdma_device->dma_pool, desc->hw,
- desc->async_tx.phys);
+ desc->txd.phys);
kfree(desc);
INIT_LIST_HEAD(&ioat_chan->free_desc);
INIT_LIST_HEAD(&ioat_chan->used_desc);
@@ -855,8 +850,7 @@ static void ioat_dma_free_chan_resources(struct dma_chan *chan)
/* one is ok since we left it on there on purpose */
if (in_use_descs > 1)
- dev_err(&ioat_chan->device->pdev->dev,
- "Freeing %d in use descriptors!\n",
+ dev_err(to_dev(ioat_chan), "Freeing %d in use descriptors!\n",
in_use_descs - 1);
ioat_chan->last_completion = ioat_chan->completion_addr = 0;
@@ -889,8 +883,7 @@ ioat1_dma_get_next_descriptor(struct ioat_dma_chan *ioat_chan)
/* try to get another desc */
new = ioat_dma_alloc_descriptor(ioat_chan, GFP_ATOMIC);
if (!new) {
- dev_err(&ioat_chan->device->pdev->dev,
- "alloc failed\n");
+ dev_err(to_dev(ioat_chan), "alloc failed\n");
return NULL;
}
}
@@ -936,16 +929,15 @@ ioat2_dma_get_next_descriptor(struct ioat_dma_chan *ioat_chan)
for (i = 16; i; i--) {
desc = ioat_dma_alloc_descriptor(ioat_chan, GFP_ATOMIC);
if (!desc) {
- dev_err(&ioat_chan->device->pdev->dev,
- "alloc failed\n");
+ dev_err(to_dev(ioat_chan), "alloc failed\n");
break;
}
list_add_tail(&desc->node, ioat_chan->used_desc.next);
desc->hw->next
- = to_ioat_desc(desc->node.next)->async_tx.phys;
+ = to_ioat_desc(desc->node.next)->txd.phys;
to_ioat_desc(desc->node.prev)->hw->next
- = desc->async_tx.phys;
+ = desc->txd.phys;
ioat_chan->desccount++;
}
@@ -962,8 +954,8 @@ ioat2_dma_get_next_descriptor(struct ioat_dma_chan *ioat_chan)
return new;
}
-static struct ioat_desc_sw *ioat_dma_get_next_descriptor(
- struct ioat_dma_chan *ioat_chan)
+static struct ioat_desc_sw *
+ioat_dma_get_next_descriptor(struct ioat_dma_chan *ioat_chan)
{
if (!ioat_chan)
return NULL;
@@ -978,12 +970,9 @@ static struct ioat_desc_sw *ioat_dma_get_next_descriptor(
return NULL;
}
-static struct dma_async_tx_descriptor *ioat1_dma_prep_memcpy(
- struct dma_chan *chan,
- dma_addr_t dma_dest,
- dma_addr_t dma_src,
- size_t len,
- unsigned long flags)
+static struct dma_async_tx_descriptor *
+ioat1_dma_prep_memcpy(struct dma_chan *chan, dma_addr_t dma_dest,
+ dma_addr_t dma_src, size_t len, unsigned long flags)
{
struct ioat_dma_chan *ioat_chan = to_ioat_chan(chan);
struct ioat_desc_sw *new;
@@ -996,22 +985,19 @@ static struct dma_async_tx_descriptor *ioat1_dma_prep_memcpy(
new->len = len;
new->dst = dma_dest;
new->src = dma_src;
- new->async_tx.flags = flags;
- return &new->async_tx;
+ new->txd.flags = flags;
+ return &new->txd;
} else {
- dev_err(&ioat_chan->device->pdev->dev,
+ dev_err(to_dev(ioat_chan),
"chan%d - get_next_desc failed: %d descs waiting, %d total desc\n",
chan_num(ioat_chan), ioat_chan->dmacount, ioat_chan->desccount);
return NULL;
}
}
-static struct dma_async_tx_descriptor *ioat2_dma_prep_memcpy(
- struct dma_chan *chan,
- dma_addr_t dma_dest,
- dma_addr_t dma_src,
- size_t len,
- unsigned long flags)
+static struct dma_async_tx_descriptor *
+ioat2_dma_prep_memcpy(struct dma_chan *chan, dma_addr_t dma_dest,
+ dma_addr_t dma_src, size_t len, unsigned long flags)
{
struct ioat_dma_chan *ioat_chan = to_ioat_chan(chan);
struct ioat_desc_sw *new;
@@ -1028,11 +1014,11 @@ static struct dma_async_tx_descriptor *ioat2_dma_prep_memcpy(
new->len = len;
new->dst = dma_dest;
new->src = dma_src;
- new->async_tx.flags = flags;
- return &new->async_tx;
+ new->txd.flags = flags;
+ return &new->txd;
} else {
spin_unlock_bh(&ioat_chan->desc_lock);
- dev_err(&ioat_chan->device->pdev->dev,
+ dev_err(to_dev(ioat_chan),
"chan%d - get_next_desc failed: %d descs waiting, %d total desc\n",
chan_num(ioat_chan), ioat_chan->dmacount, ioat_chan->desccount);
return NULL;
@@ -1050,8 +1036,8 @@ static void ioat_dma_cleanup_tasklet(unsigned long data)
static void
ioat_dma_unmap(struct ioat_dma_chan *ioat_chan, struct ioat_desc_sw *desc)
{
- if (!(desc->async_tx.flags & DMA_COMPL_SKIP_DEST_UNMAP)) {
- if (desc->async_tx.flags & DMA_COMPL_DEST_UNMAP_SINGLE)
+ if (!(desc->txd.flags & DMA_COMPL_SKIP_DEST_UNMAP)) {
+ if (desc->txd.flags & DMA_COMPL_DEST_UNMAP_SINGLE)
pci_unmap_single(ioat_chan->device->pdev,
pci_unmap_addr(desc, dst),
pci_unmap_len(desc, len),
@@ -1063,8 +1049,8 @@ ioat_dma_unmap(struct ioat_dma_chan *ioat_chan, struct ioat_desc_sw *desc)
PCI_DMA_FROMDEVICE);
}
- if (!(desc->async_tx.flags & DMA_COMPL_SKIP_SRC_UNMAP)) {
- if (desc->async_tx.flags & DMA_COMPL_SRC_UNMAP_SINGLE)
+ if (!(desc->txd.flags & DMA_COMPL_SKIP_SRC_UNMAP)) {
+ if (desc->txd.flags & DMA_COMPL_SRC_UNMAP_SINGLE)
pci_unmap_single(ioat_chan->device->pdev,
pci_unmap_addr(desc, src),
pci_unmap_len(desc, len),
@@ -1088,6 +1074,7 @@ static void ioat_dma_memcpy_cleanup(struct ioat_dma_chan *ioat_chan)
dma_cookie_t cookie = 0;
unsigned long desc_phys;
struct ioat_desc_sw *latest_desc;
+ struct dma_async_tx_descriptor *tx;
prefetch(ioat_chan->completion_virt);
@@ -1111,8 +1098,7 @@ static void ioat_dma_memcpy_cleanup(struct ioat_dma_chan *ioat_chan)
if ((ioat_chan->completion_virt->full
& IOAT_CHANSTS_DMA_TRANSFER_STATUS) ==
IOAT_CHANSTS_DMA_TRANSFER_STATUS_HALTED) {
- dev_err(&ioat_chan->device->pdev->dev,
- "Channel halted, chanerr = %x\n",
+ dev_err(to_dev(ioat_chan), "Channel halted, chanerr = %x\n",
readl(ioat_chan->reg_base + IOAT_CHANERR_OFFSET));
/* TODO do something to salvage the situation */
@@ -1145,38 +1131,38 @@ static void ioat_dma_memcpy_cleanup(struct ioat_dma_chan *ioat_chan)
case IOAT_VER_1_2:
list_for_each_entry_safe(desc, _desc,
&ioat_chan->used_desc, node) {
-
+ tx = &desc->txd;
/*
* Incoming DMA requests may use multiple descriptors,
* due to exceeding xfercap, perhaps. If so, only the
* last one will have a cookie, and require unmapping.
*/
- if (desc->async_tx.cookie) {
- cookie = desc->async_tx.cookie;
+ if (tx->cookie) {
+ cookie = tx->cookie;
ioat_dma_unmap(ioat_chan, desc);
- if (desc->async_tx.callback) {
- desc->async_tx.callback(desc->async_tx.callback_param);
- desc->async_tx.callback = NULL;
+ if (tx->callback) {
+ tx->callback(tx->callback_param);
+ tx->callback = NULL;
}
}
- if (desc->async_tx.phys != phys_complete) {
+ if (tx->phys != phys_complete) {
/*
* a completed entry, but not the last, so clean
* up if the client is done with the descriptor
*/
- if (async_tx_test_ack(&desc->async_tx)) {
+ if (async_tx_test_ack(tx)) {
list_move_tail(&desc->node,
&ioat_chan->free_desc);
} else
- desc->async_tx.cookie = 0;
+ tx->cookie = 0;
} else {
/*
* last used desc. Do not remove, so we can
* append from it, but don't look at it next
* time, either
*/
- desc->async_tx.cookie = 0;
+ tx->cookie = 0;
/* TODO check status bits? */
break;
@@ -1191,10 +1177,11 @@ static void ioat_dma_memcpy_cleanup(struct ioat_dma_chan *ioat_chan)
/* work backwards to find latest finished desc */
desc = to_ioat_desc(ioat_chan->used_desc.next);
+ tx = &desc->txd;
latest_desc = NULL;
do {
desc = to_ioat_desc(desc->node.prev);
- desc_phys = (unsigned long)desc->async_tx.phys
+ desc_phys = (unsigned long)tx->phys
& IOAT_CHANSTS_COMPLETED_DESCRIPTOR_ADDR;
if (desc_phys == phys_complete) {
latest_desc = desc;
@@ -1203,19 +1190,18 @@ static void ioat_dma_memcpy_cleanup(struct ioat_dma_chan *ioat_chan)
} while (&desc->node != ioat_chan->used_desc.prev);
if (latest_desc != NULL) {
-
/* work forwards to clear finished descriptors */
for (desc = to_ioat_desc(ioat_chan->used_desc.prev);
&desc->node != latest_desc->node.next &&
&desc->node != ioat_chan->used_desc.next;
desc = to_ioat_desc(desc->node.next)) {
- if (desc->async_tx.cookie) {
- cookie = desc->async_tx.cookie;
- desc->async_tx.cookie = 0;
+ if (tx->cookie) {
+ cookie = tx->cookie;
+ tx->cookie = 0;
ioat_dma_unmap(ioat_chan, desc);
- if (desc->async_tx.callback) {
- desc->async_tx.callback(desc->async_tx.callback_param);
- desc->async_tx.callback = NULL;
+ if (tx->callback) {
+ tx->callback(tx->callback_param);
+ tx->callback = NULL;
}
}
}
@@ -1245,10 +1231,9 @@ static void ioat_dma_memcpy_cleanup(struct ioat_dma_chan *ioat_chan)
* @done: if not %NULL, updated with last completed transaction
* @used: if not %NULL, updated with last used transaction
*/
-static enum dma_status ioat_dma_is_complete(struct dma_chan *chan,
- dma_cookie_t cookie,
- dma_cookie_t *done,
- dma_cookie_t *used)
+static enum dma_status
+ioat_dma_is_complete(struct dma_chan *chan, dma_cookie_t cookie,
+ dma_cookie_t *done, dma_cookie_t *used)
{
struct ioat_dma_chan *ioat_chan = to_ioat_chan(chan);
dma_cookie_t last_used;
@@ -1290,7 +1275,7 @@ static void ioat_dma_start_null_desc(struct ioat_dma_chan *ioat_chan)
desc = ioat_dma_get_next_descriptor(ioat_chan);
if (!desc) {
- dev_err(&ioat_chan->device->pdev->dev,
+ dev_err(to_dev(ioat_chan),
"Unable to start null desc - get next desc failed\n");
spin_unlock_bh(&ioat_chan->desc_lock);
return;
@@ -1303,15 +1288,15 @@ static void ioat_dma_start_null_desc(struct ioat_dma_chan *ioat_chan)
desc->hw->size = NULL_DESC_BUFFER_SIZE;
desc->hw->src_addr = 0;
desc->hw->dst_addr = 0;
- async_tx_ack(&desc->async_tx);
+ async_tx_ack(&desc->txd);
switch (ioat_chan->device->version) {
case IOAT_VER_1_2:
desc->hw->next = 0;
list_add_tail(&desc->node, &ioat_chan->used_desc);
- writel(((u64) desc->async_tx.phys) & 0x00000000FFFFFFFF,
+ writel(((u64) desc->txd.phys) & 0x00000000FFFFFFFF,
ioat_chan->reg_base + IOAT1_CHAINADDR_OFFSET_LOW);
- writel(((u64) desc->async_tx.phys) >> 32,
+ writel(((u64) desc->txd.phys) >> 32,
ioat_chan->reg_base + IOAT1_CHAINADDR_OFFSET_HIGH);
writeb(IOAT_CHANCMD_START, ioat_chan->reg_base
@@ -1319,9 +1304,9 @@ static void ioat_dma_start_null_desc(struct ioat_dma_chan *ioat_chan)
break;
case IOAT_VER_2_0:
case IOAT_VER_3_0:
- writel(((u64) desc->async_tx.phys) & 0x00000000FFFFFFFF,
+ writel(((u64) desc->txd.phys) & 0x00000000FFFFFFFF,
ioat_chan->reg_base + IOAT2_CHAINADDR_OFFSET_LOW);
- writel(((u64) desc->async_tx.phys) >> 32,
+ writel(((u64) desc->txd.phys) >> 32,
ioat_chan->reg_base + IOAT2_CHAINADDR_OFFSET_HIGH);
ioat_chan->dmacount++;
@@ -1352,6 +1337,8 @@ static int ioat_dma_self_test(struct ioatdma_device *device)
int i;
u8 *src;
u8 *dest;
+ struct dma_device *dma = &device->common;
+ struct device *dev = &device->pdev->dev;
struct dma_chan *dma_chan;
struct dma_async_tx_descriptor *tx;
dma_addr_t dma_dest, dma_src;
@@ -1375,26 +1362,21 @@ static int ioat_dma_self_test(struct ioatdma_device *device)
src[i] = (u8)i;
/* Start copy, using first DMA channel */
- dma_chan = container_of(device->common.channels.next,
- struct dma_chan,
+ dma_chan = container_of(dma->channels.next, struct dma_chan,
device_node);
- if (device->common.device_alloc_chan_resources(dma_chan) < 1) {
- dev_err(&device->pdev->dev,
- "selftest cannot allocate chan resource\n");
+ if (dma->device_alloc_chan_resources(dma_chan) < 1) {
+ dev_err(dev, "selftest cannot allocate chan resource\n");
err = -ENODEV;
goto out;
}
- dma_src = dma_map_single(dma_chan->device->dev, src, IOAT_TEST_SIZE,
- DMA_TO_DEVICE);
- dma_dest = dma_map_single(dma_chan->device->dev, dest, IOAT_TEST_SIZE,
- DMA_FROM_DEVICE);
+ dma_src = dma_map_single(dev, src, IOAT_TEST_SIZE, DMA_TO_DEVICE);
+ dma_dest = dma_map_single(dev, dest, IOAT_TEST_SIZE, DMA_FROM_DEVICE);
flags = DMA_COMPL_SRC_UNMAP_SINGLE | DMA_COMPL_DEST_UNMAP_SINGLE;
tx = device->common.device_prep_dma_memcpy(dma_chan, dma_dest, dma_src,
IOAT_TEST_SIZE, flags);
if (!tx) {
- dev_err(&device->pdev->dev,
- "Self-test prep failed, disabling\n");
+ dev_err(dev, "Self-test prep failed, disabling\n");
err = -ENODEV;
goto free_resources;
}
@@ -1405,32 +1387,29 @@ static int ioat_dma_self_test(struct ioatdma_device *device)
tx->callback_param = &cmp;
cookie = tx->tx_submit(tx);
if (cookie < 0) {
- dev_err(&device->pdev->dev,
- "Self-test setup failed, disabling\n");
+ dev_err(dev, "Self-test setup failed, disabling\n");
err = -ENODEV;
goto free_resources;
}
- device->common.device_issue_pending(dma_chan);
+ dma->device_issue_pending(dma_chan);
tmo = wait_for_completion_timeout(&cmp, msecs_to_jiffies(3000));
if (tmo == 0 ||
- device->common.device_is_tx_complete(dma_chan, cookie, NULL, NULL)
+ dma->device_is_tx_complete(dma_chan, cookie, NULL, NULL)
!= DMA_SUCCESS) {
- dev_err(&device->pdev->dev,
- "Self-test copy timed out, disabling\n");
+ dev_err(dev, "Self-test copy timed out, disabling\n");
err = -ENODEV;
goto free_resources;
}
if (memcmp(src, dest, IOAT_TEST_SIZE)) {
- dev_err(&device->pdev->dev,
- "Self-test copy failed compare, disabling\n");
+ dev_err(dev, "Self-test copy failed compare, disabling\n");
err = -ENODEV;
goto free_resources;
}
free_resources:
- device->common.device_free_chan_resources(dma_chan);
+ dma->device_free_chan_resources(dma_chan);
out:
kfree(src);
kfree(dest);
@@ -1483,15 +1462,14 @@ msix:
for (i = 0; i < msixcnt; i++) {
msix = &device->msix_entries[i];
- ioat_chan = ioat_lookup_chan_by_index(device, i);
+ ioat_chan = ioat_chan_by_index(device, i);
err = devm_request_irq(dev, msix->vector,
ioat_dma_do_interrupt_msix, 0,
"ioat-msix", ioat_chan);
if (err) {
for (j = 0; j < i; j++) {
msix = &device->msix_entries[j];
- ioat_chan =
- ioat_lookup_chan_by_index(device, j);
+ ioat_chan = ioat_chan_by_index(device, j);
devm_free_irq(dev, msix->vector, ioat_chan);
}
goto msix_single_vector;
@@ -1561,12 +1539,13 @@ static void ioat_disable_interrupts(struct ioatdma_device *device)
writeb(0, device->reg_base + IOAT_INTRCTRL_OFFSET);
}
-struct ioatdma_device *ioat_dma_probe(struct pci_dev *pdev,
- void __iomem *iobase)
+struct ioatdma_device *
+ioat_dma_probe(struct pci_dev *pdev, void __iomem *iobase)
{
int err;
struct device *dev = &pdev->dev;
struct ioatdma_device *device;
+ struct dma_device *dma;
device = devm_kzalloc(dev, sizeof(*device), GFP_KERNEL);
if (!device)
@@ -1574,6 +1553,7 @@ struct ioatdma_device *ioat_dma_probe(struct pci_dev *pdev,
device->pdev = pdev;
device->reg_base = iobase;
device->version = readb(device->reg_base + IOAT_VER_OFFSET);
+ dma = &device->common;
/* DMA coherent memory pool for DMA descriptor allocations */
device->dma_pool = pci_pool_create("dma_desc_pool", pdev,
@@ -1592,36 +1572,32 @@ struct ioatdma_device *ioat_dma_probe(struct pci_dev *pdev,
goto err_completion_pool;
}
- INIT_LIST_HEAD(&device->common.channels);
+ INIT_LIST_HEAD(&dma->channels);
ioat_dma_enumerate_channels(device);
- device->common.device_alloc_chan_resources =
- ioat_dma_alloc_chan_resources;
- device->common.device_free_chan_resources =
- ioat_dma_free_chan_resources;
- device->common.dev = &pdev->dev;
+ dma->device_alloc_chan_resources = ioat_dma_alloc_chan_resources;
+ dma->device_free_chan_resources = ioat_dma_free_chan_resources;
+ dma->dev = &pdev->dev;
- dma_cap_set(DMA_MEMCPY, device->common.cap_mask);
- device->common.device_is_tx_complete = ioat_dma_is_complete;
+ dma_cap_set(DMA_MEMCPY, dma->cap_mask);
+ dma->device_is_tx_complete = ioat_dma_is_complete;
switch (device->version) {
case IOAT_VER_1_2:
- device->common.device_prep_dma_memcpy = ioat1_dma_prep_memcpy;
- device->common.device_issue_pending =
- ioat1_dma_memcpy_issue_pending;
+ dma->device_prep_dma_memcpy = ioat1_dma_prep_memcpy;
+ dma->device_issue_pending = ioat1_dma_memcpy_issue_pending;
break;
case IOAT_VER_2_0:
case IOAT_VER_3_0:
- device->common.device_prep_dma_memcpy = ioat2_dma_prep_memcpy;
- device->common.device_issue_pending =
- ioat2_dma_memcpy_issue_pending;
+ dma->device_prep_dma_memcpy = ioat2_dma_prep_memcpy;
+ dma->device_issue_pending = ioat2_dma_memcpy_issue_pending;
break;
}
dev_err(dev, "Intel(R) I/OAT DMA Engine found,"
" %d channels, device version 0x%02x, driver version %s\n",
- device->common.chancnt, device->version, IOAT_DMA_VERSION);
+ dma->chancnt, device->version, IOAT_DMA_VERSION);
- if (!device->common.chancnt) {
+ if (!dma->chancnt) {
dev_err(dev, "Intel(R) I/OAT DMA Engine problem found: "
"zero channels detected\n");
goto err_setup_interrupts;
@@ -1635,7 +1611,7 @@ struct ioatdma_device *ioat_dma_probe(struct pci_dev *pdev,
if (err)
goto err_self_test;
- err = dma_async_device_register(&device->common);
+ err = dma_async_device_register(dma);
if (err)
goto err_self_test;
@@ -1663,19 +1639,19 @@ void ioat_dma_remove(struct ioatdma_device *device)
{
struct dma_chan *chan, *_chan;
struct ioat_dma_chan *ioat_chan;
+ struct dma_device *dma = &device->common;
if (device->version != IOAT_VER_3_0)
cancel_delayed_work(&device->work);
ioat_disable_interrupts(device);
- dma_async_device_unregister(&device->common);
+ dma_async_device_unregister(dma);
pci_pool_destroy(device->dma_pool);
pci_pool_destroy(device->completion_pool);
- list_for_each_entry_safe(chan, _chan,
- &device->common.channels, device_node) {
+ list_for_each_entry_safe(chan, _chan, &dma->channels, device_node) {
ioat_chan = to_ioat_chan(chan);
list_del(&chan->device_node);
}
diff --git a/drivers/dma/ioat/dma.h b/drivers/dma/ioat/dma.h
index 5e8d7cf..c5eabae 100644
--- a/drivers/dma/ioat/dma.h
+++ b/drivers/dma/ioat/dma.h
@@ -38,7 +38,8 @@
#define to_ioat_chan(chan) container_of(chan, struct ioat_dma_chan, common)
#define to_ioatdma_device(dev) container_of(dev, struct ioatdma_device, common)
#define to_ioat_desc(lh) container_of(lh, struct ioat_desc_sw, node)
-#define tx_to_ioat_desc(tx) container_of(tx, struct ioat_desc_sw, async_tx)
+#define tx_to_ioat_desc(tx) container_of(tx, struct ioat_desc_sw, txd)
+#define to_dev(ioat_chan) (&(ioat_chan)->device->pdev->dev)
#define chan_num(ch) ((int)((ch)->reg_base - (ch)->device->reg_base) / 0x80)
@@ -123,7 +124,7 @@ struct ioat_dma_chan {
* @node: this descriptor will either be on the free list,
* or attached to a transaction list (async_tx.tx_list)
* @tx_cnt: number of descriptors required to complete the transaction
- * @async_tx: the generic software descriptor for all engines
+ * @txd: the generic software descriptor for all engines
*/
struct ioat_desc_sw {
struct ioat_dma_descriptor *hw;
@@ -132,7 +133,7 @@ struct ioat_desc_sw {
size_t len;
dma_addr_t src;
dma_addr_t dst;
- struct dma_async_tx_descriptor async_tx;
+ struct dma_async_tx_descriptor txd;
};
static inline void ioat_set_tcp_copy_break(struct ioatdma_device *dev)
^ permalink raw reply related
* [PATCH 05/29] ioat: kill function prototype ifdef guards
From: Dan Williams @ 2009-09-04 2:30 UTC (permalink / raw)
To: linux-kernel; +Cc: linux-raid, netdev, Maciej Sosnowski
In-Reply-To: <20090904022733.32667.77626.stgit@dwillia2-linux.ch.intel.com>
The only .c files that utilize these protected prototypes depend on
CONFIG_INTEL_IOATDMA=y, so there is no value gained in providing empty
prototypes.
[ Impact: pure cleanup ]
Signed-off-by: Maciej Sosnowski <maciej.sosnowski@intel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
drivers/dma/ioat/dma.h | 9 ---------
1 files changed, 0 insertions(+), 9 deletions(-)
diff --git a/drivers/dma/ioat/dma.h b/drivers/dma/ioat/dma.h
index c5eabae..6e27ddb 100644
--- a/drivers/dma/ioat/dma.h
+++ b/drivers/dma/ioat/dma.h
@@ -153,19 +153,10 @@ static inline void ioat_set_tcp_copy_break(struct ioatdma_device *dev)
#endif
}
-#if defined(CONFIG_INTEL_IOATDMA) || defined(CONFIG_INTEL_IOATDMA_MODULE)
struct ioatdma_device *ioat_dma_probe(struct pci_dev *pdev,
void __iomem *iobase);
void ioat_dma_remove(struct ioatdma_device *device);
struct dca_provider *ioat_dca_init(struct pci_dev *pdev, void __iomem *iobase);
struct dca_provider *ioat2_dca_init(struct pci_dev *pdev, void __iomem *iobase);
struct dca_provider *ioat3_dca_init(struct pci_dev *pdev, void __iomem *iobase);
-#else
-#define ioat_dma_probe(pdev, iobase) NULL
-#define ioat_dma_remove(device) do { } while (0)
-#define ioat_dca_init(pdev, iobase) NULL
-#define ioat2_dca_init(pdev, iobase) NULL
-#define ioat3_dca_init(pdev, iobase) NULL
-#endif
-
#endif /* IOATDMA_H */
^ permalink raw reply related
* [PATCH 06/29] ioat: split ioat_dma_probe into core/version-specific routines
From: Dan Williams @ 2009-09-04 2:31 UTC (permalink / raw)
To: linux-kernel; +Cc: linux-raid, netdev, Maciej Sosnowski
In-Reply-To: <20090904022733.32667.77626.stgit@dwillia2-linux.ch.intel.com>
Towards the removal of ioatdma_device.version split the initialization
path into distinct versions. This conversion:
1/ moves version specific probe code to version specific routines
2/ removes the need for ioat_device
3/ turns off the ioat1 msi quirk if the device is reinitialized for intx
Signed-off-by: Maciej Sosnowski <maciej.sosnowski@intel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
drivers/dma/ioat/dma.c | 253 +++++++++++++++++++++++++++++-------------------
drivers/dma/ioat/dma.h | 23 ++--
drivers/dma/ioat/pci.c | 79 +++++++--------
3 files changed, 200 insertions(+), 155 deletions(-)
diff --git a/drivers/dma/ioat/dma.c b/drivers/dma/ioat/dma.c
index 462dae6..b750804 100644
--- a/drivers/dma/ioat/dma.c
+++ b/drivers/dma/ioat/dma.c
@@ -121,52 +121,21 @@ static int ioat_dma_enumerate_channels(struct ioatdma_device *device)
int i;
struct ioat_dma_chan *ioat_chan;
struct device *dev = &device->pdev->dev;
+ struct dma_device *dma = &device->common;
- /*
- * IOAT ver.3 workarounds
- */
- if (device->version == IOAT_VER_3_0) {
- u32 chan_err_mask;
- u16 dev_id;
- u32 dmauncerrsts;
-
- /*
- * Write CHANERRMSK_INT with 3E07h to mask out the errors
- * that can cause stability issues for IOAT ver.3
- */
- chan_err_mask = 0x3E07;
- pci_write_config_dword(device->pdev,
- IOAT_PCI_CHANERRMASK_INT_OFFSET,
- chan_err_mask);
-
- /*
- * Clear DMAUNCERRSTS Cfg-Reg Parity Error status bit
- * (workaround for spurious config parity error after restart)
- */
- pci_read_config_word(device->pdev,
- IOAT_PCI_DEVICE_ID_OFFSET,
- &dev_id);
- if (dev_id == PCI_DEVICE_ID_INTEL_IOAT_TBG0) {
- dmauncerrsts = 0x10;
- pci_write_config_dword(device->pdev,
- IOAT_PCI_DMAUNCERRSTS_OFFSET,
- dmauncerrsts);
- }
- }
-
- device->common.chancnt = readb(device->reg_base + IOAT_CHANCNT_OFFSET);
+ INIT_LIST_HEAD(&dma->channels);
+ dma->chancnt = readb(device->reg_base + IOAT_CHANCNT_OFFSET);
xfercap_scale = readb(device->reg_base + IOAT_XFERCAP_OFFSET);
xfercap = (xfercap_scale == 0 ? -1 : (1UL << xfercap_scale));
#ifdef CONFIG_I7300_IDLE_IOAT_CHANNEL
- if (i7300_idle_platform_probe(NULL, NULL, 1) == 0) {
- device->common.chancnt--;
- }
+ if (i7300_idle_platform_probe(NULL, NULL, 1) == 0)
+ dma->chancnt--;
#endif
- for (i = 0; i < device->common.chancnt; i++) {
+ for (i = 0; i < dma->chancnt; i++) {
ioat_chan = devm_kzalloc(dev, sizeof(*ioat_chan), GFP_KERNEL);
if (!ioat_chan) {
- device->common.chancnt = i;
+ dma->chancnt = i;
break;
}
@@ -175,28 +144,20 @@ static int ioat_dma_enumerate_channels(struct ioatdma_device *device)
ioat_chan->xfercap = xfercap;
ioat_chan->desccount = 0;
INIT_DELAYED_WORK(&ioat_chan->work, ioat_dma_chan_reset_part2);
- if (ioat_chan->device->version == IOAT_VER_2_0)
- writel(IOAT_DCACTRL_CMPL_WRITE_ENABLE |
- IOAT_DMA_DCA_ANY_CPU,
- ioat_chan->reg_base + IOAT_DCACTRL_OFFSET);
- else if (ioat_chan->device->version == IOAT_VER_3_0)
- writel(IOAT_DMA_DCA_ANY_CPU,
- ioat_chan->reg_base + IOAT_DCACTRL_OFFSET);
spin_lock_init(&ioat_chan->cleanup_lock);
spin_lock_init(&ioat_chan->desc_lock);
INIT_LIST_HEAD(&ioat_chan->free_desc);
INIT_LIST_HEAD(&ioat_chan->used_desc);
/* This should be made common somewhere in dmaengine.c */
ioat_chan->common.device = &device->common;
- list_add_tail(&ioat_chan->common.device_node,
- &device->common.channels);
+ list_add_tail(&ioat_chan->common.device_node, &dma->channels);
device->idx[i] = ioat_chan;
tasklet_init(&ioat_chan->cleanup_task,
ioat_dma_cleanup_tasklet,
(unsigned long) ioat_chan);
tasklet_disable(&ioat_chan->cleanup_task);
}
- return device->common.chancnt;
+ return dma->chancnt;
}
/**
@@ -1504,15 +1465,6 @@ msi:
pci_disable_msi(pdev);
goto intx;
}
- /*
- * CB 1.2 devices need a bit set in configuration space to enable MSI
- */
- if (device->version == IOAT_VER_1_2) {
- u32 dmactrl;
- pci_read_config_dword(pdev, IOAT_PCI_DMACTRL_OFFSET, &dmactrl);
- dmactrl |= IOAT_PCI_DMACTRL_MSI_EN;
- pci_write_config_dword(pdev, IOAT_PCI_DMACTRL_OFFSET, dmactrl);
- }
goto done;
intx:
@@ -1522,6 +1474,8 @@ intx:
goto err_no_irq;
done:
+ if (device->intr_quirk)
+ device->intr_quirk(device);
intrctrl |= IOAT_INTRCTRL_MASTER_INT_EN;
writeb(intrctrl, device->reg_base + IOAT_INTRCTRL_OFFSET);
return 0;
@@ -1539,21 +1493,12 @@ static void ioat_disable_interrupts(struct ioatdma_device *device)
writeb(0, device->reg_base + IOAT_INTRCTRL_OFFSET);
}
-struct ioatdma_device *
-ioat_dma_probe(struct pci_dev *pdev, void __iomem *iobase)
+static int ioat_probe(struct ioatdma_device *device)
{
- int err;
+ int err = -ENODEV;
+ struct dma_device *dma = &device->common;
+ struct pci_dev *pdev = device->pdev;
struct device *dev = &pdev->dev;
- struct ioatdma_device *device;
- struct dma_device *dma;
-
- device = devm_kzalloc(dev, sizeof(*device), GFP_KERNEL);
- if (!device)
- err = -ENOMEM;
- device->pdev = pdev;
- device->reg_base = iobase;
- device->version = readb(device->reg_base + IOAT_VER_OFFSET);
- dma = &device->common;
/* DMA coherent memory pool for DMA descriptor allocations */
device->dma_pool = pci_pool_create("dma_desc_pool", pdev,
@@ -1572,26 +1517,13 @@ ioat_dma_probe(struct pci_dev *pdev, void __iomem *iobase)
goto err_completion_pool;
}
- INIT_LIST_HEAD(&dma->channels);
ioat_dma_enumerate_channels(device);
+ dma_cap_set(DMA_MEMCPY, dma->cap_mask);
dma->device_alloc_chan_resources = ioat_dma_alloc_chan_resources;
dma->device_free_chan_resources = ioat_dma_free_chan_resources;
- dma->dev = &pdev->dev;
-
- dma_cap_set(DMA_MEMCPY, dma->cap_mask);
dma->device_is_tx_complete = ioat_dma_is_complete;
- switch (device->version) {
- case IOAT_VER_1_2:
- dma->device_prep_dma_memcpy = ioat1_dma_prep_memcpy;
- dma->device_issue_pending = ioat1_dma_memcpy_issue_pending;
- break;
- case IOAT_VER_2_0:
- case IOAT_VER_3_0:
- dma->device_prep_dma_memcpy = ioat2_dma_prep_memcpy;
- dma->device_issue_pending = ioat2_dma_memcpy_issue_pending;
- break;
- }
+ dma->dev = &pdev->dev;
dev_err(dev, "Intel(R) I/OAT DMA Engine found,"
" %d channels, device version 0x%02x, driver version %s\n",
@@ -1611,19 +1543,7 @@ ioat_dma_probe(struct pci_dev *pdev, void __iomem *iobase)
if (err)
goto err_self_test;
- err = dma_async_device_register(dma);
- if (err)
- goto err_self_test;
-
- ioat_set_tcp_copy_break(device);
-
- if (device->version != IOAT_VER_3_0) {
- INIT_DELAYED_WORK(&device->work, ioat_dma_chan_watchdog);
- schedule_delayed_work(&device->work,
- WATCHDOG_DELAY);
- }
-
- return device;
+ return 0;
err_self_test:
ioat_disable_interrupts(device);
@@ -1632,7 +1552,142 @@ err_setup_interrupts:
err_completion_pool:
pci_pool_destroy(device->dma_pool);
err_dma_pool:
- return NULL;
+ return err;
+}
+
+static int ioat_register(struct ioatdma_device *device)
+{
+ int err = dma_async_device_register(&device->common);
+
+ if (err) {
+ ioat_disable_interrupts(device);
+ pci_pool_destroy(device->completion_pool);
+ pci_pool_destroy(device->dma_pool);
+ }
+
+ return err;
+}
+
+/* ioat1_intr_quirk - fix up dma ctrl register to enable / disable msi */
+static void ioat1_intr_quirk(struct ioatdma_device *device)
+{
+ struct pci_dev *pdev = device->pdev;
+ u32 dmactrl;
+
+ pci_read_config_dword(pdev, IOAT_PCI_DMACTRL_OFFSET, &dmactrl);
+ if (pdev->msi_enabled)
+ dmactrl |= IOAT_PCI_DMACTRL_MSI_EN;
+ else
+ dmactrl &= ~IOAT_PCI_DMACTRL_MSI_EN;
+ pci_write_config_dword(pdev, IOAT_PCI_DMACTRL_OFFSET, dmactrl);
+}
+
+int ioat1_dma_probe(struct ioatdma_device *device, int dca)
+{
+ struct pci_dev *pdev = device->pdev;
+ struct dma_device *dma;
+ int err;
+
+ device->intr_quirk = ioat1_intr_quirk;
+ dma = &device->common;
+ dma->device_prep_dma_memcpy = ioat1_dma_prep_memcpy;
+ dma->device_issue_pending = ioat1_dma_memcpy_issue_pending;
+
+ err = ioat_probe(device);
+ if (err)
+ return err;
+ ioat_set_tcp_copy_break(4096);
+ err = ioat_register(device);
+ if (err)
+ return err;
+ if (dca)
+ device->dca = ioat_dca_init(pdev, device->reg_base);
+
+ INIT_DELAYED_WORK(&device->work, ioat_dma_chan_watchdog);
+ schedule_delayed_work(&device->work, WATCHDOG_DELAY);
+
+ return err;
+}
+
+int ioat2_dma_probe(struct ioatdma_device *device, int dca)
+{
+ struct pci_dev *pdev = device->pdev;
+ struct dma_device *dma;
+ struct dma_chan *chan;
+ struct ioat_dma_chan *ioat_chan;
+ int err;
+
+ dma = &device->common;
+ dma->device_prep_dma_memcpy = ioat2_dma_prep_memcpy;
+ dma->device_issue_pending = ioat2_dma_memcpy_issue_pending;
+
+ err = ioat_probe(device);
+ if (err)
+ return err;
+ ioat_set_tcp_copy_break(2048);
+
+ list_for_each_entry(chan, &dma->channels, device_node) {
+ ioat_chan = to_ioat_chan(chan);
+ writel(IOAT_DCACTRL_CMPL_WRITE_ENABLE | IOAT_DMA_DCA_ANY_CPU,
+ ioat_chan->reg_base + IOAT_DCACTRL_OFFSET);
+ }
+
+ err = ioat_register(device);
+ if (err)
+ return err;
+ if (dca)
+ device->dca = ioat2_dca_init(pdev, device->reg_base);
+
+ INIT_DELAYED_WORK(&device->work, ioat_dma_chan_watchdog);
+ schedule_delayed_work(&device->work, WATCHDOG_DELAY);
+
+ return err;
+}
+
+int ioat3_dma_probe(struct ioatdma_device *device, int dca)
+{
+ struct pci_dev *pdev = device->pdev;
+ struct dma_device *dma;
+ struct dma_chan *chan;
+ struct ioat_dma_chan *ioat_chan;
+ int err;
+ u16 dev_id;
+
+ dma = &device->common;
+ dma->device_prep_dma_memcpy = ioat2_dma_prep_memcpy;
+ dma->device_issue_pending = ioat2_dma_memcpy_issue_pending;
+
+ /* -= IOAT ver.3 workarounds =- */
+ /* Write CHANERRMSK_INT with 3E07h to mask out the errors
+ * that can cause stability issues for IOAT ver.3
+ */
+ pci_write_config_dword(pdev, IOAT_PCI_CHANERRMASK_INT_OFFSET, 0x3e07);
+
+ /* Clear DMAUNCERRSTS Cfg-Reg Parity Error status bit
+ * (workaround for spurious config parity error after restart)
+ */
+ pci_read_config_word(pdev, IOAT_PCI_DEVICE_ID_OFFSET, &dev_id);
+ if (dev_id == PCI_DEVICE_ID_INTEL_IOAT_TBG0)
+ pci_write_config_dword(pdev, IOAT_PCI_DMAUNCERRSTS_OFFSET, 0x10);
+
+ err = ioat_probe(device);
+ if (err)
+ return err;
+ ioat_set_tcp_copy_break(262144);
+
+ list_for_each_entry(chan, &dma->channels, device_node) {
+ ioat_chan = to_ioat_chan(chan);
+ writel(IOAT_DMA_DCA_ANY_CPU,
+ ioat_chan->reg_base + IOAT_DCACTRL_OFFSET);
+ }
+
+ err = ioat_register(device);
+ if (err)
+ return err;
+ if (dca)
+ device->dca = ioat3_dca_init(pdev, device->reg_base);
+
+ return err;
}
void ioat_dma_remove(struct ioatdma_device *device)
diff --git a/drivers/dma/ioat/dma.h b/drivers/dma/ioat/dma.h
index 6e27ddb..1226e35 100644
--- a/drivers/dma/ioat/dma.h
+++ b/drivers/dma/ioat/dma.h
@@ -61,6 +61,8 @@
* @version: version of ioatdma device
* @msix_entries: irq handlers
* @idx: per channel data
+ * @dca: direct cache access context
+ * @intr_quirk: interrupt setup quirk (for ioat_v1 devices)
*/
struct ioatdma_device {
@@ -73,6 +75,8 @@ struct ioatdma_device {
struct delayed_work work;
struct msix_entry msix_entries[4];
struct ioat_dma_chan *idx[4];
+ struct dca_provider *dca;
+ void (*intr_quirk)(struct ioatdma_device *device);
};
/**
@@ -136,25 +140,16 @@ struct ioat_desc_sw {
struct dma_async_tx_descriptor txd;
};
-static inline void ioat_set_tcp_copy_break(struct ioatdma_device *dev)
+static inline void ioat_set_tcp_copy_break(unsigned long copybreak)
{
#ifdef CONFIG_NET_DMA
- switch (dev->version) {
- case IOAT_VER_1_2:
- sysctl_tcp_dma_copybreak = 4096;
- break;
- case IOAT_VER_2_0:
- sysctl_tcp_dma_copybreak = 2048;
- break;
- case IOAT_VER_3_0:
- sysctl_tcp_dma_copybreak = 262144;
- break;
- }
+ sysctl_tcp_dma_copybreak = copybreak;
#endif
}
-struct ioatdma_device *ioat_dma_probe(struct pci_dev *pdev,
- void __iomem *iobase);
+int ioat1_dma_probe(struct ioatdma_device *dev, int dca);
+int ioat2_dma_probe(struct ioatdma_device *dev, int dca);
+int ioat3_dma_probe(struct ioatdma_device *dev, int dca);
void ioat_dma_remove(struct ioatdma_device *device);
struct dca_provider *ioat_dca_init(struct pci_dev *pdev, void __iomem *iobase);
struct dca_provider *ioat2_dca_init(struct pci_dev *pdev, void __iomem *iobase);
diff --git a/drivers/dma/ioat/pci.c b/drivers/dma/ioat/pci.c
index 982e38f..55414d8 100644
--- a/drivers/dma/ioat/pci.c
+++ b/drivers/dma/ioat/pci.c
@@ -60,14 +60,8 @@ static struct pci_device_id ioat_pci_tbl[] = {
{ 0, }
};
-struct ioat_device {
- struct pci_dev *pdev;
- struct ioatdma_device *dma;
- struct dca_provider *dca;
-};
-
-static int __devinit ioat_probe(struct pci_dev *pdev,
- const struct pci_device_id *id);
+static int __devinit ioat_pci_probe(struct pci_dev *pdev,
+ const struct pci_device_id *id);
static void __devexit ioat_remove(struct pci_dev *pdev);
static int ioat_dca_enabled = 1;
@@ -79,17 +73,28 @@ MODULE_PARM_DESC(ioat_dca_enabled, "control support of dca service (default: 1)"
static struct pci_driver ioat_pci_driver = {
.name = DRV_NAME,
.id_table = ioat_pci_tbl,
- .probe = ioat_probe,
+ .probe = ioat_pci_probe,
.remove = __devexit_p(ioat_remove),
};
-static int __devinit ioat_probe(struct pci_dev *pdev,
- const struct pci_device_id *id)
+static struct ioatdma_device *
+alloc_ioatdma(struct pci_dev *pdev, void __iomem *iobase)
+{
+ struct device *dev = &pdev->dev;
+ struct ioatdma_device *d = devm_kzalloc(dev, sizeof(*d), GFP_KERNEL);
+
+ if (!d)
+ return NULL;
+ d->pdev = pdev;
+ d->reg_base = iobase;
+ return d;
+}
+
+static int __devinit ioat_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
{
void __iomem * const *iomap;
- void __iomem *iobase;
struct device *dev = &pdev->dev;
- struct ioat_device *device;
+ struct ioatdma_device *device;
int err;
err = pcim_enable_device(pdev);
@@ -119,33 +124,24 @@ static int __devinit ioat_probe(struct pci_dev *pdev,
if (!device)
return -ENOMEM;
- device->pdev = pdev;
- pci_set_drvdata(pdev, device);
- iobase = iomap[IOAT_MMIO_BAR];
-
pci_set_master(pdev);
- switch (readb(iobase + IOAT_VER_OFFSET)) {
- case IOAT_VER_1_2:
- device->dma = ioat_dma_probe(pdev, iobase);
- if (device->dma && ioat_dca_enabled)
- device->dca = ioat_dca_init(pdev, iobase);
- break;
- case IOAT_VER_2_0:
- device->dma = ioat_dma_probe(pdev, iobase);
- if (device->dma && ioat_dca_enabled)
- device->dca = ioat2_dca_init(pdev, iobase);
- break;
- case IOAT_VER_3_0:
- device->dma = ioat_dma_probe(pdev, iobase);
- if (device->dma && ioat_dca_enabled)
- device->dca = ioat3_dca_init(pdev, iobase);
- break;
- default:
+ device = alloc_ioatdma(pdev, iomap[IOAT_MMIO_BAR]);
+ if (!device)
+ return -ENOMEM;
+ pci_set_drvdata(pdev, device);
+
+ device->version = readb(device->reg_base + IOAT_VER_OFFSET);
+ if (device->version == IOAT_VER_1_2)
+ err = ioat1_dma_probe(device, ioat_dca_enabled);
+ else if (device->version == IOAT_VER_2_0)
+ err = ioat2_dma_probe(device, ioat_dca_enabled);
+ else if (device->version >= IOAT_VER_3_0)
+ err = ioat3_dma_probe(device, ioat_dca_enabled);
+ else
return -ENODEV;
- }
- if (!device->dma) {
+ if (err) {
dev_err(dev, "Intel(R) I/OAT DMA Engine init failed\n");
return -ENODEV;
}
@@ -155,7 +151,10 @@ static int __devinit ioat_probe(struct pci_dev *pdev,
static void __devexit ioat_remove(struct pci_dev *pdev)
{
- struct ioat_device *device = pci_get_drvdata(pdev);
+ struct ioatdma_device *device = pci_get_drvdata(pdev);
+
+ if (!device)
+ return;
dev_err(&pdev->dev, "Removing dma and dca services\n");
if (device->dca) {
@@ -163,11 +162,7 @@ static void __devexit ioat_remove(struct pci_dev *pdev)
free_dca_provider(device->dca);
device->dca = NULL;
}
-
- if (device->dma) {
- ioat_dma_remove(device->dma);
- device->dma = NULL;
- }
+ ioat_dma_remove(device);
}
static int __init ioat_init_module(void)
^ permalink raw reply related
* [PATCH 07/29] ioat: fix type mismatch for ->dmacount
From: Dan Williams @ 2009-09-04 2:31 UTC (permalink / raw)
To: linux-kernel; +Cc: linux-raid, netdev, Maciej Sosnowski
In-Reply-To: <20090904022733.32667.77626.stgit@dwillia2-linux.ch.intel.com>
->dmacount tracks the sequence number of active descriptors. It is
written to the DMACOUNT register to update the channel's view of pending
descriptors in the chain. The register is 16-bits so ->dmacount should
be unsigned and 16-bit as well. Also modify ->desccount to maintain
alignment.
This was never a problem in practice because we never compared dmacount
values, but this is a bug waiting to happen.
Signed-off-by: Maciej Sosnowski <maciej.sosnowski@intel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
drivers/dma/ioat/dma.h | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/dma/ioat/dma.h b/drivers/dma/ioat/dma.h
index 1226e35..9f0c853 100644
--- a/drivers/dma/ioat/dma.h
+++ b/drivers/dma/ioat/dma.h
@@ -102,8 +102,8 @@ struct ioat_dma_chan {
struct delayed_work work;
int pending;
- int dmacount;
- int desccount;
+ u16 dmacount;
+ u16 desccount;
struct ioatdma_device *device;
struct dma_chan common;
^ permalink raw reply related
* [PATCH 08/29] ioat: define descriptor control bit-field
From: Dan Williams @ 2009-09-04 2:31 UTC (permalink / raw)
To: linux-kernel; +Cc: linux-raid, netdev, Maciej Sosnowski
In-Reply-To: <20090904022733.32667.77626.stgit@dwillia2-linux.ch.intel.com>
This cleans up a mess of and'ing and or'ing bit definitions, and allows
simple assignments from the specified dma_ctrl_flags parameter.
Signed-off-by: Maciej Sosnowski <maciej.sosnowski@intel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
drivers/dma/ioat/dma.c | 28 ++++++++++++++++------------
drivers/dma/ioat/hw.h | 38 ++++++++++++++++++--------------------
2 files changed, 34 insertions(+), 32 deletions(-)
diff --git a/drivers/dma/ioat/dma.c b/drivers/dma/ioat/dma.c
index b750804..4840d48 100644
--- a/drivers/dma/ioat/dma.c
+++ b/drivers/dma/ioat/dma.c
@@ -472,9 +472,9 @@ static dma_cookie_t ioat1_tx_submit(struct dma_async_tx_descriptor *tx)
return -ENOMEM;
}
- hw->ctl = IOAT_DMA_DESCRIPTOR_CTL_CP_STS;
+ hw->ctl_f.compl_write = 1;
if (first->txd.callback) {
- hw->ctl |= IOAT_DMA_DESCRIPTOR_CTL_INT_GN;
+ hw->ctl_f.int_en = 1;
if (first != new) {
/* move callback into to last desc */
new->txd.callback = first->txd.callback;
@@ -563,9 +563,9 @@ static dma_cookie_t ioat2_tx_submit(struct dma_async_tx_descriptor *tx)
return -ENOMEM;
}
- hw->ctl |= IOAT_DMA_DESCRIPTOR_CTL_CP_STS;
+ hw->ctl_f.compl_write = 1;
if (first->txd.callback) {
- hw->ctl |= IOAT_DMA_DESCRIPTOR_CTL_INT_GN;
+ hw->ctl_f.int_en = 1;
if (first != new) {
/* move callback into to last desc */
new->txd.callback = first->txd.callback;
@@ -878,7 +878,8 @@ ioat2_dma_get_next_descriptor(struct ioat_dma_chan *ioat_chan)
noop_desc = to_ioat_desc(ioat_chan->used_desc.next);
/* set size to non-zero value (channel returns error when size is 0) */
noop_desc->hw->size = NULL_DESC_BUFFER_SIZE;
- noop_desc->hw->ctl = IOAT_DMA_DESCRIPTOR_NUL;
+ noop_desc->hw->ctl = 0;
+ noop_desc->hw->ctl_f.null = 1;
noop_desc->hw->src_addr = 0;
noop_desc->hw->dst_addr = 0;
@@ -1230,6 +1231,7 @@ ioat_dma_is_complete(struct dma_chan *chan, dma_cookie_t cookie,
static void ioat_dma_start_null_desc(struct ioat_dma_chan *ioat_chan)
{
struct ioat_desc_sw *desc;
+ struct ioat_dma_descriptor *hw;
spin_lock_bh(&ioat_chan->desc_lock);
@@ -1242,17 +1244,19 @@ static void ioat_dma_start_null_desc(struct ioat_dma_chan *ioat_chan)
return;
}
- desc->hw->ctl = IOAT_DMA_DESCRIPTOR_NUL
- | IOAT_DMA_DESCRIPTOR_CTL_INT_GN
- | IOAT_DMA_DESCRIPTOR_CTL_CP_STS;
+ hw = desc->hw;
+ hw->ctl = 0;
+ hw->ctl_f.null = 1;
+ hw->ctl_f.int_en = 1;
+ hw->ctl_f.compl_write = 1;
/* set size to non-zero value (channel returns error when size is 0) */
- desc->hw->size = NULL_DESC_BUFFER_SIZE;
- desc->hw->src_addr = 0;
- desc->hw->dst_addr = 0;
+ hw->size = NULL_DESC_BUFFER_SIZE;
+ hw->src_addr = 0;
+ hw->dst_addr = 0;
async_tx_ack(&desc->txd);
switch (ioat_chan->device->version) {
case IOAT_VER_1_2:
- desc->hw->next = 0;
+ hw->next = 0;
list_add_tail(&desc->node, &ioat_chan->used_desc);
writel(((u64) desc->txd.phys) & 0x00000000FFFFFFFF,
diff --git a/drivers/dma/ioat/hw.h b/drivers/dma/ioat/hw.h
index 1438fa5..e13f3ed 100644
--- a/drivers/dma/ioat/hw.h
+++ b/drivers/dma/ioat/hw.h
@@ -40,7 +40,24 @@
struct ioat_dma_descriptor {
uint32_t size;
- uint32_t ctl;
+ union {
+ uint32_t ctl;
+ struct {
+ unsigned int int_en:1;
+ unsigned int src_snoop_dis:1;
+ unsigned int dest_snoop_dis:1;
+ unsigned int compl_write:1;
+ unsigned int fence:1;
+ unsigned int null:1;
+ unsigned int src_brk:1;
+ unsigned int dest_brk:1;
+ unsigned int bundle:1;
+ unsigned int dest_dca:1;
+ unsigned int hint:1;
+ unsigned int rsvd2:13;
+ unsigned int op:8;
+ } ctl_f;
+ };
uint64_t src_addr;
uint64_t dst_addr;
uint64_t next;
@@ -49,23 +66,4 @@ struct ioat_dma_descriptor {
uint64_t user1;
uint64_t user2;
};
-
-#define IOAT_DMA_DESCRIPTOR_CTL_INT_GN 0x00000001
-#define IOAT_DMA_DESCRIPTOR_CTL_SRC_SN 0x00000002
-#define IOAT_DMA_DESCRIPTOR_CTL_DST_SN 0x00000004
-#define IOAT_DMA_DESCRIPTOR_CTL_CP_STS 0x00000008
-#define IOAT_DMA_DESCRIPTOR_CTL_FRAME 0x00000010
-#define IOAT_DMA_DESCRIPTOR_NUL 0x00000020
-#define IOAT_DMA_DESCRIPTOR_CTL_SP_BRK 0x00000040
-#define IOAT_DMA_DESCRIPTOR_CTL_DP_BRK 0x00000080
-#define IOAT_DMA_DESCRIPTOR_CTL_BNDL 0x00000100
-#define IOAT_DMA_DESCRIPTOR_CTL_DCA 0x00000200
-#define IOAT_DMA_DESCRIPTOR_CTL_BUFHINT 0x00000400
-
-#define IOAT_DMA_DESCRIPTOR_CTL_OPCODE_CONTEXT 0xFF000000
-#define IOAT_DMA_DESCRIPTOR_CTL_OPCODE_DMA 0x00000000
-
-#define IOAT_DMA_DESCRIPTOR_CTL_CONTEXT_DCA 0x00000001
-#define IOAT_DMA_DESCRIPTOR_CTL_OPCODE_MASK 0xFF000000
-
#endif
^ permalink raw reply related
* [PATCH 09/29] ioat1: move descriptor allocation from submit to prep
From: Dan Williams @ 2009-09-04 2:31 UTC (permalink / raw)
To: linux-kernel; +Cc: linux-raid, netdev, Maciej Sosnowski
In-Reply-To: <20090904022733.32667.77626.stgit@dwillia2-linux.ch.intel.com>
The async_tx api assumes that after a successful ->prep a subsequent
->submit will not fail due to a lack of resources.
This also fixes a bug in the allocation failure case. Previously the
descriptors allocated prior to the allocation failure would not be
returned to the free list.
Signed-off-by: Maciej Sosnowski <maciej.sosnowski@intel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
drivers/dma/ioat/dma.c | 154 ++++++++++++++++++++----------------------------
1 files changed, 65 insertions(+), 89 deletions(-)
diff --git a/drivers/dma/ioat/dma.c b/drivers/dma/ioat/dma.c
index 4840d48..c4333be 100644
--- a/drivers/dma/ioat/dma.c
+++ b/drivers/dma/ioat/dma.c
@@ -420,95 +420,29 @@ static void ioat_dma_chan_watchdog(struct work_struct *work)
static dma_cookie_t ioat1_tx_submit(struct dma_async_tx_descriptor *tx)
{
struct ioat_dma_chan *ioat_chan = to_ioat_chan(tx->chan);
- struct ioat_desc_sw *first = tx_to_ioat_desc(tx);
- struct ioat_desc_sw *prev, *new;
- struct ioat_dma_descriptor *hw;
+ struct ioat_desc_sw *desc = tx_to_ioat_desc(tx);
+ struct ioat_desc_sw *first;
+ struct ioat_desc_sw *chain_tail;
dma_cookie_t cookie;
- LIST_HEAD(new_chain);
- u32 copy;
- size_t len;
- dma_addr_t src, dst;
- unsigned long orig_flags;
- unsigned int desc_count = 0;
-
- /* src and dest and len are stored in the initial descriptor */
- len = first->len;
- src = first->src;
- dst = first->dst;
- orig_flags = first->txd.flags;
- new = first;
spin_lock_bh(&ioat_chan->desc_lock);
- prev = to_ioat_desc(ioat_chan->used_desc.prev);
- prefetch(prev->hw);
- do {
- copy = min_t(size_t, len, ioat_chan->xfercap);
-
- async_tx_ack(&new->txd);
-
- hw = new->hw;
- hw->size = copy;
- hw->ctl = 0;
- hw->src_addr = src;
- hw->dst_addr = dst;
- hw->next = 0;
-
- /* chain together the physical address list for the HW */
- wmb();
- prev->hw->next = (u64) new->txd.phys;
-
- len -= copy;
- dst += copy;
- src += copy;
-
- list_add_tail(&new->node, &new_chain);
- desc_count++;
- prev = new;
- } while (len && (new = ioat1_dma_get_next_descriptor(ioat_chan)));
-
- if (!new) {
- dev_err(to_dev(ioat_chan), "tx submit failed\n");
- spin_unlock_bh(&ioat_chan->desc_lock);
- return -ENOMEM;
- }
-
- hw->ctl_f.compl_write = 1;
- if (first->txd.callback) {
- hw->ctl_f.int_en = 1;
- if (first != new) {
- /* move callback into to last desc */
- new->txd.callback = first->txd.callback;
- new->txd.callback_param
- = first->txd.callback_param;
- first->txd.callback = NULL;
- first->txd.callback_param = NULL;
- }
- }
-
- new->tx_cnt = desc_count;
- new->txd.flags = orig_flags; /* client is in control of this ack */
-
- /* store the original values for use in later cleanup */
- if (new != first) {
- new->src = first->src;
- new->dst = first->dst;
- new->len = first->len;
- }
-
/* cookie incr and addition to used_list must be atomic */
cookie = ioat_chan->common.cookie;
cookie++;
if (cookie < 0)
cookie = 1;
- ioat_chan->common.cookie = new->txd.cookie = cookie;
+ ioat_chan->common.cookie = tx->cookie = cookie;
/* write address into NextDescriptor field of last desc in chain */
- to_ioat_desc(ioat_chan->used_desc.prev)->hw->next =
- first->txd.phys;
- list_splice_tail(&new_chain, &ioat_chan->used_desc);
-
- ioat_chan->dmacount += desc_count;
- ioat_chan->pending += desc_count;
+ first = to_ioat_desc(tx->tx_list.next);
+ chain_tail = to_ioat_desc(ioat_chan->used_desc.prev);
+ /* make descriptor updates globally visible before chaining */
+ wmb();
+ chain_tail->hw->next = first->txd.phys;
+ list_splice_tail_init(&tx->tx_list, &ioat_chan->used_desc);
+
+ ioat_chan->dmacount += desc->tx_cnt;
+ ioat_chan->pending += desc->tx_cnt;
if (ioat_chan->pending >= ioat_pending_level)
__ioat1_dma_memcpy_issue_pending(ioat_chan);
spin_unlock_bh(&ioat_chan->desc_lock);
@@ -937,24 +871,66 @@ ioat1_dma_prep_memcpy(struct dma_chan *chan, dma_addr_t dma_dest,
dma_addr_t dma_src, size_t len, unsigned long flags)
{
struct ioat_dma_chan *ioat_chan = to_ioat_chan(chan);
- struct ioat_desc_sw *new;
+ struct ioat_desc_sw *desc;
+ size_t copy;
+ LIST_HEAD(chain);
+ dma_addr_t src = dma_src;
+ dma_addr_t dest = dma_dest;
+ size_t total_len = len;
+ struct ioat_dma_descriptor *hw = NULL;
+ int tx_cnt = 0;
spin_lock_bh(&ioat_chan->desc_lock);
- new = ioat_dma_get_next_descriptor(ioat_chan);
- spin_unlock_bh(&ioat_chan->desc_lock);
+ desc = ioat_dma_get_next_descriptor(ioat_chan);
+ do {
+ if (!desc)
+ break;
- if (new) {
- new->len = len;
- new->dst = dma_dest;
- new->src = dma_src;
- new->txd.flags = flags;
- return &new->txd;
- } else {
+ tx_cnt++;
+ copy = min_t(size_t, len, ioat_chan->xfercap);
+
+ hw = desc->hw;
+ hw->size = copy;
+ hw->ctl = 0;
+ hw->src_addr = src;
+ hw->dst_addr = dest;
+
+ list_add_tail(&desc->node, &chain);
+
+ len -= copy;
+ dest += copy;
+ src += copy;
+ if (len) {
+ struct ioat_desc_sw *next;
+
+ async_tx_ack(&desc->txd);
+ next = ioat_dma_get_next_descriptor(ioat_chan);
+ hw->next = next ? next->txd.phys : 0;
+ desc = next;
+ } else
+ hw->next = 0;
+ } while (len);
+
+ if (!desc) {
dev_err(to_dev(ioat_chan),
"chan%d - get_next_desc failed: %d descs waiting, %d total desc\n",
chan_num(ioat_chan), ioat_chan->dmacount, ioat_chan->desccount);
+ list_splice(&chain, &ioat_chan->free_desc);
+ spin_unlock_bh(&ioat_chan->desc_lock);
return NULL;
}
+ spin_unlock_bh(&ioat_chan->desc_lock);
+
+ desc->txd.flags = flags;
+ desc->tx_cnt = tx_cnt;
+ desc->src = dma_src;
+ desc->dst = dma_dest;
+ desc->len = total_len;
+ list_splice(&chain, &desc->txd.tx_list);
+ hw->ctl_f.int_en = !!(flags & DMA_PREP_INTERRUPT);
+ hw->ctl_f.compl_write = 1;
+
+ return &desc->txd;
}
static struct dma_async_tx_descriptor *
^ permalink raw reply related
* [PATCH 10/29] ioat: fix self test interrupts
From: Dan Williams @ 2009-09-04 2:31 UTC (permalink / raw)
To: linux-kernel; +Cc: linux-raid, netdev, Maciej Sosnowski
In-Reply-To: <20090904022733.32667.77626.stgit@dwillia2-linux.ch.intel.com>
If a callback is to be attached to a descriptor the channel needs to
know at ->prep time so it can set the interrupt enable bit. This is in
preparation for moving descriptor ioat2 descriptor preparation from
->submit to ->prep.
Signed-off-by: Maciej Sosnowski <maciej.sosnowski@intel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
drivers/dma/ioat/dma.c | 3 ++-
1 files changed, 2 insertions(+), 1 deletions(-)
diff --git a/drivers/dma/ioat/dma.c b/drivers/dma/ioat/dma.c
index c4333be..cc5c557 100644
--- a/drivers/dma/ioat/dma.c
+++ b/drivers/dma/ioat/dma.c
@@ -1313,7 +1313,8 @@ static int ioat_dma_self_test(struct ioatdma_device *device)
dma_src = dma_map_single(dev, src, IOAT_TEST_SIZE, DMA_TO_DEVICE);
dma_dest = dma_map_single(dev, dest, IOAT_TEST_SIZE, DMA_FROM_DEVICE);
- flags = DMA_COMPL_SRC_UNMAP_SINGLE | DMA_COMPL_DEST_UNMAP_SINGLE;
+ flags = DMA_COMPL_SRC_UNMAP_SINGLE | DMA_COMPL_DEST_UNMAP_SINGLE |
+ DMA_PREP_INTERRUPT;
tx = device->common.device_prep_dma_memcpy(dma_chan, dma_dest, dma_src,
IOAT_TEST_SIZE, flags);
if (!tx) {
^ permalink raw reply related
* [PATCH 11/29] ioat: prepare the code for ioat[12]_dma_chan split
From: Dan Williams @ 2009-09-04 2:31 UTC (permalink / raw)
To: linux-kernel; +Cc: linux-raid, netdev, Maciej Sosnowski
In-Reply-To: <20090904022733.32667.77626.stgit@dwillia2-linux.ch.intel.com>
Prepare the code for the conversion of the ioat2 linked-list-ring into a
native ring buffer. After this conversion ioat2 channels will share
less of the ioat1 infrastructure, but there will still be places where
sharing is possible. struct ioat_chan_common is created to house the
channel attributes that will remain common between ioat1 and ioat2
channels.
For every routine that accesses both common and hardware specific fields
the old unified 'ioat_chan' pointer is split into an 'ioat' and 'chan'
pointer. Where 'chan' references common fields and 'ioat' the
hardware/version specific.
[ Impact: pure structure member movement/variable renames, no logic changes ]
Signed-off-by: Maciej Sosnowski <maciej.sosnowski@intel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
drivers/dma/ioat/dma.c | 711 ++++++++++++++++++++++++------------------------
drivers/dma/ioat/dma.h | 49 ++-
2 files changed, 390 insertions(+), 370 deletions(-)
diff --git a/drivers/dma/ioat/dma.c b/drivers/dma/ioat/dma.c
index cc5c557..2e81e0c 100644
--- a/drivers/dma/ioat/dma.c
+++ b/drivers/dma/ioat/dma.c
@@ -47,15 +47,15 @@ static void ioat_dma_chan_reset_part2(struct work_struct *work);
static void ioat_dma_chan_watchdog(struct work_struct *work);
/* internal functions */
-static void ioat_dma_start_null_desc(struct ioat_dma_chan *ioat_chan);
-static void ioat_dma_memcpy_cleanup(struct ioat_dma_chan *ioat_chan);
+static void ioat_dma_start_null_desc(struct ioat_dma_chan *ioat);
+static void ioat_dma_memcpy_cleanup(struct ioat_dma_chan *ioat);
static struct ioat_desc_sw *
-ioat1_dma_get_next_descriptor(struct ioat_dma_chan *ioat_chan);
+ioat1_dma_get_next_descriptor(struct ioat_dma_chan *ioat);
static struct ioat_desc_sw *
-ioat2_dma_get_next_descriptor(struct ioat_dma_chan *ioat_chan);
+ioat2_dma_get_next_descriptor(struct ioat_dma_chan *ioat);
-static inline struct ioat_dma_chan *
+static inline struct ioat_chan_common *
ioat_chan_by_index(struct ioatdma_device *device, int index)
{
return device->idx[index];
@@ -69,7 +69,7 @@ ioat_chan_by_index(struct ioatdma_device *device, int index)
static irqreturn_t ioat_dma_do_interrupt(int irq, void *data)
{
struct ioatdma_device *instance = data;
- struct ioat_dma_chan *ioat_chan;
+ struct ioat_chan_common *chan;
unsigned long attnstatus;
int bit;
u8 intrctrl;
@@ -86,8 +86,8 @@ static irqreturn_t ioat_dma_do_interrupt(int irq, void *data)
attnstatus = readl(instance->reg_base + IOAT_ATTNSTATUS_OFFSET);
for_each_bit(bit, &attnstatus, BITS_PER_LONG) {
- ioat_chan = ioat_chan_by_index(instance, bit);
- tasklet_schedule(&ioat_chan->cleanup_task);
+ chan = ioat_chan_by_index(instance, bit);
+ tasklet_schedule(&chan->cleanup_task);
}
writeb(intrctrl, instance->reg_base + IOAT_INTRCTRL_OFFSET);
@@ -101,9 +101,9 @@ static irqreturn_t ioat_dma_do_interrupt(int irq, void *data)
*/
static irqreturn_t ioat_dma_do_interrupt_msix(int irq, void *data)
{
- struct ioat_dma_chan *ioat_chan = data;
+ struct ioat_chan_common *chan = data;
- tasklet_schedule(&ioat_chan->cleanup_task);
+ tasklet_schedule(&chan->cleanup_task);
return IRQ_HANDLED;
}
@@ -119,7 +119,8 @@ static int ioat_dma_enumerate_channels(struct ioatdma_device *device)
u8 xfercap_scale;
u32 xfercap;
int i;
- struct ioat_dma_chan *ioat_chan;
+ struct ioat_chan_common *chan;
+ struct ioat_dma_chan *ioat;
struct device *dev = &device->pdev->dev;
struct dma_device *dma = &device->common;
@@ -133,29 +134,30 @@ static int ioat_dma_enumerate_channels(struct ioatdma_device *device)
dma->chancnt--;
#endif
for (i = 0; i < dma->chancnt; i++) {
- ioat_chan = devm_kzalloc(dev, sizeof(*ioat_chan), GFP_KERNEL);
- if (!ioat_chan) {
+ ioat = devm_kzalloc(dev, sizeof(*ioat), GFP_KERNEL);
+ if (!ioat) {
dma->chancnt = i;
break;
}
- ioat_chan->device = device;
- ioat_chan->reg_base = device->reg_base + (0x80 * (i + 1));
- ioat_chan->xfercap = xfercap;
- ioat_chan->desccount = 0;
- INIT_DELAYED_WORK(&ioat_chan->work, ioat_dma_chan_reset_part2);
- spin_lock_init(&ioat_chan->cleanup_lock);
- spin_lock_init(&ioat_chan->desc_lock);
- INIT_LIST_HEAD(&ioat_chan->free_desc);
- INIT_LIST_HEAD(&ioat_chan->used_desc);
+ chan = &ioat->base;
+ chan->device = device;
+ chan->reg_base = device->reg_base + (0x80 * (i + 1));
+ ioat->xfercap = xfercap;
+ ioat->desccount = 0;
+ INIT_DELAYED_WORK(&chan->work, ioat_dma_chan_reset_part2);
+ spin_lock_init(&chan->cleanup_lock);
+ spin_lock_init(&ioat->desc_lock);
+ INIT_LIST_HEAD(&ioat->free_desc);
+ INIT_LIST_HEAD(&ioat->used_desc);
/* This should be made common somewhere in dmaengine.c */
- ioat_chan->common.device = &device->common;
- list_add_tail(&ioat_chan->common.device_node, &dma->channels);
- device->idx[i] = ioat_chan;
- tasklet_init(&ioat_chan->cleanup_task,
+ chan->common.device = &device->common;
+ list_add_tail(&chan->common.device_node, &dma->channels);
+ device->idx[i] = chan;
+ tasklet_init(&chan->cleanup_task,
ioat_dma_cleanup_tasklet,
- (unsigned long) ioat_chan);
- tasklet_disable(&ioat_chan->cleanup_task);
+ (unsigned long) ioat);
+ tasklet_disable(&chan->cleanup_task);
}
return dma->chancnt;
}
@@ -166,39 +168,42 @@ static int ioat_dma_enumerate_channels(struct ioatdma_device *device)
* @chan: DMA channel handle
*/
static inline void
-__ioat1_dma_memcpy_issue_pending(struct ioat_dma_chan *ioat_chan)
+__ioat1_dma_memcpy_issue_pending(struct ioat_dma_chan *ioat)
{
- ioat_chan->pending = 0;
- writeb(IOAT_CHANCMD_APPEND, ioat_chan->reg_base + IOAT1_CHANCMD_OFFSET);
+ void __iomem *reg_base = ioat->base.reg_base;
+
+ ioat->pending = 0;
+ writeb(IOAT_CHANCMD_APPEND, reg_base + IOAT1_CHANCMD_OFFSET);
}
static void ioat1_dma_memcpy_issue_pending(struct dma_chan *chan)
{
- struct ioat_dma_chan *ioat_chan = to_ioat_chan(chan);
+ struct ioat_dma_chan *ioat = to_ioat_chan(chan);
- if (ioat_chan->pending > 0) {
- spin_lock_bh(&ioat_chan->desc_lock);
- __ioat1_dma_memcpy_issue_pending(ioat_chan);
- spin_unlock_bh(&ioat_chan->desc_lock);
+ if (ioat->pending > 0) {
+ spin_lock_bh(&ioat->desc_lock);
+ __ioat1_dma_memcpy_issue_pending(ioat);
+ spin_unlock_bh(&ioat->desc_lock);
}
}
static inline void
-__ioat2_dma_memcpy_issue_pending(struct ioat_dma_chan *ioat_chan)
+__ioat2_dma_memcpy_issue_pending(struct ioat_dma_chan *ioat)
{
- ioat_chan->pending = 0;
- writew(ioat_chan->dmacount,
- ioat_chan->reg_base + IOAT_CHAN_DMACOUNT_OFFSET);
+ void __iomem *reg_base = ioat->base.reg_base;
+
+ ioat->pending = 0;
+ writew(ioat->dmacount, reg_base + IOAT_CHAN_DMACOUNT_OFFSET);
}
static void ioat2_dma_memcpy_issue_pending(struct dma_chan *chan)
{
- struct ioat_dma_chan *ioat_chan = to_ioat_chan(chan);
+ struct ioat_dma_chan *ioat = to_ioat_chan(chan);
- if (ioat_chan->pending > 0) {
- spin_lock_bh(&ioat_chan->desc_lock);
- __ioat2_dma_memcpy_issue_pending(ioat_chan);
- spin_unlock_bh(&ioat_chan->desc_lock);
+ if (ioat->pending > 0) {
+ spin_lock_bh(&ioat->desc_lock);
+ __ioat2_dma_memcpy_issue_pending(ioat);
+ spin_unlock_bh(&ioat->desc_lock);
}
}
@@ -208,84 +213,88 @@ static void ioat2_dma_memcpy_issue_pending(struct dma_chan *chan)
*/
static void ioat_dma_chan_reset_part2(struct work_struct *work)
{
- struct ioat_dma_chan *ioat_chan =
- container_of(work, struct ioat_dma_chan, work.work);
+ struct ioat_chan_common *chan;
+ struct ioat_dma_chan *ioat;
struct ioat_desc_sw *desc;
- spin_lock_bh(&ioat_chan->cleanup_lock);
- spin_lock_bh(&ioat_chan->desc_lock);
+ chan = container_of(work, struct ioat_chan_common, work.work);
+ ioat = container_of(chan, struct ioat_dma_chan, base);
+ spin_lock_bh(&chan->cleanup_lock);
+ spin_lock_bh(&ioat->desc_lock);
- ioat_chan->completion_virt->low = 0;
- ioat_chan->completion_virt->high = 0;
- ioat_chan->pending = 0;
+ chan->completion_virt->low = 0;
+ chan->completion_virt->high = 0;
+ ioat->pending = 0;
/*
* count the descriptors waiting, and be sure to do it
* right for both the CB1 line and the CB2 ring
*/
- ioat_chan->dmacount = 0;
- if (ioat_chan->used_desc.prev) {
- desc = to_ioat_desc(ioat_chan->used_desc.prev);
+ ioat->dmacount = 0;
+ if (ioat->used_desc.prev) {
+ desc = to_ioat_desc(ioat->used_desc.prev);
do {
- ioat_chan->dmacount++;
+ ioat->dmacount++;
desc = to_ioat_desc(desc->node.next);
- } while (&desc->node != ioat_chan->used_desc.next);
+ } while (&desc->node != ioat->used_desc.next);
}
/*
* write the new starting descriptor address
* this puts channel engine into ARMED state
*/
- desc = to_ioat_desc(ioat_chan->used_desc.prev);
- switch (ioat_chan->device->version) {
+ desc = to_ioat_desc(ioat->used_desc.prev);
+ switch (chan->device->version) {
case IOAT_VER_1_2:
writel(((u64) desc->txd.phys) & 0x00000000FFFFFFFF,
- ioat_chan->reg_base + IOAT1_CHAINADDR_OFFSET_LOW);
+ chan->reg_base + IOAT1_CHAINADDR_OFFSET_LOW);
writel(((u64) desc->txd.phys) >> 32,
- ioat_chan->reg_base + IOAT1_CHAINADDR_OFFSET_HIGH);
+ chan->reg_base + IOAT1_CHAINADDR_OFFSET_HIGH);
- writeb(IOAT_CHANCMD_START, ioat_chan->reg_base
- + IOAT_CHANCMD_OFFSET(ioat_chan->device->version));
+ writeb(IOAT_CHANCMD_START, chan->reg_base
+ + IOAT_CHANCMD_OFFSET(chan->device->version));
break;
case IOAT_VER_2_0:
writel(((u64) desc->txd.phys) & 0x00000000FFFFFFFF,
- ioat_chan->reg_base + IOAT2_CHAINADDR_OFFSET_LOW);
+ chan->reg_base + IOAT2_CHAINADDR_OFFSET_LOW);
writel(((u64) desc->txd.phys) >> 32,
- ioat_chan->reg_base + IOAT2_CHAINADDR_OFFSET_HIGH);
+ chan->reg_base + IOAT2_CHAINADDR_OFFSET_HIGH);
/* tell the engine to go with what's left to be done */
- writew(ioat_chan->dmacount,
- ioat_chan->reg_base + IOAT_CHAN_DMACOUNT_OFFSET);
+ writew(ioat->dmacount,
+ chan->reg_base + IOAT_CHAN_DMACOUNT_OFFSET);
break;
}
- dev_err(to_dev(ioat_chan),
+ dev_err(to_dev(chan),
"chan%d reset - %d descs waiting, %d total desc\n",
- chan_num(ioat_chan), ioat_chan->dmacount, ioat_chan->desccount);
+ chan_num(chan), ioat->dmacount, ioat->desccount);
- spin_unlock_bh(&ioat_chan->desc_lock);
- spin_unlock_bh(&ioat_chan->cleanup_lock);
+ spin_unlock_bh(&ioat->desc_lock);
+ spin_unlock_bh(&chan->cleanup_lock);
}
/**
* ioat_dma_reset_channel - restart a channel
- * @ioat_chan: IOAT DMA channel handle
+ * @ioat: IOAT DMA channel handle
*/
-static void ioat_dma_reset_channel(struct ioat_dma_chan *ioat_chan)
+static void ioat_dma_reset_channel(struct ioat_dma_chan *ioat)
{
+ struct ioat_chan_common *chan = &ioat->base;
+ void __iomem *reg_base = chan->reg_base;
u32 chansts, chanerr;
- if (!ioat_chan->used_desc.prev)
+ if (!ioat->used_desc.prev)
return;
- chanerr = readl(ioat_chan->reg_base + IOAT_CHANERR_OFFSET);
- chansts = (ioat_chan->completion_virt->low
+ chanerr = readl(reg_base + IOAT_CHANERR_OFFSET);
+ chansts = (chan->completion_virt->low
& IOAT_CHANSTS_DMA_TRANSFER_STATUS);
if (chanerr) {
- dev_err(to_dev(ioat_chan),
+ dev_err(to_dev(chan),
"chan%d, CHANSTS = 0x%08x CHANERR = 0x%04x, clearing\n",
- chan_num(ioat_chan), chansts, chanerr);
- writel(chanerr, ioat_chan->reg_base + IOAT_CHANERR_OFFSET);
+ chan_num(chan), chansts, chanerr);
+ writel(chanerr, reg_base + IOAT_CHANERR_OFFSET);
}
/*
@@ -296,15 +305,14 @@ static void ioat_dma_reset_channel(struct ioat_dma_chan *ioat_chan)
* while we're waiting.
*/
- spin_lock_bh(&ioat_chan->desc_lock);
- ioat_chan->pending = INT_MIN;
+ spin_lock_bh(&ioat->desc_lock);
+ ioat->pending = INT_MIN;
writeb(IOAT_CHANCMD_RESET,
- ioat_chan->reg_base
- + IOAT_CHANCMD_OFFSET(ioat_chan->device->version));
- spin_unlock_bh(&ioat_chan->desc_lock);
+ reg_base + IOAT_CHANCMD_OFFSET(chan->device->version));
+ spin_unlock_bh(&ioat->desc_lock);
/* schedule the 2nd half instead of sleeping a long time */
- schedule_delayed_work(&ioat_chan->work, RESET_DELAY);
+ schedule_delayed_work(&chan->work, RESET_DELAY);
}
/**
@@ -314,7 +322,8 @@ static void ioat_dma_chan_watchdog(struct work_struct *work)
{
struct ioatdma_device *device =
container_of(work, struct ioatdma_device, work.work);
- struct ioat_dma_chan *ioat_chan;
+ struct ioat_dma_chan *ioat;
+ struct ioat_chan_common *chan;
int i;
union {
@@ -327,23 +336,21 @@ static void ioat_dma_chan_watchdog(struct work_struct *work)
unsigned long compl_desc_addr_hw;
for (i = 0; i < device->common.chancnt; i++) {
- ioat_chan = ioat_chan_by_index(device, i);
+ chan = ioat_chan_by_index(device, i);
+ ioat = container_of(chan, struct ioat_dma_chan, base);
- if (ioat_chan->device->version == IOAT_VER_1_2
+ if (chan->device->version == IOAT_VER_1_2
/* have we started processing anything yet */
- && ioat_chan->last_completion
+ && chan->last_completion
/* have we completed any since last watchdog cycle? */
- && (ioat_chan->last_completion ==
- ioat_chan->watchdog_completion)
+ && (chan->last_completion == chan->watchdog_completion)
/* has TCP stuck on one cookie since last watchdog? */
- && (ioat_chan->watchdog_tcp_cookie ==
- ioat_chan->watchdog_last_tcp_cookie)
- && (ioat_chan->watchdog_tcp_cookie !=
- ioat_chan->completed_cookie)
+ && (chan->watchdog_tcp_cookie == chan->watchdog_last_tcp_cookie)
+ && (chan->watchdog_tcp_cookie != chan->completed_cookie)
/* is there something in the chain to be processed? */
/* CB1 chain always has at least the last one processed */
- && (ioat_chan->used_desc.prev != ioat_chan->used_desc.next)
- && ioat_chan->pending == 0) {
+ && (ioat->used_desc.prev != ioat->used_desc.next)
+ && ioat->pending == 0) {
/*
* check CHANSTS register for completed
@@ -360,10 +367,10 @@ static void ioat_dma_chan_watchdog(struct work_struct *work)
* try resetting the channel
*/
- completion_hw.low = readl(ioat_chan->reg_base +
- IOAT_CHANSTS_OFFSET_LOW(ioat_chan->device->version));
- completion_hw.high = readl(ioat_chan->reg_base +
- IOAT_CHANSTS_OFFSET_HIGH(ioat_chan->device->version));
+ completion_hw.low = readl(chan->reg_base +
+ IOAT_CHANSTS_OFFSET_LOW(chan->device->version));
+ completion_hw.high = readl(chan->reg_base +
+ IOAT_CHANSTS_OFFSET_HIGH(chan->device->version));
#if (BITS_PER_LONG == 64)
compl_desc_addr_hw =
completion_hw.full
@@ -374,15 +381,15 @@ static void ioat_dma_chan_watchdog(struct work_struct *work)
#endif
if ((compl_desc_addr_hw != 0)
- && (compl_desc_addr_hw != ioat_chan->watchdog_completion)
- && (compl_desc_addr_hw != ioat_chan->last_compl_desc_addr_hw)) {
- ioat_chan->last_compl_desc_addr_hw = compl_desc_addr_hw;
- ioat_chan->completion_virt->low = completion_hw.low;
- ioat_chan->completion_virt->high = completion_hw.high;
+ && (compl_desc_addr_hw != chan->watchdog_completion)
+ && (compl_desc_addr_hw != chan->last_compl_desc_addr_hw)) {
+ chan->last_compl_desc_addr_hw = compl_desc_addr_hw;
+ chan->completion_virt->low = completion_hw.low;
+ chan->completion_virt->high = completion_hw.high;
} else {
- ioat_dma_reset_channel(ioat_chan);
- ioat_chan->watchdog_completion = 0;
- ioat_chan->last_compl_desc_addr_hw = 0;
+ ioat_dma_reset_channel(ioat);
+ chan->watchdog_completion = 0;
+ chan->last_compl_desc_addr_hw = 0;
}
/*
@@ -393,25 +400,22 @@ static void ioat_dma_chan_watchdog(struct work_struct *work)
* else
* try resetting the channel
*/
- } else if (ioat_chan->device->version == IOAT_VER_2_0
- && ioat_chan->used_desc.prev
- && ioat_chan->last_completion
- && ioat_chan->last_completion == ioat_chan->watchdog_completion) {
+ } else if (chan->device->version == IOAT_VER_2_0
+ && ioat->used_desc.prev
+ && chan->last_completion
+ && chan->last_completion == chan->watchdog_completion) {
- if (ioat_chan->pending < ioat_pending_level)
- ioat2_dma_memcpy_issue_pending(&ioat_chan->common);
+ if (ioat->pending < ioat_pending_level)
+ ioat2_dma_memcpy_issue_pending(&chan->common);
else {
- ioat_dma_reset_channel(ioat_chan);
- ioat_chan->watchdog_completion = 0;
+ ioat_dma_reset_channel(ioat);
+ chan->watchdog_completion = 0;
}
} else {
- ioat_chan->last_compl_desc_addr_hw = 0;
- ioat_chan->watchdog_completion
- = ioat_chan->last_completion;
+ chan->last_compl_desc_addr_hw = 0;
+ chan->watchdog_completion = chan->last_completion;
}
-
- ioat_chan->watchdog_last_tcp_cookie =
- ioat_chan->watchdog_tcp_cookie;
+ chan->watchdog_last_tcp_cookie = chan->watchdog_tcp_cookie;
}
schedule_delayed_work(&device->work, WATCHDOG_DELAY);
@@ -419,40 +423,42 @@ static void ioat_dma_chan_watchdog(struct work_struct *work)
static dma_cookie_t ioat1_tx_submit(struct dma_async_tx_descriptor *tx)
{
- struct ioat_dma_chan *ioat_chan = to_ioat_chan(tx->chan);
+ struct dma_chan *c = tx->chan;
+ struct ioat_dma_chan *ioat = to_ioat_chan(c);
struct ioat_desc_sw *desc = tx_to_ioat_desc(tx);
struct ioat_desc_sw *first;
struct ioat_desc_sw *chain_tail;
dma_cookie_t cookie;
- spin_lock_bh(&ioat_chan->desc_lock);
+ spin_lock_bh(&ioat->desc_lock);
/* cookie incr and addition to used_list must be atomic */
- cookie = ioat_chan->common.cookie;
+ cookie = c->cookie;
cookie++;
if (cookie < 0)
cookie = 1;
- ioat_chan->common.cookie = tx->cookie = cookie;
+ c->cookie = cookie;
+ tx->cookie = cookie;
/* write address into NextDescriptor field of last desc in chain */
first = to_ioat_desc(tx->tx_list.next);
- chain_tail = to_ioat_desc(ioat_chan->used_desc.prev);
+ chain_tail = to_ioat_desc(ioat->used_desc.prev);
/* make descriptor updates globally visible before chaining */
wmb();
chain_tail->hw->next = first->txd.phys;
- list_splice_tail_init(&tx->tx_list, &ioat_chan->used_desc);
+ list_splice_tail_init(&tx->tx_list, &ioat->used_desc);
- ioat_chan->dmacount += desc->tx_cnt;
- ioat_chan->pending += desc->tx_cnt;
- if (ioat_chan->pending >= ioat_pending_level)
- __ioat1_dma_memcpy_issue_pending(ioat_chan);
- spin_unlock_bh(&ioat_chan->desc_lock);
+ ioat->dmacount += desc->tx_cnt;
+ ioat->pending += desc->tx_cnt;
+ if (ioat->pending >= ioat_pending_level)
+ __ioat1_dma_memcpy_issue_pending(ioat);
+ spin_unlock_bh(&ioat->desc_lock);
return cookie;
}
static dma_cookie_t ioat2_tx_submit(struct dma_async_tx_descriptor *tx)
{
- struct ioat_dma_chan *ioat_chan = to_ioat_chan(tx->chan);
+ struct ioat_dma_chan *ioat = to_ioat_chan(tx->chan);
struct ioat_desc_sw *first = tx_to_ioat_desc(tx);
struct ioat_desc_sw *new;
struct ioat_dma_descriptor *hw;
@@ -471,11 +477,11 @@ static dma_cookie_t ioat2_tx_submit(struct dma_async_tx_descriptor *tx)
new = first;
/*
- * ioat_chan->desc_lock is still in force in version 2 path
+ * ioat->desc_lock is still in force in version 2 path
* it gets unlocked at end of this function
*/
do {
- copy = min_t(size_t, len, ioat_chan->xfercap);
+ copy = min_t(size_t, len, ioat->xfercap);
async_tx_ack(&new->txd);
@@ -489,11 +495,11 @@ static dma_cookie_t ioat2_tx_submit(struct dma_async_tx_descriptor *tx)
dst += copy;
src += copy;
desc_count++;
- } while (len && (new = ioat2_dma_get_next_descriptor(ioat_chan)));
+ } while (len && (new = ioat2_dma_get_next_descriptor(ioat)));
if (!new) {
- dev_err(to_dev(ioat_chan), "tx submit failed\n");
- spin_unlock_bh(&ioat_chan->desc_lock);
+ dev_err(to_dev(&ioat->base), "tx submit failed\n");
+ spin_unlock_bh(&ioat->desc_lock);
return -ENOMEM;
}
@@ -521,35 +527,35 @@ static dma_cookie_t ioat2_tx_submit(struct dma_async_tx_descriptor *tx)
}
/* cookie incr and addition to used_list must be atomic */
- cookie = ioat_chan->common.cookie;
+ cookie = ioat->base.common.cookie;
cookie++;
if (cookie < 0)
cookie = 1;
- ioat_chan->common.cookie = new->txd.cookie = cookie;
+ ioat->base.common.cookie = new->txd.cookie = cookie;
- ioat_chan->dmacount += desc_count;
- ioat_chan->pending += desc_count;
- if (ioat_chan->pending >= ioat_pending_level)
- __ioat2_dma_memcpy_issue_pending(ioat_chan);
- spin_unlock_bh(&ioat_chan->desc_lock);
+ ioat->dmacount += desc_count;
+ ioat->pending += desc_count;
+ if (ioat->pending >= ioat_pending_level)
+ __ioat2_dma_memcpy_issue_pending(ioat);
+ spin_unlock_bh(&ioat->desc_lock);
return cookie;
}
/**
* ioat_dma_alloc_descriptor - allocate and return a sw and hw descriptor pair
- * @ioat_chan: the channel supplying the memory pool for the descriptors
+ * @ioat: the channel supplying the memory pool for the descriptors
* @flags: allocation flags
*/
static struct ioat_desc_sw *
-ioat_dma_alloc_descriptor(struct ioat_dma_chan *ioat_chan, gfp_t flags)
+ioat_dma_alloc_descriptor(struct ioat_dma_chan *ioat, gfp_t flags)
{
struct ioat_dma_descriptor *desc;
struct ioat_desc_sw *desc_sw;
struct ioatdma_device *ioatdma_device;
dma_addr_t phys;
- ioatdma_device = to_ioatdma_device(ioat_chan->common.device);
+ ioatdma_device = ioat->base.device;
desc = pci_pool_alloc(ioatdma_device->dma_pool, flags, &phys);
if (unlikely(!desc))
return NULL;
@@ -561,8 +567,8 @@ ioat_dma_alloc_descriptor(struct ioat_dma_chan *ioat_chan, gfp_t flags)
}
memset(desc, 0, sizeof(*desc));
- dma_async_tx_descriptor_init(&desc_sw->txd, &ioat_chan->common);
- switch (ioat_chan->device->version) {
+ dma_async_tx_descriptor_init(&desc_sw->txd, &ioat->base.common);
+ switch (ioatdma_device->version) {
case IOAT_VER_1_2:
desc_sw->txd.tx_submit = ioat1_tx_submit;
break;
@@ -585,26 +591,26 @@ MODULE_PARM_DESC(ioat_initial_desc_count,
/**
* ioat2_dma_massage_chan_desc - link the descriptors into a circle
- * @ioat_chan: the channel to be massaged
+ * @ioat: the channel to be massaged
*/
-static void ioat2_dma_massage_chan_desc(struct ioat_dma_chan *ioat_chan)
+static void ioat2_dma_massage_chan_desc(struct ioat_dma_chan *ioat)
{
struct ioat_desc_sw *desc, *_desc;
/* setup used_desc */
- ioat_chan->used_desc.next = ioat_chan->free_desc.next;
- ioat_chan->used_desc.prev = NULL;
+ ioat->used_desc.next = ioat->free_desc.next;
+ ioat->used_desc.prev = NULL;
/* pull free_desc out of the circle so that every node is a hw
* descriptor, but leave it pointing to the list
*/
- ioat_chan->free_desc.prev->next = ioat_chan->free_desc.next;
- ioat_chan->free_desc.next->prev = ioat_chan->free_desc.prev;
+ ioat->free_desc.prev->next = ioat->free_desc.next;
+ ioat->free_desc.next->prev = ioat->free_desc.prev;
/* circle link the hw descriptors */
- desc = to_ioat_desc(ioat_chan->free_desc.next);
+ desc = to_ioat_desc(ioat->free_desc.next);
desc->hw->next = to_ioat_desc(desc->node.next)->txd.phys;
- list_for_each_entry_safe(desc, _desc, ioat_chan->free_desc.next, node) {
+ list_for_each_entry_safe(desc, _desc, ioat->free_desc.next, node) {
desc->hw->next = to_ioat_desc(desc->node.next)->txd.phys;
}
}
@@ -613,9 +619,10 @@ static void ioat2_dma_massage_chan_desc(struct ioat_dma_chan *ioat_chan)
* ioat_dma_alloc_chan_resources - returns the number of allocated descriptors
* @chan: the channel to be filled out
*/
-static int ioat_dma_alloc_chan_resources(struct dma_chan *chan)
+static int ioat_dma_alloc_chan_resources(struct dma_chan *c)
{
- struct ioat_dma_chan *ioat_chan = to_ioat_chan(chan);
+ struct ioat_dma_chan *ioat = to_ioat_chan(c);
+ struct ioat_chan_common *chan = &ioat->base;
struct ioat_desc_sw *desc;
u16 chanctrl;
u32 chanerr;
@@ -623,89 +630,87 @@ static int ioat_dma_alloc_chan_resources(struct dma_chan *chan)
LIST_HEAD(tmp_list);
/* have we already been set up? */
- if (!list_empty(&ioat_chan->free_desc))
- return ioat_chan->desccount;
+ if (!list_empty(&ioat->free_desc))
+ return ioat->desccount;
/* Setup register to interrupt and write completion status on error */
chanctrl = IOAT_CHANCTRL_ERR_INT_EN |
IOAT_CHANCTRL_ANY_ERR_ABORT_EN |
IOAT_CHANCTRL_ERR_COMPLETION_EN;
- writew(chanctrl, ioat_chan->reg_base + IOAT_CHANCTRL_OFFSET);
+ writew(chanctrl, chan->reg_base + IOAT_CHANCTRL_OFFSET);
- chanerr = readl(ioat_chan->reg_base + IOAT_CHANERR_OFFSET);
+ chanerr = readl(chan->reg_base + IOAT_CHANERR_OFFSET);
if (chanerr) {
- dev_err(to_dev(ioat_chan), "CHANERR = %x, clearing\n", chanerr);
- writel(chanerr, ioat_chan->reg_base + IOAT_CHANERR_OFFSET);
+ dev_err(to_dev(chan), "CHANERR = %x, clearing\n", chanerr);
+ writel(chanerr, chan->reg_base + IOAT_CHANERR_OFFSET);
}
/* Allocate descriptors */
for (i = 0; i < ioat_initial_desc_count; i++) {
- desc = ioat_dma_alloc_descriptor(ioat_chan, GFP_KERNEL);
+ desc = ioat_dma_alloc_descriptor(ioat, GFP_KERNEL);
if (!desc) {
- dev_err(to_dev(ioat_chan),
- "Only %d initial descriptors\n", i);
+ dev_err(to_dev(chan), "Only %d initial descriptors\n", i);
break;
}
list_add_tail(&desc->node, &tmp_list);
}
- spin_lock_bh(&ioat_chan->desc_lock);
- ioat_chan->desccount = i;
- list_splice(&tmp_list, &ioat_chan->free_desc);
- if (ioat_chan->device->version != IOAT_VER_1_2)
- ioat2_dma_massage_chan_desc(ioat_chan);
- spin_unlock_bh(&ioat_chan->desc_lock);
+ spin_lock_bh(&ioat->desc_lock);
+ ioat->desccount = i;
+ list_splice(&tmp_list, &ioat->free_desc);
+ if (chan->device->version != IOAT_VER_1_2)
+ ioat2_dma_massage_chan_desc(ioat);
+ spin_unlock_bh(&ioat->desc_lock);
/* allocate a completion writeback area */
/* doing 2 32bit writes to mmio since 1 64b write doesn't work */
- ioat_chan->completion_virt =
- pci_pool_alloc(ioat_chan->device->completion_pool,
- GFP_KERNEL,
- &ioat_chan->completion_addr);
- memset(ioat_chan->completion_virt, 0,
- sizeof(*ioat_chan->completion_virt));
- writel(((u64) ioat_chan->completion_addr) & 0x00000000FFFFFFFF,
- ioat_chan->reg_base + IOAT_CHANCMP_OFFSET_LOW);
- writel(((u64) ioat_chan->completion_addr) >> 32,
- ioat_chan->reg_base + IOAT_CHANCMP_OFFSET_HIGH);
-
- tasklet_enable(&ioat_chan->cleanup_task);
- ioat_dma_start_null_desc(ioat_chan); /* give chain to dma device */
- return ioat_chan->desccount;
+ chan->completion_virt = pci_pool_alloc(chan->device->completion_pool,
+ GFP_KERNEL,
+ &chan->completion_addr);
+ memset(chan->completion_virt, 0,
+ sizeof(*chan->completion_virt));
+ writel(((u64) chan->completion_addr) & 0x00000000FFFFFFFF,
+ chan->reg_base + IOAT_CHANCMP_OFFSET_LOW);
+ writel(((u64) chan->completion_addr) >> 32,
+ chan->reg_base + IOAT_CHANCMP_OFFSET_HIGH);
+
+ tasklet_enable(&chan->cleanup_task);
+ ioat_dma_start_null_desc(ioat); /* give chain to dma device */
+ return ioat->desccount;
}
/**
* ioat_dma_free_chan_resources - release all the descriptors
* @chan: the channel to be cleaned
*/
-static void ioat_dma_free_chan_resources(struct dma_chan *chan)
+static void ioat_dma_free_chan_resources(struct dma_chan *c)
{
- struct ioat_dma_chan *ioat_chan = to_ioat_chan(chan);
- struct ioatdma_device *ioatdma_device = to_ioatdma_device(chan->device);
+ struct ioat_dma_chan *ioat = to_ioat_chan(c);
+ struct ioat_chan_common *chan = &ioat->base;
+ struct ioatdma_device *ioatdma_device = chan->device;
struct ioat_desc_sw *desc, *_desc;
int in_use_descs = 0;
/* Before freeing channel resources first check
* if they have been previously allocated for this channel.
*/
- if (ioat_chan->desccount == 0)
+ if (ioat->desccount == 0)
return;
- tasklet_disable(&ioat_chan->cleanup_task);
- ioat_dma_memcpy_cleanup(ioat_chan);
+ tasklet_disable(&chan->cleanup_task);
+ ioat_dma_memcpy_cleanup(ioat);
/* Delay 100ms after reset to allow internal DMA logic to quiesce
* before removing DMA descriptor resources.
*/
writeb(IOAT_CHANCMD_RESET,
- ioat_chan->reg_base
- + IOAT_CHANCMD_OFFSET(ioat_chan->device->version));
+ chan->reg_base + IOAT_CHANCMD_OFFSET(chan->device->version));
mdelay(100);
- spin_lock_bh(&ioat_chan->desc_lock);
- switch (ioat_chan->device->version) {
+ spin_lock_bh(&ioat->desc_lock);
+ switch (chan->device->version) {
case IOAT_VER_1_2:
list_for_each_entry_safe(desc, _desc,
- &ioat_chan->used_desc, node) {
+ &ioat->used_desc, node) {
in_use_descs++;
list_del(&desc->node);
pci_pool_free(ioatdma_device->dma_pool, desc->hw,
@@ -713,7 +718,7 @@ static void ioat_dma_free_chan_resources(struct dma_chan *chan)
kfree(desc);
}
list_for_each_entry_safe(desc, _desc,
- &ioat_chan->free_desc, node) {
+ &ioat->free_desc, node) {
list_del(&desc->node);
pci_pool_free(ioatdma_device->dma_pool, desc->hw,
desc->txd.phys);
@@ -723,62 +728,61 @@ static void ioat_dma_free_chan_resources(struct dma_chan *chan)
case IOAT_VER_2_0:
case IOAT_VER_3_0:
list_for_each_entry_safe(desc, _desc,
- ioat_chan->free_desc.next, node) {
+ ioat->free_desc.next, node) {
list_del(&desc->node);
pci_pool_free(ioatdma_device->dma_pool, desc->hw,
desc->txd.phys);
kfree(desc);
}
- desc = to_ioat_desc(ioat_chan->free_desc.next);
+ desc = to_ioat_desc(ioat->free_desc.next);
pci_pool_free(ioatdma_device->dma_pool, desc->hw,
desc->txd.phys);
kfree(desc);
- INIT_LIST_HEAD(&ioat_chan->free_desc);
- INIT_LIST_HEAD(&ioat_chan->used_desc);
+ INIT_LIST_HEAD(&ioat->free_desc);
+ INIT_LIST_HEAD(&ioat->used_desc);
break;
}
- spin_unlock_bh(&ioat_chan->desc_lock);
+ spin_unlock_bh(&ioat->desc_lock);
pci_pool_free(ioatdma_device->completion_pool,
- ioat_chan->completion_virt,
- ioat_chan->completion_addr);
+ chan->completion_virt,
+ chan->completion_addr);
/* one is ok since we left it on there on purpose */
if (in_use_descs > 1)
- dev_err(to_dev(ioat_chan), "Freeing %d in use descriptors!\n",
+ dev_err(to_dev(chan), "Freeing %d in use descriptors!\n",
in_use_descs - 1);
- ioat_chan->last_completion = ioat_chan->completion_addr = 0;
- ioat_chan->pending = 0;
- ioat_chan->dmacount = 0;
- ioat_chan->desccount = 0;
- ioat_chan->watchdog_completion = 0;
- ioat_chan->last_compl_desc_addr_hw = 0;
- ioat_chan->watchdog_tcp_cookie =
- ioat_chan->watchdog_last_tcp_cookie = 0;
+ chan->last_completion = chan->completion_addr = 0;
+ chan->watchdog_completion = 0;
+ chan->last_compl_desc_addr_hw = 0;
+ chan->watchdog_tcp_cookie = chan->watchdog_last_tcp_cookie = 0;
+ ioat->pending = 0;
+ ioat->dmacount = 0;
+ ioat->desccount = 0;
}
/**
- * ioat_dma_get_next_descriptor - return the next available descriptor
- * @ioat_chan: IOAT DMA channel handle
+ * ioat1_dma_get_next_descriptor - return the next available descriptor
+ * @ioat: IOAT DMA channel handle
*
* Gets the next descriptor from the chain, and must be called with the
* channel's desc_lock held. Allocates more descriptors if the channel
* has run out.
*/
static struct ioat_desc_sw *
-ioat1_dma_get_next_descriptor(struct ioat_dma_chan *ioat_chan)
+ioat1_dma_get_next_descriptor(struct ioat_dma_chan *ioat)
{
struct ioat_desc_sw *new;
- if (!list_empty(&ioat_chan->free_desc)) {
- new = to_ioat_desc(ioat_chan->free_desc.next);
+ if (!list_empty(&ioat->free_desc)) {
+ new = to_ioat_desc(ioat->free_desc.next);
list_del(&new->node);
} else {
/* try to get another desc */
- new = ioat_dma_alloc_descriptor(ioat_chan, GFP_ATOMIC);
+ new = ioat_dma_alloc_descriptor(ioat, GFP_ATOMIC);
if (!new) {
- dev_err(to_dev(ioat_chan), "alloc failed\n");
+ dev_err(to_dev(&ioat->base), "alloc failed\n");
return NULL;
}
}
@@ -788,7 +792,7 @@ ioat1_dma_get_next_descriptor(struct ioat_dma_chan *ioat_chan)
}
static struct ioat_desc_sw *
-ioat2_dma_get_next_descriptor(struct ioat_dma_chan *ioat_chan)
+ioat2_dma_get_next_descriptor(struct ioat_dma_chan *ioat)
{
struct ioat_desc_sw *new;
@@ -801,15 +805,15 @@ ioat2_dma_get_next_descriptor(struct ioat_dma_chan *ioat_chan)
* linking in a new set of descriptors, since the device
* has probably already read the pointer to it
*/
- if (ioat_chan->used_desc.prev &&
- ioat_chan->used_desc.next == ioat_chan->used_desc.prev->prev) {
+ if (ioat->used_desc.prev &&
+ ioat->used_desc.next == ioat->used_desc.prev->prev) {
struct ioat_desc_sw *desc;
struct ioat_desc_sw *noop_desc;
int i;
/* set up the noop descriptor */
- noop_desc = to_ioat_desc(ioat_chan->used_desc.next);
+ noop_desc = to_ioat_desc(ioat->used_desc.next);
/* set size to non-zero value (channel returns error when size is 0) */
noop_desc->hw->size = NULL_DESC_BUFFER_SIZE;
noop_desc->hw->ctl = 0;
@@ -817,60 +821,61 @@ ioat2_dma_get_next_descriptor(struct ioat_dma_chan *ioat_chan)
noop_desc->hw->src_addr = 0;
noop_desc->hw->dst_addr = 0;
- ioat_chan->used_desc.next = ioat_chan->used_desc.next->next;
- ioat_chan->pending++;
- ioat_chan->dmacount++;
+ ioat->used_desc.next = ioat->used_desc.next->next;
+ ioat->pending++;
+ ioat->dmacount++;
/* try to get a few more descriptors */
for (i = 16; i; i--) {
- desc = ioat_dma_alloc_descriptor(ioat_chan, GFP_ATOMIC);
+ desc = ioat_dma_alloc_descriptor(ioat, GFP_ATOMIC);
if (!desc) {
- dev_err(to_dev(ioat_chan), "alloc failed\n");
+ dev_err(to_dev(&ioat->base),
+ "alloc failed\n");
break;
}
- list_add_tail(&desc->node, ioat_chan->used_desc.next);
+ list_add_tail(&desc->node, ioat->used_desc.next);
desc->hw->next
= to_ioat_desc(desc->node.next)->txd.phys;
to_ioat_desc(desc->node.prev)->hw->next
= desc->txd.phys;
- ioat_chan->desccount++;
+ ioat->desccount++;
}
- ioat_chan->used_desc.next = noop_desc->node.next;
+ ioat->used_desc.next = noop_desc->node.next;
}
- new = to_ioat_desc(ioat_chan->used_desc.next);
+ new = to_ioat_desc(ioat->used_desc.next);
prefetch(new);
- ioat_chan->used_desc.next = new->node.next;
+ ioat->used_desc.next = new->node.next;
- if (ioat_chan->used_desc.prev == NULL)
- ioat_chan->used_desc.prev = &new->node;
+ if (ioat->used_desc.prev == NULL)
+ ioat->used_desc.prev = &new->node;
prefetch(new->hw);
return new;
}
static struct ioat_desc_sw *
-ioat_dma_get_next_descriptor(struct ioat_dma_chan *ioat_chan)
+ioat_dma_get_next_descriptor(struct ioat_dma_chan *ioat)
{
- if (!ioat_chan)
+ if (!ioat)
return NULL;
- switch (ioat_chan->device->version) {
+ switch (ioat->base.device->version) {
case IOAT_VER_1_2:
- return ioat1_dma_get_next_descriptor(ioat_chan);
+ return ioat1_dma_get_next_descriptor(ioat);
case IOAT_VER_2_0:
case IOAT_VER_3_0:
- return ioat2_dma_get_next_descriptor(ioat_chan);
+ return ioat2_dma_get_next_descriptor(ioat);
}
return NULL;
}
static struct dma_async_tx_descriptor *
-ioat1_dma_prep_memcpy(struct dma_chan *chan, dma_addr_t dma_dest,
+ioat1_dma_prep_memcpy(struct dma_chan *c, dma_addr_t dma_dest,
dma_addr_t dma_src, size_t len, unsigned long flags)
{
- struct ioat_dma_chan *ioat_chan = to_ioat_chan(chan);
+ struct ioat_dma_chan *ioat = to_ioat_chan(c);
struct ioat_desc_sw *desc;
size_t copy;
LIST_HEAD(chain);
@@ -880,14 +885,14 @@ ioat1_dma_prep_memcpy(struct dma_chan *chan, dma_addr_t dma_dest,
struct ioat_dma_descriptor *hw = NULL;
int tx_cnt = 0;
- spin_lock_bh(&ioat_chan->desc_lock);
- desc = ioat_dma_get_next_descriptor(ioat_chan);
+ spin_lock_bh(&ioat->desc_lock);
+ desc = ioat_dma_get_next_descriptor(ioat);
do {
if (!desc)
break;
tx_cnt++;
- copy = min_t(size_t, len, ioat_chan->xfercap);
+ copy = min_t(size_t, len, ioat->xfercap);
hw = desc->hw;
hw->size = copy;
@@ -904,7 +909,7 @@ ioat1_dma_prep_memcpy(struct dma_chan *chan, dma_addr_t dma_dest,
struct ioat_desc_sw *next;
async_tx_ack(&desc->txd);
- next = ioat_dma_get_next_descriptor(ioat_chan);
+ next = ioat_dma_get_next_descriptor(ioat);
hw->next = next ? next->txd.phys : 0;
desc = next;
} else
@@ -912,14 +917,16 @@ ioat1_dma_prep_memcpy(struct dma_chan *chan, dma_addr_t dma_dest,
} while (len);
if (!desc) {
- dev_err(to_dev(ioat_chan),
+ struct ioat_chan_common *chan = &ioat->base;
+
+ dev_err(to_dev(chan),
"chan%d - get_next_desc failed: %d descs waiting, %d total desc\n",
- chan_num(ioat_chan), ioat_chan->dmacount, ioat_chan->desccount);
- list_splice(&chain, &ioat_chan->free_desc);
- spin_unlock_bh(&ioat_chan->desc_lock);
+ chan_num(chan), ioat->dmacount, ioat->desccount);
+ list_splice(&chain, &ioat->free_desc);
+ spin_unlock_bh(&ioat->desc_lock);
return NULL;
}
- spin_unlock_bh(&ioat_chan->desc_lock);
+ spin_unlock_bh(&ioat->desc_lock);
desc->txd.flags = flags;
desc->tx_cnt = tx_cnt;
@@ -934,17 +941,17 @@ ioat1_dma_prep_memcpy(struct dma_chan *chan, dma_addr_t dma_dest,
}
static struct dma_async_tx_descriptor *
-ioat2_dma_prep_memcpy(struct dma_chan *chan, dma_addr_t dma_dest,
+ioat2_dma_prep_memcpy(struct dma_chan *c, dma_addr_t dma_dest,
dma_addr_t dma_src, size_t len, unsigned long flags)
{
- struct ioat_dma_chan *ioat_chan = to_ioat_chan(chan);
+ struct ioat_dma_chan *ioat = to_ioat_chan(c);
struct ioat_desc_sw *new;
- spin_lock_bh(&ioat_chan->desc_lock);
- new = ioat2_dma_get_next_descriptor(ioat_chan);
+ spin_lock_bh(&ioat->desc_lock);
+ new = ioat2_dma_get_next_descriptor(ioat);
/*
- * leave ioat_chan->desc_lock set in ioat 2 path
+ * leave ioat->desc_lock set in ioat 2 path
* it will get unlocked at end of tx_submit
*/
@@ -955,10 +962,12 @@ ioat2_dma_prep_memcpy(struct dma_chan *chan, dma_addr_t dma_dest,
new->txd.flags = flags;
return &new->txd;
} else {
- spin_unlock_bh(&ioat_chan->desc_lock);
- dev_err(to_dev(ioat_chan),
+ struct ioat_chan_common *chan = &ioat->base;
+
+ spin_unlock_bh(&ioat->desc_lock);
+ dev_err(to_dev(chan),
"chan%d - get_next_desc failed: %d descs waiting, %d total desc\n",
- chan_num(ioat_chan), ioat_chan->dmacount, ioat_chan->desccount);
+ chan_num(chan), ioat->dmacount, ioat->desccount);
return NULL;
}
}
@@ -968,20 +977,20 @@ static void ioat_dma_cleanup_tasklet(unsigned long data)
struct ioat_dma_chan *chan = (void *)data;
ioat_dma_memcpy_cleanup(chan);
writew(IOAT_CHANCTRL_INT_DISABLE,
- chan->reg_base + IOAT_CHANCTRL_OFFSET);
+ chan->base.reg_base + IOAT_CHANCTRL_OFFSET);
}
static void
-ioat_dma_unmap(struct ioat_dma_chan *ioat_chan, struct ioat_desc_sw *desc)
+ioat_dma_unmap(struct ioat_chan_common *chan, struct ioat_desc_sw *desc)
{
if (!(desc->txd.flags & DMA_COMPL_SKIP_DEST_UNMAP)) {
if (desc->txd.flags & DMA_COMPL_DEST_UNMAP_SINGLE)
- pci_unmap_single(ioat_chan->device->pdev,
+ pci_unmap_single(chan->device->pdev,
pci_unmap_addr(desc, dst),
pci_unmap_len(desc, len),
PCI_DMA_FROMDEVICE);
else
- pci_unmap_page(ioat_chan->device->pdev,
+ pci_unmap_page(chan->device->pdev,
pci_unmap_addr(desc, dst),
pci_unmap_len(desc, len),
PCI_DMA_FROMDEVICE);
@@ -989,12 +998,12 @@ ioat_dma_unmap(struct ioat_dma_chan *ioat_chan, struct ioat_desc_sw *desc)
if (!(desc->txd.flags & DMA_COMPL_SKIP_SRC_UNMAP)) {
if (desc->txd.flags & DMA_COMPL_SRC_UNMAP_SINGLE)
- pci_unmap_single(ioat_chan->device->pdev,
+ pci_unmap_single(chan->device->pdev,
pci_unmap_addr(desc, src),
pci_unmap_len(desc, len),
PCI_DMA_TODEVICE);
else
- pci_unmap_page(ioat_chan->device->pdev,
+ pci_unmap_page(chan->device->pdev,
pci_unmap_addr(desc, src),
pci_unmap_len(desc, len),
PCI_DMA_TODEVICE);
@@ -1005,8 +1014,9 @@ ioat_dma_unmap(struct ioat_dma_chan *ioat_chan, struct ioat_desc_sw *desc)
* ioat_dma_memcpy_cleanup - cleanup up finished descriptors
* @chan: ioat channel to be cleaned up
*/
-static void ioat_dma_memcpy_cleanup(struct ioat_dma_chan *ioat_chan)
+static void ioat_dma_memcpy_cleanup(struct ioat_dma_chan *ioat)
{
+ struct ioat_chan_common *chan = &ioat->base;
unsigned long phys_complete;
struct ioat_desc_sw *desc, *_desc;
dma_cookie_t cookie = 0;
@@ -1014,9 +1024,9 @@ static void ioat_dma_memcpy_cleanup(struct ioat_dma_chan *ioat_chan)
struct ioat_desc_sw *latest_desc;
struct dma_async_tx_descriptor *tx;
- prefetch(ioat_chan->completion_virt);
+ prefetch(chan->completion_virt);
- if (!spin_trylock_bh(&ioat_chan->cleanup_lock))
+ if (!spin_trylock_bh(&chan->cleanup_lock))
return;
/* The completion writeback can happen at any time,
@@ -1026,49 +1036,47 @@ static void ioat_dma_memcpy_cleanup(struct ioat_dma_chan *ioat_chan)
#if (BITS_PER_LONG == 64)
phys_complete =
- ioat_chan->completion_virt->full
+ chan->completion_virt->full
& IOAT_CHANSTS_COMPLETED_DESCRIPTOR_ADDR;
#else
- phys_complete =
- ioat_chan->completion_virt->low & IOAT_LOW_COMPLETION_MASK;
+ phys_complete = chan->completion_virt->low & IOAT_LOW_COMPLETION_MASK;
#endif
- if ((ioat_chan->completion_virt->full
+ if ((chan->completion_virt->full
& IOAT_CHANSTS_DMA_TRANSFER_STATUS) ==
IOAT_CHANSTS_DMA_TRANSFER_STATUS_HALTED) {
- dev_err(to_dev(ioat_chan), "Channel halted, chanerr = %x\n",
- readl(ioat_chan->reg_base + IOAT_CHANERR_OFFSET));
+ dev_err(to_dev(chan), "Channel halted, chanerr = %x\n",
+ readl(chan->reg_base + IOAT_CHANERR_OFFSET));
/* TODO do something to salvage the situation */
}
- if (phys_complete == ioat_chan->last_completion) {
- spin_unlock_bh(&ioat_chan->cleanup_lock);
+ if (phys_complete == chan->last_completion) {
+ spin_unlock_bh(&chan->cleanup_lock);
/*
* perhaps we're stuck so hard that the watchdog can't go off?
* try to catch it after 2 seconds
*/
- if (ioat_chan->device->version != IOAT_VER_3_0) {
+ if (chan->device->version != IOAT_VER_3_0) {
if (time_after(jiffies,
- ioat_chan->last_completion_time + HZ*WATCHDOG_DELAY)) {
- ioat_dma_chan_watchdog(&(ioat_chan->device->work.work));
- ioat_chan->last_completion_time = jiffies;
+ chan->last_completion_time + HZ*WATCHDOG_DELAY)) {
+ ioat_dma_chan_watchdog(&(chan->device->work.work));
+ chan->last_completion_time = jiffies;
}
}
return;
}
- ioat_chan->last_completion_time = jiffies;
+ chan->last_completion_time = jiffies;
cookie = 0;
- if (!spin_trylock_bh(&ioat_chan->desc_lock)) {
- spin_unlock_bh(&ioat_chan->cleanup_lock);
+ if (!spin_trylock_bh(&ioat->desc_lock)) {
+ spin_unlock_bh(&chan->cleanup_lock);
return;
}
- switch (ioat_chan->device->version) {
+ switch (chan->device->version) {
case IOAT_VER_1_2:
- list_for_each_entry_safe(desc, _desc,
- &ioat_chan->used_desc, node) {
+ list_for_each_entry_safe(desc, _desc, &ioat->used_desc, node) {
tx = &desc->txd;
/*
* Incoming DMA requests may use multiple descriptors,
@@ -1077,7 +1085,7 @@ static void ioat_dma_memcpy_cleanup(struct ioat_dma_chan *ioat_chan)
*/
if (tx->cookie) {
cookie = tx->cookie;
- ioat_dma_unmap(ioat_chan, desc);
+ ioat_dma_unmap(chan, desc);
if (tx->callback) {
tx->callback(tx->callback_param);
tx->callback = NULL;
@@ -1091,7 +1099,7 @@ static void ioat_dma_memcpy_cleanup(struct ioat_dma_chan *ioat_chan)
*/
if (async_tx_test_ack(tx)) {
list_move_tail(&desc->node,
- &ioat_chan->free_desc);
+ &ioat->free_desc);
} else
tx->cookie = 0;
} else {
@@ -1110,11 +1118,11 @@ static void ioat_dma_memcpy_cleanup(struct ioat_dma_chan *ioat_chan)
case IOAT_VER_2_0:
case IOAT_VER_3_0:
/* has some other thread has already cleaned up? */
- if (ioat_chan->used_desc.prev == NULL)
+ if (ioat->used_desc.prev == NULL)
break;
/* work backwards to find latest finished desc */
- desc = to_ioat_desc(ioat_chan->used_desc.next);
+ desc = to_ioat_desc(ioat->used_desc.next);
tx = &desc->txd;
latest_desc = NULL;
do {
@@ -1125,18 +1133,18 @@ static void ioat_dma_memcpy_cleanup(struct ioat_dma_chan *ioat_chan)
latest_desc = desc;
break;
}
- } while (&desc->node != ioat_chan->used_desc.prev);
+ } while (&desc->node != ioat->used_desc.prev);
if (latest_desc != NULL) {
/* work forwards to clear finished descriptors */
- for (desc = to_ioat_desc(ioat_chan->used_desc.prev);
+ for (desc = to_ioat_desc(ioat->used_desc.prev);
&desc->node != latest_desc->node.next &&
- &desc->node != ioat_chan->used_desc.next;
+ &desc->node != ioat->used_desc.next;
desc = to_ioat_desc(desc->node.next)) {
if (tx->cookie) {
cookie = tx->cookie;
tx->cookie = 0;
- ioat_dma_unmap(ioat_chan, desc);
+ ioat_dma_unmap(chan, desc);
if (tx->callback) {
tx->callback(tx->callback_param);
tx->callback = NULL;
@@ -1145,21 +1153,21 @@ static void ioat_dma_memcpy_cleanup(struct ioat_dma_chan *ioat_chan)
}
/* move used.prev up beyond those that are finished */
- if (&desc->node == ioat_chan->used_desc.next)
- ioat_chan->used_desc.prev = NULL;
+ if (&desc->node == ioat->used_desc.next)
+ ioat->used_desc.prev = NULL;
else
- ioat_chan->used_desc.prev = &desc->node;
+ ioat->used_desc.prev = &desc->node;
}
break;
}
- spin_unlock_bh(&ioat_chan->desc_lock);
+ spin_unlock_bh(&ioat->desc_lock);
- ioat_chan->last_completion = phys_complete;
+ chan->last_completion = phys_complete;
if (cookie != 0)
- ioat_chan->completed_cookie = cookie;
+ chan->completed_cookie = cookie;
- spin_unlock_bh(&ioat_chan->cleanup_lock);
+ spin_unlock_bh(&chan->cleanup_lock);
}
/**
@@ -1170,17 +1178,18 @@ static void ioat_dma_memcpy_cleanup(struct ioat_dma_chan *ioat_chan)
* @used: if not %NULL, updated with last used transaction
*/
static enum dma_status
-ioat_dma_is_complete(struct dma_chan *chan, dma_cookie_t cookie,
+ioat_dma_is_complete(struct dma_chan *c, dma_cookie_t cookie,
dma_cookie_t *done, dma_cookie_t *used)
{
- struct ioat_dma_chan *ioat_chan = to_ioat_chan(chan);
+ struct ioat_dma_chan *ioat = to_ioat_chan(c);
+ struct ioat_chan_common *chan = &ioat->base;
dma_cookie_t last_used;
dma_cookie_t last_complete;
enum dma_status ret;
- last_used = chan->cookie;
- last_complete = ioat_chan->completed_cookie;
- ioat_chan->watchdog_tcp_cookie = cookie;
+ last_used = c->cookie;
+ last_complete = chan->completed_cookie;
+ chan->watchdog_tcp_cookie = cookie;
if (done)
*done = last_complete;
@@ -1191,10 +1200,10 @@ ioat_dma_is_complete(struct dma_chan *chan, dma_cookie_t cookie,
if (ret == DMA_SUCCESS)
return ret;
- ioat_dma_memcpy_cleanup(ioat_chan);
+ ioat_dma_memcpy_cleanup(ioat);
- last_used = chan->cookie;
- last_complete = ioat_chan->completed_cookie;
+ last_used = c->cookie;
+ last_complete = chan->completed_cookie;
if (done)
*done = last_complete;
@@ -1204,19 +1213,20 @@ ioat_dma_is_complete(struct dma_chan *chan, dma_cookie_t cookie,
return dma_async_is_complete(cookie, last_complete, last_used);
}
-static void ioat_dma_start_null_desc(struct ioat_dma_chan *ioat_chan)
+static void ioat_dma_start_null_desc(struct ioat_dma_chan *ioat)
{
+ struct ioat_chan_common *chan = &ioat->base;
struct ioat_desc_sw *desc;
struct ioat_dma_descriptor *hw;
- spin_lock_bh(&ioat_chan->desc_lock);
+ spin_lock_bh(&ioat->desc_lock);
- desc = ioat_dma_get_next_descriptor(ioat_chan);
+ desc = ioat_dma_get_next_descriptor(ioat);
if (!desc) {
- dev_err(to_dev(ioat_chan),
+ dev_err(to_dev(chan),
"Unable to start null desc - get next desc failed\n");
- spin_unlock_bh(&ioat_chan->desc_lock);
+ spin_unlock_bh(&ioat->desc_lock);
return;
}
@@ -1230,31 +1240,31 @@ static void ioat_dma_start_null_desc(struct ioat_dma_chan *ioat_chan)
hw->src_addr = 0;
hw->dst_addr = 0;
async_tx_ack(&desc->txd);
- switch (ioat_chan->device->version) {
+ switch (chan->device->version) {
case IOAT_VER_1_2:
hw->next = 0;
- list_add_tail(&desc->node, &ioat_chan->used_desc);
+ list_add_tail(&desc->node, &ioat->used_desc);
writel(((u64) desc->txd.phys) & 0x00000000FFFFFFFF,
- ioat_chan->reg_base + IOAT1_CHAINADDR_OFFSET_LOW);
+ chan->reg_base + IOAT1_CHAINADDR_OFFSET_LOW);
writel(((u64) desc->txd.phys) >> 32,
- ioat_chan->reg_base + IOAT1_CHAINADDR_OFFSET_HIGH);
+ chan->reg_base + IOAT1_CHAINADDR_OFFSET_HIGH);
- writeb(IOAT_CHANCMD_START, ioat_chan->reg_base
- + IOAT_CHANCMD_OFFSET(ioat_chan->device->version));
+ writeb(IOAT_CHANCMD_START, chan->reg_base
+ + IOAT_CHANCMD_OFFSET(chan->device->version));
break;
case IOAT_VER_2_0:
case IOAT_VER_3_0:
writel(((u64) desc->txd.phys) & 0x00000000FFFFFFFF,
- ioat_chan->reg_base + IOAT2_CHAINADDR_OFFSET_LOW);
+ chan->reg_base + IOAT2_CHAINADDR_OFFSET_LOW);
writel(((u64) desc->txd.phys) >> 32,
- ioat_chan->reg_base + IOAT2_CHAINADDR_OFFSET_HIGH);
+ chan->reg_base + IOAT2_CHAINADDR_OFFSET_HIGH);
- ioat_chan->dmacount++;
- __ioat2_dma_memcpy_issue_pending(ioat_chan);
+ ioat->dmacount++;
+ __ioat2_dma_memcpy_issue_pending(ioat);
break;
}
- spin_unlock_bh(&ioat_chan->desc_lock);
+ spin_unlock_bh(&ioat->desc_lock);
}
/*
@@ -1371,7 +1381,7 @@ MODULE_PARM_DESC(ioat_interrupt_style,
*/
static int ioat_dma_setup_interrupts(struct ioatdma_device *device)
{
- struct ioat_dma_chan *ioat_chan;
+ struct ioat_chan_common *chan;
struct pci_dev *pdev = device->pdev;
struct device *dev = &pdev->dev;
struct msix_entry *msix;
@@ -1404,15 +1414,15 @@ msix:
for (i = 0; i < msixcnt; i++) {
msix = &device->msix_entries[i];
- ioat_chan = ioat_chan_by_index(device, i);
+ chan = ioat_chan_by_index(device, i);
err = devm_request_irq(dev, msix->vector,
ioat_dma_do_interrupt_msix, 0,
- "ioat-msix", ioat_chan);
+ "ioat-msix", chan);
if (err) {
for (j = 0; j < i; j++) {
msix = &device->msix_entries[j];
- ioat_chan = ioat_chan_by_index(device, j);
- devm_free_irq(dev, msix->vector, ioat_chan);
+ chan = ioat_chan_by_index(device, j);
+ devm_free_irq(dev, msix->vector, chan);
}
goto msix_single_vector;
}
@@ -1594,8 +1604,8 @@ int ioat2_dma_probe(struct ioatdma_device *device, int dca)
{
struct pci_dev *pdev = device->pdev;
struct dma_device *dma;
- struct dma_chan *chan;
- struct ioat_dma_chan *ioat_chan;
+ struct dma_chan *c;
+ struct ioat_chan_common *chan;
int err;
dma = &device->common;
@@ -1607,10 +1617,10 @@ int ioat2_dma_probe(struct ioatdma_device *device, int dca)
return err;
ioat_set_tcp_copy_break(2048);
- list_for_each_entry(chan, &dma->channels, device_node) {
- ioat_chan = to_ioat_chan(chan);
+ list_for_each_entry(c, &dma->channels, device_node) {
+ chan = to_chan_common(c);
writel(IOAT_DCACTRL_CMPL_WRITE_ENABLE | IOAT_DMA_DCA_ANY_CPU,
- ioat_chan->reg_base + IOAT_DCACTRL_OFFSET);
+ chan->reg_base + IOAT_DCACTRL_OFFSET);
}
err = ioat_register(device);
@@ -1629,8 +1639,8 @@ int ioat3_dma_probe(struct ioatdma_device *device, int dca)
{
struct pci_dev *pdev = device->pdev;
struct dma_device *dma;
- struct dma_chan *chan;
- struct ioat_dma_chan *ioat_chan;
+ struct dma_chan *c;
+ struct ioat_chan_common *chan;
int err;
u16 dev_id;
@@ -1656,10 +1666,10 @@ int ioat3_dma_probe(struct ioatdma_device *device, int dca)
return err;
ioat_set_tcp_copy_break(262144);
- list_for_each_entry(chan, &dma->channels, device_node) {
- ioat_chan = to_ioat_chan(chan);
+ list_for_each_entry(c, &dma->channels, device_node) {
+ chan = to_chan_common(c);
writel(IOAT_DMA_DCA_ANY_CPU,
- ioat_chan->reg_base + IOAT_DCACTRL_OFFSET);
+ chan->reg_base + IOAT_DCACTRL_OFFSET);
}
err = ioat_register(device);
@@ -1673,8 +1683,6 @@ int ioat3_dma_probe(struct ioatdma_device *device, int dca)
void ioat_dma_remove(struct ioatdma_device *device)
{
- struct dma_chan *chan, *_chan;
- struct ioat_dma_chan *ioat_chan;
struct dma_device *dma = &device->common;
if (device->version != IOAT_VER_3_0)
@@ -1687,9 +1695,6 @@ void ioat_dma_remove(struct ioatdma_device *device)
pci_pool_destroy(device->dma_pool);
pci_pool_destroy(device->completion_pool);
- list_for_each_entry_safe(chan, _chan, &dma->channels, device_node) {
- ioat_chan = to_ioat_chan(chan);
- list_del(&chan->device_node);
- }
+ INIT_LIST_HEAD(&dma->channels);
}
diff --git a/drivers/dma/ioat/dma.h b/drivers/dma/ioat/dma.h
index 9f0c853..5b31db7 100644
--- a/drivers/dma/ioat/dma.h
+++ b/drivers/dma/ioat/dma.h
@@ -35,7 +35,6 @@
#define IOAT_DMA_DCA_ANY_CPU ~0
#define IOAT_WATCHDOG_PERIOD (2 * HZ)
-#define to_ioat_chan(chan) container_of(chan, struct ioat_dma_chan, common)
#define to_ioatdma_device(dev) container_of(dev, struct ioatdma_device, common)
#define to_ioat_desc(lh) container_of(lh, struct ioat_desc_sw, node)
#define tx_to_ioat_desc(tx) container_of(tx, struct ioat_desc_sw, txd)
@@ -74,37 +73,24 @@ struct ioatdma_device {
u8 version;
struct delayed_work work;
struct msix_entry msix_entries[4];
- struct ioat_dma_chan *idx[4];
+ struct ioat_chan_common *idx[4];
struct dca_provider *dca;
void (*intr_quirk)(struct ioatdma_device *device);
};
-/**
- * struct ioat_dma_chan - internal representation of a DMA channel
- */
-struct ioat_dma_chan {
-
+struct ioat_chan_common {
void __iomem *reg_base;
- dma_cookie_t completed_cookie;
unsigned long last_completion;
unsigned long last_completion_time;
- size_t xfercap; /* XFERCAP register value expanded out */
-
spinlock_t cleanup_lock;
- spinlock_t desc_lock;
- struct list_head free_desc;
- struct list_head used_desc;
+ dma_cookie_t completed_cookie;
unsigned long watchdog_completion;
int watchdog_tcp_cookie;
u32 watchdog_last_tcp_cookie;
struct delayed_work work;
- int pending;
- u16 dmacount;
- u16 desccount;
-
struct ioatdma_device *device;
struct dma_chan common;
@@ -120,6 +106,35 @@ struct ioat_dma_chan {
struct tasklet_struct cleanup_task;
};
+/**
+ * struct ioat_dma_chan - internal representation of a DMA channel
+ */
+struct ioat_dma_chan {
+ struct ioat_chan_common base;
+
+ size_t xfercap; /* XFERCAP register value expanded out */
+
+ spinlock_t desc_lock;
+ struct list_head free_desc;
+ struct list_head used_desc;
+
+ int pending;
+ u16 dmacount;
+ u16 desccount;
+};
+
+static inline struct ioat_chan_common *to_chan_common(struct dma_chan *c)
+{
+ return container_of(c, struct ioat_chan_common, common);
+}
+
+static inline struct ioat_dma_chan *to_ioat_chan(struct dma_chan *c)
+{
+ struct ioat_chan_common *chan = to_chan_common(c);
+
+ return container_of(chan, struct ioat_dma_chan, base);
+}
+
/* wrapper around hardware descriptor format + additional software fields */
/**
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox