ipoib lockdep warning

[Zach Brown <zach.brown@oracle.com>]

I get an awfully verbose lockdep warning when I bring up ipoib devices
and wanted to make sure it didn't go unreported.  I've put it up at the
following URL:

  http://oss.oracle.com/~zab/ipoib-multicast-lockdep-warning.txt

I looked into it a bit and it seems to be on to something.  There might
be a AB, BC, CA ordering deadlock.

AB:

priv->lock is held while acquiring query_idr.lock

ipoib_mcast_send()
  spin_lock(&priv->lock);
  ipoib_mcast_sendonly_join()
    ib_sa_mcmember_rec_query()
      send_mad()
        idr_pre_get(&query_idr)
          spin_lock(&idp->lock);

BC:

query_idr.lock is taken with interrupts enabled and so is implicitly
ordered before dev->_xmit_lock which is taken in interrupt context.

ipoib_mcast_join_task()
  ipoib_mcast_join()
    ib_sa_mcmember_rec_query()
      send_mad()
        idr_pre_get(&query_idr)
          spin_lock(&idp->lock)

CA:

dev->_xmit_lock is held while acquiring priv->lock.  This triggers the
lockdep warning that adding the dep between dev->_xmit_lock and
priv->lock connects a soft-irq-safe lock to a soft-irq-unsafe one.

ipoib_mcast_restart_task()
  local_irq_save(flags);
  netif_tx_lock(dev)
    spin_lock(&dev->_xmit_lock);
  spin_lock(&priv->lock);

I can imagine all sorts of potential fixes (block ints when calling idr?
 reorder acquiry in ipoib_mcast_restart_task()?) but I'm operating on a
partial view of the paths here so I wasn't comfortable suggesting a fix.
 I wouldn't be surprised to hear that there are circumstances that both
lockdep and I don't know about that stop this from being a problem :).

In any case, it'd be fantastic if someone who knows this code could sit
down with lockdep and some ipoib regression suite to shake out lockdep's
complaints.

- z