* Re: [patch] infiniband: checking the wrong variable
From: Roland Dreier @ 2010-04-08 16:00 UTC (permalink / raw)
To: Bart Van Assche
Cc: Dan Carpenter, Roland Dreier, Sean Hefty, Hal Rosenstock,
Tejun Heo, Christoph Lameter, Jack Morgenstein,
linux-rdma-u79uwXL29TY76Z2rM5mHXA,
kernel-janitors-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <r2le2e108261004080807i36ad0755gad9843224b0c42fc-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
> It would be great if someone could explain the impact of this issue --
> the impact is not yet clear to me.
Um, it's a one-line patch -- pretty clear if the allocation fails then
we check the wrong value and probably crash.
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* RE: [PATCH 11/37] librdmacm: add zalloc call
From: Sean Hefty @ 2010-04-08 16:14 UTC (permalink / raw)
To: 'Jason Gunthorpe'; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20100407191938.GF15629-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
>Maybe use calloc(,1 sizeof(x)); instead?
yes - I'll update the patches, but won't repost for this change.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* IB_CQ_VECTOR_LEAST_ATTACHED in mainline?
From: Andy Grover @ 2010-04-08 19:50 UTC (permalink / raw)
To: Yevgeny Petrilin, linux-rdma
Hi Yevgeny,
When will this be targeted for mainline?
I'm working on using multiple vectors, and it would be great to see it
there, since I need to work against both OFED and mainline. Otherwise I
need to have my ulp do round robin assignment up to
caps.num_comp_vectors, I guess? VECTOR_LEAST_ATTACHED seems better.
Thanks -- Regards -- Andy
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2 12/51] IB/qib: Add qib_dma.c
From: Ralph Campbell @ 2010-04-08 20:40 UTC (permalink / raw)
To: Roland Dreier; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <adaljcz43n9.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
OK. This will be in v3.
On Wed, 2010-03-31 at 16:18 -0700, Roland Dreier wrote:
> > +struct ib_dma_mapping_ops qib_dma_mapping_ops = {
> > + qib_mapping_error,
> > + qib_dma_map_single,
> > + qib_dma_unmap_single,
> > + qib_dma_map_page,
> > + qib_dma_unmap_page,
> > + qib_map_sg,
> > + qib_unmap_sg,
> > + qib_sg_dma_address,
> > + qib_sg_dma_len,
> > + qib_sync_single_for_cpu,
> > + qib_sync_single_for_device,
> > + qib_dma_alloc_coherent,
> > + qib_dma_free_coherent
> > +};
>
> I think it would be better to use designated initializers (".xxx = yyy,")
> here to avoid problems if this struct ever changes layout.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2] IB/core: allow HCAs to create IB port sysfs files
From: Roland Dreier @ 2010-04-08 20:49 UTC (permalink / raw)
To: Ralph Campbell; +Cc: linux-rdma
In-Reply-To: <1258486848.992.407.camel-/vjeY7uYZjrPXfVEPVhPGq6RkeBMCJyt@public.gmane.org>
Rather than have the device driver call a new function that then calls
it back, how about if we just add a parameter to the
ib_register_device() call that is the callback to add the sysfs stuff?
This would let us avoid the race where the device is registered but not
all the sysfs stuff is created yet.
- R.
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2 38/51] IB/qib: Add qib_sysfs.c
From: Roland Dreier @ 2010-04-08 20:50 UTC (permalink / raw)
To: Ralph Campbell; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20091203190624.29507.22110.stgit-/vjeY7uYZjrPXfVEPVhPGq6RkeBMCJyt@public.gmane.org>
I didn't audit everything (so there may be more problems here too), but
certainly this violates the "one value per sysfs file" rule rather
egregiously:
> +static ssize_t show_stats(struct device *device, struct device_attribute *attr,
> + char *buf)
> +{
> + struct qib_ibdev *dev =
> + container_of(device, struct qib_ibdev, ibdev.dev);
> + struct qib_devdata *dd = dd_from_dev(dev);
> + unsigned pidx;
> + unsigned i;
> + int len = 0;
> + unsigned long flags;
> +
> + for (pidx = 0; pidx < dd->num_pports; ++pidx) {
> + struct qib_ibport *ibp = &dd->pport[pidx].ibport_data;
> +
> + len += sprintf(buf + len,
> + "Port %d:\n"
> + "RC timeouts %d\n"
> + "RC resends %d\n"
> + "RC QACKs %d\n"
> + "RC SEQ NAKs %d\n"
> + "RC RDMA seq %d\n"
> + "RC RNR NAKs %d\n"
> + "RC OTH NAKs %d\n"
> + "RC DComp %d\n"
> + "RCr dup req %d\n"
> + "RCr SEQ NAK %d\n"
> + "wait piobuf %d\n"
> + "wait DMA %d\n"
> + "wait TX %d\n"
> + "unaligned %d\n"
> + "loop pkts %d\n"
> + "PKT drops %d\n",
> + dd->pport[pidx].port,
> + ibp->n_timeouts, ibp->n_rc_resends,
> + ibp->n_rc_qacks, ibp->n_seq_naks,
> + ibp->n_rdma_seq, ibp->n_rnr_naks,
> + ibp->n_other_naks, ibp->n_rc_delayed_comp,
> + ibp->n_rc_dupreq, ibp->n_rc_seqnak,
> + dev->n_piowait, ibp->n_dmawait, dev->n_txwait,
> + ibp->n_unaligned, ibp->n_loop_pkts,
> + ibp->n_pkt_drops);
> + for (i = 0; i < ARRAY_SIZE(ibp->opstats); i++) {
> + const struct qib_opcode_stats *si = &ibp->opstats[i];
> +
> + if (!si->n_packets && !si->n_bytes)
> + continue;
> + len += sprintf(buf + len, "%02x %llu/%llu\n", i,
> + (unsigned long long) si->n_packets,
> + (unsigned long long) si->n_bytes);
> + }
> + }
> + len += sprintf(buf + len, "Ctx:npkts");
> + for (i = 0; i < dd->first_user_ctxt; i++) {
> + if (!dd->rcd[i])
> + continue;
> + len += sprintf(buf + len, " %u:%u", i,
> + dd->rcd[i]->pkt_count);
> + }
> + len += sprintf(buf + len, "\n");
> + spin_lock_irqsave(&dev->qpt_lock, flags);
> + for (i = 0; i < dev->qp_table_size; i++) {
> + struct qib_qp *qp;
> + for (qp = dev->qp_table[i]; qp != NULL; qp = qp->next) {
> + struct qib_swqe *wqe;
> +
> + if (qp->s_last == qp->s_acked &&
> + qp->s_acked == qp->s_cur &&
> + qp->s_cur == qp->s_tail &&
> + qp->s_tail == qp->s_head)
> + continue;
> + if (len + 128 >= PAGE_SIZE)
> + break;
> + wqe = get_swqe_ptr(qp, qp->s_last);
> + len += sprintf(buf + len,
> + "QP%u %s %u %u %u f=%x %u %u %u %u %u "
> + "PSN %x %x %x %x %x "
> + "(%u %u %u %u %u %u) QP%u LID %x\n",
> + qp->ibqp.qp_num,
> + qp_type_str[qp->ibqp.qp_type],
> + qp->state,
> + wqe->wr.opcode,
> + qp->s_hdrwords,
> + qp->s_flags,
> + atomic_read(&qp->s_dma_busy),
> + !list_empty(&qp->iowait),
> + qp->timeout,
> + wqe->ssn,
> + qp->s_lsn,
> + qp->s_last_psn,
> + qp->s_psn, qp->s_next_psn,
> + qp->s_sending_psn, qp->s_sending_hpsn,
> + qp->s_last, qp->s_acked, qp->s_cur,
> + qp->s_tail, qp->s_head, qp->s_size,
> + qp->remote_qpn,
> + qp->remote_ah_attr.dlid);
> + }
> + }
> + spin_unlock_irqrestore(&dev->qpt_lock, flags);
> + return len;
> +}
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2] IB/core: allow HCAs to create IB port sysfs files
From: Roland Dreier @ 2010-04-08 20:51 UTC (permalink / raw)
To: Ralph Campbell; +Cc: linux-rdma
In-Reply-To: <adamxxdwtys.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
> Rather than have the device driver call a new function that then calls
> it back, how about if we just add a parameter to the
> ib_register_device() call that is the callback to add the sysfs stuff?
> This would let us avoid the race where the device is registered but not
> all the sysfs stuff is created yet.
And as an improvement, maybe someday we could convert the existing
per-device driver-specific sysfs stuff to a similar approach.
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2 38/51] IB/qib: Add qib_sysfs.c
From: Ralph Campbell @ 2010-04-08 21:29 UTC (permalink / raw)
To: Roland Dreier; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <adaiq81wtw3.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
This was for debugging and clearly could use a better method.
Some stats are definitely useful and having files per value
would make scripting easier too.
I could add /sys/class/infiniband/qib0/ports/1/diag_counters/*
One other place that has multiple values is dumping the QSFP
data from the IB cable.
On Thu, 2010-04-08 at 13:50 -0700, Roland Dreier wrote:
> I didn't audit everything (so there may be more problems here too), but
> certainly this violates the "one value per sysfs file" rule rather
> egregiously:
>
> > +static ssize_t show_stats(struct device *device, struct device_attribute *attr,
> > + char *buf)
> > +{
> > + struct qib_ibdev *dev =
> > + container_of(device, struct qib_ibdev, ibdev.dev);
> > + struct qib_devdata *dd = dd_from_dev(dev);
> > + unsigned pidx;
> > + unsigned i;
> > + int len = 0;
> > + unsigned long flags;
> > +
> > + for (pidx = 0; pidx < dd->num_pports; ++pidx) {
> > + struct qib_ibport *ibp = &dd->pport[pidx].ibport_data;
> > +
> > + len += sprintf(buf + len,
> > + "Port %d:\n"
> > + "RC timeouts %d\n"
> > + "RC resends %d\n"
> > + "RC QACKs %d\n"
> > + "RC SEQ NAKs %d\n"
> > + "RC RDMA seq %d\n"
> > + "RC RNR NAKs %d\n"
> > + "RC OTH NAKs %d\n"
> > + "RC DComp %d\n"
> > + "RCr dup req %d\n"
> > + "RCr SEQ NAK %d\n"
> > + "wait piobuf %d\n"
> > + "wait DMA %d\n"
> > + "wait TX %d\n"
> > + "unaligned %d\n"
> > + "loop pkts %d\n"
> > + "PKT drops %d\n",
> > + dd->pport[pidx].port,
> > + ibp->n_timeouts, ibp->n_rc_resends,
> > + ibp->n_rc_qacks, ibp->n_seq_naks,
> > + ibp->n_rdma_seq, ibp->n_rnr_naks,
> > + ibp->n_other_naks, ibp->n_rc_delayed_comp,
> > + ibp->n_rc_dupreq, ibp->n_rc_seqnak,
> > + dev->n_piowait, ibp->n_dmawait, dev->n_txwait,
> > + ibp->n_unaligned, ibp->n_loop_pkts,
> > + ibp->n_pkt_drops);
> > + for (i = 0; i < ARRAY_SIZE(ibp->opstats); i++) {
> > + const struct qib_opcode_stats *si = &ibp->opstats[i];
> > +
> > + if (!si->n_packets && !si->n_bytes)
> > + continue;
> > + len += sprintf(buf + len, "%02x %llu/%llu\n", i,
> > + (unsigned long long) si->n_packets,
> > + (unsigned long long) si->n_bytes);
> > + }
> > + }
> > + len += sprintf(buf + len, "Ctx:npkts");
> > + for (i = 0; i < dd->first_user_ctxt; i++) {
> > + if (!dd->rcd[i])
> > + continue;
> > + len += sprintf(buf + len, " %u:%u", i,
> > + dd->rcd[i]->pkt_count);
> > + }
> > + len += sprintf(buf + len, "\n");
> > + spin_lock_irqsave(&dev->qpt_lock, flags);
> > + for (i = 0; i < dev->qp_table_size; i++) {
> > + struct qib_qp *qp;
> > + for (qp = dev->qp_table[i]; qp != NULL; qp = qp->next) {
> > + struct qib_swqe *wqe;
> > +
> > + if (qp->s_last == qp->s_acked &&
> > + qp->s_acked == qp->s_cur &&
> > + qp->s_cur == qp->s_tail &&
> > + qp->s_tail == qp->s_head)
> > + continue;
> > + if (len + 128 >= PAGE_SIZE)
> > + break;
> > + wqe = get_swqe_ptr(qp, qp->s_last);
> > + len += sprintf(buf + len,
> > + "QP%u %s %u %u %u f=%x %u %u %u %u %u "
> > + "PSN %x %x %x %x %x "
> > + "(%u %u %u %u %u %u) QP%u LID %x\n",
> > + qp->ibqp.qp_num,
> > + qp_type_str[qp->ibqp.qp_type],
> > + qp->state,
> > + wqe->wr.opcode,
> > + qp->s_hdrwords,
> > + qp->s_flags,
> > + atomic_read(&qp->s_dma_busy),
> > + !list_empty(&qp->iowait),
> > + qp->timeout,
> > + wqe->ssn,
> > + qp->s_lsn,
> > + qp->s_last_psn,
> > + qp->s_psn, qp->s_next_psn,
> > + qp->s_sending_psn, qp->s_sending_hpsn,
> > + qp->s_last, qp->s_acked, qp->s_cur,
> > + qp->s_tail, qp->s_head, qp->s_size,
> > + qp->remote_qpn,
> > + qp->remote_ah_attr.dlid);
> > + }
> > + }
> > + spin_unlock_irqrestore(&dev->qpt_lock, flags);
> > + return len;
> > +}
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2 38/51] IB/qib: Add qib_sysfs.c
From: Roland Dreier @ 2010-04-08 21:33 UTC (permalink / raw)
To: Ralph Campbell; +Cc: linux-rdma@vger.kernel.org
In-Reply-To: <1270762193.2278.25.camel-/vjeY7uYZjrPXfVEPVhPGq6RkeBMCJyt@public.gmane.org>
> This was for debugging and clearly could use a better method.
> Some stats are definitely useful and having files per value
> would make scripting easier too.
> I could add /sys/class/infiniband/qib0/ports/1/diag_counters/*
>
> One other place that has multiple values is dumping the QSFP
> data from the IB cable.
You can stick debugging data under debugfs. Or just kill it for now and
figure out how to add it back later.
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2 38/51] IB/qib: Add qib_sysfs.c
From: Jason Gunthorpe @ 2010-04-08 22:08 UTC (permalink / raw)
To: Ralph Campbell
Cc: Roland Dreier, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <1270762193.2278.25.camel-/vjeY7uYZjrPXfVEPVhPGq6RkeBMCJyt@public.gmane.org>
On Thu, Apr 08, 2010 at 02:29:53PM -0700, Ralph Campbell wrote:
> One other place that has multiple values is dumping the QSFP
> data from the IB cable.
I was going to comment that building your own I2C subsystem is kinda
strange, can't the in-kernel stuff be used?
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2 38/51] IB/qib: Add qib_sysfs.c
From: Ira Weiny @ 2010-04-08 22:12 UTC (permalink / raw)
To: Roland Dreier; +Cc: Ralph Campbell, linux-rdma@vger.kernel.org
In-Reply-To: <adask75vdbs.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
On Thu, 08 Apr 2010 14:33:59 -0700
Roland Dreier <rdreier-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> wrote:
> > This was for debugging and clearly could use a better method.
> > Some stats are definitely useful and having files per value
> > would make scripting easier too.
> > I could add /sys/class/infiniband/qib0/ports/1/diag_counters/*
> >
> > One other place that has multiple values is dumping the QSFP
> > data from the IB cable.
>
> You can stick debugging data under debugfs. Or just kill it for now and
> figure out how to add it back later.
I for one would rather not see this die. We have debugged some critical
issues using this data. The sysfs entries above are what Mellanox uses.
Should those also be changed?
Ira
> --
> Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
> http://*www.*cisco.com/web/about/doing_business/legal/cri/index.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://*vger.kernel.org/majordomo-info.html
>
--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
weiny2-i2BcT+NCU+M@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2 38/51] IB/qib: Add qib_sysfs.c
From: Ralph Campbell @ 2010-04-08 22:26 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Roland Dreier, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <20100408220840.GO9769-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
On Thu, 2010-04-08 at 15:08 -0700, Jason Gunthorpe wrote:
> On Thu, Apr 08, 2010 at 02:29:53PM -0700, Ralph Campbell wrote:
>
> > One other place that has multiple values is dumping the QSFP
> > data from the IB cable.
>
> I was going to comment that building your own I2C subsystem is kinda
> strange, can't the in-kernel stuff be used?
>
> Jason
It is a bit strange but we have had this discussion before with
the ib_ipath driver. Basically, the devices connected to the bus
weren't really I2C compliant. Since we had to support the
non-standard parts, it was easier to have only one set of code
that could handle both.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2 38/51] IB/qib: Add qib_sysfs.c
From: Jason Gunthorpe @ 2010-04-08 22:50 UTC (permalink / raw)
To: Ralph Campbell
Cc: Roland Dreier, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <1270765601.2278.31.camel-/vjeY7uYZjrPXfVEPVhPGq6RkeBMCJyt@public.gmane.org>
On Thu, Apr 08, 2010 at 03:26:41PM -0700, Ralph Campbell wrote:
> On Thu, 2010-04-08 at 15:08 -0700, Jason Gunthorpe wrote:
> > On Thu, Apr 08, 2010 at 02:29:53PM -0700, Ralph Campbell wrote:
> >
> > > One other place that has multiple values is dumping the QSFP
> > > data from the IB cable.
> >
> > I was going to comment that building your own I2C subsystem is kinda
> > strange, can't the in-kernel stuff be used?
> It is a bit strange but we have had this discussion before with
> the ib_ipath driver. Basically, the devices connected to the bus
> weren't really I2C compliant. Since we had to support the
> non-standard parts, it was easier to have only one set of code
> that could handle both.
Is this still true on qib? I didn't notice anything too obviously
weird, and QSFPs are clearly standard i2c..
The i2c layer has a pretty flexible definition of i2c these days..
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2 38/51] IB/qib: Add qib_sysfs.c
From: Ralph Campbell @ 2010-04-08 23:05 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Roland Dreier, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <20100408225002.GS9769-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
On Thu, 2010-04-08 at 15:50 -0700, Jason Gunthorpe wrote:
> On Thu, Apr 08, 2010 at 03:26:41PM -0700, Ralph Campbell wrote:
> > On Thu, 2010-04-08 at 15:08 -0700, Jason Gunthorpe wrote:
> > > On Thu, Apr 08, 2010 at 02:29:53PM -0700, Ralph Campbell wrote:
> > >
> > > > One other place that has multiple values is dumping the QSFP
> > > > data from the IB cable.
> > >
> > > I was going to comment that building your own I2C subsystem is kinda
> > > strange, can't the in-kernel stuff be used?
>
> > It is a bit strange but we have had this discussion before with
> > the ib_ipath driver. Basically, the devices connected to the bus
> > weren't really I2C compliant. Since we had to support the
> > non-standard parts, it was easier to have only one set of code
> > that could handle both.
>
> Is this still true on qib? I didn't notice anything too obviously
> weird, and QSFPs are clearly standard i2c..
>
> The i2c layer has a pretty flexible definition of i2c these days..
>
> Jason
I will take another look but don't be surprised if it takes me
awhile since I'm also working on addressing the other comments
too.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2 38/51] IB/qib: Add qib_sysfs.c
From: Roland Dreier @ 2010-04-08 23:29 UTC (permalink / raw)
To: Ira Weiny; +Cc: Ralph Campbell, linux-rdma@vger.kernel.org
In-Reply-To: <20100408151234.43f4bfc6.weiny2-i2BcT+NCU+M@public.gmane.org>
> I for one would rather not see this die. We have debugged some critical
> issues using this data. The sysfs entries above are what Mellanox uses.
> Should those also be changed?
Which sysfs entries? I don't see any likely looking code in either of
the Mellanox drivers.
- R.
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2] IB/core: allow HCAs to create IB port sysfs files
From: Ralph Campbell @ 2010-04-08 23:36 UTC (permalink / raw)
To: Roland Dreier; +Cc: linux-rdma
In-Reply-To: <adamxxdwtys.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
On Thu, 2010-04-08 at 13:49 -0700, Roland Dreier wrote:
> Rather than have the device driver call a new function that then calls
> it back, how about if we just add a parameter to the
> ib_register_device() call that is the callback to add the sysfs stuff?
> This would let us avoid the race where the device is registered but not
> all the sysfs stuff is created yet.
>
> - R.
I'm not sure this is any less of a race than what I proposed
or the calls to device_create_file() that most IB device drivers
make after calling ib_register_device().
Is the following what you would like to see?
Note that all the hw/*/ drivers have to be changed to call
ib_register_device() with the new parameter.
diff --git a/drivers/infiniband/core/core_priv.h b/drivers/infiniband/core/core_priv.h
index 05ac36e..e70c809 100644
--- a/drivers/infiniband/core/core_priv.h
+++ b/drivers/infiniband/core/core_priv.h
@@ -38,7 +38,7 @@
#include <rdma/ib_verbs.h>
-int ib_device_register_sysfs(struct ib_device *device);
+int ib_device_register_sysfs(struct ib_device *device, sysfs_cb cb);
void ib_device_unregister_sysfs(struct ib_device *device);
int ib_sysfs_setup(void);
diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index d1fba41..9ef8093 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -267,7 +267,7 @@ out:
* callback for each device that is added. @device must be allocated
* with ib_alloc_device().
*/
-int ib_register_device(struct ib_device *device)
+int ib_register_device(struct ib_device *device, sysfs_cb cb)
{
int ret;
@@ -296,7 +296,7 @@ int ib_register_device(struct ib_device *device)
goto out;
}
- ret = ib_device_register_sysfs(device);
+ ret = ib_device_register_sysfs(device, cb);
if (ret) {
printk(KERN_WARNING "Couldn't register device %s with driver model\n",
device->name);
diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c
index f901957..a47bac8 100644
--- a/drivers/infiniband/core/sysfs.c
+++ b/drivers/infiniband/core/sysfs.c
@@ -754,7 +754,23 @@ static struct attribute_group iw_stats_group = {
.attrs = iw_proto_stats_attrs,
};
-int ib_device_register_sysfs(struct ib_device *device)
+static int sysfs_create_port_files(struct ib_device *device, sysfs_cb cb)
+{
+ struct kobject *p;
+ struct ib_port *port;
+ int ret = 0;
+
+ list_for_each_entry(p, &device->port_list, entry) {
+ port = container_of(p, struct ib_port, kobj);
+ ret = cb(device, port->port_num, &port->kobj);
+ if (ret)
+ break;
+ }
+
+ return ret;
+}
+
+int ib_device_register_sysfs(struct ib_device *device, sysfs_cb cb)
{
struct device *class_dev = &device->dev;
int ret;
@@ -802,6 +818,12 @@ int ib_device_register_sysfs(struct ib_device *device)
goto err_put;
}
+ if (cb) {
+ ret = sysfs_create_port_files(device, cb);
+ if (ret)
+ goto err_put;
+ }
+
return 0;
err_put:
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 6ecd85c..4e446e4 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -1189,7 +1189,10 @@ struct ib_client {
struct ib_device *ib_alloc_device(size_t size);
void ib_dealloc_device(struct ib_device *device);
-int ib_register_device (struct ib_device *device);
+typedef int (*sysfs_cb)(struct ib_device *dev, u8 port_num,
+ struct kobject *kobj);
+
+int ib_register_device (struct ib_device *device, sysfs_cb cb);
void ib_unregister_device(struct ib_device *device);
int ib_register_client (struct ib_client *client);
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* Re: [PATCH v2 38/51] IB/qib: Add qib_sysfs.c
From: Ralph Campbell @ 2010-04-08 23:42 UTC (permalink / raw)
To: Roland Dreier
Cc: Ira Weiny, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <adapr29ttfh.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
On Thu, 2010-04-08 at 16:29 -0700, Roland Dreier wrote:
> > I for one would rather not see this die. We have debugged some critical
> > issues using this data. The sysfs entries above are what Mellanox uses.
> > Should those also be changed?
>
> Which sysfs entries? I don't see any likely looking code in either of
> the Mellanox drivers.
>
> - R.
I think Ira means:
# ls /sys/class/infiniband/mlx4_0/diag_counters/
clear_diag rq_num_lqpoe rq_num_udsdprd sq_num_lqpoe sq_num_rree
num_baddb rq_num_mce rq_num_wrfe sq_num_mwbe sq_num_rsync
num_cqovf rq_num_oos sq_num_bre sq_num_oos sq_num_tree
num_eqovf rq_num_rae sq_num_ieecne sq_num_rabrte sq_num_wrfe
rq_num_lae rq_num_rire sq_num_ieecse sq_num_rae
rq_num_leeoe rq_num_rnr sq_num_leeoe sq_num_rire
rq_num_lle rq_num_rsync sq_num_lle sq_num_rnr
rq_num_lpe rq_num_ucsdprd sq_num_lpe sq_num_roe
I couldn't find where these entries are added in the kernel.org code.
It is in the OFED tree though.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2 38/51] IB/qib: Add qib_sysfs.c
From: Ira Weiny @ 2010-04-08 23:53 UTC (permalink / raw)
To: Roland Dreier; +Cc: Ralph Campbell, linux-rdma@vger.kernel.org
In-Reply-To: <adapr29ttfh.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
On Thu, 08 Apr 2010 16:29:06 -0700
Roland Dreier <rdreier-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> wrote:
> > I for one would rather not see this die. We have debugged some critical
> > issues using this data. The sysfs entries above are what Mellanox uses.
> > Should those also be changed?
>
> Which sysfs entries? I don't see any likely looking code in either of
> the Mellanox drivers.
My apologies. It appears this an OFED thing.[*]
16:31:04 > ls -la /sys/class/infiniband/mlx4_0/diag_counters/
total 0
drwxr-xr-x 2 root root 0 Apr 8 16:31 ./
drwxr-xr-x 4 root root 0 Apr 8 16:31 ../
--w--w--w- 1 root root 4096 Apr 8 16:31 clear_diag
-r--r--r-- 1 root root 4096 Apr 8 16:31 num_baddb
-r--r--r-- 1 root root 4096 Apr 8 16:31 num_cqovf
...
This is with the current RHEL5.4 drivers.
<sigh>
So is there an equivalent for mlx4 in the upstream kernel? I don't see them.
Do you feel it is appropriate for the port counters to be in sysfs?
16:46:19 > ls -la /sys/class/infiniband/mlx4_0/ports/1/counters/
total 0
drwxr-xr-x 2 root root 0 Apr 8 16:46 ./
drwxr-xr-x 5 root root 0 Apr 6 09:33 ../
-r--r--r-- 1 root root 4096 Apr 8 16:46 VL15_dropped
-r--r--r-- 1 root root 4096 Apr 8 16:46 excessive_buffer_overrun_errors
...
Although these "diag_counters" are internal to the cards they do track counts
which are part of the IB spec like RNR NAK's. So I am not sure they are
really "debug" in the strictest sense.
I will rephrase my statement, I would like to see these counters included as
they give valuable information about what is going on in a cluster.
Ira
* OFED MUST die!
>
> - R.
>
> --
> Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
> http://*www.*cisco.com/web/about/doing_business/legal/cri/index.html
>
--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
weiny2-i2BcT+NCU+M@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Credit Inquiries Database - 1 million Full Data Records all with emails
From: Ava Connelly @ 2010-04-09 2:16 UTC (permalink / raw)
To: linux-parport-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
linux-pci-u79uwXL29TY76Z2rM5mHXA,
linux-pcmcia-owner-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
linux-pcmcia-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
linux-pm-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA
I have many good quality American Contact Lists at much lower prices than most sites. Contact me here: Addie.Cho-5YPRQHlKf8JHCvwvKO8SlA@public.gmane.org for a complete catalogue of what we have.
Best Regards
Laurence Page
to adjust your subscription status email to rembox-5YPRQHlKf8JHCvwvKO8SlA@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [GIT PULL] please pull infiniband.git
From: Roland Dreier @ 2010-04-09 16:13 UTC (permalink / raw)
To: torvalds-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b,
akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA
Linus, please pull from
git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git for-linus
This will get a few small fixes that have trickled in:
Chien Tung (1):
RDMA/nes: Correct cap.max_inline_data assignment in nes_query_qp()
Dan Carpenter (1):
IB/mlx4: Check correct variable for allocation failure
Jani Nikula (1):
IB/cm: Fix device_create() return value check
Roland Dreier (1):
Merge branches 'cma', 'misc', 'mlx4' and 'nes' into for-linus
shefty (1):
RDMA/cm: Set num_paths when manually assigning path records
drivers/infiniband/core/cm.c | 2 +-
drivers/infiniband/core/cma.c | 1 +
drivers/infiniband/hw/mlx4/mr.c | 2 +-
drivers/infiniband/hw/nes/nes_verbs.c | 9 ++++-----
4 files changed, 7 insertions(+), 7 deletions(-)
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [infiniband-diags] fix libibnetdisc corner case when user tries to re-cache a loaded cache
From: Al Chu @ 2010-04-09 17:24 UTC (permalink / raw)
To: sashak-smomgflXvOZWk0Htik3J/w@public.gmane.org
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
[-- Attachment #1: Type: text/plain, Size: 367 bytes --]
Hey Sasha,
Noticed this corner case. If I load a ibnetdiscover cache, then try to
re-cache it, the cache doesn't store things properly b/c I didn't
reconstruct one fabric data structure properly. This fixes that bug.
Al
--
Albert Chu
chu11-i2BcT+NCU+M@public.gmane.org
Computer Scientist
High Performance Systems Division
Lawrence Livermore National Laboratory
[-- Attachment #2: 0001-fix-libibnetdisc-corner-case-when-user-tries-to-re-c.patch --]
[-- Type: message/rfc822, Size: 1143 bytes --]
From: Albert Chu <chu11-i2BcT+NCU+M@public.gmane.org>
Subject: [PATCH] fix libibnetdisc corner case when user tries to re-cache a loaded cache
Date: Fri, 9 Apr 2010 10:12:47 -0700
Message-ID: <1270833600.17987.15.camel-X2zTWyBD0EhliZ7u+bvwcg@public.gmane.org>
Signed-off-by: Albert Chu <chu11-i2BcT+NCU+M@public.gmane.org>
---
.../libibnetdisc/src/ibnetdisc_cache.c | 4 ++++
1 files changed, 4 insertions(+), 0 deletions(-)
diff --git a/infiniband-diags/libibnetdisc/src/ibnetdisc_cache.c b/infiniband-diags/libibnetdisc/src/ibnetdisc_cache.c
index d94b85a..9cd5981 100644
--- a/infiniband-diags/libibnetdisc/src/ibnetdisc_cache.c
+++ b/infiniband-diags/libibnetdisc/src/ibnetdisc_cache.c
@@ -501,6 +501,10 @@ static int _fill_port(ibnd_fabric_cache_t * fabric_cache, ibnd_node_t * node,
node->ports[port_cache->port->portnum] = port_cache->port;
+ /* achu: needed if user wishes to re-cache a loaded fabric.
+ * Otherwise, mostly unnecessary to do this.
+ */
+ add_to_portguid_hash(port_cache->port, fabric_cache->fabric->portstbl);
return 0;
}
--
1.5.4.5
^ permalink raw reply related
* [PATCH 26/37 v2] librdmacm: set src_addr in rdma_getaddrinfo
From: Sean Hefty @ 2010-04-09 18:54 UTC (permalink / raw)
To: Hefty, Sean, linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <B7B5C575DF7D4C51AB84AFFAC7A5D5C5-Zpru7NauK7drdx17CPfAsdBPR1lH4CV8@public.gmane.org>
If the user provides a source address as a hint, copy
the address to rdma_addrinfo ai_src_addr.
Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
Changes from v1:
Only copy the source address if one is provided by the user though the hints.
Do not assign a source address otherwise. The kernel will resolve the
source address for us via rdma_resolve_addr().
src/addrinfo.c | 12 ++++++++++++
1 files changed, 12 insertions(+), 0 deletions(-)
diff --git a/src/addrinfo.c b/src/addrinfo.c
index 15ae071..f5f86a0 100644
--- a/src/addrinfo.c
+++ b/src/addrinfo.c
@@ -39,6 +39,7 @@
#include <sys/types.h>
#include <sys/socket.h>
#include <netdb.h>
+#include <unistd.h>
#include "cma.h"
#include <rdma/rdma_cma.h>
@@ -159,6 +160,17 @@ int rdma_getaddrinfo(char *node, char *service,
if (ret)
goto err2;
+ if (!rai->ai_src_len && hints && hints->ai_src_len) {
+ rai->ai_src_addr = calloc(1, hints->ai_src_len);
+ if (!rai->ai_src_addr) {
+ ret = ERR(ENOMEM);
+ goto err2;
+ }
+ memcpy(rai->ai_src_addr, hints->ai_src_addr,
+ hints->ai_src_len);
+ rai->ai_src_len = hints->ai_src_len;
+ }
+
freeaddrinfo(ai);
*res = rai;
return 0;
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH 27/37 v2] librdmacm: add support for IB ACM service
From: Sean Hefty @ 2010-04-09 18:56 UTC (permalink / raw)
To: Hefty, Sean, linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <63A656D773734ACAA7EA63BB2C02DDD6-Zpru7NauK7drdx17CPfAsdBPR1lH4CV8@public.gmane.org>
Allow the librdmacm to contact a service via sockets to obtain
address mapping and path record data. The use of the service
is controlled through a build option (with-ib_acm). If the
library fails to contact the service, it falls back to using
the kernel services to resolve address and routing data.
Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
Changes from v1:
ACM was modified to resolve the address and route using only the
destination address. If a source address is not given, then ACM
returns the source address that was used to resolve the route.
The source address belongs to the same address family as that
specified by the destination address.
Makefile.am | 2 -
configure.in | 14 ++++
src/acm.c | 195 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
src/addrinfo.c | 3 +
src/cma.c | 9 ++-
src/cma.h | 13 +++-
6 files changed, 232 insertions(+), 4 deletions(-)
diff --git a/Makefile.am b/Makefile.am
index be53c78..8d86045 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -12,7 +12,7 @@ else
librdmacm_version_script =
endif
-src_librdmacm_la_SOURCES = src/cma.c src/addrinfo.c
+src_librdmacm_la_SOURCES = src/cma.c src/addrinfo.c src/acm.c
src_librdmacm_la_LDFLAGS = -version-info 1 -export-dynamic \
$(librdmacm_version_script)
src_librdmacm_la_DEPENDENCIES = $(srcdir)/src/librdmacm.map
diff --git a/configure.in b/configure.in
index 1122966..3db4247 100644
--- a/configure.in
+++ b/configure.in
@@ -21,6 +21,15 @@ if test "$with_valgrind" != "" && test "$with_valgrind" != "no"; then
fi
fi
+AC_ARG_WITH([ib_acm],
+ AC_HELP_STRING([--with-ib_acm],
+ [Use IB ACM for route resolution - default NO]))
+
+if test "$with_ib_acm" != "" && test "$with_ib_acm" != "no"; then
+ AC_DEFINE([USE_IB_ACM], 1,
+ [Define to 1 to use IB ACM for endpoint resolution])
+fi
+
AC_ARG_ENABLE(libcheck, [ --disable-libcheck do not test for presence of ib libraries],
[ if test "$enableval" = "no"; then
disable_libcheck=yes
@@ -51,6 +60,11 @@ AC_CHECK_HEADER(valgrind/memcheck.h, [],
AC_MSG_ERROR([valgrind requested but <valgrind/memcheck.h> not found.]))
fi
+if test "$with_ib_acm" != "" && test "$with_ib_acm" != "no"; then
+AC_CHECK_HEADER(infiniband/acm.h, [],
+ AC_MSG_ERROR([IB ACM requested but <infiniband/acm.h> not found.]))
+fi
+
fi
AC_CACHE_CHECK(whether ld accepts --version-script, ac_cv_version_script,
diff --git a/src/acm.c b/src/acm.c
new file mode 100644
index 0000000..6fdd72e
--- /dev/null
+++ b/src/acm.c
@@ -0,0 +1,195 @@
+/*
+ * Copyright (c) 2010 Intel Corporation. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#if HAVE_CONFIG_H
+# include <config.h>
+#endif /* HAVE_CONFIG_H */
+
+#include <sys/types.h>
+#include <sys/socket.h>
+#include <netdb.h>
+#include <unistd.h>
+
+#include "cma.h"
+#include <rdma/rdma_cma.h>
+#include <infiniband/ib.h>
+#include <infiniband/sa.h>
+
+#ifdef USE_IB_ACM
+#include <infiniband/acm.h>
+
+static pthread_mutex_t acm_lock = PTHREAD_MUTEX_INITIALIZER;
+static int sock;
+static short server_port = 6125;
+
+void ucma_ib_init(void)
+{
+ struct sockaddr_in addr;
+ int ret;
+
+ sock = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
+ if (sock < 0)
+ return;
+
+ memset(&addr, 0, sizeof addr);
+ addr.sin_family = AF_INET;
+ addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
+ addr.sin_port = htons(server_port);
+ ret = connect(sock, (struct sockaddr *) &addr, sizeof(addr));
+ if (ret)
+ goto err;
+
+ return;
+
+err:
+ close(sock);
+ sock = 0;
+}
+
+void ucma_ib_cleanup(void)
+{
+ if (sock > 0) {
+ shutdown(sock, SHUT_RDWR);
+ close(sock);
+ }
+}
+
+static void ucma_ib_save_resp(struct rdma_addrinfo *rai, struct acm_resolve_msg *msg)
+{
+ struct ib_path_data *path_data = NULL;
+ int i, cnt, path_cnt;
+
+ cnt = (msg->hdr.length - ACM_MSG_HDR_LENGTH) / ACM_MSG_EP_LENGTH;
+ for (i = 0; i < cnt; i++) {
+ switch (msg->data[i].type) {
+ case ACM_EP_INFO_PATH:
+ msg->data[i].type = 0;
+ if (!path_data)
+ path_data = (struct ib_path_data *) &msg->data[i];
+ path_cnt++;
+ break;
+ case ACM_EP_INFO_ADDRESS_IP:
+ if (!(msg->data[i].flags & ACM_EP_FLAG_SOURCE) || rai->ai_src_len)
+ break;
+
+ rai->ai_src_addr = calloc(1, sizeof(struct sockaddr_in));
+ if (!rai->ai_src_addr)
+ break;
+
+ rai->ai_src_len = sizeof(struct sockaddr_in);
+ memcpy(&((struct sockaddr_in *) rai->ai_src_addr)->sin_addr,
+ &msg->data[i].info.addr, 4);
+ break;
+ case ACM_EP_INFO_ADDRESS_IP6:
+ if (!(msg->data[i].flags & ACM_EP_FLAG_SOURCE) || rai->ai_src_len)
+ break;
+
+ rai->ai_src_addr = calloc(1, sizeof(struct sockaddr_in6));
+ if (!rai->ai_src_addr)
+ break;
+
+ rai->ai_src_len = sizeof(struct sockaddr_in6);
+ memcpy(&((struct sockaddr_in6 *) rai->ai_src_addr)->sin6_addr,
+ &msg->data[i].info.addr, 16);
+ break;
+ default:
+ break;
+ }
+ }
+
+ rai->ai_route = calloc(path_cnt, sizeof(*path_data));
+ if (rai->ai_route) {
+ memcpy(rai->ai_route, path_data, path_cnt * sizeof(*path_data));
+ rai->ai_route_len = path_cnt * sizeof(*path_data);
+ }
+}
+
+void ucma_ib_resolve(struct rdma_addrinfo *rai)
+{
+ struct acm_msg msg;
+ struct acm_resolve_msg *resolve_msg = (struct acm_resolve_msg *) &msg;
+ struct acm_ep_addr_data *src_data, *dst_data;
+ int ret;
+
+ if (sock <= 0)
+ return;
+
+ memset(&msg, 0, sizeof msg);
+ msg.hdr.version = ACM_VERSION;
+ msg.hdr.opcode = ACM_OP_RESOLVE;
+
+ if (rai->ai_src_len) {
+ src_data = &resolve_msg->data[0];
+ src_data->flags = ACM_EP_FLAG_SOURCE;
+ if (rai->ai_family == AF_INET) {
+ src_data->type = ACM_EP_INFO_ADDRESS_IP;
+ memcpy(src_data->info.addr,
+ &((struct sockaddr_in *) rai->ai_src_addr)->sin_addr, 4);
+ } else {
+ src_data->type = ACM_EP_INFO_ADDRESS_IP6;
+ memcpy(src_data->info.addr,
+ &((struct sockaddr_in6 *) rai->ai_src_addr)->sin6_addr, 16);
+ }
+ dst_data = &resolve_msg->data[1];
+ msg.hdr.length = ACM_MSG_HDR_LENGTH + (2 * ACM_MSG_EP_LENGTH);
+ } else {
+ dst_data = &resolve_msg->data[0];
+ msg.hdr.length = ACM_MSG_HDR_LENGTH + ACM_MSG_EP_LENGTH;
+ }
+
+ dst_data->flags = ACM_EP_FLAG_DEST;
+ if (rai->ai_family == AF_INET) {
+ dst_data->type = ACM_EP_INFO_ADDRESS_IP;
+ memcpy(dst_data->info.addr,
+ &((struct sockaddr_in *) rai->ai_dst_addr)->sin_addr, 4);
+ } else {
+ dst_data->type = ACM_EP_INFO_ADDRESS_IP6;
+ memcpy(dst_data->info.addr,
+ &((struct sockaddr_in6 *) rai->ai_dst_addr)->sin6_addr, 16);
+ }
+
+ pthread_mutex_lock(&acm_lock);
+ ret = send(sock, (char *) &msg, msg.hdr.length, 0);
+ if (ret != msg.hdr.length) {
+ pthread_mutex_unlock(&acm_lock);
+ return;
+ }
+
+ ret = recv(sock, (char *) &msg, sizeof msg, 0);
+ pthread_mutex_unlock(&acm_lock);
+ if (ret < ACM_MSG_HDR_LENGTH || ret != msg.hdr.length || msg.hdr.status)
+ return;
+
+ ucma_ib_save_resp(rai, resolve_msg);
+}
+
+#endif /* USE_IB_ACM */
diff --git a/src/addrinfo.c b/src/addrinfo.c
index f5f86a0..c8d9f0c 100644
--- a/src/addrinfo.c
+++ b/src/addrinfo.c
@@ -171,6 +171,9 @@ int rdma_getaddrinfo(char *node, char *service,
rai->ai_src_len = hints->ai_src_len;
}
+ if (!(rai->ai_flags & RAI_PASSIVE))
+ ucma_ib_resolve(rai);
+
freeaddrinfo(ai);
*res = rai;
return 0;
diff --git a/src/cma.c b/src/cma.c
index a23e2eb..6dfb87c 100644
--- a/src/cma.c
+++ b/src/cma.c
@@ -149,6 +149,8 @@ int af_ib_support;
static void ucma_cleanup(void)
{
+ ucma_ib_cleanup();
+
if (cma_dev_cnt) {
while (cma_dev_cnt--) {
ibv_dealloc_pd(cma_dev_array[cma_dev_cnt].pd);
@@ -196,7 +198,7 @@ int ucma_init(void)
struct ibv_device **dev_list = NULL;
struct cma_device *cma_dev;
struct ibv_device_attr attr;
- int i, ret, dev_cnt;
+ int i, ret, dev_cnt, ib;
/* Quick check without lock to see if we're already initialized */
if (cma_dev_cnt)
@@ -225,7 +227,7 @@ int ucma_init(void)
goto err2;
}
- for (i = 0; dev_list[i];) {
+ for (i = 0, ib = 0; dev_list[i];) {
cma_dev = &cma_dev_array[i];
cma_dev->guid = ibv_get_device_guid(dev_list[i]);
@@ -253,8 +255,11 @@ int ucma_init(void)
cma_dev->port_cnt = attr.phys_port_cnt;
cma_dev->max_initiator_depth = (uint8_t) attr.max_qp_init_rd_atom;
cma_dev->max_responder_resources = (uint8_t) attr.max_qp_rd_atom;
+ ib += (cma_dev->verbs->device->transport_type == IBV_TRANSPORT_IB);
}
+ if (ib)
+ ucma_ib_init();
cma_dev_cnt = dev_cnt;
pthread_mutex_unlock(&mut);
ibv_free_device_list(dev_list);
diff --git a/src/cma.h b/src/cma.h
index 4e2312f..ba62456 100644
--- a/src/cma.h
+++ b/src/cma.h
@@ -43,6 +43,8 @@
#include <endian.h>
#include <byteswap.h>
+#include <rdma/rdma_cma.h>
+
#ifdef INCLUDE_VALGRIND
# include <valgrind/memcheck.h>
# ifndef VALGRIND_MAKE_MEM_DEFINED
@@ -74,5 +76,14 @@ static inline int ERR(int err)
int ucma_init();
-#endif /* CMA_H */
+#ifdef USE_IB_ACM
+void ucma_ib_init();
+void ucma_ib_cleanup();
+void ucma_ib_resolve(struct rdma_addrinfo *rai);
+#else
+#define ucma_ib_init()
+#define ucma_ib_cleanup()
+#define ucma_ib_resolve(x)
+#endif
+#endif /* CMA_H */
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* IB_QPT_RAW_ETY
From: Steve Wise @ 2010-04-09 18:57 UTC (permalink / raw)
To: Roland Dreier; +Cc: linux-rdma
Hey Roland,
Is QP type IB_QPT_RAW_ETY intended to provide a raw Ethernet service for
sending/receiving Ethernet frames?
Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2 26/51] IB/qib: Add qib_mr.c
From: Ralph Campbell @ 2010-04-09 19:04 UTC (permalink / raw)
To: Roland Dreier; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <adapr2b6gqa.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
On Tue, 2010-04-06 at 21:15 -0700, Roland Dreier wrote:
> > +struct ib_fmr *qib_alloc_fmr(struct ib_pd *pd, int mr_access_flags,
> > + struct ib_fmr_attr *fmr_attr)
>
> Instead of the crufty old FMR API, I think it would be preferable to
> implement the real IB spec fast registration through a work queue
> stuff. I understand that it might take some effort to do that but if we
> end up with our newest driver stuck with FMRs, we'll never be able to
> move away from that.
It looks like the IB_WR_FAST_REG_MR operation isn't too hard to
implement so I will add this support.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox