* Re: [PATCH] IB/hfi1: Define platform_config_table_limits once
From: Doug Ledford @ 2016-12-12 20:19 UTC (permalink / raw)
To: Bart Van Assche
Cc: Dennis Dalessandro, Dean Luick,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <870f083a-df8e-8a22-52c2-e4d2dde46b9d-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
[-- Attachment #1.1: Type: text/plain, Size: 696 bytes --]
On 12/5/2016 7:48 PM, Bart Van Assche wrote:
> Defining static data structures in a header file is wrong because
> this causes the data structure to be instantiated once in every .c
> file it is included in. Hence move the definition of a static
> array from a header file into the only .c file in which it is used.
>
> Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> Cc: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Cc: Dean Luick <dean.luick-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Applied, thanks.
--
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
GPG Key ID: 0E572FDD
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]
^ permalink raw reply
* Re: [PATCH 00/23] IB/hfi1,qib,rdmavt: Patches for 4.10
From: Doug Ledford @ 2016-12-12 20:18 UTC (permalink / raw)
To: Dennis Dalessandro
Cc: Mike Marciniszyn, Dean Luick, Jakub Pawlak, Tadeusz Struk,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, Ira Weiny, Mitko Haralanov,
Brian Welty, Kaike Wan, Ashutosh Dixit, Easwar Hariharan,
Harish Chegondi, Don Hiatt, Sebastian Sanchez, Jianxin Xiong
In-Reply-To: <20161208032312.21135.66056.stgit-9QXIwq+3FY+1XWohqUldA0EOCMrvLtNR@public.gmane.org>
[-- Attachment #1.1: Type: text/plain, Size: 594 bytes --]
On 12/7/2016 10:32 PM, Dennis Dalessandro wrote:
> Doug,
>
> Here is a set of patches for our drivers for 4.10. This is a
> collection of bug fixes, perf improvements, and clean ups.
> The patches apply on top of the last series I submitted. However
> if things do not apply cleanly let me know I can rebase.
>
> Can also be found in GitHub repo at:
> https://github.com/ddalessa/kernel/tree/for-4.10
Hi Dennis, I pulled all of these in plus a couple others from other people.
--
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
GPG Key ID: 0E572FDD
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]
^ permalink raw reply
* [PATCH] IB/vmw_pvrdma: Update the driver name to vmw_pvrdma
From: Adit Ranadive @ 2016-12-12 19:52 UTC (permalink / raw)
To: dledford-H+wXaHxf7aLQT0dZR+AlfA,
linux-rdma-u79uwXL29TY76Z2rM5mHXA
Cc: Adit Ranadive, pv-drivers-pghWNbHTmq7QT0dZR+AlfA
Hi Doug,
This patch got left out of the pvrdma->vmw_pvrdma series I sent out
before. Sorry about that.
Thanks,
Adit
---
Show the right driver name with the renamed module.
Signed-off-by: Adit Ranadive <aditr-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
---
drivers/infiniband/hw/vmw_pvrdma/pvrdma_main.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_main.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_main.c
index 5510c86..231a1ce 100644
--- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_main.c
+++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_main.c
@@ -55,7 +55,7 @@
#include "pvrdma.h"
-#define DRV_NAME "pvrdma"
+#define DRV_NAME "vmw_pvrdma"
#define DRV_VERSION "1.0.0.0-k"
static DEFINE_MUTEX(pvrdma_device_list_lock);
@@ -77,7 +77,7 @@ static int pvrdma_del_gid(struct ib_device *ibdev,
static ssize_t show_hca(struct device *device, struct device_attribute *attr,
char *buf)
{
- return sprintf(buf, "PVRDMA-%s\n", DRV_VERSION);
+ return sprintf(buf, "VMW_PVRDMA-%s\n", DRV_VERSION);
}
static ssize_t show_rev(struct device *device, struct device_attribute *attr,
@@ -166,7 +166,7 @@ static int pvrdma_register_device(struct pvrdma_dev *dev)
int ret = -1;
int i = 0;
- strlcpy(dev->ib_dev.name, "pvrdma%d", IB_DEVICE_NAME_MAX);
+ strlcpy(dev->ib_dev.name, "vmw_pvrdma%d", IB_DEVICE_NAME_MAX);
dev->ib_dev.node_guid = dev->dsr->caps.node_guid;
dev->sys_image_guid = dev->dsr->caps.sys_image_guid;
dev->flags = 0;
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* Re: [PATCH V2 13/22] bnxt_re: Support QP verbs
From: Leon Romanovsky @ 2016-12-12 18:27 UTC (permalink / raw)
To: Selvin Xavier
Cc: dledford-H+wXaHxf7aLQT0dZR+AlfA,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, netdev-u79uwXL29TY76Z2rM5mHXA,
Eddie Wai, Devesh Sharma, Somnath Kotur, Sriharsha Basavapatna
In-Reply-To: <1481266096-23331-14-git-send-email-selvin.xavier-dY08KVG/lbpWk0Htik3J/w@public.gmane.org>
[-- Attachment #1: Type: text/plain, Size: 68951 bytes --]
On Thu, Dec 08, 2016 at 10:48:07PM -0800, Selvin Xavier wrote:
> This patch implements create_qp, destroy_qp, query_qp and modify_qp verbs.
>
> v2: Fixed sparse warnings
>
> Signed-off-by: Eddie Wai <eddie.wai-dY08KVG/lbpWk0Htik3J/w@public.gmane.org>
> Signed-off-by: Devesh Sharma <devesh.sharma-dY08KVG/lbpWk0Htik3J/w@public.gmane.org>
> Signed-off-by: Somnath Kotur <somnath.kotur-dY08KVG/lbpWk0Htik3J/w@public.gmane.org>
> Signed-off-by: Sriharsha Basavapatna <sriharsha.basavapatna-dY08KVG/lbpWk0Htik3J/w@public.gmane.org>
> Signed-off-by: Selvin Xavier <selvin.xavier-dY08KVG/lbpWk0Htik3J/w@public.gmane.org>
> ---
> drivers/infiniband/hw/bnxtre/bnxt_qplib_fp.c | 873 ++++++++++++++++++++++++
> drivers/infiniband/hw/bnxtre/bnxt_qplib_fp.h | 250 +++++++
> drivers/infiniband/hw/bnxtre/bnxt_re.h | 14 +
> drivers/infiniband/hw/bnxtre/bnxt_re_ib_verbs.c | 762 +++++++++++++++++++++
> drivers/infiniband/hw/bnxtre/bnxt_re_ib_verbs.h | 21 +
> drivers/infiniband/hw/bnxtre/bnxt_re_main.c | 6 +
> include/uapi/rdma/bnxt_re_uverbs_abi.h | 10 +
> 7 files changed, 1936 insertions(+)
>
> diff --git a/drivers/infiniband/hw/bnxtre/bnxt_qplib_fp.c b/drivers/infiniband/hw/bnxtre/bnxt_qplib_fp.c
> index 636306f..edc9411 100644
> --- a/drivers/infiniband/hw/bnxtre/bnxt_qplib_fp.c
> +++ b/drivers/infiniband/hw/bnxtre/bnxt_qplib_fp.c
> @@ -50,6 +50,69 @@
> #include "bnxt_qplib_fp.h"
>
> static void bnxt_qplib_arm_cq_enable(struct bnxt_qplib_cq *cq);
> +
> +static void bnxt_qplib_free_qp_hdr_buf(struct bnxt_qplib_res *res,
> + struct bnxt_qplib_qp *qp)
> +{
> + struct bnxt_qplib_q *rq = &qp->rq;
> + struct bnxt_qplib_q *sq = &qp->sq;
> +
> + if (qp->rq_hdr_buf)
> + dma_free_coherent(&res->pdev->dev,
> + rq->hwq.max_elements * qp->rq_hdr_buf_size,
> + qp->rq_hdr_buf, qp->rq_hdr_buf_map);
> + if (qp->sq_hdr_buf)
> + dma_free_coherent(&res->pdev->dev,
> + sq->hwq.max_elements * qp->sq_hdr_buf_size,
> + qp->sq_hdr_buf, qp->sq_hdr_buf_map);
> + qp->rq_hdr_buf = NULL;
> + qp->sq_hdr_buf = NULL;
> + qp->rq_hdr_buf_map = 0;
> + qp->sq_hdr_buf_map = 0;
> + qp->sq_hdr_buf_size = 0;
> + qp->rq_hdr_buf_size = 0;
> +}
> +
> +static int bnxt_qplib_alloc_qp_hdr_buf(struct bnxt_qplib_res *res,
> + struct bnxt_qplib_qp *qp)
> +{
> + struct bnxt_qplib_q *rq = &qp->rq;
> + struct bnxt_qplib_q *sq = &qp->rq;
> + int rc = 0;
> +
> + if (qp->sq_hdr_buf_size && sq->hwq.max_elements) {
> + qp->sq_hdr_buf = dma_alloc_coherent(&res->pdev->dev,
> + sq->hwq.max_elements *
> + qp->sq_hdr_buf_size,
> + &qp->sq_hdr_buf_map, GFP_KERNEL);
> + if (!qp->sq_hdr_buf) {
> + rc = -ENOMEM;
> + dev_err(&res->pdev->dev,
> + "QPLIB: Failed to create sq_hdr_buf");
> + goto fail;
> + }
> + }
> +
> + if (qp->rq_hdr_buf_size && rq->hwq.max_elements) {
> + qp->rq_hdr_buf = dma_alloc_coherent(&res->pdev->dev,
> + rq->hwq.max_elements *
> + qp->rq_hdr_buf_size,
> + &qp->rq_hdr_buf_map,
> + GFP_KERNEL);
> + if (!qp->rq_hdr_buf) {
> + rc = -ENOMEM;
> + dev_err(&res->pdev->dev,
> + "QPLIB: Failed to create rq_hdr_buf");
> + goto fail;
> + }
> + }
> + return 0;
> +
> +fail:
> + bnxt_qplib_free_qp_hdr_buf(res, qp);
> + return rc;
> +}
> +
> static void bnxt_qplib_service_nq(unsigned long data)
> {
> struct bnxt_qplib_nq *nq = (struct bnxt_qplib_nq *)data;
> @@ -215,6 +278,816 @@ int bnxt_qplib_alloc_nq(struct pci_dev *pdev, struct bnxt_qplib_nq *nq)
> return 0;
> }
>
> +/* QP */
> +int bnxt_qplib_create_qp1(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp)
> +{
> + struct bnxt_qplib_rcfw *rcfw = res->rcfw;
> + struct cmdq_create_qp1 req;
> + struct creq_create_qp1_resp *resp;
> + struct bnxt_qplib_pbl *pbl;
> + struct bnxt_qplib_q *sq = &qp->sq;
> + struct bnxt_qplib_q *rq = &qp->rq;
> + int rc;
> + u16 cmd_flags = 0;
> + u32 qp_flags = 0;
> +
> + RCFW_CMD_PREP(req, CREATE_QP1, cmd_flags);
> +
> + /* General */
> + req.type = qp->type;
> + req.dpi = cpu_to_le32(qp->dpi->dpi);
> + req.qp_handle = cpu_to_le64(qp->qp_handle);
> +
> + /* SQ */
> + sq->hwq.max_elements = sq->max_wqe;
> + rc = bnxt_qplib_alloc_init_hwq(res->pdev, &sq->hwq, NULL, 0,
> + &sq->hwq.max_elements,
> + BNXT_QPLIB_MAX_SQE_ENTRY_SIZE, 0,
> + PAGE_SIZE, HWQ_TYPE_QUEUE);
> + if (rc)
> + goto exit;
> +
> + sq->swq = kcalloc(sq->hwq.max_elements, sizeof(*sq->swq), GFP_KERNEL);
> + if (!sq->swq) {
> + rc = -ENOMEM;
> + goto fail_sq;
> + }
> + pbl = &sq->hwq.pbl[PBL_LVL_0];
> + req.sq_pbl = cpu_to_le64(pbl->pg_map_arr[0]);
> + req.sq_pg_size_sq_lvl =
> + ((sq->hwq.level & CMDQ_CREATE_QP1_SQ_LVL_MASK)
> + << CMDQ_CREATE_QP1_SQ_LVL_SFT) |
> + (pbl->pg_size == ROCE_PG_SIZE_4K ?
> + CMDQ_CREATE_QP1_SQ_PG_SIZE_PG_4K :
> + pbl->pg_size == ROCE_PG_SIZE_8K ?
> + CMDQ_CREATE_QP1_SQ_PG_SIZE_PG_8K :
> + pbl->pg_size == ROCE_PG_SIZE_64K ?
> + CMDQ_CREATE_QP1_SQ_PG_SIZE_PG_64K :
> + pbl->pg_size == ROCE_PG_SIZE_2M ?
> + CMDQ_CREATE_QP1_SQ_PG_SIZE_PG_2M :
> + pbl->pg_size == ROCE_PG_SIZE_8M ?
> + CMDQ_CREATE_QP1_SQ_PG_SIZE_PG_8M :
> + pbl->pg_size == ROCE_PG_SIZE_1G ?
> + CMDQ_CREATE_QP1_SQ_PG_SIZE_PG_1G :
> + CMDQ_CREATE_QP1_SQ_PG_SIZE_PG_4K);
> +
> + if (qp->scq)
> + req.scq_cid = cpu_to_le32(qp->scq->id);
> +
> + qp_flags |= CMDQ_CREATE_QP1_QP_FLAGS_RESERVED_LKEY_ENABLE;
> +
> + /* RQ */
> + if (rq->max_wqe) {
> + rq->hwq.max_elements = qp->rq.max_wqe;
> + rc = bnxt_qplib_alloc_init_hwq(res->pdev, &rq->hwq, NULL, 0,
> + &rq->hwq.max_elements,
> + BNXT_QPLIB_MAX_RQE_ENTRY_SIZE, 0,
> + PAGE_SIZE, HWQ_TYPE_QUEUE);
> + if (rc)
> + goto fail_sq;
> +
> + rq->swq = kcalloc(rq->hwq.max_elements, sizeof(*rq->swq),
> + GFP_KERNEL);
> + if (!rq->swq) {
> + rc = -ENOMEM;
> + goto fail_rq;
> + }
> + pbl = &rq->hwq.pbl[PBL_LVL_0];
> + req.rq_pbl = cpu_to_le64(pbl->pg_map_arr[0]);
> + req.rq_pg_size_rq_lvl =
> + ((rq->hwq.level & CMDQ_CREATE_QP1_RQ_LVL_MASK) <<
> + CMDQ_CREATE_QP1_RQ_LVL_SFT) |
> + (pbl->pg_size == ROCE_PG_SIZE_4K ?
> + CMDQ_CREATE_QP1_RQ_PG_SIZE_PG_4K :
> + pbl->pg_size == ROCE_PG_SIZE_8K ?
> + CMDQ_CREATE_QP1_RQ_PG_SIZE_PG_8K :
> + pbl->pg_size == ROCE_PG_SIZE_64K ?
> + CMDQ_CREATE_QP1_RQ_PG_SIZE_PG_64K :
> + pbl->pg_size == ROCE_PG_SIZE_2M ?
> + CMDQ_CREATE_QP1_RQ_PG_SIZE_PG_2M :
> + pbl->pg_size == ROCE_PG_SIZE_8M ?
> + CMDQ_CREATE_QP1_RQ_PG_SIZE_PG_8M :
> + pbl->pg_size == ROCE_PG_SIZE_1G ?
> + CMDQ_CREATE_QP1_RQ_PG_SIZE_PG_1G :
> + CMDQ_CREATE_QP1_RQ_PG_SIZE_PG_4K);
> + if (qp->rcq)
> + req.rcq_cid = cpu_to_le32(qp->rcq->id);
> + }
> +
> + /* Header buffer - allow hdr_buf pass in */
> + rc = bnxt_qplib_alloc_qp_hdr_buf(res, qp);
> + if (rc) {
> + rc = -ENOMEM;
> + goto fail;
> + }
> + req.qp_flags = cpu_to_le32(qp_flags);
> + req.sq_size = cpu_to_le32(sq->hwq.max_elements);
> + req.rq_size = cpu_to_le32(rq->hwq.max_elements);
> +
> + req.sq_fwo_sq_sge =
> + cpu_to_le16((sq->max_sge & CMDQ_CREATE_QP1_SQ_SGE_MASK) <<
> + CMDQ_CREATE_QP1_SQ_SGE_SFT);
> + req.rq_fwo_rq_sge =
> + cpu_to_le16((rq->max_sge & CMDQ_CREATE_QP1_RQ_SGE_MASK) <<
> + CMDQ_CREATE_QP1_RQ_SGE_SFT);
> +
> + req.pd_id = cpu_to_le32(qp->pd->id);
> +
> + resp = (struct creq_create_qp1_resp *)
> + bnxt_qplib_rcfw_send_message(rcfw, (void *)&req,
> + NULL, 0);
> + if (!resp) {
> + dev_err(&res->pdev->dev, "QPLIB: FP: CREATE_QP1 send failed");
> + rc = -EINVAL;
> + goto fail;
> + }
> + /**/
It looks like you forgot to add a text into comment section.
> + if (!bnxt_qplib_rcfw_wait_for_resp(rcfw, le16_to_cpu(req.cookie))) {
> + /* Cmd timed out */
> + dev_err(&rcfw->pdev->dev, "QPLIB: FP: CREATE_QP1 timed out");
> + rc = -ETIMEDOUT;
> + goto fail;
> + }
> + if (RCFW_RESP_STATUS(resp) ||
> + RCFW_RESP_COOKIE(resp) != RCFW_CMDQ_COOKIE(req)) {
> + dev_err(&rcfw->pdev->dev, "QPLIB: FP: CREATE_QP1 failed ");
> + dev_err(&rcfw->pdev->dev,
> + "QPLIB: with status 0x%x cmdq 0x%x resp 0x%x",
> + RCFW_RESP_STATUS(resp), RCFW_CMDQ_COOKIE(req),
> + RCFW_RESP_COOKIE(resp));
> + rc = -EINVAL;
> + goto fail;
> + }
> + qp->id = le32_to_cpu(resp->xid);
> + qp->cur_qp_state = CMDQ_MODIFY_QP_NEW_STATE_RESET;
> + sq->flush_in_progress = false;
> + rq->flush_in_progress = false;
> +
> + return 0;
> +
> +fail:
> + bnxt_qplib_free_qp_hdr_buf(res, qp);
> +fail_rq:
> + bnxt_qplib_free_hwq(res->pdev, &rq->hwq);
> + kfree(rq->swq);
> +fail_sq:
> + bnxt_qplib_free_hwq(res->pdev, &sq->hwq);
> + kfree(sq->swq);
> +exit:
> + return rc;
> +}
> +
> +int bnxt_qplib_create_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp)
> +{
> + struct bnxt_qplib_rcfw *rcfw = res->rcfw;
> + struct sq_send *hw_sq_send_hdr, **hw_sq_send_ptr;
> + struct cmdq_create_qp req;
> + struct creq_create_qp_resp *resp;
> + struct bnxt_qplib_pbl *pbl;
> + struct sq_psn_search **psn_search_ptr;
> + unsigned long long int psn_search, poff = 0;
> + struct bnxt_qplib_q *sq = &qp->sq;
> + struct bnxt_qplib_q *rq = &qp->rq;
> + struct bnxt_qplib_hwq *xrrq;
> + int i, rc, req_size, psn_sz;
> + u16 cmd_flags = 0, max_ssge;
> + u32 sw_prod, qp_flags = 0;
> +
> + RCFW_CMD_PREP(req, CREATE_QP, cmd_flags);
> +
> + /* General */
> + req.type = qp->type;
> + req.dpi = cpu_to_le32(qp->dpi->dpi);
> + req.qp_handle = cpu_to_le64(qp->qp_handle);
> +
> + /* SQ */
> + psn_sz = (qp->type == CMDQ_CREATE_QP_TYPE_RC) ?
> + sizeof(struct sq_psn_search) : 0;
> + sq->hwq.max_elements = sq->max_wqe;
> + rc = bnxt_qplib_alloc_init_hwq(res->pdev, &sq->hwq, sq->sglist,
> + sq->nmap, &sq->hwq.max_elements,
> + BNXT_QPLIB_MAX_SQE_ENTRY_SIZE,
> + psn_sz,
> + PAGE_SIZE, HWQ_TYPE_QUEUE);
> + if (rc)
> + goto exit;
> +
> + sq->swq = kcalloc(sq->hwq.max_elements, sizeof(*sq->swq), GFP_KERNEL);
> + if (!sq->swq) {
> + rc = -ENOMEM;
> + goto fail_sq;
> + }
> + hw_sq_send_ptr = (struct sq_send **)sq->hwq.pbl_ptr;
> + if (psn_sz) {
> + psn_search_ptr = (struct sq_psn_search **)
> + &hw_sq_send_ptr[SQE_PG(sq->hwq.max_elements)];
> + psn_search = (unsigned long long int)
> + &hw_sq_send_ptr[SQE_PG(sq->hwq.max_elements)]
> + [SQE_IDX(sq->hwq.max_elements)];
> + if (psn_search & ~PAGE_MASK) {
> + /* If the psn_search does not start on a page boundary,
> + * then calculate the offset
> + */
> + poff = (psn_search & ~PAGE_MASK) /
> + BNXT_QPLIB_MAX_PSNE_ENTRY_SIZE;
> + }
> + for (i = 0; i < sq->hwq.max_elements; i++)
> + sq->swq[i].psn_search =
> + &psn_search_ptr[PSNE_PG(i + poff)]
> + [PSNE_IDX(i + poff)];
> + }
> + pbl = &sq->hwq.pbl[PBL_LVL_0];
> + req.sq_pbl = cpu_to_le64(pbl->pg_map_arr[0]);
> + req.sq_pg_size_sq_lvl =
> + ((sq->hwq.level & CMDQ_CREATE_QP_SQ_LVL_MASK)
> + << CMDQ_CREATE_QP_SQ_LVL_SFT) |
> + (pbl->pg_size == ROCE_PG_SIZE_4K ?
> + CMDQ_CREATE_QP_SQ_PG_SIZE_PG_4K :
> + pbl->pg_size == ROCE_PG_SIZE_8K ?
> + CMDQ_CREATE_QP_SQ_PG_SIZE_PG_8K :
> + pbl->pg_size == ROCE_PG_SIZE_64K ?
> + CMDQ_CREATE_QP_SQ_PG_SIZE_PG_64K :
> + pbl->pg_size == ROCE_PG_SIZE_2M ?
> + CMDQ_CREATE_QP_SQ_PG_SIZE_PG_2M :
> + pbl->pg_size == ROCE_PG_SIZE_8M ?
> + CMDQ_CREATE_QP_SQ_PG_SIZE_PG_8M :
> + pbl->pg_size == ROCE_PG_SIZE_1G ?
> + CMDQ_CREATE_QP_SQ_PG_SIZE_PG_1G :
> + CMDQ_CREATE_QP_SQ_PG_SIZE_PG_4K);
> +
> + /* initialize all SQ WQEs to LOCAL_INVALID (sq prep for hw fetch) */
> + hw_sq_send_ptr = (struct sq_send **)sq->hwq.pbl_ptr;
> + for (sw_prod = 0; sw_prod < sq->hwq.max_elements; sw_prod++) {
> + hw_sq_send_hdr = &hw_sq_send_ptr[SQE_PG(sw_prod)]
> + [SQE_IDX(sw_prod)];
> + hw_sq_send_hdr->wqe_type = SQ_BASE_WQE_TYPE_LOCAL_INVALID;
> + }
> +
> + if (qp->scq)
> + req.scq_cid = cpu_to_le32(qp->scq->id);
> +
> + qp_flags |= CMDQ_CREATE_QP_QP_FLAGS_RESERVED_LKEY_ENABLE;
> + qp_flags |= CMDQ_CREATE_QP_QP_FLAGS_FR_PMR_ENABLED;
> + if (qp->sig_type)
> + qp_flags |= CMDQ_CREATE_QP_QP_FLAGS_FORCE_COMPLETION;
> +
> + /* RQ */
> + if (rq->max_wqe) {
> + rq->hwq.max_elements = rq->max_wqe;
> + rc = bnxt_qplib_alloc_init_hwq(res->pdev, &rq->hwq, rq->sglist,
> + rq->nmap, &rq->hwq.max_elements,
> + BNXT_QPLIB_MAX_RQE_ENTRY_SIZE, 0,
> + PAGE_SIZE, HWQ_TYPE_QUEUE);
> + if (rc)
> + goto fail_sq;
> +
> + rq->swq = kcalloc(rq->hwq.max_elements, sizeof(*rq->swq),
> + GFP_KERNEL);
> + if (!rq->swq) {
> + rc = -ENOMEM;
> + goto fail_rq;
> + }
> + pbl = &rq->hwq.pbl[PBL_LVL_0];
> + req.rq_pbl = cpu_to_le64(pbl->pg_map_arr[0]);
> + req.rq_pg_size_rq_lvl =
> + ((rq->hwq.level & CMDQ_CREATE_QP_RQ_LVL_MASK) <<
> + CMDQ_CREATE_QP_RQ_LVL_SFT) |
> + (pbl->pg_size == ROCE_PG_SIZE_4K ?
> + CMDQ_CREATE_QP_RQ_PG_SIZE_PG_4K :
> + pbl->pg_size == ROCE_PG_SIZE_8K ?
> + CMDQ_CREATE_QP_RQ_PG_SIZE_PG_8K :
> + pbl->pg_size == ROCE_PG_SIZE_64K ?
> + CMDQ_CREATE_QP_RQ_PG_SIZE_PG_64K :
> + pbl->pg_size == ROCE_PG_SIZE_2M ?
> + CMDQ_CREATE_QP_RQ_PG_SIZE_PG_2M :
> + pbl->pg_size == ROCE_PG_SIZE_8M ?
> + CMDQ_CREATE_QP_RQ_PG_SIZE_PG_8M :
> + pbl->pg_size == ROCE_PG_SIZE_1G ?
> + CMDQ_CREATE_QP_RQ_PG_SIZE_PG_1G :
> + CMDQ_CREATE_QP_RQ_PG_SIZE_PG_4K);
> + }
> +
> + if (qp->rcq)
> + req.rcq_cid = cpu_to_le32(qp->rcq->id);
> + req.qp_flags = cpu_to_le32(qp_flags);
> + req.sq_size = cpu_to_le32(sq->hwq.max_elements);
> + req.rq_size = cpu_to_le32(rq->hwq.max_elements);
> + qp->sq_hdr_buf = NULL;
> + qp->rq_hdr_buf = NULL;
> +
> + rc = bnxt_qplib_alloc_qp_hdr_buf(res, qp);
> + if (rc)
> + goto fail_rq;
> +
> + /* CTRL-22434: Irrespective of the requested SGE count on the SQ
> + * always create the QP with max send sges possible if the requested
> + * inline size is greater than 0.
> + */
> + max_ssge = qp->max_inline_data ? 6 : sq->max_sge;
> + req.sq_fwo_sq_sge = cpu_to_le16(
> + ((max_ssge & CMDQ_CREATE_QP_SQ_SGE_MASK)
> + << CMDQ_CREATE_QP_SQ_SGE_SFT) | 0);
> + req.rq_fwo_rq_sge = cpu_to_le16(
> + ((rq->max_sge & CMDQ_CREATE_QP_RQ_SGE_MASK)
> + << CMDQ_CREATE_QP_RQ_SGE_SFT) | 0);
> + /* ORRQ and IRRQ */
> + if (psn_sz) {
> + xrrq = &qp->orrq;
> + xrrq->max_elements =
> + ORD_LIMIT_TO_ORRQ_SLOTS(qp->max_rd_atomic);
> + req_size = xrrq->max_elements *
> + BNXT_QPLIB_MAX_ORRQE_ENTRY_SIZE + PAGE_SIZE - 1;
> + req_size &= ~(PAGE_SIZE - 1);
> + rc = bnxt_qplib_alloc_init_hwq(res->pdev, xrrq, NULL, 0,
> + &xrrq->max_elements,
> + BNXT_QPLIB_MAX_ORRQE_ENTRY_SIZE,
> + 0, req_size, HWQ_TYPE_CTX);
> + if (rc)
> + goto fail_buf_free;
> + pbl = &xrrq->pbl[PBL_LVL_0];
> + req.orrq_addr = cpu_to_le64(pbl->pg_map_arr[0]);
> +
> + xrrq = &qp->irrq;
> + xrrq->max_elements = IRD_LIMIT_TO_IRRQ_SLOTS(
> + qp->max_dest_rd_atomic);
> + req_size = xrrq->max_elements *
> + BNXT_QPLIB_MAX_IRRQE_ENTRY_SIZE + PAGE_SIZE - 1;
> + req_size &= ~(PAGE_SIZE - 1);
> +
> + rc = bnxt_qplib_alloc_init_hwq(res->pdev, xrrq, NULL, 0,
> + &xrrq->max_elements,
> + BNXT_QPLIB_MAX_IRRQE_ENTRY_SIZE,
> + 0, req_size, HWQ_TYPE_CTX);
> + if (rc)
> + goto fail_orrq;
> +
> + pbl = &xrrq->pbl[PBL_LVL_0];
> + req.irrq_addr = cpu_to_le64(pbl->pg_map_arr[0]);
> + }
> + req.pd_id = cpu_to_le32(qp->pd->id);
> +
> + resp = (struct creq_create_qp_resp *)
> + bnxt_qplib_rcfw_send_message(rcfw, (void *)&req,
> + NULL, 0);
> + if (!resp) {
> + dev_err(&rcfw->pdev->dev, "QPLIB: FP: CREATE_QP send failed");
> + rc = -EINVAL;
> + goto fail;
> + }
> + /**/
> + if (!bnxt_qplib_rcfw_wait_for_resp(rcfw, le16_to_cpu(req.cookie))) {
> + /* Cmd timed out */
> + dev_err(&rcfw->pdev->dev, "QPLIB: FP: CREATE_QP timed out");
> + rc = -ETIMEDOUT;
> + goto fail;
> + }
> + if (RCFW_RESP_STATUS(resp) ||
> + RCFW_RESP_COOKIE(resp) != RCFW_CMDQ_COOKIE(req)) {
> + dev_err(&rcfw->pdev->dev, "QPLIB: FP: CREATE_QP failed ");
> + dev_err(&rcfw->pdev->dev,
> + "QPLIB: with status 0x%x cmdq 0x%x resp 0x%x",
> + RCFW_RESP_STATUS(resp), RCFW_CMDQ_COOKIE(req),
> + RCFW_RESP_COOKIE(resp));
> + rc = -EINVAL;
> + goto fail;
> + }
> + qp->id = le32_to_cpu(resp->xid);
> + qp->cur_qp_state = CMDQ_MODIFY_QP_NEW_STATE_RESET;
> + sq->flush_in_progress = false;
> + rq->flush_in_progress = false;
> +
> + return 0;
> +
> +fail:
> + if (qp->irrq.max_elements)
> + bnxt_qplib_free_hwq(res->pdev, &qp->irrq);
> +fail_orrq:
> + if (qp->orrq.max_elements)
> + bnxt_qplib_free_hwq(res->pdev, &qp->orrq);
> +fail_buf_free:
> + bnxt_qplib_free_qp_hdr_buf(res, qp);
> +fail_rq:
> + bnxt_qplib_free_hwq(res->pdev, &rq->hwq);
> + kfree(rq->swq);
> +fail_sq:
> + bnxt_qplib_free_hwq(res->pdev, &sq->hwq);
> + kfree(sq->swq);
> +exit:
> + return rc;
> +}
> +
> +static void __filter_modify_flags(struct bnxt_qplib_qp *qp)
> +{
It can help to review if you break this function into smaller pieces and
get rid of switch->switch->if construction.
> + switch (qp->cur_qp_state) {
> + case CMDQ_MODIFY_QP_NEW_STATE_RESET:
> + switch (qp->state) {
> + case CMDQ_MODIFY_QP_NEW_STATE_INIT:
> + break;
> + default:
> + break;
> + }
> + break;
> + case CMDQ_MODIFY_QP_NEW_STATE_INIT:
> + switch (qp->state) {
> + case CMDQ_MODIFY_QP_NEW_STATE_RTR:
> + /* INIT->RTR, configure the path_mtu to the default
> + * 2048 if not being requested
> + */
> + if (!(qp->modify_flags &
> + CMDQ_MODIFY_QP_MODIFY_MASK_PATH_MTU)) {
> + qp->modify_flags |=
> + CMDQ_MODIFY_QP_MODIFY_MASK_PATH_MTU;
> + qp->path_mtu = CMDQ_MODIFY_QP_PATH_MTU_MTU_2048;
> + }
> + qp->modify_flags &=
> + ~CMDQ_MODIFY_QP_MODIFY_MASK_VLAN_ID;
> + /* Bono FW requires the max_dest_rd_atomic to be >= 1 */
> + if (qp->max_dest_rd_atomic < 1)
> + qp->max_dest_rd_atomic = 1;
> + qp->modify_flags &= ~CMDQ_MODIFY_QP_MODIFY_MASK_SRC_MAC;
> + /* Bono FW 20.6.5 requires SGID_INDEX configuration */
> + if (!(qp->modify_flags &
> + CMDQ_MODIFY_QP_MODIFY_MASK_SGID_INDEX)) {
> + qp->modify_flags |=
> + CMDQ_MODIFY_QP_MODIFY_MASK_SGID_INDEX;
> + qp->ah.sgid_index = 0;
> + }
> + break;
> + default:
> + break;
> + }
> + break;
> + case CMDQ_MODIFY_QP_NEW_STATE_RTR:
> + switch (qp->state) {
> + case CMDQ_MODIFY_QP_NEW_STATE_RTS:
> + /* Bono FW requires the max_rd_atomic to be >= 1 */
> + if (qp->max_rd_atomic < 1)
> + qp->max_rd_atomic = 1;
> + /* Bono FW does not allow PKEY_INDEX,
> + * DGID, FLOW_LABEL, SGID_INDEX, HOP_LIMIT,
> + * TRAFFIC_CLASS, DEST_MAC, PATH_MTU, RQ_PSN,
> + * MIN_RNR_TIMER, MAX_DEST_RD_ATOMIC, DEST_QP_ID
> + * modification
> + */
> + qp->modify_flags &=
> + ~(CMDQ_MODIFY_QP_MODIFY_MASK_PKEY |
> + CMDQ_MODIFY_QP_MODIFY_MASK_DGID |
> + CMDQ_MODIFY_QP_MODIFY_MASK_FLOW_LABEL |
> + CMDQ_MODIFY_QP_MODIFY_MASK_SGID_INDEX |
> + CMDQ_MODIFY_QP_MODIFY_MASK_HOP_LIMIT |
> + CMDQ_MODIFY_QP_MODIFY_MASK_TRAFFIC_CLASS |
> + CMDQ_MODIFY_QP_MODIFY_MASK_DEST_MAC |
> + CMDQ_MODIFY_QP_MODIFY_MASK_PATH_MTU |
> + CMDQ_MODIFY_QP_MODIFY_MASK_RQ_PSN |
> + CMDQ_MODIFY_QP_MODIFY_MASK_MIN_RNR_TIMER |
> + CMDQ_MODIFY_QP_MODIFY_MASK_MAX_DEST_RD_ATOMIC
> + | CMDQ_MODIFY_QP_MODIFY_MASK_DEST_QP_ID);
> + break;
> + default:
> + break;
> + }
> + break;
> + case CMDQ_MODIFY_QP_NEW_STATE_RTS:
> + break;
> + case CMDQ_MODIFY_QP_NEW_STATE_SQD:
> + break;
> + case CMDQ_MODIFY_QP_NEW_STATE_SQE:
> + break;
> + case CMDQ_MODIFY_QP_NEW_STATE_ERR:
> + break;
> + default:
> + break;
> + }
> +}
> +
> +int bnxt_qplib_modify_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp)
> +{
> + struct bnxt_qplib_rcfw *rcfw = res->rcfw;
> + struct cmdq_modify_qp req;
> + struct creq_modify_qp_resp *resp;
> + u16 cmd_flags = 0, pkey;
> + u32 temp32[4];
> + u32 bmask;
> +
> + RCFW_CMD_PREP(req, MODIFY_QP, cmd_flags);
> +
> + /* Filter out the qp_attr_mask based on the state->new transition */
> + __filter_modify_flags(qp);
> + bmask = qp->modify_flags;
> + req.modify_mask = cpu_to_le64(qp->modify_flags);
> + req.qp_cid = cpu_to_le32(qp->id);
> + if (bmask & CMDQ_MODIFY_QP_MODIFY_MASK_STATE) {
> + req.network_type_en_sqd_async_notify_new_state =
> + (qp->state & CMDQ_MODIFY_QP_NEW_STATE_MASK) |
> + (qp->en_sqd_async_notify ?
> + CMDQ_MODIFY_QP_EN_SQD_ASYNC_NOTIFY : 0);
> + }
> + req.network_type_en_sqd_async_notify_new_state |= qp->nw_type;
> +
> + if (bmask & CMDQ_MODIFY_QP_MODIFY_MASK_ACCESS)
> + req.access = qp->access;
> +
> + if (bmask & CMDQ_MODIFY_QP_MODIFY_MASK_PKEY) {
> + if (!bnxt_qplib_get_pkey(res, &res->pkey_tbl,
> + qp->pkey_index, &pkey))
> + req.pkey = cpu_to_le16(pkey);
> + }
> + if (bmask & CMDQ_MODIFY_QP_MODIFY_MASK_QKEY)
> + req.qkey = cpu_to_le32(qp->qkey);
> +
> + if (bmask & CMDQ_MODIFY_QP_MODIFY_MASK_DGID) {
> + memcpy(temp32, qp->ah.dgid.data, sizeof(struct bnxt_qplib_gid));
> + req.dgid[0] = cpu_to_le32(temp32[0]);
> + req.dgid[1] = cpu_to_le32(temp32[1]);
> + req.dgid[2] = cpu_to_le32(temp32[2]);
> + req.dgid[3] = cpu_to_le32(temp32[3]);
> + }
> + if (bmask & CMDQ_MODIFY_QP_MODIFY_MASK_FLOW_LABEL)
> + req.flow_label = cpu_to_le32(qp->ah.flow_label);
> +
> + if (bmask & CMDQ_MODIFY_QP_MODIFY_MASK_SGID_INDEX)
> + req.sgid_index = cpu_to_le16(res->sgid_tbl.hw_id
> + [qp->ah.sgid_index]);
> +
> + if (bmask & CMDQ_MODIFY_QP_MODIFY_MASK_HOP_LIMIT)
> + req.hop_limit = qp->ah.hop_limit;
> +
> + if (bmask & CMDQ_MODIFY_QP_MODIFY_MASK_TRAFFIC_CLASS)
> + req.traffic_class = qp->ah.traffic_class;
> +
> + if (bmask & CMDQ_MODIFY_QP_MODIFY_MASK_DEST_MAC)
> + memcpy(req.dest_mac, qp->ah.dmac, 6);
> +
> + if (bmask & CMDQ_MODIFY_QP_MODIFY_MASK_PATH_MTU)
> + req.path_mtu = cpu_to_le16(qp->path_mtu);
> +
> + if (bmask & CMDQ_MODIFY_QP_MODIFY_MASK_TIMEOUT)
> + req.timeout = qp->timeout;
> +
> + if (bmask & CMDQ_MODIFY_QP_MODIFY_MASK_RETRY_CNT)
> + req.retry_cnt = qp->retry_cnt;
> +
> + if (bmask & CMDQ_MODIFY_QP_MODIFY_MASK_RNR_RETRY)
> + req.rnr_retry = qp->rnr_retry;
> +
> + if (bmask & CMDQ_MODIFY_QP_MODIFY_MASK_MIN_RNR_TIMER)
> + req.min_rnr_timer = qp->min_rnr_timer;
> +
> + if (bmask & CMDQ_MODIFY_QP_MODIFY_MASK_RQ_PSN)
> + req.rq_psn = cpu_to_le32(qp->rq.psn);
> +
> + if (bmask & CMDQ_MODIFY_QP_MODIFY_MASK_SQ_PSN)
> + req.sq_psn = cpu_to_le32(qp->sq.psn);
> +
> + if (bmask & CMDQ_MODIFY_QP_MODIFY_MASK_MAX_RD_ATOMIC)
> + req.max_rd_atomic =
> + ORD_LIMIT_TO_ORRQ_SLOTS(qp->max_rd_atomic);
> +
> + if (bmask & CMDQ_MODIFY_QP_MODIFY_MASK_MAX_DEST_RD_ATOMIC)
> + req.max_dest_rd_atomic =
> + IRD_LIMIT_TO_IRRQ_SLOTS(qp->max_dest_rd_atomic);
> +
> + req.sq_size = cpu_to_le32(qp->sq.hwq.max_elements);
> + req.rq_size = cpu_to_le32(qp->rq.hwq.max_elements);
> + req.sq_sge = cpu_to_le16(qp->sq.max_sge);
> + req.rq_sge = cpu_to_le16(qp->rq.max_sge);
> + req.max_inline_data = cpu_to_le32(qp->max_inline_data);
> + if (bmask & CMDQ_MODIFY_QP_MODIFY_MASK_DEST_QP_ID)
> + req.dest_qp_id = cpu_to_le32(qp->dest_qpn);
> +
> + req.vlan_pcp_vlan_dei_vlan_id = cpu_to_le16(qp->vlan_id);
> +
> + resp = (struct creq_modify_qp_resp *)
> + bnxt_qplib_rcfw_send_message(rcfw, (void *)&req,
> + NULL, 0);
> + if (!resp) {
> + dev_err(&rcfw->pdev->dev, "QPLIB: FP: MODIFY_QP send failed");
> + return -EINVAL;
> + }
> + /**/
> + if (!bnxt_qplib_rcfw_wait_for_resp(rcfw, le16_to_cpu(req.cookie))) {
> + /* Cmd timed out */
> + dev_err(&rcfw->pdev->dev, "QPLIB: FP: MODIFY_QP timed out");
> + return -ETIMEDOUT;
> + }
> + if (RCFW_RESP_STATUS(resp) ||
> + RCFW_RESP_COOKIE(resp) != RCFW_CMDQ_COOKIE(req)) {
> + dev_err(&rcfw->pdev->dev, "QPLIB: FP: MODIFY_QP failed ");
> + dev_err(&rcfw->pdev->dev,
> + "QPLIB: with status 0x%x cmdq 0x%x resp 0x%x",
> + RCFW_RESP_STATUS(resp), RCFW_CMDQ_COOKIE(req),
> + RCFW_RESP_COOKIE(resp));
> + return -EINVAL;
> + }
> + qp->cur_qp_state = qp->state;
> + return 0;
> +}
> +
> +int bnxt_qplib_query_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp)
> +{
> + struct bnxt_qplib_rcfw *rcfw = res->rcfw;
> + struct cmdq_query_qp req;
> + struct creq_query_qp_resp *resp;
> + struct creq_query_qp_resp_sb *sb;
> + u16 cmd_flags = 0;
> + u32 temp32[4];
> + int i;
> +
> + RCFW_CMD_PREP(req, QUERY_QP, cmd_flags);
> +
> + req.qp_cid = cpu_to_le32(qp->id);
> + req.resp_size = sizeof(*sb) / BNXT_QPLIB_CMDQE_UNITS;
> + resp = (struct creq_query_qp_resp *)
> + bnxt_qplib_rcfw_send_message(rcfw, (void *)&req,
> + (void **)&sb, 0);
> + if (!resp) {
> + dev_err(&rcfw->pdev->dev, "QPLIB: FP: QUERY_QP send failed");
> + return -EINVAL;
> + }
> + /**/
> + if (!bnxt_qplib_rcfw_wait_for_resp(rcfw, le16_to_cpu(req.cookie))) {
> + /* Cmd timed out */
> + dev_err(&rcfw->pdev->dev, "QPLIB: FP: QUERY_QP timed out");
> + return -ETIMEDOUT;
> + }
> + if (RCFW_RESP_STATUS(resp) ||
> + RCFW_RESP_COOKIE(resp) != RCFW_CMDQ_COOKIE(req)) {
> + dev_err(&rcfw->pdev->dev, "QPLIB: FP: QUERY_QP failed ");
> + dev_err(&rcfw->pdev->dev,
> + "QPLIB: with status 0x%x cmdq 0x%x resp 0x%x",
> + RCFW_RESP_STATUS(resp), RCFW_CMDQ_COOKIE(req),
> + RCFW_RESP_COOKIE(resp));
> + return -EINVAL;
> + }
> + /* Extract the context from the side buffer */
> + qp->state = sb->en_sqd_async_notify_state &
> + CREQ_QUERY_QP_RESP_SB_STATE_MASK;
> + qp->en_sqd_async_notify = sb->en_sqd_async_notify_state &
> + CREQ_QUERY_QP_RESP_SB_EN_SQD_ASYNC_NOTIFY ?
> + true : false;
> + qp->access = sb->access;
> + qp->pkey_index = le16_to_cpu(sb->pkey);
> + qp->qkey = le32_to_cpu(sb->qkey);
> +
> + temp32[0] = le32_to_cpu(sb->dgid[0]);
> + temp32[1] = le32_to_cpu(sb->dgid[1]);
> + temp32[2] = le32_to_cpu(sb->dgid[2]);
> + temp32[3] = le32_to_cpu(sb->dgid[3]);
> + memcpy(qp->ah.dgid.data, temp32, sizeof(qp->ah.dgid.data));
> +
> + qp->ah.flow_label = le32_to_cpu(sb->flow_label);
> +
> + qp->ah.sgid_index = 0;
> + for (i = 0; i < res->sgid_tbl.max; i++) {
> + if (res->sgid_tbl.hw_id[i] == le16_to_cpu(sb->sgid_index)) {
> + qp->ah.sgid_index = i;
> + break;
> + }
> + }
> + if (i == res->sgid_tbl.max)
> + dev_warn(&res->pdev->dev, "QPLIB: SGID not found??");
> +
> + qp->ah.hop_limit = sb->hop_limit;
> + qp->ah.traffic_class = sb->traffic_class;
> + memcpy(qp->ah.dmac, sb->dest_mac, 6);
> + qp->ah.vlan_id = le16_to_cpu((sb->path_mtu_dest_vlan_id &
> + CREQ_QUERY_QP_RESP_SB_VLAN_ID_MASK) >>
> + CREQ_QUERY_QP_RESP_SB_VLAN_ID_SFT);
> + qp->path_mtu = sb->path_mtu_dest_vlan_id &
> + CREQ_QUERY_QP_RESP_SB_PATH_MTU_MASK;
> + qp->timeout = sb->timeout;
> + qp->retry_cnt = sb->retry_cnt;
> + qp->rnr_retry = sb->rnr_retry;
> + qp->min_rnr_timer = sb->min_rnr_timer;
> + qp->rq.psn = le32_to_cpu(sb->rq_psn);
> + qp->max_rd_atomic = ORRQ_SLOTS_TO_ORD_LIMIT(sb->max_rd_atomic);
> + qp->sq.psn = le32_to_cpu(sb->sq_psn);
> + qp->max_dest_rd_atomic =
> + IRRQ_SLOTS_TO_IRD_LIMIT(sb->max_dest_rd_atomic);
> + qp->sq.max_wqe = qp->sq.hwq.max_elements;
> + qp->rq.max_wqe = qp->rq.hwq.max_elements;
> + qp->sq.max_sge = le16_to_cpu(sb->sq_sge);
> + qp->rq.max_sge = le32_to_cpu(sb->rq_sge);
> + qp->max_inline_data = le32_to_cpu(sb->max_inline_data);
> + qp->dest_qpn = le32_to_cpu(sb->dest_qp_id);
> + memcpy(qp->smac, sb->src_mac, 6);
> + qp->vlan_id = le16_to_cpu(sb->vlan_pcp_vlan_dei_vlan_id);
> + return 0;
> +}
> +
> +static void __clean_cq(struct bnxt_qplib_cq *cq, u64 qp)
> +{
> + struct bnxt_qplib_hwq *cq_hwq = &cq->hwq;
> + struct cq_base *hw_cqe, **hw_cqe_ptr;
> + int i;
> +
> + for (i = 0; i < cq_hwq->max_elements; i++) {
> + hw_cqe_ptr = (struct cq_base **)cq_hwq->pbl_ptr;
> + hw_cqe = &hw_cqe_ptr[CQE_PG(i)][CQE_IDX(i)];
> + if (!CQE_CMP_VALID(hw_cqe, i, cq_hwq->max_elements))
> + continue;
> + switch (hw_cqe->cqe_type_toggle & CQ_BASE_CQE_TYPE_MASK) {
> + case CQ_BASE_CQE_TYPE_REQ:
> + case CQ_BASE_CQE_TYPE_TERMINAL:
> + {
> + struct cq_req *cqe = (struct cq_req *)hw_cqe;
> +
> + if (qp == le64_to_cpu(cqe->qp_handle))
> + cqe->qp_handle = 0;
> + break;
> + }
> + case CQ_BASE_CQE_TYPE_RES_RC:
> + case CQ_BASE_CQE_TYPE_RES_UD:
> + case CQ_BASE_CQE_TYPE_RES_RAWETH_QP1:
> + {
> + struct cq_res_rc *cqe = (struct cq_res_rc *)hw_cqe;
> +
> + if (qp == le64_to_cpu(cqe->qp_handle))
> + cqe->qp_handle = 0;
> + break;
> + }
> + default:
> + break;
> + }
> + }
> +}
> +
> +static unsigned long bnxt_qplib_lock_cqs(struct bnxt_qplib_qp *qp)
> +{
> + unsigned long flags;
> +
> + spin_lock_irqsave(&qp->scq->hwq.lock, flags);
> + if (qp->rcq && qp->rcq != qp->scq)
> + spin_lock(&qp->rcq->hwq.lock);
> +
> + return flags;
> +}
> +
> +static void bnxt_qplib_unlock_cqs(struct bnxt_qplib_qp *qp,
> + unsigned long flags)
> +{
> + if (qp->rcq && qp->rcq != qp->scq)
> + spin_unlock(&qp->rcq->hwq.lock);
> + spin_unlock_irqrestore(&qp->scq->hwq.lock, flags);
> +}
> +
> +int bnxt_qplib_destroy_qp(struct bnxt_qplib_res *res,
> + struct bnxt_qplib_qp *qp)
> +{
> + struct bnxt_qplib_rcfw *rcfw = res->rcfw;
> + struct cmdq_destroy_qp req;
> + struct creq_destroy_qp_resp *resp;
> + unsigned long flags;
> + u16 cmd_flags = 0;
> +
> + RCFW_CMD_PREP(req, DESTROY_QP, cmd_flags);
> +
> + req.qp_cid = cpu_to_le32(qp->id);
> + resp = (struct creq_destroy_qp_resp *)
> + bnxt_qplib_rcfw_send_message(rcfw, (void *)&req,
> + NULL, 0);
> + if (!resp) {
> + dev_err(&rcfw->pdev->dev, "QPLIB: FP: DESTROY_QP send failed");
> + return -EINVAL;
> + }
> + /**/
> + if (!bnxt_qplib_rcfw_wait_for_resp(rcfw, le16_to_cpu(req.cookie))) {
> + /* Cmd timed out */
> + dev_err(&rcfw->pdev->dev, "QPLIB: FP: DESTROY_QP timed out");
> + return -ETIMEDOUT;
> + }
> + if (RCFW_RESP_STATUS(resp) ||
> + RCFW_RESP_COOKIE(resp) != RCFW_CMDQ_COOKIE(req)) {
> + dev_err(&rcfw->pdev->dev, "QPLIB: FP: DESTROY_QP failed ");
> + dev_err(&rcfw->pdev->dev,
> + "QPLIB: with status 0x%x cmdq 0x%x resp 0x%x",
> + RCFW_RESP_STATUS(resp), RCFW_CMDQ_COOKIE(req),
> + RCFW_RESP_COOKIE(resp));
> + return -EINVAL;
> + }
> +
> + /* Must walk the associated CQs to nullified the QP ptr */
> + flags = bnxt_qplib_lock_cqs(qp);
> + __clean_cq(qp->scq, (u64)qp);
> + if (qp->rcq != qp->scq)
> + __clean_cq(qp->rcq, (u64)qp);
> + bnxt_qplib_unlock_cqs(qp, flags);
> +
> + bnxt_qplib_free_qp_hdr_buf(res, qp);
> + bnxt_qplib_free_hwq(res->pdev, &qp->sq.hwq);
> + kfree(qp->sq.swq);
> +
> + bnxt_qplib_free_hwq(res->pdev, &qp->rq.hwq);
> + kfree(qp->rq.swq);
> +
> + if (qp->irrq.max_elements)
> + bnxt_qplib_free_hwq(res->pdev, &qp->irrq);
> + if (qp->orrq.max_elements)
> + bnxt_qplib_free_hwq(res->pdev, &qp->orrq);
> +
> + return 0;
> +}
> +
> /* CQ */
>
> /* Spinlock must be held */
> diff --git a/drivers/infiniband/hw/bnxtre/bnxt_qplib_fp.h b/drivers/infiniband/hw/bnxtre/bnxt_qplib_fp.h
> index 1991eaa..f6d2be5 100644
> --- a/drivers/infiniband/hw/bnxtre/bnxt_qplib_fp.h
> +++ b/drivers/infiniband/hw/bnxtre/bnxt_qplib_fp.h
> @@ -38,8 +38,246 @@
>
> #ifndef __BNXT_QPLIB_FP_H__
> #define __BNXT_QPLIB_FP_H__
> +struct bnxt_qplib_sge {
> + u64 addr;
> + u32 lkey;
> + u32 size;
> +};
> +
> +#define BNXT_QPLIB_MAX_SQE_ENTRY_SIZE sizeof(struct sq_send)
> +
> +#define SQE_CNT_PER_PG (PAGE_SIZE / BNXT_QPLIB_MAX_SQE_ENTRY_SIZE)
> +#define SQE_MAX_IDX_PER_PG (SQE_CNT_PER_PG - 1)
> +#define SQE_PG(x) (((x) & ~SQE_MAX_IDX_PER_PG) / SQE_CNT_PER_PG)
> +#define SQE_IDX(x) ((x) & SQE_MAX_IDX_PER_PG)
> +
> +#define BNXT_QPLIB_MAX_PSNE_ENTRY_SIZE sizeof(struct sq_psn_search)
> +
> +#define PSNE_CNT_PER_PG (PAGE_SIZE / BNXT_QPLIB_MAX_PSNE_ENTRY_SIZE)
> +#define PSNE_MAX_IDX_PER_PG (PSNE_CNT_PER_PG - 1)
> +#define PSNE_PG(x) (((x) & ~PSNE_MAX_IDX_PER_PG) / PSNE_CNT_PER_PG)
> +#define PSNE_IDX(x) ((x) & PSNE_MAX_IDX_PER_PG)
> +
> +#define BNXT_QPLIB_QP_MAX_SGL 6
> +
> +struct bnxt_qplib_swq {
> + u64 wr_id;
> + u8 type;
> + u8 flags;
> + u32 start_psn;
> + u32 next_psn;
> + struct sq_psn_search *psn_search;
> +};
> +
> +struct bnxt_qplib_swqe {
> + /* General */
> + u64 wr_id;
> + u8 reqs_type;
> + u8 type;
> +#define BNXT_QPLIB_SWQE_TYPE_SEND 0
> +#define BNXT_QPLIB_SWQE_TYPE_SEND_WITH_IMM 1
> +#define BNXT_QPLIB_SWQE_TYPE_SEND_WITH_INV 2
> +#define BNXT_QPLIB_SWQE_TYPE_RDMA_WRITE 4
> +#define BNXT_QPLIB_SWQE_TYPE_RDMA_WRITE_WITH_IMM 5
> +#define BNXT_QPLIB_SWQE_TYPE_RDMA_READ 6
> +#define BNXT_QPLIB_SWQE_TYPE_ATOMIC_CMP_AND_SWP 8
> +#define BNXT_QPLIB_SWQE_TYPE_ATOMIC_FETCH_AND_ADD 11
> +#define BNXT_QPLIB_SWQE_TYPE_LOCAL_INV 12
> +#define BNXT_QPLIB_SWQE_TYPE_FAST_REG_MR 13
> +#define BNXT_QPLIB_SWQE_TYPE_REG_MR 13
> +#define BNXT_QPLIB_SWQE_TYPE_BIND_MW 14
> +#define BNXT_QPLIB_SWQE_TYPE_RECV 128
> +#define BNXT_QPLIB_SWQE_TYPE_RECV_RDMA_IMM 129
> + u8 flags;
> +#define BNXT_QPLIB_SWQE_FLAGS_SIGNAL_COMP BIT(0)
> +#define BNXT_QPLIB_SWQE_FLAGS_RD_ATOMIC_FENCE BIT(1)
> +#define BNXT_QPLIB_SWQE_FLAGS_UC_FENCE BIT(2)
> +#define BNXT_QPLIB_SWQE_FLAGS_SOLICIT_EVENT BIT(3)
> +#define BNXT_QPLIB_SWQE_FLAGS_INLINE BIT(4)
> + struct bnxt_qplib_sge sg_list[BNXT_QPLIB_QP_MAX_SGL];
> + int num_sge;
> + /* Max inline data is 96 bytes */
> + u32 inline_len;
> +#define BNXT_QPLIB_SWQE_MAX_INLINE_LENGTH 96
> + u8 inline_data[BNXT_QPLIB_SWQE_MAX_INLINE_LENGTH];
> +
> + union {
> + /* Send, with imm, inval key */
> + struct {
> + u32 imm_data_or_inv_key;
> + u32 q_key;
> + u32 dst_qp;
> + u16 avid;
> + } send;
> +
> + /* Send Raw Ethernet and QP1 */
> + struct {
> + u16 lflags;
> + u16 cfa_action;
> + u32 cfa_meta;
> + } rawqp1;
> +
> + /* RDMA write, with imm, read */
> + struct {
> + u32 imm_data_or_inv_key;
> + u64 remote_va;
> + u32 r_key;
> + } rdma;
> +
> + /* Atomic cmp/swap, fetch/add */
> + struct {
> + u64 remote_va;
> + u32 r_key;
> + u64 swap_data;
> + u64 cmp_data;
> + } atomic;
> +
> + /* Local Invalidate */
> + struct {
> + u32 inv_l_key;
> + } local_inv;
> +
> + /* FR-PMR */
> + struct {
> + u8 access_cntl;
> + u8 pg_sz_log;
> + bool zero_based;
> + u32 l_key;
> + u32 length;
> + u8 pbl_pg_sz_log;
> +#define BNXT_QPLIB_SWQE_PAGE_SIZE_4K 0
> +#define BNXT_QPLIB_SWQE_PAGE_SIZE_8K 1
> +#define BNXT_QPLIB_SWQE_PAGE_SIZE_64K 4
> +#define BNXT_QPLIB_SWQE_PAGE_SIZE_256K 6
> +#define BNXT_QPLIB_SWQE_PAGE_SIZE_1M 8
> +#define BNXT_QPLIB_SWQE_PAGE_SIZE_2M 9
> +#define BNXT_QPLIB_SWQE_PAGE_SIZE_4M 10
> +#define BNXT_QPLIB_SWQE_PAGE_SIZE_1G 18
> + u8 levels;
> +#define PAGE_SHIFT_4K 12
> + u64 *pbl_ptr;
> + dma_addr_t pbl_dma_ptr;
> + u64 *page_list;
> + u16 page_list_len;
> + u64 va;
> + } frmr;
> +
> + /* Bind */
> + struct {
> + u8 access_cntl;
> +#define BNXT_QPLIB_BIND_SWQE_ACCESS_LOCAL_WRITE BIT(0)
> +#define BNXT_QPLIB_BIND_SWQE_ACCESS_REMOTE_READ BIT(1)
> +#define BNXT_QPLIB_BIND_SWQE_ACCESS_REMOTE_WRITE BIT(2)
> +#define BNXT_QPLIB_BIND_SWQE_ACCESS_REMOTE_ATOMIC BIT(3)
> +#define BNXT_QPLIB_BIND_SWQE_ACCESS_WINDOW_BIND BIT(4)
> + bool zero_based;
> + u8 mw_type;
> + u32 parent_l_key;
> + u32 r_key;
> + u64 va;
> + u32 length;
> + } bind;
> + };
> +};
> +
> +#define BNXT_QPLIB_MAX_RQE_ENTRY_SIZE sizeof(struct rq_wqe)
> +
> +#define RQE_CNT_PER_PG (PAGE_SIZE / BNXT_QPLIB_MAX_RQE_ENTRY_SIZE)
> +#define RQE_MAX_IDX_PER_PG (RQE_CNT_PER_PG - 1)
> +#define RQE_PG(x) (((x) & ~RQE_MAX_IDX_PER_PG) / RQE_CNT_PER_PG)
> +#define RQE_IDX(x) ((x) & RQE_MAX_IDX_PER_PG)
> +
> +struct bnxt_qplib_q {
> + struct bnxt_qplib_hwq hwq;
> + struct bnxt_qplib_swq *swq;
> + struct scatterlist *sglist;
> + u32 nmap;
> + u32 max_wqe;
> + u16 max_sge;
> + u32 psn;
> + bool flush_in_progress;
> +};
> +
> +struct bnxt_qplib_qp {
> + struct bnxt_qplib_pd *pd;
> + struct bnxt_qplib_dpi *dpi;
> + u64 qp_handle;
> + u32 id;
> + u8 type;
> + u8 sig_type;
> + u64 modify_flags;
> + u8 state;
> + u8 cur_qp_state;
> + u32 max_inline_data;
> + u32 mtu;
> + u32 path_mtu;
> + bool en_sqd_async_notify;
> + u16 pkey_index;
> + u32 qkey;
> + u32 dest_qp_id;
> + u8 access;
> + u8 timeout;
> + u8 retry_cnt;
> + u8 rnr_retry;
> + u32 min_rnr_timer;
> + u32 max_rd_atomic;
> + u32 max_dest_rd_atomic;
> + u32 dest_qpn;
> + u8 smac[6];
> + u16 vlan_id;
> + u8 nw_type;
> + struct bnxt_qplib_ah ah;
> +
> +#define BTH_PSN_MASK ((1 << 24) - 1)
> + /* SQ */
> + struct bnxt_qplib_q sq;
> + /* RQ */
> + struct bnxt_qplib_q rq;
> + /* SRQ */
> + struct bnxt_qplib_srq *srq;
> + /* CQ */
> + struct bnxt_qplib_cq *scq;
> + struct bnxt_qplib_cq *rcq;
> + /* IRRQ and ORRQ */
> + struct bnxt_qplib_hwq irrq;
> + struct bnxt_qplib_hwq orrq;
> + /* Header buffer for QP1 */
> + int sq_hdr_buf_size;
> + int rq_hdr_buf_size;
> +/*
> + * Buffer space for ETH(14), IP or GRH(40), UDP header(8)
> + * and ib_bth + ib_deth (20).
> + * Max required is 82 when RoCE V2 is enabled
> + */
> +#define BNXT_QPLIB_MAX_QP1_SQ_HDR_SIZE_V2 86
> + /* Ethernet header = 14 */
> + /* ib_grh = 40 (provided by MAD) */
> + /* ib_bth + ib_deth = 20 */
> + /* MAD = 256 (provided by MAD) */
> + /* iCRC = 4 */
> +#define BNXT_QPLIB_MAX_QP1_RQ_ETH_HDR_SIZE 14
> +#define BNXT_QPLIB_MAX_QP1_RQ_HDR_SIZE_V2 512
> +#define BNXT_QPLIB_MAX_GRH_HDR_SIZE_IPV4 20
> +#define BNXT_QPLIB_MAX_GRH_HDR_SIZE_IPV6 40
> +#define BNXT_QPLIB_MAX_QP1_RQ_BDETH_HDR_SIZE 20
> + void *sq_hdr_buf;
> + dma_addr_t sq_hdr_buf_map;
> + void *rq_hdr_buf;
> + dma_addr_t rq_hdr_buf_map;
> +};
> +
> #define BNXT_QPLIB_MAX_CQE_ENTRY_SIZE sizeof(struct cq_base)
>
> +#define CQE_CNT_PER_PG (PAGE_SIZE / BNXT_QPLIB_MAX_CQE_ENTRY_SIZE)
> +#define CQE_MAX_IDX_PER_PG (CQE_CNT_PER_PG - 1)
> +#define CQE_PG(x) (((x) & ~CQE_MAX_IDX_PER_PG) / CQE_CNT_PER_PG)
> +#define CQE_IDX(x) ((x) & CQE_MAX_IDX_PER_PG)
> +
> +#define ROCE_CQE_CMP_V 0
> +#define CQE_CMP_VALID(hdr, raw_cons, cp_bit) \
> + (!!((hdr)->cqe_type_toggle & CQ_BASE_TOGGLE) == \
> + !((raw_cons) & (cp_bit)))
> +
> struct bnxt_qplib_cqe {
> u8 status;
> u8 type;
> @@ -82,6 +320,13 @@ struct bnxt_qplib_cq {
> wait_queue_head_t waitq;
> };
>
> +#define BNXT_QPLIB_MAX_IRRQE_ENTRY_SIZE sizeof(struct xrrq_irrq)
> +#define BNXT_QPLIB_MAX_ORRQE_ENTRY_SIZE sizeof(struct xrrq_orrq)
> +#define IRD_LIMIT_TO_IRRQ_SLOTS(x) (2 * (x) + 2)
> +#define IRRQ_SLOTS_TO_IRD_LIMIT(s) (((s) >> 1) - 1)
> +#define ORD_LIMIT_TO_ORRQ_SLOTS(x) ((x) + 1)
> +#define ORRQ_SLOTS_TO_ORD_LIMIT(s) ((s) - 1)
> +
> #define BNXT_QPLIB_MAX_NQE_ENTRY_SIZE sizeof(struct nq_base)
>
> #define NQE_CNT_PER_PG (PAGE_SIZE / BNXT_QPLIB_MAX_NQE_ENTRY_SIZE)
> @@ -140,6 +385,11 @@ int bnxt_qplib_enable_nq(struct pci_dev *pdev, struct bnxt_qplib_nq *nq,
> int (*srqn_handler)(struct bnxt_qplib_nq *nq,
> void *srq,
> u8 event));
> +int bnxt_qplib_create_qp1(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp);
> +int bnxt_qplib_create_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp);
> +int bnxt_qplib_modify_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp);
> +int bnxt_qplib_query_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp);
> +int bnxt_qplib_destroy_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp);
> int bnxt_qplib_create_cq(struct bnxt_qplib_res *res, struct bnxt_qplib_cq *cq);
> int bnxt_qplib_destroy_cq(struct bnxt_qplib_res *res, struct bnxt_qplib_cq *cq);
>
> diff --git a/drivers/infiniband/hw/bnxtre/bnxt_re.h b/drivers/infiniband/hw/bnxtre/bnxt_re.h
> index 3a93a88..84af86b 100644
> --- a/drivers/infiniband/hw/bnxtre/bnxt_re.h
> +++ b/drivers/infiniband/hw/bnxtre/bnxt_re.h
> @@ -64,6 +64,14 @@ struct bnxt_re_work {
> struct net_device *vlan_dev;
> };
>
> +struct bnxt_re_sqp_entries {
> + struct bnxt_qplib_sge sge;
> + u64 wrid;
> + /* For storing the actual qp1 cqe */
> + struct bnxt_qplib_cqe cqe;
> + struct bnxt_re_qp *qp1_qp;
> +};
> +
> #define BNXT_RE_MIN_MSIX 2
> #define BNXT_RE_MAX_MSIX 16
> #define BNXT_RE_AEQ_IDX 0
> @@ -112,6 +120,12 @@ struct bnxt_re_dev {
> atomic_t mw_count;
> /* Max of 2 lossless traffic class supported per port */
> u16 cosq[2];
> +
> + /* QP for for handling QP1 packets */
> + u32 sqp_id;
> + struct bnxt_re_qp *qp1_sqp;
> + struct bnxt_re_ah *sqp_ah;
> + struct bnxt_re_sqp_entries sqp_tbl[1024];
> };
>
> #define to_bnxt_re(ptr, type, member) \
> diff --git a/drivers/infiniband/hw/bnxtre/bnxt_re_ib_verbs.c b/drivers/infiniband/hw/bnxtre/bnxt_re_ib_verbs.c
> index 5e41317..77860a2 100644
> --- a/drivers/infiniband/hw/bnxtre/bnxt_re_ib_verbs.c
> +++ b/drivers/infiniband/hw/bnxtre/bnxt_re_ib_verbs.c
> @@ -649,6 +649,481 @@ int bnxt_re_query_ah(struct ib_ah *ib_ah, struct ib_ah_attr *ah_attr)
> return 0;
> }
>
> +/* Queue Pairs */
> +int bnxt_re_destroy_qp(struct ib_qp *ib_qp)
> +{
> + struct bnxt_re_qp *qp = to_bnxt_re(ib_qp, struct bnxt_re_qp, ib_qp);
> + struct bnxt_re_dev *rdev = qp->rdev;
> + int rc;
> +
> + rc = bnxt_qplib_destroy_qp(&rdev->qplib_res, &qp->qplib_qp);
> + if (rc) {
> + dev_err(rdev_to_dev(rdev), "Failed to destroy HW QP");
> + return rc;
> + }
> + if (ib_qp->qp_type == IB_QPT_GSI && rdev->qp1_sqp) {
> + rc = bnxt_qplib_destroy_ah(&rdev->qplib_res,
> + &rdev->sqp_ah->qplib_ah);
> + if (rc) {
> + dev_err(rdev_to_dev(rdev),
> + "Failed to destroy HW AH for shadow QP");
> + return rc;
> + }
> +
> + rc = bnxt_qplib_destroy_qp(&rdev->qplib_res,
> + &rdev->qp1_sqp->qplib_qp);
> + if (rc) {
> + dev_err(rdev_to_dev(rdev),
> + "Failed to destroy Shadow QP");
> + return rc;
> + }
> + mutex_lock(&rdev->qp_lock);
> + list_del(&rdev->qp1_sqp->list);
> + atomic_dec(&rdev->qp_count);
> + mutex_unlock(&rdev->qp_lock);
> +
> + kfree(rdev->sqp_ah);
> + kfree(rdev->qp1_sqp);
> + }
> +
> + if (qp->rumem && !IS_ERR(qp->rumem))
> + ib_umem_release(qp->rumem);
> + if (qp->sumem && !IS_ERR(qp->sumem))
> + ib_umem_release(qp->sumem);
> +
> + mutex_lock(&rdev->qp_lock);
> + list_del(&qp->list);
> + atomic_dec(&rdev->qp_count);
> + mutex_unlock(&rdev->qp_lock);
> + kfree(qp);
> + return 0;
> +}
> +
> +static u8 __from_ib_qp_type(enum ib_qp_type type)
> +{
> + switch (type) {
> + case IB_QPT_GSI:
> + return CMDQ_CREATE_QP1_TYPE_GSI;
> + case IB_QPT_RC:
> + return CMDQ_CREATE_QP_TYPE_RC;
> + case IB_QPT_UD:
> + return CMDQ_CREATE_QP_TYPE_UD;
> + case IB_QPT_RAW_ETHERTYPE:
> + return CMDQ_CREATE_QP_TYPE_RAW_ETHERTYPE;
> + default:
> + return IB_QPT_MAX;
> + }
> +}
> +
> +static int bnxt_re_init_user_qp(struct bnxt_re_dev *rdev, struct bnxt_re_pd *pd,
> + struct bnxt_re_qp *qp, struct ib_udata *udata)
> +{
> + struct bnxt_re_qp_req ureq;
> + struct bnxt_qplib_qp *qplib_qp = &qp->qplib_qp;
> + struct ib_umem *umem;
> + int bytes = 0;
> + struct ib_ucontext *context = pd->ib_pd.uobject->context;
> + struct bnxt_re_ucontext *cntx = to_bnxt_re(context,
> + struct bnxt_re_ucontext,
> + ib_uctx);
> + if (ib_copy_from_udata(&ureq, udata, sizeof(ureq)))
> + return -EFAULT;
> +
> + bytes = (qplib_qp->sq.max_wqe * BNXT_QPLIB_MAX_SQE_ENTRY_SIZE);
> + /* Consider mapping PSN search memory only for RC QPs. */
> + if (qplib_qp->type == CMDQ_CREATE_QP_TYPE_RC)
> + bytes += (qplib_qp->sq.max_wqe * sizeof(struct sq_psn_search));
> + bytes = PAGE_ALIGN(bytes);
> + umem = ib_umem_get(context, ureq.qpsva, bytes,
> + IB_ACCESS_LOCAL_WRITE, 1);
> + if (IS_ERR(umem))
> + return PTR_ERR(umem);
> +
> + qp->sumem = umem;
> + qplib_qp->sq.sglist = umem->sg_head.sgl;
> + qplib_qp->sq.nmap = umem->nmap;
> + qplib_qp->qp_handle = ureq.qp_handle;
> +
> + if (!qp->qplib_qp.srq) {
> + bytes = (qplib_qp->rq.max_wqe * BNXT_QPLIB_MAX_RQE_ENTRY_SIZE);
> + bytes = PAGE_ALIGN(bytes);
> + umem = ib_umem_get(context, ureq.qprva, bytes,
> + IB_ACCESS_LOCAL_WRITE, 1);
> + if (IS_ERR(umem))
> + goto rqfail;
> + qp->rumem = umem;
> + qplib_qp->rq.sglist = umem->sg_head.sgl;
> + qplib_qp->rq.nmap = umem->nmap;
> + }
> +
> + qplib_qp->dpi = cntx->dpi;
> + return 0;
> +rqfail:
> + ib_umem_release(qp->sumem);
> + qp->sumem = NULL;
> + qplib_qp->sq.sglist = NULL;
> + qplib_qp->sq.nmap = 0;
> +
> + return PTR_ERR(umem);
> +}
> +
> +static struct bnxt_re_ah *bnxt_re_create_shadow_qp_ah(struct bnxt_re_pd *pd,
> + struct bnxt_qplib_res *qp1_res,
> + struct bnxt_qplib_qp *qp1_qp)
> +{
> + struct bnxt_re_dev *rdev = pd->rdev;
> + struct bnxt_re_ah *ah;
> + union ib_gid sgid;
> + int rc;
> +
> + ah = kzalloc(sizeof(*ah), GFP_KERNEL);
> + if (!ah)
> + return NULL;
> +
> + memset(ah, 0, sizeof(*ah));
> + ah->rdev = rdev;
> + ah->qplib_ah.pd = &pd->qplib_pd;
> +
> + rc = bnxt_re_query_gid(&rdev->ibdev, 1, 0, &sgid);
> + if (rc)
> + goto fail;
> +
> + /* supply the dgid data same as sgid */
> + memcpy(ah->qplib_ah.dgid.data, &sgid.raw,
> + sizeof(union ib_gid));
> + ah->qplib_ah.sgid_index = 0;
> +
> + ah->qplib_ah.traffic_class = 0;
> + ah->qplib_ah.flow_label = 0;
> + ah->qplib_ah.hop_limit = 1;
> + ah->qplib_ah.sl = 0;
> + /* Have DMAC same as SMAC */
> + ether_addr_copy(ah->qplib_ah.dmac, rdev->netdev->dev_addr);
> +
> + rc = bnxt_qplib_create_ah(&rdev->qplib_res, &ah->qplib_ah);
> + if (rc) {
> + dev_err(rdev_to_dev(rdev),
> + "Failed to allocate HW AH for Shadow QP");
> + goto fail;
> + }
> +
> + return ah;
> +
> +fail:
> + kfree(ah);
> + return NULL;
> +}
> +
> +static struct bnxt_re_qp *bnxt_re_create_shadow_qp(struct bnxt_re_pd *pd,
> + struct bnxt_qplib_res *qp1_res,
> + struct bnxt_qplib_qp *qp1_qp)
> +{
> + struct bnxt_re_dev *rdev = pd->rdev;
> + struct bnxt_re_qp *qp;
> + int rc;
> +
> + qp = kzalloc(sizeof(*qp), GFP_KERNEL);
> + if (!qp)
> + return NULL;
> +
> + memset(qp, 0, sizeof(*qp));
> + qp->rdev = rdev;
> +
> + /* Initialize the shadow QP structure from the QP1 values */
> + ether_addr_copy(qp->qplib_qp.smac, rdev->netdev->dev_addr);
> +
> + qp->qplib_qp.pd = &pd->qplib_pd;
> + qp->qplib_qp.qp_handle = (u64)&qp->qplib_qp;
> + qp->qplib_qp.type = IB_QPT_UD;
> +
> + qp->qplib_qp.max_inline_data = 0;
> + qp->qplib_qp.sig_type = true;
> +
> + /* Shadow QP SQ depth should be same as QP1 RQ depth */
> + qp->qplib_qp.sq.max_wqe = qp1_qp->rq.max_wqe;
> + qp->qplib_qp.sq.max_sge = 2;
> +
> + qp->qplib_qp.scq = qp1_qp->scq;
> + qp->qplib_qp.rcq = qp1_qp->rcq;
> +
> + qp->qplib_qp.rq.max_wqe = qp1_qp->rq.max_wqe;
> + qp->qplib_qp.rq.max_sge = qp1_qp->rq.max_sge;
> +
> + qp->qplib_qp.mtu = qp1_qp->mtu;
> +
> + qp->qplib_qp.sq_hdr_buf_size = 0;
> + qp->qplib_qp.rq_hdr_buf_size = BNXT_QPLIB_MAX_GRH_HDR_SIZE_IPV6;
> + qp->qplib_qp.dpi = &rdev->dpi_privileged;
> +
> + rc = bnxt_qplib_create_qp(qp1_res, &qp->qplib_qp);
> + if (rc)
> + goto fail;
> +
> + rdev->sqp_id = qp->qplib_qp.id;
> +
> + spin_lock_init(&qp->sq_lock);
> + INIT_LIST_HEAD(&qp->list);
> + mutex_lock(&rdev->qp_lock);
> + list_add_tail(&qp->list, &rdev->qp_list);
> + atomic_inc(&rdev->qp_count);
> + mutex_unlock(&rdev->qp_lock);
> + return qp;
> +fail:
> + kfree(qp);
> + return NULL;
> +}
> +
> +struct ib_qp *bnxt_re_create_qp(struct ib_pd *ib_pd,
> + struct ib_qp_init_attr *qp_init_attr,
> + struct ib_udata *udata)
> +{
> + struct bnxt_re_pd *pd = to_bnxt_re(ib_pd, struct bnxt_re_pd, ib_pd);
> + struct bnxt_re_dev *rdev = pd->rdev;
> + struct bnxt_qplib_dev_attr *dev_attr = &rdev->dev_attr;
> + struct bnxt_re_qp *qp;
> + struct bnxt_re_srq *srq;
> + struct bnxt_re_cq *cq;
> + int rc, entries;
> +
> + if ((qp_init_attr->cap.max_send_wr > dev_attr->max_qp_wqes) ||
> + (qp_init_attr->cap.max_recv_wr > dev_attr->max_qp_wqes) ||
> + (qp_init_attr->cap.max_send_sge > dev_attr->max_qp_sges) ||
> + (qp_init_attr->cap.max_recv_sge > dev_attr->max_qp_sges) ||
> + (qp_init_attr->cap.max_inline_data > dev_attr->max_inline_data))
> + return ERR_PTR(-EINVAL);
> +
> + qp = kzalloc(sizeof(*qp), GFP_KERNEL);
> + if (!qp)
> + return ERR_PTR(-ENOMEM);
> +
> + qp->rdev = rdev;
> + ether_addr_copy(qp->qplib_qp.smac, rdev->netdev->dev_addr);
> + qp->qplib_qp.pd = &pd->qplib_pd;
> + qp->qplib_qp.qp_handle = (u64)&qp->qplib_qp;
> + qp->qplib_qp.type = __from_ib_qp_type(qp_init_attr->qp_type);
> + if (qp->qplib_qp.type == IB_QPT_MAX) {
> + dev_err(rdev_to_dev(rdev), "QP type 0x%x not supported",
> + qp->qplib_qp.type);
> + rc = -EINVAL;
> + goto fail;
> + }
> + qp->qplib_qp.max_inline_data = qp_init_attr->cap.max_inline_data;
> + qp->qplib_qp.sig_type = ((qp_init_attr->sq_sig_type ==
> + IB_SIGNAL_ALL_WR) ? true : false);
> +
> + entries = roundup_pow_of_two(qp_init_attr->cap.max_send_wr + 1);
> + if (entries > dev_attr->max_qp_wqes + 1)
> + entries = dev_attr->max_qp_wqes + 1;
> + qp->qplib_qp.sq.max_wqe = entries;
> +
> + qp->qplib_qp.sq.max_sge = qp_init_attr->cap.max_send_sge;
> + if (qp->qplib_qp.sq.max_sge > dev_attr->max_qp_sges)
> + qp->qplib_qp.sq.max_sge = dev_attr->max_qp_sges;
> +
> + if (qp_init_attr->send_cq) {
> + cq = to_bnxt_re(qp_init_attr->send_cq, struct bnxt_re_cq,
> + ib_cq);
> + if (!cq) {
> + dev_err(rdev_to_dev(rdev), "Send CQ not found");
> + rc = -EINVAL;
> + goto fail;
> + }
> + qp->qplib_qp.scq = &cq->qplib_cq;
> + }
> +
> + if (qp_init_attr->recv_cq) {
> + cq = to_bnxt_re(qp_init_attr->recv_cq, struct bnxt_re_cq,
> + ib_cq);
> + if (!cq) {
> + dev_err(rdev_to_dev(rdev), "Receive CQ not found");
> + rc = -EINVAL;
> + goto fail;
> + }
> + qp->qplib_qp.rcq = &cq->qplib_cq;
> + }
> +
> + if (qp_init_attr->srq) {
> + dev_err(rdev_to_dev(rdev), "SRQ not supported");
> + rc = -ENOTSUPP;
> + goto fail;
> + } else {
> + /* Allocate 1 more than what's provided so posting max doesn't
> + * mean empty
> + */
> + entries = roundup_pow_of_two(qp_init_attr->cap.max_recv_wr + 1);
> + if (entries > dev_attr->max_qp_wqes + 1)
> + entries = dev_attr->max_qp_wqes + 1;
> + qp->qplib_qp.rq.max_wqe = entries;
> +
> + qp->qplib_qp.rq.max_sge = qp_init_attr->cap.max_recv_sge;
> + if (qp->qplib_qp.rq.max_sge > dev_attr->max_qp_sges)
> + qp->qplib_qp.rq.max_sge = dev_attr->max_qp_sges;
> + }
> +
> + qp->qplib_qp.mtu = ib_mtu_enum_to_int(iboe_get_mtu(rdev->netdev->mtu));
> +
> + if (qp_init_attr->qp_type == IB_QPT_GSI) {
> + qp->qplib_qp.rq.max_sge = dev_attr->max_qp_sges;
> + if (qp->qplib_qp.rq.max_sge > dev_attr->max_qp_sges)
> + qp->qplib_qp.rq.max_sge = dev_attr->max_qp_sges;
> + qp->qplib_qp.sq.max_sge++;
> + if (qp->qplib_qp.sq.max_sge > dev_attr->max_qp_sges)
> + qp->qplib_qp.sq.max_sge = dev_attr->max_qp_sges;
> +
> + qp->qplib_qp.rq_hdr_buf_size =
> + BNXT_QPLIB_MAX_QP1_RQ_HDR_SIZE_V2;
> +
> + qp->qplib_qp.sq_hdr_buf_size =
> + BNXT_QPLIB_MAX_QP1_SQ_HDR_SIZE_V2;
> + qp->qplib_qp.dpi = &rdev->dpi_privileged;
> + rc = bnxt_qplib_create_qp1(&rdev->qplib_res, &qp->qplib_qp);
> + if (rc) {
> + dev_err(rdev_to_dev(rdev), "Failed to create HW QP1");
> + goto fail;
> + }
> + /* Create a shadow QP to handle the QP1 traffic */
> + rdev->qp1_sqp = bnxt_re_create_shadow_qp(pd, &rdev->qplib_res,
> + &qp->qplib_qp);
> + if (!rdev->qp1_sqp) {
> + rc = -EINVAL;
> + dev_err(rdev_to_dev(rdev),
> + "Failed to create Shadow QP for QP1");
> + goto qp_destroy;
> + }
> + rdev->sqp_ah = bnxt_re_create_shadow_qp_ah(pd, &rdev->qplib_res,
> + &qp->qplib_qp);
> + if (!rdev->sqp_ah) {
> + bnxt_qplib_destroy_qp(&rdev->qplib_res,
> + &rdev->qp1_sqp->qplib_qp);
> + rc = -EINVAL;
> + dev_err(rdev_to_dev(rdev),
> + "Failed to create AH entry for ShadowQP");
> + goto qp_destroy;
> + }
> +
> + } else {
> + qp->qplib_qp.max_rd_atomic = dev_attr->max_qp_rd_atom;
> + qp->qplib_qp.max_dest_rd_atomic = dev_attr->max_qp_init_rd_atom;
> + if (udata) {
> + rc = bnxt_re_init_user_qp(rdev, pd, qp, udata);
> + if (rc)
> + goto fail;
> + } else {
> + qp->qplib_qp.dpi = &rdev->dpi_privileged;
> + }
> +
> + rc = bnxt_qplib_create_qp(&rdev->qplib_res, &qp->qplib_qp);
> + if (rc) {
> + dev_err(rdev_to_dev(rdev), "Failed to create HW QP");
> + goto fail;
> + }
> + }
> +
> + qp->ib_qp.qp_num = qp->qplib_qp.id;
> + spin_lock_init(&qp->sq_lock);
> +
> + if (udata) {
> + struct bnxt_re_qp_resp resp;
> +
> + resp.qpid = qp->ib_qp.qp_num;
> + rc = bnxt_re_copy_to_udata(rdev, &resp, sizeof(resp), udata);
> + if (rc) {
> + dev_err(rdev_to_dev(rdev), "Failed to copy QP udata");
> + goto qp_destroy;
> + }
> + }
> + INIT_LIST_HEAD(&qp->list);
> + mutex_lock(&rdev->qp_lock);
> + list_add_tail(&qp->list, &rdev->qp_list);
> + atomic_inc(&rdev->qp_count);
> + mutex_unlock(&rdev->qp_lock);
> +
> + return &qp->ib_qp;
> +qp_destroy:
> + bnxt_qplib_destroy_qp(&rdev->qplib_res, &qp->qplib_qp);
> +fail:
> + kfree(qp);
> + return ERR_PTR(rc);
> +}
> +
> +static u8 __from_ib_qp_state(enum ib_qp_state state)
> +{
> + switch (state) {
> + case IB_QPS_RESET:
> + return CMDQ_MODIFY_QP_NEW_STATE_RESET;
> + case IB_QPS_INIT:
> + return CMDQ_MODIFY_QP_NEW_STATE_INIT;
> + case IB_QPS_RTR:
> + return CMDQ_MODIFY_QP_NEW_STATE_RTR;
> + case IB_QPS_RTS:
> + return CMDQ_MODIFY_QP_NEW_STATE_RTS;
> + case IB_QPS_SQD:
> + return CMDQ_MODIFY_QP_NEW_STATE_SQD;
> + case IB_QPS_SQE:
> + return CMDQ_MODIFY_QP_NEW_STATE_SQE;
> + case IB_QPS_ERR:
> + default:
> + return CMDQ_MODIFY_QP_NEW_STATE_ERR;
> + }
> +}
> +
> +static enum ib_qp_state __to_ib_qp_state(u8 state)
> +{
> + switch (state) {
> + case CMDQ_MODIFY_QP_NEW_STATE_RESET:
> + return IB_QPS_RESET;
> + case CMDQ_MODIFY_QP_NEW_STATE_INIT:
> + return IB_QPS_INIT;
> + case CMDQ_MODIFY_QP_NEW_STATE_RTR:
> + return IB_QPS_RTR;
> + case CMDQ_MODIFY_QP_NEW_STATE_RTS:
> + return IB_QPS_RTS;
> + case CMDQ_MODIFY_QP_NEW_STATE_SQD:
> + return IB_QPS_SQD;
> + case CMDQ_MODIFY_QP_NEW_STATE_SQE:
> + return IB_QPS_SQE;
> + case CMDQ_MODIFY_QP_NEW_STATE_ERR:
> + default:
> + return IB_QPS_ERR;
> + }
> +}
> +
> +static u32 __from_ib_mtu(enum ib_mtu mtu)
> +{
> + switch (mtu) {
> + case IB_MTU_256:
> + return CMDQ_MODIFY_QP_PATH_MTU_MTU_256;
> + case IB_MTU_512:
> + return CMDQ_MODIFY_QP_PATH_MTU_MTU_512;
> + case IB_MTU_1024:
> + return CMDQ_MODIFY_QP_PATH_MTU_MTU_1024;
> + case IB_MTU_2048:
> + return CMDQ_MODIFY_QP_PATH_MTU_MTU_2048;
> + case IB_MTU_4096:
> + return CMDQ_MODIFY_QP_PATH_MTU_MTU_4096;
> + default:
> + return CMDQ_MODIFY_QP_PATH_MTU_MTU_2048;
> + }
> +}
> +
> +static enum ib_mtu __to_ib_mtu(u32 mtu)
> +{
> + switch (mtu & CREQ_QUERY_QP_RESP_SB_PATH_MTU_MASK) {
> + case CMDQ_MODIFY_QP_PATH_MTU_MTU_256:
> + return IB_MTU_256;
> + case CMDQ_MODIFY_QP_PATH_MTU_MTU_512:
> + return IB_MTU_512;
> + case CMDQ_MODIFY_QP_PATH_MTU_MTU_1024:
> + return IB_MTU_1024;
> + case CMDQ_MODIFY_QP_PATH_MTU_MTU_2048:
> + return IB_MTU_2048;
> + case CMDQ_MODIFY_QP_PATH_MTU_MTU_4096:
> + return IB_MTU_4096;
> + default:
> + return IB_MTU_2048;
> + }
> +}
> +
> static int __from_ib_access_flags(int iflags)
> {
> int qflags = 0;
> @@ -690,6 +1165,293 @@ static enum ib_access_flags __to_ib_access_flags(int qflags)
> iflags |= IB_ACCESS_ON_DEMAND;
> return iflags;
> };
> +
> +static int bnxt_re_modify_shadow_qp(struct bnxt_re_dev *rdev,
> + struct bnxt_re_qp *qp1_qp,
> + int qp_attr_mask)
> +{
> + struct bnxt_re_qp *qp = rdev->qp1_sqp;
> + int rc = 0;
> +
> + if (qp_attr_mask & IB_QP_STATE) {
> + qp->qplib_qp.modify_flags |= CMDQ_MODIFY_QP_MODIFY_MASK_STATE;
> + qp->qplib_qp.state = qp1_qp->qplib_qp.state;
> + }
> + if (qp_attr_mask & IB_QP_PKEY_INDEX) {
> + qp->qplib_qp.modify_flags |= CMDQ_MODIFY_QP_MODIFY_MASK_PKEY;
> + qp->qplib_qp.pkey_index = qp1_qp->qplib_qp.pkey_index;
> + }
> +
> + if (qp_attr_mask & IB_QP_QKEY) {
> + qp->qplib_qp.modify_flags |= CMDQ_MODIFY_QP_MODIFY_MASK_QKEY;
> + /* Using a Random QKEY */
> + qp->qplib_qp.qkey = 0x81818181;
> + }
> + if (qp_attr_mask & IB_QP_SQ_PSN) {
> + qp->qplib_qp.modify_flags |= CMDQ_MODIFY_QP_MODIFY_MASK_SQ_PSN;
> + qp->qplib_qp.sq.psn = qp1_qp->qplib_qp.sq.psn;
> + }
> +
> + rc = bnxt_qplib_modify_qp(&rdev->qplib_res, &qp->qplib_qp);
> + if (rc)
> + dev_err(rdev_to_dev(rdev),
> + "Failed to modify Shadow QP for QP1");
> + return rc;
> +}
> +
> +int bnxt_re_modify_qp(struct ib_qp *ib_qp, struct ib_qp_attr *qp_attr,
> + int qp_attr_mask, struct ib_udata *udata)
> +{
> + struct bnxt_re_qp *qp = to_bnxt_re(ib_qp, struct bnxt_re_qp, ib_qp);
> + struct bnxt_re_dev *rdev = qp->rdev;
> + struct bnxt_qplib_dev_attr *dev_attr = &rdev->dev_attr;
> + enum ib_qp_state curr_qp_state, new_qp_state;
> + int rc, entries;
> + int status;
> + union ib_gid sgid;
> + struct ib_gid_attr sgid_attr;
> + u8 nw_type;
> +
> + qp->qplib_qp.modify_flags = 0;
> + if (qp_attr_mask & IB_QP_STATE) {
> + curr_qp_state = __to_ib_qp_state(qp->qplib_qp.cur_qp_state);
> + new_qp_state = qp_attr->qp_state;
> + if (!ib_modify_qp_is_ok(curr_qp_state, new_qp_state,
> + ib_qp->qp_type, qp_attr_mask,
> + IB_LINK_LAYER_ETHERNET)) {
> + dev_err(rdev_to_dev(rdev),
> + "Invalid attribute mask: %#x specified ",
> + qp_attr_mask);
> + dev_err(rdev_to_dev(rdev),
> + "for qpn: %#x type: %#x",
> + ib_qp->qp_num, ib_qp->qp_type);
> + dev_err(rdev_to_dev(rdev),
> + "curr_qp_state=0x%x, new_qp_state=0x%x\n",
> + curr_qp_state, new_qp_state);
> + return -EINVAL;
> + }
> + qp->qplib_qp.modify_flags |= CMDQ_MODIFY_QP_MODIFY_MASK_STATE;
> + qp->qplib_qp.state = __from_ib_qp_state(qp_attr->qp_state);
> + }
> + if (qp_attr_mask & IB_QP_EN_SQD_ASYNC_NOTIFY) {
> + qp->qplib_qp.modify_flags |=
> + CMDQ_MODIFY_QP_MODIFY_MASK_EN_SQD_ASYNC_NOTIFY;
> + qp->qplib_qp.en_sqd_async_notify = true;
> + }
> + if (qp_attr_mask & IB_QP_ACCESS_FLAGS) {
> + qp->qplib_qp.modify_flags |= CMDQ_MODIFY_QP_MODIFY_MASK_ACCESS;
> + qp->qplib_qp.access =
> + __from_ib_access_flags(qp_attr->qp_access_flags);
> + /* LOCAL_WRITE access must be set to allow RC receive */
> + qp->qplib_qp.access |= BNXT_QPLIB_ACCESS_LOCAL_WRITE;
> + }
> + if (qp_attr_mask & IB_QP_PKEY_INDEX) {
> + qp->qplib_qp.modify_flags |= CMDQ_MODIFY_QP_MODIFY_MASK_PKEY;
> + qp->qplib_qp.pkey_index = qp_attr->pkey_index;
> + }
> + if (qp_attr_mask & IB_QP_QKEY) {
> + qp->qplib_qp.modify_flags |= CMDQ_MODIFY_QP_MODIFY_MASK_QKEY;
> + qp->qplib_qp.qkey = qp_attr->qkey;
> + }
> + if (qp_attr_mask & IB_QP_AV) {
> + qp->qplib_qp.modify_flags |= CMDQ_MODIFY_QP_MODIFY_MASK_DGID |
> + CMDQ_MODIFY_QP_MODIFY_MASK_FLOW_LABEL |
> + CMDQ_MODIFY_QP_MODIFY_MASK_SGID_INDEX |
> + CMDQ_MODIFY_QP_MODIFY_MASK_HOP_LIMIT |
> + CMDQ_MODIFY_QP_MODIFY_MASK_TRAFFIC_CLASS |
> + CMDQ_MODIFY_QP_MODIFY_MASK_DEST_MAC |
> + CMDQ_MODIFY_QP_MODIFY_MASK_VLAN_ID;
> + memcpy(qp->qplib_qp.ah.dgid.data, qp_attr->ah_attr.grh.dgid.raw,
> + sizeof(qp->qplib_qp.ah.dgid.data));
> + qp->qplib_qp.ah.flow_label = qp_attr->ah_attr.grh.flow_label;
> + /* If RoCE V2 is enabled, stack will have two entries for
> + * each GID entry. Avoiding this duplicte entry in HW. Dividing
> + * the GID index by 2 for RoCE V2
> + */
> + qp->qplib_qp.ah.sgid_index =
> + qp_attr->ah_attr.grh.sgid_index / 2;
> + qp->qplib_qp.ah.host_sgid_index =
> + qp_attr->ah_attr.grh.sgid_index;
> + qp->qplib_qp.ah.hop_limit = qp_attr->ah_attr.grh.hop_limit;
> + qp->qplib_qp.ah.traffic_class =
> + qp_attr->ah_attr.grh.traffic_class;
> + qp->qplib_qp.ah.sl = qp_attr->ah_attr.sl;
> + ether_addr_copy(qp->qplib_qp.ah.dmac, qp_attr->ah_attr.dmac);
> +
> + status = ib_get_cached_gid(&rdev->ibdev, 1,
> + qp_attr->ah_attr.grh.sgid_index,
> + &sgid, &sgid_attr);
> + if (!status && sgid_attr.ndev) {
> + memcpy(qp->qplib_qp.smac, sgid_attr.ndev->dev_addr,
> + ETH_ALEN);
> + dev_put(sgid_attr.ndev);
> + nw_type = ib_gid_to_network_type(sgid_attr.gid_type,
> + &sgid);
> + switch (nw_type) {
> + case RDMA_NETWORK_IPV4:
> + qp->qplib_qp.nw_type =
> + CMDQ_MODIFY_QP_NETWORK_TYPE_ROCEV2_IPV4;
> + break;
> + case RDMA_NETWORK_IPV6:
> + qp->qplib_qp.nw_type =
> + CMDQ_MODIFY_QP_NETWORK_TYPE_ROCEV2_IPV6;
> + break;
> + default:
> + qp->qplib_qp.nw_type =
> + CMDQ_MODIFY_QP_NETWORK_TYPE_ROCEV1;
> + break;
> + }
> + }
> + }
> +
> + if (qp_attr_mask & IB_QP_PATH_MTU) {
> + qp->qplib_qp.modify_flags |=
> + CMDQ_MODIFY_QP_MODIFY_MASK_PATH_MTU;
> + qp->qplib_qp.path_mtu = __from_ib_mtu(qp_attr->path_mtu);
> + } else if (qp_attr->qp_state == IB_QPS_RTR) {
> + qp->qplib_qp.modify_flags |=
> + CMDQ_MODIFY_QP_MODIFY_MASK_PATH_MTU;
> + qp->qplib_qp.path_mtu =
> + __from_ib_mtu(iboe_get_mtu(rdev->netdev->mtu));
> + }
> +
> + if (qp_attr_mask & IB_QP_TIMEOUT) {
> + qp->qplib_qp.modify_flags |= CMDQ_MODIFY_QP_MODIFY_MASK_TIMEOUT;
> + qp->qplib_qp.timeout = qp_attr->timeout;
> + }
> + if (qp_attr_mask & IB_QP_RETRY_CNT) {
> + qp->qplib_qp.modify_flags |=
> + CMDQ_MODIFY_QP_MODIFY_MASK_RETRY_CNT;
> + qp->qplib_qp.retry_cnt = qp_attr->retry_cnt;
> + }
> + if (qp_attr_mask & IB_QP_RNR_RETRY) {
> + qp->qplib_qp.modify_flags |=
> + CMDQ_MODIFY_QP_MODIFY_MASK_RNR_RETRY;
> + qp->qplib_qp.rnr_retry = qp_attr->rnr_retry;
> + }
> + if (qp_attr_mask & IB_QP_MIN_RNR_TIMER) {
> + qp->qplib_qp.modify_flags |=
> + CMDQ_MODIFY_QP_MODIFY_MASK_MIN_RNR_TIMER;
> + qp->qplib_qp.min_rnr_timer = qp_attr->min_rnr_timer;
> + }
> + if (qp_attr_mask & IB_QP_RQ_PSN) {
> + qp->qplib_qp.modify_flags |= CMDQ_MODIFY_QP_MODIFY_MASK_RQ_PSN;
> + qp->qplib_qp.rq.psn = qp_attr->rq_psn;
> + }
> + if (qp_attr_mask & IB_QP_MAX_QP_RD_ATOMIC) {
> + qp->qplib_qp.modify_flags |=
> + CMDQ_MODIFY_QP_MODIFY_MASK_MAX_RD_ATOMIC;
> + qp->qplib_qp.max_rd_atomic = qp_attr->max_rd_atomic;
> + }
> + if (qp_attr_mask & IB_QP_SQ_PSN) {
> + qp->qplib_qp.modify_flags |= CMDQ_MODIFY_QP_MODIFY_MASK_SQ_PSN;
> + qp->qplib_qp.sq.psn = qp_attr->sq_psn;
> + }
> + if (qp_attr_mask & IB_QP_MAX_DEST_RD_ATOMIC) {
> + qp->qplib_qp.modify_flags |=
> + CMDQ_MODIFY_QP_MODIFY_MASK_MAX_DEST_RD_ATOMIC;
> + qp->qplib_qp.max_dest_rd_atomic = qp_attr->max_dest_rd_atomic;
> + }
> + if (qp_attr_mask & IB_QP_CAP) {
> + qp->qplib_qp.modify_flags |=
> + CMDQ_MODIFY_QP_MODIFY_MASK_SQ_SIZE |
> + CMDQ_MODIFY_QP_MODIFY_MASK_RQ_SIZE |
> + CMDQ_MODIFY_QP_MODIFY_MASK_SQ_SGE |
> + CMDQ_MODIFY_QP_MODIFY_MASK_RQ_SGE |
> + CMDQ_MODIFY_QP_MODIFY_MASK_MAX_INLINE_DATA;
> + if ((qp_attr->cap.max_send_wr >= dev_attr->max_qp_wqes) ||
> + (qp_attr->cap.max_recv_wr >= dev_attr->max_qp_wqes) ||
> + (qp_attr->cap.max_send_sge >= dev_attr->max_qp_sges) ||
> + (qp_attr->cap.max_recv_sge >= dev_attr->max_qp_sges) ||
> + (qp_attr->cap.max_inline_data >=
> + dev_attr->max_inline_data)) {
> + dev_err(rdev_to_dev(rdev),
> + "Create QP failed - max exceeded");
> + return -EINVAL;
> + }
> + entries = roundup_pow_of_two(qp_attr->cap.max_send_wr);
> + if (entries > dev_attr->max_qp_wqes)
> + entries = dev_attr->max_qp_wqes;
> + qp->qplib_qp.sq.max_wqe = entries;
> + qp->qplib_qp.sq.max_sge = qp_attr->cap.max_send_sge;
> + if (qp->qplib_qp.rq.max_wqe) {
> + entries = roundup_pow_of_two(qp_attr->cap.max_recv_wr);
> + if (entries > dev_attr->max_qp_wqes)
> + entries = dev_attr->max_qp_wqes;
> + qp->qplib_qp.rq.max_wqe = entries;
> + qp->qplib_qp.rq.max_sge = qp_attr->cap.max_recv_sge;
> + } else {
> + /* SRQ was used prior, just ignore the RQ caps */
> + }
> + }
> + if (qp_attr_mask & IB_QP_DEST_QPN) {
> + qp->qplib_qp.modify_flags |=
> + CMDQ_MODIFY_QP_MODIFY_MASK_DEST_QP_ID;
> + qp->qplib_qp.dest_qpn = qp_attr->dest_qp_num;
> + }
> + rc = bnxt_qplib_modify_qp(&rdev->qplib_res, &qp->qplib_qp);
> + if (rc) {
> + dev_err(rdev_to_dev(rdev), "Failed to modify HW QP");
> + return rc;
> + }
> + if (ib_qp->qp_type == IB_QPT_GSI && rdev->qp1_sqp)
> + rc = bnxt_re_modify_shadow_qp(rdev, qp, qp_attr_mask);
> + return rc;
> +}
> +
> +int bnxt_re_query_qp(struct ib_qp *ib_qp, struct ib_qp_attr *qp_attr,
> + int qp_attr_mask, struct ib_qp_init_attr *qp_init_attr)
> +{
> + struct bnxt_re_qp *qp = to_bnxt_re(ib_qp, struct bnxt_re_qp, ib_qp);
> + struct bnxt_re_dev *rdev = qp->rdev;
> + struct bnxt_qplib_qp qplib_qp;
> + int rc;
> +
> + memset(&qplib_qp, 0, sizeof(struct bnxt_qplib_qp));
> + qplib_qp.id = qp->qplib_qp.id;
> + qplib_qp.ah.host_sgid_index = qp->qplib_qp.ah.host_sgid_index;
> +
> + rc = bnxt_qplib_query_qp(&rdev->qplib_res, &qplib_qp);
> + if (rc) {
> + dev_err(rdev_to_dev(rdev), "Failed to query HW QP");
> + return rc;
> + }
> + qp_attr->qp_state = __to_ib_qp_state(qplib_qp.state);
> + qp_attr->en_sqd_async_notify = qplib_qp.en_sqd_async_notify ? 1 : 0;
> + qp_attr->qp_access_flags = __to_ib_access_flags(qplib_qp.access);
> + qp_attr->pkey_index = qplib_qp.pkey_index;
> + qp_attr->qkey = qplib_qp.qkey;
> + memcpy(qp_attr->ah_attr.grh.dgid.raw, qplib_qp.ah.dgid.data,
> + sizeof(qplib_qp.ah.dgid.data));
> + qp_attr->ah_attr.grh.flow_label = qplib_qp.ah.flow_label;
> + qp_attr->ah_attr.grh.sgid_index = qplib_qp.ah.host_sgid_index;
> + qp_attr->ah_attr.grh.hop_limit = qplib_qp.ah.hop_limit;
> + qp_attr->ah_attr.grh.traffic_class = qplib_qp.ah.traffic_class;
> + qp_attr->ah_attr.sl = qplib_qp.ah.sl;
> + ether_addr_copy(qp_attr->ah_attr.dmac, qplib_qp.ah.dmac);
> + qp_attr->path_mtu = __to_ib_mtu(qplib_qp.path_mtu);
> + qp_attr->timeout = qplib_qp.timeout;
> + qp_attr->retry_cnt = qplib_qp.retry_cnt;
> + qp_attr->rnr_retry = qplib_qp.rnr_retry;
> + qp_attr->min_rnr_timer = qplib_qp.min_rnr_timer;
> + qp_attr->rq_psn = qplib_qp.rq.psn;
> + qp_attr->max_rd_atomic = qplib_qp.max_rd_atomic;
> + qp_attr->sq_psn = qplib_qp.sq.psn;
> + qp_attr->max_dest_rd_atomic = qplib_qp.max_dest_rd_atomic;
> + qp_init_attr->sq_sig_type = qplib_qp.sig_type ? IB_SIGNAL_ALL_WR :
> + IB_SIGNAL_REQ_WR;
> + qp_attr->dest_qp_num = qplib_qp.dest_qpn;
> +
> + qp_attr->cap.max_send_wr = qp->qplib_qp.sq.max_wqe;
> + qp_attr->cap.max_send_sge = qp->qplib_qp.sq.max_sge;
> + qp_attr->cap.max_recv_wr = qp->qplib_qp.rq.max_wqe;
> + qp_attr->cap.max_recv_sge = qp->qplib_qp.rq.max_sge;
> + qp_attr->cap.max_inline_data = qp->qplib_qp.max_inline_data;
> + qp_init_attr->cap = qp_attr->cap;
> +
> + return 0;
> +}
> +
> /* Completion Queues */
> int bnxt_re_destroy_cq(struct ib_cq *ib_cq)
> {
> diff --git a/drivers/infiniband/hw/bnxtre/bnxt_re_ib_verbs.h b/drivers/infiniband/hw/bnxtre/bnxt_re_ib_verbs.h
> index ba9a4c9..75ee88a 100644
> --- a/drivers/infiniband/hw/bnxtre/bnxt_re_ib_verbs.h
> +++ b/drivers/infiniband/hw/bnxtre/bnxt_re_ib_verbs.h
> @@ -57,6 +57,19 @@ struct bnxt_re_ah {
> struct bnxt_qplib_ah qplib_ah;
> };
>
> +struct bnxt_re_qp {
> + struct list_head list;
> + struct bnxt_re_dev *rdev;
> + struct ib_qp ib_qp;
> + spinlock_t sq_lock; /* protect sq */
> + struct bnxt_qplib_qp qplib_qp;
> + struct ib_umem *sumem;
> + struct ib_umem *rumem;
> + /* QP1 */
> + u32 send_psn;
> + struct ib_ud_header qp1_hdr;
> +};
> +
> struct bnxt_re_cq {
> struct bnxt_re_dev *rdev;
> spinlock_t cq_lock; /* protect cq */
> @@ -141,6 +154,14 @@ struct ib_ah *bnxt_re_create_ah(struct ib_pd *pd,
> int bnxt_re_modify_ah(struct ib_ah *ah, struct ib_ah_attr *ah_attr);
> int bnxt_re_query_ah(struct ib_ah *ah, struct ib_ah_attr *ah_attr);
> int bnxt_re_destroy_ah(struct ib_ah *ah);
> +struct ib_qp *bnxt_re_create_qp(struct ib_pd *pd,
> + struct ib_qp_init_attr *qp_init_attr,
> + struct ib_udata *udata);
> +int bnxt_re_modify_qp(struct ib_qp *qp, struct ib_qp_attr *qp_attr,
> + int qp_attr_mask, struct ib_udata *udata);
> +int bnxt_re_query_qp(struct ib_qp *qp, struct ib_qp_attr *qp_attr,
> + int qp_attr_mask, struct ib_qp_init_attr *qp_init_attr);
> +int bnxt_re_destroy_qp(struct ib_qp *qp);
> struct ib_cq *bnxt_re_create_cq(struct ib_device *ibdev,
> const struct ib_cq_init_attr *attr,
> struct ib_ucontext *context,
> diff --git a/drivers/infiniband/hw/bnxtre/bnxt_re_main.c b/drivers/infiniband/hw/bnxtre/bnxt_re_main.c
> index 3d1504e..5facacc 100644
> --- a/drivers/infiniband/hw/bnxtre/bnxt_re_main.c
> +++ b/drivers/infiniband/hw/bnxtre/bnxt_re_main.c
> @@ -445,6 +445,12 @@ static int bnxt_re_register_ib(struct bnxt_re_dev *rdev)
> ibdev->modify_ah = bnxt_re_modify_ah;
> ibdev->query_ah = bnxt_re_query_ah;
> ibdev->destroy_ah = bnxt_re_destroy_ah;
> +
> + ibdev->create_qp = bnxt_re_create_qp;
> + ibdev->modify_qp = bnxt_re_modify_qp;
> + ibdev->query_qp = bnxt_re_query_qp;
> + ibdev->destroy_qp = bnxt_re_destroy_qp;
> +
> ibdev->create_cq = bnxt_re_create_cq;
> ibdev->destroy_cq = bnxt_re_destroy_cq;
> ibdev->req_notify_cq = bnxt_re_req_notify_cq;
> diff --git a/include/uapi/rdma/bnxt_re_uverbs_abi.h b/include/uapi/rdma/bnxt_re_uverbs_abi.h
> index 5444eff..e6732f8 100644
> --- a/include/uapi/rdma/bnxt_re_uverbs_abi.h
> +++ b/include/uapi/rdma/bnxt_re_uverbs_abi.h
> @@ -66,6 +66,16 @@ struct bnxt_re_cq_resp {
> __u32 phase;
> } __packed;
>
> +struct bnxt_re_qp_req {
> + __u64 qpsva;
> + __u64 qprva;
> + __u64 qp_handle;
> +} __packed;
> +
> +struct bnxt_re_qp_resp {
> + __u32 qpid;
> +} __packed;
> +
> enum bnxt_re_shpg_offt {
> BNXT_RE_BEG_RESV_OFFT = 0x00,
> BNXT_RE_AVID_OFFT = 0x10,
> --
> 2.5.5
>
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* [LSF/MM TOPIC][LSF/MM ATTEND] Enabling Peer-to-Peer DMAs between PCIe devices
From: Stephen Bates @ 2016-12-12 17:51 UTC (permalink / raw)
To: lsf-pc; +Cc: linux-kernel, linux-mm, linux-fsdevel, linux-rdma, linux-nvme
Hi
I'd like to discuss the topic of how best to enable DMAs between PCIe
devices in the Linux kernel.
There have been many attempts to add to the kernel the ability to DMA
between two PCIe devices. However, to date, none of these have been
accepted. However as PCIe devices like NICs, NVMe SSDs and GPGPUs continue
to get faster the desire to move data directly between these devices (as
opposed to having to using a temporary buffer in system memory) is
increasing. Out of tree solutions like GPU-Direct are one illustration of
the popularity of this functionality. A recent discussion on this topic
provides a good summary of where things stand [1].
I would like to propose a session at LFS/MM to discuss some of the
different use cases for these P2P DMAs and also to discuss the pros and
cons of these approaches. The desire would be to try and form a consensus
on how best to move forward to an upstreamable solution to this problem.
In addition I would also be interested in participating in the following
topics:
* Anything related to PMEM and DAX.
* Integrating the block-layer polling capability into file-systems.
* New feature integration into the NVMe driver (e.g. fabrics, CMBs, IO
tags etc.)
Cheers
Stephen
[1] http://marc.info/?l=linux-pci&m=147976059431355&w=2 (and subsequent
thread).
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH V2 00/22] Broadcom RoCE Driver (bnxt_re)
From: Jason Gunthorpe @ 2016-12-12 17:07 UTC (permalink / raw)
To: Selvin Xavier
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
netdev-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <CA+sbYW1ZEa5fndGkvN8OXr-orcUx4jaL73Di8zBJQX_uCdK=Ww-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
On Sat, Dec 10, 2016 at 11:06:58AM +0530, Selvin Xavier wrote:
> On Fri, Dec 9, 2016 at 12:17 PM, Selvin Xavier
> <selvin.xavier-dY08KVG/lbpWk0Htik3J/w@public.gmane.org> wrote:
> > I am preparing a git repository with these changes as per Jason's
> > comment and will share the details later today.
>
> Please use bnxt_re branch in this git repository.
>
> https://github.com/Broadcom/linux-rdma-nxt.git
Why are you using __packed in bnxt_re_uverbs_abi.h ? that doesn't seem
necessary. It is a good idea to make sure all those structures are a
multiple of 64 bits (add explicit reserved fields), and make sure you
test 32 bit verbs as well.
Why are you using debugfs just to export counters? Isn't the core code
counter framework good enough?
Please try and avoid writing functions as defines (eg rdev_to_dev,
to_bnxt_re, SQE_PG, RCFW_CMDQ_COOKIE, PTR_PG etc)
There is something wrong with the tabs and spaces (see
https://github.com/Broadcom/linux-rdma-nxt/blob/03e23b087f7e86ea28656273994e065827210ce5/drivers/infiniband/hw/bnxtre/bnxt_re_hsi.h)
FWIW, I really dislike the column alignment style, it is so hard to
maintain..
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH V2 00/22] Broadcom RoCE Driver (bnxt_re)
From: Jonathan Toppins @ 2016-12-12 16:54 UTC (permalink / raw)
To: Selvin Xavier, dledford-H+wXaHxf7aLQT0dZR+AlfA,
linux-rdma-u79uwXL29TY76Z2rM5mHXA
Cc: netdev-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <1481266096-23331-1-git-send-email-selvin.xavier-dY08KVG/lbpWk0Htik3J/w@public.gmane.org>
On 12/09/2016 01:47 AM, Selvin Xavier wrote:
> This series introduces the RoCE driver for the Broadcom
> NetXtreme-E 10/25/40/50 gigabit RoCE HCAs.
> This driver is dependent on the bnxt_en NIC driver and is
> based on the bnxt_re branch in Doug's repository. bnxt_en changes
> required for this patch series is already available in this branch.
>
> I am preparing a git repository with these changes as per Jason's
> comment and will share the details later today.
>
> v1-> v2:
> * The license text in each file updated to reflect Dual license.
> * Makefile and Kconfig changes are pushed to the last patch
> * Moved bnxt_re_uverbs_abi.h to include/uapi/rdma folder
> * Remove duplicate structure definitions from bnxt_re_hsi.h as
> it is available in the corresponding bnxt_en header file (bnxt_hsi.h)
> * Removed some unused code reported during code review.
> * Fixed few sparse warnings
>
I get the following sparse errors (filtered for only bnxt_re ones),
please let me know if they are false positives:
$ make C=2 drivers/net/ethernet/broadcom/bnxt/bnxt_en.ko
drivers/infiniband/hw/bnxtre/bnxt_re.ko
CHK include/config/kernel.release
CHK include/generated/uapi/linux/version.h
CHK include/generated/utsrelease.h
CHECK arch/x86/purgatory/purgatory.c
[...]
CHECK arch/x86/purgatory/sha256.c
CHECK arch/x86/purgatory/string.c
[...]
CHK include/generated/bounds.h
CHK include/generated/timeconst.h
CHK include/generated/asm-offsets.h
CALL scripts/checksyscalls.sh
CHECK scripts/mod/empty.c
CHECK drivers/net/ethernet/broadcom/bnxt/bnxt.c
CHECK drivers/net/ethernet/broadcom/bnxt/bnxt_sriov.c
CHECK drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c
CHECK drivers/net/ethernet/broadcom/bnxt/bnxt_dcb.c
CHECK drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c
MODPOST 2 modules
CHECK drivers/infiniband/hw/bnxtre/bnxt_re_main.c
CHECK drivers/infiniband/hw/bnxtre/bnxt_re_ib_verbs.c
[...]
CHECK drivers/infiniband/hw/bnxtre/bnxt_re_debugfs.c
CHECK drivers/infiniband/hw/bnxtre/bnxt_qplib_res.c
drivers/infiniband/hw/bnxtre/bnxt_qplib_res.c:729:6: warning: symbol
'bnxt_qplib_cleanup_pkey_tbl' was not declared. Should it be static?
CHECK drivers/infiniband/hw/bnxtre/bnxt_qplib_rcfw.c
CHECK drivers/infiniband/hw/bnxtre/bnxt_qplib_sp.c
CHECK drivers/infiniband/hw/bnxtre/bnxt_qplib_fp.c
drivers/infiniband/hw/bnxtre/bnxt_qplib_fp.c:1015:22: warning: context
imbalance in 'bnxt_qplib_lock_cqs' - wrong count at exit
drivers/infiniband/hw/bnxtre/bnxt_qplib_fp.c:1030:28: warning: context
imbalance in 'bnxt_qplib_unlock_cqs' - unexpected unlock
MODPOST 2 modules
-Jon
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* SR-IOV with mlx4 on ConnectX-2 fails with DMAR errors
From: Joshua McBeth @ 2016-12-12 16:04 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA
[-- Attachment #1: Type: text/plain, Size: 5328 bytes --]
I am having some issues getting SR-IOV working with a Mellanox
ConnectX-2 in a Supermicro X8DTH-6F.
The Infiniband adapter has been flashed with the latest compatible
firmware with SR-IOV enabled and SR-IOV/virtualization is enabled in
the BIOS and working for other hardware (with 2 gigabit ethernet, 1
wireless ethernet, 1 nVidia GPU passed through to qemu/kvm guests).
The Infiniband adapter functions as expected if SR-IOV is not enabled
in the driver.
When I enable SR-IOV in the mlx4 driver ( mlx4_core.port_type_array=1
mlx4_core.num_vfs=8 mlx4_core.probe_vf=0 ), the ib0 interface does not
function. The link is never reported as up in dmesg or ibstat, but a
DMAR error is reported around when the link would be expected to come
up. Attempts to use ibping result in additional DMAR errors and no
responses are received by the ibping application. The DMAR errors are
reported for a different bus address than the iommu seems to be
getting configured for so I am thinking this is a driver error.
I have excerpted the issue below.
Here the devices added to the iommu are 0000:05:0x.x - 0000:05:01.0
------ dmesg excerpts
[ 44.410799] mlx4_core 0000:05:00.0: Enabling SR-IOV with 8 VFs
[ 44.512772] pci 0000:05:00.1: [15b3:1002] type 00 class 0x0c0600
[ 44.513052] pci 0000:05:00.1: Max Payload Size set to 256 (was 128, max 256)
[ 44.513520] iommu: Adding device 0000:05:00.1 to group 44
[ 44.513722] mlx4_core: Initializing 0000:05:00.1
[ 44.513891] mlx4_core 0000:05:00.1: enabling device (0000 -> 0002)
[ 44.514081] mlx4_core 0000:05:00.1: Skipping virtual function:1
[ 44.514332] pci 0000:05:00.2: [15b3:1002] type 00 class 0x0c0600
[ 44.514604] pci 0000:05:00.2: Max Payload Size set to 256 (was 128, max 256)
[ 44.515047] iommu: Adding device 0000:05:00.2 to group 45
[ 44.515225] mlx4_core: Initializing 0000:05:00.2
[ 44.515388] mlx4_core 0000:05:00.2: enabling device (0000 -> 0002)
[ 44.515572] mlx4_core 0000:05:00.2: Skipping virtual function:2
...
[ 44.523297] pci 0000:05:01.0: [15b3:1002] type 00 class 0x0c0600
[ 44.523570] pci 0000:05:01.0: Max Payload Size set to 256 (was 128, max 256)
[ 44.524007] iommu: Adding device 0000:05:01.0 to group 51
[ 44.524194] mlx4_core: Initializing 0000:05:01.0
[ 44.524363] mlx4_core 0000:05:01.0: enabling device (0000 -> 0002)
[ 44.524554] mlx4_core 0000:05:01.0: Skipping virtual function:8
[ 44.524746] mlx4_core 0000:05:00.0: Running in master mode
[ 46.867330] mlx4_core 0000:05:00.0: PCIe link speed is 5.0GT/s,
device supports 5.0GT/s
[ 46.867613] mlx4_core 0000:05:00.0: PCIe link width is x8, device supports x8
[ 46.910736] mlx4_core: Initializing 0000:05:00.1
[ 46.910913] mlx4_core 0000:05:00.1: enabling device (0000 -> 0002)
[ 46.911102] mlx4_core 0000:05:00.1: Skipping virtual function:1
...
[ 46.915085] mlx4_core: Initializing 0000:05:01.0
[ 46.915257] mlx4_core 0000:05:01.0: enabling device (0000 -> 0002)
[ 46.915440] mlx4_core 0000:05:01.0: Skipping virtual function:8
---
Interface is brought up here by init scripts and what I assume is the
link state notification seems to be eaten by iommu
The adapter seems to now have the bus address [0000:]05:06.1?
---
[ 71.631199] DMAR: DRHD: handling fault status reg 2
[ 71.631204] DMAR: [DMA Read] Request device [05:06.1] fault addr
c2652b000 [fault reason 02] Present bit in context entry is clear
[ 72.020267] ib0: enabling connected mode will cause multicast packet drops
[ 72.020307] ib0: mtu > 2044 will cause multicast packet drops.
------
Here I attempt to ibping another node 3 times and each packet results
in a DMAR error, again with a different bus address than was added to
the IOMMU:
------ dmesg excerpt continues
[ 103.134429] DMAR: DRHD: handling fault status reg 102
[ 103.134434] DMAR: [DMA Read] Request device [05:06.1] fault addr
81b081000 [fault reason 02] Present bit in context entry is clear
[ 105.135927] DMAR: DRHD: handling fault status reg 202
[ 105.136013] DMAR: [DMA Read] Request device [05:06.1] fault addr
81b081000 [fault reason 02] Present bit in context entry is clear
[ 107.137479] DMAR: DRHD: handling fault status reg 302
[ 107.137484] DMAR: [DMA Read] Request device [05:06.1] fault addr
81b081000 [fault reason 02] Present bit in context entry is clear
------ uname -a
Linux cuprum 4.8.1-gentoo #1 SMP Sun Dec 11 00:05:06 UTC 2016 x86_64
Intel(R) Xeon(R) CPU X5650 @ 2.67GHz GenuineIntel GNU/Linux
------- lspci excerpt with SR-IOV disabled
05:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe
2.0 5GT/s - IB QDR / 10GigE] (rev b0)
Subsystem: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s
- IB QDR / 10GigE]
Flags: bus master, fast devsel, latency 0, IRQ 49, NUMA node 0
Memory at fae00000 (64-bit, non-prefetchable) [size=1M]
Memory at f8800000 (64-bit, prefetchable) [size=8M]
Capabilities: [40] Power Management version 3
Capabilities: [48] Vital Product Data
Capabilities: [9c] MSI-X: Enable+ Count=128 Masked-
Capabilities: [60] Express Endpoint, MSI 00
Capabilities: [100] Alternative Routing-ID Interpretation (ARI)
Capabilities: [148] Device Serial Number 00-02-c9-03-00-07-7d-2e
Capabilities: [108] Single Root I/O Virtualization (SR-IOV)
Kernel driver in use: mlx4_core
Kernel modules: mlx4_core
------ full dmesg is attached
[-- Attachment #2: dmesg.log --]
[-- Type: text/x-log, Size: 144938 bytes --]
[ 0.000000] Linux version 4.8.1-gentoo (root@cuprum) (gcc version 4.9.3 (Gentoo 4.9.3 p1.5, pie-0.6.4) ) #1 SMP Sat Dec 3 05:50:35 UTC 2016
[ 0.000000] Command line: BOOT_IMAGE=/boot/kernel-genkernel-x86_64-4.8.1-gentoo root=/dev/mapper/local_mirror-root ro intel_iommu=on iommu=pt dolvm domdadm mlx4_core.port_type_array=1 mlx4_core.num_vfs=8 mlx4_core.probe_vf=0
[ 0.000000] x86/fpu: Legacy x87 FPU detected.
[ 0.000000] x86/fpu: Using 'eager' FPU context switches.
[ 0.000000] e820: BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009bfff] usable
[ 0.000000] BIOS-e820: [mem 0x000000000009c000-0x000000000009ffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bf77ffff] usable
[ 0.000000] BIOS-e820: [mem 0x00000000bf78e000-0x00000000bf78ffff] type 9
[ 0.000000] BIOS-e820: [mem 0x00000000bf790000-0x00000000bf79dfff] ACPI data
[ 0.000000] BIOS-e820: [mem 0x00000000bf79e000-0x00000000bf7cffff] ACPI NVS
[ 0.000000] BIOS-e820: [mem 0x00000000bf7d0000-0x00000000bf7dffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000bf7ec000-0x00000000bfffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000e0000000-0x00000000efffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000ffc00000-0x00000000ffffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000c3fffffff] usable
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] SMBIOS 2.6 present.
[ 0.000000] DMI: Supermicro X8DTH-i/6/iF/6F/X8DTH, BIOS 2.1b 05/04/12
[ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[ 0.000000] e820: last_pfn = 0xc40000 max_arch_pfn = 0x400000000
[ 0.000000] MTRR default type: uncachable
[ 0.000000] MTRR fixed ranges enabled:
[ 0.000000] 00000-9FFFF write-back
[ 0.000000] A0000-BFFFF uncachable
[ 0.000000] C0000-CBFFF write-protect
[ 0.000000] CC000-DFFFF uncachable
[ 0.000000] E0000-E3FFF write-protect
[ 0.000000] E4000-EFFFF write-through
[ 0.000000] F0000-FFFFF write-protect
[ 0.000000] MTRR variable ranges enabled:
[ 0.000000] 0 base 0000000000 mask F800000000 write-back
[ 0.000000] 1 base 0800000000 mask FC00000000 write-back
[ 0.000000] 2 base 0C00000000 mask FFC0000000 write-back
[ 0.000000] 3 base 00C0000000 mask FFC0000000 uncachable
[ 0.000000] 4 base 00BF800000 mask FFFF800000 uncachable
[ 0.000000] 5 disabled
[ 0.000000] 6 disabled
[ 0.000000] 7 disabled
[ 0.000000] 8 disabled
[ 0.000000] 9 disabled
[ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WC UC- WT
[ 0.000000] e820: update [mem 0xbf800000-0xffffffff] usable ==> reserved
[ 0.000000] e820: last_pfn = 0xbf780 max_arch_pfn = 0x400000000
[ 0.000000] found SMP MP-table at [mem 0x000ff780-0x000ff78f] mapped at [ffff8800000ff780]
[ 0.000000] Scanning 1 areas for low memory corruption
[ 0.000000] Base memory trampoline at [ffff880000096000] 96000 size 24576
[ 0.000000] Using GB pages for direct mapping
[ 0.000000] BRK [0x0237f000, 0x0237ffff] PGTABLE
[ 0.000000] BRK [0x02380000, 0x02380fff] PGTABLE
[ 0.000000] BRK [0x02381000, 0x02381fff] PGTABLE
[ 0.000000] RAMDISK: [mem 0x377b7000-0x37bd2fff]
[ 0.000000] ACPI: Early table checksum verification disabled
[ 0.000000] ACPI: RSDP 0x00000000000FAE30 000024 (v02 ACPIAM)
[ 0.000000] ACPI: XSDT 0x00000000BF790100 000094 (v01 SMCI 20120504 MSFT 00000097)
[ 0.000000] ACPI: FACP 0x00000000BF790290 0000F4 (v03 SUPERM FACP1635 20120504 MSFT 00000097)
[ 0.000000] ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 128/64 (20160422/tbfadt-624)
[ 0.000000] ACPI: DSDT 0x00000000BF790700 0077F7 (v01 10400 10400000 00000000 INTL 20051117)
[ 0.000000] ACPI: FACS 0x00000000BF79E000 000040
[ 0.000000] ACPI: FACS 0x00000000BF79E000 000040
[ 0.000000] ACPI: APIC 0x00000000BF790390 00012A (v01 SUPERM APIC1635 20120504 MSFT 00000097)
[ 0.000000] ACPI: MCFG 0x00000000BF7904C0 00003C (v01 SUPERM OEMMCFG 20120504 MSFT 00000097)
[ 0.000000] ACPI: SLIT 0x00000000BF790500 000030 (v01 SUPERM OEMSLIT 20120504 MSFT 00000097)
[ 0.000000] ACPI: SPMI 0x00000000BF790530 000041 (v05 SUPERM OEMSPMI 20120504 MSFT 00000097)
[ 0.000000] ACPI: OEMB 0x00000000BF79E040 0000A4 (v01 SUPERM OEMB1635 20120504 MSFT 00000097)
[ 0.000000] ACPI: SRAT 0x00000000BF79A700 000250 (v01 SUPERM OEMSRAT 00000001 INTL 00000001)
[ 0.000000] ACPI: HPET 0x00000000BF79A950 000038 (v01 SUPERM OEMHPET 20120504 MSFT 00000097)
[ 0.000000] ACPI: DMAR 0x00000000BF79E0F0 000228 (v01 AMI OEMDMAR 00000001 MSFT 00000097)
[ 0.000000] ACPI: SSDT 0x00000000BF7A4370 000363 (v01 DpgPmm CpuPm 00000012 INTL 20051117)
[ 0.000000] ACPI: EINJ 0x00000000BF79A990 000130 (v01 AMIER AMI_EINJ 20120504 MSFT 00000097)
[ 0.000000] ACPI: BERT 0x00000000BF79AB20 000030 (v01 AMIER AMI_BERT 20120504 MSFT 00000097)
[ 0.000000] ACPI: ERST 0x00000000BF79AB50 0001B0 (v01 AMIER AMI_ERST 20120504 MSFT 00000097)
[ 0.000000] ACPI: HEST 0x00000000BF79AD00 0000A8 (v01 AMIER ABC_HEST 20120504 MSFT 00000097)
[ 0.000000] ACPI: Local APIC address 0xfee00000
[ 0.000000] SRAT: PXM 0 -> APIC 0x00 -> Node 0
[ 0.000000] SRAT: PXM 0 -> APIC 0x02 -> Node 0
[ 0.000000] SRAT: PXM 0 -> APIC 0x04 -> Node 0
[ 0.000000] SRAT: PXM 0 -> APIC 0x10 -> Node 0
[ 0.000000] SRAT: PXM 0 -> APIC 0x12 -> Node 0
[ 0.000000] SRAT: PXM 0 -> APIC 0x14 -> Node 0
[ 0.000000] SRAT: PXM 0 -> APIC 0x01 -> Node 0
[ 0.000000] SRAT: PXM 0 -> APIC 0x03 -> Node 0
[ 0.000000] SRAT: PXM 0 -> APIC 0x05 -> Node 0
[ 0.000000] SRAT: PXM 0 -> APIC 0x11 -> Node 0
[ 0.000000] SRAT: PXM 0 -> APIC 0x13 -> Node 0
[ 0.000000] SRAT: PXM 0 -> APIC 0x15 -> Node 0
[ 0.000000] SRAT: PXM 1 -> APIC 0x20 -> Node 1
[ 0.000000] SRAT: PXM 1 -> APIC 0x22 -> Node 1
[ 0.000000] SRAT: PXM 1 -> APIC 0x24 -> Node 1
[ 0.000000] SRAT: PXM 1 -> APIC 0x30 -> Node 1
[ 0.000000] SRAT: PXM 1 -> APIC 0x32 -> Node 1
[ 0.000000] SRAT: PXM 1 -> APIC 0x34 -> Node 1
[ 0.000000] SRAT: PXM 1 -> APIC 0x21 -> Node 1
[ 0.000000] SRAT: PXM 1 -> APIC 0x23 -> Node 1
[ 0.000000] SRAT: PXM 1 -> APIC 0x25 -> Node 1
[ 0.000000] SRAT: PXM 1 -> APIC 0x31 -> Node 1
[ 0.000000] SRAT: PXM 1 -> APIC 0x33 -> Node 1
[ 0.000000] SRAT: PXM 1 -> APIC 0x35 -> Node 1
[ 0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x0009ffff]
[ 0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x00100000-0xbfffffff]
[ 0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x83fffffff]
[ 0.000000] ACPI: SRAT: Node 1 PXM 1 [mem 0x840000000-0xc3fffffff]
[ 0.000000] NUMA: Initialized distance table, cnt=2
[ 0.000000] NUMA: Node 0 [mem 0x00000000-0x0009ffff] + [mem 0x00100000-0xbfffffff] -> [mem 0x00000000-0xbfffffff]
[ 0.000000] NUMA: Node 0 [mem 0x00000000-0xbfffffff] + [mem 0x100000000-0x83fffffff] -> [mem 0x00000000-0x83fffffff]
[ 0.000000] NODE_DATA(0) allocated [mem 0x83fffc000-0x83fffffff]
[ 0.000000] NODE_DATA(1) allocated [mem 0xc3fff9000-0xc3fffcfff]
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff]
[ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff]
[ 0.000000] Normal [mem 0x0000000100000000-0x0000000c3fffffff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009bfff]
[ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bf77ffff]
[ 0.000000] node 0: [mem 0x0000000100000000-0x000000083fffffff]
[ 0.000000] node 1: [mem 0x0000000840000000-0x0000000c3fffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x000000083fffffff]
[ 0.000000] On node 0 totalpages: 8386331
[ 0.000000] DMA zone: 64 pages used for memmap
[ 0.000000] DMA zone: 21 pages reserved
[ 0.000000] DMA zone: 3995 pages, LIFO batch:0
[ 0.000000] DMA32 zone: 12190 pages used for memmap
[ 0.000000] DMA32 zone: 780160 pages, LIFO batch:31
[ 0.000000] Normal zone: 118784 pages used for memmap
[ 0.000000] Normal zone: 7602176 pages, LIFO batch:31
[ 0.000000] Initmem setup node 1 [mem 0x0000000840000000-0x0000000c3fffffff]
[ 0.000000] On node 1 totalpages: 4194304
[ 0.000000] Normal zone: 65536 pages used for memmap
[ 0.000000] Normal zone: 4194304 pages, LIFO batch:31
[ 0.000000] ACPI: PM-Timer IO Port: 0x808
[ 0.000000] ACPI: Local APIC address 0xfee00000
[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
[ 0.000000] IOAPIC[0]: apic_id 6, version 32, address 0xfec00000, GSI 0-23
[ 0.000000] IOAPIC[1]: apic_id 7, version 32, address 0xfec8a000, GSI 24-47
[ 0.000000] IOAPIC[2]: apic_id 8, version 32, address 0xfec9a000, GSI 48-71
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[ 0.000000] ACPI: IRQ0 used by override.
[ 0.000000] ACPI: IRQ9 used by override.
[ 0.000000] Using ACPI (MADT) for SMP configuration information
[ 0.000000] ACPI: HPET id: 0x8086a301 base: 0xfed00000
[ 0.000000] smpboot: Allowing 24 CPUs, 0 hotplug CPUs
[ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff]
[ 0.000000] PM: Registered nosave memory: [mem 0x0009c000-0x0009ffff]
[ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000dffff]
[ 0.000000] PM: Registered nosave memory: [mem 0x000e0000-0x000fffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xbf780000-0xbf78dfff]
[ 0.000000] PM: Registered nosave memory: [mem 0xbf78e000-0xbf78ffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xbf790000-0xbf79dfff]
[ 0.000000] PM: Registered nosave memory: [mem 0xbf79e000-0xbf7cffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xbf7d0000-0xbf7dffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xbf7e0000-0xbf7ebfff]
[ 0.000000] PM: Registered nosave memory: [mem 0xbf7ec000-0xbfffffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xdfffffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xe0000000-0xefffffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xf0000000-0xfedfffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xfee00000-0xfee00fff]
[ 0.000000] PM: Registered nosave memory: [mem 0xfee01000-0xffbfffff]
[ 0.000000] PM: Registered nosave memory: [mem 0xffc00000-0xffffffff]
[ 0.000000] e820: [mem 0xc0000000-0xdfffffff] available for PCI devices
[ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns
[ 0.000000] setup_percpu: NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:24 nr_node_ids:2
[ 0.000000] percpu: Embedded 34 pages/cpu @ffff88081fa00000 s101400 r8192 d29672 u262144
[ 0.000000] pcpu-alloc: s101400 r8192 d29672 u262144 alloc=1*2097152
[ 0.000000] pcpu-alloc: [0] 00 01 02 03 04 05 12 13 [0] 14 15 16 17 -- -- -- --
[ 0.000000] pcpu-alloc: [1] 06 07 08 09 10 11 18 19 [1] 20 21 22 23 -- -- -- --
[ 0.000000] Built 2 zonelists in Node order, mobility grouping on. Total pages: 12384040
[ 0.000000] Policy zone: Normal
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/kernel-genkernel-x86_64-4.8.1-gentoo root=/dev/mapper/local_mirror-root ro intel_iommu=on iommu=pt dolvm domdadm mlx4_core.port_type_array=1 mlx4_core.num_vfs=8 mlx4_core.probe_vf=0
[ 0.000000] DMAR: IOMMU enabled
[ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.000000] Calgary: detecting Calgary via BIOS EBDA area
[ 0.000000] Calgary: Unable to locate Rio Grande table in EBDA - bailing!
[ 0.000000] Memory: 49442208K/50322540K available (11254K kernel code, 1372K rwdata, 3576K rodata, 1172K init, 976K bss, 880332K reserved, 0K cma-reserved)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=24, Nodes=2
[ 0.000000] Hierarchical RCU implementation.
[ 0.000000] Build-time adjustment of leaf fanout to 64.
[ 0.000000] RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=24.
[ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=24
[ 0.000000] NR_IRQS:4352 nr_irqs:1432 16
[ 0.000000] Console: colour VGA+ 80x25
[ 0.000000] console [tty0] enabled
[ 0.000000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 133484882848 ns
[ 0.000000] hpet clockevent registered
[ 0.000000] tsc: Fast TSC calibration using PIT
[ 0.000000] tsc: Detected 2666.861 MHz processor
[ 0.000007] Calibrating delay loop (skipped), value calculated using timer frequency.. 5333.72 BogoMIPS (lpj=2666861)
[ 0.000319] pid_max: default: 32768 minimum: 301
[ 0.000487] ACPI: Core revision 20160422
[ 0.004109] ACPI: 2 ACPI AML tables successfully acquired and loaded
[ 0.004508] Security Framework initialized
[ 0.004675] SELinux: Initializing.
[ 0.004840] SELinux: Starting in permissive mode
[ 0.007538] Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
[ 0.020993] Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
[ 0.027132] Mount-cache hash table entries: 131072 (order: 8, 1048576 bytes)
[ 0.027347] Mountpoint-cache hash table entries: 131072 (order: 8, 1048576 bytes)
[ 0.028207] CPU: Physical Processor ID: 0
[ 0.028362] CPU: Processor Core ID: 0
[ 0.028519] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
[ 0.028685] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8)
[ 0.028845] mce: CPU supports 9 MCE banks
[ 0.029003] CPU0: Thermal monitoring enabled (TM1)
[ 0.029166] process: using mwait in idle threads
[ 0.031827] Last level iTLB entries: 4KB 512, 2MB 7, 4MB 7
[ 0.031989] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32, 1GB 0
[ 0.032510] Freeing SMP alternatives memory: 44K (ffffffff8227e000 - ffffffff82289000)
[ 0.056185] smpboot: APIC(0) Converting physical 0 to logical package 0
[ 0.056345] smpboot: APIC(20) Converting physical 1 to logical package 1
[ 0.056505] smpboot: Max logical packages: 4
[ 0.056678] DMAR: Host address width 40
[ 0.056834] DMAR: DRHD base: 0x000000debfe000 flags: 0x0
[ 0.056995] DMAR: dmar0: reg_base_addr debfe000 ver 1:0 cap c90780106f0462 ecap f020f6
[ 0.057262] DMAR: DRHD base: 0x000000fecfe000 flags: 0x1
[ 0.057422] DMAR: dmar1: reg_base_addr fecfe000 ver 1:0 cap c90780106f0462 ecap f020f6
[ 0.057695] DMAR: RMRR base: 0x000000000ec000 end: 0x000000000effff
[ 0.057857] DMAR: RMRR base: 0x000000bf7ec000 end: 0x000000bf7fffff
[ 0.058021] DMAR: ATSR flags: 0x0
[ 0.058180] DMAR: ATSR flags: 0x0
[ 0.058338] DMAR: RHSA base: 0x000000fecfe000 proximity domain: 0x0
[ 0.058502] DMAR: RHSA base: 0x000000debfe000 proximity domain: 0x1
[ 0.058672] Switched APIC routing to physical flat.
[ 0.059539] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[ 0.171636] smpboot: CPU0: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz (family: 0x6, model: 0x2c, stepping: 0x2)
[ 0.172031] Performance Events: PEBS fmt1+, Westmere events, 16-deep LBR, Intel PMU driver.
[ 0.172487] core: CPUID marked event: 'bus cycles' unavailable
[ 0.172658] ... version: 3
[ 0.172818] ... bit width: 48
[ 0.172977] ... generic registers: 4
[ 0.173136] ... value mask: 0000ffffffffffff
[ 0.173298] ... max period: 000000007fffffff
[ 0.173460] ... fixed-purpose events: 3
[ 0.173619] ... event mask: 000000070000000f
[ 0.174291] x86: Booting SMP configuration:
[ 0.174453] .... node #0, CPUs: #1 #2 #3 #4 #5
[ 0.484918] .... node #1, CPUs: #6 #7 #8 #9 #10 #11
[ 0.874858] .... node #0, CPUs: #12 #13 #14 #15 #16 #17
[ 1.247834] .... node #1, CPUs: #18 #19 #20 #21 #22 #23
[ 1.620551] x86: Booted up 2 nodes, 24 CPUs
[ 1.620859] smpboot: Total of 24 processors activated (128126.79 BogoMIPS)
[ 1.648782] devtmpfs: initialized
[ 1.649285] PM: Registering ACPI NVS region [mem 0xbf79e000-0xbf7cffff] (204800 bytes)
[ 1.650870] kworker/u49:0 (132) used greatest stack depth: 13960 bytes left
[ 1.650881] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
[ 1.650962] xor: measuring software checksum speed
[ 1.660469] prefetch64-sse: 13944.000 MB/sec
[ 1.670467] generic_sse: 11916.000 MB/sec
[ 1.670622] xor: using function: prefetch64-sse (13944.000 MB/sec)
[ 1.670806] RTC time: 17:08:04, date: 12/10/16
[ 1.671043] NET: Registered protocol family 16
[ 1.676468] cpuidle: using governor ladder
[ 1.683467] cpuidle: using governor menu
[ 1.683733] ACPI: bus type PCI registered
[ 1.683948] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000)
[ 1.684225] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in E820
[ 1.684394] pmd_set_huge: Cannot satisfy [mem 0xe0000000-0xe0200000] with a huge-page mapping due to MTRR override.
[ 1.684895] PCI: Using configuration type 1 for base access
[ 1.695624] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[ 1.712471] raid6: sse2x1 gen() 6574 MB/s
[ 1.729462] raid6: sse2x1 xor() 5205 MB/s
[ 1.746458] raid6: sse2x2 gen() 7710 MB/s
[ 1.763458] raid6: sse2x2 xor() 6132 MB/s
[ 1.780456] raid6: sse2x4 gen() 9066 MB/s
[ 1.797452] raid6: sse2x4 xor() 6642 MB/s
[ 1.797636] raid6: using algorithm sse2x4 gen() 9066 MB/s
[ 1.797820] raid6: .... xor() 6642 MB/s, rmw enabled
[ 1.798004] raid6: using ssse3x2 recovery algorithm
[ 1.798263] ACPI: Added _OSI(Module Device)
[ 1.798445] ACPI: Added _OSI(Processor Device)
[ 1.798631] ACPI: Added _OSI(3.0 _SCP Extensions)
[ 1.798814] ACPI: Added _OSI(Processor Aggregator Device)
[ 1.799194] ACPI: Executed 1 blocks of module-level executable AML code
[ 1.803605] ACPI: Dynamic OEM Table Load:
[ 1.803898] ACPI: SSDT 0xFFFF88081BF10000 0053DC (v01 DpgPmm P001Ist 00000011 INTL 20051117)
[ 1.805308] ACPI: Dynamic OEM Table Load:
[ 1.805589] ACPI: SSDT 0xFFFF88081BEE3000 000C69 (v01 PmRef P001Cst 00003001 INTL 20051117)
[ 1.806805] ACPI: Interpreter enabled
[ 1.807003] ACPI: (supports S0 S1 S4 S5)
[ 1.807188] ACPI: Using IOAPIC for interrupt routing
[ 1.807403] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[ 1.815625] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7f])
[ 1.815817] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
[ 1.816296] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug AER]
[ 1.816769] acpi PNP0A08:00: _OSC: OS now controls [PME PCIeCapability]
[ 1.817071] acpi PNP0A08:00: host bridge window expanded to [io 0x0d00-0xcfff window]; [io 0x0d00-0xcfff window] ignored
[ 1.817429] PCI host bridge to bus 0000:00
[ 1.817615] pci_bus 0000:00: root bus resource [io 0x0000-0x03af window]
[ 1.817804] pci_bus 0000:00: root bus resource [io 0x03e0-0x03ff window]
[ 1.817993] pci_bus 0000:00: root bus resource [io 0x03b0-0x03bb window]
[ 1.818183] pci_bus 0000:00: root bus resource [io 0x03c0-0x03df window]
[ 1.818373] pci_bus 0000:00: root bus resource [io 0x0400-0x041f window]
[ 1.818564] pci_bus 0000:00: root bus resource [io 0x0d00-0xcfff window]
[ 1.818753] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[ 1.819069] pci_bus 0000:00: root bus resource [mem 0x000d0000-0x000dffff window]
[ 1.819383] pci_bus 0000:00: root bus resource [mem 0xf0000000-0xfbffffff window]
[ 1.819701] pci_bus 0000:00: root bus resource [bus 00-7f]
[ 1.819897] pci 0000:00:00.0: [8086:3406] type 00 class 0x060000
[ 1.819961] pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
[ 1.820050] pci 0000:00:01.0: [8086:3408] type 01 class 0x060400
[ 1.820110] pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
[ 1.820164] pci 0000:00:01.0: System wakeup disabled by ACPI
[ 1.820391] pci 0000:00:03.0: [8086:340a] type 01 class 0x060400
[ 1.820450] pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
[ 1.820508] pci 0000:00:03.0: System wakeup disabled by ACPI
[ 1.820736] pci 0000:00:05.0: [8086:340c] type 01 class 0x060400
[ 1.820790] pci 0000:00:05.0: PME# supported from D0 D3hot D3cold
[ 1.820843] pci 0000:00:05.0: System wakeup disabled by ACPI
[ 1.821069] pci 0000:00:07.0: [8086:340e] type 01 class 0x060400
[ 1.821128] pci 0000:00:07.0: PME# supported from D0 D3hot D3cold
[ 1.821180] pci 0000:00:07.0: System wakeup disabled by ACPI
[ 1.821408] pci 0000:00:09.0: [8086:3410] type 01 class 0x060400
[ 1.821465] pci 0000:00:09.0: PME# supported from D0 D3hot D3cold
[ 1.821522] pci 0000:00:09.0: System wakeup disabled by ACPI
[ 1.821751] pci 0000:00:13.0: [8086:342d] type 00 class 0x080020
[ 1.821761] pci 0000:00:13.0: reg 0x10: [mem 0xfec8a000-0xfec8afff]
[ 1.821811] pci 0000:00:13.0: PME# supported from D0 D3hot D3cold
[ 1.821887] pci 0000:00:14.0: [8086:342e] type 00 class 0x080000
[ 1.822004] pci 0000:00:14.1: [8086:3422] type 00 class 0x080000
[ 1.822121] pci 0000:00:14.2: [8086:3423] type 00 class 0x080000
[ 1.822233] pci 0000:00:14.3: [8086:3438] type 00 class 0x080000
[ 1.822339] pci 0000:00:16.0: [8086:3430] type 00 class 0x088000
[ 1.822352] pci 0000:00:16.0: reg 0x10: [mem 0xfbef8000-0xfbefbfff 64bit]
[ 1.822475] pci 0000:00:16.1: [8086:3431] type 00 class 0x088000
[ 1.822494] pci 0000:00:16.1: reg 0x10: [mem 0xfbef4000-0xfbef7fff 64bit]
[ 1.822618] pci 0000:00:16.2: [8086:3432] type 00 class 0x088000
[ 1.822631] pci 0000:00:16.2: reg 0x10: [mem 0xfbef0000-0xfbef3fff 64bit]
[ 1.822753] pci 0000:00:16.3: [8086:3433] type 00 class 0x088000
[ 1.822766] pci 0000:00:16.3: reg 0x10: [mem 0xfbeec000-0xfbeeffff 64bit]
[ 1.822888] pci 0000:00:16.4: [8086:3429] type 00 class 0x088000
[ 1.822902] pci 0000:00:16.4: reg 0x10: [mem 0xfbee8000-0xfbeebfff 64bit]
[ 1.823025] pci 0000:00:16.5: [8086:342a] type 00 class 0x088000
[ 1.823038] pci 0000:00:16.5: reg 0x10: [mem 0xfbee4000-0xfbee7fff 64bit]
[ 1.823161] pci 0000:00:16.6: [8086:342b] type 00 class 0x088000
[ 1.823175] pci 0000:00:16.6: reg 0x10: [mem 0xfbee0000-0xfbee3fff 64bit]
[ 1.823297] pci 0000:00:16.7: [8086:342c] type 00 class 0x088000
[ 1.823311] pci 0000:00:16.7: reg 0x10: [mem 0xfbedc000-0xfbedffff 64bit]
[ 1.823436] pci 0000:00:1a.0: [8086:3a37] type 00 class 0x0c0300
[ 1.823477] pci 0000:00:1a.0: reg 0x20: [io 0x8f80-0x8f9f]
[ 1.823572] pci 0000:00:1a.0: System wakeup disabled by ACPI
[ 1.823797] pci 0000:00:1a.1: [8086:3a38] type 00 class 0x0c0300
[ 1.823839] pci 0000:00:1a.1: reg 0x20: [io 0x8f40-0x8f5f]
[ 1.823930] pci 0000:00:1a.1: System wakeup disabled by ACPI
[ 1.824154] pci 0000:00:1a.2: [8086:3a39] type 00 class 0x0c0300
[ 1.824196] pci 0000:00:1a.2: reg 0x20: [io 0x8f20-0x8f3f]
[ 1.824285] pci 0000:00:1a.2: System wakeup disabled by ACPI
[ 1.824521] pci 0000:00:1a.7: [8086:3a3c] type 00 class 0x0c0320
[ 1.824539] pci 0000:00:1a.7: reg 0x10: [mem 0xfbeda000-0xfbeda3ff]
[ 1.824625] pci 0000:00:1a.7: PME# supported from D0 D3hot D3cold
[ 1.824673] pci 0000:00:1a.7: System wakeup disabled by ACPI
[ 1.824900] pci 0000:00:1d.0: [8086:3a34] type 00 class 0x0c0300
[ 1.824942] pci 0000:00:1d.0: reg 0x20: [io 0x8f00-0x8f1f]
[ 1.825032] pci 0000:00:1d.0: System wakeup disabled by ACPI
[ 1.825257] pci 0000:00:1d.1: [8086:3a35] type 00 class 0x0c0300
[ 1.825299] pci 0000:00:1d.1: reg 0x20: [io 0x8ec0-0x8edf]
[ 1.825390] pci 0000:00:1d.1: System wakeup disabled by ACPI
[ 1.825619] pci 0000:00:1d.2: [8086:3a36] type 00 class 0x0c0300
[ 1.825661] pci 0000:00:1d.2: reg 0x20: [io 0x8ea0-0x8ebf]
[ 1.825753] pci 0000:00:1d.2: System wakeup disabled by ACPI
[ 1.825985] pci 0000:00:1d.7: [8086:3a3a] type 00 class 0x0c0320
[ 1.826003] pci 0000:00:1d.7: reg 0x10: [mem 0xfbed8000-0xfbed83ff]
[ 1.826089] pci 0000:00:1d.7: PME# supported from D0 D3hot D3cold
[ 1.826134] pci 0000:00:1d.7: System wakeup disabled by ACPI
[ 1.826357] pci 0000:00:1e.0: [8086:244e] type 01 class 0x060401
[ 1.826447] pci 0000:00:1e.0: System wakeup disabled by ACPI
[ 1.826676] pci 0000:00:1f.0: [8086:3a16] type 00 class 0x060100
[ 1.826756] pci 0000:00:1f.0: can't claim BAR 13 [io 0x0800-0x087f]: no compatible bridge window
[ 1.827084] pci 0000:00:1f.0: can't claim BAR 14 [io 0x0500-0x053f]: no compatible bridge window
[ 1.827409] pci 0000:00:1f.0: ICH7 LPC Generic IO decode 1 PIO at 0a00 (mask 00ff)
[ 1.827729] pci 0000:00:1f.0: ICH7 LPC Generic IO decode 3 PIO at 164c (mask 0007)
[ 1.828044] pci 0000:00:1f.0: ICH7 LPC Generic IO decode 4 PIO at 0ca0 (mask 000f)
[ 1.828456] pci 0000:00:1f.2: [8086:3a22] type 00 class 0x010601
[ 1.828471] pci 0000:00:1f.2: reg 0x10: [io 0x8ff0-0x8ff7]
[ 1.828478] pci 0000:00:1f.2: reg 0x14: [io 0x8fac-0x8faf]
[ 1.828490] pci 0000:00:1f.2: reg 0x18: [io 0x8fe0-0x8fe7]
[ 1.828497] pci 0000:00:1f.2: reg 0x1c: [io 0x8fa8-0x8fab]
[ 1.828504] pci 0000:00:1f.2: reg 0x20: [io 0x8e80-0x8e9f]
[ 1.828511] pci 0000:00:1f.2: reg 0x24: [mem 0xfbed6000-0xfbed67ff]
[ 1.828555] pci 0000:00:1f.2: PME# supported from D3hot
[ 1.828633] pci 0000:00:1f.3: [8086:3a30] type 00 class 0x0c0500
[ 1.828649] pci 0000:00:1f.3: reg 0x10: [mem 0xfbed4000-0xfbed40ff 64bit]
[ 1.828669] pci 0000:00:1f.3: reg 0x20: [io 0x0400-0x041f]
[ 1.828803] pci 0000:01:00.0: [8086:10c9] type 00 class 0x020000
[ 1.828818] pci 0000:01:00.0: reg 0x10: [mem 0xfa7e0000-0xfa7fffff]
[ 1.828826] pci 0000:01:00.0: reg 0x14: [mem 0xfa7c0000-0xfa7dffff]
[ 1.828835] pci 0000:01:00.0: reg 0x18: [io 0xac00-0xac1f]
[ 1.828843] pci 0000:01:00.0: reg 0x1c: [mem 0xfa79c000-0xfa79ffff]
[ 1.828866] pci 0000:01:00.0: reg 0x30: [mem 0xfa7a0000-0xfa7bffff pref]
[ 1.828923] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[ 1.828956] pci 0000:01:00.0: reg 0x184: [mem 0xfa760000-0xfa763fff 64bit]
[ 1.828958] pci 0000:01:00.0: VF(n) BAR0 space: [mem 0xfa760000-0xfa77ffff 64bit] (contains BAR0 for 8 VFs)
[ 1.829295] pci 0000:01:00.0: reg 0x190: [mem 0xfa740000-0xfa743fff 64bit]
[ 1.829296] pci 0000:01:00.0: VF(n) BAR3 space: [mem 0xfa740000-0xfa75ffff 64bit] (contains BAR3 for 8 VFs)
[ 1.829720] pci 0000:01:00.1: [8086:10c9] type 00 class 0x020000
[ 1.829734] pci 0000:01:00.1: reg 0x10: [mem 0xfa720000-0xfa73ffff]
[ 1.829743] pci 0000:01:00.1: reg 0x14: [mem 0xfa700000-0xfa71ffff]
[ 1.829751] pci 0000:01:00.1: reg 0x18: [io 0xa800-0xa81f]
[ 1.829760] pci 0000:01:00.1: reg 0x1c: [mem 0xfa6dc000-0xfa6dffff]
[ 1.829783] pci 0000:01:00.1: reg 0x30: [mem 0xfa6e0000-0xfa6fffff pref]
[ 1.829840] pci 0000:01:00.1: PME# supported from D0 D3hot D3cold
[ 1.829868] pci 0000:01:00.1: reg 0x184: [mem 0xfa6a0000-0xfa6a3fff 64bit]
[ 1.829869] pci 0000:01:00.1: VF(n) BAR0 space: [mem 0xfa6a0000-0xfa6bffff 64bit] (contains BAR0 for 8 VFs)
[ 1.830208] pci 0000:01:00.1: reg 0x190: [mem 0xfa680000-0xfa683fff 64bit]
[ 1.830210] pci 0000:01:00.1: VF(n) BAR3 space: [mem 0xfa680000-0xfa69ffff 64bit] (contains BAR3 for 8 VFs)
[ 1.833498] pci 0000:00:01.0: PCI bridge to [bus 01-02]
[ 1.833686] pci 0000:00:01.0: bridge window [io 0xa000-0xafff]
[ 1.833688] pci 0000:00:01.0: bridge window [mem 0xfa600000-0xfa7fffff]
[ 1.833739] pci 0000:03:00.0: [1000:0072] type 00 class 0x010700
[ 1.833750] pci 0000:03:00.0: reg 0x10: [io 0xb800-0xb8ff]
[ 1.833760] pci 0000:03:00.0: reg 0x14: [mem 0xfad3c000-0xfad3ffff 64bit]
[ 1.833770] pci 0000:03:00.0: reg 0x1c: [mem 0xfad40000-0xfad7ffff 64bit]
[ 1.833782] pci 0000:03:00.0: reg 0x30: [mem 0xfad80000-0xfadfffff pref]
[ 1.833829] pci 0000:03:00.0: supports D1 D2
[ 1.833851] pci 0000:03:00.0: reg 0x174: [mem 0xfacc0000-0xfacc3fff 64bit]
[ 1.833852] pci 0000:03:00.0: VF(n) BAR0 space: [mem 0xfacc0000-0xfacfffff 64bit] (contains BAR0 for 16 VFs)
[ 1.834181] pci 0000:03:00.0: reg 0x17c: [mem 0xfa800000-0xfa83ffff 64bit]
[ 1.834183] pci 0000:03:00.0: VF(n) BAR2 space: [mem 0xfa800000-0xfabfffff 64bit] (contains BAR2 for 16 VFs)
[ 1.837494] pci 0000:00:03.0: PCI bridge to [bus 03]
[ 1.837680] pci 0000:00:03.0: bridge window [io 0xb000-0xbfff]
[ 1.837683] pci 0000:00:03.0: bridge window [mem 0xfa800000-0xfadfffff]
[ 1.837724] pci 0000:00:05.0: PCI bridge to [bus 04]
[ 1.838145] pci 0000:05:00.0: [15b3:673c] type 00 class 0x0c0600
[ 1.838428] pci 0000:05:00.0: reg 0x10: [mem 0xfae00000-0xfaefffff 64bit]
[ 1.838629] pci 0000:05:00.0: reg 0x18: [mem 0xf8800000-0xf8ffffff 64bit pref]
[ 1.840491] pci 0000:05:00.0: reg 0x134: [mem 0xf0000000-0xf07fffff 64bit pref]
[ 1.840493] pci 0000:05:00.0: VF(n) BAR2 space: [mem 0xf0000000-0xf7ffffff 64bit pref] (contains BAR2 for 16 VFs)
[ 1.844559] pci 0000:00:07.0: PCI bridge to [bus 05]
[ 1.844747] pci 0000:00:07.0: bridge window [mem 0xfae00000-0xfaefffff]
[ 1.844751] pci 0000:00:07.0: bridge window [mem 0xf0000000-0xf8ffffff 64bit pref]
[ 1.844798] pci 0000:06:00.0: [1000:0072] type 00 class 0x010700
[ 1.844809] pci 0000:06:00.0: reg 0x10: [io 0xc800-0xc8ff]
[ 1.844819] pci 0000:06:00.0: reg 0x14: [mem 0xfbd3c000-0xfbd3ffff 64bit]
[ 1.844829] pci 0000:06:00.0: reg 0x1c: [mem 0xfbd40000-0xfbd7ffff 64bit]
[ 1.844841] pci 0000:06:00.0: reg 0x30: [mem 0xfbd80000-0xfbdfffff pref]
[ 1.844888] pci 0000:06:00.0: supports D1 D2
[ 1.844910] pci 0000:06:00.0: reg 0x174: [mem 0xfbcc0000-0xfbcc3fff 64bit]
[ 1.844911] pci 0000:06:00.0: VF(n) BAR0 space: [mem 0xfbcc0000-0xfbcfffff 64bit] (contains BAR0 for 16 VFs)
[ 1.845239] pci 0000:06:00.0: reg 0x17c: [mem 0xfb800000-0xfb83ffff 64bit]
[ 1.845241] pci 0000:06:00.0: VF(n) BAR2 space: [mem 0xfb800000-0xfbbfffff 64bit] (contains BAR2 for 16 VFs)
[ 1.848493] pci 0000:00:09.0: PCI bridge to [bus 06]
[ 1.848679] pci 0000:00:09.0: bridge window [io 0xc000-0xcfff]
[ 1.848682] pci 0000:00:09.0: bridge window [mem 0xfb800000-0xfbdfffff]
[ 1.848731] pci 0000:07:04.0: [102b:0532] type 00 class 0x030000
[ 1.848746] pci 0000:07:04.0: reg 0x10: [mem 0xf9000000-0xf9ffffff pref]
[ 1.848755] pci 0000:07:04.0: reg 0x14: [mem 0xfaffc000-0xfaffffff]
[ 1.848764] pci 0000:07:04.0: reg 0x18: [mem 0xfb000000-0xfb7fffff]
[ 1.848888] pci 0000:00:1e.0: PCI bridge to [bus 07] (subtractive decode)
[ 1.849081] pci 0000:00:1e.0: bridge window [mem 0xfaf00000-0xfb7fffff]
[ 1.849086] pci 0000:00:1e.0: bridge window [mem 0xf9000000-0xf9ffffff 64bit pref]
[ 1.849087] pci 0000:00:1e.0: bridge window [io 0x0000-0x03af window] (subtractive decode)
[ 1.849088] pci 0000:00:1e.0: bridge window [io 0x03e0-0x03ff window] (subtractive decode)
[ 1.849090] pci 0000:00:1e.0: bridge window [io 0x03b0-0x03bb window] (subtractive decode)
[ 1.849091] pci 0000:00:1e.0: bridge window [io 0x03c0-0x03df window] (subtractive decode)
[ 1.849092] pci 0000:00:1e.0: bridge window [io 0x0400-0x041f window] (subtractive decode)
[ 1.849093] pci 0000:00:1e.0: bridge window [io 0x0d00-0xcfff window] (subtractive decode)
[ 1.849094] pci 0000:00:1e.0: bridge window [mem 0x000a0000-0x000bffff window] (subtractive decode)
[ 1.849095] pci 0000:00:1e.0: bridge window [mem 0x000d0000-0x000dffff window] (subtractive decode)
[ 1.849096] pci 0000:00:1e.0: bridge window [mem 0xf0000000-0xfbffffff window] (subtractive decode)
[ 1.849128] pci_bus 0000:00: on NUMA node 0
[ 1.849649] ACPI: PCI Root Bridge [BR50] (domain 0000 [bus 80-fb])
[ 1.849839] acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
[ 1.850317] acpi PNP0A08:01: _OSC: platform does not support [PCIeHotplug AER]
[ 1.850789] acpi PNP0A08:01: _OSC: OS now controls [PME PCIeCapability]
[ 1.851106] PCI host bridge to bus 0000:80
[ 1.851290] pci_bus 0000:80: root bus resource [io 0xd000-0xffff window]
[ 1.851480] pci_bus 0000:80: root bus resource [mem 0xce000000-0xdfffffff window]
[ 1.851799] pci_bus 0000:80: root bus resource [bus 80-fb]
[ 1.851994] pci 0000:80:00.0: [8086:3420] type 01 class 0x060400
[ 1.852041] pci 0000:80:00.0: PME# supported from D0 D3hot D3cold
[ 1.852108] pci 0000:80:01.0: [8086:3408] type 01 class 0x060400
[ 1.852154] pci 0000:80:01.0: PME# supported from D0 D3hot D3cold
[ 1.852186] pci 0000:80:01.0: System wakeup disabled by ACPI
[ 1.852410] pci 0000:80:03.0: [8086:340a] type 01 class 0x060400
[ 1.852460] pci 0000:80:03.0: PME# supported from D0 D3hot D3cold
[ 1.852496] pci 0000:80:03.0: System wakeup disabled by ACPI
[ 1.852721] pci 0000:80:05.0: [8086:340c] type 01 class 0x060400
[ 1.852770] pci 0000:80:05.0: PME# supported from D0 D3hot D3cold
[ 1.852799] pci 0000:80:05.0: System wakeup disabled by ACPI
[ 1.853023] pci 0000:80:07.0: [8086:340e] type 01 class 0x060400
[ 1.853073] pci 0000:80:07.0: PME# supported from D0 D3hot D3cold
[ 1.853104] pci 0000:80:07.0: System wakeup disabled by ACPI
[ 1.853328] pci 0000:80:09.0: [8086:3410] type 01 class 0x060400
[ 1.853373] pci 0000:80:09.0: PME# supported from D0 D3hot D3cold
[ 1.853403] pci 0000:80:09.0: System wakeup disabled by ACPI
[ 1.853632] pci 0000:80:13.0: [8086:342d] type 00 class 0x080020
[ 1.853641] pci 0000:80:13.0: reg 0x10: [mem 0xfec9a000-0xfec9afff]
[ 1.853684] pci 0000:80:13.0: PME# supported from D0 D3hot D3cold
[ 1.853739] pci 0000:80:14.0: [8086:342e] type 00 class 0x080000
[ 1.853828] pci 0000:80:14.1: [8086:3422] type 00 class 0x080000
[ 1.853916] pci 0000:80:14.2: [8086:3423] type 00 class 0x080000
[ 1.854002] pci 0000:80:14.3: [8086:3438] type 00 class 0x080000
[ 1.854079] pci 0000:80:16.0: [8086:3430] type 00 class 0x088000
[ 1.854090] pci 0000:80:16.0: reg 0x10: [mem 0xdebf8000-0xdebfbfff 64bit]
[ 1.854183] pci 0000:80:16.1: [8086:3431] type 00 class 0x088000
[ 1.854195] pci 0000:80:16.1: reg 0x10: [mem 0xdebf4000-0xdebf7fff 64bit]
[ 1.854287] pci 0000:80:16.2: [8086:3432] type 00 class 0x088000
[ 1.854299] pci 0000:80:16.2: reg 0x10: [mem 0xdebf0000-0xdebf3fff 64bit]
[ 1.854392] pci 0000:80:16.3: [8086:3433] type 00 class 0x088000
[ 1.854404] pci 0000:80:16.3: reg 0x10: [mem 0xdebec000-0xdebeffff 64bit]
[ 1.854504] pci 0000:80:16.4: [8086:3429] type 00 class 0x088000
[ 1.854515] pci 0000:80:16.4: reg 0x10: [mem 0xdebe8000-0xdebebfff 64bit]
[ 1.854609] pci 0000:80:16.5: [8086:342a] type 00 class 0x088000
[ 1.854621] pci 0000:80:16.5: reg 0x10: [mem 0xdebe4000-0xdebe7fff 64bit]
[ 1.854714] pci 0000:80:16.6: [8086:342b] type 00 class 0x088000
[ 1.854725] pci 0000:80:16.6: reg 0x10: [mem 0xdebe0000-0xdebe3fff 64bit]
[ 1.854819] pci 0000:80:16.7: [8086:342c] type 00 class 0x088000
[ 1.854831] pci 0000:80:16.7: reg 0x10: [mem 0xdebdc000-0xdebdffff 64bit]
[ 1.854954] pci 0000:80:00.0: PCI bridge to [bus 81]
[ 1.855177] pci 0000:80:01.0: PCI bridge to [bus 82]
[ 1.855412] pci 0000:83:00.0: [168c:002b] type 00 class 0x028000
[ 1.855431] pci 0000:83:00.0: reg 0x10: [mem 0xdecf0000-0xdecfffff 64bit]
[ 1.855523] pci 0000:83:00.0: supports D1
[ 1.855524] pci 0000:83:00.0: PME# supported from D0 D1 D3hot
[ 1.858493] pci 0000:80:03.0: PCI bridge to [bus 83]
[ 1.858681] pci 0000:80:03.0: bridge window [mem 0xdec00000-0xdecfffff]
[ 1.858729] pci 0000:84:00.0: [1b21:1182] type 01 class 0x060400
[ 1.858797] pci 0000:84:00.0: PME# supported from D0 D3hot D3cold
[ 1.861491] pci 0000:80:05.0: PCI bridge to [bus 84-87]
[ 1.861677] pci 0000:80:05.0: bridge window [io 0xd000-0xefff]
[ 1.861680] pci 0000:80:05.0: bridge window [mem 0xded00000-0xdeefffff]
[ 1.861684] pci 0000:80:05.0: bridge window [mem 0xdde00000-0xddffffff 64bit pref]
[ 1.861734] pci 0000:85:03.0: [1b21:1182] type 01 class 0x060400
[ 1.861804] pci 0000:85:03.0: PME# supported from D0 D3hot D3cold
[ 1.861865] pci 0000:85:07.0: [1b21:1182] type 01 class 0x060400
[ 1.861935] pci 0000:85:07.0: PME# supported from D0 D3hot D3cold
[ 1.862002] pci 0000:84:00.0: PCI bridge to [bus 85-87]
[ 1.862191] pci 0000:84:00.0: bridge window [io 0xd000-0xefff]
[ 1.862194] pci 0000:84:00.0: bridge window [mem 0xded00000-0xdeefffff]
[ 1.862199] pci 0000:84:00.0: bridge window [mem 0xdde00000-0xddffffff 64bit pref]
[ 1.862257] pci 0000:86:00.0: [10ec:8168] type 00 class 0x020000
[ 1.862278] pci 0000:86:00.0: reg 0x10: [io 0xdc00-0xdcff]
[ 1.862310] pci 0000:86:00.0: reg 0x18: [mem 0xdedfe000-0xdedfefff 64bit]
[ 1.862329] pci 0000:86:00.0: reg 0x20: [mem 0xddefc000-0xddefffff 64bit pref]
[ 1.862433] pci 0000:86:00.0: supports D1 D2
[ 1.862434] pci 0000:86:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 1.865497] pci 0000:85:03.0: PCI bridge to [bus 86]
[ 1.868691] pci 0000:85:03.0: bridge window [io 0xd000-0xdfff]
[ 1.868693] pci 0000:85:03.0: bridge window [mem 0xded00000-0xdedfffff]
[ 1.868698] pci 0000:85:03.0: bridge window [mem 0xdde00000-0xddefffff 64bit pref]
[ 1.868757] pci 0000:87:00.0: [10ec:8168] type 00 class 0x020000
[ 1.868778] pci 0000:87:00.0: reg 0x10: [io 0xec00-0xecff]
[ 1.868809] pci 0000:87:00.0: reg 0x18: [mem 0xdeefe000-0xdeefefff 64bit]
[ 1.868828] pci 0000:87:00.0: reg 0x20: [mem 0xddffc000-0xddffffff 64bit pref]
[ 1.868930] pci 0000:87:00.0: supports D1 D2
[ 1.868931] pci 0000:87:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 1.871496] pci 0000:85:07.0: PCI bridge to [bus 87]
[ 1.871683] pci 0000:85:07.0: bridge window [io 0xe000-0xefff]
[ 1.871686] pci 0000:85:07.0: bridge window [mem 0xdee00000-0xdeefffff]
[ 1.871691] pci 0000:85:07.0: bridge window [mem 0xddf00000-0xddffffff 64bit pref]
[ 1.871749] pci 0000:88:00.0: [10de:1180] type 00 class 0x030000
[ 1.871759] pci 0000:88:00.0: reg 0x10: [mem 0xdf000000-0xdfffffff]
[ 1.871767] pci 0000:88:00.0: reg 0x14: [mem 0xd0000000-0xd7ffffff 64bit pref]
[ 1.871775] pci 0000:88:00.0: reg 0x1c: [mem 0xce000000-0xcfffffff 64bit pref]
[ 1.871781] pci 0000:88:00.0: reg 0x24: [io 0xfc00-0xfc7f]
[ 1.871787] pci 0000:88:00.0: reg 0x30: [mem 0xdef80000-0xdeffffff pref]
[ 1.871872] pci 0000:88:00.1: [10de:0e0a] type 00 class 0x040300
[ 1.871882] pci 0000:88:00.1: reg 0x10: [mem 0xdef7c000-0xdef7ffff]
[ 1.874492] pci 0000:80:07.0: PCI bridge to [bus 88]
[ 1.874678] pci 0000:80:07.0: bridge window [io 0xf000-0xffff]
[ 1.874681] pci 0000:80:07.0: bridge window [mem 0xdef00000-0xdfffffff]
[ 1.874684] pci 0000:80:07.0: bridge window [mem 0xce000000-0xd7ffffff 64bit pref]
[ 1.874720] pci 0000:80:09.0: PCI bridge to [bus 89]
[ 1.874937] pci_bus 0000:80: on NUMA node 1
[ 1.875025] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 6 7 *10 11 12 14 15)
[ 1.875761] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 6 7 10 11 12 14 *15)
[ 1.876495] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 6 7 10 11 12 *14 15)
[ 1.877230] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 6 7 10 *11 12 14 15)
[ 1.877965] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 6 7 10 11 12 14 15) *0, disabled.
[ 1.878917] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 6 *7 10 11 12 14 15)
[ 1.879650] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 6 7 10 11 12 14 15) *0, disabled.
[ 1.880601] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 *6 7 10 11 12 14 15)
[ 1.881390] ACPI: Enabled 2 GPEs in block 00 to 3F
[ 1.881797] vgaarb: setting as boot device: PCI:0000:07:04.0
[ 1.881985] vgaarb: device added: PCI:0000:07:04.0,decodes=io+mem,owns=io+mem,locks=none
[ 1.882306] vgaarb: device added: PCI:0000:88:00.0,decodes=io+mem,owns=none,locks=none
[ 1.882626] vgaarb: loaded
[ 1.882803] vgaarb: bridge control possible 0000:88:00.0
[ 1.882987] vgaarb: bridge control possible 0000:07:04.0
[ 1.883243] SCSI subsystem initialized
[ 1.883493] libata version 3.00 loaded.
[ 1.883532] ACPI: bus type USB registered
[ 1.883738] usbcore: registered new interface driver usbfs
[ 1.883936] usbcore: registered new interface driver hub
[ 1.884176] usbcore: registered new device driver usb
[ 1.884389] pps_core: LinuxPPS API ver. 1 registered
[ 1.884575] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[ 1.884898] PTP clock support registered
[ 1.885200] Advanced Linux Sound Architecture Driver Initialized.
[ 1.885408] PCI: Using ACPI for IRQ routing
[ 1.889990] PCI: Discovered peer bus fe
[ 1.890170] PCI: root bus fe: using default resources
[ 1.890171] PCI: Probing PCI hardware (bus fe)
[ 1.890195] ACPI: \: failed to evaluate _DSM (0x1001)
[ 1.890380] PCI host bridge to bus 0000:fe
[ 1.890565] pci_bus 0000:fe: root bus resource [io 0x0000-0xffff]
[ 1.890753] pci_bus 0000:fe: root bus resource [mem 0x00000000-0xffffffffff]
[ 1.890943] pci_bus 0000:fe: No busn resource found for root bus, will use [bus fe-ff]
[ 1.891263] pci 0000:fe:00.0: [8086:2c70] type 00 class 0x060000
[ 1.891306] pci 0000:fe:00.1: [8086:2d81] type 00 class 0x060000
[ 1.891352] pci 0000:fe:02.0: [8086:2d90] type 00 class 0x060000
[ 1.891394] pci 0000:fe:02.1: [8086:2d91] type 00 class 0x060000
[ 1.891436] pci 0000:fe:02.2: [8086:2d92] type 00 class 0x060000
[ 1.891481] pci 0000:fe:02.3: [8086:2d93] type 00 class 0x060000
[ 1.891523] pci 0000:fe:02.4: [8086:2d94] type 00 class 0x060000
[ 1.891564] pci 0000:fe:02.5: [8086:2d95] type 00 class 0x060000
[ 1.891606] pci 0000:fe:03.0: [8086:2d98] type 00 class 0x060000
[ 1.891649] pci 0000:fe:03.1: [8086:2d99] type 00 class 0x060000
[ 1.891695] pci 0000:fe:03.2: [8086:2d9a] type 00 class 0x060000
[ 1.891737] pci 0000:fe:03.4: [8086:2d9c] type 00 class 0x060000
[ 1.891780] pci 0000:fe:04.0: [8086:2da0] type 00 class 0x060000
[ 1.891821] pci 0000:fe:04.1: [8086:2da1] type 00 class 0x060000
[ 1.891862] pci 0000:fe:04.2: [8086:2da2] type 00 class 0x060000
[ 1.891903] pci 0000:fe:04.3: [8086:2da3] type 00 class 0x060000
[ 1.891947] pci 0000:fe:05.0: [8086:2da8] type 00 class 0x060000
[ 1.891988] pci 0000:fe:05.1: [8086:2da9] type 00 class 0x060000
[ 1.892029] pci 0000:fe:05.2: [8086:2daa] type 00 class 0x060000
[ 1.892071] pci 0000:fe:05.3: [8086:2dab] type 00 class 0x060000
[ 1.892115] pci 0000:fe:06.0: [8086:2db0] type 00 class 0x060000
[ 1.892155] pci 0000:fe:06.1: [8086:2db1] type 00 class 0x060000
[ 1.892196] pci 0000:fe:06.2: [8086:2db2] type 00 class 0x060000
[ 1.892238] pci 0000:fe:06.3: [8086:2db3] type 00 class 0x060000
[ 1.892292] pci_bus 0000:fe: busn_res: [bus fe-ff] end is updated to fe
[ 1.892299] PCI: Discovered peer bus ff
[ 1.892483] PCI: root bus ff: using default resources
[ 1.892484] PCI: Probing PCI hardware (bus ff)
[ 1.892505] ACPI: \: failed to evaluate _DSM (0x1001)
[ 1.892691] PCI host bridge to bus 0000:ff
[ 1.892873] pci_bus 0000:ff: root bus resource [io 0x0000-0xffff]
[ 1.893060] pci_bus 0000:ff: root bus resource [mem 0x00000000-0xffffffffff]
[ 1.893251] pci_bus 0000:ff: No busn resource found for root bus, will use [bus ff-ff]
[ 1.893574] pci 0000:ff:00.0: [8086:2c70] type 00 class 0x060000
[ 1.893618] pci 0000:ff:00.1: [8086:2d81] type 00 class 0x060000
[ 1.893665] pci 0000:ff:02.0: [8086:2d90] type 00 class 0x060000
[ 1.893707] pci 0000:ff:02.1: [8086:2d91] type 00 class 0x060000
[ 1.893749] pci 0000:ff:02.2: [8086:2d92] type 00 class 0x060000
[ 1.893791] pci 0000:ff:02.3: [8086:2d93] type 00 class 0x060000
[ 1.893833] pci 0000:ff:02.4: [8086:2d94] type 00 class 0x060000
[ 1.893874] pci 0000:ff:02.5: [8086:2d95] type 00 class 0x060000
[ 1.893917] pci 0000:ff:03.0: [8086:2d98] type 00 class 0x060000
[ 1.893960] pci 0000:ff:03.1: [8086:2d99] type 00 class 0x060000
[ 1.894003] pci 0000:ff:03.2: [8086:2d9a] type 00 class 0x060000
[ 1.894047] pci 0000:ff:03.4: [8086:2d9c] type 00 class 0x060000
[ 1.894091] pci 0000:ff:04.0: [8086:2da0] type 00 class 0x060000
[ 1.894133] pci 0000:ff:04.1: [8086:2da1] type 00 class 0x060000
[ 1.894176] pci 0000:ff:04.2: [8086:2da2] type 00 class 0x060000
[ 1.894219] pci 0000:ff:04.3: [8086:2da3] type 00 class 0x060000
[ 1.894265] pci 0000:ff:05.0: [8086:2da8] type 00 class 0x060000
[ 1.894307] pci 0000:ff:05.1: [8086:2da9] type 00 class 0x060000
[ 1.894350] pci 0000:ff:05.2: [8086:2daa] type 00 class 0x060000
[ 1.894394] pci 0000:ff:05.3: [8086:2dab] type 00 class 0x060000
[ 1.894439] pci 0000:ff:06.0: [8086:2db0] type 00 class 0x060000
[ 1.894485] pci 0000:ff:06.1: [8086:2db1] type 00 class 0x060000
[ 1.894528] pci 0000:ff:06.2: [8086:2db2] type 00 class 0x060000
[ 1.894572] pci 0000:ff:06.3: [8086:2db3] type 00 class 0x060000
[ 1.894627] pci_bus 0000:ff: busn_res: [bus ff] end is updated to ff
[ 1.894633] PCI: pci_cache_line_size set to 64 bytes
[ 1.894668] pci 0000:00:13.0: can't claim BAR 0 [mem 0xfec8a000-0xfec8afff]: no compatible bridge window
[ 1.895032] pci 0000:80:13.0: can't claim BAR 0 [mem 0xfec9a000-0xfec9afff]: no compatible bridge window
[ 1.895467] e820: reserve RAM buffer [mem 0x0009c000-0x0009ffff]
[ 1.895468] e820: reserve RAM buffer [mem 0xbf780000-0xbfffffff]
[ 1.895681] NetLabel: Initializing
[ 1.895862] NetLabel: domain hash size = 128
[ 1.896044] NetLabel: protocols = UNLABELED CIPSOv4
[ 1.896245] NetLabel: unlabeled traffic allowed by default
[ 1.896581] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0
[ 1.896998] hpet0: 4 comparators, 64-bit 14.318180 MHz counter
[ 1.899253] clocksource: Switched to clocksource hpet
[ 1.907470] VFS: Disk quotas dquot_6.6.0
[ 1.907686] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ 1.907964] pnp: PnP ACPI init
[ 1.908232] system 00:00: [mem 0xfed1c000-0xfed1ffff] has been reserved
[ 1.908435] system 00:00: Plug and Play ACPI device, IDs PNP0c01 (active)
[ 1.908525] pnp 00:01: Plug and Play ACPI device, IDs PNP0b00 (active)
[ 1.908658] system 00:02: [io 0x0a10-0x0a1f] has been reserved
[ 1.908848] system 00:02: Plug and Play ACPI device, IDs PNP0c02 (active)
[ 1.908986] pnp 00:03: [dma 0 disabled]
[ 1.909034] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active)
[ 1.909163] pnp 00:04: [dma 0 disabled]
[ 1.909209] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active)
[ 1.909421] system 00:05: [io 0x0cf8-0x0cff] could not be reserved
[ 1.909611] system 00:05: [io 0x04d0-0x04d1] has been reserved
[ 1.909799] system 00:05: [io 0x0800-0x087f] has been reserved
[ 1.909986] system 00:05: [io 0x0500-0x057f] has been reserved
[ 1.910173] system 00:05: [mem 0x00000400-0x000004ff] could not be reserved
[ 1.910373] system 00:05: [mem 0xfed1c000-0xfed1ffff] has been reserved
[ 1.910562] system 00:05: [mem 0xfed20000-0xfed3ffff] has been reserved
[ 1.910751] system 00:05: [mem 0xfed40000-0xfed8ffff] has been reserved
[ 1.910942] system 00:05: Plug and Play ACPI device, IDs PNP0c02 (active)
[ 1.911080] system 00:06: [mem 0xfec00000-0xfec00fff] could not be reserved
[ 1.911281] system 00:06: [mem 0xfee00000-0xfee00fff] has been reserved
[ 1.911472] system 00:06: Plug and Play ACPI device, IDs PNP0c02 (active)
[ 1.911561] system 00:07: [mem 0xe0000000-0xefffffff] has been reserved
[ 1.911752] system 00:07: Plug and Play ACPI device, IDs PNP0c02 (active)
[ 1.911883] system 00:08: [mem 0x000c0000-0x000cffff] could not be reserved
[ 1.912075] system 00:08: [mem 0x000e0000-0x000fffff] could not be reserved
[ 1.912278] system 00:08: [mem 0xfed90000-0xffffffff] could not be reserved
[ 1.912471] system 00:08: Plug and Play ACPI device, IDs PNP0c01 (active)
[ 1.912545] system 00:09: [mem 0xdebfe000-0xdebfffff] could not be reserved
[ 1.912736] system 00:09: Plug and Play ACPI device, IDs PNP0c01 (active)
[ 1.912869] pnp: PnP ACPI: found 10 devices
[ 1.919183] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[ 1.919561] pci 0000:00:1f.0: BAR 13: [io size 0x0080] has bogus alignment
[ 1.919752] pci 0000:00:1f.0: BAR 14: [io size 0x0040] has bogus alignment
[ 1.919943] pci 0000:00:01.0: PCI bridge to [bus 01-02]
[ 1.920129] pci 0000:00:01.0: bridge window [io 0xa000-0xafff]
[ 1.920329] pci 0000:00:01.0: bridge window [mem 0xfa600000-0xfa7fffff]
[ 1.920524] pci 0000:00:03.0: PCI bridge to [bus 03]
[ 1.920708] pci 0000:00:03.0: bridge window [io 0xb000-0xbfff]
[ 1.920898] pci 0000:00:03.0: bridge window [mem 0xfa800000-0xfadfffff]
[ 1.921091] pci 0000:00:05.0: PCI bridge to [bus 04]
[ 1.921293] pci 0000:00:07.0: PCI bridge to [bus 05]
[ 1.921479] pci 0000:00:07.0: bridge window [mem 0xfae00000-0xfaefffff]
[ 1.921671] pci 0000:00:07.0: bridge window [mem 0xf0000000-0xf8ffffff 64bit pref]
[ 1.921991] pci 0000:00:09.0: PCI bridge to [bus 06]
[ 1.922175] pci 0000:00:09.0: bridge window [io 0xc000-0xcfff]
[ 1.922375] pci 0000:00:09.0: bridge window [mem 0xfb800000-0xfbdfffff]
[ 1.922570] pci 0000:00:1e.0: PCI bridge to [bus 07]
[ 1.922756] pci 0000:00:1e.0: bridge window [mem 0xfaf00000-0xfb7fffff]
[ 1.922948] pci 0000:00:1e.0: bridge window [mem 0xf9000000-0xf9ffffff 64bit pref]
[ 1.923280] pci_bus 0000:00: resource 4 [io 0x0000-0x03af window]
[ 1.923282] pci_bus 0000:00: resource 5 [io 0x03e0-0x03ff window]
[ 1.923283] pci_bus 0000:00: resource 6 [io 0x03b0-0x03bb window]
[ 1.923284] pci_bus 0000:00: resource 7 [io 0x03c0-0x03df window]
[ 1.923285] pci_bus 0000:00: resource 8 [io 0x0400-0x041f window]
[ 1.923286] pci_bus 0000:00: resource 9 [io 0x0d00-0xcfff window]
[ 1.923288] pci_bus 0000:00: resource 10 [mem 0x000a0000-0x000bffff window]
[ 1.923289] pci_bus 0000:00: resource 11 [mem 0x000d0000-0x000dffff window]
[ 1.923290] pci_bus 0000:00: resource 12 [mem 0xf0000000-0xfbffffff window]
[ 1.923291] pci_bus 0000:01: resource 0 [io 0xa000-0xafff]
[ 1.923292] pci_bus 0000:01: resource 1 [mem 0xfa600000-0xfa7fffff]
[ 1.923294] pci_bus 0000:03: resource 0 [io 0xb000-0xbfff]
[ 1.923295] pci_bus 0000:03: resource 1 [mem 0xfa800000-0xfadfffff]
[ 1.923296] pci_bus 0000:05: resource 1 [mem 0xfae00000-0xfaefffff]
[ 1.923297] pci_bus 0000:05: resource 2 [mem 0xf0000000-0xf8ffffff 64bit pref]
[ 1.923298] pci_bus 0000:06: resource 0 [io 0xc000-0xcfff]
[ 1.923299] pci_bus 0000:06: resource 1 [mem 0xfb800000-0xfbdfffff]
[ 1.923301] pci_bus 0000:07: resource 1 [mem 0xfaf00000-0xfb7fffff]
[ 1.923302] pci_bus 0000:07: resource 2 [mem 0xf9000000-0xf9ffffff 64bit pref]
[ 1.923303] pci_bus 0000:07: resource 4 [io 0x0000-0x03af window]
[ 1.923304] pci_bus 0000:07: resource 5 [io 0x03e0-0x03ff window]
[ 1.923305] pci_bus 0000:07: resource 6 [io 0x03b0-0x03bb window]
[ 1.923306] pci_bus 0000:07: resource 7 [io 0x03c0-0x03df window]
[ 1.923307] pci_bus 0000:07: resource 8 [io 0x0400-0x041f window]
[ 1.923308] pci_bus 0000:07: resource 9 [io 0x0d00-0xcfff window]
[ 1.923309] pci_bus 0000:07: resource 10 [mem 0x000a0000-0x000bffff window]
[ 1.923310] pci_bus 0000:07: resource 11 [mem 0x000d0000-0x000dffff window]
[ 1.923311] pci_bus 0000:07: resource 12 [mem 0xf0000000-0xfbffffff window]
[ 1.923357] pci 0000:80:00.0: PCI bridge to [bus 81]
[ 1.923546] pci 0000:80:01.0: PCI bridge to [bus 82]
[ 1.923735] pci 0000:80:03.0: PCI bridge to [bus 83]
[ 1.923920] pci 0000:80:03.0: bridge window [mem 0xdec00000-0xdecfffff]
[ 1.924113] pci 0000:85:03.0: PCI bridge to [bus 86]
[ 1.924307] pci 0000:85:03.0: bridge window [io 0xd000-0xdfff]
[ 1.924496] pci 0000:85:03.0: bridge window [mem 0xded00000-0xdedfffff]
[ 1.924688] pci 0000:85:03.0: bridge window [mem 0xdde00000-0xddefffff 64bit pref]
[ 1.925006] pci 0000:85:07.0: PCI bridge to [bus 87]
[ 1.925191] pci 0000:85:07.0: bridge window [io 0xe000-0xefff]
[ 1.925394] pci 0000:85:07.0: bridge window [mem 0xdee00000-0xdeefffff]
[ 1.925585] pci 0000:85:07.0: bridge window [mem 0xddf00000-0xddffffff 64bit pref]
[ 1.925903] pci 0000:84:00.0: PCI bridge to [bus 85-87]
[ 1.926089] pci 0000:84:00.0: bridge window [io 0xd000-0xefff]
[ 1.926289] pci 0000:84:00.0: bridge window [mem 0xded00000-0xdeefffff]
[ 1.926479] pci 0000:84:00.0: bridge window [mem 0xdde00000-0xddffffff 64bit pref]
[ 1.926805] pci 0000:80:05.0: PCI bridge to [bus 84-87]
[ 1.926992] pci 0000:80:05.0: bridge window [io 0xd000-0xefff]
[ 1.927184] pci 0000:80:05.0: bridge window [mem 0xded00000-0xdeefffff]
[ 1.927386] pci 0000:80:05.0: bridge window [mem 0xdde00000-0xddffffff 64bit pref]
[ 1.927704] pci 0000:80:07.0: PCI bridge to [bus 88]
[ 1.927887] pci 0000:80:07.0: bridge window [io 0xf000-0xffff]
[ 1.928075] pci 0000:80:07.0: bridge window [mem 0xdef00000-0xdfffffff]
[ 1.928277] pci 0000:80:07.0: bridge window [mem 0xce000000-0xd7ffffff 64bit pref]
[ 1.928592] pci 0000:80:09.0: PCI bridge to [bus 89]
[ 1.928782] pci_bus 0000:80: resource 4 [io 0xd000-0xffff window]
[ 1.928783] pci_bus 0000:80: resource 5 [mem 0xce000000-0xdfffffff window]
[ 1.928784] pci_bus 0000:83: resource 1 [mem 0xdec00000-0xdecfffff]
[ 1.928785] pci_bus 0000:84: resource 0 [io 0xd000-0xefff]
[ 1.928786] pci_bus 0000:84: resource 1 [mem 0xded00000-0xdeefffff]
[ 1.928787] pci_bus 0000:84: resource 2 [mem 0xdde00000-0xddffffff 64bit pref]
[ 1.928788] pci_bus 0000:85: resource 0 [io 0xd000-0xefff]
[ 1.928789] pci_bus 0000:85: resource 1 [mem 0xded00000-0xdeefffff]
[ 1.928791] pci_bus 0000:85: resource 2 [mem 0xdde00000-0xddffffff 64bit pref]
[ 1.928792] pci_bus 0000:86: resource 0 [io 0xd000-0xdfff]
[ 1.928793] pci_bus 0000:86: resource 1 [mem 0xded00000-0xdedfffff]
[ 1.928794] pci_bus 0000:86: resource 2 [mem 0xdde00000-0xddefffff 64bit pref]
[ 1.928795] pci_bus 0000:87: resource 0 [io 0xe000-0xefff]
[ 1.928796] pci_bus 0000:87: resource 1 [mem 0xdee00000-0xdeefffff]
[ 1.928797] pci_bus 0000:87: resource 2 [mem 0xddf00000-0xddffffff 64bit pref]
[ 1.928798] pci_bus 0000:88: resource 0 [io 0xf000-0xffff]
[ 1.928799] pci_bus 0000:88: resource 1 [mem 0xdef00000-0xdfffffff]
[ 1.928800] pci_bus 0000:88: resource 2 [mem 0xce000000-0xd7ffffff 64bit pref]
[ 1.928804] pci_bus 0000:fe: resource 4 [io 0x0000-0xffff]
[ 1.928805] pci_bus 0000:fe: resource 5 [mem 0x00000000-0xffffffffff]
[ 1.928808] pci_bus 0000:ff: resource 4 [io 0x0000-0xffff]
[ 1.928809] pci_bus 0000:ff: resource 5 [mem 0x00000000-0xffffffffff]
[ 1.928858] NET: Registered protocol family 2
[ 1.929431] TCP established hash table entries: 524288 (order: 10, 4194304 bytes)
[ 1.930650] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[ 1.930994] TCP: Hash tables configured (established 524288 bind 65536)
[ 1.931272] UDP hash table entries: 32768 (order: 8, 1048576 bytes)
[ 1.931665] UDP-Lite hash table entries: 32768 (order: 8, 1048576 bytes)
[ 1.932116] NET: Registered protocol family 1
[ 1.932475] RPC: Registered named UNIX socket transport module.
[ 1.932661] RPC: Registered udp transport module.
[ 1.932844] RPC: Registered tcp transport module.
[ 1.933027] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 1.934907] pci 0000:07:04.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[ 1.935347] PCI: CLS 256 bytes, default 64
[ 1.935400] Trying to unpack rootfs image as initramfs...
[ 2.421170] Freeing initrd memory: 4208K (ffff8800377b7000 - ffff880037bd3000)
[ 2.426294] DMAR: dmar0: Using Queued invalidation
[ 2.426498] DMAR: dmar1: Using Queued invalidation
[ 2.426911] DMAR: Hardware identity mapping for device 0000:00:00.0
[ 2.427105] DMAR: Hardware identity mapping for device 0000:00:01.0
[ 2.427303] DMAR: Hardware identity mapping for device 0000:00:03.0
[ 2.427493] DMAR: Hardware identity mapping for device 0000:00:05.0
[ 2.427683] DMAR: Hardware identity mapping for device 0000:00:07.0
[ 2.427872] DMAR: Hardware identity mapping for device 0000:00:09.0
[ 2.428059] DMAR: Hardware identity mapping for device 0000:00:13.0
[ 2.428252] DMAR: Hardware identity mapping for device 0000:00:14.0
[ 2.428440] DMAR: Hardware identity mapping for device 0000:00:14.1
[ 2.428628] DMAR: Hardware identity mapping for device 0000:00:14.2
[ 2.428816] DMAR: Hardware identity mapping for device 0000:00:14.3
[ 2.429005] DMAR: Hardware identity mapping for device 0000:00:16.0
[ 2.429193] DMAR: Hardware identity mapping for device 0000:00:16.1
[ 2.429384] DMAR: Hardware identity mapping for device 0000:00:16.2
[ 2.429573] DMAR: Hardware identity mapping for device 0000:00:16.3
[ 2.429761] DMAR: Hardware identity mapping for device 0000:00:16.4
[ 2.429949] DMAR: Hardware identity mapping for device 0000:00:16.5
[ 2.430139] DMAR: Hardware identity mapping for device 0000:00:16.6
[ 2.430332] DMAR: Hardware identity mapping for device 0000:00:16.7
[ 2.430520] DMAR: Hardware identity mapping for device 0000:00:1a.0
[ 2.430709] DMAR: Hardware identity mapping for device 0000:00:1a.1
[ 2.430897] DMAR: Hardware identity mapping for device 0000:00:1a.2
[ 2.431085] DMAR: Hardware identity mapping for device 0000:00:1a.7
[ 2.431278] DMAR: Hardware identity mapping for device 0000:00:1d.0
[ 2.431467] DMAR: Hardware identity mapping for device 0000:00:1d.1
[ 2.431656] DMAR: Hardware identity mapping for device 0000:00:1d.2
[ 2.431844] DMAR: Hardware identity mapping for device 0000:00:1d.7
[ 2.432033] DMAR: Hardware identity mapping for device 0000:00:1f.0
[ 2.432221] DMAR: Hardware identity mapping for device 0000:00:1f.2
[ 2.432413] DMAR: Hardware identity mapping for device 0000:00:1f.3
[ 2.435588] DMAR: Hardware identity mapping for device 0000:01:00.0
[ 2.435780] DMAR: Hardware identity mapping for device 0000:01:00.1
[ 2.435979] DMAR: Hardware identity mapping for device 0000:03:00.0
[ 2.436230] DMAR: Hardware identity mapping for device 0000:05:00.0
[ 2.436430] DMAR: Hardware identity mapping for device 0000:06:00.0
[ 2.436630] DMAR: Hardware identity mapping for device 0000:80:00.0
[ 2.436826] DMAR: Hardware identity mapping for device 0000:80:01.0
[ 2.437017] DMAR: Hardware identity mapping for device 0000:80:03.0
[ 2.437206] DMAR: Hardware identity mapping for device 0000:80:05.0
[ 2.437399] DMAR: Hardware identity mapping for device 0000:80:07.0
[ 2.437588] DMAR: Hardware identity mapping for device 0000:80:09.0
[ 2.437776] DMAR: Hardware identity mapping for device 0000:80:13.0
[ 2.437966] DMAR: Hardware identity mapping for device 0000:80:14.0
[ 2.438154] DMAR: Hardware identity mapping for device 0000:80:14.1
[ 2.438347] DMAR: Hardware identity mapping for device 0000:80:14.2
[ 2.438536] DMAR: Hardware identity mapping for device 0000:80:14.3
[ 2.438725] DMAR: Hardware identity mapping for device 0000:80:16.0
[ 2.438913] DMAR: Hardware identity mapping for device 0000:80:16.1
[ 2.439101] DMAR: Hardware identity mapping for device 0000:80:16.2
[ 2.439294] DMAR: Hardware identity mapping for device 0000:80:16.3
[ 2.439483] DMAR: Hardware identity mapping for device 0000:80:16.4
[ 2.439673] DMAR: Hardware identity mapping for device 0000:80:16.5
[ 2.439861] DMAR: Hardware identity mapping for device 0000:80:16.6
[ 2.440049] DMAR: Hardware identity mapping for device 0000:80:16.7
[ 2.440250] DMAR: Hardware identity mapping for device 0000:83:00.0
[ 2.440445] DMAR: Hardware identity mapping for device 0000:84:00.0
[ 2.440641] DMAR: Hardware identity mapping for device 0000:85:03.0
[ 2.440831] DMAR: Hardware identity mapping for device 0000:85:07.0
[ 2.441028] DMAR: Hardware identity mapping for device 0000:86:00.0
[ 2.441231] DMAR: Hardware identity mapping for device 0000:87:00.0
[ 2.441427] DMAR: Hardware identity mapping for device 0000:88:00.0
[ 2.441616] DMAR: Hardware identity mapping for device 0000:88:00.1
[ 2.441813] DMAR: Hardware identity mapping for device 0000:fe:00.0
[ 2.442003] DMAR: Hardware identity mapping for device 0000:fe:00.1
[ 2.442191] DMAR: Hardware identity mapping for device 0000:fe:02.0
[ 2.442383] DMAR: Hardware identity mapping for device 0000:fe:02.1
[ 2.442572] DMAR: Hardware identity mapping for device 0000:fe:02.2
[ 2.442760] DMAR: Hardware identity mapping for device 0000:fe:02.3
[ 2.442947] DMAR: Hardware identity mapping for device 0000:fe:02.4
[ 2.443135] DMAR: Hardware identity mapping for device 0000:fe:02.5
[ 2.443328] DMAR: Hardware identity mapping for device 0000:fe:03.0
[ 2.443515] DMAR: Hardware identity mapping for device 0000:fe:03.1
[ 2.443703] DMAR: Hardware identity mapping for device 0000:fe:03.2
[ 2.443892] DMAR: Hardware identity mapping for device 0000:fe:03.4
[ 2.444080] DMAR: Hardware identity mapping for device 0000:fe:04.0
[ 2.444272] DMAR: Hardware identity mapping for device 0000:fe:04.1
[ 2.444460] DMAR: Hardware identity mapping for device 0000:fe:04.2
[ 2.444649] DMAR: Hardware identity mapping for device 0000:fe:04.3
[ 2.444837] DMAR: Hardware identity mapping for device 0000:fe:05.0
[ 2.445026] DMAR: Hardware identity mapping for device 0000:fe:05.1
[ 2.445214] DMAR: Hardware identity mapping for device 0000:fe:05.2
[ 2.445412] DMAR: Hardware identity mapping for device 0000:fe:05.3
[ 2.445601] DMAR: Hardware identity mapping for device 0000:fe:06.0
[ 2.445791] DMAR: Hardware identity mapping for device 0000:fe:06.1
[ 2.445979] DMAR: Hardware identity mapping for device 0000:fe:06.2
[ 2.446167] DMAR: Hardware identity mapping for device 0000:fe:06.3
[ 2.446369] DMAR: Hardware identity mapping for device 0000:ff:00.0
[ 2.446557] DMAR: Hardware identity mapping for device 0000:ff:00.1
[ 2.446748] DMAR: Hardware identity mapping for device 0000:ff:02.0
[ 2.446938] DMAR: Hardware identity mapping for device 0000:ff:02.1
[ 2.447129] DMAR: Hardware identity mapping for device 0000:ff:02.2
[ 2.447321] DMAR: Hardware identity mapping for device 0000:ff:02.3
[ 2.447509] DMAR: Hardware identity mapping for device 0000:ff:02.4
[ 2.447697] DMAR: Hardware identity mapping for device 0000:ff:02.5
[ 2.447885] DMAR: Hardware identity mapping for device 0000:ff:03.0
[ 2.448074] DMAR: Hardware identity mapping for device 0000:ff:03.1
[ 2.448266] DMAR: Hardware identity mapping for device 0000:ff:03.2
[ 2.448454] DMAR: Hardware identity mapping for device 0000:ff:03.4
[ 2.448643] DMAR: Hardware identity mapping for device 0000:ff:04.0
[ 2.448831] DMAR: Hardware identity mapping for device 0000:ff:04.1
[ 2.449020] DMAR: Hardware identity mapping for device 0000:ff:04.2
[ 2.449208] DMAR: Hardware identity mapping for device 0000:ff:04.3
[ 2.449400] DMAR: Hardware identity mapping for device 0000:ff:05.0
[ 2.449588] DMAR: Hardware identity mapping for device 0000:ff:05.1
[ 2.449776] DMAR: Hardware identity mapping for device 0000:ff:05.2
[ 2.449965] DMAR: Hardware identity mapping for device 0000:ff:05.3
[ 2.450153] DMAR: Hardware identity mapping for device 0000:ff:06.0
[ 2.450346] DMAR: Hardware identity mapping for device 0000:ff:06.1
[ 2.450534] DMAR: Hardware identity mapping for device 0000:ff:06.2
[ 2.450722] DMAR: Hardware identity mapping for device 0000:ff:06.3
[ 2.450910] DMAR: Setting RMRR:
[ 2.451089] DMAR: Ignoring identity map for HW passthrough device 0000:00:1a.0 [0xbf7ec000 - 0xbf7fffff]
[ 2.451413] DMAR: Ignoring identity map for HW passthrough device 0000:00:1a.1 [0xbf7ec000 - 0xbf7fffff]
[ 2.451732] DMAR: Ignoring identity map for HW passthrough device 0000:00:1a.2 [0xbf7ec000 - 0xbf7fffff]
[ 2.452051] DMAR: Ignoring identity map for HW passthrough device 0000:00:1a.7 [0xbf7ec000 - 0xbf7fffff]
[ 2.452376] DMAR: Ignoring identity map for HW passthrough device 0000:00:1d.0 [0xbf7ec000 - 0xbf7fffff]
[ 2.452695] DMAR: Ignoring identity map for HW passthrough device 0000:00:1d.1 [0xbf7ec000 - 0xbf7fffff]
[ 2.453015] DMAR: Ignoring identity map for HW passthrough device 0000:00:1d.2 [0xbf7ec000 - 0xbf7fffff]
[ 2.453338] DMAR: Ignoring identity map for HW passthrough device 0000:00:1d.7 [0xbf7ec000 - 0xbf7fffff]
[ 2.453658] DMAR: Ignoring identity map for HW passthrough device 0000:00:1a.0 [0xec000 - 0xeffff]
[ 2.453977] DMAR: Ignoring identity map for HW passthrough device 0000:00:1a.1 [0xec000 - 0xeffff]
[ 2.454299] DMAR: Ignoring identity map for HW passthrough device 0000:00:1a.2 [0xec000 - 0xeffff]
[ 2.454617] DMAR: Ignoring identity map for HW passthrough device 0000:00:1a.7 [0xec000 - 0xeffff]
[ 2.454935] DMAR: Ignoring identity map for HW passthrough device 0000:00:1d.0 [0xec000 - 0xeffff]
[ 2.455259] DMAR: Ignoring identity map for HW passthrough device 0000:00:1d.1 [0xec000 - 0xeffff]
[ 2.455577] DMAR: Ignoring identity map for HW passthrough device 0000:00:1d.2 [0xec000 - 0xeffff]
[ 2.455895] DMAR: Ignoring identity map for HW passthrough device 0000:00:1d.7 [0xec000 - 0xeffff]
[ 2.456214] DMAR: Prepare 0-16MiB unity mapping for LPC
[ 2.456402] DMAR: Ignoring identity map for HW passthrough device 0000:00:1f.0 [0x0 - 0xffffff]
[ 2.456779] DMAR: Intel(R) Virtualization Technology for Directed I/O
[ 2.457051] iommu: Adding device 0000:00:00.0 to group 0
[ 2.457262] iommu: Adding device 0000:00:01.0 to group 1
[ 2.457463] iommu: Adding device 0000:00:03.0 to group 2
[ 2.457664] iommu: Adding device 0000:00:05.0 to group 3
[ 2.457866] iommu: Adding device 0000:00:07.0 to group 4
[ 2.458072] iommu: Adding device 0000:00:09.0 to group 5
[ 2.458291] iommu: Adding device 0000:00:13.0 to group 6
[ 2.458541] iommu: Adding device 0000:00:14.0 to group 7
[ 2.458744] iommu: Adding device 0000:00:14.1 to group 7
[ 2.458947] iommu: Adding device 0000:00:14.2 to group 7
[ 2.459149] iommu: Adding device 0000:00:14.3 to group 7
[ 2.459456] iommu: Adding device 0000:00:16.0 to group 8
[ 2.459659] iommu: Adding device 0000:00:16.1 to group 8
[ 2.459863] iommu: Adding device 0000:00:16.2 to group 8
[ 2.460066] iommu: Adding device 0000:00:16.3 to group 8
[ 2.460274] iommu: Adding device 0000:00:16.4 to group 8
[ 2.460477] iommu: Adding device 0000:00:16.5 to group 8
[ 2.460681] iommu: Adding device 0000:00:16.6 to group 8
[ 2.460884] iommu: Adding device 0000:00:16.7 to group 8
[ 2.461130] iommu: Adding device 0000:00:1a.0 to group 9
[ 2.461341] iommu: Adding device 0000:00:1a.1 to group 9
[ 2.461543] iommu: Adding device 0000:00:1a.2 to group 9
[ 2.461744] iommu: Adding device 0000:00:1a.7 to group 9
[ 2.461991] iommu: Adding device 0000:00:1d.0 to group 10
[ 2.462193] iommu: Adding device 0000:00:1d.1 to group 10
[ 2.462401] iommu: Adding device 0000:00:1d.2 to group 10
[ 2.462603] iommu: Adding device 0000:00:1d.7 to group 10
[ 2.462807] iommu: Adding device 0000:00:1e.0 to group 11
[ 2.463043] iommu: Adding device 0000:00:1f.0 to group 12
[ 2.463254] iommu: Adding device 0000:00:1f.2 to group 12
[ 2.463457] iommu: Adding device 0000:00:1f.3 to group 12
[ 2.463663] iommu: Adding device 0000:01:00.0 to group 13
[ 2.463868] iommu: Adding device 0000:01:00.1 to group 14
[ 2.464074] iommu: Adding device 0000:03:00.0 to group 15
[ 2.464286] iommu: Adding device 0000:05:00.0 to group 16
[ 2.464490] iommu: Adding device 0000:06:00.0 to group 17
[ 2.464679] iommu: Adding device 0000:07:04.0 to group 11
[ 2.464881] iommu: Adding device 0000:80:00.0 to group 18
[ 2.465083] iommu: Adding device 0000:80:01.0 to group 19
[ 2.465289] iommu: Adding device 0000:80:03.0 to group 20
[ 2.465492] iommu: Adding device 0000:80:05.0 to group 21
[ 2.465694] iommu: Adding device 0000:80:07.0 to group 22
[ 2.465895] iommu: Adding device 0000:80:09.0 to group 23
[ 2.466107] iommu: Adding device 0000:80:13.0 to group 24
[ 2.466362] iommu: Adding device 0000:80:14.0 to group 25
[ 2.466570] iommu: Adding device 0000:80:14.1 to group 25
[ 2.466780] iommu: Adding device 0000:80:14.2 to group 25
[ 2.466988] iommu: Adding device 0000:80:14.3 to group 25
[ 2.467295] iommu: Adding device 0000:80:16.0 to group 26
[ 2.467502] iommu: Adding device 0000:80:16.1 to group 26
[ 2.467708] iommu: Adding device 0000:80:16.2 to group 26
[ 2.467914] iommu: Adding device 0000:80:16.3 to group 26
[ 2.468121] iommu: Adding device 0000:80:16.4 to group 26
[ 2.468334] iommu: Adding device 0000:80:16.5 to group 26
[ 2.468540] iommu: Adding device 0000:80:16.6 to group 26
[ 2.468746] iommu: Adding device 0000:80:16.7 to group 26
[ 2.468951] iommu: Adding device 0000:83:00.0 to group 27
[ 2.469155] iommu: Adding device 0000:84:00.0 to group 28
[ 2.469369] iommu: Adding device 0000:85:03.0 to group 29
[ 2.469576] iommu: Adding device 0000:85:07.0 to group 30
[ 2.469769] iommu: Adding device 0000:86:00.0 to group 29
[ 2.469960] iommu: Adding device 0000:87:00.0 to group 30
[ 2.470190] iommu: Adding device 0000:88:00.0 to group 31
[ 2.473330] iommu: Adding device 0000:88:00.1 to group 31
[ 2.473555] iommu: Adding device 0000:fe:00.0 to group 32
[ 2.473761] iommu: Adding device 0000:fe:00.1 to group 32
[ 2.474029] iommu: Adding device 0000:fe:02.0 to group 33
[ 2.474248] iommu: Adding device 0000:fe:02.1 to group 33
[ 2.474455] iommu: Adding device 0000:fe:02.2 to group 33
[ 2.474661] iommu: Adding device 0000:fe:02.3 to group 33
[ 2.474867] iommu: Adding device 0000:fe:02.4 to group 33
[ 2.475073] iommu: Adding device 0000:fe:02.5 to group 33
[ 2.475330] iommu: Adding device 0000:fe:03.0 to group 34
[ 2.475537] iommu: Adding device 0000:fe:03.1 to group 34
[ 2.475744] iommu: Adding device 0000:fe:03.2 to group 34
[ 2.475952] iommu: Adding device 0000:fe:03.4 to group 34
[ 2.476199] iommu: Adding device 0000:fe:04.0 to group 35
[ 2.476411] iommu: Adding device 0000:fe:04.1 to group 35
[ 2.476618] iommu: Adding device 0000:fe:04.2 to group 35
[ 2.476828] iommu: Adding device 0000:fe:04.3 to group 35
[ 2.477078] iommu: Adding device 0000:fe:05.0 to group 36
[ 2.477290] iommu: Adding device 0000:fe:05.1 to group 36
[ 2.477499] iommu: Adding device 0000:fe:05.2 to group 36
[ 2.477706] iommu: Adding device 0000:fe:05.3 to group 36
[ 2.477952] iommu: Adding device 0000:fe:06.0 to group 37
[ 2.478161] iommu: Adding device 0000:fe:06.1 to group 37
[ 2.478376] iommu: Adding device 0000:fe:06.2 to group 37
[ 2.478583] iommu: Adding device 0000:fe:06.3 to group 37
[ 2.478807] iommu: Adding device 0000:ff:00.0 to group 38
[ 2.479016] iommu: Adding device 0000:ff:00.1 to group 38
[ 2.479292] iommu: Adding device 0000:ff:02.0 to group 39
[ 2.479501] iommu: Adding device 0000:ff:02.1 to group 39
[ 2.479709] iommu: Adding device 0000:ff:02.2 to group 39
[ 2.479918] iommu: Adding device 0000:ff:02.3 to group 39
[ 2.480127] iommu: Adding device 0000:ff:02.4 to group 39
[ 2.480340] iommu: Adding device 0000:ff:02.5 to group 39
[ 2.480585] iommu: Adding device 0000:ff:03.0 to group 40
[ 2.480795] iommu: Adding device 0000:ff:03.1 to group 40
[ 2.481005] iommu: Adding device 0000:ff:03.2 to group 40
[ 2.481213] iommu: Adding device 0000:ff:03.4 to group 40
[ 2.481464] iommu: Adding device 0000:ff:04.0 to group 41
[ 2.481674] iommu: Adding device 0000:ff:04.1 to group 41
[ 2.481883] iommu: Adding device 0000:ff:04.2 to group 41
[ 2.482091] iommu: Adding device 0000:ff:04.3 to group 41
[ 2.482342] iommu: Adding device 0000:ff:05.0 to group 42
[ 2.482554] iommu: Adding device 0000:ff:05.1 to group 42
[ 2.482764] iommu: Adding device 0000:ff:05.2 to group 42
[ 2.482974] iommu: Adding device 0000:ff:05.3 to group 42
[ 2.483219] iommu: Adding device 0000:ff:06.0 to group 43
[ 2.483435] iommu: Adding device 0000:ff:06.1 to group 43
[ 2.483645] iommu: Adding device 0000:ff:06.2 to group 43
[ 2.483856] iommu: Adding device 0000:ff:06.3 to group 43
[ 2.486486] kvm: VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL does not work properly. Using workaround
[ 2.488397] Scanning for low memory corruption every 60 seconds
[ 2.489065] futex hash table entries: 8192 (order: 7, 524288 bytes)
[ 2.489370] audit: initializing netlink subsys (disabled)
[ 2.489577] audit: type=2000 audit(1481389684.294:1): initialized
[ 2.490423] workingset: timestamp_bits=40 max_order=24 bucket_order=0
[ 2.492876] NFS: Registering the id_resolver key type
[ 2.493069] Key type id_resolver registered
[ 2.493269] Key type id_legacy registered
[ 2.493451] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[ 2.494013] SELinux: Registering netfilter hooks
[ 2.495040] async_tx: api initialized (async)
[ 2.495291] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
[ 2.495607] io scheduler noop registered
[ 2.495787] io scheduler deadline registered
[ 2.495973] io scheduler cfq registered (default)
[ 2.497493] pcieport 0000:00:01.0: Signaling PME through PCIe PME interrupt
[ 2.497685] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
[ 2.497873] pci 0000:01:00.1: Signaling PME through PCIe PME interrupt
[ 2.498064] pcie_pme 0000:00:01.0:pcie001: service driver pcie_pme loaded
[ 2.498082] pcieport 0000:00:03.0: Signaling PME through PCIe PME interrupt
[ 2.498304] pci 0000:03:00.0: Signaling PME through PCIe PME interrupt
[ 2.498495] pcie_pme 0000:00:03.0:pcie001: service driver pcie_pme loaded
[ 2.498514] pcieport 0000:00:05.0: Signaling PME through PCIe PME interrupt
[ 2.498705] pcie_pme 0000:00:05.0:pcie001: service driver pcie_pme loaded
[ 2.498723] pcieport 0000:00:07.0: Signaling PME through PCIe PME interrupt
[ 2.498912] pci 0000:05:00.0: Signaling PME through PCIe PME interrupt
[ 2.499102] pcie_pme 0000:00:07.0:pcie001: service driver pcie_pme loaded
[ 2.499119] pcieport 0000:00:09.0: Signaling PME through PCIe PME interrupt
[ 2.499337] pci 0000:06:00.0: Signaling PME through PCIe PME interrupt
[ 2.499527] pcie_pme 0000:00:09.0:pcie001: service driver pcie_pme loaded
[ 2.499544] pcieport 0000:80:00.0: Signaling PME through PCIe PME interrupt
[ 2.499735] pcie_pme 0000:80:00.0:pcie001: service driver pcie_pme loaded
[ 2.499749] pcieport 0000:80:01.0: Signaling PME through PCIe PME interrupt
[ 2.499940] pcie_pme 0000:80:01.0:pcie001: service driver pcie_pme loaded
[ 2.499954] pcieport 0000:80:03.0: Signaling PME through PCIe PME interrupt
[ 2.500143] pci 0000:83:00.0: Signaling PME through PCIe PME interrupt
[ 2.500357] pcie_pme 0000:80:03.0:pcie001: service driver pcie_pme loaded
[ 2.500372] pcieport 0000:80:05.0: Signaling PME through PCIe PME interrupt
[ 2.500562] pcieport 0000:84:00.0: Signaling PME through PCIe PME interrupt
[ 2.500751] pcieport 0000:85:03.0: Signaling PME through PCIe PME interrupt
[ 2.500940] pci 0000:86:00.0: Signaling PME through PCIe PME interrupt
[ 2.501129] pcieport 0000:85:07.0: Signaling PME through PCIe PME interrupt
[ 2.501345] pci 0000:87:00.0: Signaling PME through PCIe PME interrupt
[ 2.501534] pcie_pme 0000:80:05.0:pcie001: service driver pcie_pme loaded
[ 2.501549] pcieport 0000:80:07.0: Signaling PME through PCIe PME interrupt
[ 2.501738] pci 0000:88:00.0: Signaling PME through PCIe PME interrupt
[ 2.501927] pci 0000:88:00.1: Signaling PME through PCIe PME interrupt
[ 2.502117] pcie_pme 0000:80:07.0:pcie001: service driver pcie_pme loaded
[ 2.502131] pcieport 0000:80:09.0: Signaling PME through PCIe PME interrupt
[ 2.502349] pcie_pme 0000:80:09.0:pcie001: service driver pcie_pme loaded
[ 2.502356] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[ 2.502651] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input0
[ 2.502972] ACPI: Power Button [PWRB]
[ 2.503184] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input1
[ 2.503528] ACPI: Power Button [PWRF]
[ 2.505054] Monitor-Mwait will be used to enter C-1 state
[ 2.505063] Monitor-Mwait will be used to enter C-2 state
[ 2.505071] Monitor-Mwait will be used to enter C-3 state
[ 2.508865] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 2.529647] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[ 2.550600] 00:04: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A
[ 2.551317] Non-volatile memory driver v1.3
[ 2.551523] Linux agpgart interface v0.103
[ 2.551798] [drm] Initialized drm 1.1.0 20060810
[ 2.557690] brd: module loaded
[ 2.559674] loop: module loaded
[ 2.559900] mpt3sas version 13.100.00.00 loaded
[ 2.560453] mpt2sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (49512412 kB)
[ 2.615636] mpt2sas_cm0: MSI-X vectors supported: 1, no of cores: 24, max_msix_vectors: -1
[ 2.615952] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 43
[ 2.616113] mpt2sas_cm0: iomem(0x00000000fad3c000), mapped(0xffffc90006a80000), size(16384)
[ 2.616397] mpt2sas_cm0: ioport(0x000000000000b800), size(256)
[ 2.699196] DMAR: 32bit 0000:03:00.0 uses non-identity mapping
[ 2.719928] mpt2sas_cm0: Allocated physical memory: size(8800 kB)
[ 2.720093] mpt2sas_cm0: Current Controller Queue Depth(3905),Max Controller Queue Depth(3992)
[ 2.720375] mpt2sas_cm0: Scatter Gather Elements per IO(128)
[ 2.765703] mpt2sas_cm0: log_info(0x30030100): originator(IOP), code(0x03), sub_code(0x0100)
[ 2.766008] mpt2sas_cm0: log_info(0x30030100): originator(IOP), code(0x03), sub_code(0x0100)
[ 2.766292] mpt2sas_cm0: LSISAS2008: FWVersion(09.00.00.00), ChipRevision(0x03), BiosVersion(07.27.00.00)
[ 2.766571] mpt2sas_cm0: Protocol=(
[ 2.766619] Initiator
[ 2.766821] ,Target
[ 2.766863] ),
[ 2.767061] Capabilities=(
[ 2.767105] TLR
[ 2.767308] ,EEDP
[ 2.767350] ,Snapshot Buffer
[ 2.767549] ,Diag Trace Buffer
[ 2.767593] ,Task Set Full
[ 2.767790] ,NCQ
[ 2.767832] )
[ 2.768190] scsi host0: Fusion MPT SAS Host
[ 2.768763] mpt2sas_cm0: sending port enable !!
[ 3.490244] tsc: Refined TSC clocksource calibration: 2666.760 MHz
[ 3.491292] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x267095f0616, max_idle_ns: 440795338298 ns
[ 4.281969] mpt2sas_cm0: host_add: handle(0x0001), sas_addr(0x500605b003e93e60), phys(8)
[ 4.283475] random: fast init done
[ 4.286453] mpt2sas_cm0: expander_add: handle(0x0009), parent(0x0001), sas_addr(0x5001c4500029b4bf), phys(38)
[ 4.514604] clocksource: Switched to clocksource tsc
[ 9.788019] mpt2sas_cm0: port enable: SUCCESS
[ 10.278095] scsi 0:0:0:0: Direct-Access ATA Hitachi HUA72302 A840 PQ: 0 ANSI: 5
[ 10.278377] scsi 0:0:0:0: SATA: handle(0x000a), sas_addr(0x5001c4500029b48c), phy(12), device_name(0x0000000000000000)
[ 10.278656] scsi 0:0:0:0: SATA: enclosure_logical_id(0x5001c4500029b4bf), slot(0)
[ 10.279119] scsi 0:0:0:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[ 11.528128] scsi 0:0:1:0: Direct-Access ATA Hitachi HUA72302 A840 PQ: 0 ANSI: 5
[ 11.528404] scsi 0:0:1:0: SATA: handle(0x000b), sas_addr(0x5001c4500029b48d), phy(13), device_name(0x0000000000000000)
[ 11.528675] scsi 0:0:1:0: SATA: enclosure_logical_id(0x5001c4500029b4bf), slot(1)
[ 11.529123] scsi 0:0:1:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[ 12.778140] scsi 0:0:2:0: Direct-Access ATA Hitachi HUA72302 A840 PQ: 0 ANSI: 5
[ 12.778424] scsi 0:0:2:0: SATA: handle(0x000c), sas_addr(0x5001c4500029b48e), phy(14), device_name(0x0000000000000000)
[ 12.778702] scsi 0:0:2:0: SATA: enclosure_logical_id(0x5001c4500029b4bf), slot(2)
[ 12.779187] scsi 0:0:2:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[ 14.028168] scsi 0:0:3:0: Direct-Access ATA Hitachi HUA72302 A840 PQ: 0 ANSI: 5
[ 14.028444] scsi 0:0:3:0: SATA: handle(0x000d), sas_addr(0x5001c4500029b48f), phy(15), device_name(0x0000000000000000)
[ 14.028716] scsi 0:0:3:0: SATA: enclosure_logical_id(0x5001c4500029b4bf), slot(3)
[ 14.029164] scsi 0:0:3:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[ 15.278185] scsi 0:0:4:0: Direct-Access ATA Hitachi HUA72302 A840 PQ: 0 ANSI: 5
[ 15.280968] scsi 0:0:4:0: SATA: handle(0x000e), sas_addr(0x5001c4500029b494), phy(20), device_name(0x0000000000000000)
[ 15.281251] scsi 0:0:4:0: SATA: enclosure_logical_id(0x5001c4500029b4bf), slot(8)
[ 15.281722] scsi 0:0:4:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[ 16.528227] scsi 0:0:5:0: Direct-Access ATA Hitachi HUA72302 A840 PQ: 0 ANSI: 5
[ 16.528503] scsi 0:0:5:0: SATA: handle(0x000f), sas_addr(0x5001c4500029b495), phy(21), device_name(0x0000000000000000)
[ 16.528775] scsi 0:0:5:0: SATA: enclosure_logical_id(0x5001c4500029b4bf), slot(9)
[ 16.529245] scsi 0:0:5:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[ 17.778240] scsi 0:0:6:0: Direct-Access ATA Hitachi HUA72302 A840 PQ: 0 ANSI: 5
[ 17.778524] scsi 0:0:6:0: SATA: handle(0x0010), sas_addr(0x5001c4500029b496), phy(22), device_name(0x0000000000000000)
[ 17.778801] scsi 0:0:6:0: SATA: enclosure_logical_id(0x5001c4500029b4bf), slot(10)
[ 17.779262] scsi 0:0:6:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[ 19.028261] scsi 0:0:7:0: Direct-Access ATA Hitachi HUA72302 A840 PQ: 0 ANSI: 5
[ 19.028539] scsi 0:0:7:0: SATA: handle(0x0011), sas_addr(0x5001c4500029b497), phy(23), device_name(0x0000000000000000)
[ 19.028811] scsi 0:0:7:0: SATA: enclosure_logical_id(0x5001c4500029b4bf), slot(11)
[ 19.029284] scsi 0:0:7:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[ 20.278287] scsi 0:0:8:0: Direct-Access ATA Hitachi HUA72302 A840 PQ: 0 ANSI: 5
[ 20.278578] scsi 0:0:8:0: SATA: handle(0x0012), sas_addr(0x5001c4500029b498), phy(24), device_name(0x0000000000000000)
[ 20.278859] scsi 0:0:8:0: SATA: enclosure_logical_id(0x5001c4500029b4bf), slot(12)
[ 20.279331] scsi 0:0:8:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[ 21.528313] scsi 0:0:9:0: Direct-Access ATA Hitachi HUA72302 A840 PQ: 0 ANSI: 5
[ 21.528601] scsi 0:0:9:0: SATA: handle(0x0013), sas_addr(0x5001c4500029b499), phy(25), device_name(0x0000000000000000)
[ 21.528873] scsi 0:0:9:0: SATA: enclosure_logical_id(0x5001c4500029b4bf), slot(13)
[ 21.529341] scsi 0:0:9:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[ 22.778332] scsi 0:0:10:0: Direct-Access ATA Hitachi HUA72302 A840 PQ: 0 ANSI: 5
[ 22.778626] scsi 0:0:10:0: SATA: handle(0x0014), sas_addr(0x5001c4500029b49a), phy(26), device_name(0x0000000000000000)
[ 22.778905] scsi 0:0:10:0: SATA: enclosure_logical_id(0x5001c4500029b4bf), slot(14)
[ 22.779376] scsi 0:0:10:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[ 24.028359] scsi 0:0:11:0: Direct-Access ATA Hitachi HUA72302 A840 PQ: 0 ANSI: 5
[ 24.028654] scsi 0:0:11:0: SATA: handle(0x0015), sas_addr(0x5001c4500029b49b), phy(27), device_name(0x0000000000000000)
[ 24.028926] scsi 0:0:11:0: SATA: enclosure_logical_id(0x5001c4500029b4bf), slot(15)
[ 24.029393] scsi 0:0:11:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[ 25.278378] scsi 0:0:12:0: Direct-Access ATA Hitachi HUA72302 A840 PQ: 0 ANSI: 5
[ 25.278665] scsi 0:0:12:0: SATA: handle(0x0016), sas_addr(0x5001c4500029b49c), phy(28), device_name(0x0000000000000000)
[ 25.278943] scsi 0:0:12:0: SATA: enclosure_logical_id(0x5001c4500029b4bf), slot(16)
[ 25.279421] scsi 0:0:12:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[ 26.528410] scsi 0:0:13:0: Direct-Access ATA Hitachi HUA72302 A840 PQ: 0 ANSI: 5
[ 26.528696] scsi 0:0:13:0: SATA: handle(0x0017), sas_addr(0x5001c4500029b49d), phy(29), device_name(0x0000000000000000)
[ 26.528968] scsi 0:0:13:0: SATA: enclosure_logical_id(0x5001c4500029b4bf), slot(17)
[ 26.529410] scsi 0:0:13:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[ 27.778422] scsi 0:0:14:0: Direct-Access ATA Hitachi HUA72302 A840 PQ: 0 ANSI: 5
[ 27.778708] scsi 0:0:14:0: SATA: handle(0x0018), sas_addr(0x5001c4500029b49e), phy(30), device_name(0x0000000000000000)
[ 27.778988] scsi 0:0:14:0: SATA: enclosure_logical_id(0x5001c4500029b4bf), slot(18)
[ 27.779463] scsi 0:0:14:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[ 29.028450] scsi 0:0:15:0: Direct-Access ATA Hitachi HUA72302 A840 PQ: 0 ANSI: 5
[ 29.028737] scsi 0:0:15:0: SATA: handle(0x0019), sas_addr(0x5001c4500029b49f), phy(31), device_name(0x0000000000000000)
[ 29.029010] scsi 0:0:15:0: SATA: enclosure_logical_id(0x5001c4500029b4bf), slot(19)
[ 29.029462] scsi 0:0:15:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[ 30.278467] scsi 0:0:16:0: Direct-Access ATA Hitachi HUA72302 A840 PQ: 0 ANSI: 5
[ 30.278754] scsi 0:0:16:0: SATA: handle(0x001a), sas_addr(0x5001c4500029b4a0), phy(32), device_name(0x0000000000000000)
[ 30.279033] scsi 0:0:16:0: SATA: enclosure_logical_id(0x5001c4500029b4bf), slot(20)
[ 30.279508] scsi 0:0:16:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[ 31.528520] scsi 0:0:17:0: Direct-Access ATA Hitachi HUA72302 A840 PQ: 0 ANSI: 5
[ 31.528804] scsi 0:0:17:0: SATA: handle(0x001b), sas_addr(0x5001c4500029b4a1), phy(33), device_name(0x0000000000000000)
[ 31.529079] scsi 0:0:17:0: SATA: enclosure_logical_id(0x5001c4500029b4bf), slot(21)
[ 31.529549] scsi 0:0:17:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[ 32.778510] scsi 0:0:18:0: Direct-Access ATA Hitachi HUA72302 A840 PQ: 0 ANSI: 5
[ 32.778799] scsi 0:0:18:0: SATA: handle(0x001c), sas_addr(0x5001c4500029b4a2), phy(34), device_name(0x0000000000000000)
[ 32.779080] scsi 0:0:18:0: SATA: enclosure_logical_id(0x5001c4500029b4bf), slot(22)
[ 32.779563] scsi 0:0:18:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[ 34.028369] scsi 0:0:19:0: Direct-Access ATA Hitachi HUA72302 AA10 PQ: 0 ANSI: 5
[ 34.028653] scsi 0:0:19:0: SATA: handle(0x001d), sas_addr(0x5001c4500029b4a3), phy(35), device_name(0x0000000000000000)
[ 34.028935] scsi 0:0:19:0: SATA: enclosure_logical_id(0x5001c4500029b4bf), slot(23)
[ 34.029404] scsi 0:0:19:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[ 34.785562] scsi 0:0:20:0: Enclosure 80H10462301A0 0703 PQ: 0 ANSI: 5
[ 34.785844] scsi 0:0:20:0: set ignore_delay_remove for handle(0x001e)
[ 34.786010] scsi 0:0:20:0: SES: handle(0x001e), sas_addr(0x5001c4500029b4bd), phy(36), device_name(0x50c40150bdb42900)
[ 34.786287] scsi 0:0:20:0: SES: enclosure_logical_id(0x5001c4500029b4bf), slot(0)
[ 34.805402] mpt2sas_cm1: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (49512412 kB)
[ 34.860882] mpt2sas_cm1: MSI-X vectors supported: 1, no of cores: 24, max_msix_vectors: 8
[ 34.861206] mpt2sas1-msix0: PCI-MSI-X enabled: IRQ 45
[ 34.861366] mpt2sas_cm1: iomem(0x00000000fbd3c000), mapped(0xffffc90006a88000), size(16384)
[ 34.861639] mpt2sas_cm1: ioport(0x000000000000c800), size(256)
[ 34.945783] DMAR: 32bit 0000:06:00.0 uses non-identity mapping
[ 34.953413] mpt2sas_cm1: Allocated physical memory: size(4178 kB)
[ 34.953576] mpt2sas_cm1: Current Controller Queue Depth(1832),Max Controller Queue Depth(1952)
[ 34.953848] mpt2sas_cm1: Scatter Gather Elements per IO(128)
[ 34.999735] mpt2sas_cm1: LSISAS2008: FWVersion(14.00.00.00), ChipRevision(0x03), BiosVersion(07.27.00.00)
[ 35.000022] mpt2sas_cm1: Protocol=(
[ 35.000069] Initiator
[ 35.000266] ),
[ 35.000308] Capabilities=(
[ 35.000506] Raid
[ 35.000548] ,TLR
[ 35.000739] ,EEDP
[ 35.000781] ,Snapshot Buffer
[ 35.000973] ,Diag Trace Buffer
[ 35.001019] ,Task Set Full
[ 35.001211] ,NCQ
[ 35.001253] )
[ 35.001597] scsi host1: Fusion MPT SAS Host
[ 35.002062] mpt2sas_cm1: sending port enable !!
[ 37.515497] mpt2sas_cm1: host_add: handle(0x0001), sas_addr(0x500304800463b200), phys(8)
[ 42.645711] mpt2sas_cm1: port enable: SUCCESS
[ 42.646253] sd 0:0:0:0: tag#0 CDB: Test Unit Ready 00 00 00 00 00 00
[ 42.646421] mpt2sas_cm0: sas_address(0x5001c4500029b48c), phy(12)
[ 42.646586] mpt2sas_cm0: enclosure_logical_id(0x5001c4500029b4bf),slot(0)
[ 42.646751] mpt2sas_cm0: handle(0x000a), ioc_status(success)(0x0000), smid(1)
[ 42.646959] sd 0:0:0:0: Attached scsi generic sg0 type 0
[ 42.647122] sd 0:0:1:0: Attached scsi generic sg1 type 0
[ 42.647328] sd 0:0:2:0: Attached scsi generic sg2 type 0
[ 42.647504] mpt2sas_cm0: request_len(0), underflow(0), resid(0)
[ 42.647572] sd 0:0:3:0: Attached scsi generic sg3 type 0
[ 42.647732] sd 0:0:4:0: Attached scsi generic sg4 type 0
[ 42.647878] sd 0:0:5:0: Attached scsi generic sg5 type 0
[ 42.648026] sd 0:0:6:0: Attached scsi generic sg6 type 0
[ 42.648118] sd 0:0:7:0: Attached scsi generic sg7 type 0
[ 42.648168] sd 0:0:8:0: Attached scsi generic sg8 type 0
[ 42.648215] sd 0:0:9:0: Attached scsi generic sg9 type 0
[ 42.648284] sd 0:0:10:0: Attached scsi generic sg10 type 0
[ 42.648358] sd 0:0:11:0: Attached scsi generic sg11 type 0
[ 42.648394] sd 0:0:12:0: Attached scsi generic sg12 type 0
[ 42.648423] sd 0:0:13:0: Attached scsi generic sg13 type 0
[ 42.648453] sd 0:0:14:0: Attached scsi generic sg14 type 0
[ 42.648479] sd 0:0:15:0: Attached scsi generic sg15 type 0
[ 42.648506] sd 0:0:16:0: Attached scsi generic sg16 type 0
[ 42.648536] sd 0:0:17:0: Attached scsi generic sg17 type 0
[ 42.648562] sd 0:0:18:0: Attached scsi generic sg18 type 0
[ 42.648589] sd 0:0:19:0: Attached scsi generic sg19 type 0
[ 42.648619] scsi 0:0:20:0: Attached scsi generic sg20 type 13
[ 42.653044] mpt2sas_cm0: tag(65535), transfer_count(0), sc->result(0x00000000)
[ 42.653315] mpt2sas_cm0: scsi_status(check condition)(0x02), scsi_state(autosense valid )(0x01)
[ 42.653588] mpt2sas_cm0: [sense_key,asc,ascq]: [0x06,0x29,0x00], count(18)
[ 42.653752] sd 0:0:1:0: tag#1 CDB: Test Unit Ready 00 00 00 00 00 00
[ 42.653914] mpt2sas_cm0: sas_address(0x5001c4500029b48d), phy(13)
[ 42.654076] mpt2sas_cm0: enclosure_logical_id(0x5001c4500029b4bf),slot(1)
[ 42.654239] mpt2sas_cm0: handle(0x000b), ioc_status(success)(0x0000), smid(2)
[ 42.654511] mpt2sas_cm0: request_len(0), underflow(0), resid(0)
[ 42.654672] mpt2sas_cm0: tag(65535), transfer_count(0), sc->result(0x00000000)
[ 42.654942] mpt2sas_cm0: scsi_status(check condition)(0x02), scsi_state(autosense valid )(0x01)
[ 42.655215] mpt2sas_cm0: [sense_key,asc,ascq]: [0x06,0x29,0x00], count(18)
[ 42.655380] sd 0:0:2:0: tag#2 CDB: Test Unit Ready 00 00 00 00 00 00
[ 42.655544] mpt2sas_cm0: sas_address(0x5001c4500029b48e), phy(14)
[ 42.655707] mpt2sas_cm0: enclosure_logical_id(0x5001c4500029b4bf),slot(2)
[ 42.655871] mpt2sas_cm0: handle(0x000c), ioc_status(success)(0x0000), smid(3)
[ 42.656144] mpt2sas_cm0: request_len(0), underflow(0), resid(0)
[ 42.656306] mpt2sas_cm0: tag(65535), transfer_count(0), sc->result(0x00000000)
[ 42.656586] mpt2sas_cm0: scsi_status(check condition)(0x02), scsi_state(autosense valid )(0x01)
[ 42.656868] mpt2sas_cm0: [sense_key,asc,ascq]: [0x06,0x29,0x00], count(18)
[ 42.657036] sd 0:0:3:0: tag#3 CDB: Test Unit Ready 00 00 00 00 00 00
[ 42.657200] mpt2sas_cm0: sas_address(0x5001c4500029b48f), phy(15)
[ 42.657360] mpt2sas_cm0: enclosure_logical_id(0x5001c4500029b4bf),slot(3)
[ 42.657525] mpt2sas_cm0: handle(0x000d), ioc_status(success)(0x0000), smid(4)
[ 42.657797] mpt2sas_cm0: request_len(0), underflow(0), resid(0)
[ 42.657958] mpt2sas_cm0: tag(65535), transfer_count(0), sc->result(0x00000000)
[ 42.658223] mpt2sas_cm0: scsi_status(check condition)(0x02), scsi_state(autosense valid )(0x01)
[ 42.658490] mpt2sas_cm0: [sense_key,asc,ascq]: [0x06,0x29,0x00], count(18)
[ 42.658651] sd 0:0:4:0: tag#4 CDB: Test Unit Ready 00 00 00 00 00 00
[ 42.658809] mpt2sas_cm0: sas_address(0x5001c4500029b494), phy(20)
[ 42.658966] mpt2sas_cm0: enclosure_logical_id(0x5001c4500029b4bf),slot(8)
[ 42.659125] mpt2sas_cm0: handle(0x000e), ioc_status(success)(0x0000), smid(5)
[ 42.659390] mpt2sas_cm0: request_len(0), underflow(0), resid(0)
[ 42.659550] mpt2sas_cm0: tag(65535), transfer_count(0), sc->result(0x00000000)
[ 42.659828] mpt2sas_cm0: scsi_status(check condition)(0x02), scsi_state(autosense valid )(0x01)
[ 42.660104] mpt2sas_cm0: [sense_key,asc,ascq]: [0x06,0x29,0x00], count(18)
[ 42.660268] sd 0:0:5:0: tag#5 CDB: Test Unit Ready 00 00 00 00 00 00
[ 42.660429] mpt2sas_cm0: sas_address(0x5001c4500029b495), phy(21)
[ 42.660590] mpt2sas_cm0: enclosure_logical_id(0x5001c4500029b4bf),slot(9)
[ 42.660756] mpt2sas_cm0: handle(0x000f), ioc_status(success)(0x0000), smid(6)
[ 42.661034] mpt2sas_cm0: request_len(0), underflow(0), resid(0)
[ 42.661198] mpt2sas_cm0: tag(65535), transfer_count(0), sc->result(0x00000000)
[ 42.661473] mpt2sas_cm0: scsi_status(check condition)(0x02), scsi_state(autosense valid )(0x01)
[ 42.661747] mpt2sas_cm0: [sense_key,asc,ascq]: [0x06,0x29,0x00], count(18)
[ 42.661912] sd 0:0:6:0: tag#6 CDB: Test Unit Ready 00 00 00 00 00 00
[ 42.662074] mpt2sas_cm0: sas_address(0x5001c4500029b496), phy(22)
[ 42.662238] mpt2sas_cm0: enclosure_logical_id(0x5001c4500029b4bf),slot(10)
[ 42.662404] mpt2sas_cm0: handle(0x0010), ioc_status(success)(0x0000), smid(7)
[ 42.662677] mpt2sas_cm0: request_len(0), underflow(0), resid(0)
[ 42.662838] mpt2sas_cm0: tag(65535), transfer_count(0), sc->result(0x00000000)
[ 42.663108] mpt2sas_cm0: scsi_status(check condition)(0x02), scsi_state(autosense valid )(0x01)
[ 42.663388] mpt2sas_cm0: [sense_key,asc,ascq]: [0x06,0x29,0x00], count(18)
[ 42.663555] sd 0:0:7:0: tag#7 CDB: Test Unit Ready 00 00 00 00 00 00
[ 42.663720] mpt2sas_cm0: sas_address(0x5001c4500029b497), phy(23)
[ 42.663884] mpt2sas_cm0: enclosure_logical_id(0x5001c4500029b4bf),slot(11)
[ 42.664049] mpt2sas_cm0: handle(0x0011), ioc_status(success)(0x0000), smid(8)
[ 42.664323] mpt2sas_cm0: request_len(0), underflow(0), resid(0)
[ 42.664485] mpt2sas_cm0: tag(65535), transfer_count(0), sc->result(0x00000000)
[ 42.664763] mpt2sas_cm0: scsi_status(check condition)(0x02), scsi_state(autosense valid )(0x01)
[ 42.665040] mpt2sas_cm0: [sense_key,asc,ascq]: [0x06,0x29,0x00], count(18)
[ 42.665207] sd 0:0:8:0: tag#8 CDB: Test Unit Ready 00 00 00 00 00 00
[ 42.665371] mpt2sas_cm0: sas_address(0x5001c4500029b498), phy(24)
[ 42.665535] mpt2sas_cm0: enclosure_logical_id(0x5001c4500029b4bf),slot(12)
[ 42.665696] mpt2sas_cm0: handle(0x0012), ioc_status(success)(0x0000), smid(9)
[ 42.665966] mpt2sas_cm0: request_len(0), underflow(0), resid(0)
[ 42.666127] mpt2sas_cm0: tag(65535), transfer_count(0), sc->result(0x00000000)
[ 42.666404] mpt2sas_cm0: scsi_status(check condition)(0x02), scsi_state(autosense valid )(0x01)
[ 42.666693] mpt2sas_cm0: [sense_key,asc,ascq]: [0x06,0x29,0x00], count(18)
[ 42.666894] ahci 0000:00:1f.2: version 3.0
[ 42.667035] sd 0:0:9:0: tag#9 CDB: Test Unit Ready 00 00 00 00 00 00
[ 42.667202] mpt2sas_cm0: sas_address(0x5001c4500029b499), phy(25)
[ 42.667368] mpt2sas_cm0: enclosure_logical_id(0x5001c4500029b4bf),slot(13)
[ 42.667533] mpt2sas_cm0: handle(0x0013), ioc_status(success)(0x0000), smid(10)
[ 42.667812] mpt2sas_cm0: request_len(0), underflow(0), resid(0)
[ 42.667976] mpt2sas_cm0: tag(65535), transfer_count(0), sc->result(0x00000000)
[ 42.668251] mpt2sas_cm0: scsi_status(check condition)(0x02), scsi_state(autosense valid )(0x01)
[ 42.668524] mpt2sas_cm0: [sense_key,asc,ascq]: [0x06,0x29,0x00], count(18)
[ 42.668693] sd 0:0:10:0: tag#10 CDB: Test Unit Ready 00 00 00 00 00 00
[ 42.668857] mpt2sas_cm0: sas_address(0x5001c4500029b49a), phy(26)
[ 42.669021] mpt2sas_cm0: enclosure_logical_id(0x5001c4500029b4bf),slot(14)
[ 42.669186] mpt2sas_cm0: handle(0x0014), ioc_status(success)(0x0000), smid(11)
[ 42.669463] mpt2sas_cm0: request_len(0), underflow(0), resid(0)
[ 42.669627] mpt2sas_cm0: tag(65535), transfer_count(0), sc->result(0x00000000)
[ 42.669904] mpt2sas_cm0: scsi_status(check condition)(0x02), scsi_state(autosense valid )(0x01)
[ 42.670183] mpt2sas_cm0: [sense_key,asc,ascq]: [0x06,0x29,0x00], count(18)
[ 42.670350] sd 0:0:11:0: tag#11 CDB: Test Unit Ready 00 00 00 00 00 00
[ 42.670515] mpt2sas_cm0: sas_address(0x5001c4500029b49b), phy(27)
[ 42.670679] mpt2sas_cm0: enclosure_logical_id(0x5001c4500029b4bf),slot(15)
[ 42.670845] mpt2sas_cm0: handle(0x0015), ioc_status(success)(0x0000), smid(12)
[ 42.671122] mpt2sas_cm0: request_len(0), underflow(0), resid(0)
[ 42.671282] mpt2sas_cm0: tag(65535), transfer_count(0), sc->result(0x00000000)
[ 42.671553] mpt2sas_cm0: scsi_status(check condition)(0x02), scsi_state(autosense valid )(0x01)
[ 42.671831] mpt2sas_cm0: [sense_key,asc,ascq]: [0x06,0x29,0x00], count(18)
[ 42.671997] sd 0:0:12:0: tag#12 CDB: Test Unit Ready 00 00 00 00 00 00
[ 42.672161] mpt2sas_cm0: sas_address(0x5001c4500029b49c), phy(28)
[ 42.672318] mpt2sas_cm0: enclosure_logical_id(0x5001c4500029b4bf),slot(16)
[ 42.672477] mpt2sas_cm0: handle(0x0016), ioc_status(success)(0x0000), smid(13)
[ 42.672741] mpt2sas_cm0: request_len(0), underflow(0), resid(0)
[ 42.672899] mpt2sas_cm0: tag(65535), transfer_count(0), sc->result(0x00000000)
[ 42.673164] mpt2sas_cm0: scsi_status(check condition)(0x02), scsi_state(autosense valid )(0x01)
[ 42.673431] mpt2sas_cm0: [sense_key,asc,ascq]: [0x06,0x29,0x00], count(18)
[ 42.673592] sd 0:0:13:0: tag#13 CDB: Test Unit Ready 00 00 00 00 00 00
[ 42.673750] mpt2sas_cm0: sas_address(0x5001c4500029b49d), phy(29)
[ 42.673915] mpt2sas_cm0: enclosure_logical_id(0x5001c4500029b4bf),slot(17)
[ 42.674080] mpt2sas_cm0: handle(0x0017), ioc_status(success)(0x0000), smid(14)
[ 42.674357] mpt2sas_cm0: request_len(0), underflow(0), resid(0)
[ 42.674521] mpt2sas_cm0: tag(65535), transfer_count(0), sc->result(0x00000000)
[ 42.674796] mpt2sas_cm0: scsi_status(check condition)(0x02), scsi_state(autosense valid )(0x01)
[ 42.675075] mpt2sas_cm0: [sense_key,asc,ascq]: [0x06,0x29,0x00], count(18)
[ 42.675242] sd 0:0:14:0: tag#14 CDB: Test Unit Ready 00 00 00 00 00 00
[ 42.675405] mpt2sas_cm0: sas_address(0x5001c4500029b49e), phy(30)
[ 42.675568] mpt2sas_cm0: enclosure_logical_id(0x5001c4500029b4bf),slot(18)
[ 42.675732] mpt2sas_cm0: handle(0x0018), ioc_status(success)(0x0000), smid(15)
[ 42.676009] mpt2sas_cm0: request_len(0), underflow(0), resid(0)
[ 42.676172] mpt2sas_cm0: tag(65535), transfer_count(0), sc->result(0x00000000)
[ 42.676447] mpt2sas_cm0: scsi_status(check condition)(0x02), scsi_state(autosense valid )(0x01)
[ 42.676729] mpt2sas_cm0: [sense_key,asc,ascq]: [0x06,0x29,0x00], count(18)
[ 42.676896] sd 0:0:15:0: tag#15 CDB: Test Unit Ready 00 00 00 00 00 00
[ 42.677062] mpt2sas_cm0: sas_address(0x5001c4500029b49f), phy(31)
[ 42.677227] mpt2sas_cm0: enclosure_logical_id(0x5001c4500029b4bf),slot(19)
[ 42.677392] mpt2sas_cm0: handle(0x0019), ioc_status(success)(0x0000), smid(16)
[ 42.677668] mpt2sas_cm0: request_len(0), underflow(0), resid(0)
[ 42.677832] mpt2sas_cm0: tag(65535), transfer_count(0), sc->result(0x00000000)
[ 42.678103] mpt2sas_cm0: scsi_status(check condition)(0x02), scsi_state(autosense valid )(0x01)
[ 42.678377] mpt2sas_cm0: [sense_key,asc,ascq]: [0x06,0x29,0x00], count(18)
[ 42.678544] sd 0:0:16:0: tag#16 CDB: Test Unit Ready 00 00 00 00 00 00
[ 42.678709] mpt2sas_cm0: sas_address(0x5001c4500029b4a0), phy(32)
[ 42.678873] mpt2sas_cm0: enclosure_logical_id(0x5001c4500029b4bf),slot(20)
[ 42.681531] mpt2sas_cm0: handle(0x001a), ioc_status(success)(0x0000), smid(17)
[ 42.681807] mpt2sas_cm0: request_len(0), underflow(0), resid(0)
[ 42.681968] mpt2sas_cm0: tag(65535), transfer_count(0), sc->result(0x00000000)
[ 42.682239] mpt2sas_cm0: scsi_status(check condition)(0x02), scsi_state(autosense valid )(0x01)
[ 42.682513] mpt2sas_cm0: [sense_key,asc,ascq]: [0x06,0x29,0x00], count(18)
[ 42.682680] sd 0:0:17:0: tag#17 CDB: Test Unit Ready 00 00 00 00 00 00
[ 42.682845] mpt2sas_cm0: sas_address(0x5001c4500029b4a1), phy(33)
[ 42.683009] mpt2sas_cm0: enclosure_logical_id(0x5001c4500029b4bf),slot(21)
[ 42.683174] mpt2sas_cm0: handle(0x001b), ioc_status(success)(0x0000), smid(18)
[ 42.683451] mpt2sas_cm0: request_len(0), underflow(0), resid(0)
[ 42.683613] mpt2sas_cm0: tag(65535), transfer_count(0), sc->result(0x00000000)
[ 42.683886] mpt2sas_cm0: scsi_status(check condition)(0x02), scsi_state(autosense valid )(0x01)
[ 42.684164] mpt2sas_cm0: [sense_key,asc,ascq]: [0x06,0x29,0x00], count(18)
[ 42.684331] sd 0:0:18:0: tag#18 CDB: Test Unit Ready 00 00 00 00 00 00
[ 42.684495] mpt2sas_cm0: sas_address(0x5001c4500029b4a2), phy(34)
[ 42.684656] mpt2sas_cm0: enclosure_logical_id(0x5001c4500029b4bf),slot(22)
[ 42.684818] mpt2sas_cm0: handle(0x001c), ioc_status(success)(0x0000), smid(19)
[ 42.685093] mpt2sas_cm0: request_len(0), underflow(0), resid(0)
[ 42.685256] mpt2sas_cm0: tag(65535), transfer_count(0), sc->result(0x00000000)
[ 42.685533] mpt2sas_cm0: scsi_status(check condition)(0x02), scsi_state(autosense valid )(0x01)
[ 42.685813] mpt2sas_cm0: [sense_key,asc,ascq]: [0x06,0x29,0x00], count(18)
[ 42.685979] sd 0:0:19:0: tag#19 CDB: Test Unit Ready 00 00 00 00 00 00
[ 42.686143] mpt2sas_cm0: sas_address(0x5001c4500029b4a3), phy(35)
[ 42.686307] mpt2sas_cm0: enclosure_logical_id(0x5001c4500029b4bf),slot(23)
[ 42.686473] mpt2sas_cm0: handle(0x001d), ioc_status(success)(0x0000), smid(20)
[ 42.686737] mpt2sas_cm0: request_len(0), underflow(0), resid(0)
[ 42.686895] mpt2sas_cm0: tag(65535), transfer_count(0), sc->result(0x00000000)
[ 42.687160] mpt2sas_cm0: scsi_status(check condition)(0x02), scsi_state(autosense valid )(0x01)
[ 42.687427] mpt2sas_cm0: [sense_key,asc,ascq]: [0x06,0x29,0x00], count(18)
[ 42.687679] ahci 0000:00:1f.2: SSS flag set, parallel bus scan disabled
[ 42.687899] ahci 0000:00:1f.2: AHCI 0001.0200 32 slots 6 ports 3 Gbps 0x3f impl SATA mode
[ 42.688185] ahci 0000:00:1f.2: flags: 64bit ncq sntf stag pm led clo pio slum part ccc ems sxs
[ 42.688978] sd 0:0:2:0: [sdc] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[ 42.688979] sd 0:0:3:0: [sdd] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[ 42.689011] sd 0:0:7:0: [sdh] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[ 42.689046] sd 0:0:19:0: [sdt] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[ 42.689133] sd 0:0:4:0: [sde] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[ 42.689134] sd 0:0:8:0: [sdi] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[ 42.689192] sd 0:0:6:0: [sdg] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[ 42.689196] sd 0:0:5:0: [sdf] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[ 42.689199] sd 0:0:15:0: [sdp] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[ 42.689242] sd 0:0:1:0: [sdb] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[ 42.689268] sd 0:0:17:0: [sdr] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[ 42.689268] sd 0:0:0:0: [sda] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[ 42.689270] sd 0:0:18:0: [sds] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[ 42.689320] sd 0:0:16:0: [sdq] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[ 42.689352] sd 0:0:12:0: [sdm] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[ 42.689357] sd 0:0:9:0: [sdj] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[ 42.689504] sd 0:0:14:0: [sdo] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[ 42.689506] sd 0:0:11:0: [sdl] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[ 42.689508] sd 0:0:13:0: [sdn] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[ 42.689562] sd 0:0:10:0: [sdk] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[ 42.695031] sd 0:0:3:0: [sdd] Write Protect is off
[ 42.695033] sd 0:0:3:0: [sdd] Mode Sense: 7f 00 00 08
[ 42.695061] sd 0:0:7:0: [sdh] Write Protect is off
[ 42.695063] sd 0:0:7:0: [sdh] Mode Sense: 7f 00 00 08
[ 42.695137] sd 0:0:15:0: [sdp] Write Protect is off
[ 42.695138] sd 0:0:15:0: [sdp] Mode Sense: 7f 00 00 08
[ 42.695176] sd 0:0:8:0: [sdi] Write Protect is off
[ 42.695178] sd 0:0:8:0: [sdi] Mode Sense: 7f 00 00 08
[ 42.695204] sd 0:0:5:0: [sdf] Write Protect is off
[ 42.695205] sd 0:0:5:0: [sdf] Mode Sense: 7f 00 00 08
[ 42.695340] sd 0:0:6:0: [sdg] Write Protect is off
[ 42.695342] sd 0:0:6:0: [sdg] Mode Sense: 7f 00 00 08
[ 42.695464] sd 0:0:18:0: [sds] Write Protect is off
[ 42.695465] sd 0:0:4:0: [sde] Write Protect is off
[ 42.695466] sd 0:0:18:0: [sds] Mode Sense: 7f 00 00 08
[ 42.695467] sd 0:0:4:0: [sde] Mode Sense: 7f 00 00 08
[ 42.695542] sd 0:0:16:0: [sdq] Write Protect is off
[ 42.695544] sd 0:0:16:0: [sdq] Mode Sense: 7f 00 00 08
[ 42.695556] sd 0:0:9:0: [sdj] Write Protect is off
[ 42.695558] sd 0:0:9:0: [sdj] Mode Sense: 7f 00 00 08
[ 42.695567] sd 0:0:0:0: [sda] Write Protect is off
[ 42.695568] sd 0:0:0:0: [sda] Mode Sense: 7f 00 00 08
[ 42.695578] sd 0:0:12:0: [sdm] Write Protect is off
[ 42.695580] sd 0:0:12:0: [sdm] Mode Sense: 7f 00 00 08
[ 42.695642] sd 0:0:11:0: [sdl] Write Protect is off
[ 42.695643] sd 0:0:11:0: [sdl] Mode Sense: 7f 00 00 08
[ 42.695670] sd 0:0:14:0: [sdo] Write Protect is off
[ 42.695672] sd 0:0:14:0: [sdo] Mode Sense: 7f 00 00 08
[ 42.695683] sd 0:0:17:0: [sdr] Write Protect is off
[ 42.695685] sd 0:0:17:0: [sdr] Mode Sense: 7f 00 00 08
[ 42.695740] sd 0:0:10:0: [sdk] Write Protect is off
[ 42.695742] sd 0:0:10:0: [sdk] Mode Sense: 7f 00 00 08
[ 42.695747] sd 0:0:13:0: [sdn] Write Protect is off
[ 42.695749] sd 0:0:13:0: [sdn] Mode Sense: 7f 00 00 08
[ 42.696180] sd 0:0:1:0: [sdb] Write Protect is off
[ 42.696181] sd 0:0:1:0: [sdb] Mode Sense: 7f 00 00 08
[ 42.697372] sd 0:0:3:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 42.697504] sd 0:0:15:0: [sdp] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 42.697524] sd 0:0:7:0: [sdh] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 42.697528] sd 0:0:8:0: [sdi] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 42.697568] sd 0:0:5:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 42.697804] sd 0:0:6:0: [sdg] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 42.697914] sd 0:0:18:0: [sds] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 42.697924] sd 0:0:16:0: [sdq] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 42.697965] sd 0:0:4:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 42.698000] sd 0:0:12:0: [sdm] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 42.698037] sd 0:0:9:0: [sdj] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 42.698065] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 42.698141] sd 0:0:11:0: [sdl] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 42.698162] sd 0:0:14:0: [sdo] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 42.698223] sd 0:0:17:0: [sdr] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 42.698256] sd 0:0:10:0: [sdk] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 42.698263] sd 0:0:13:0: [sdn] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 42.698981] sd 0:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 42.703272] scsi host2: ahci
[ 42.703488] scsi host3: ahci
[ 42.703564] scsi host4: ahci
[ 42.703657] scsi host5: ahci
[ 42.703850] scsi host6: ahci
[ 42.704020] scsi host7: ahci
[ 42.704057] ata1: SATA max UDMA/133 abar m2048@0xfbed6000 port 0xfbed6100 irq 46
[ 42.704059] ata2: SATA max UDMA/133 abar m2048@0xfbed6000 port 0xfbed6180 irq 46
[ 42.704060] ata3: SATA max UDMA/133 abar m2048@0xfbed6000 port 0xfbed6200 irq 46
[ 42.704061] ata4: SATA max UDMA/133 abar m2048@0xfbed6000 port 0xfbed6280 irq 46
[ 42.704063] ata5: SATA max UDMA/133 abar m2048@0xfbed6000 port 0xfbed6300 irq 46
[ 42.704064] ata6: SATA max UDMA/133 abar m2048@0xfbed6000 port 0xfbed6380 irq 46
[ 42.704307] tun: Universal TUN/TAP device driver, 1.6
[ 42.704308] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[ 42.704412] e100: Intel(R) PRO/100 Network Driver, 3.5.24-k2-NAPI
[ 42.704412] e100: Copyright(c) 1999-2006 Intel Corporation
[ 42.704438] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
[ 42.704439] e1000: Copyright (c) 1999-2006 Intel Corporation.
[ 42.704463] sky2: driver version 1.30
[ 42.704621] mlx4_core: Mellanox ConnectX core driver v2.2-1 (Feb, 2014)
[ 42.704642] mlx4_core: Initializing 0000:05:00.0
[ 42.715075] sd 0:0:2:0: [sdc] Write Protect is off
[ 42.715266] sd 0:0:2:0: [sdc] Mode Sense: 7f 00 00 08
[ 42.717745] sd 0:0:2:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 42.732078] random: crng init done
[ 42.752380] sdf: sdf1 sdf9
[ 42.752786] sdg: sdg1 sdg9
[ 42.754366] sdn: sdn1 sdn9
[ 42.755512] sdk: sdk1 sdk9
[ 42.756761] sdq: sdq1 sdq9
[ 42.757438] sdl: sdl1 sdl9
[ 42.757812] sda: sda1 sda9
[ 42.759219] sdj: sdj1 sdj9
[ 42.759615] sdi: sdi1 sdi9
[ 42.760350] sdh: sdh1 sdh9
[ 42.761539] sds: sds1 sds9
[ 42.763081] sdm: sdm1 sdm9
[ 42.763548] sdb: sdb1 sdb9
[ 42.763839] sdr: sdr1 sdr9
[ 42.763900] sdp: sdp1 sdp9
[ 42.764112] sdd: sdd1 sdd9
[ 42.764780] sdo: sdo1 sdo9
[ 42.765766] sde: sde1 sde9
[ 42.775869] sdc: sdc1 sdc9
[ 42.779447] sd 0:0:5:0: [sdf] Attached SCSI disk
[ 42.782511] sd 0:0:6:0: [sdg] Attached SCSI disk
[ 42.784002] sd 0:0:13:0: [sdn] Attached SCSI disk
[ 42.785638] sd 0:0:16:0: [sdq] Attached SCSI disk
[ 42.786652] sd 0:0:10:0: [sdk] Attached SCSI disk
[ 42.787075] sd 0:0:11:0: [sdl] Attached SCSI disk
[ 42.787423] sd 0:0:9:0: [sdj] Attached SCSI disk
[ 42.788335] sd 0:0:8:0: [sdi] Attached SCSI disk
[ 42.788824] sd 0:0:18:0: [sds] Attached SCSI disk
[ 42.788933] sd 0:0:0:0: [sda] Attached SCSI disk
[ 42.790468] sd 0:0:12:0: [sdm] Attached SCSI disk
[ 42.792146] sd 0:0:1:0: [sdb] Attached SCSI disk
[ 42.792486] sd 0:0:7:0: [sdh] Attached SCSI disk
[ 42.793782] sd 0:0:17:0: [sdr] Attached SCSI disk
[ 42.794042] sd 0:0:4:0: [sde] Attached SCSI disk
[ 42.794743] sd 0:0:14:0: [sdo] Attached SCSI disk
[ 42.797127] sd 0:0:3:0: [sdd] Attached SCSI disk
[ 42.802078] sd 0:0:15:0: [sdp] Attached SCSI disk
[ 42.807718] sd 0:0:2:0: [sdc] Attached SCSI disk
[ 42.903893] sd 0:0:19:0: [sdt] Write Protect is off
[ 42.904086] sd 0:0:19:0: [sdt] Mode Sense: 7f 00 00 08
[ 42.906275] sd 0:0:19:0: [sdt] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 43.117727] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 43.118100] ata1.00: ATA-10: Patriot Torch, S9FM02.8, max UDMA/133
[ 43.118271] ata1.00: 117231408 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
[ 43.118613] ata1.00: configured for UDMA/133
[ 43.119036] scsi 2:0:0:0: Direct-Access ATA Patriot Torch 02.8 PQ: 0 ANSI: 5
[ 43.132990] sd 2:0:0:0: Attached scsi generic sg21 type 0
[ 43.133067] sd 2:0:0:0: [sdu] 117231408 512-byte logical blocks: (60.0 GB/55.9 GiB)
[ 43.133486] sd 2:0:0:0: [sdu] Write Protect is off
[ 43.133656] sd 2:0:0:0: [sdu] Mode Sense: 00 3a 00 00
[ 43.133798] sd 2:0:0:0: [sdu] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 43.135294] sdu: sdu1
[ 43.136527] sd 2:0:0:0: [sdu] Attached SCSI disk
[ 43.145700] sdt: sdt1 sdt9
[ 43.375000] sd 0:0:19:0: [sdt] Attached SCSI disk
[ 43.439041] ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 43.439453] ata2.00: ATA-10: Patriot Torch, S9FM02.8, max UDMA/133
[ 43.439620] ata2.00: 117231408 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
[ 43.440047] ata2.00: configured for UDMA/133
[ 43.440668] scsi 3:0:0:0: Direct-Access ATA Patriot Torch 02.8 PQ: 0 ANSI: 5
[ 43.454984] sd 3:0:0:0: Attached scsi generic sg22 type 0
[ 43.455068] sd 3:0:0:0: [sdv] 117231408 512-byte logical blocks: (60.0 GB/55.9 GiB)
[ 43.455569] sd 3:0:0:0: [sdv] Write Protect is off
[ 43.455746] sd 3:0:0:0: [sdv] Mode Sense: 00 3a 00 00
[ 43.455878] sd 3:0:0:0: [sdv] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 43.456567] sdv: sdv1
[ 43.457047] sd 3:0:0:0: [sdv] Attached SCSI disk
[ 43.766815] ata3: SATA link down (SStatus 0 SControl 300)
[ 44.078775] ata4: SATA link down (SStatus 0 SControl 300)
[ 44.390759] ata5: SATA link down (SStatus 0 SControl 300)
[ 44.410799] mlx4_core 0000:05:00.0: Enabling SR-IOV with 8 VFs
[ 44.512772] pci 0000:05:00.1: [15b3:1002] type 00 class 0x0c0600
[ 44.513052] pci 0000:05:00.1: Max Payload Size set to 256 (was 128, max 256)
[ 44.513520] iommu: Adding device 0000:05:00.1 to group 44
[ 44.513722] mlx4_core: Initializing 0000:05:00.1
[ 44.513891] mlx4_core 0000:05:00.1: enabling device (0000 -> 0002)
[ 44.514081] mlx4_core 0000:05:00.1: Skipping virtual function:1
[ 44.514332] pci 0000:05:00.2: [15b3:1002] type 00 class 0x0c0600
[ 44.514604] pci 0000:05:00.2: Max Payload Size set to 256 (was 128, max 256)
[ 44.515047] iommu: Adding device 0000:05:00.2 to group 45
[ 44.515225] mlx4_core: Initializing 0000:05:00.2
[ 44.515388] mlx4_core 0000:05:00.2: enabling device (0000 -> 0002)
[ 44.515572] mlx4_core 0000:05:00.2: Skipping virtual function:2
[ 44.515819] pci 0000:05:00.3: [15b3:1002] type 00 class 0x0c0600
[ 44.516092] pci 0000:05:00.3: Max Payload Size set to 256 (was 128, max 256)
[ 44.516521] iommu: Adding device 0000:05:00.3 to group 46
[ 44.516700] mlx4_core: Initializing 0000:05:00.3
[ 44.516870] mlx4_core 0000:05:00.3: enabling device (0000 -> 0002)
[ 44.517061] mlx4_core 0000:05:00.3: Skipping virtual function:3
[ 44.517310] pci 0000:05:00.4: [15b3:1002] type 00 class 0x0c0600
[ 44.517583] pci 0000:05:00.4: Max Payload Size set to 256 (was 128, max 256)
[ 44.518020] iommu: Adding device 0000:05:00.4 to group 47
[ 44.518197] mlx4_core: Initializing 0000:05:00.4
[ 44.518365] mlx4_core 0000:05:00.4: enabling device (0000 -> 0002)
[ 44.518554] mlx4_core 0000:05:00.4: Skipping virtual function:4
[ 44.518806] pci 0000:05:00.5: [15b3:1002] type 00 class 0x0c0600
[ 44.519078] pci 0000:05:00.5: Max Payload Size set to 256 (was 128, max 256)
[ 44.519514] iommu: Adding device 0000:05:00.5 to group 48
[ 44.519698] mlx4_core: Initializing 0000:05:00.5
[ 44.519865] mlx4_core 0000:05:00.5: enabling device (0000 -> 0002)
[ 44.520052] mlx4_core 0000:05:00.5: Skipping virtual function:5
[ 44.520297] pci 0000:05:00.6: [15b3:1002] type 00 class 0x0c0600
[ 44.520568] pci 0000:05:00.6: Max Payload Size set to 256 (was 128, max 256)
[ 44.521007] iommu: Adding device 0000:05:00.6 to group 49
[ 44.521186] mlx4_core: Initializing 0000:05:00.6
[ 44.521354] mlx4_core 0000:05:00.6: enabling device (0000 -> 0002)
[ 44.521543] mlx4_core 0000:05:00.6: Skipping virtual function:6
[ 44.521797] pci 0000:05:00.7: [15b3:1002] type 00 class 0x0c0600
[ 44.522069] pci 0000:05:00.7: Max Payload Size set to 256 (was 128, max 256)
[ 44.522504] iommu: Adding device 0000:05:00.7 to group 50
[ 44.522688] mlx4_core: Initializing 0000:05:00.7
[ 44.522858] mlx4_core 0000:05:00.7: enabling device (0000 -> 0002)
[ 44.523047] mlx4_core 0000:05:00.7: Skipping virtual function:7
[ 44.523297] pci 0000:05:01.0: [15b3:1002] type 00 class 0x0c0600
[ 44.523570] pci 0000:05:01.0: Max Payload Size set to 256 (was 128, max 256)
[ 44.524007] iommu: Adding device 0000:05:01.0 to group 51
[ 44.524194] mlx4_core: Initializing 0000:05:01.0
[ 44.524363] mlx4_core 0000:05:01.0: enabling device (0000 -> 0002)
[ 44.524554] mlx4_core 0000:05:01.0: Skipping virtual function:8
[ 44.524746] mlx4_core 0000:05:00.0: Running in master mode
[ 44.702999] ata6: SATA link down (SStatus 0 SControl 300)
[ 46.867330] mlx4_core 0000:05:00.0: PCIe link speed is 5.0GT/s, device supports 5.0GT/s
[ 46.867613] mlx4_core 0000:05:00.0: PCIe link width is x8, device supports x8
[ 46.910736] mlx4_core: Initializing 0000:05:00.1
[ 46.910913] mlx4_core 0000:05:00.1: enabling device (0000 -> 0002)
[ 46.911102] mlx4_core 0000:05:00.1: Skipping virtual function:1
[ 46.911380] mlx4_core: Initializing 0000:05:00.2
[ 46.911560] mlx4_core 0000:05:00.2: enabling device (0000 -> 0002)
[ 46.911751] mlx4_core 0000:05:00.2: Skipping virtual function:2
[ 46.912031] mlx4_core: Initializing 0000:05:00.3
[ 46.912203] mlx4_core 0000:05:00.3: enabling device (0000 -> 0002)
[ 46.912390] mlx4_core 0000:05:00.3: Skipping virtual function:3
[ 46.912650] mlx4_core: Initializing 0000:05:00.4
[ 46.912823] mlx4_core 0000:05:00.4: enabling device (0000 -> 0002)
[ 46.913011] mlx4_core 0000:05:00.4: Skipping virtual function:4
[ 46.913260] mlx4_core: Initializing 0000:05:00.5
[ 46.913432] mlx4_core 0000:05:00.5: enabling device (0000 -> 0002)
[ 46.913625] mlx4_core 0000:05:00.5: Skipping virtual function:5
[ 46.913869] mlx4_core: Initializing 0000:05:00.6
[ 46.914041] mlx4_core 0000:05:00.6: enabling device (0000 -> 0002)
[ 46.914227] mlx4_core 0000:05:00.6: Skipping virtual function:6
[ 46.914472] mlx4_core: Initializing 0000:05:00.7
[ 46.914649] mlx4_core 0000:05:00.7: enabling device (0000 -> 0002)
[ 46.914836] mlx4_core 0000:05:00.7: Skipping virtual function:7
[ 46.915085] mlx4_core: Initializing 0000:05:01.0
[ 46.915257] mlx4_core 0000:05:01.0: enabling device (0000 -> 0002)
[ 46.915440] mlx4_core 0000:05:01.0: Skipping virtual function:8
[ 46.915738] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[ 46.916303] DMAR: 32bit 0000:86:00.0 uses non-identity mapping
[ 46.916756] r8169 0000:86:00.0 eth0: RTL8168evl/8111evl at 0xffffc90006b6c000, 00:13:3b:0f:12:11, XID 0c900800 IRQ 52
[ 46.917084] r8169 0000:86:00.0 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko]
[ 46.917409] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[ 46.917933] DMAR: 32bit 0000:87:00.0 uses non-identity mapping
[ 46.918389] r8169 0000:87:00.0 eth1: RTL8168evl/8111evl at 0xffffc90006b6e000, 00:13:3b:0f:12:12, XID 0c900800 IRQ 53
[ 46.918718] r8169 0000:87:00.0 eth1: jumbo features [frames: 9200 bytes, tx checksumming: ko]
[ 46.919706] VFIO - User Level meta-driver version: 0.3
[ 46.920949] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 46.921145] ehci-pci: EHCI PCI platform driver
[ 46.921514] ehci-pci 0000:00:1a.7: EHCI Host Controller
[ 46.921787] ehci-pci 0000:00:1a.7: new USB bus registered, assigned bus number 1
[ 46.922076] ehci-pci 0000:00:1a.7: debug port 1
[ 46.922244] DMAR: 32bit 0000:00:1a.7 uses non-identity mapping
[ 46.922554] DMAR: Setting identity map for device 0000:00:1a.7 [0xbf7ec000 - 0xbf7fffff]
[ 46.922847] DMAR: Setting identity map for device 0000:00:1a.7 [0xec000 - 0xeffff]
[ 46.927052] ehci-pci 0000:00:1a.7: cache line size of 256 is not supported
[ 46.927061] ehci-pci 0000:00:1a.7: irq 18, io mem 0xfbeda000
[ 46.933555] ehci-pci 0000:00:1a.7: USB 2.0 started, EHCI 1.00
[ 46.933761] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[ 46.933926] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 46.934202] usb usb1: Product: EHCI Host Controller
[ 46.934364] usb usb1: Manufacturer: Linux 4.8.1-gentoo ehci_hcd
[ 46.934526] usb usb1: SerialNumber: 0000:00:1a.7
[ 46.934859] hub 1-0:1.0: USB hub found
[ 46.935028] hub 1-0:1.0: 6 ports detected
[ 46.935513] ehci-pci 0000:00:1d.7: EHCI Host Controller
[ 46.935775] ehci-pci 0000:00:1d.7: new USB bus registered, assigned bus number 2
[ 46.936065] ehci-pci 0000:00:1d.7: debug port 1
[ 46.936232] DMAR: 32bit 0000:00:1d.7 uses non-identity mapping
[ 46.936540] DMAR: Setting identity map for device 0000:00:1d.7 [0xbf7ec000 - 0xbf7fffff]
[ 46.936841] DMAR: Setting identity map for device 0000:00:1d.7 [0xec000 - 0xeffff]
[ 46.941029] ehci-pci 0000:00:1d.7: cache line size of 256 is not supported
[ 46.941039] ehci-pci 0000:00:1d.7: irq 23, io mem 0xfbed8000
[ 46.947555] ehci-pci 0000:00:1d.7: USB 2.0 started, EHCI 1.00
[ 46.950253] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
[ 46.950417] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 46.950695] usb usb2: Product: EHCI Host Controller
[ 46.950856] usb usb2: Manufacturer: Linux 4.8.1-gentoo ehci_hcd
[ 46.951019] usb usb2: SerialNumber: 0000:00:1d.7
[ 46.951353] hub 2-0:1.0: USB hub found
[ 46.951518] hub 2-0:1.0: 6 ports detected
[ 46.951869] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[ 46.952068] ohci-pci: OHCI PCI platform driver
[ 46.952267] uhci_hcd: USB Universal Host Controller Interface driver
[ 46.952607] uhci_hcd 0000:00:1a.0: UHCI Host Controller
[ 46.952866] uhci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 3
[ 46.953169] uhci_hcd 0000:00:1a.0: irq 16, io base 0x00008f80
[ 46.953336] DMAR: 32bit 0000:00:1a.0 uses non-identity mapping
[ 46.953654] DMAR: Setting identity map for device 0000:00:1a.0 [0xbf7ec000 - 0xbf7fffff]
[ 46.953943] DMAR: Setting identity map for device 0000:00:1a.0 [0xec000 - 0xeffff]
[ 46.954272] usb usb3: New USB device found, idVendor=1d6b, idProduct=0001
[ 46.954436] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 46.954715] usb usb3: Product: UHCI Host Controller
[ 46.954876] usb usb3: Manufacturer: Linux 4.8.1-gentoo uhci_hcd
[ 46.955040] usb usb3: SerialNumber: 0000:00:1a.0
[ 46.955362] hub 3-0:1.0: USB hub found
[ 46.955530] hub 3-0:1.0: 2 ports detected
[ 46.955962] uhci_hcd 0000:00:1a.1: UHCI Host Controller
[ 46.956222] uhci_hcd 0000:00:1a.1: new USB bus registered, assigned bus number 4
[ 46.956528] uhci_hcd 0000:00:1a.1: irq 21, io base 0x00008f40
[ 46.956700] DMAR: 32bit 0000:00:1a.1 uses non-identity mapping
[ 46.957015] DMAR: Setting identity map for device 0000:00:1a.1 [0xbf7ec000 - 0xbf7fffff]
[ 46.957311] DMAR: Setting identity map for device 0000:00:1a.1 [0xec000 - 0xeffff]
[ 46.957643] usb usb4: New USB device found, idVendor=1d6b, idProduct=0001
[ 46.957809] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 46.958083] usb usb4: Product: UHCI Host Controller
[ 46.958238] usb usb4: Manufacturer: Linux 4.8.1-gentoo uhci_hcd
[ 46.958395] usb usb4: SerialNumber: 0000:00:1a.1
[ 46.958680] hub 4-0:1.0: USB hub found
[ 46.958837] hub 4-0:1.0: 2 ports detected
[ 46.959254] uhci_hcd 0000:00:1a.2: UHCI Host Controller
[ 46.959506] uhci_hcd 0000:00:1a.2: new USB bus registered, assigned bus number 5
[ 46.959807] uhci_hcd 0000:00:1a.2: irq 19, io base 0x00008f20
[ 46.959971] DMAR: 32bit 0000:00:1a.2 uses non-identity mapping
[ 46.960282] DMAR: Setting identity map for device 0000:00:1a.2 [0xbf7ec000 - 0xbf7fffff]
[ 46.960578] DMAR: Setting identity map for device 0000:00:1a.2 [0xec000 - 0xeffff]
[ 46.960912] usb usb5: New USB device found, idVendor=1d6b, idProduct=0001
[ 46.961078] usb usb5: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 46.961353] usb usb5: Product: UHCI Host Controller
[ 46.961513] usb usb5: Manufacturer: Linux 4.8.1-gentoo uhci_hcd
[ 46.961678] usb usb5: SerialNumber: 0000:00:1a.2
[ 46.961997] hub 5-0:1.0: USB hub found
[ 46.962163] hub 5-0:1.0: 2 ports detected
[ 46.962586] uhci_hcd 0000:00:1d.0: UHCI Host Controller
[ 46.962847] uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 6
[ 46.963149] uhci_hcd 0000:00:1d.0: irq 23, io base 0x00008f00
[ 46.963316] DMAR: 32bit 0000:00:1d.0 uses non-identity mapping
[ 46.963633] DMAR: Setting identity map for device 0000:00:1d.0 [0xbf7ec000 - 0xbf7fffff]
[ 46.963924] DMAR: Setting identity map for device 0000:00:1d.0 [0xec000 - 0xeffff]
[ 46.964249] usb usb6: New USB device found, idVendor=1d6b, idProduct=0001
[ 46.964413] usb usb6: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 46.964691] usb usb6: Product: UHCI Host Controller
[ 46.964853] usb usb6: Manufacturer: Linux 4.8.1-gentoo uhci_hcd
[ 46.965016] usb usb6: SerialNumber: 0000:00:1d.0
[ 46.965337] hub 6-0:1.0: USB hub found
[ 46.965505] hub 6-0:1.0: 2 ports detected
[ 46.965932] uhci_hcd 0000:00:1d.1: UHCI Host Controller
[ 46.966183] uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 7
[ 46.966484] uhci_hcd 0000:00:1d.1: irq 19, io base 0x00008ec0
[ 46.966657] DMAR: 32bit 0000:00:1d.1 uses non-identity mapping
[ 46.966975] DMAR: Setting identity map for device 0000:00:1d.1 [0xbf7ec000 - 0xbf7fffff]
[ 46.967263] DMAR: Setting identity map for device 0000:00:1d.1 [0xec000 - 0xeffff]
[ 46.967597] usb usb7: New USB device found, idVendor=1d6b, idProduct=0001
[ 46.967762] usb usb7: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 46.968040] usb usb7: Product: UHCI Host Controller
[ 46.968202] usb usb7: Manufacturer: Linux 4.8.1-gentoo uhci_hcd
[ 46.968365] usb usb7: SerialNumber: 0000:00:1d.1
[ 46.968650] hub 7-0:1.0: USB hub found
[ 46.968811] hub 7-0:1.0: 2 ports detected
[ 46.969231] uhci_hcd 0000:00:1d.2: UHCI Host Controller
[ 46.969490] uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 8
[ 46.969796] uhci_hcd 0000:00:1d.2: irq 18, io base 0x00008ea0
[ 46.969963] DMAR: 32bit 0000:00:1d.2 uses non-identity mapping
[ 46.970279] DMAR: Setting identity map for device 0000:00:1d.2 [0xbf7ec000 - 0xbf7fffff]
[ 46.970570] DMAR: Setting identity map for device 0000:00:1d.2 [0xec000 - 0xeffff]
[ 46.970899] usb usb8: New USB device found, idVendor=1d6b, idProduct=0001
[ 46.971065] usb usb8: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 46.971337] usb usb8: Product: UHCI Host Controller
[ 46.971499] usb usb8: Manufacturer: Linux 4.8.1-gentoo uhci_hcd
[ 46.971664] usb usb8: SerialNumber: 0000:00:1d.2
[ 46.971987] hub 8-0:1.0: USB hub found
[ 46.972155] hub 8-0:1.0: 2 ports detected
[ 46.972497] usbcore: registered new interface driver usblp
[ 46.972729] usbcore: registered new interface driver usb-storage
[ 46.972943] i8042: PNP: No PS/2 controller found. Probing ports directly.
[ 46.975869] serio: i8042 KBD port at 0x60,0x64 irq 1
[ 46.976057] serio: i8042 AUX port at 0x60,0x64 irq 12
[ 46.976469] mousedev: PS/2 mouse device common for all mice
[ 46.977274] rtc_cmos 00:01: RTC can wake from S4
[ 46.977704] rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0
[ 46.977925] rtc_cmos 00:01: alarms up to one month, y3k, 114 bytes nvram, hpet irqs
[ 46.978408] i801_smbus 0000:00:1f.3: SMBus using PCI interrupt
[ 46.978664] md: linear personality registered for level -1
[ 46.978857] md: raid0 personality registered for level 0
[ 46.979041] md: raid1 personality registered for level 1
[ 46.979225] md: raid10 personality registered for level 10
[ 46.979565] md: raid6 personality registered for level 6
[ 46.979756] md: raid5 personality registered for level 5
[ 46.979940] md: raid4 personality registered for level 4
[ 46.980124] md: multipath personality registered for level -4
[ 46.980309] md: faulty personality registered for level -5
[ 46.980678] device-mapper: ioctl: 4.35.0-ioctl (2016-06-23) initialised: dm-devel@redhat.com
[ 46.981388] device-mapper: multipath round-robin: version 1.1.0 loaded
[ 46.981584] device-mapper: multipath queue-length: version 0.2.0 loaded
[ 46.981772] device-mapper: multipath service-time: version 0.3.0 loaded
[ 46.981968] device-mapper: dm-log-userspace: version 1.3.0 loaded
[ 46.982156] device-mapper: raid: Loading target version 1.9.0
[ 46.984417] <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v2.2-1 (Feb 2014)
[ 46.985031] <mlx4_ib> mlx4_ib_add: counter index 0 for port 1 allocated 0
[ 47.025669] mlx4_core 0000:05:00.0: mlx4_ib: multi-function enabled
[ 47.027665] mlx4_core 0000:05:00.0: mlx4_ib: initializing demux service for 128 qp1 clients
[ 47.045355] hidraw: raw HID events driver (C) Jiri Kosina
[ 47.045784] usbcore: registered new interface driver usbhid
[ 47.045969] usbhid: USB HID core driver
[ 47.046700] snd_hda_intel 0000:88:00.1: Disabling MSI
[ 47.046971] Netfilter messages via NETLINK v0.30.
[ 47.047189] DMAR: 32bit 0000:88:00.1 uses non-identity mapping
[ 47.047752] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
[ 47.047996] ctnetlink v0.93: registering with nfnetlink.
[ 47.048263] ip_tables: (C) 2000-2006 Netfilter Core Team
[ 47.048521] Initializing XFRM netlink socket
[ 47.048851] NET: Registered protocol family 10
[ 47.049474] ip6_tables: (C) 2000-2006 Netfilter Core Team
[ 47.049800] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[ 47.050110] NET: Registered protocol family 17
[ 47.050301] bridge: automatic filtering via arp/ip/ip6tables has been deprecated. Update your scripts to load br_netfilter if you need this.
[ 47.050806] RPC: Registered rdma transport module.
[ 47.050992] RPC: Registered rdma backchannel transport module.
[ 47.051197] Key type dns_resolver registered
[ 47.052110] microcode: sig=0x206c2, pf=0x1, revision=0x14
[ 47.053402] microcode: Microcode Update Driver: v2.01 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
[ 47.054007] registered taskstats version 1
[ 47.055195] Magic number: 12:655:137
[ 47.055372] usb usb4-port2: hash matches
[ 47.055639] block ram2: hash matches
[ 47.055904] console [netcon0] enabled
[ 47.056063] netconsole: network logging started
[ 47.063309] PM: Hibernation image not present or could not be loaded.
[ 47.063310] ALSA device list:
[ 47.063468] No soundcards found.
[ 47.067144] Freeing unused kernel memory: 1172K (ffffffff82159000 - ffffffff8227e000)
[ 47.067421] Write protecting the kernel read-only data: 16384k
[ 47.072250] Freeing unused kernel memory: 1004K (ffff880001b05000 - ffff880001c00000)
[ 47.078084] Freeing unused kernel memory: 520K (ffff880001f7e000 - ffff880002000000)
[ 47.306702] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:80/0000:80:07.0/0000:88:00.1/sound/card0/input5
[ 47.307265] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:80/0000:80:07.0/0000:88:00.1/sound/card0/input6
[ 47.307714] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:80/0000:80:07.0/0000:88:00.1/sound/card0/input7
[ 47.308186] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:80/0000:80:07.0/0000:88:00.1/sound/card0/input8
[ 47.356442] ata_id (3517) used greatest stack depth: 13832 bytes left
[ 47.432532] usb 4-2: new full-speed USB device number 2 using uhci_hcd
[ 47.449530] usb 6-1: new full-speed USB device number 2 using uhci_hcd
[ 47.599622] usb 4-2: New USB device found, idVendor=046b, idProduct=ff10
[ 47.599795] usb 4-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 47.600077] usb 4-2: Product: Virtual Keyboard and Mouse
[ 47.600251] usb 4-2: Manufacturer: American Megatrends Inc.
[ 47.600493] usb 4-2: SerialNumber: serial
[ 47.610582] usb 6-1: New USB device found, idVendor=051d, idProduct=0003
[ 47.610752] usb 6-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 47.610917] usb 6-1: Product: Smart-UPS X 1500 FW:UPS 09.1 / ID=20
[ 47.611082] usb 6-1: Manufacturer: American Power Conversion
[ 47.611245] usb 6-1: SerialNumber: AS1220224995
[ 47.641886] input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1a.1/usb4/4-2/4-2:1.0/0003:046B:FF10.0002/input/input9
[ 47.651900] hid-generic 0003:051D:0003.0001: hiddev0,hidraw0: USB HID v1.00 Device [American Power Conversion Smart-UPS X 1500 FW:UPS 09.1 / ID=20] on usb-0000:00:1d.0-1/input0
[ 47.694101] hid-generic 0003:046B:FF10.0002: input,hidraw1: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.1-2/input0
[ 47.700860] input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1a.1/usb4/4-2/4-2:1.1/0003:046B:FF10.0003/input/input10
[ 47.701719] hid-generic 0003:046B:FF10.0003: input,hidraw2: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.1-2/input1
[ 47.863575] usb 6-2: new full-speed USB device number 3 using uhci_hcd
[ 48.016592] usb 6-2: New USB device found, idVendor=1532, idProduct=0118
[ 48.016759] usb 6-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 48.016925] usb 6-2: Product: Razer DeathStalker
[ 48.017088] usb 6-2: Manufacturer: Razer
[ 48.020846] input: Razer Razer DeathStalker as /devices/pci0000:00/0000:00:1d.0/usb6/6-2/6-2:1.0/0003:1532:0118.0004/input/input11
[ 48.073039] hid-generic 0003:1532:0118.0004: input,hidraw3: USB HID v1.11 Keyboard [Razer Razer DeathStalker] on usb-0000:00:1d.0-2/input0
[ 48.080736] input: Razer Razer DeathStalker as /devices/pci0000:00/0000:00:1d.0/usb6/6-2/6-2:1.1/0003:1532:0118.0005/input/input12
[ 48.132948] hid-generic 0003:1532:0118.0005: input,hidraw4: USB HID v1.11 Keyboard [Razer Razer DeathStalker] on usb-0000:00:1d.0-2/input1
[ 48.137655] input: Razer Razer DeathStalker as /devices/pci0000:00/0000:00:1d.0/usb6/6-2/6-2:1.2/0003:1532:0118.0006/input/input13
[ 48.138509] hid-generic 0003:1532:0118.0006: input,hidraw5: USB HID v1.11 Mouse [Razer Razer DeathStalker] on usb-0000:00:1d.0-2/input2
[ 48.207307] systemd-udevd (2776) used greatest stack depth: 13744 bytes left
[ 48.207559] systemd-udevd (3278) used greatest stack depth: 13688 bytes left
[ 48.562640] spl: loading out-of-tree module taints kernel.
[ 48.565838] SPL: Loaded module v0.6.5.8-r0-gentoo
[ 48.566059] znvpair: module license 'CDDL' taints kernel.
[ 48.566220] Disabling lock debugging due to kernel taint
[ 48.598546] ZFS: Loaded module v0.6.5.8-r0-gentoo, ZFS pool version 5000, ZFS filesystem version 5
[ 48.598993] modprobe (4709) used greatest stack depth: 13680 bytes left
[ 48.701318] md: md127 stopped.
[ 48.702106] md: bind<sdu1>
[ 48.702403] md: bind<sdv1>
[ 48.703056] md/raid1:md127: active with 2 out of 2 mirrors
[ 48.703234] md127: detected capacity change from 0 to 59987853312
[ 53.367532] mdadm (5036) used greatest stack depth: 13216 bytes left
[ 53.540973] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null)
[ 53.639115] init-early.sh (5131) used greatest stack depth: 12696 bytes left
[ 54.111029] i5500_temp 0000:00:14.3: Sensor seems to be disabled
[ 54.111085] i5500_temp 0000:80:14.3: Sensor seems to be disabled
[ 54.128981] r8169 0000:86:00.0 enp134s0: renamed from eth0
[ 54.146432] r8169 0000:87:00.0 enp135s0: renamed from eth1
[ 55.465194] EXT4-fs (dm-0): re-mounted. Opts: discard
[ 55.611294] Adding 4194300k swap on /dev/mapper/local_mirror-swap. Priority:-1 extents:1 across:4194300k Dsc
[ 56.425131] SPL: using hostid 0x00000000
[ 57.322488] dmu_objset_find (7192) used greatest stack depth: 12648 bytes left
[ 57.322867] dmu_objset_find (7196) used greatest stack depth: 12488 bytes left
[ 58.291003] zpool (5900) used greatest stack depth: 10960 bytes left
[ 61.330973] IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready
[ 61.519463] device lan_br entered promiscuous mode
[ 63.740350] device wan_br entered promiscuous mode
[ 67.497627] NFSD: starting 90-second grace period (net ffffffff820f88c0)
[ 69.111836] lan_br: port 1(vnet0) entered blocking state
[ 69.111838] lan_br: port 1(vnet0) entered disabled state
[ 69.111900] device vnet0 entered promiscuous mode
[ 69.119856] lan_br: port 1(vnet0) entered blocking state
[ 69.119858] lan_br: port 1(vnet0) entered forwarding state
[ 69.139552] cgroup: libvirtd (9216) created nested cgroup for controller "memory" which has incomplete hierarchy support. Nested cgroups may change behavior in the future.
[ 69.139553] cgroup: "memory" requires setting use_hierarchy to 1 on the root
[ 69.770699] wan_br: port 1(vnet1) entered blocking state
[ 69.770700] wan_br: port 1(vnet1) entered disabled state
[ 69.770771] device vnet1 entered promiscuous mode
[ 69.778742] wan_br: port 1(vnet1) entered blocking state
[ 69.778743] wan_br: port 1(vnet1) entered forwarding state
[ 69.822764] lan_br: port 2(vnet2) entered blocking state
[ 69.822766] lan_br: port 2(vnet2) entered disabled state
[ 69.822822] device vnet2 entered promiscuous mode
[ 69.830794] lan_br: port 2(vnet2) entered blocking state
[ 69.830795] lan_br: port 2(vnet2) entered forwarding state
[ 69.892907] vfio_iommu_type1_attach_group: No interrupt remapping support. Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
[ 69.899949] lan_br: port 2(vnet2) entered disabled state
[ 69.900871] device vnet2 left promiscuous mode
[ 69.900872] lan_br: port 2(vnet2) entered disabled state
[ 69.918830] wan_br: port 1(vnet1) entered disabled state
[ 69.919814] device vnet1 left promiscuous mode
[ 69.919815] wan_br: port 1(vnet1) entered disabled state
[ 69.921410] kvm: zapping shadow pages for mmio generation wraparound
[ 69.926547] kvm: zapping shadow pages for mmio generation wraparound
[ 69.945921] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[ 69.946752] r8169 0000:87:00.0 eth0: RTL8168evl/8111evl at 0xffffc9001e42e000, 00:13:3b:0f:12:12, XID 0c900800 IRQ 53
[ 69.946753] r8169 0000:87:00.0 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko]
[ 69.947957] r8169 0000:87:00.0 enp135s0: renamed from eth0
[ 69.995773] wan_br: port 1(vnet1) entered blocking state
[ 69.995776] wan_br: port 1(vnet1) entered disabled state
[ 69.995855] device vnet1 entered promiscuous mode
[ 70.003826] wan_br: port 1(vnet1) entered blocking state
[ 70.003828] wan_br: port 1(vnet1) entered forwarding state
[ 70.155120] vfio_iommu_type1_attach_group: No interrupt remapping support. Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
[ 70.162880] wan_br: port 1(vnet1) entered disabled state
[ 70.163827] device vnet1 left promiscuous mode
[ 70.163828] wan_br: port 1(vnet1) entered disabled state
[ 70.260753] wan_br: port 1(vnet1) entered blocking state
[ 70.260755] wan_br: port 1(vnet1) entered disabled state
[ 70.260812] device vnet1 entered promiscuous mode
[ 70.272723] wan_br: port 1(vnet1) entered blocking state
[ 70.272724] wan_br: port 1(vnet1) entered forwarding state
[ 70.396500] vfio_iommu_type1_attach_group: No interrupt remapping support. Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
[ 70.403874] wan_br: port 1(vnet1) entered disabled state
[ 70.404945] device vnet1 left promiscuous mode
[ 70.404948] wan_br: port 1(vnet1) entered disabled state
[ 70.427821] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[ 70.428546] r8169 0000:86:00.0 eth0: RTL8168evl/8111evl at 0xffffc9001edfe000, 00:13:3b:0f:12:11, XID 0c900800 IRQ 52
[ 70.428548] r8169 0000:86:00.0 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko]
[ 70.429687] r8169 0000:86:00.0 enp134s0: renamed from eth0
[ 70.588762] wan_br: port 1(vnet1) entered blocking state
[ 70.588765] wan_br: port 1(vnet1) entered disabled state
[ 70.588840] device vnet1 entered promiscuous mode
[ 70.596816] wan_br: port 1(vnet1) entered blocking state
[ 70.596818] wan_br: port 1(vnet1) entered forwarding state
[ 70.663854] vfio_iommu_type1_attach_group: No interrupt remapping support. Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
[ 70.671872] wan_br: port 1(vnet1) entered disabled state
[ 70.672800] device vnet1 left promiscuous mode
[ 70.672803] wan_br: port 1(vnet1) entered disabled state
[ 70.787715] wan_br: port 1(vnet1) entered blocking state
[ 70.787717] wan_br: port 1(vnet1) entered disabled state
[ 70.787773] device vnet1 entered promiscuous mode
[ 70.795759] wan_br: port 1(vnet1) entered blocking state
[ 70.795760] wan_br: port 1(vnet1) entered forwarding state
[ 70.837715] lan_br: port 2(vnet2) entered blocking state
[ 70.837716] lan_br: port 2(vnet2) entered disabled state
[ 70.837770] device vnet2 entered promiscuous mode
[ 70.845748] lan_br: port 2(vnet2) entered blocking state
[ 70.845750] lan_br: port 2(vnet2) entered forwarding state
[ 70.907353] vfio_iommu_type1_attach_group: No interrupt remapping support. Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
[ 70.913879] lan_br: port 2(vnet2) entered disabled state
[ 70.914795] device vnet2 left promiscuous mode
[ 70.914799] lan_br: port 2(vnet2) entered disabled state
[ 70.936846] wan_br: port 1(vnet1) entered disabled state
[ 70.937789] device vnet1 left promiscuous mode
[ 70.937791] wan_br: port 1(vnet1) entered disabled state
[ 70.962807] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[ 70.963553] r8169 0000:87:00.0 eth0: RTL8168evl/8111evl at 0xffffc9001f2e4000, 00:13:3b:0f:12:12, XID 0c900800 IRQ 53
[ 70.963555] r8169 0000:87:00.0 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko]
[ 70.964731] r8169 0000:87:00.0 enp135s0: renamed from eth0
[ 71.126707] wan_br: port 1(vnet1) entered blocking state
[ 71.126708] wan_br: port 1(vnet1) entered disabled state
[ 71.126763] device vnet1 entered promiscuous mode
[ 71.134744] wan_br: port 1(vnet1) entered blocking state
[ 71.134746] wan_br: port 1(vnet1) entered forwarding state
[ 71.196894] vfio_iommu_type1_attach_group: No interrupt remapping support. Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
[ 71.203893] wan_br: port 1(vnet1) entered disabled state
[ 71.204808] device vnet1 left promiscuous mode
[ 71.204810] wan_br: port 1(vnet1) entered disabled state
[ 71.234791] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[ 71.235528] r8169 0000:86:00.0 eth0: RTL8168evl/8111evl at 0xffffc9001f51e000, 00:13:3b:0f:12:11, XID 0c900800 IRQ 52
[ 71.235529] r8169 0000:86:00.0 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko]
[ 71.236561] r8169 0000:86:00.0 enp134s0: renamed from eth0
[ 71.631199] DMAR: DRHD: handling fault status reg 2
[ 71.631204] DMAR: [DMA Read] Request device [05:06.1] fault addr c2652b000 [fault reason 02] Present bit in context entry is clear
[ 72.020267] ib0: enabling connected mode will cause multicast packet drops
[ 72.020307] ib0: mtu > 2044 will cause multicast packet drops.
[ 103.134429] DMAR: DRHD: handling fault status reg 102
[ 103.134434] DMAR: [DMA Read] Request device [05:06.1] fault addr 81b081000 [fault reason 02] Present bit in context entry is clear
[ 105.135927] DMAR: DRHD: handling fault status reg 202
[ 105.136013] DMAR: [DMA Read] Request device [05:06.1] fault addr 81b081000 [fault reason 02] Present bit in context entry is clear
[ 107.137479] DMAR: DRHD: handling fault status reg 302
[ 107.137484] DMAR: [DMA Read] Request device [05:06.1] fault addr 81b081000 [fault reason 02] Present bit in context entry is clear
^ permalink raw reply
* Re: [bug report] IB/mlx5: Support IB_WR_REG_SIG_MR
From: Sagi Grimberg @ 2016-12-12 14:34 UTC (permalink / raw)
To: Leon Romanovsky, Dan Carpenter
Cc: sagig-VPRAkNaXOzVWk0Htik3J/w, linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20161212125205.GB6503-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org>
>>>> 3624 case IB_SIG_TYPE_T10_DIF:
>>>> 3625 psv_seg->transient_sig = cpu_to_be32(domain->sig.dif.bg << 16 |
>>>> 3626 domain->sig.dif.app_tag);
>>>> 3627 psv_seg->ref_tag = cpu_to_be32(domain->sig.dif.ref_tag);
>>>> 3628 break;
>>>> 3629 default:
>>>> 3630 pr_err("Bad signature type given.\n");
>>>> 3631 return 1;
>>>>
>>>> Why do we return 1 here instead of a negative error code?
>>>
>>> Yes, probably better to return something like EINVAL, instead of 1.
>>> do you want us to handle it?
>>>
>>
>> Yes, please.
>
> Thanks,
> We will send it as part of our fixes series for 4.10.
Thanks Leon. Thanks Dan for reporting.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 2/4] IB/core: export ib_get_cached_port_state
From: Jinpu Wang @ 2016-12-12 13:31 UTC (permalink / raw)
To: Doug Ledford, Hefty, Sean, Hal Rosenstock,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Michael Wang,
Jason Gunthorpe, Mark Bloch
In-Reply-To: <CAMGffEnCU-8sodNH_i-KKPAhe3ekkt+Yd34wiUVqvXJ+ujSRwQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
[-- Attachment #1: Type: text/plain, Size: 3460 bytes --]
Add Mark in reply.
Hi, Mark,
On Mon, Dec 12, 2016 at 12:45 PM, Jinpu Wang
<jinpu.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org> wrote:
> From 1cf770be635e987e31feed2f7c82b9f4f3bd316a Mon Sep 17 00:00:00 2001
> From: Jack Wang <jinpu.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>
> Date: Mon, 12 Dec 2016 10:12:39 +0100
> Subject: [PATCH 2/4] IB/core: export ib_get_cached_port_state
>
> Export function for rdma_cm, patch will follow.
>
> Signed-off-by: Jack Wang <jinpu.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>
> Reviewed-by: Michael Wang <yun.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>
> ---
> drivers/infiniband/core/cache.c | 18 ++++++++++++++++++
> include/rdma/ib_cache.h | 13 +++++++++++++
> 2 files changed, 31 insertions(+)
>
> diff --git a/drivers/infiniband/core/cache.c b/drivers/infiniband/core/cache.c
> index 025db27..1b064cf 100644
> --- a/drivers/infiniband/core/cache.c
> +++ b/drivers/infiniband/core/cache.c
> @@ -1025,6 +1025,24 @@ int ib_get_cached_lmc(struct ib_device *device,
> }
> EXPORT_SYMBOL(ib_get_cached_lmc);
>
> +int ib_get_cached_port_state(struct ib_device *device,
> + u8 port_num,
> + enum ib_port_state *port_state)
Q: Can you please fix the style (indentation)
A: format mangled by gmail, I attached the patch in attachment.
> +{
> + unsigned long flags;
> + int ret = 0;
> +
> + if (port_num < rdma_start_port(device) || port_num > rdma_end_port(device))
> + return -EINVAL;
> +
> + read_lock_irqsave(&device->cache.lock, flags);
> + *port_state = device->cache.port_state_cache[port_num -
> rdma_start_port(device)];
> + read_unlock_irqrestore(&device->cache.lock, flags);
> +
> + return ret;
> +}
> +EXPORT_SYMBOL(ib_get_cached_port_state);
> +
Q: Why do you need ret? just return 0.
A: We just need to keep the same style as ib_get_cached_lmc(), ret could be
reserved, indentation should be consistent if not.
> static void ib_cache_update(struct ib_device *device,
> u8 port)
> {
> diff --git a/include/rdma/ib_cache.h b/include/rdma/ib_cache.h
> index e30f19b..385ec88 100644
> --- a/include/rdma/ib_cache.h
> +++ b/include/rdma/ib_cache.h
> @@ -165,4 +165,17 @@ int ib_get_cached_lmc(struct ib_device *device,
> u8 port_num,
> u8 *lmc);
>
> +/**
> + * ib_get_cached_port_state - Returns a cached port state table entry
> + * @device: The device to query.
> + * @port_num: The port number of the device to query.
> + * @port_state: port_state for the specified port for that device.
> + *
> + * ib_get_cached_port_state() fetches the specified port_state table
> entry stored in
> + * the local software cache.
> + */
> +int ib_get_cached_port_state(struct ib_device *device,
> + u8 port_num,
> + enum ib_port_state *port_active);
> +
> #endif /* _IB_CACHE_H */
> --
> 2.7.4
>
>
Thanks!
--
Jinpu Wang
Linux Kernel Developer
ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin
Tel: +49 30 577 008 042
Fax: +49 30 577 008 299
Email: jinpu.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org
URL: https://www.profitbricks.de
Sitz der Gesellschaft: Berlin
Registergericht: Amtsgericht Charlottenburg, HRB 125506 B
Geschäftsführer: Achim Weiss
[-- Attachment #2: 0002-IB-core-export-ib_get_cached_port_state.patch --]
[-- Type: text/x-patch, Size: 2271 bytes --]
From 1cf770be635e987e31feed2f7c82b9f4f3bd316a Mon Sep 17 00:00:00 2001
From: Jack Wang <jinpu.wang@profitbricks.com>
Date: Mon, 12 Dec 2016 10:12:39 +0100
Subject: [PATCH 2/4] IB/core: export ib_get_cached_port_state
Export function for rdma_cm, patch will follow.
Signed-off-by: Jack Wang <jinpu.wang@profitbricks.com>
Reviewed-by: Michael Wang <yun.wang@profitbricks.com>
---
drivers/infiniband/core/cache.c | 18 ++++++++++++++++++
include/rdma/ib_cache.h | 13 +++++++++++++
2 files changed, 31 insertions(+)
diff --git a/drivers/infiniband/core/cache.c b/drivers/infiniband/core/cache.c
index 025db27..1b064cf 100644
--- a/drivers/infiniband/core/cache.c
+++ b/drivers/infiniband/core/cache.c
@@ -1025,6 +1025,24 @@ int ib_get_cached_lmc(struct ib_device *device,
}
EXPORT_SYMBOL(ib_get_cached_lmc);
+int ib_get_cached_port_state(struct ib_device *device,
+ u8 port_num,
+ enum ib_port_state *port_state)
+{
+ unsigned long flags;
+ int ret = 0;
+
+ if (port_num < rdma_start_port(device) || port_num > rdma_end_port(device))
+ return -EINVAL;
+
+ read_lock_irqsave(&device->cache.lock, flags);
+ *port_state = device->cache.port_state_cache[port_num - rdma_start_port(device)];
+ read_unlock_irqrestore(&device->cache.lock, flags);
+
+ return ret;
+}
+EXPORT_SYMBOL(ib_get_cached_port_state);
+
static void ib_cache_update(struct ib_device *device,
u8 port)
{
diff --git a/include/rdma/ib_cache.h b/include/rdma/ib_cache.h
index e30f19b..385ec88 100644
--- a/include/rdma/ib_cache.h
+++ b/include/rdma/ib_cache.h
@@ -165,4 +165,17 @@ int ib_get_cached_lmc(struct ib_device *device,
u8 port_num,
u8 *lmc);
+/**
+ * ib_get_cached_port_state - Returns a cached port state table entry
+ * @device: The device to query.
+ * @port_num: The port number of the device to query.
+ * @port_state: port_state for the specified port for that device.
+ *
+ * ib_get_cached_port_state() fetches the specified port_state table entry stored in
+ * the local software cache.
+ */
+int ib_get_cached_port_state(struct ib_device *device,
+ u8 port_num,
+ enum ib_port_state *port_active);
+
#endif /* _IB_CACHE_H */
--
2.7.4
^ permalink raw reply related
* Re: [bug report] IB/mlx5: Support IB_WR_REG_SIG_MR
From: Leon Romanovsky @ 2016-12-12 12:52 UTC (permalink / raw)
To: Dan Carpenter
Cc: sagig-VPRAkNaXOzVWk0Htik3J/w, linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20161212090910.GS8244@mwanda>
[-- Attachment #1: Type: text/plain, Size: 924 bytes --]
On Mon, Dec 12, 2016 at 12:33:55PM +0300, Dan Carpenter wrote:
> On Sun, Dec 11, 2016 at 11:23:26AM +0200, Leon Romanovsky wrote:
> > > 3624 case IB_SIG_TYPE_T10_DIF:
> > > 3625 psv_seg->transient_sig = cpu_to_be32(domain->sig.dif.bg << 16 |
> > > 3626 domain->sig.dif.app_tag);
> > > 3627 psv_seg->ref_tag = cpu_to_be32(domain->sig.dif.ref_tag);
> > > 3628 break;
> > > 3629 default:
> > > 3630 pr_err("Bad signature type given.\n");
> > > 3631 return 1;
> > >
> > > Why do we return 1 here instead of a negative error code?
> >
> > Yes, probably better to return something like EINVAL, instead of 1.
> > do you want us to handle it?
> >
>
> Yes, please.
Thanks,
We will send it as part of our fixes series for 4.10.
>
> regards,
> dan carpenter
>
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* [Bug 190141] New: v4.9: Running SRP on top of SoftRoCE driver triggers OOM
From: bugzilla-daemon-590EEB7GvNiWaY/ihj7yzEB+6BGkLq7r @ 2016-12-12 12:44 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA
https://bugzilla.kernel.org/show_bug.cgi?id=190141
Bug ID: 190141
Summary: v4.9: Running SRP on top of SoftRoCE driver triggers
OOM
Product: Drivers
Version: 2.5
Kernel Version: v4.9
Hardware: All
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Infiniband/RDMA
Assignee: drivers_infiniband-rdma-ztI5WcYan/vQLgFONoPN62D2FQJk+8+b@public.gmane.org
Reporter: bvanassche-HInyCGIudOg@public.gmane.org
Regression: No
Test description:
* Clone https://github.com/bvanassche/srp-test.
* Enable ib_srp, ib_srpt and rdma_rxe in the kernel config.
* Build and install kernel v4.9, e.g. in a virtual machine.
* Boot kernel v4.9 and run the following commands:
modprobe rdma_rxe
echo eth0 > /sys/module/rdma_rxe/parameters/add
cd srp-test
./run-tests
Result on the console:
gmain invoked oom-killer: gfp_mask=0x24201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD),
nodemask=0, order=0, oom_score_adj=0
Call Trace:
[<ffffffff812a4943>] dump_stack+0x86/0xc3
[<ffffffff811ac69f>] dump_header+0x76/0x24a
[<ffffffff8112a6ce>] oom_kill_process+0x1fe/0x3e0
[<ffffffff8112ab81>] out_of_memory+0x101/0x530
[<ffffffff8112ff4b>] __alloc_pages_slowpath+0xafb/0xb40
[<ffffffff8113021e>] __alloc_pages_nodemask+0x28e/0x2d0
[<ffffffff8117c81c>] alloc_pages_current+0x9c/0x1e0
[<ffffffff81124ee7>] __page_cache_alloc+0x137/0x160
[<ffffffff811284c3>] filemap_fault+0x323/0x5f0
[<ffffffffa0168551>] ext4_filemap_fault+0x31/0x50 [ext4]
[<ffffffff811557ad>] __do_fault+0x6d/0x150
[<ffffffff8115a0e5>] handle_mm_fault+0x1075/0x14b0
[<ffffffff8104d29b>] __do_page_fault+0x20b/0x460
[<ffffffff8104d542>] do_page_fault+0x22/0x30
[<ffffffff81048b44>] do_async_page_fault+0x14/0x70
[<ffffffff8165a5d8>] async_page_fault+0x28/0x30
Mem-Info:
active_anon:6056 inactive_anon:1337 isolated_anon:0
active_file:42 inactive_file:46 isolated_file:0
unevictable:2637 dirty:50 writeback:0 unstable:0
slab_reclaimable:3546 slab_unreclaimable:51680
mapped:2019 shmem:1419 pagetables:888 bounce:0
free:21709 free_pcp:90 free_cma:0
Node 0 active_anon:24224kB inactive_anon:5348kB active_file:168kB
inactive_file:184kB unevictable:10548kB isolated(anon):0kB isolated(file):0kB
mapped:8076kB dirty:200kB writeback:0kB shmem:0kB shmem_thp: 0kB
shmem_pmdmapped: 0kB anon_thp: 5676kB writeback_tmp:0kB unstable:0kB
pages_scanned:0 all_unreclaimable? no
Node 0 DMA free:15908kB min:264kB low:328kB high:392kB active_anon:0kB
inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB
writepending:0kB present:15992kB managed:15908kB mlocked:0kB
slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB
bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
lowmem_reserve[]: 0 2952 3911 3911
Node 0 DMA32 free:54640kB min:50812kB low:63512kB high:76212kB
active_anon:14684kB inactive_anon:92kB active_file:56kB inactive_file:116kB
unevictable:1880kB writepending:88kB present:3129200kB managed:1716180kB
mlocked:1880kB slab_reclaimable:5696kB slab_unreclaimable:127384kB
kernel_stack:6320kB pagetables:1904kB bounce:0kB free_pcp:240kB local_pcp:116kB
free_cma:0kB
Node 0 Normal free:16288kB min:16504kB low:20628kB high:24752kB
active_anon:9540kB inactive_anon:5256kB active_file:112kB inactive_file:68kB
unevictable:8668kB writepending:112kB present:1048576kB managed:208592kB
mlocked:8668kB slab_reclaimable:8488kB slab_unreclaimable:79336kB
kernel_stack:2048kB pagetables:1648kB bounce:0kB free_pcp:120kB local_pcp:0kB
free_cma:0kB
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB
(U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
Node 0 DMA32: 1398*4kB (M) 769*8kB (UME) 531*16kB (UME) 425*32kB (UME) 125*64kB
(UME) 18*128kB (UE) 5*256kB (UE) 6*512kB (UE) 4*1024kB (E) 1*2048kB (E)
0*4096kB = 54640kB
Node 0 Normal: 1046*4kB (UMH) 457*8kB (UMH) 246*16kB (UMEH) 129*32kB (UMEH)
6*64kB (UMH) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 16288kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0
hugepages_size=1048576kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0
hugepages_size=2048kB
3346 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
1048442 pages RAM
--
You are receiving this mail because:
You are watching the assignee of the bug.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 1/4] IB/core: add port state cache
From: Jinpu Wang @ 2016-12-12 12:31 UTC (permalink / raw)
To: Mark Bloch
Cc: Doug Ledford, Hefty, Sean, Hal Rosenstock,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Michael Wang,
Jason Gunthorpe
In-Reply-To: <6b60d6b1-96ce-3710-7716-0df4b1895c44-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
On Mon, Dec 12, 2016 at 1:11 PM, Mark Bloch <markb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:
> Hi,
>
> On 12/12/2016 1:43 PM, Jinpu Wang wrote:
>> From fc80c1730d94a38934c5c60f9b9561745723dfd9 Mon Sep 17 00:00:00 2001
>> From: Jack Wang <jinpu.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>
>> Date: Mon, 12 Dec 2016 09:52:42 +0100
>> Subject: [PATCH 1/4] IB/core: add port state cache
>>
>> We need a port state cache in ib_core, later we will use in rdma_cm.
>>
>> Signed-off-by: Jack Wang <jinpu.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>
>> Reviewed-by: Michael Wang <yun.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>
>> ---
>> drivers/infiniband/core/cache.c | 9 ++++++++-
>> include/rdma/ib_verbs.h | 1 +
>> 2 files changed, 9 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/infiniband/core/cache.c b/drivers/infiniband/core/cache.c
>> index 1a2984c..025db27 100644
>> --- a/drivers/infiniband/core/cache.c
>> +++ b/drivers/infiniband/core/cache.c
>> @@ -1109,6 +1109,8 @@ static void ib_cache_update(struct ib_device *device,
>> }
>>
>> device->cache.lmc_cache[port - rdma_start_port(device)] = tprops->lmc;
>> + device->cache.port_state_cache[port - rdma_start_port(device)] =
>> + tprops->state;
>>
>> write_unlock_irq(&device->cache.lock);
>>
>> @@ -1168,7 +1170,11 @@ int ib_cache_setup_one(struct ib_device *device)
>> (rdma_end_port(device) -
>> rdma_start_port(device) + 1),
>> GFP_KERNEL);
>> - if (!device->cache.pkey_cache ||
>> + device->cache.port_state_cache = kmalloc(sizeof
>> *device->cache.port_state_cache *
>> + (rdma_end_port(device) -
>> + rdma_start_port(device) + 1),
>> + GFP_KERNEL);
>> + if (!device->cache.pkey_cache || !device->cache.port_state_cache ||
> Looking at the code, I see we can leak memory here.
> if pkey_cache is allocated but lms_cache is not, we will return without freeing pkey_cache (or the other way around)
> your code adds port_state_cache to the mix but with the same issue.
> I guess that should be fixed.
>
Hi Mark,
Not really. The release was done by ib_dealloc_device() when put the
last ref of kobj,
dev_release -> ib_device_release -> ib_cache_release_one, driver was supposed
to use it to destroy the device when register failed.
Cheers,
Jinpu
>> !device->cache.lmc_cache) {
>> pr_warn("Couldn't allocate cache for %s\n", device->name);
>> return -ENOMEM;
>> @@ -1213,6 +1219,7 @@ void ib_cache_release_one(struct ib_device *device)
>> gid_table_release_one(device);
>> kfree(device->cache.pkey_cache);
>> kfree(device->cache.lmc_cache);
>> + kfree(device->cache.port_state_cache);
>> }
>>
>> void ib_cache_cleanup_one(struct ib_device *device)
>> diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
>> index 5ad43a4..a167ae0 100644
>> --- a/include/rdma/ib_verbs.h
>> +++ b/include/rdma/ib_verbs.h
>> @@ -1761,6 +1761,7 @@ struct ib_cache {
>> struct ib_pkey_cache **pkey_cache;
>> struct ib_gid_table **gid_cache;
>> u8 *lmc_cache;
>> + enum ib_port_state *port_state_cache;
>> };
>>
>> struct ib_dma_mapping_ops {
>>
>
> Mark.
--
Jinpu Wang
Linux Kernel Developer
ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin
Tel: +49 30 577 008 042
Fax: +49 30 577 008 299
Email: jinpu.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org
URL: https://www.profitbricks.de
Sitz der Gesellschaft: Berlin
Registergericht: Amtsgericht Charlottenburg, HRB 125506 B
Geschäftsführer: Achim Weiss
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH 2/4] IB/core: export ib_get_cached_port_state
From: Mark Bloch @ 2016-12-12 12:19 UTC (permalink / raw)
To: Doug Ledford, Hefty, Sean, Hal Rosenstock,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Michael Wang,
Jason Gunthorpe
Hi,
diff --git a/drivers/infiniband/core/cache.c b/drivers/infiniband/core/cache.c
index 025db27..1b064cf 100644
--- a/drivers/infiniband/core/cache.c
+++ b/drivers/infiniband/core/cache.c
@@ -1025,6 +1025,24 @@ int ib_get_cached_lmc(struct ib_device *device,
}
EXPORT_SYMBOL(ib_get_cached_lmc);
+int ib_get_cached_port_state(struct ib_device *device,
+ u8 port_num,
+ enum ib_port_state *port_state)
Can you please fix the style (indentation)
+{
+ unsigned long flags;
+ int ret = 0;
+
+ if (port_num < rdma_start_port(device) || port_num > rdma_end_port(device))
+ return -EINVAL;
+
+ read_lock_irqsave(&device->cache.lock, flags);
+ *port_state = device->cache.port_state_cache[port_num -
rdma_start_port(device)];
+ read_unlock_irqrestore(&device->cache.lock, flags);
+
+ return ret;
Why do you need ret? just return 0.
+}
+EXPORT_SYMBOL(ib_get_cached_port_state);
+
static void ib_cache_update(struct ib_device *device,
u8 port)
{
diff --git a/include/rdma/ib_cache.h b/include/rdma/ib_cache.h
index e30f19b..385ec88 100644
--- a/include/rdma/ib_cache.h
+++ b/include/rdma/ib_cache.h
@@ -165,4 +165,17 @@ int ib_get_cached_lmc(struct ib_device *device,
u8 port_num,
u8 *lmc);
+/**
+ * ib_get_cached_port_state - Returns a cached port state table entry
+ * @device: The device to query.
+ * @port_num: The port number of the device to query.
+ * @port_state: port_state for the specified port for that device.
+ *
+ * ib_get_cached_port_state() fetches the specified port_state table
entry stored in
+ * the local software cache.
+ */
+int ib_get_cached_port_state(struct ib_device *device,
+ u8 port_num,
+ enum ib_port_state *port_active);
+
#endif /* _IB_CACHE_H */
--
2.7.4
Mark.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* Re: [PATCH 1/4] IB/core: add port state cache
From: Mark Bloch @ 2016-12-12 12:11 UTC (permalink / raw)
To: Jinpu Wang, Doug Ledford, Hefty, Sean, Hal Rosenstock,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Michael Wang,
Jason Gunthorpe
In-Reply-To: <CAMGffE=gytHRs+SVOmNPCkWX6tZ9hkeFznqeQQs-TFJMMJi97g-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
Hi,
On 12/12/2016 1:43 PM, Jinpu Wang wrote:
> From fc80c1730d94a38934c5c60f9b9561745723dfd9 Mon Sep 17 00:00:00 2001
> From: Jack Wang <jinpu.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>
> Date: Mon, 12 Dec 2016 09:52:42 +0100
> Subject: [PATCH 1/4] IB/core: add port state cache
>
> We need a port state cache in ib_core, later we will use in rdma_cm.
>
> Signed-off-by: Jack Wang <jinpu.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>
> Reviewed-by: Michael Wang <yun.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>
> ---
> drivers/infiniband/core/cache.c | 9 ++++++++-
> include/rdma/ib_verbs.h | 1 +
> 2 files changed, 9 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/infiniband/core/cache.c b/drivers/infiniband/core/cache.c
> index 1a2984c..025db27 100644
> --- a/drivers/infiniband/core/cache.c
> +++ b/drivers/infiniband/core/cache.c
> @@ -1109,6 +1109,8 @@ static void ib_cache_update(struct ib_device *device,
> }
>
> device->cache.lmc_cache[port - rdma_start_port(device)] = tprops->lmc;
> + device->cache.port_state_cache[port - rdma_start_port(device)] =
> + tprops->state;
>
> write_unlock_irq(&device->cache.lock);
>
> @@ -1168,7 +1170,11 @@ int ib_cache_setup_one(struct ib_device *device)
> (rdma_end_port(device) -
> rdma_start_port(device) + 1),
> GFP_KERNEL);
> - if (!device->cache.pkey_cache ||
> + device->cache.port_state_cache = kmalloc(sizeof
> *device->cache.port_state_cache *
> + (rdma_end_port(device) -
> + rdma_start_port(device) + 1),
> + GFP_KERNEL);
> + if (!device->cache.pkey_cache || !device->cache.port_state_cache ||
Looking at the code, I see we can leak memory here.
if pkey_cache is allocated but lms_cache is not, we will return without freeing pkey_cache (or the other way around)
your code adds port_state_cache to the mix but with the same issue.
I guess that should be fixed.
> !device->cache.lmc_cache) {
> pr_warn("Couldn't allocate cache for %s\n", device->name);
> return -ENOMEM;
> @@ -1213,6 +1219,7 @@ void ib_cache_release_one(struct ib_device *device)
> gid_table_release_one(device);
> kfree(device->cache.pkey_cache);
> kfree(device->cache.lmc_cache);
> + kfree(device->cache.port_state_cache);
> }
>
> void ib_cache_cleanup_one(struct ib_device *device)
> diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
> index 5ad43a4..a167ae0 100644
> --- a/include/rdma/ib_verbs.h
> +++ b/include/rdma/ib_verbs.h
> @@ -1761,6 +1761,7 @@ struct ib_cache {
> struct ib_pkey_cache **pkey_cache;
> struct ib_gid_table **gid_cache;
> u8 *lmc_cache;
> + enum ib_port_state *port_state_cache;
> };
>
> struct ib_dma_mapping_ops {
>
Mark.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH 4/4] cma: use cached port state when bind loopback
From: Jinpu Wang @ 2016-12-12 11:49 UTC (permalink / raw)
To: Doug Ledford, Hefty, Sean, Hal Rosenstock,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Michael Wang,
Jason Gunthorpe
>From f2083c81f7738841cc696feda5f0ff5a96beeb01 Mon Sep 17 00:00:00 2001
From: Jack Wang <jinpu.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>
Date: Mon, 12 Dec 2016 10:29:00 +0100
Subject: [PATCH 4/4] cma: use cached port state when bind loopback
Signed-off-by: Jack Wang <jinpu.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>
Reviewed-by: Michael Wang <yun.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>
---
drivers/infiniband/core/cma.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index 3413aec..f57d017 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -2613,8 +2613,8 @@ static void cma_set_loopback(struct sockaddr *addr)
static int cma_bind_loopback(struct rdma_id_private *id_priv)
{
struct cma_device *cma_dev, *cur_dev;
- struct ib_port_attr port_attr;
union ib_gid gid;
+ enum ib_port_state port_state;
u16 pkey;
int ret;
u8 p;
@@ -2630,8 +2630,8 @@ static int cma_bind_loopback(struct
rdma_id_private *id_priv)
cma_dev = cur_dev;
for (p = 1; p <= cur_dev->device->phys_port_cnt; ++p) {
- if (!ib_query_port(cur_dev->device, p, &port_attr) &&
- port_attr.state == IB_PORT_ACTIVE) {
+ if (!ib_get_cached_port_state(cur_dev->device, p, &port_state) &&
+ port_state == IB_PORT_ACTIVE) {
cma_dev = cur_dev;
goto port_found;
}
--
2.7.4
--
Jinpu Wang
Linux Kernel Developer
ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin
Tel: +49 30 577 008 042
Fax: +49 30 577 008 299
Email: jinpu.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org
URL: https://www.profitbricks.de
Sitz der Gesellschaft: Berlin
Registergericht: Amtsgericht Charlottenburg, HRB 125506 B
Geschäftsführer: Achim Weiss
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH 3/4] cma: resolve to first active ib port
From: Jinpu Wang @ 2016-12-12 11:48 UTC (permalink / raw)
To: Doug Ledford, Hefty, Sean, Hal Rosenstock,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Michael Wang,
Jason Gunthorpe
>From 77323a633072966e358c8f684e5ff4b665a27881 Mon Sep 17 00:00:00 2001
From: Jack Wang <jinpu.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>
Date: Mon, 12 Dec 2016 10:23:55 +0100
Subject: [PATCH 3/4] cma: resolve to first active ib port
When resolve addr if we don't give src addr, cma core will try to resolve to
ib device on itself, current logic is only check if it has same
subnet_prefix, which is not enough if we use default well known gid,
we should also check if port is active.
v3: use cached port_active state in ib_core.
Signed-off-by: Jack Wang <jinpu.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>
Reviewed-by: Michael Wang <yun.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>
---
drivers/infiniband/core/cma.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index 2a6fc47..3413aec 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -666,6 +666,7 @@ static int cma_resolve_ib_dev(struct
rdma_id_private *id_priv)
union ib_gid gid, sgid, *dgid;
u16 pkey, index;
u8 p;
+ enum ib_port_state port_state;
int i;
cma_dev = NULL;
@@ -681,6 +682,8 @@ static int cma_resolve_ib_dev(struct
rdma_id_private *id_priv)
if (ib_find_cached_pkey(cur_dev->device, p, pkey, &index))
continue;
+ if (ib_get_cached_port_state(cur_dev->device, p, &port_state))
+ continue;
for (i = 0; !ib_get_cached_gid(cur_dev->device, p, i,
&gid, NULL);
i++) {
@@ -692,7 +695,8 @@ static int cma_resolve_ib_dev(struct
rdma_id_private *id_priv)
}
if (!cma_dev && (gid.global.subnet_prefix ==
- dgid->global.subnet_prefix)) {
+ dgid->global.subnet_prefix) &&
+ port_state == IB_PORT_ACTIVE) {
cma_dev = cur_dev;
sgid = gid;
id_priv->id.port_num = p;
--
2.7.4
--
Jinpu Wang
Linux Kernel Developer
ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin
Tel: +49 30 577 008 042
Fax: +49 30 577 008 299
Email: jinpu.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org
URL: https://www.profitbricks.de
Sitz der Gesellschaft: Berlin
Registergericht: Amtsgericht Charlottenburg, HRB 125506 B
Geschäftsführer: Achim Weiss
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH 2/4] IB/core: export ib_get_cached_port_state
From: Jinpu Wang @ 2016-12-12 11:45 UTC (permalink / raw)
To: Doug Ledford, Hefty, Sean, Hal Rosenstock,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Michael Wang,
Jason Gunthorpe
>From 1cf770be635e987e31feed2f7c82b9f4f3bd316a Mon Sep 17 00:00:00 2001
From: Jack Wang <jinpu.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>
Date: Mon, 12 Dec 2016 10:12:39 +0100
Subject: [PATCH 2/4] IB/core: export ib_get_cached_port_state
Export function for rdma_cm, patch will follow.
Signed-off-by: Jack Wang <jinpu.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>
Reviewed-by: Michael Wang <yun.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>
---
drivers/infiniband/core/cache.c | 18 ++++++++++++++++++
include/rdma/ib_cache.h | 13 +++++++++++++
2 files changed, 31 insertions(+)
diff --git a/drivers/infiniband/core/cache.c b/drivers/infiniband/core/cache.c
index 025db27..1b064cf 100644
--- a/drivers/infiniband/core/cache.c
+++ b/drivers/infiniband/core/cache.c
@@ -1025,6 +1025,24 @@ int ib_get_cached_lmc(struct ib_device *device,
}
EXPORT_SYMBOL(ib_get_cached_lmc);
+int ib_get_cached_port_state(struct ib_device *device,
+ u8 port_num,
+ enum ib_port_state *port_state)
+{
+ unsigned long flags;
+ int ret = 0;
+
+ if (port_num < rdma_start_port(device) || port_num > rdma_end_port(device))
+ return -EINVAL;
+
+ read_lock_irqsave(&device->cache.lock, flags);
+ *port_state = device->cache.port_state_cache[port_num -
rdma_start_port(device)];
+ read_unlock_irqrestore(&device->cache.lock, flags);
+
+ return ret;
+}
+EXPORT_SYMBOL(ib_get_cached_port_state);
+
static void ib_cache_update(struct ib_device *device,
u8 port)
{
diff --git a/include/rdma/ib_cache.h b/include/rdma/ib_cache.h
index e30f19b..385ec88 100644
--- a/include/rdma/ib_cache.h
+++ b/include/rdma/ib_cache.h
@@ -165,4 +165,17 @@ int ib_get_cached_lmc(struct ib_device *device,
u8 port_num,
u8 *lmc);
+/**
+ * ib_get_cached_port_state - Returns a cached port state table entry
+ * @device: The device to query.
+ * @port_num: The port number of the device to query.
+ * @port_state: port_state for the specified port for that device.
+ *
+ * ib_get_cached_port_state() fetches the specified port_state table
entry stored in
+ * the local software cache.
+ */
+int ib_get_cached_port_state(struct ib_device *device,
+ u8 port_num,
+ enum ib_port_state *port_active);
+
#endif /* _IB_CACHE_H */
--
2.7.4
--
Jinpu Wang
Linux Kernel Developer
ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin
Tel: +49 30 577 008 042
Fax: +49 30 577 008 299
Email: jinpu.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org
URL: https://www.profitbricks.de
Sitz der Gesellschaft: Berlin
Registergericht: Amtsgericht Charlottenburg, HRB 125506 B
Geschäftsführer: Achim Weiss
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH 1/4] IB/core: add port state cache
From: Jinpu Wang @ 2016-12-12 11:43 UTC (permalink / raw)
To: Doug Ledford, Hefty, Sean, Hal Rosenstock,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Michael Wang,
Jason Gunthorpe
>From fc80c1730d94a38934c5c60f9b9561745723dfd9 Mon Sep 17 00:00:00 2001
From: Jack Wang <jinpu.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>
Date: Mon, 12 Dec 2016 09:52:42 +0100
Subject: [PATCH 1/4] IB/core: add port state cache
We need a port state cache in ib_core, later we will use in rdma_cm.
Signed-off-by: Jack Wang <jinpu.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>
Reviewed-by: Michael Wang <yun.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>
---
drivers/infiniband/core/cache.c | 9 ++++++++-
include/rdma/ib_verbs.h | 1 +
2 files changed, 9 insertions(+), 1 deletion(-)
diff --git a/drivers/infiniband/core/cache.c b/drivers/infiniband/core/cache.c
index 1a2984c..025db27 100644
--- a/drivers/infiniband/core/cache.c
+++ b/drivers/infiniband/core/cache.c
@@ -1109,6 +1109,8 @@ static void ib_cache_update(struct ib_device *device,
}
device->cache.lmc_cache[port - rdma_start_port(device)] = tprops->lmc;
+ device->cache.port_state_cache[port - rdma_start_port(device)] =
+ tprops->state;
write_unlock_irq(&device->cache.lock);
@@ -1168,7 +1170,11 @@ int ib_cache_setup_one(struct ib_device *device)
(rdma_end_port(device) -
rdma_start_port(device) + 1),
GFP_KERNEL);
- if (!device->cache.pkey_cache ||
+ device->cache.port_state_cache = kmalloc(sizeof
*device->cache.port_state_cache *
+ (rdma_end_port(device) -
+ rdma_start_port(device) + 1),
+ GFP_KERNEL);
+ if (!device->cache.pkey_cache || !device->cache.port_state_cache ||
!device->cache.lmc_cache) {
pr_warn("Couldn't allocate cache for %s\n", device->name);
return -ENOMEM;
@@ -1213,6 +1219,7 @@ void ib_cache_release_one(struct ib_device *device)
gid_table_release_one(device);
kfree(device->cache.pkey_cache);
kfree(device->cache.lmc_cache);
+ kfree(device->cache.port_state_cache);
}
void ib_cache_cleanup_one(struct ib_device *device)
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 5ad43a4..a167ae0 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -1761,6 +1761,7 @@ struct ib_cache {
struct ib_pkey_cache **pkey_cache;
struct ib_gid_table **gid_cache;
u8 *lmc_cache;
+ enum ib_port_state *port_state_cache;
};
struct ib_dma_mapping_ops {
--
2.7.4
--
Jinpu Wang
Linux Kernel Developer
ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin
Tel: +49 30 577 008 042
Fax: +49 30 577 008 299
Email: jinpu.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org
URL: https://www.profitbricks.de
Sitz der Gesellschaft: Berlin
Registergericht: Amtsgericht Charlottenburg, HRB 125506 B
Geschäftsführer: Achim Weiss
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH 0/4] IB cache port state, use it when resolve address
From: Jinpu Wang @ 2016-12-12 11:40 UTC (permalink / raw)
To: Doug Ledford, Hefty, Sean, Hal Rosenstock,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Jason Gunthorpe
Cc: Michael Wang
Hi Folks,
This is my third attempt to fix rdma_revolve_addr may resolve to first
port even it's down[1].
As Jason and Hefty suggested, I now cache port_state in ib_core, and
update cma_resolve_ib_dev and cma_bind_loopback to use it.
0001-IB-core-add-port-state-cache.patch
0002-IB-core-export-ib_get_cached_port_state.patch
0003-cma-resolve-to-first-active-ib-port.patch
0004-cma-use-cached-port-state-when-bind-loopback.patch
[1]http://marc.info/?l=linux-rdma&m=148102306632521&w=2
--
Jinpu Wang
Linux Kernel Developer
ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin
Tel: +49 30 577 008 042
Fax: +49 30 577 008 299
Email: jinpu.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org
URL: https://www.profitbricks.de
Sitz der Gesellschaft: Berlin
Registergericht: Amtsgericht Charlottenburg, HRB 125506 B
Geschäftsführer: Achim Weiss
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [bug report] IB/mlx5: Support IB_WR_REG_SIG_MR
From: Dan Carpenter @ 2016-12-12 9:33 UTC (permalink / raw)
To: Leon Romanovsky
Cc: sagig-VPRAkNaXOzVWk0Htik3J/w, linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20161211092326.GA4479-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org>
On Sun, Dec 11, 2016 at 11:23:26AM +0200, Leon Romanovsky wrote:
> > 3624 case IB_SIG_TYPE_T10_DIF:
> > 3625 psv_seg->transient_sig = cpu_to_be32(domain->sig.dif.bg << 16 |
> > 3626 domain->sig.dif.app_tag);
> > 3627 psv_seg->ref_tag = cpu_to_be32(domain->sig.dif.ref_tag);
> > 3628 break;
> > 3629 default:
> > 3630 pr_err("Bad signature type given.\n");
> > 3631 return 1;
> >
> > Why do we return 1 here instead of a negative error code?
>
> Yes, probably better to return something like EINVAL, instead of 1.
> do you want us to handle it?
>
Yes, please.
regards,
dan carpenter
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] cma: resolve to first active IB port
From: Jinpu Wang @ 2016-12-12 8:43 UTC (permalink / raw)
To: Hefty, Sean
Cc: Jason Gunthorpe, Doug Ledford, Hal Rosenstock,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Michael Wang
In-Reply-To: <1828884A29C6694DAF28B7E6B8A82373AB0BCED3-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>
On Fri, Dec 9, 2016 at 6:17 PM, Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
>> > Sure, I updated the v2 patch as attached, is this what you want?
>> >
>> > Note: in cma_resolve_loopback, it does the same, I leave it like that
>> > for now, I can also update this part if you like.
>>
>> I feel like we already have a cache for subnet prefix, or maybe
>> another patch series added one? Can't remember, sorry.
>
> Yep - see core/cache.c.
>
> Jinpu, port state caching should go here as well. If you can update the loopback call as well, that would be great.
Thanks Jason and Hefty, good idea!
I will send a patchset to cache port state in cache module, will also
update loopback call as well.
--
Jinpu Wang
Linux Kernel Developer
ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin
Tel: +49 30 577 008 042
Fax: +49 30 577 008 299
Email: jinpu.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org
URL: https://www.profitbricks.de
Sitz der Gesellschaft: Berlin
Registergericht: Amtsgericht Charlottenburg, HRB 125506 B
Geschäftsführer: Achim Weiss
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2] infiniband: mthca: Replace pci_pool_alloc by pci_pool_zalloc
From: Souptick Joarder @ 2016-12-12 4:45 UTC (permalink / raw)
To: Yuval Shaia
Cc: dledford-H+wXaHxf7aLQT0dZR+AlfA,
sean.hefty-ral2JQCrhuEAvxtiuMwx3w, Hal Rosenstock,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, Rameshwar Sahu
In-Reply-To: <CAFqt6zarZHK3+7QH3tfuHXr2YtV_bL8WavKuPt97+YaqQoB=1w-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
On Mon, Dec 5, 2016 at 11:54 AM, Souptick Joarder <jrdr.linux-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Any further comment on this patch?
>
> On Fri, Dec 2, 2016 at 12:43 AM, Yuval Shaia <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> wrote:
>> Reviewed-by: Yuval Shaia <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
>>
>> On Fri, Dec 02, 2016 at 12:11:59AM +0530, Souptick Joarder wrote:
>>> In mthca_create_ah(), pci_pool_alloc() followed by memset will be
>>> replaced by pci_pool_zalloc()
>>>
>>> Signed-off-by: Souptick joarder <jrdr.linux-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
>>> ---
>>> v2:
>>> - Alignment changes
>>> - Remove empty lines
>>>
>>> drivers/infiniband/hw/mthca/mthca_av.c | 6 ++----
>>> 1 file changed, 2 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/drivers/infiniband/hw/mthca/mthca_av.c b/drivers/infiniband/hw/mthca/mthca_av.c
>>> index bcac294..c9f0f36 100644
>>> --- a/drivers/infiniband/hw/mthca/mthca_av.c
>>> +++ b/drivers/infiniband/hw/mthca/mthca_av.c
>>> @@ -186,8 +186,8 @@ int mthca_create_ah(struct mthca_dev *dev,
>>>
>>> on_hca_fail:
>>> if (ah->type == MTHCA_AH_PCI_POOL) {
>>> - ah->av = pci_pool_alloc(dev->av_table.pool,
>>> - GFP_ATOMIC, &ah->avdma);
>>> + ah->av = pci_pool_zalloc(dev->av_table.pool,
>>> + GFP_ATOMIC, &ah->avdma);
>>> if (!ah->av)
>>> return -ENOMEM;
>>>
>>> @@ -196,8 +196,6 @@ int mthca_create_ah(struct mthca_dev *dev,
>>>
>>> ah->key = pd->ntmr.ibmr.lkey;
>>>
>>> - memset(av, 0, MTHCA_AV_SIZE);
>>> -
>>> av->port_pd = cpu_to_be32(pd->pd_num | (ah_attr->port_num << 24));
>>> av->g_slid = ah_attr->src_path_bits;
>>> av->dlid = cpu_to_be16(ah_attr->dlid);
>>> --
>>> 1.9.1
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
Any comment on this patch?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v6 12/16] IB/pvrdma: Add Queue Pair support
From: Yuval Shaia @ 2016-12-11 21:13 UTC (permalink / raw)
To: Adit Ranadive
Cc: dledford-H+wXaHxf7aLQT0dZR+AlfA,
linux-rdma-u79uwXL29TY76Z2rM5mHXA,
pv-drivers-pghWNbHTmq7QT0dZR+AlfA, jhansen-pghWNbHTmq7QT0dZR+AlfA,
asarwade-pghWNbHTmq7QT0dZR+AlfA,
georgezhang-pghWNbHTmq7QT0dZR+AlfA,
bryantan-pghWNbHTmq7QT0dZR+AlfA
In-Reply-To: <6a643e92376856394d45638d80a90619d3abac37.1475458407.git.aditr-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
+
> +/**
> + * pvrdma_post_send - post send work request entries on a QP
> + * @ibqp: the QP
> + * @wr: work request list to post
> + * @bad_wr: the first bad WR returned
> + *
> + * @return: 0 on success, otherwise errno returned.
> + */
> +int pvrdma_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr,
> + struct ib_send_wr **bad_wr)
> +{
> + struct pvrdma_qp *qp = to_vqp(ibqp);
> + struct pvrdma_dev *dev = to_vdev(ibqp->device);
> + unsigned long flags;
> + struct pvrdma_sq_wqe_hdr *wqe_hdr;
> + struct pvrdma_sge *sge;
> + int i, index;
> + int nreq;
> + int ret;
> +
> + /*
> + * In states lower than RTS, we can fail immediately. In other states,
> + * just post and let the device figure it out.
> + */
> + if (qp->state < IB_QPS_RTS) {
> + *bad_wr = wr;
> + return -EINVAL;
> + }
> +
> + spin_lock_irqsave(&qp->sq.lock, flags);
> +
> + index = pvrdma_idx(&qp->sq.ring->prod_tail, qp->sq.wqe_cnt);
> + for (nreq = 0; wr; nreq++, wr = wr->next) {
> + unsigned int tail;
> +
> + if (unlikely(!pvrdma_idx_ring_has_space(
> + qp->sq.ring, qp->sq.wqe_cnt, &tail))) {
> + dev_warn_ratelimited(&dev->pdev->dev,
> + "send queue is full\n");
> + *bad_wr = wr;
> + ret = -ENOMEM;
> + goto out;
> + }
Correct me if i'm wrong but at this point index == tail, right?
If yes then "index" is redundant.
> +
> + if (unlikely(wr->num_sge > qp->sq.max_sg || wr->num_sge < 0)) {
> + dev_warn_ratelimited(&dev->pdev->dev,
> + "send SGE overflow\n");
> + *bad_wr = wr;
> + ret = -EINVAL;
> + goto out;
> + }
> +
> + if (unlikely(wr->opcode < 0)) {
> + dev_warn_ratelimited(&dev->pdev->dev,
> + "invalid send opcode\n");
> + *bad_wr = wr;
> + ret = -EINVAL;
> + goto out;
> + }
> +
> + /*
> + * Only support UD, RC.
> + * Need to check opcode table for thorough checking.
> + * opcode _UD _UC _RC
> + * _SEND x x x
> + * _SEND_WITH_IMM x x x
> + * _RDMA_WRITE x x
> + * _RDMA_WRITE_WITH_IMM x x
> + * _LOCAL_INV x x
> + * _SEND_WITH_INV x x
> + * _RDMA_READ x
> + * _ATOMIC_CMP_AND_SWP x
> + * _ATOMIC_FETCH_AND_ADD x
> + * _MASK_ATOMIC_CMP_AND_SWP x
> + * _MASK_ATOMIC_FETCH_AND_ADD x
> + * _REG_MR x
> + *
> + */
> + if (qp->ibqp.qp_type != IB_QPT_UD &&
> + qp->ibqp.qp_type != IB_QPT_RC &&
> + wr->opcode != IB_WR_SEND) {
> + dev_warn_ratelimited(&dev->pdev->dev,
> + "unsupported queuepair type\n");
> + *bad_wr = wr;
> + ret = -EINVAL;
> + goto out;
> + } else if (qp->ibqp.qp_type == IB_QPT_UD ||
> + qp->ibqp.qp_type == IB_QPT_GSI) {
> + if (wr->opcode != IB_WR_SEND &&
> + wr->opcode != IB_WR_SEND_WITH_IMM) {
> + dev_warn_ratelimited(&dev->pdev->dev,
> + "invalid send opcode\n");
> + *bad_wr = wr;
> + ret = -EINVAL;
> + goto out;
> + }
> + }
> +
> + wqe_hdr = (struct pvrdma_sq_wqe_hdr *)get_sq_wqe(qp, index);
> + memset(wqe_hdr, 0, sizeof(*wqe_hdr));
> + wqe_hdr->wr_id = wr->wr_id;
> + wqe_hdr->num_sge = wr->num_sge;
> + wqe_hdr->opcode = ib_wr_opcode_to_pvrdma(wr->opcode);
> + wqe_hdr->send_flags = ib_send_flags_to_pvrdma(wr->send_flags);
> + if (wr->opcode == IB_WR_SEND_WITH_IMM ||
> + wr->opcode == IB_WR_RDMA_WRITE_WITH_IMM)
> + wqe_hdr->ex.imm_data = wr->ex.imm_data;
> +
> + switch (qp->ibqp.qp_type) {
> + case IB_QPT_GSI:
> + case IB_QPT_UD:
> + if (unlikely(!ud_wr(wr)->ah)) {
> + dev_warn_ratelimited(&dev->pdev->dev,
> + "invalid address handle\n");
> + *bad_wr = wr;
> + ret = -EINVAL;
> + goto out;
> + }
> +
> + /*
> + * Use qkey from qp context if high order bit set,
> + * otherwise from work request.
> + */
> + wqe_hdr->wr.ud.remote_qpn = ud_wr(wr)->remote_qpn;
> + wqe_hdr->wr.ud.remote_qkey =
> + ud_wr(wr)->remote_qkey & 0x80000000 ?
> + qp->qkey : ud_wr(wr)->remote_qkey;
> + wqe_hdr->wr.ud.av = to_vah(ud_wr(wr)->ah)->av;
> +
> + break;
> + case IB_QPT_RC:
> + switch (wr->opcode) {
> + case IB_WR_RDMA_READ:
> + case IB_WR_RDMA_WRITE:
> + case IB_WR_RDMA_WRITE_WITH_IMM:
> + wqe_hdr->wr.rdma.remote_addr =
> + rdma_wr(wr)->remote_addr;
> + wqe_hdr->wr.rdma.rkey = rdma_wr(wr)->rkey;
> + break;
> + case IB_WR_LOCAL_INV:
> + case IB_WR_SEND_WITH_INV:
> + wqe_hdr->ex.invalidate_rkey =
> + wr->ex.invalidate_rkey;
> + break;
> + case IB_WR_ATOMIC_CMP_AND_SWP:
> + case IB_WR_ATOMIC_FETCH_AND_ADD:
> + wqe_hdr->wr.atomic.remote_addr =
> + atomic_wr(wr)->remote_addr;
> + wqe_hdr->wr.atomic.rkey = atomic_wr(wr)->rkey;
> + wqe_hdr->wr.atomic.compare_add =
> + atomic_wr(wr)->compare_add;
> + if (wr->opcode == IB_WR_ATOMIC_CMP_AND_SWP)
> + wqe_hdr->wr.atomic.swap =
> + atomic_wr(wr)->swap;
> + break;
> + case IB_WR_REG_MR:
> + ret = set_reg_seg(wqe_hdr, reg_wr(wr));
> + if (ret < 0) {
> + dev_warn_ratelimited(&dev->pdev->dev,
> + "Failed to set fast register work request\n");
> + *bad_wr = wr;
> + goto out;
> + }
> + break;
> + default:
> + break;
> + }
> +
> + break;
> + default:
> + dev_warn_ratelimited(&dev->pdev->dev,
> + "invalid queuepair type\n");
> + ret = -EINVAL;
> + *bad_wr = wr;
> + goto out;
> + }
> +
> + sge = (struct pvrdma_sge *)(wqe_hdr + 1);
> + for (i = 0; i < wr->num_sge; i++) {
> + /* Need to check wqe_size 0 or max size */
> + sge->addr = wr->sg_list[i].addr;
> + sge->length = wr->sg_list[i].length;
> + sge->lkey = wr->sg_list[i].lkey;
> + sge++;
> + }
> +
> + /* Make sure wqe is written before index update */
> + smp_wmb();
> +
> + index++;
> + if (unlikely(index >= qp->sq.wqe_cnt))
> + index = 0;
If my assumption above is correct then the above 3 lines are not needed.
> + /* Update shared sq ring */
> + pvrdma_idx_ring_inc(&qp->sq.ring->prod_tail,
> + qp->sq.wqe_cnt);
> + }
> +
> + ret = 0;
> +
> +out:
> + spin_unlock_irqrestore(&qp->sq.lock, flags);
> +
> + if (!ret)
> + pvrdma_write_uar_qp(dev, PVRDMA_UAR_QP_SEND | qp->qp_handle);
> +
> + return ret;
> +}
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* RE: [PATCH rdma-core 0/4] libqedr: multiple bug fixes
From: Amrani, Ram @ 2016-12-11 16:02 UTC (permalink / raw)
To: Leon Romanovsky
Cc: dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Elior, Ariel,
Kalderon, Michal
In-Reply-To: <20161211145031.GD20460-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org>
>
> Thanks, applied.
>
That was fast. Thanks!
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox