* Re: [PATCH 1/7] AMSO1100 Makefiles and Kconfig changes.
From: Steve Wise @ 2006-05-31 20:39 UTC (permalink / raw)
To: Roland Dreier; +Cc: mshefty, linux-kernel, netdev, openib-general
In-Reply-To: <ada8xoix76r.fsf@cisco.com>
On Wed, 2006-05-31 at 13:36 -0700, Roland Dreier wrote:
> Can you reorder things so these changes go last? Otherwise after this
> patch we're left with a kernel tree that has a Makefile that refers to
> sources that don't exist yet. It's not really a practical issue but
> it is neater to do that way.
>
> (It's easy to do in stgit -- just pop all the patches and then use
> "stg push <name>" to push them in a different order)
>
> - R.
will do.
^ permalink raw reply
* [PATCH 3/7] AMSO1100 WR / Event Definitions.
From: Steve Wise @ 2006-05-31 20:50 UTC (permalink / raw)
To: rdreier; +Cc: linux-kernel, openib-general, netdev
In-Reply-To: <20060531182733.3652.54755.stgit@stevo-desktop>
[-- Attachment #1: Type: text/plain, Size: 100 bytes --]
I've sent this twice in-line and it is not going through. So here it is
as an attachment.
Steve.
[-- Attachment #2: amso1100_wr --]
[-- Type: text/plain, Size: 46539 bytes --]
AMSO1100 WR / Event Definitions.
From: Steve Wise <swise@opengridcomputing.com>
---
drivers/infiniband/hw/amso1100/c2_ae.h | 108 ++
drivers/infiniband/hw/amso1100/c2_status.h | 158 +++
drivers/infiniband/hw/amso1100/c2_wr.h | 1523 ++++++++++++++++++++++++++++
3 files changed, 1789 insertions(+), 0 deletions(-)
diff --git a/drivers/infiniband/hw/amso1100/c2_ae.h b/drivers/infiniband/hw/amso1100/c2_ae.h
new file mode 100644
index 0000000..3a065c3
--- /dev/null
+++ b/drivers/infiniband/hw/amso1100/c2_ae.h
@@ -0,0 +1,108 @@
+/*
+ * Copyright (c) 2005 Ammasso, Inc. All rights reserved.
+ * Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+#ifndef _C2_AE_H_
+#define _C2_AE_H_
+
+/*
+ * WARNING: If you change this file, also bump C2_IVN_BASE
+ * in common/include/clustercore/c2_ivn.h.
+ */
+
+/*
+ * Asynchronous Event Identifiers
+ *
+ * These start at 0x80 only so it's obvious from inspection that
+ * they are not work-request statuses. This isn't critical.
+ *
+ * NOTE: these event id's must fit in eight bits.
+ */
+enum c2_event_id {
+ CCAE_REMOTE_SHUTDOWN = 0x80,
+ CCAE_ACTIVE_CONNECT_RESULTS,
+ CCAE_CONNECTION_REQUEST,
+ CCAE_LLP_CLOSE_COMPLETE,
+ CCAE_TERMINATE_MESSAGE_RECEIVED,
+ CCAE_LLP_CONNECTION_RESET,
+ CCAE_LLP_CONNECTION_LOST,
+ CCAE_LLP_SEGMENT_SIZE_INVALID,
+ CCAE_LLP_INVALID_CRC,
+ CCAE_LLP_BAD_FPDU,
+ CCAE_INVALID_DDP_VERSION,
+ CCAE_INVALID_RDMA_VERSION,
+ CCAE_UNEXPECTED_OPCODE,
+ CCAE_INVALID_DDP_QUEUE_NUMBER,
+ CCAE_RDMA_READ_NOT_ENABLED,
+ CCAE_RDMA_WRITE_NOT_ENABLED,
+ CCAE_RDMA_READ_TOO_SMALL,
+ CCAE_NO_L_BIT,
+ CCAE_TAGGED_INVALID_STAG,
+ CCAE_TAGGED_BASE_BOUNDS_VIOLATION,
+ CCAE_TAGGED_ACCESS_RIGHTS_VIOLATION,
+ CCAE_TAGGED_INVALID_PD,
+ CCAE_WRAP_ERROR,
+ CCAE_BAD_CLOSE,
+ CCAE_BAD_LLP_CLOSE,
+ CCAE_INVALID_MSN_RANGE,
+ CCAE_INVALID_MSN_GAP,
+ CCAE_IRRQ_OVERFLOW,
+ CCAE_IRRQ_MSN_GAP,
+ CCAE_IRRQ_MSN_RANGE,
+ CCAE_IRRQ_INVALID_STAG,
+ CCAE_IRRQ_BASE_BOUNDS_VIOLATION,
+ CCAE_IRRQ_ACCESS_RIGHTS_VIOLATION,
+ CCAE_IRRQ_INVALID_PD,
+ CCAE_IRRQ_WRAP_ERROR,
+ CCAE_CQ_SQ_COMPLETION_OVERFLOW,
+ CCAE_CQ_RQ_COMPLETION_ERROR,
+ CCAE_QP_SRQ_WQE_ERROR,
+ CCAE_QP_LOCAL_CATASTROPHIC_ERROR,
+ CCAE_CQ_OVERFLOW,
+ CCAE_CQ_OPERATION_ERROR,
+ CCAE_SRQ_LIMIT_REACHED,
+ CCAE_QP_RQ_LIMIT_REACHED,
+ CCAE_SRQ_CATASTROPHIC_ERROR,
+ CCAE_RNIC_CATASTROPHIC_ERROR
+/* WARNING If you add more id's, make sure their values fit in eight bits. */
+};
+
+/*
+ * Resource Indicators and Identifiers
+ */
+enum c2_resource_indicator {
+ C2_RES_IND_QP = 1,
+ C2_RES_IND_EP,
+ C2_RES_IND_CQ,
+ C2_RES_IND_SRQ,
+};
+
+#endif /* _C2_AE_H_ */
diff --git a/drivers/infiniband/hw/amso1100/c2_status.h b/drivers/infiniband/hw/amso1100/c2_status.h
new file mode 100644
index 0000000..6ee4aa9
--- /dev/null
+++ b/drivers/infiniband/hw/amso1100/c2_status.h
@@ -0,0 +1,158 @@
+/*
+ * Copyright (c) 2005 Ammasso, Inc. All rights reserved.
+ * Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+#ifndef _C2_STATUS_H_
+#define _C2_STATUS_H_
+
+/*
+ * Verbs Status Codes
+ */
+enum c2_status {
+ C2_OK = 0, /* This must be zero */
+ CCERR_INSUFFICIENT_RESOURCES = 1,
+ CCERR_INVALID_MODIFIER = 2,
+ CCERR_INVALID_MODE = 3,
+ CCERR_IN_USE = 4,
+ CCERR_INVALID_RNIC = 5,
+ CCERR_INTERRUPTED_OPERATION = 6,
+ CCERR_INVALID_EH = 7,
+ CCERR_INVALID_CQ = 8,
+ CCERR_CQ_EMPTY = 9,
+ CCERR_NOT_IMPLEMENTED = 10,
+ CCERR_CQ_DEPTH_TOO_SMALL = 11,
+ CCERR_PD_IN_USE = 12,
+ CCERR_INVALID_PD = 13,
+ CCERR_INVALID_SRQ = 14,
+ CCERR_INVALID_ADDRESS = 15,
+ CCERR_INVALID_NETMASK = 16,
+ CCERR_INVALID_QP = 17,
+ CCERR_INVALID_QP_STATE = 18,
+ CCERR_TOO_MANY_WRS_POSTED = 19,
+ CCERR_INVALID_WR_TYPE = 20,
+ CCERR_INVALID_SGL_LENGTH = 21,
+ CCERR_INVALID_SQ_DEPTH = 22,
+ CCERR_INVALID_RQ_DEPTH = 23,
+ CCERR_INVALID_ORD = 24,
+ CCERR_INVALID_IRD = 25,
+ CCERR_QP_ATTR_CANNOT_CHANGE = 26,
+ CCERR_INVALID_STAG = 27,
+ CCERR_QP_IN_USE = 28,
+ CCERR_OUTSTANDING_WRS = 29,
+ CCERR_STAG_IN_USE = 30,
+ CCERR_INVALID_STAG_INDEX = 31,
+ CCERR_INVALID_SGL_FORMAT = 32,
+ CCERR_ADAPTER_TIMEOUT = 33,
+ CCERR_INVALID_CQ_DEPTH = 34,
+ CCERR_INVALID_PRIVATE_DATA_LENGTH = 35,
+ CCERR_INVALID_EP = 36,
+ CCERR_MR_IN_USE = CCERR_STAG_IN_USE,
+ CCERR_FLUSHED = 38,
+ CCERR_INVALID_WQE = 39,
+ CCERR_LOCAL_QP_CATASTROPHIC_ERROR = 40,
+ CCERR_REMOTE_TERMINATION_ERROR = 41,
+ CCERR_BASE_AND_BOUNDS_VIOLATION = 42,
+ CCERR_ACCESS_VIOLATION = 43,
+ CCERR_INVALID_PD_ID = 44,
+ CCERR_WRAP_ERROR = 45,
+ CCERR_INV_STAG_ACCESS_ERROR = 46,
+ CCERR_ZERO_RDMA_READ_RESOURCES = 47,
+ CCERR_QP_NOT_PRIVILEGED = 48,
+ CCERR_STAG_STATE_NOT_INVALID = 49,
+ CCERR_INVALID_PAGE_SIZE = 50,
+ CCERR_INVALID_BUFFER_SIZE = 51,
+ CCERR_INVALID_PBE = 52,
+ CCERR_INVALID_FBO = 53,
+ CCERR_INVALID_LENGTH = 54,
+ CCERR_INVALID_ACCESS_RIGHTS = 55,
+ CCERR_PBL_TOO_BIG = 56,
+ CCERR_INVALID_VA = 57,
+ CCERR_INVALID_REGION = 58,
+ CCERR_INVALID_WINDOW = 59,
+ CCERR_TOTAL_LENGTH_TOO_BIG = 60,
+ CCERR_INVALID_QP_ID = 61,
+ CCERR_ADDR_IN_USE = 62,
+ CCERR_ADDR_NOT_AVAIL = 63,
+ CCERR_NET_DOWN = 64,
+ CCERR_NET_UNREACHABLE = 65,
+ CCERR_CONN_ABORTED = 66,
+ CCERR_CONN_RESET = 67,
+ CCERR_NO_BUFS = 68,
+ CCERR_CONN_TIMEDOUT = 69,
+ CCERR_CONN_REFUSED = 70,
+ CCERR_HOST_UNREACHABLE = 71,
+ CCERR_INVALID_SEND_SGL_DEPTH = 72,
+ CCERR_INVALID_RECV_SGL_DEPTH = 73,
+ CCERR_INVALID_RDMA_WRITE_SGL_DEPTH = 74,
+ CCERR_INSUFFICIENT_PRIVILEGES = 75,
+ CCERR_STACK_ERROR = 76,
+ CCERR_INVALID_VERSION = 77,
+ CCERR_INVALID_MTU = 78,
+ CCERR_INVALID_IMAGE = 79,
+ CCERR_PENDING = 98, /* not an error; user internally by adapter */
+ CCERR_DEFER = 99, /* not an error; used internally by adapter */
+ CCERR_FAILED_WRITE = 100,
+ CCERR_FAILED_ERASE = 101,
+ CCERR_FAILED_VERIFICATION = 102,
+ CCERR_NOT_FOUND = 103,
+
+};
+
+/*
+ * CCAE_ACTIVE_CONNECT_RESULTS status result codes.
+ */
+enum c2_connect_status {
+ C2_CONN_STATUS_SUCCESS = C2_OK,
+ C2_CONN_STATUS_NO_MEM = CCERR_INSUFFICIENT_RESOURCES,
+ C2_CONN_STATUS_TIMEDOUT = CCERR_CONN_TIMEDOUT,
+ C2_CONN_STATUS_REFUSED = CCERR_CONN_REFUSED,
+ C2_CONN_STATUS_NETUNREACH = CCERR_NET_UNREACHABLE,
+ C2_CONN_STATUS_HOSTUNREACH = CCERR_HOST_UNREACHABLE,
+ C2_CONN_STATUS_INVALID_RNIC = CCERR_INVALID_RNIC,
+ C2_CONN_STATUS_INVALID_QP = CCERR_INVALID_QP,
+ C2_CONN_STATUS_INVALID_QP_STATE = CCERR_INVALID_QP_STATE,
+ C2_CONN_STATUS_REJECTED = CCERR_CONN_RESET,
+ C2_CONN_STATUS_ADDR_NOT_AVAIL = CCERR_ADDR_NOT_AVAIL,
+};
+
+/*
+ * Flash programming status codes.
+ */
+enum c2_flash_status {
+ C2_FLASH_STATUS_SUCCESS = 0x0000,
+ C2_FLASH_STATUS_VERIFY_ERR = 0x0002,
+ C2_FLASH_STATUS_IMAGE_ERR = 0x0004,
+ C2_FLASH_STATUS_ECLBS = 0x0400,
+ C2_FLASH_STATUS_PSLBS = 0x0800,
+ C2_FLASH_STATUS_VPENS = 0x1000,
+};
+
+#endif /* _C2_STATUS_H_ */
diff --git a/drivers/infiniband/hw/amso1100/c2_wr.h b/drivers/infiniband/hw/amso1100/c2_wr.h
new file mode 100644
index 0000000..9d6468d
--- /dev/null
+++ b/drivers/infiniband/hw/amso1100/c2_wr.h
@@ -0,0 +1,1523 @@
+/*
+ * Copyright (c) 2005 Ammasso, Inc. All rights reserved.
+ * Copyright (c) 2005 Open Grid Computing, Inc. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+#ifndef _C2_WR_H_
+#define _C2_WR_H_
+
+#ifdef CCDEBUG
+#define CCWR_MAGIC 0xb07700b0
+#endif
+
+#define C2_QP_NO_ATTR_CHANGE 0xFFFFFFFF
+
+/* Maximum allowed size in bytes of private_data exchange
+ * on connect.
+ */
+#define C2_MAX_PRIVATE_DATA_SIZE 200
+
+/*
+ * These types are shared among the adapter, host, and CCIL consumer.
+ */
+enum c2_cq_notification_type {
+ C2_CQ_NOTIFICATION_TYPE_NONE = 1,
+ C2_CQ_NOTIFICATION_TYPE_NEXT,
+ C2_CQ_NOTIFICATION_TYPE_NEXT_SE
+};
+
+enum c2_setconfig_cmd {
+ C2_CFG_ADD_ADDR = 1,
+ C2_CFG_DEL_ADDR = 2,
+ C2_CFG_ADD_ROUTE = 3,
+ C2_CFG_DEL_ROUTE = 4
+};
+
+enum c2_getconfig_cmd {
+ C2_GETCONFIG_ROUTES = 1,
+ C2_GETCONFIG_ADDRS
+};
+
+/*
+ * CCIL Work Request Identifiers
+ */
+enum c2wr_ids {
+ CCWR_RNIC_OPEN = 1,
+ CCWR_RNIC_QUERY,
+ CCWR_RNIC_SETCONFIG,
+ CCWR_RNIC_GETCONFIG,
+ CCWR_RNIC_CLOSE,
+ CCWR_CQ_CREATE,
+ CCWR_CQ_QUERY,
+ CCWR_CQ_MODIFY,
+ CCWR_CQ_DESTROY,
+ CCWR_QP_CONNECT,
+ CCWR_PD_ALLOC,
+ CCWR_PD_DEALLOC,
+ CCWR_SRQ_CREATE,
+ CCWR_SRQ_QUERY,
+ CCWR_SRQ_MODIFY,
+ CCWR_SRQ_DESTROY,
+ CCWR_QP_CREATE,
+ CCWR_QP_QUERY,
+ CCWR_QP_MODIFY,
+ CCWR_QP_DESTROY,
+ CCWR_NSMR_STAG_ALLOC,
+ CCWR_NSMR_REGISTER,
+ CCWR_NSMR_PBL,
+ CCWR_STAG_DEALLOC,
+ CCWR_NSMR_REREGISTER,
+ CCWR_SMR_REGISTER,
+ CCWR_MR_QUERY,
+ CCWR_MW_ALLOC,
+ CCWR_MW_QUERY,
+ CCWR_EP_CREATE,
+ CCWR_EP_GETOPT,
+ CCWR_EP_SETOPT,
+ CCWR_EP_DESTROY,
+ CCWR_EP_BIND,
+ CCWR_EP_CONNECT,
+ CCWR_EP_LISTEN,
+ CCWR_EP_SHUTDOWN,
+ CCWR_EP_LISTEN_CREATE,
+ CCWR_EP_LISTEN_DESTROY,
+ CCWR_EP_QUERY,
+ CCWR_CR_ACCEPT,
+ CCWR_CR_REJECT,
+ CCWR_CONSOLE,
+ CCWR_TERM,
+ CCWR_FLASH_INIT,
+ CCWR_FLASH,
+ CCWR_BUF_ALLOC,
+ CCWR_BUF_FREE,
+ CCWR_FLASH_WRITE,
+ CCWR_INIT, /* WARNING: Don't move this ever again! */
+
+
+
+ /* Add new IDs here */
+
+
+
+ /*
+ * WARNING: CCWR_LAST must always be the last verbs id defined!
+ * All the preceding IDs are fixed, and must not change.
+ * You can add new IDs, but must not remove or reorder
+ * any IDs. If you do, YOU will ruin any hope of
+ * compatability between versions.
+ */
+ CCWR_LAST,
+
+ /*
+ * Start over at 1 so that arrays indexed by user wr id's
+ * begin at 1. This is OK since the verbs and user wr id's
+ * are always used on disjoint sets of queues.
+ */
+ /*
+ * The order of the CCWR_SEND_XX verbs must
+ * match the order of the RDMA_OPs
+ */
+ CCWR_SEND = 1,
+ CCWR_SEND_INV,
+ CCWR_SEND_SE,
+ CCWR_SEND_SE_INV,
+ CCWR_RDMA_WRITE,
+ CCWR_RDMA_READ,
+ CCWR_RDMA_READ_INV,
+ CCWR_MW_BIND,
+ CCWR_NSMR_FASTREG,
+ CCWR_STAG_INVALIDATE,
+ CCWR_RECV,
+ CCWR_NOP,
+ CCWR_UNIMPL,
+/* WARNING: This must always be the last user wr id defined! */
+};
+#define RDMA_SEND_OPCODE_FROM_WR_ID(x) (x+2)
+
+/*
+ * SQ/RQ Work Request Types
+ */
+enum c2_wr_type {
+ C2_WR_TYPE_SEND = CCWR_SEND,
+ C2_WR_TYPE_SEND_SE = CCWR_SEND_SE,
+ C2_WR_TYPE_SEND_INV = CCWR_SEND_INV,
+ C2_WR_TYPE_SEND_SE_INV = CCWR_SEND_SE_INV,
+ C2_WR_TYPE_RDMA_WRITE = CCWR_RDMA_WRITE,
+ C2_WR_TYPE_RDMA_READ = CCWR_RDMA_READ,
+ C2_WR_TYPE_RDMA_READ_INV_STAG = CCWR_RDMA_READ_INV,
+ C2_WR_TYPE_BIND_MW = CCWR_MW_BIND,
+ C2_WR_TYPE_FASTREG_NSMR = CCWR_NSMR_FASTREG,
+ C2_WR_TYPE_INV_STAG = CCWR_STAG_INVALIDATE,
+ C2_WR_TYPE_RECV = CCWR_RECV,
+ C2_WR_TYPE_NOP = CCWR_NOP,
+};
+
+struct c2_netaddr {
+ u32 ip_addr;
+ u32 netmask;
+ u32 mtu;
+};
+
+struct c2_route {
+ u32 ip_addr; /* 0 indicates the default route */
+ u32 netmask; /* netmask associated with dst */
+ u32 flags;
+ union {
+ u32 ipaddr; /* address of the nexthop interface */
+ u8 enaddr[6];
+ } nexthop;
+};
+
+/*
+ * A Scatter Gather Entry.
+ */
+struct c2_data_addr {
+ u32 stag;
+ u32 length;
+ u64 to;
+};
+
+/*
+ * MR and MW flags used by the consumer, RI, and RNIC.
+ */
+enum c2_mm_flags {
+ MEM_REMOTE = 0x0001, /* allow mw binds with remote access. */
+ MEM_VA_BASED = 0x0002, /* Not Zero-based */
+ MEM_PBL_COMPLETE = 0x0004, /* PBL array is complete in this msg */
+ MEM_LOCAL_READ = 0x0008, /* allow local reads */
+ MEM_LOCAL_WRITE = 0x0010, /* allow local writes */
+ MEM_REMOTE_READ = 0x0020, /* allow remote reads */
+ MEM_REMOTE_WRITE = 0x0040, /* allow remote writes */
+ MEM_WINDOW_BIND = 0x0080, /* binds allowed */
+ MEM_SHARED = 0x0100, /* set if MR is shared */
+ MEM_STAG_VALID = 0x0200 /* set if STAG is in valid state */
+};
+
+/*
+ * CCIL API ACF flags defined in terms of the low level mem flags.
+ * This minimizes translation needed in the user API
+ */
+enum c2_acf {
+ C2_ACF_LOCAL_READ = MEM_LOCAL_READ,
+ C2_ACF_LOCAL_WRITE = MEM_LOCAL_WRITE,
+ C2_ACF_REMOTE_READ = MEM_REMOTE_READ,
+ C2_ACF_REMOTE_WRITE = MEM_REMOTE_WRITE,
+ C2_ACF_WINDOW_BIND = MEM_WINDOW_BIND
+};
+
+/*
+ * Image types of objects written to flash
+ */
+#define C2_FLASH_IMG_BITFILE 1
+#define C2_FLASH_IMG_OPTION_ROM 2
+#define C2_FLASH_IMG_VPD 3
+
+/*
+ * to fix bug 1815 we define the max size allowable of the
+ * terminate message (per the IETF spec).Refer to the IETF
+ * protocal specification, section 12.1.6, page 64)
+ * The message is prefixed by 20 types of DDP info.
+ *
+ * Then the message has 6 bytes for the terminate control
+ * and DDP segment length info plus a DDP header (either
+ * 14 or 18 byts) plus 28 bytes for the RDMA header.
+ * Thus the max size in:
+ * 20 + (6 + 18 + 28) = 72
+ */
+#define C2_MAX_TERMINATE_MESSAGE_SIZE (72)
+
+/*
+ * Build String Length. It must be the same as C2_BUILD_STR_LEN in ccil_api.h
+ */
+#define WR_BUILD_STR_LEN 64
+
+/*
+ * WARNING: All of these structs need to align any 64bit types on
+ * 64 bit boundaries! 64bit types include u64 and u64.
+ */
+
+/*
+ * Clustercore Work Request Header. Be sensitive to field layout
+ * and alignment.
+ */
+struct c2wr_hdr {
+ /* wqe_count is part of the cqe. It is put here so the
+ * adapter can write to it while the wr is pending without
+ * clobbering part of the wr. This word need not be dma'd
+ * from the host to adapter by libccil, but we copy it anyway
+ * to make the memcpy to the adapter better aligned.
+ */
+ u32 wqe_count;
+
+ /* Put these fields next so that later 32- and 64-bit
+ * quantities are naturally aligned.
+ */
+ u8 id;
+ u8 result; /* adapter -> host */
+ u8 sge_count; /* host -> adapter */
+ u8 flags; /* host -> adapter */
+
+ u64 context;
+#ifdef CCMSGMAGIC
+ u32 magic;
+ u32 pad;
+#endif
+} __attribute__((packed));
+
+/*
+ *------------------------ RNIC ------------------------
+ */
+
+/*
+ * WR_RNIC_OPEN
+ */
+
+/*
+ * Flags for the RNIC WRs
+ */
+enum c2_rnic_flags {
+ RNIC_IRD_STATIC = 0x0001,
+ RNIC_ORD_STATIC = 0x0002,
+ RNIC_QP_STATIC = 0x0004,
+ RNIC_SRQ_SUPPORTED = 0x0008,
+ RNIC_PBL_BLOCK_MODE = 0x0010,
+ RNIC_SRQ_MODEL_ARRIVAL = 0x0020,
+ RNIC_CQ_OVF_DETECTED = 0x0040,
+ RNIC_PRIV_MODE = 0x0080
+};
+
+struct c2wr_rnic_open_req {
+ struct c2wr_hdr hdr;
+ u64 user_context;
+ u16 flags; /* See enum c2_rnic_flags */
+ u16 port_num;
+} __attribute__((packed));
+
+struct c2wr_rnic_open_rep {
+ struct c2wr_hdr hdr;
+ u32 rnic_handle;
+} __attribute__((packed));
+
+union c2wr_rnic_open {
+ struct c2wr_rnic_open_req req;
+ struct c2wr_rnic_open_rep rep;
+} __attribute__((packed));
+
+struct c2wr_rnic_query_req {
+ struct c2wr_hdr hdr;
+ u32 rnic_handle;
+} __attribute__((packed));
+
+/*
+ * WR_RNIC_QUERY
+ */
+struct c2wr_rnic_query_rep {
+ struct c2wr_hdr hdr;
+ u64 user_context;
+ u32 vendor_id;
+ u32 part_number;
+ u32 hw_version;
+ u32 fw_ver_major;
+ u32 fw_ver_minor;
+ u32 fw_ver_patch;
+ char fw_ver_build_str[WR_BUILD_STR_LEN];
+ u32 max_qps;
+ u32 max_qp_depth;
+ u32 max_srq_depth;
+ u32 max_send_sgl_depth;
+ u32 max_rdma_sgl_depth;
+ u32 max_cqs;
+ u32 max_cq_depth;
+ u32 max_cq_event_handlers;
+ u32 max_mrs;
+ u32 max_pbl_depth;
+ u32 max_pds;
+ u32 max_global_ird;
+ u32 max_global_ord;
+ u32 max_qp_ird;
+ u32 max_qp_ord;
+ u32 flags;
+ u32 max_mws;
+ u32 pbe_range_low;
+ u32 pbe_range_high;
+ u32 max_srqs;
+ u32 page_size;
+} __attribute__((packed));
+
+union c2wr_rnic_query {
+ struct c2wr_rnic_query_req req;
+ struct c2wr_rnic_query_rep rep;
+} __attribute__((packed));
+
+/*
+ * WR_RNIC_GETCONFIG
+ */
+
+struct c2wr_rnic_getconfig_req {
+ struct c2wr_hdr hdr;
+ u32 rnic_handle;
+ u32 option; /* see c2_getconfig_cmd_t */
+ u64 reply_buf;
+ u32 reply_buf_len;
+} __attribute__((packed)) ;
+
+struct c2wr_rnic_getconfig_rep {
+ struct c2wr_hdr hdr;
+ u32 option; /* see c2_getconfig_cmd_t */
+ u32 count_len; /* length of the number of addresses configured */
+} __attribute__((packed)) ;
+
+union c2wr_rnic_getconfig {
+ struct c2wr_rnic_getconfig_req req;
+ struct c2wr_rnic_getconfig_rep rep;
+} __attribute__((packed)) ;
+
+/*
+ * WR_RNIC_SETCONFIG
+ */
+struct c2wr_rnic_setconfig_req {
+ struct c2wr_hdr hdr;
+ u32 rnic_handle;
+ u32 option; /* See c2_setconfig_cmd_t */
+ /* variable data and pad. See c2_netaddr and c2_route */
+ u8 data[0];
+} __attribute__((packed)) ;
+
+struct c2wr_rnic_setconfig_rep {
+ struct c2wr_hdr hdr;
+} __attribute__((packed)) ;
+
+union c2wr_rnic_setconfig {
+ struct c2wr_rnic_setconfig_req req;
+ struct c2wr_rnic_setconfig_rep rep;
+} __attribute__((packed)) ;
+
+/*
+ * WR_RNIC_CLOSE
+ */
+struct c2wr_rnic_close_req {
+ struct c2wr_hdr hdr;
+ u32 rnic_handle;
+} __attribute__((packed)) ;
+
+struct c2wr_rnic_close_rep {
+ struct c2wr_hdr hdr;
+} __attribute__((packed)) ;
+
+union c2wr_rnic_close {
+ struct c2wr_rnic_close_req req;
+ struct c2wr_rnic_close_rep rep;
+} __attribute__((packed)) ;
+
+/*
+ *------------------------ CQ ------------------------
+ */
+struct c2wr_cq_create_req {
+ struct c2wr_hdr hdr;
+ u64 shared_ht;
+ u64 user_context;
+ u64 msg_pool;
+ u32 rnic_handle;
+ u32 msg_size;
+ u32 depth;
+} __attribute__((packed)) ;
+
+struct c2wr_cq_create_rep {
+ struct c2wr_hdr hdr;
+ u32 mq_index;
+ u32 adapter_shared;
+ u32 cq_handle;
+} __attribute__((packed)) ;
+
+union c2wr_cq_create {
+ struct c2wr_cq_create_req req;
+ struct c2wr_cq_create_rep rep;
+} __attribute__((packed)) ;
+
+struct c2wr_cq_modify_req {
+ struct c2wr_hdr hdr;
+ u32 rnic_handle;
+ u32 cq_handle;
+ u32 new_depth;
+ u64 new_msg_pool;
+} __attribute__((packed)) ;
+
+struct c2wr_cq_modify_rep {
+ struct c2wr_hdr hdr;
+} __attribute__((packed)) ;
+
+union c2wr_cq_modify {
+ struct c2wr_cq_modify_req req;
+ struct c2wr_cq_modify_rep rep;
+} __attribute__((packed)) ;
+
+struct c2wr_cq_destroy_req {
+ struct c2wr_hdr hdr;
+ u32 rnic_handle;
+ u32 cq_handle;
+} __attribute__((packed)) ;
+
+struct c2wr_cq_destroy_rep {
+ struct c2wr_hdr hdr;
+} __attribute__((packed)) ;
+
+union c2wr_cq_destroy {
+ struct c2wr_cq_destroy_req req;
+ struct c2wr_cq_destroy_rep rep;
+} __attribute__((packed)) ;
+
+/*
+ *------------------------ PD ------------------------
+ */
+struct c2wr_pd_alloc_req {
+ struct c2wr_hdr hdr;
+ u32 rnic_handle;
+ u32 pd_id;
+} __attribute__((packed)) ;
+
+struct c2wr_pd_alloc_rep {
+ struct c2wr_hdr hdr;
+} __attribute__((packed)) ;
+
+union c2wr_pd_alloc {
+ struct c2wr_pd_alloc_req req;
+ struct c2wr_pd_alloc_rep rep;
+} __attribute__((packed)) ;
+
+struct c2wr_pd_dealloc_req {
+ struct c2wr_hdr hdr;
+ u32 rnic_handle;
+ u32 pd_id;
+} __attribute__((packed)) ;
+
+struct c2wr_pd_dealloc_rep {
+ struct c2wr_hdr hdr;
+} __attribute__((packed)) ;
+
+union c2wr_pd_dealloc {
+ struct c2wr_pd_dealloc_req req;
+ struct c2wr_pd_dealloc_rep rep;
+} __attribute__((packed)) ;
+
+/*
+ *------------------------ SRQ ------------------------
+ */
+struct c2wr_srq_create_req {
+ struct c2wr_hdr hdr;
+ u64 shared_ht;
+ u64 user_context;
+ u32 rnic_handle;
+ u32 srq_depth;
+ u32 srq_limit;
+ u32 sgl_depth;
+ u32 pd_id;
+} __attribute__((packed)) ;
+
+struct c2wr_srq_create_rep {
+ struct c2wr_hdr hdr;
+ u32 srq_depth;
+ u32 sgl_depth;
+ u32 msg_size;
+ u32 mq_index;
+ u32 mq_start;
+ u32 srq_handle;
+} __attribute__((packed)) ;
+
+union c2wr_srq_create {
+ struct c2wr_srq_create_req req;
+ struct c2wr_srq_create_rep rep;
+} __attribute__((packed)) ;
+
+struct c2wr_srq_destroy_req {
+ struct c2wr_hdr hdr;
+ u32 rnic_handle;
+ u32 srq_handle;
+} __attribute__((packed)) ;
+
+struct c2wr_srq_destroy_rep {
+ struct c2wr_hdr hdr;
+} __attribute__((packed)) ;
+
+union c2wr_srq_destroy {
+ struct c2wr_srq_destroy_req req;
+ struct c2wr_srq_destroy_rep rep;
+} __attribute__((packed)) ;
+
+/*
+ *------------------------ QP ------------------------
+ */
+enum c2wr_qp_flags {
+ QP_RDMA_READ = 0x00000001, /* RDMA read enabled? */
+ QP_RDMA_WRITE = 0x00000002, /* RDMA write enabled? */
+ QP_MW_BIND = 0x00000004, /* MWs enabled */
+ QP_ZERO_STAG = 0x00000008, /* enabled? */
+ QP_REMOTE_TERMINATION = 0x00000010, /* remote end terminated */
+ QP_RDMA_READ_RESPONSE = 0x00000020 /* Remote RDMA read */
+ /* enabled? */
+};
+
+struct c2wr_qp_create_req {
+ struct c2wr_hdr hdr;
+ u64 shared_sq_ht;
+ u64 shared_rq_ht;
+ u64 user_context;
+ u32 rnic_handle;
+ u32 sq_cq_handle;
+ u32 rq_cq_handle;
+ u32 sq_depth;
+ u32 rq_depth;
+ u32 srq_handle;
+ u32 srq_limit;
+ u32 flags; /* see enum c2wr_qp_flags */
+ u32 send_sgl_depth;
+ u32 recv_sgl_depth;
+ u32 rdma_write_sgl_depth;
+ u32 ord;
+ u32 ird;
+ u32 pd_id;
+} __attribute__((packed)) ;
+
+struct c2wr_qp_create_rep {
+ struct c2wr_hdr hdr;
+ u32 sq_depth;
+ u32 rq_depth;
+ u32 send_sgl_depth;
+ u32 recv_sgl_depth;
+ u32 rdma_write_sgl_depth;
+ u32 ord;
+ u32 ird;
+ u32 sq_msg_size;
+ u32 sq_mq_index;
+ u32 sq_mq_start;
+ u32 rq_msg_size;
+ u32 rq_mq_index;
+ u32 rq_mq_start;
+ u32 qp_handle;
+} __attribute__((packed)) ;
+
+union c2wr_qp_create {
+ struct c2wr_qp_create_req req;
+ struct c2wr_qp_create_rep rep;
+} __attribute__((packed)) ;
+
+struct c2wr_qp_query_req {
+ struct c2wr_hdr hdr;
+ u32 rnic_handle;
+ u32 qp_handle;
+} __attribute__((packed)) ;
+
+struct c2wr_qp_query_rep {
+ struct c2wr_hdr hdr;
+ u64 user_context;
+ u32 rnic_handle;
+ u32 sq_depth;
+ u32 rq_depth;
+ u32 send_sgl_depth;
+ u32 rdma_write_sgl_depth;
+ u32 recv_sgl_depth;
+ u32 ord;
+ u32 ird;
+ u16 qp_state;
+ u16 flags; /* see c2wr_qp_flags_t */
+ u32 qp_id;
+ u32 local_addr;
+ u32 remote_addr;
+ u16 local_port;
+ u16 remote_port;
+ u32 terminate_msg_length; /* 0 if not present */
+ u8 data[0];
+ /* Terminate Message in-line here. */
+} __attribute__((packed)) ;
+
+union c2wr_qp_query {
+ struct c2wr_qp_query_req req;
+ struct c2wr_qp_query_rep rep;
+} __attribute__((packed)) ;
+
+struct c2wr_qp_modify_req {
+ struct c2wr_hdr hdr;
+ u64 stream_msg;
+ u32 stream_msg_length;
+ u32 rnic_handle;
+ u32 qp_handle;
+ u32 next_qp_state;
+ u32 ord;
+ u32 ird;
+ u32 sq_depth;
+ u32 rq_depth;
+ u32 llp_ep_handle;
+} __attribute__((packed)) ;
+
+struct c2wr_qp_modify_rep {
+ struct c2wr_hdr hdr;
+ u32 ord;
+ u32 ird;
+ u32 sq_depth;
+ u32 rq_depth;
+ u32 sq_msg_size;
+ u32 sq_mq_index;
+ u32 sq_mq_start;
+ u32 rq_msg_size;
+ u32 rq_mq_index;
+ u32 rq_mq_start;
+} __attribute__((packed)) ;
+
+union c2wr_qp_modify {
+ struct c2wr_qp_modify_req req;
+ struct c2wr_qp_modify_rep rep;
+} __attribute__((packed)) ;
+
+struct c2wr_qp_destroy_req {
+ struct c2wr_hdr hdr;
+ u32 rnic_handle;
+ u32 qp_handle;
+} __attribute__((packed)) ;
+
+struct c2wr_qp_destroy_rep {
+ struct c2wr_hdr hdr;
+} __attribute__((packed)) ;
+
+union c2wr_qp_destroy {
+ struct c2wr_qp_destroy_req req;
+ struct c2wr_qp_destroy_rep rep;
+} __attribute__((packed)) ;
+
+/*
+ * The CCWR_QP_CONNECT msg is posted on the verbs request queue. It can
+ * only be posted when a QP is in IDLE state. After the connect request is
+ * submitted to the LLP, the adapter moves the QP to CONNECT_PENDING state.
+ * No synchronous reply from adapter to this WR. The results of
+ * connection are passed back in an async event CCAE_ACTIVE_CONNECT_RESULTS
+ * See c2wr_ae_active_connect_results_t
+ */
+struct c2wr_qp_connect_req {
+ struct c2wr_hdr hdr;
+ u32 rnic_handle;
+ u32 qp_handle;
+ u32 remote_addr;
+ u16 remote_port;
+ u16 pad;
+ u32 private_data_length;
+ u8 private_data[0]; /* Private data in-line. */
+} __attribute__((packed)) ;
+
+struct c2wr_qp_connect {
+ struct c2wr_qp_connect_req req;
+ /* no synchronous reply. */
+} __attribute__((packed)) ;
+
+
+/*
+ *------------------------ MM ------------------------
+ */
+
+struct c2wr_nsmr_stag_alloc_req {
+ struct c2wr_hdr hdr;
+ u32 rnic_handle;
+ u32 pbl_depth;
+ u32 pd_id;
+ u32 flags;
+} __attribute__((packed)) ;
+
+struct c2wr_nsmr_stag_alloc_rep {
+ struct c2wr_hdr hdr;
+ u32 pbl_depth;
+ u32 stag_index;
+} __attribute__((packed)) ;
+
+union c2wr_nsmr_stag_alloc {
+ struct c2wr_nsmr_stag_alloc_req req;
+ struct c2wr_nsmr_stag_alloc_rep rep;
+} __attribute__((packed)) ;
+
+struct c2wr_nsmr_register_req {
+ struct c2wr_hdr hdr;
+ u64 va;
+ u32 rnic_handle;
+ u16 flags;
+ u8 stag_key;
+ u8 pad;
+ u32 pd_id;
+ u32 pbl_depth;
+ u32 pbe_size;
+ u32 fbo;
+ u32 length;
+ u32 addrs_length;
+ /* array of paddrs (must be aligned on a 64bit boundary) */
+ u64 paddrs[0];
+} __attribute__((packed)) ;
+
+struct c2wr_nsmr_register_rep {
+ struct c2wr_hdr hdr;
+ u32 pbl_depth;
+ u32 stag_index;
+} __attribute__((packed)) ;
+
+union c2wr_nsmr_register {
+ struct c2wr_nsmr_register_req req;
+ struct c2wr_nsmr_register_rep rep;
+} __attribute__((packed)) ;
+
+struct c2wr_nsmr_pbl_req {
+ struct c2wr_hdr hdr;
+ u32 rnic_handle;
+ u32 flags;
+ u32 stag_index;
+ u32 addrs_length;
+ /* array of paddrs (must be aligned on a 64bit boundary) */
+ u64 paddrs[0];
+} __attribute__((packed)) ;
+
+struct c2wr_nsmr_pbl_rep {
+ struct c2wr_hdr hdr;
+} __attribute__((packed)) ;
+
+union c2wr_nsmr_pbl {
+ struct c2wr_nsmr_pbl_req req;
+ struct c2wr_nsmr_pbl_rep rep;
+} __attribute__((packed)) ;
+
+struct c2wr_mr_query_req {
+ struct c2wr_hdr hdr;
+ u32 rnic_handle;
+ u32 stag_index;
+} __attribute__((packed)) ;
+
+struct c2wr_mr_query_rep {
+ struct c2wr_hdr hdr;
+ u8 stag_key;
+ u8 pad[3];
+ u32 pd_id;
+ u32 flags;
+ u32 pbl_depth;
+} __attribute__((packed)) ;
+
+union c2wr_mr_query {
+ struct c2wr_mr_query_req req;
+ struct c2wr_mr_query_rep rep;
+} __attribute__((packed)) ;
+
+struct c2wr_mw_query_req {
+ struct c2wr_hdr hdr;
+ u32 rnic_handle;
+ u32 stag_index;
+} __attribute__((packed)) ;
+
+struct c2wr_mw_query_rep {
+ struct c2wr_hdr hdr;
+ u8 stag_key;
+ u8 pad[3];
+ u32 pd_id;
+ u32 flags;
+} __attribute__((packed)) ;
+
+union c2wr_mw_query {
+ struct c2wr_mw_query_req req;
+ struct c2wr_mw_query_rep rep;
+} __attribute__((packed)) ;
+
+
+struct c2wr_stag_dealloc_req {
+ struct c2wr_hdr hdr;
+ u32 rnic_handle;
+ u32 stag_index;
+} __attribute__((packed)) ;
+
+struct c2wr_stag_dealloc_rep {
+ struct c2wr_hdr hdr;
+} __attribute__((packed)) ;
+
+union c2wr_stag_dealloc {
+ struct c2wr_stag_dealloc_req req;
+ struct c2wr_stag_dealloc_rep rep;
+} __attribute__((packed)) ;
+
+struct c2wr_nsmr_reregister_req {
+ struct c2wr_hdr hdr;
+ u64 va;
+ u32 rnic_handle;
+ u16 flags;
+ u8 stag_key;
+ u8 pad;
+ u32 stag_index;
+ u32 pd_id;
+ u32 pbl_depth;
+ u32 pbe_size;
+ u32 fbo;
+ u32 length;
+ u32 addrs_length;
+ u32 pad1;
+ /* array of paddrs (must be aligned on a 64bit boundary) */
+ u64 paddrs[0];
+} __attribute__((packed)) ;
+
+struct c2wr_nsmr_reregister_rep {
+ struct c2wr_hdr hdr;
+ u32 pbl_depth;
+ u32 stag_index;
+} __attribute__((packed)) ;
+
+union c2wr_nsmr_reregister {
+ struct c2wr_nsmr_reregister_req req;
+ struct c2wr_nsmr_reregister_rep rep;
+} __attribute__((packed)) ;
+
+struct c2wr_smr_register_req {
+ struct c2wr_hdr hdr;
+ u64 va;
+ u32 rnic_handle;
+ u16 flags;
+ u8 stag_key;
+ u8 pad;
+ u32 stag_index;
+ u32 pd_id;
+} __attribute__((packed)) ;
+
+struct c2wr_smr_register_rep {
+ struct c2wr_hdr hdr;
+ u32 stag_index;
+} __attribute__((packed)) ;
+
+union c2wr_smr_register {
+ struct c2wr_smr_register_req req;
+ struct c2wr_smr_register_rep rep;
+} __attribute__((packed)) ;
+
+struct c2wr_mw_alloc_req {
+ struct c2wr_hdr hdr;
+ u32 rnic_handle;
+ u32 pd_id;
+} __attribute__((packed)) ;
+
+struct c2wr_mw_alloc_rep {
+ struct c2wr_hdr hdr;
+ u32 stag_index;
+} __attribute__((packed)) ;
+
+union c2wr_mw_alloc {
+ struct c2wr_mw_alloc_req req;
+ struct c2wr_mw_alloc_rep rep;
+} __attribute__((packed)) ;
+
+/*
+ *------------------------ WRs -----------------------
+ */
+
+struct c2wr_user_hdr {
+ struct c2wr_hdr hdr; /* Has status and WR Type */
+} __attribute__((packed)) ;
+
+enum c2_qp_state {
+ C2_QP_STATE_IDLE = 0x01,
+ C2_QP_STATE_CONNECTING = 0x02,
+ C2_QP_STATE_RTS = 0x04,
+ C2_QP_STATE_CLOSING = 0x08,
+ C2_QP_STATE_TERMINATE = 0x10,
+ C2_QP_STATE_ERROR = 0x20,
+};
+
+/* Completion queue entry. */
+struct c2wr_ce {
+ struct c2wr_hdr hdr; /* Has status and WR Type */
+ u64 qp_user_context; /* c2_user_qp_t * */
+ u32 qp_state; /* Current QP State */
+ u32 handle; /* QPID or EP Handle */
+ u32 bytes_rcvd; /* valid for RECV WCs */
+ u32 stag;
+} __attribute__((packed)) ;
+
+
+/*
+ * Flags used for all post-sq WRs. These must fit in the flags
+ * field of the struct c2wr_hdr (eight bits).
+ */
+enum {
+ SQ_SIGNALED = 0x01,
+ SQ_READ_FENCE = 0x02,
+ SQ_FENCE = 0x04,
+};
+
+/*
+ * Common fields for all post-sq WRs. Namely the standard header and a
+ * secondary header with fields common to all post-sq WRs.
+ */
+struct c2_sq_hdr {
+ struct c2wr_user_hdr user_hdr;
+} __attribute__((packed));
+
+/*
+ * Same as above but for post-rq WRs.
+ */
+struct c2_rq_hdr {
+ struct c2wr_user_hdr user_hdr;
+} __attribute__((packed));
+
+/*
+ * use the same struct for all sends.
+ */
+struct c2wr_send_req {
+ struct c2_sq_hdr sq_hdr;
+ u32 sge_len;
+ u32 remote_stag;
+ u8 data[0]; /* SGE array */
+} __attribute__((packed));
+/* XXX c2wr_send_req_t, c2wr_send_se_req_t, c2wr_send_inv_req_t,
+ c2wr_send_se_inv_req_t;*/
+
+union c2wr_send {
+ struct c2wr_send_req req;
+ struct c2wr_ce rep;
+} __attribute__((packed));
+
+struct c2wr_rdma_write_req {
+ struct c2_sq_hdr sq_hdr;
+ u64 remote_to;
+ u32 remote_stag;
+ u32 sge_len;
+ u8 data[0]; /* SGE array */
+} __attribute__((packed));
+
+union c2wr_rdma_write {
+ struct c2wr_rdma_write_req req;
+ struct c2wr_ce rep;
+} __attribute__((packed));
+
+struct c2wr_rdma_read_req {
+ struct c2_sq_hdr sq_hdr;
+ u64 local_to;
+ u64 remote_to;
+ u32 local_stag;
+ u32 remote_stag;
+ u32 length;
+} __attribute__((packed));
+
+union c2wr_rdma_read {
+ struct c2wr_rdma_read_req req;
+ struct c2wr_ce rep;
+} __attribute__((packed));
+
+struct c2wr_mw_bind_req {
+ struct c2_sq_hdr sq_hdr;
+ u64 va;
+ u8 stag_key;
+ u8 pad[3];
+ u32 mw_stag_index;
+ u32 mr_stag_index;
+ u32 length;
+ u32 flags;
+} __attribute__((packed));
+
+union c2wr_mw_bind {
+ struct c2wr_mw_bind_req req;
+ struct c2wr_ce rep;
+} __attribute__((packed));
+
+struct c2wr_nsmr_fastreg_req {
+ struct c2_sq_hdr sq_hdr;
+ u64 va;
+ u8 stag_key;
+ u8 pad[3];
+ u32 stag_index;
+ u32 pbe_size;
+ u32 fbo;
+ u32 length;
+ u32 addrs_length;
+ /* array of paddrs (must be aligned on a 64bit boundary) */
+ u64 paddrs[0];
+} __attribute__((packed));
+
+union c2wr_nsmr_fastreg {
+ struct c2wr_nsmr_fastreg_req req;
+ struct c2wr_ce rep;
+} __attribute__((packed));
+
+struct c2wr_stag_invalidate_req {
+ struct c2_sq_hdr sq_hdr;
+ u8 stag_key;
+ u8 pad[3];
+ u32 stag_index;
+} __attribute__((packed));
+
+union c2wr_stag_invalidate {
+ struct c2wr_stag_invalidate_req req;
+ struct c2wr_ce rep;
+} __attribute__((packed));
+
+union c2wr_sqwr {
+ struct c2_sq_hdr sq_hdr;
+ struct c2wr_send_req send;
+ struct c2wr_send_req send_se;
+ struct c2wr_send_req send_inv;
+ struct c2wr_send_req send_se_inv;
+ struct c2wr_rdma_write_req rdma_write;
+ struct c2wr_rdma_read_req rdma_read;
+ struct c2wr_mw_bind_req mw_bind;
+ struct c2wr_nsmr_fastreg_req nsmr_fastreg;
+ struct c2wr_stag_invalidate_req stag_inv;
+} __attribute__((packed));
+
+
+/*
+ * RQ WRs
+ */
+struct c2wr_rqwr {
+ struct c2_rq_hdr rq_hdr;
+ u8 data[0]; /* array of SGEs */
+} __attribute__((packed));
+/* XXX c2wr_rqwr_t, c2wr_recv_req_t; */
+
+union c2wr_recv {
+ struct c2wr_rqwr req;
+ struct c2wr_ce rep;
+} __attribute__((packed));
+
+/*
+ * All AEs start with this header. Most AEs only need to convey the
+ * information in the header. Some, like LLP connection events, need
+ * more info. The union typdef c2wr_ae_t has all the possible AEs.
+ *
+ * hdr.context is the user_context from the rnic_open WR. NULL If this
+ * is not affiliated with an rnic
+ *
+ * hdr.id is the AE identifier (eg; CCAE_REMOTE_SHUTDOWN,
+ * CCAE_LLP_CLOSE_COMPLETE)
+ *
+ * resource_type is one of: C2_RES_IND_QP, C2_RES_IND_CQ, C2_RES_IND_SRQ
+ *
+ * user_context is the context passed down when the host created the resource.
+ */
+struct c2wr_ae_hdr {
+ struct c2wr_hdr hdr;
+ u64 user_context; /* user context for this res. */
+ u32 resource_type; /* see enum c2_resource_indicator */
+ u32 resource; /* handle for resource */
+ u32 qp_state; /* current QP State */
+} __attribute__((packed));
+
+/*
+ * After submitting the CCAE_ACTIVE_CONNECT_RESULTS message on the AEQ,
+ * the adapter moves the QP into RTS state
+ */
+struct c2wr_ae_active_connect_results {
+ struct c2wr_ae_hdr ae_hdr;
+ u32 laddr;
+ u32 raddr;
+ u16 lport;
+ u16 rport;
+ u32 private_data_length;
+ u8 private_data[0]; /* data is in-line in the msg. */
+} __attribute__((packed));
+
+/*
+ * When connections are established by the stack (and the private data
+ * MPA frame is received), the adapter will generate an event to the host.
+ * The details of the connection, any private data, and the new connection
+ * request handle is passed up via the CCAE_CONNECTION_REQUEST msg on the
+ * AE queue:
+ */
+struct c2wr_ae_connection_request {
+ struct c2wr_ae_hdr ae_hdr;
+ u32 cr_handle; /* connreq handle (sock ptr) */
+ u32 laddr;
+ u32 raddr;
+ u16 lport;
+ u16 rport;
+ u32 private_data_length;
+ u8 private_data[0]; /* data is in-line in the msg. */
+} __attribute__((packed));
+
+union c2wr_ae {
+ struct c2wr_ae_hdr ae_generic;
+ struct c2wr_ae_active_connect_results ae_active_connect_results;
+ struct c2wr_ae_connection_request ae_connection_request;
+} __attribute__((packed));
+
+struct c2wr_init_req {
+ struct c2wr_hdr hdr;
+ u64 hint_count;
+ u64 q0_host_shared;
+ u64 q1_host_shared;
+ u64 q1_host_msg_pool;
+ u64 q2_host_shared;
+ u64 q2_host_msg_pool;
+} __attribute__((packed));
+
+struct c2wr_init_rep {
+ struct c2wr_hdr hdr;
+} __attribute__((packed));
+
+union c2wr_init {
+ struct c2wr_init_req req;
+ struct c2wr_init_rep rep;
+} __attribute__((packed));
+
+/*
+ * For upgrading flash.
+ */
+
+struct c2wr_flash_init_req {
+ struct c2wr_hdr hdr;
+ u32 rnic_handle;
+} __attribute__((packed));
+
+struct c2wr_flash_init_rep {
+ struct c2wr_hdr hdr;
+ u32 adapter_flash_buf_offset;
+ u32 adapter_flash_len;
+} __attribute__((packed));
+
+union c2wr_flash_init {
+ struct c2wr_flash_init_req req;
+ struct c2wr_flash_init_rep rep;
+} __attribute__((packed));
+
+struct c2wr_flash_req {
+ struct c2wr_hdr hdr;
+ u32 rnic_handle;
+ u32 len;
+} __attribute__((packed));
+
+struct c2wr_flash_rep {
+ struct c2wr_hdr hdr;
+ u32 status;
+} __attribute__((packed));
+
+union c2wr_flash {
+ struct c2wr_flash_req req;
+ struct c2wr_flash_rep rep;
+} __attribute__((packed));
+
+struct c2wr_buf_alloc_req {
+ struct c2wr_hdr hdr;
+ u32 rnic_handle;
+ u32 size;
+} __attribute__((packed));
+
+struct c2wr_buf_alloc_rep {
+ struct c2wr_hdr hdr;
+ u32 offset; /* 0 if mem not available */
+ u32 size; /* 0 if mem not available */
+} __attribute__((packed));
+
+union c2wr_buf_alloc {
+ struct c2wr_buf_alloc_req req;
+ struct c2wr_buf_alloc_rep rep;
+} __attribute__((packed));
+
+struct c2wr_buf_free_req {
+ struct c2wr_hdr hdr;
+ u32 rnic_handle;
+ u32 offset; /* Must match value from alloc */
+ u32 size; /* Must match value from alloc */
+} __attribute__((packed));
+
+struct c2wr_buf_free_rep {
+ struct c2wr_hdr hdr;
+} __attribute__((packed));
+
+union c2wr_buf_free {
+ struct c2wr_buf_free_req req;
+ struct c2wr_ce rep;
+} __attribute__((packed));
+
+struct c2wr_flash_write_req {
+ struct c2wr_hdr hdr;
+ u32 rnic_handle;
+ u32 offset;
+ u32 size;
+ u32 type;
+ u32 flags;
+} __attribute__((packed));
+
+struct c2wr_flash_write_rep {
+ struct c2wr_hdr hdr;
+ u32 status;
+} __attribute__((packed));
+
+union c2wr_flash_write {
+ struct c2wr_flash_write_req req;
+ struct c2wr_flash_write_rep rep;
+} __attribute__((packed));
+
+/*
+ * Messages for LLP connection setup.
+ */
+
+/*
+ * Listen Request. This allocates a listening endpoint to allow passive
+ * connection setup. Newly established LLP connections are passed up
+ * via an AE. See c2wr_ae_connection_request_t
+ */
+struct c2wr_ep_listen_create_req {
+ struct c2wr_hdr hdr;
+ u64 user_context; /* returned in AEs. */
+ u32 rnic_handle;
+ u32 local_addr; /* local addr, or 0 */
+ u16 local_port; /* 0 means "pick one" */
+ u16 pad;
+ u32 backlog; /* tradional tcp listen bl */
+} __attribute__((packed));
+
+struct c2wr_ep_listen_create_rep {
+ struct c2wr_hdr hdr;
+ u32 ep_handle; /* handle to new listening ep */
+ u16 local_port; /* resulting port... */
+ u16 pad;
+} __attribute__((packed));
+
+union c2wr_ep_listen_create {
+ struct c2wr_ep_listen_create_req req;
+ struct c2wr_ep_listen_create_rep rep;
+} __attribute__((packed));
+
+struct c2wr_ep_listen_destroy_req {
+ struct c2wr_hdr hdr;
+ u32 rnic_handle;
+ u32 ep_handle;
+} __attribute__((packed));
+
+struct c2wr_ep_listen_destroy_rep {
+ struct c2wr_hdr hdr;
+} __attribute__((packed));
+
+union c2wr_ep_listen_destroy {
+ struct c2wr_ep_listen_destroy_req req;
+ struct c2wr_ep_listen_destroy_rep rep;
+} __attribute__((packed));
+
+struct c2wr_ep_query_req {
+ struct c2wr_hdr hdr;
+ u32 rnic_handle;
+ u32 ep_handle;
+} __attribute__((packed));
+
+struct c2wr_ep_query_rep {
+ struct c2wr_hdr hdr;
+ u32 rnic_handle;
+ u32 local_addr;
+ u32 remote_addr;
+ u16 local_port;
+ u16 remote_port;
+} __attribute__((packed));
+
+union c2wr_ep_query {
+ struct c2wr_ep_query_req req;
+ struct c2wr_ep_query_rep rep;
+} __attribute__((packed));
+
+
+/*
+ * The host passes this down to indicate acceptance of a pending iWARP
+ * connection. The cr_handle was obtained from the CONNECTION_REQUEST
+ * AE passed up by the adapter. See c2wr_ae_connection_request_t.
+ */
+struct c2wr_cr_accept_req {
+ struct c2wr_hdr hdr;
+ u32 rnic_handle;
+ u32 qp_handle; /* QP to bind to this LLP conn */
+ u32 ep_handle; /* LLP handle to accept */
+ u32 private_data_length;
+ u8 private_data[0]; /* data in-line in msg. */
+} __attribute__((packed));
+
+/*
+ * adapter sends reply when private data is successfully submitted to
+ * the LLP.
+ */
+struct c2wr_cr_accept_rep {
+ struct c2wr_hdr hdr;
+} __attribute__((packed));
+
+union c2wr_cr_accept {
+ struct c2wr_cr_accept_req req;
+ struct c2wr_cr_accept_rep rep;
+} __attribute__((packed));
+
+/*
+ * The host sends this down if a given iWARP connection request was
+ * rejected by the consumer. The cr_handle was obtained from a
+ * previous c2wr_ae_connection_request_t AE sent by the adapter.
+ */
+struct c2wr_cr_reject_req {
+ struct c2wr_hdr hdr;
+ u32 rnic_handle;
+ u32 ep_handle; /* LLP handle to reject */
+} __attribute__((packed));
+
+/*
+ * Dunno if this is needed, but we'll add it for now. The adapter will
+ * send the reject_reply after the LLP endpoint has been destroyed.
+ */
+struct c2wr_cr_reject_rep {
+ struct c2wr_hdr hdr;
+} __attribute__((packed));
+
+union c2wr_cr_reject {
+ struct c2wr_cr_reject_req req;
+ struct c2wr_cr_reject_rep rep;
+} __attribute__((packed));
+
+/*
+ * console command. Used to implement a debug console over the verbs
+ * request and reply queues.
+ */
+
+/*
+ * Console request message. It contains:
+ * - message hdr with id = CCWR_CONSOLE
+ * - the physaddr/len of host memory to be used for the reply.
+ * - the command string. eg: "netstat -s" or "zoneinfo"
+ */
+struct c2wr_console_req {
+ struct c2wr_hdr hdr; /* id = CCWR_CONSOLE */
+ u64 reply_buf; /* pinned host buf for reply */
+ u32 reply_buf_len; /* length of reply buffer */
+ u8 command[0]; /* NUL terminated ascii string */
+ /* containing the command req */
+} __attribute__((packed));
+
+/*
+ * flags used in the console reply.
+ */
+enum c2_console_flags {
+ CONS_REPLY_TRUNCATED = 0x00000001 /* reply was truncated */
+} __attribute__((packed));
+
+/*
+ * Console reply message.
+ * hdr.result contains the c2_status_t error if the reply was _not_ generated,
+ * or C2_OK if the reply was generated.
+ */
+struct c2wr_console_rep {
+ struct c2wr_hdr hdr; /* id = CCWR_CONSOLE */
+ u32 flags;
+} __attribute__((packed));
+
+union c2wr_console {
+ struct c2wr_console_req req;
+ struct c2wr_console_rep rep;
+} __attribute__((packed));
+
+
+/*
+ * Giant union with all WRs. Makes life easier...
+ */
+union c2wr {
+ struct c2wr_hdr hdr;
+ struct c2wr_user_hdr user_hdr;
+ union c2wr_rnic_open rnic_open;
+ union c2wr_rnic_query rnic_query;
+ union c2wr_rnic_getconfig rnic_getconfig;
+ union c2wr_rnic_setconfig rnic_setconfig;
+ union c2wr_rnic_close rnic_close;
+ union c2wr_cq_create cq_create;
+ union c2wr_cq_modify cq_modify;
+ union c2wr_cq_destroy cq_destroy;
+ union c2wr_pd_alloc pd_alloc;
+ union c2wr_pd_dealloc pd_dealloc;
+ union c2wr_srq_create srq_create;
+ union c2wr_srq_destroy srq_destroy;
+ union c2wr_qp_create qp_create;
+ union c2wr_qp_query qp_query;
+ union c2wr_qp_modify qp_modify;
+ union c2wr_qp_destroy qp_destroy;
+ struct c2wr_qp_connect qp_connect;
+ union c2wr_nsmr_stag_alloc nsmr_stag_alloc;
+ union c2wr_nsmr_register nsmr_register;
+ union c2wr_nsmr_pbl nsmr_pbl;
+ union c2wr_mr_query mr_query;
+ union c2wr_mw_query mw_query;
+ union c2wr_stag_dealloc stag_dealloc;
+ union c2wr_sqwr sqwr;
+ struct c2wr_rqwr rqwr;
+ struct c2wr_ce ce;
+ union c2wr_ae ae;
+ union c2wr_init init;
+ union c2wr_ep_listen_create ep_listen_create;
+ union c2wr_ep_listen_destroy ep_listen_destroy;
+ union c2wr_cr_accept cr_accept;
+ union c2wr_cr_reject cr_reject;
+ union c2wr_console console;
+ union c2wr_flash_init flash_init;
+ union c2wr_flash flash;
+ union c2wr_buf_alloc buf_alloc;
+ union c2wr_buf_free buf_free;
+ union c2wr_flash_write flash_write;
+} __attribute__((packed));
+
+
+/*
+ * Accessors for the wr fields that are packed together tightly to
+ * reduce the wr message size. The wr arguments are void* so that
+ * either a struct c2wr*, a struct c2wr_hdr*, or a pointer to any of the types
+ * in the struct c2wr union can be passed in.
+ */
+static __inline__ u8 c2_wr_get_id(void *wr)
+{
+ return ((struct c2wr_hdr *) wr)->id;
+}
+static __inline__ void c2_wr_set_id(void *wr, u8 id)
+{
+ ((struct c2wr_hdr *) wr)->id = id;
+}
+static __inline__ u8 c2_wr_get_result(void *wr)
+{
+ return ((struct c2wr_hdr *) wr)->result;
+}
+static __inline__ void c2_wr_set_result(void *wr, u8 result)
+{
+ ((struct c2wr_hdr *) wr)->result = result;
+}
+static __inline__ u8 c2_wr_get_flags(void *wr)
+{
+ return ((struct c2wr_hdr *) wr)->flags;
+}
+static __inline__ void c2_wr_set_flags(void *wr, u8 flags)
+{
+ ((struct c2wr_hdr *) wr)->flags = flags;
+}
+static __inline__ u8 c2_wr_get_sge_count(void *wr)
+{
+ return ((struct c2wr_hdr *) wr)->sge_count;
+}
+static __inline__ void c2_wr_set_sge_count(void *wr, u8 sge_count)
+{
+ ((struct c2wr_hdr *) wr)->sge_count = sge_count;
+}
+static __inline__ u32 c2_wr_get_wqe_count(void *wr)
+{
+ return ((struct c2wr_hdr *) wr)->wqe_count;
+}
+static __inline__ void c2_wr_set_wqe_count(void *wr, u32 wqe_count)
+{
+ ((struct c2wr_hdr *) wr)->wqe_count = wqe_count;
+}
+
+#endif /* _C2_WR_H_ */
[-- Attachment #3: Type: text/plain, Size: 0 bytes --]
^ permalink raw reply related
* Re: [PATCH 2/2] iWARP Core Changes.
From: Sean Hefty @ 2006-05-31 21:26 UTC (permalink / raw)
To: Steve Wise; +Cc: rdreier, linux-kernel, netdev, openib-general
In-Reply-To: <20060531182654.3308.41372.stgit@stevo-desktop>
Mainly nits...
Steve Wise wrote:
> -static int copy_addr(struct rdma_dev_addr *dev_addr, struct net_device *dev,
> +int copy_addr(struct rdma_dev_addr *dev_addr, struct net_device *dev,
> unsigned char *dst_dev_addr)
Might want to rename this to something like rdma_copy_addr if you're going to
export it.
> +static int cma_iw_handler(struct iw_cm_id *iw_id, struct iw_cm_event *iw_event)
> +{
> + struct rdma_id_private *id_priv = iw_id->context;
> + enum rdma_cm_event_type event = 0;
> + struct sockaddr_in *sin;
> + int ret = 0;
> +
> + atomic_inc(&id_priv->dev_remove);
> +
> + switch (iw_event->event) {
> + case IW_CM_EVENT_CLOSE:
> + event = RDMA_CM_EVENT_DISCONNECTED;
> + break;
> + case IW_CM_EVENT_CONNECT_REPLY:
> + sin = (struct sockaddr_in*)&id_priv->id.route.addr.src_addr;
> + *sin = iw_event->local_addr;
> + sin = (struct sockaddr_in*)&id_priv->id.route.addr.dst_addr;
spacing nit - (struct sockaddr_in *) &id_priv->...
> +struct net_device *ip_dev_find(u32 ip);
Just include header file with definition.
> + sin = (struct sockaddr_in*)&new_cm_id->route.addr.src_addr;
> + *sin = iw_event->local_addr;
> + sin = (struct sockaddr_in*)&new_cm_id->route.addr.dst_addr;
same spacing nit... appears in a couple other places as well.
> +static inline union ib_gid* iw_addr_get_sgid(struct rdma_dev_addr* rda)
> +{
> + return (union ib_gid*)rda->src_dev_addr;
> +}
> +
> +static inline union ib_gid* iw_addr_get_dgid(struct rdma_dev_addr* rda)
> +{
> + return (union ib_gid*)rda->dst_dev_addr;
> +}
spacing nits
> +struct iw_cm_verbs;
> struct ib_device {
> struct device *dma_device;
>
> @@ -846,6 +873,8 @@ struct ib_device {
>
> u32 flags;
>
> + struct iw_cm_verbs* iwcm;
> +
'*' placement nit
- Sean
^ permalink raw reply
* DPRINTKs in e1000 code (2.6.x kernels)
From: Amit K Arora @ 2006-05-31 9:01 UTC (permalink / raw)
To: netdev; +Cc: auke-jan.h.kok
Hello,
I am wondering why this patch "e1000: Remove PM warning DPRINTKs
breaking 2.4.x kernels"
(http://www.mail-archive.com/netdev@vger.kernel.org/msg10803.html) was
meant only for 2.4 kernels, and _not_ for 2.6 kernels.
These DPRINTKs in e1000_suspend() are currently resulting in following
messages while rebooting a system (2.6 kernel) which has e1000 adapter:
Shutting down MD Raid
done
Stopping udevd: done
proc umounted
The system will be halted immediately.
md: stopping all md devices.
e1000: eth3: e1000_suspend: Error enabling D3 wake <<==
e1000: eth3: e1000_suspend: Error enabling D3 cold wake <<==
Power down.
Should these DPRINTKs be removed from the 2.6.x e1000 code as well ?
Regards,
Amit Arora
^ permalink raw reply
* Re: TCP connection hang problem with 2.6.16.16, e1000 (found work-around)
From: Ben Greear @ 2006-05-31 21:45 UTC (permalink / raw)
Cc: NetDev
In-Reply-To: <447DF085.2050604@candelatech.com>
Ok, I think I found the problem. It appears to be at least partially
self-inflicted. I was modifying the kernel buffer settings via /proc/*
to improve performance. I noticed that when I used kernel-defaults,
the TCP connection hangs stopped (and ran ~20% slower). My personal opinion
is that this is still a bug somewhere in TCP, but at least I have a work-around.
Since these settings appear somewhat delicate, it might be nice if
the kernel could warn about configurations that could cause TCP deadlocks
and such.
My old settings that exhibitted the problem were:
my $netdev_max_backlog = 5000;
my $wmem_max = 4096000;
my $wmem_default = 128000;
my $rmem_max = 8096000;
my $rmem_default = 128000;
my $tcp_rmem_min = 4096;
my $tcp_rmem_default = 256000;
my $tcp_rmem_max = 30000000;
my $tcp_wmem_min = 4096;
my $tcp_wmem_default = 256000;
my $tcp_wmem_max = 30000000;
my $tcp_mem_lo = 20000000;
my $tcp_mem_pressure = 30000000;
my $tcp_mem_high = 60000000;
The new settings that appear to work around the TCP connection hang,
and still give decent performance are below. Documentation being thin,
I used the suggested values in Documentation/networking/ixgb.txt as
hints:
my $netdev_max_backlog = 30000;
my $wmem_max = 524287;
my $wmem_default = 128000;
my $rmem_max = 524287;
my $rmem_default = 128000;
my $tcp_rmem_min = 4096;
my $tcp_rmem_default = 256000;
my $tcp_rmem_max = 5000000;
my $tcp_wmem_min = 4096;
my $tcp_wmem_default = 256000;
my $tcp_wmem_max = 5000000;
my $tcp_mem_lo = 10000000;
my $tcp_mem_pressure = 20000000;
my $tcp_mem_high = 30000000;
Thanks,
Ben
--
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc http://www.candelatech.com
^ permalink raw reply
* Re: TCP connection hang problem with 2.6.16.16, e1000 (found work-around)
From: Stephen Hemminger @ 2006-05-31 22:10 UTC (permalink / raw)
To: Ben Greear; +Cc: NetDev
In-Reply-To: <447E0E69.3080006@candelatech.com>
On Wed, 31 May 2006 14:45:13 -0700
Ben Greear <greearb@candelatech.com> wrote:
> Ok, I think I found the problem. It appears to be at least partially
> self-inflicted. I was modifying the kernel buffer settings via /proc/*
> to improve performance. I noticed that when I used kernel-defaults,
> the TCP connection hangs stopped (and ran ~20% slower). My personal opinion
> is that this is still a bug somewhere in TCP, but at least I have a work-around.
If you set bogus values, you can end up consuming all the memory on your system.
> Since these settings appear somewhat delicate, it might be nice if
> the kernel could warn about configurations that could cause TCP deadlocks
> and such.
>
There is little or no range checking on most all sysctl values. Whether or
not that is a good thing is debatable.
> My old settings that exhibitted the problem were:
>
> my $netdev_max_backlog = 5000;
This only matters for non-NAPI devices.
> my $wmem_max = 4096000;
> my $wmem_default = 128000;
> my $rmem_max = 8096000;
> my $rmem_default = 128000;
>
> my $tcp_rmem_min = 4096;
> my $tcp_rmem_default = 256000;
Receiver auto-tuning makes setting this unnecessary
in most cases.
> my $tcp_rmem_max = 30000000;
>
> my $tcp_wmem_min = 4096;
> my $tcp_wmem_default = 256000;
That's too big with current auto-tuning.
> my $tcp_wmem_max = 30000000;
>
> my $tcp_mem_lo = 20000000;
> my $tcp_mem_pressure = 30000000;
> my $tcp_mem_high = 60000000;
>
>
> The new settings that appear to work around the TCP connection hang,
> and still give decent performance are below. Documentation being thin,
> I used the suggested values in Documentation/networking/ixgb.txt as
> hints:
Settings don't belong in this Documentation file because it a general
thing not specific to that hardware.
> my $netdev_max_backlog = 30000;
> my $wmem_max = 524287;
> my $wmem_default = 128000;
> my $rmem_max = 524287;
> my $rmem_default = 128000;
>
> my $tcp_rmem_min = 4096;
> my $tcp_rmem_default = 256000;
> my $tcp_rmem_max = 5000000;
>
> my $tcp_wmem_min = 4096;
> my $tcp_wmem_default = 256000;
> my $tcp_wmem_max = 5000000;
>
> my $tcp_mem_lo = 10000000;
> my $tcp_mem_pressure = 20000000;
> my $tcp_mem_high = 30000000;
>
> Thanks,
> Ben
>
^ permalink raw reply
* Re: [PATCH 1/2] iWARP Connection Manager.
From: Sean Hefty @ 2006-05-31 22:22 UTC (permalink / raw)
To: Steve Wise; +Cc: rdreier, linux-kernel, netdev, openib-general
In-Reply-To: <20060531182652.3308.1244.stgit@stevo-desktop>
Steve Wise wrote:
> +/*
> + * Release a reference on cm_id. If the last reference is being removed
> + * and iw_destroy_cm_id is waiting, wake up the waiting thread.
> + */
> +static int iwcm_deref_id(struct iwcm_id_private *cm_id_priv)
> +{
> + int ret = 0;
> +
> + BUG_ON(atomic_read(&cm_id_priv->refcount)==0);
> + if (atomic_dec_and_test(&cm_id_priv->refcount)) {
> + BUG_ON(!list_empty(&cm_id_priv->work_list));
> + if (waitqueue_active(&cm_id_priv->destroy_wait)) {
> + BUG_ON(cm_id_priv->state != IW_CM_STATE_DESTROYING);
> + BUG_ON(test_bit(IWCM_F_CALLBACK_DESTROY,
> + &cm_id_priv->flags));
> + ret = 1;
> + wake_up(&cm_id_priv->destroy_wait);
We recently changed the RDMA CM, IB CM, and a couple of other modules from using
wait objects to completions. This avoids a race condition between decrementing
the reference count, which allows destruction to proceed, and calling wake_up on
a freed cm_id. My guess is that you may need to do the same.
Can you also explain the use of the return value here? It's ignored below in
rem_ref() and destroy_cm_id().
> +static void add_ref(struct iw_cm_id *cm_id)
> +{
> + struct iwcm_id_private *cm_id_priv;
> + cm_id_priv = container_of(cm_id, struct iwcm_id_private, id);
> + atomic_inc(&cm_id_priv->refcount);
> +}
> +
> +static void rem_ref(struct iw_cm_id *cm_id)
> +{
> + struct iwcm_id_private *cm_id_priv;
> + cm_id_priv = container_of(cm_id, struct iwcm_id_private, id);
> + iwcm_deref_id(cm_id_priv);
> +}
> +
> +/*
> + * CM_ID <-- CLOSING
> + *
> + * Block if a passive or active connection is currenlty being processed. Then
> + * process the event as follows:
> + * - If we are ESTABLISHED, move to CLOSING and modify the QP state
> + * based on the abrupt flag
> + * - If the connection is already in the CLOSING or IDLE state, the peer is
> + * disconnecting concurrently with us and we've already seen the
> + * DISCONNECT event -- ignore the request and return 0
> + * - Disconnect on a listening endpoint returns -EINVAL
> + */
> +int iw_cm_disconnect(struct iw_cm_id *cm_id, int abrupt)
> +{
> + struct iwcm_id_private *cm_id_priv;
> + unsigned long flags;
> + int ret = 0;
> +
> + cm_id_priv = container_of(cm_id, struct iwcm_id_private, id);
> + /* Wait if we're currently in a connect or accept downcall */
> + wait_event(cm_id_priv->connect_wait,
> + !test_bit(IWCM_F_CONNECT_WAIT, &cm_id_priv->flags));
Am I understanding this check correctly? You're checking to see if the user has
called iw_cm_disconnect() at the same time that they called iw_cm_connect() or
iw_cm_accept(). Are connect / accept blocking, or are you just waiting for an
event?
> +
> + spin_lock_irqsave(&cm_id_priv->lock, flags);
> + switch (cm_id_priv->state) {
> + case IW_CM_STATE_ESTABLISHED:
> + cm_id_priv->state = IW_CM_STATE_CLOSING;
> + spin_unlock_irqrestore(&cm_id_priv->lock, flags);
> + if (cm_id_priv->qp) { /* QP could be <nul> for user-mode client */
> + if (abrupt)
> + ret = iwcm_modify_qp_err(cm_id_priv->qp);
> + else
> + ret = iwcm_modify_qp_sqd(cm_id_priv->qp);
> + /*
> + * If both sides are disconnecting the QP could
> + * already be in ERR or SQD states
> + */
> + ret = 0;
> + }
> + else
> + ret = -EINVAL;
> + break;
> + case IW_CM_STATE_LISTEN:
> + spin_unlock_irqrestore(&cm_id_priv->lock, flags);
> + ret = -EINVAL;
> + break;
> + case IW_CM_STATE_CLOSING:
> + /* remote peer closed first */
> + case IW_CM_STATE_IDLE:
> + /* accept or connect returned !0 */
> + spin_unlock_irqrestore(&cm_id_priv->lock, flags);
> + break;
> + case IW_CM_STATE_CONN_RECV:
> + /*
> + * App called disconnect before/without calling accept after
> + * connect_request event delivered.
> + */
> + spin_unlock_irqrestore(&cm_id_priv->lock, flags);
> + break;
> + case IW_CM_STATE_CONN_SENT:
> + /* Can only get here if wait above fails */
> + default:
> + BUG_ON(1);
> + }
> +
> + return ret;
> +}
> +EXPORT_SYMBOL(iw_cm_disconnect);
> +static void destroy_cm_id(struct iw_cm_id *cm_id)
> +{
> + struct iwcm_id_private *cm_id_priv;
> + unsigned long flags;
> + int ret;
> +
> + cm_id_priv = container_of(cm_id, struct iwcm_id_private, id);
> + /* Wait if we're currently in a connect or accept downcall. A
> + * listening endpoint should never block here. */
> + wait_event(cm_id_priv->connect_wait,
> + !test_bit(IWCM_F_CONNECT_WAIT, &cm_id_priv->flags));
Same question/comment as above.
> +
> + spin_lock_irqsave(&cm_id_priv->lock, flags);
> + switch (cm_id_priv->state) {
> + case IW_CM_STATE_LISTEN:
> + cm_id_priv->state = IW_CM_STATE_DESTROYING;
> + spin_unlock_irqrestore(&cm_id_priv->lock, flags);
> + /* destroy the listening endpoint */
> + ret = cm_id->device->iwcm->destroy_listen(cm_id);
> + break;
> + case IW_CM_STATE_ESTABLISHED:
> + cm_id_priv->state = IW_CM_STATE_DESTROYING;
> + spin_unlock_irqrestore(&cm_id_priv->lock, flags);
> + /* Abrupt close of the connection */
> + (void)iwcm_modify_qp_err(cm_id_priv->qp);
> + break;
> + case IW_CM_STATE_IDLE:
> + case IW_CM_STATE_CLOSING:
> + cm_id_priv->state = IW_CM_STATE_DESTROYING;
> + spin_unlock_irqrestore(&cm_id_priv->lock, flags);
> + break;
> + case IW_CM_STATE_CONN_RECV:
> + /*
> + * App called destroy before/without calling accept after
> + * receiving connection request event notification.
> + */
> + cm_id_priv->state = IW_CM_STATE_DESTROYING;
> + spin_unlock_irqrestore(&cm_id_priv->lock, flags);
> + break;
> + case IW_CM_STATE_CONN_SENT:
> + case IW_CM_STATE_DESTROYING:
> + default:
> + BUG_ON(1);
> + break;
> + }
> +
> + spin_lock_irqsave(&cm_id_priv->lock, flags);
As an alternative, you could hold the lock from above, an let the LISTEN /
ESTABLISHED state checks release and reacquire.
> + if (cm_id_priv->qp) {
> + cm_id_priv->id.device->iwcm->rem_ref(cm_id_priv->qp);
> + cm_id_priv->qp = NULL;
> + }
> + spin_unlock_irqrestore(&cm_id_priv->lock, flags);
> +
> + (void)iwcm_deref_id(cm_id_priv);
> +}
> +
> +/*
> + * This function is only called by the application thread and cannot
> + * be called by the event thread. The function will wait for all
> + * references to be released on the cm_id and then kfree the cm_id
> + * object.
> + */
> +void iw_destroy_cm_id(struct iw_cm_id *cm_id)
> +{
> + struct iwcm_id_private *cm_id_priv;
> +
> + cm_id_priv = container_of(cm_id, struct iwcm_id_private, id);
> + BUG_ON(test_bit(IWCM_F_CALLBACK_DESTROY, &cm_id_priv->flags));
> +
> + destroy_cm_id(cm_id);
> +
> + wait_event(cm_id_priv->destroy_wait,
> + !atomic_read(&cm_id_priv->refcount));
> +
> + kfree(cm_id_priv);
> +}
> +EXPORT_SYMBOL(iw_destroy_cm_id);
> +
> +/*
> + * CM_ID <-- LISTEN
> + *
> + * Start listening for connect requests. Generates one CONNECT_REQUEST
> + * event for each inbound connect request.
> + */
> +int iw_cm_listen(struct iw_cm_id *cm_id, int backlog)
> +{
> + struct iwcm_id_private *cm_id_priv;
> + unsigned long flags;
> + int ret = 0;
> +
> + cm_id_priv = container_of(cm_id, struct iwcm_id_private, id);
> + spin_lock_irqsave(&cm_id_priv->lock, flags);
> + switch (cm_id_priv->state) {
> + case IW_CM_STATE_IDLE:
> + cm_id_priv->state = IW_CM_STATE_LISTEN;
> + spin_unlock_irqrestore(&cm_id_priv->lock, flags);
> + ret = cm_id->device->iwcm->create_listen(cm_id, backlog);
> + if (ret)
> + cm_id_priv->state = IW_CM_STATE_IDLE;
> + break;
> + default:
> + spin_unlock_irqrestore(&cm_id_priv->lock, flags);
> + ret = -EINVAL;
> + }
> +
> + return ret;
> +}
> +EXPORT_SYMBOL(iw_cm_listen);
> +
> +/*
> + * CM_ID <-- IDLE
> + *
> + * Rejects an inbound connection request. No events are generated.
> + */
> +int iw_cm_reject(struct iw_cm_id *cm_id,
> + const void *private_data,
> + u8 private_data_len)
> +{
> + struct iwcm_id_private *cm_id_priv;
> + unsigned long flags;
> + int ret;
> +
> + cm_id_priv = container_of(cm_id, struct iwcm_id_private, id);
> + set_bit(IWCM_F_CONNECT_WAIT, &cm_id_priv->flags);
> +
> + spin_lock_irqsave(&cm_id_priv->lock, flags);
> + if (cm_id_priv->state != IW_CM_STATE_CONN_RECV) {
> + spin_unlock_irqrestore(&cm_id_priv->lock, flags);
> + clear_bit(IWCM_F_CONNECT_WAIT, &cm_id_priv->flags);
> + wake_up_all(&cm_id_priv->connect_wait);
> + return -EINVAL;
> + }
> + cm_id_priv->state = IW_CM_STATE_IDLE;
> + spin_unlock_irqrestore(&cm_id_priv->lock, flags);
> +
> + ret = cm_id->device->iwcm->reject(cm_id, private_data,
> + private_data_len);
> +
> + clear_bit(IWCM_F_CONNECT_WAIT, &cm_id_priv->flags);
> + wake_up_all(&cm_id_priv->connect_wait);
> +
> + return ret;
> +}
> +EXPORT_SYMBOL(iw_cm_reject);
> +
> +/*
> + * CM_ID <-- ESTABLISHED
> + *
> + * Accepts an inbound connection request and generates an ESTABLISHED
> + * event. Callers of iw_cm_disconnect and iw_destroy_cm_id will block
> + * until the ESTABLISHED event is received from the provider.
> + */
This makes it sound like we're just waiting for an event.
> +int iw_cm_accept(struct iw_cm_id *cm_id,
> + struct iw_cm_conn_param *iw_param)
> +{
> + struct iwcm_id_private *cm_id_priv;
> + struct ib_qp *qp;
> + unsigned long flags;
> + int ret;
> +
> + cm_id_priv = container_of(cm_id, struct iwcm_id_private, id);
> + set_bit(IWCM_F_CONNECT_WAIT, &cm_id_priv->flags);
> +
> + spin_lock_irqsave(&cm_id_priv->lock, flags);
> + if (cm_id_priv->state != IW_CM_STATE_CONN_RECV) {
> + spin_unlock_irqrestore(&cm_id_priv->lock, flags);
> + clear_bit(IWCM_F_CONNECT_WAIT, &cm_id_priv->flags);
> + wake_up_all(&cm_id_priv->connect_wait);
> + return -EINVAL;
> + }
> + /* Get the ib_qp given the QPN */
> + qp = cm_id->device->iwcm->get_qp(cm_id->device, iw_param->qpn);
> + if (!qp) {
> + spin_unlock_irqrestore(&cm_id_priv->lock, flags);
> + return -EINVAL;
> + }
> + cm_id->device->iwcm->add_ref(qp);
> + cm_id_priv->qp = qp;
> + spin_unlock_irqrestore(&cm_id_priv->lock, flags);
> +
> + ret = cm_id->device->iwcm->accept(cm_id, iw_param);
> + if (ret) {
> + /* An error on accept precludes provider events */
> + BUG_ON(cm_id_priv->state != IW_CM_STATE_CONN_RECV);
> + cm_id_priv->state = IW_CM_STATE_IDLE;
> + spin_lock_irqsave(&cm_id_priv->lock, flags);
> + if (cm_id_priv->qp) {
> + cm_id->device->iwcm->rem_ref(qp);
> + cm_id_priv->qp = NULL;
> + }
> + spin_unlock_irqrestore(&cm_id_priv->lock, flags);
> + printk("Accept failed, ret=%d\n", ret);
> + clear_bit(IWCM_F_CONNECT_WAIT, &cm_id_priv->flags);
> + wake_up_all(&cm_id_priv->connect_wait);
> + }
> +
> + return ret;
> +}
> +EXPORT_SYMBOL(iw_cm_accept);
> +
> +/*
> + * Active Side: CM_ID <-- CONN_SENT
> + *
> + * If successful, results in the generation of a CONNECT_REPLY
> + * event. iw_cm_disconnect and iw_cm_destroy will block until the
> + * CONNECT_REPLY event is received from the provider.
> + */
> +int iw_cm_connect(struct iw_cm_id *cm_id, struct iw_cm_conn_param *iw_param)
> +{
> + struct iwcm_id_private *cm_id_priv;
> + int ret = 0;
> + unsigned long flags;
> + struct ib_qp *qp;
> +
> + cm_id_priv = container_of(cm_id, struct iwcm_id_private, id);
> + set_bit(IWCM_F_CONNECT_WAIT, &cm_id_priv->flags);
> +
> + spin_lock_irqsave(&cm_id_priv->lock, flags);
> + if (cm_id_priv->state != IW_CM_STATE_IDLE) {
> + spin_unlock_irqrestore(&cm_id_priv->lock, flags);
> + clear_bit(IWCM_F_CONNECT_WAIT, &cm_id_priv->flags);
> + wake_up_all(&cm_id_priv->connect_wait);
> + return -EINVAL;
> + }
> +
> + /* Get the ib_qp given the QPN */
> + qp = cm_id->device->iwcm->get_qp(cm_id->device, iw_param->qpn);
> + if (!qp) {
> + spin_unlock_irqrestore(&cm_id_priv->lock, flags);
> + return -EINVAL;
> + }
> + cm_id->device->iwcm->add_ref(qp);
> + cm_id_priv->qp = qp;
> + cm_id_priv->state = IW_CM_STATE_CONN_SENT;
> + spin_unlock_irqrestore(&cm_id_priv->lock, flags);
> +
> + ret = cm_id->device->iwcm->connect(cm_id, iw_param);
> + if (ret) {
> + spin_lock_irqsave(&cm_id_priv->lock, flags);
> + if (cm_id_priv->qp) {
> + cm_id->device->iwcm->rem_ref(qp);
> + cm_id_priv->qp = NULL;
> + }
> + spin_unlock_irqrestore(&cm_id_priv->lock, flags);
> + BUG_ON(cm_id_priv->state != IW_CM_STATE_CONN_SENT);
> + cm_id_priv->state = IW_CM_STATE_IDLE;
> + printk("Connect failed, ret=%d\n", ret);
> + clear_bit(IWCM_F_CONNECT_WAIT, &cm_id_priv->flags);
> + wake_up_all(&cm_id_priv->connect_wait);
> + }
> +
> + return ret;
> +}
> +EXPORT_SYMBOL(iw_cm_connect);
> +
> +/*
> + * Passive Side: new CM_ID <-- CONN_RECV
> + *
> + * Handles an inbound connect request. The function creates a new
> + * iw_cm_id to represent the new connection and inherits the client
> + * callback function and other attributes from the listening parent.
> + *
> + * The work item contains a pointer to the listen_cm_id and the event. The
> + * listen_cm_id contains the client cm_handler, context and
> + * device. These are copied when the device is cloned. The event
> + * contains the new four tuple.
> + *
> + * An error on the child should not affect the parent, so this
> + * function does not return a value.
> + */
> +static void cm_conn_req_handler(struct iwcm_id_private *listen_id_priv,
> + struct iw_cm_event *iw_event)
> +{
> + unsigned long flags;
> + struct iw_cm_id *cm_id;
> + struct iwcm_id_private *cm_id_priv;
> + int ret;
> +
> + /* The provider should never generate a connection request
> + * event with a bad status.
> + */
> + BUG_ON(iw_event->status);
> +
> + /* We could be destroying the listening id. If so, ignore this
> + * upcall. */
> + spin_lock_irqsave(&listen_id_priv->lock, flags);
> + if (listen_id_priv->state != IW_CM_STATE_LISTEN) {
> + spin_unlock_irqrestore(&listen_id_priv->lock, flags);
> + return;
> + }
> + spin_unlock_irqrestore(&listen_id_priv->lock, flags);
> +
> + cm_id = iw_create_cm_id(listen_id_priv->id.device,
> + listen_id_priv->id.cm_handler,
> + listen_id_priv->id.context);
> + /* If the cm_id could not be created, ignore the request */
> + if (IS_ERR(cm_id))
> + return;
> +
> + cm_id->provider_data = iw_event->provider_data;
> + cm_id->local_addr = iw_event->local_addr;
> + cm_id->remote_addr = iw_event->remote_addr;
> +
> + cm_id_priv = container_of(cm_id, struct iwcm_id_private, id);
> + cm_id_priv->state = IW_CM_STATE_CONN_RECV;
> +
> + /* Call the client CM handler */
> + ret = cm_id->cm_handler(cm_id, iw_event);
> + if (ret) {
> + printk("destroying child id %p, ret=%d\n",
> + cm_id, ret);
We probably don't always want to print a message here.
> + set_bit(IWCM_F_CALLBACK_DESTROY, &cm_id_priv->flags);
> + destroy_cm_id(cm_id);
> + if (atomic_read(&cm_id_priv->refcount)==0)
> + kfree(cm_id);
> + }
> +}
> +
> +/*
> + * Passive Side: CM_ID <-- ESTABLISHED
> + *
> + * The provider generated an ESTABLISHED event which means that
> + * the MPA negotion has completed successfully and we are now in MPA
> + * FPDU mode.
> + *
> + * This event can only be received in the CONN_RECV state. If the
> + * remote peer closed, the ESTABLISHED event would be received followed
> + * by the CLOSE event. If the app closes, it will block until we wake
> + * it up after processing this event.
> + */
> +static int cm_conn_est_handler(struct iwcm_id_private *cm_id_priv,
> + struct iw_cm_event *iw_event)
> +{
> + unsigned long flags;
> + int ret = 0;
> +
> + spin_lock_irqsave(&cm_id_priv->lock, flags);
> +
> + /* We clear the CONNECT_WAIT bit here to allow the callback
> + * function to call iw_cm_disconnect. Calling iw_destroy_cm_id
> + * from a callback handler is not allowed */
> + clear_bit(IWCM_F_CONNECT_WAIT, &cm_id_priv->flags);
> + switch (cm_id_priv->state) {
> + case IW_CM_STATE_CONN_RECV:
> + cm_id_priv->state = IW_CM_STATE_ESTABLISHED;
> + spin_unlock_irqrestore(&cm_id_priv->lock, flags);
> + ret = cm_id_priv->id.cm_handler(&cm_id_priv->id, iw_event);
> + break;
> + default:
> + BUG_ON(1);
Can just BUG_ON the state and avoid the switch. Same comment applies below.
> + }
> + wake_up_all(&cm_id_priv->connect_wait);
> +
> + return ret;
> +}
> +
> +/*
> + * Active Side: CM_ID <-- ESTABLISHED
> + *
> + * The app has called connect and is waiting for the established event to
> + * post it's requests to the server. This event will wake up anyone
> + * blocked in iw_cm_disconnect or iw_destroy_id.
> + */
> +static int cm_conn_rep_handler(struct iwcm_id_private *cm_id_priv,
> + struct iw_cm_event *iw_event)
> +{
> + unsigned long flags;
> + int ret = 0;
> +
> + spin_lock_irqsave(&cm_id_priv->lock, flags);
> + /* Clear the connect wait bit so a callback function calling
> + * iw_cm_disconnect will not wait and deadlock this thread */
> + clear_bit(IWCM_F_CONNECT_WAIT, &cm_id_priv->flags);
> + switch (cm_id_priv->state) {
> + case IW_CM_STATE_CONN_SENT:
> + if (iw_event->status == IW_CM_EVENT_STATUS_ACCEPTED) {
> + cm_id_priv->id.local_addr = iw_event->local_addr;
> + cm_id_priv->id.remote_addr = iw_event->remote_addr;
> + cm_id_priv->state = IW_CM_STATE_ESTABLISHED;
> + } else {
> + /* REJECTED or RESET */
> + cm_id_priv->id.device->iwcm->rem_ref(cm_id_priv->qp);
> + cm_id_priv->qp = NULL;
> + cm_id_priv->state = IW_CM_STATE_IDLE;
> + }
> + spin_unlock_irqrestore(&cm_id_priv->lock, flags);
> + ret = cm_id_priv->id.cm_handler(&cm_id_priv->id, iw_event);
> + break;
> + default:
> + BUG_ON(1);
> + }
> + /* Wake up waiters on connect complete */
> + wake_up_all(&cm_id_priv->connect_wait);
> +
> + return ret;
> +}
> +
> +/*
> + * CM_ID <-- CLOSING
> + *
> + * If in the ESTABLISHED state, move to CLOSING.
> + */
> +static void cm_disconnect_handler(struct iwcm_id_private *cm_id_priv,
> + struct iw_cm_event *iw_event)
> +{
> + unsigned long flags;
> +
> + spin_lock_irqsave(&cm_id_priv->lock, flags);
> + if (cm_id_priv->state == IW_CM_STATE_ESTABLISHED)
> + cm_id_priv->state = IW_CM_STATE_CLOSING;
> + spin_unlock_irqrestore(&cm_id_priv->lock, flags);
> +}
> +
> +/*
> + * CM_ID <-- IDLE
> + *
> + * If in the ESTBLISHED or CLOSING states, the QP will have have been
> + * moved by the provider to the ERR state. Disassociate the CM_ID from
> + * the QP, move to IDLE, and remove the 'connected' reference.
> + *
> + * If in some other state, the cm_id was destroyed asynchronously.
> + * This is the last reference that will result in waking up
> + * the app thread blocked in iw_destroy_cm_id.
> + */
> +static int cm_close_handler(struct iwcm_id_private *cm_id_priv,
> + struct iw_cm_event *iw_event)
> +{
> + unsigned long flags;
> + int ret = 0;
> + /* TT */printk("%s:%d cm_id_priv=%p, state=%d\n",
> + __FUNCTION__, __LINE__,
> + cm_id_priv,cm_id_priv->state);
Will want to remove this.
- Sean
^ permalink raw reply
* Re: TCP connection hang problem with 2.6.16.16, e1000 (found work-around)
From: Ben Greear @ 2006-05-31 22:38 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: NetDev
In-Reply-To: <20060531151018.17110684@localhost.localdomain>
Stephen Hemminger wrote:
> On Wed, 31 May 2006 14:45:13 -0700
> Ben Greear <greearb@candelatech.com> wrote:
>
>
>>Ok, I think I found the problem. It appears to be at least partially
>>self-inflicted. I was modifying the kernel buffer settings via /proc/*
>>to improve performance. I noticed that when I used kernel-defaults,
>>the TCP connection hangs stopped (and ran ~20% slower). My personal opinion
>>is that this is still a bug somewhere in TCP, but at least I have a work-around.
>
>
> If you set bogus values, you can end up consuming all the memory on your system.
I still had some sort of memory..ie I could run other applications and could
start a second TCP connection on the same interface. It is true that you can
truly OOM systems with bad settings here, but my machine has 2GB of RAM, and
I think my setting were low enough that it should not have run out. It also
had enough memory left to spam the duplicate TCP acks at around 200kpps in
both directions! Manually over-riding the write buffer size had little or
no effect: It would still hang.
>>The new settings that appear to work around the TCP connection hang,
>>and still give decent performance are below. Documentation being thin,
>>I used the suggested values in Documentation/networking/ixgb.txt as
>>hints:
>
>
> Settings don't belong in this Documentation file because it a general
> thing not specific to that hardware.
The file was describing how to tune for good performance using their
NICs.
At least most of the TCP settings are described in the networking/ip-sysctl.txt
file, but I could find nothing on the core.rmem_max setting, except in examples
in the 10Gbe NIC help texts.
It would be nice to either have the core.* values described in the
ip-sysctl.txt file, or perhaps add a new file.
There are tuning tips all over the web too...often quite badly
outdated it seems. Might be a good thing to add to the linux-net wiki.
I'd give it a try, but obviously my info is suspect! :P
Ben
--
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc http://www.candelatech.com
^ permalink raw reply
* Re: netif_tx_disable and lockless TX
From: Michael Chan @ 2006-05-31 21:20 UTC (permalink / raw)
To: David Miller; +Cc: herbert, jgarzik, netdev
In-Reply-To: <20060530.221117.85394939.davem@davemloft.net>
On Tue, 2006-05-30 at 22:11 -0700, David Miller wrote:
> From: Herbert Xu <herbert@gondor.apana.org.au>
> Date: Wed, 31 May 2006 14:58:11 +1000
>
> > Yes, TG3 does not disable IRQs when taking its TX lock. So do you see
> > any problems with replacing the TG3 TX lock using xmit_lock?
>
> I don't see any.
>
David, So do we want to fix this issue as proposed by Herbert to replace
tx_lock with xmit_lock? It seems quite straightforward to do. For this
change to work, netpoll also needs to be fixed up a bit to check for
LLTX before getting the xmit_lock.
^ permalink raw reply
* Re: DPRINTKs in e1000 code (2.6.x kernels)
From: Auke Kok @ 2006-05-31 23:30 UTC (permalink / raw)
To: Amit K Arora; +Cc: netdev, auke-jan.h.kok, sofar
In-Reply-To: <447D5B51.7030100@in.ibm.com>
On Wed, 31 May 2006 14:31:05 +0530, Amit K Arora <aarora@in.ibm.com> wrote:
> Hello,
>
> I am wondering why this patch "e1000: Remove PM warning DPRINTKs
> breaking 2.4.x kernels"
> (http://www.mail-archive.com/netdev@vger.kernel.org/msg10803.html) was
> meant only for 2.4 kernels, and _not_ for 2.6 kernels.
this code was removed from our generic driver first which is available for 2.4 kernels as well. we try to keep the in-kernel 2.4 and 2.6-driver as close to the general module because we spend a lot of time testing that particular driver.
The removal from the 2.6 kernel has two motivations:
- sync with the standalone tarball driver on e1000.sf.net
- pci subsystem smartness should take care of warning on those operations
> These DPRINTKs in e1000_suspend() are currently resulting in following
> messages while rebooting a system (2.6 kernel) which has e1000 adapter:
>
> Shutting down MD Raid
> done
> Stopping udevd:
> done
> proc umounted
> The system will be halted immediately.
> md: stopping all md devices.
> e1000: eth3: e1000_suspend: Error enabling D3 wake <<==
> e1000: eth3: e1000_suspend: Error enabling D3 cold wake <<==
> Power down.
>
>
> Should these DPRINTKs be removed from the 2.6.x e1000 code as well ?
they already are. the patch was merged in 7.0.38-k2 or so which is over a month ago.
Also, if you are getting these errors there are several fixes in 7.0.38+ in the kernel that might be related. especially the WoL fix that re-enables the shutdown handler correctly might fix this issue for you. Please give this kernel/module a try (see http://www.mail-archive.com/netdev@vger.kernel.org/msg12689.html).
Cheers,
Auke
^ permalink raw reply
* 2.6.17-rc5-mm1 lockdep output
From: Jesse Brandeburg @ 2006-06-01 0:04 UTC (permalink / raw)
To: Linux-Kernel,, Andrew Morton, mingo, Arjan van de Ven,
NetDEV list
[-- Attachment #1: Type: text/plain, Size: 1820 bytes --]
well, when running e1000 through some code paths on FC4 +
2.6.17-rc5-mm1 + ingo's latest rollup patch, with this lockdep debug
option enabled I got this:
e1000: eth1: e1000_watchdog_task: NIC Link is Up 1000 Mbps Full Duplex
======================================
[ BUG: bad unlock ordering detected! ]
--------------------------------------
mDNSResponder/2361 is trying to release lock (&in_dev->mc_list_lock) at:
[<ffffffff81233f5a>] ip_mc_add_src+0x85/0x1f8
but the next lock to release is:
(&im->lock){-+..}, at: [<ffffffff81233f52>] ip_mc_add_src+0x7d/0x1f8
other info that might help us debug this:
2 locks held by mDNSResponder/2361:
#0: (rtnl_mutex){--..}, at: [<ffffffff81253741>] mutex_lock+0x27/0x2c
#1: (&in_dev->mc_list_lock){-.-?}, at: [<ffffffff81233f14>]
ip_mc_add_src+0x3f/0x1f8
stack backtrace:
Call Trace:
[<ffffffff8100a8c0>] show_trace+0xa4/0x204
[<ffffffff8100ac32>] dump_stack+0x15/0x17
[<ffffffff8104495b>] lockdep_release+0x197/0x329
[<ffffffff81254e45>] _read_unlock+0x1c/0x28
[<ffffffff81233f5a>] ip_mc_add_src+0x85/0x1f8
[<ffffffff812342f5>] ip_mc_inc_group+0x1b5/0x220
[<ffffffff8123493e>] ip_mc_join_group+0xcf/0xf3
[<ffffffff81214ca7>] do_ip_setsockopt+0x697/0x99d
[<ffffffff81215059>] ip_setsockopt+0x24/0x8b
[<ffffffff8122c5bc>] udp_setsockopt+0xe/0x1c
[<ffffffff811ec236>] sock_common_setsockopt+0xf/0x11
[<ffffffff811ea425>] sys_setsockopt+0x8f/0xb4
[<ffffffff810093c2>] tracesys+0xd1/0xdb
e1000: eth1: e1000_watchdog_task: NIC Link is Down
e1000: eth1: e1000_watchdog_task: NIC Link is Up 1000 Mbps Full Duplex
.config attached.
As an aside, when I first tried to enable this I turned on the
selftest but turned off all the lockdep options besides spinlock and
the selftest fails.
[-- Attachment #2: config.gz --]
[-- Type: application/x-gzip, Size: 11074 bytes --]
^ permalink raw reply
* Re: netif_tx_disable and lockless TX
From: David Miller @ 2006-06-01 0:09 UTC (permalink / raw)
To: mchan; +Cc: herbert, jgarzik, netdev
In-Reply-To: <1149110429.3078.5.camel@rh4>
From: "Michael Chan" <mchan@broadcom.com>
Date: Wed, 31 May 2006 14:20:29 -0700
> David, So do we want to fix this issue as proposed by Herbert to replace
> tx_lock with xmit_lock? It seems quite straightforward to do. For this
> change to work, netpoll also needs to be fixed up a bit to check for
> LLTX before getting the xmit_lock.
Oh yes, netpoll needs changes, thanks for noticing that.
That's why I had the idea to make xmit_lock a pointer, so
that things like that netpoll case could be transparent.
I think this netpoll wrinkle means we also have to make
sure to set the xmit_lock_owner across the board.
^ permalink raw reply
* Re: netif_tx_disable and lockless TX
From: Herbert Xu @ 2006-06-01 0:25 UTC (permalink / raw)
To: David Miller; +Cc: mchan, jgarzik, netdev
In-Reply-To: <20060531.170908.46214247.davem@davemloft.net>
On Wed, May 31, 2006 at 05:09:08PM -0700, David Miller wrote:
> From: "Michael Chan" <mchan@broadcom.com>
> Date: Wed, 31 May 2006 14:20:29 -0700
>
> > David, So do we want to fix this issue as proposed by Herbert to replace
> > tx_lock with xmit_lock? It seems quite straightforward to do. For this
> > change to work, netpoll also needs to be fixed up a bit to check for
> > LLTX before getting the xmit_lock.
>
> Oh yes, netpoll needs changes, thanks for noticing that.
Why does it need to change? The idea is to take the LLTX flag off tg3
after the conversion to xmit_lock.
> I think this netpoll wrinkle means we also have to make
> sure to set the xmit_lock_owner across the board.
You're right. In fact this can deadlock today for those drivers that
already make use of xmit_lock without setting the owner. So I suppose
something like net_xmit_lock to obtain xmit_lock is called for.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: netif_tx_disable and lockless TX
From: Michael Chan @ 2006-05-31 23:01 UTC (permalink / raw)
To: Herbert Xu; +Cc: David Miller, jgarzik, netdev
In-Reply-To: <20060601002525.GA19220@gondor.apana.org.au>
On Thu, 2006-06-01 at 10:25 +1000, Herbert Xu wrote:
> On Wed, May 31, 2006 at 05:09:08PM -0700, David Miller wrote:
> > From: "Michael Chan" <mchan@broadcom.com>
> > Date: Wed, 31 May 2006 14:20:29 -0700
> >
> > > David, So do we want to fix this issue as proposed by Herbert to replace
> > > tx_lock with xmit_lock? It seems quite straightforward to do. For this
> > > change to work, netpoll also needs to be fixed up a bit to check for
> > > LLTX before getting the xmit_lock.
> >
> > Oh yes, netpoll needs changes, thanks for noticing that.
>
> Why does it need to change? The idea is to take the LLTX flag off tg3
> after the conversion to xmit_lock.
Oh, I thought your idea was to keep the LLTX and just replace tx_lock
with xmit_lock in tg3. But I suppose we can also clear LLTX, remove the
tx_lock in hard_start_xmit and convert the rest to avoid changing
netpoll.
^ permalink raw reply
* Re: netif_tx_disable and lockless TX
From: Herbert Xu @ 2006-06-01 0:42 UTC (permalink / raw)
To: Michael Chan; +Cc: David Miller, jgarzik, netdev
In-Reply-To: <1149116466.4022.5.camel@rh4>
On Wed, May 31, 2006 at 04:01:06PM -0700, Michael Chan wrote:
>
> Oh, I thought your idea was to keep the LLTX and just replace tx_lock
> with xmit_lock in tg3. But I suppose we can also clear LLTX, remove the
> tx_lock in hard_start_xmit and convert the rest to avoid changing
> netpoll.
That's what I meant. This has the advantage that things like AF_PACKET
won't see duplicate packets.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: netif_tx_disable and lockless TX
From: Michael Chan @ 2006-06-01 0:27 UTC (permalink / raw)
To: Herbert Xu; +Cc: David Miller, jgarzik, netdev
In-Reply-To: <20060601004235.GA19459@gondor.apana.org.au>
On Thu, 2006-06-01 at 10:42 +1000, Herbert Xu wrote:
> On Wed, May 31, 2006 at 04:01:06PM -0700, Michael Chan wrote:
> >
> > Oh, I thought your idea was to keep the LLTX and just replace tx_lock
> > with xmit_lock in tg3. But I suppose we can also clear LLTX, remove the
> > tx_lock in hard_start_xmit and convert the rest to avoid changing
> > netpoll.
>
> That's what I meant. This has the advantage that things like AF_PACKET
> won't see duplicate packets.
Now that I think about it some more, we can eliminate most of the
tx_lock in tg3 without converting to xmit_lock. In most places, we get
the tx_lock after we call netif_tx_disable(), etc, and quiesce the chip.
So the tx_lock or xmit_lock is really not necessary in most places. The
tp->lock is sufficient after we stop everything.
The only few remaining places where we need it is around
netif_stop_queue/netif_wake_queue in hard_start_xmit and tx completion.
We can either keep the tx_lock or convert to xmit_lock and either way
makes little difference because these are not in the normal fast path.
If we keep the tx_lock, it will be just like BNX2.
^ permalink raw reply
* Re: netif_tx_disable and lockless TX
From: Herbert Xu @ 2006-06-01 2:27 UTC (permalink / raw)
To: Michael Chan; +Cc: David Miller, jgarzik, netdev
In-Reply-To: <1149121673.4022.22.camel@rh4>
On Wed, May 31, 2006 at 05:27:53PM -0700, Michael Chan wrote:
>
> Now that I think about it some more, we can eliminate most of the
> tx_lock in tg3 without converting to xmit_lock. In most places, we get
> the tx_lock after we call netif_tx_disable(), etc, and quiesce the chip.
> So the tx_lock or xmit_lock is really not necessary in most places. The
> tp->lock is sufficient after we stop everything.
Sounds good.
> The only few remaining places where we need it is around
> netif_stop_queue/netif_wake_queue in hard_start_xmit and tx completion.
> We can either keep the tx_lock or convert to xmit_lock and either way
> makes little difference because these are not in the normal fast path.
> If we keep the tx_lock, it will be just like BNX2.
Now that I look at it the BNX2 method looks perfect.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: TCP connection hang problem with 2.6.16.16, e1000.
From: Herbert Xu @ 2006-06-01 2:56 UTC (permalink / raw)
To: David S. Miller, Ben Greear; +Cc: NetDev
In-Reply-To: <447DC830.4080201@candelatech.com>
[-- Attachment #1: Type: text/plain, Size: 2041 bytes --]
On Wed, May 31, 2006 at 04:45:36PM +0000, Ben Greear wrote:
>
> May 31 08:07:00 xeon-dt kernel: irqbalance: page allocation failure. order:1, mode:0x20
> May 31 08:07:00 xeon-dt kernel: [<78147393>] __alloc_pages+0x1d8/0x2b8
> May 31 08:07:00 xeon-dt kernel: [<7815c525>] kmem_getpages+0x2f/0x8c
> May 31 08:07:00 xeon-dt kernel: [<7815d297>] cache_grow+0xb1/0x14e
> May 31 08:07:00 xeon-dt kernel: [<7815d48e>] cache_alloc_refill+0x15a/0x206
> May 31 08:07:00 xeon-dt kernel: [<7815d7c5>] __kmalloc+0x68/0x80
> May 31 08:07:00 xeon-dt kernel: [<782b3ba2>] __alloc_skb+0x4c/0xf6
> May 31 08:07:00 xeon-dt kernel: [<782e63af>] tcp_collapse+0xe9/0x34e
We specifically calculate the packet size here to avoid a 2-page
allocation, yet we still do. This is the bug.
[NET]: Fix SKB_MAX_ORDER calculation
The calculation in SKB_MAX_ORDER is incorrect in that it can cause
an overflow across a page boundary which is what it's meant to prevent.
In particular, the header length (X) should not be lumped together with
skb_shared_info. The latter needs to be aligned properly while the header
has no choice but to sit in front of wherever the payload is.
Therefore the correct calculation is to take away the aligned size of
skb_shared_info, and then subtract the header length. The resulting
quantity L satisfies the following inequality:
SKB_DATA_ALIGN(L + X) + sizeof(struct skb_shared_info) <= PAGE_SIZE
This is the quantity used by alloc_skb to do the actual allocation.
While we're at it, there is no need to round up the header size in
sk_stream_alloc_pskb. The only thing this achieves is to make the
data payload cache-aligned. However, the data payload is just a
byte stream with no alignment requirements. So we should take away
the rounding.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
[-- Attachment #2: skb-max-order.patch --]
[-- Type: text/plain, Size: 950 bytes --]
index f8f2347..8761000 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -39,9 +39,8 @@ #define CHECKSUM_UNNECESSARY 2
#define SKB_DATA_ALIGN(X) (((X) + (SMP_CACHE_BYTES - 1)) & \
~(SMP_CACHE_BYTES - 1))
-#define SKB_MAX_ORDER(X, ORDER) (((PAGE_SIZE << (ORDER)) - (X) - \
- sizeof(struct skb_shared_info)) & \
- ~(SMP_CACHE_BYTES - 1))
+#define SKB_MAX_ORDER(X, ORDER) ((PAGE_SIZE << (ORDER)) - (X) - \
+ SKB_DATA_ALIGN(sizeof(struct skb_shared_info)))
#define SKB_MAX_HEAD(X) (SKB_MAX_ORDER((X), 0))
#define SKB_MAX_ALLOC (SKB_MAX_ORDER(0, 2))
index c9fad6f..9694fdf 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -1162,7 +1162,7 @@ static inline struct sk_buff *sk_stream_
struct sk_buff *skb;
int hdr_len;
- hdr_len = SKB_DATA_ALIGN(sk->sk_prot->max_header);
+ hdr_len = sk->sk_prot->max_header;
skb = alloc_skb_fclone(size + hdr_len, gfp);
if (skb) {
skb->truesize += mem;
^ permalink raw reply related
* netfilter reg problem
From: varun @ 2006-06-01 6:03 UTC (permalink / raw)
To: netdev
Hi all,
I have a setup where iam using iptables. I have squid
running on my statem. Now the scenario is that this sytem is my router.
I have eth0 connected to LAN, eth1 to Wan 1 and eth2 to Wan2. In the
same box i also have a propratery FW which has Load Balancing feature.
Now my intention is i start iptables first so that it registers with
netfilter hook first.
So policy is like this
iptables -t nat -D PREROUTING -i eth0 -p tcp -j REDIRECT --to-port
8080
In the port 8080 i have squid running.
As far as filtering of iptables is concerned it is allow all.
So my basic FW functionality is concerned it is taken care by My
Propratery FW. It will do the natting for both wan interfaces in Load
balance.
Now what i expect is that iptables will get all the packets first and
then packet will go to my firewall and might get dropped if ACL matches
else it is success. Once success i expect the packet to be redirected to
port 8080 where squid can take control.
Now my problem is that even though iptables is registered first when i
log the packets using iptables for the above rule i find that no packets
are comming to iptables there by no packets are going to proxy. I get
only some multicast packets.
From lan iam able to browse well and all my traffic is going tru my FW
rather that first through iptables. Why?
As far as my knowledge first registered module get the packet first and
if the action is NF_ACCEPT the the next registered module to the hook
gets it right? If yes why is it not happening? Another thing is what
exactly does redirect do does it mangel the packet?
Please help me in this?
Varun
^ permalink raw reply
* Re: Question about tcp hash function tcp_hashfn()
From: Evgeniy Polyakov @ 2006-06-01 6:04 UTC (permalink / raw)
To: David Miller; +Cc: draghuram, linux-kernel, netdev, Brian F. G. Bidulock
In-Reply-To: <20060531.114127.14356069.davem@davemloft.net>
[-- Attachment #1: Type: text/plain, Size: 1092 bytes --]
On Wed, May 31, 2006 at 11:41:27AM -0700, David Miller (davem@davemloft.net) wrote:
> > > Worse: he folded the jenkins algorith result with
> > >
> > > h ^= h >> 16;
> > > h ^= h >> 8;
> > >
> > > Destroying the coverage of the function.
> >
> > It was done to simulate socket code which uses the same folding.
> > Leaving 32bit space is just wrong, consider hash table size with that
> > index.
>
> You absolutely show not do this shifting on the jenkins hash
> result, you destroy the distribution entirely. Just mask
> it with the hash mask and that's all you need to do.
>
> Brian is right, this is absolutely critical to using the Jenkins hash
> correctly. You're "unmixing" the bits it worked so hard to mix.
That is wrong. And I have a code and picture to show that,
and you dont - prove me wrong :)
Attached code and picture which shows _exactly_ the same distribution for
folded and not folded Jenkins hash distribution, and it's artifact
compared to XOR hash.
Fairly distributed values being XORed produce still fairly distributed
value.
--
Evgeniy Polyakov
[-- Attachment #2: hash_comparison.png --]
[-- Type: image/png, Size: 7231 bytes --]
[-- Attachment #3: test.c --]
[-- Type: text/plain, Size: 2015 bytes --]
#include <stdlib.h>
#include <stdio.h>
#include <errno.h>
typedef unsigned int u32;
typedef unsigned short u16;
typedef unsigned char u8;
typedef unsigned int __u32;
typedef unsigned short __u16;
typedef unsigned char __u8;
#include "jhash.h"
struct hash_entry
{
unsigned long counter;
};
static inline num2ip(__u8 a1, __u8 a2, __u8 a3, __u8 a4)
{
__u32 a = 0;
a |= a1;
a << 8;
a |= a2;
a << 8;
a |= a3;
a << 8;
a |= a4;
return a;
}
static inline __u8 get_random_byte(void)
{
return 1 + (int) (255.0 * (rand() / (RAND_MAX + 1.0)));
}
static inline __u16 get_random_word(void)
{
return 1 + (int) (65536.0 * (rand() / (RAND_MAX + 1.0)));
}
unsigned int hash_addr(__u32 faddr, __u16 fport, __u32 laddr, __u16 lport)
{
unsigned int h = (laddr ^ lport) ^ (faddr ^ fport);
h ^= h >> 16;
h ^= h >> 8;
return h;
}
unsigned int hash_addr1(__u32 faddr, __u16 fport, __u32 laddr, __u16 lport)
{
u32 ports;
unsigned int h;
ports = lport;
ports <<= 16;
ports |= fport;
h = jhash_2words(faddr, laddr, ports);
//h ^= h >> 16;
//h ^= h >> 8;
return h;
}
int main()
{
struct hash_entry *table;
__u32 saddr, daddr;
__u16 sport, dport;
unsigned int hash, i, *results;
unsigned int hash_size = 65536, iter_num = 100;
table = malloc(hash_size * sizeof(struct hash_entry));
if (!table)
return -ENOMEM;
results = malloc(hash_size * sizeof(unsigned int));
if (!results)
return -ENOMEM;
for (i=0; i<hash_size; ++i) {
results[i] = 0;
table[i].counter = 0;
}
srand(time(NULL));
daddr = num2ip(192, 168, 0, 1);
dport = htons(80);
for (i=0; i<hash_size*iter_num; ++i) {
saddr = num2ip(get_random_byte(), get_random_byte(), get_random_byte(), get_random_byte());
sport = get_random_word();
hash = hash_addr(saddr, sport, daddr, dport);
hash &= (hash_size - 1);
table[hash].counter++;
}
for (i=0; i<hash_size; ++i)
results[table[i].counter]++;
for (i=0; i<hash_size; ++i)
printf("%u %u\n", i, results[i]);
//printf("%u %u\n", i, table[i].counter);
}
^ permalink raw reply
* Re: Question about tcp hash function tcp_hashfn()
From: Evgeniy Polyakov @ 2006-06-01 6:12 UTC (permalink / raw)
To: David Miller, draghuram, linux-kernel, netdev; +Cc: Brian F. G. Bidulock
In-Reply-To: <20060531122955.B10147@openss7.org>
On Wed, May 31, 2006 at 12:29:55PM -0600, Brian F. G. Bidulock (bidulock@openss7.org) wrote:
> Evgeniy,
>
> On Wed, 31 May 2006, Evgeniy Polyakov wrote:
> > > > > Worse: he folded the jenkins algorith result with
> > > > >
> > > > > h ^= h >> 16;
> > > > > h ^= h >> 8;
> > > > >
> > > > > Destroying the coverage of the function.
> > > >
> > > > It was done to simulate socket code which uses the same folding.
> > > > Leaving 32bit space is just wrong, consider hash table size with that
> > > > index.
> >
> > Btw, that probably requires some clarification.
> > Since hash table size is definitely less than returned hash value, so
> > higher bits are removed, for that case above folding is done both in
> > XOR hash and my test case.
> > It is possible to just remove higher bits, but fairly ditributed parts
> > being xored produce still fairly distributed value.
>
> > > > > h ^= h >> 16;
> > > > > h ^= h >> 8;
>
> This does not remove high order bits in either function.
> Your comparison results are simply invalid with these two lines in place.
It is hash function, but not function which creates index inside hash
table. Index is created by removing high order bits with (& 0xffff).
I've present the new simple code and test results which show
that folded and not folded Jenkins hashes _do_ produce _exactly_ the
same distribution.
I think I've already said that fairly distributed values being xored
produce still fairly distributed value, so parts of 32bit fairly
distributed hash after being xored with each other still produce fairly
distributed 32bit space.
> --brian
--
Evgeniy Polyakov
^ permalink raw reply
* Re: Question about tcp hash function tcp_hashfn()
From: David Miller @ 2006-06-01 6:18 UTC (permalink / raw)
To: johnpol; +Cc: draghuram, linux-kernel, netdev, bidulock
In-Reply-To: <20060601061234.GB28087@2ka.mipt.ru>
From: Evgeniy Polyakov <johnpol@2ka.mipt.ru>
Date: Thu, 1 Jun 2006 10:12:36 +0400
> I've present the new simple code and test results which show
> that folded and not folded Jenkins hashes _do_ produce _exactly_ the
> same distribution.
Ok I believe you now :)
> I think I've already said that fairly distributed values being xored
> produce still fairly distributed value, so parts of 32bit fairly
> distributed hash after being xored with each other still produce fairly
> distributed 32bit space.
It would make a good research paper for someone mathmatically
inclined enough :)
^ permalink raw reply
* Re: Question about tcp hash function tcp_hashfn()
From: Brian F. G. Bidulock @ 2006-06-01 6:18 UTC (permalink / raw)
To: Evgeniy Polyakov; +Cc: David Miller, draghuram, linux-kernel, netdev
In-Reply-To: <20060601060424.GA28087@2ka.mipt.ru>
Evgeniy,
On Thu, 01 Jun 2006, Evgeniy Polyakov wrote:
>
> for (i=0; i<hash_size*iter_num; ++i) {
> saddr = num2ip(get_random_byte(), get_random_byte(), get_random_byte(), get_random_byte());
> sport = get_random_word();
You still have a problem: you cannot use a pseudo-random number
generator to generate the sample set as the pseudo-random number
generator function itself can interact with the hash.
Try iterating through all 2**48 values or at least a sizeable
representative subset.
^ permalink raw reply
* Re: Question about tcp hash function tcp_hashfn()
From: Brian F. G. Bidulock @ 2006-06-01 6:22 UTC (permalink / raw)
To: David Miller; +Cc: johnpol, draghuram, linux-kernel, netdev
In-Reply-To: <20060531.231839.10909081.davem@davemloft.net>
David,
On Wed, 31 May 2006, David Miller wrote:
>
> Ok I believe you now :)
>
I'll believe it if he interates through a subset and gets the
same results instead of using a pseudo-random number generator.
I thought you said you were considering jenkins_3word(), not
jenkins_2word()?
^ permalink raw reply
* Re: Question about tcp hash function tcp_hashfn()
From: David Miller @ 2006-06-01 6:24 UTC (permalink / raw)
To: bidulock; +Cc: johnpol, draghuram, linux-kernel, netdev
In-Reply-To: <20060601002221.B21730@openss7.org>
From: "Brian F. G. Bidulock" <bidulock@openss7.org>
Date: Thu, 1 Jun 2006 00:22:21 -0600
> I thought you said you were considering jenkins_3word(), not
> jenkins_2word()?
We could xor some of the inputs in order to use jenkins_2word().
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox