From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5C0BA3E63B8 for ; Mon, 1 Jun 2026 17:54:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780336493; cv=none; b=FaLYqkjT/PzJ1mQ5z01ALgIxgev5WEfBR/2rQZbpWCG7Ll5ZRw44eEQSPAa5jQHC0fiILwjSmK9/o74amDyIeZLq0GcMdNWEghbfxBhJhd+ZO4XlaV4dISdeanb/hQHmEeqWlFdrNuDR75kw3KGWSJDwP7fy1//q6zPsmLypdLM= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780336493; c=relaxed/simple; bh=FDTgzU5jwYPlxFUPB7xXeAJa14ZzKu6xRGMoXI/hdgw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=DHy797Hr3sfnlAMrckjQUMG4DC3pvFgDx56iTg04v4F9OSi9YM4gSUuWPMkQJaiBmm5xDjelsbsqExGy2QA5QKgQWujbJHU5EHSc/AxgqIM5LSmyyiK1d76sBDnCm1EovaifXHXS/6eJPeoQKyZOm38QJ7AUbLyXOZCiZyTLTT8= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--hramamurthy.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=VOSxabMZ; arc=none smtp.client-ip=209.85.215.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--hramamurthy.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="VOSxabMZ" Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-c8592a62f32so1247327a12.0 for ; Mon, 01 Jun 2026 10:54:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780336489; x=1780941289; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=LRE30N2+FLjU6j+wC3395cQ0eD+GH8qgnBq77kKy2+A=; b=VOSxabMZNTFoMMTo/knqtXaj25F8sUMSoYfpRdQQMlQOqv10kQPFEfaexjtoucjhtj DxzuQazmJJqEIbaum25pjUeVEG6Q9wfl2tTjg994nMegJTcN/ungnPfMNUwO74A+jw0m zQf93X/K1zNLM8Y53KxcZCzH9AHzg08A+yJUAO79e/a5CbcUTRcK+fmgCV9ryF6qLPNR TmsXdc427iy/fYtHxy34y4ePmkROvblquTB+KVYwYcyArIWxGCX+Bavc9cLkQaAOMpj8 lK/mKQ6N2Bi8XZYD/UAU9uFxAkB8jpWWYDTlSJcV9MIdmTQgpg8ClntPxkXQgkH9VeuY bfJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780336489; x=1780941289; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=LRE30N2+FLjU6j+wC3395cQ0eD+GH8qgnBq77kKy2+A=; b=iIjscobVugcfWZ92ot7NImCE/7Uf9CnQNz/kFw/wDyiATka+D2bzhC5kc6Wk+HRuMZ docNFtWiDLRaFKM6uncIwupf+ubkdyWy8BY5tw5KZyT/KTk2wFvYvMZkTU3jffpSUOK+ EyieJTA7lksBoVOAmUIMgrloklrZpN2bCkMi55/kF26WEhqL3HfMN/JvVDZKZypo+3bI lu0Okhh//i0zD98SGpq3x9FDuopFwJKsPAgYMZ23sDXMzCpfMvVuSHUBiFXGzfA5yh9p PZ0XcD80L0JmPMVQmGITuTcnmSFdnAw8zQusGHKawRM45vAjw0YSkGv9WVN0Av6LqLYV HHSA== X-Gm-Message-State: AOJu0YyZZ9qdBOIx78wB7rIzwhnOxpTepPgY2tQlfunoeaPMiQAFtt/P UD3EMXOiEwOI+ijHaCxhwxHPg+uVou3H5ZPXtAm+Eo3aAKHNL4uZNX0rWYCwDXgRFQC3Htdbh9p 6FKQWkEdW/Roju/z+iskR1GzrU1VeSH228l9w2R3KtZBR3iIg3nB1gturzaIMLxTyL5ECb0iET+ 97Pg/6xGCdkUi0CFOJ/Kt0rLZfuaO6nStVH36wVk2hX34ALklbCq+WG2k//s52mAI= X-Received: from pfbfx22.prod.google.com ([2002:a05:6a00:8216:b0:838:4903:1f13]) (user=hramamurthy job=prod-delivery.src-stubby-dispatcher) by 2002:aa7:9a87:0:b0:836:5c3c:2054 with SMTP id d2e1a72fcca58-842254979a0mr12281388b3a.37.1780336489188; Mon, 01 Jun 2026 10:54:49 -0700 (PDT) Date: Mon, 1 Jun 2026 17:54:34 +0000 In-Reply-To: <20260601175437.3767283-1-hramamurthy@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260601175437.3767283-1-hramamurthy@google.com> X-Mailer: git-send-email 2.54.0.1013.g208068f2d8-goog Message-ID: <20260601175437.3767283-13-hramamurthy@google.com> Subject: [PATCH net-next 12/15] gve: introduce new methods to handle IRQ doorbells From: Harshitha Ramamurthy To: netdev@vger.kernel.org Cc: joshwash@google.com, hramamurthy@google.com, andrew+netdev@lunn.ch, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, ast@kernel.org, daniel@iogearbox.net, hawk@kernel.org, john.fastabend@gmail.com, sdf@fomichev.me, willemb@google.com, jordanrhee@google.com, jfraker@google.com, nktgrg@google.com, bpf@vger.kernel.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" From: Joshua Washington Introduce `request_db_info` and `free_db_resources` to `struct gve_ctrl_ops`. These encapsulate the configuration of device resources (counter arrays and IRQ doorbell indices) which vary between Admin Queue and Mailbox modes. All behaviors related to the IRQ doorbell indices will be managed by these new methods instead of occurring directly in notify_block setup/teardown methods. Similarly, GQ ring counters will be managed in `request_db_info`. Reviewed-by: Willem de Bruijn Reviewed-by: Jordan Rhee Signed-off-by: Joshua Washington Signed-off-by: Harshitha Ramamurthy --- drivers/net/ethernet/google/gve/gve.h | 12 ++++ drivers/net/ethernet/google/gve/gve_adminq.c | 71 ++++++++++++++++++++ drivers/net/ethernet/google/gve/gve_adminq.h | 2 + drivers/net/ethernet/google/gve/gve_main.c | 70 +++++-------------- 4 files changed, 103 insertions(+), 52 deletions(-) diff --git a/drivers/net/ethernet/google/gve/gve.h b/drivers/net/ethernet/google/gve/gve.h index 12591a1fd746..2cc66dfb6098 100644 --- a/drivers/net/ethernet/google/gve/gve.h +++ b/drivers/net/ethernet/google/gve/gve.h @@ -836,6 +836,8 @@ struct gve_device_info { * structures stored in @priv to be used during initialization. * @set_num_ntfy_blks: Sets no. of vectors into @priv to be used during * initialization. + * @request_db_info: Request and store doorbell information into @priv + * @free_db_resources: Free DMA memory holding doorbell info (AdminQ only) * @get_ptype_map: Learn packet type map from device and store it in @priv * @configure_rss: Set up default RSS configuration * @setup_stats_report: Set up DMA region for stats report (AdminQ only) @@ -846,6 +848,8 @@ struct gve_ctrl_ops { void (*unmap_db_bar)(struct gve_priv *priv); void (*set_num_queues)(struct gve_priv *priv); int (*set_num_ntfy_blks)(struct gve_priv *priv); + int (*request_db_info)(struct gve_priv *priv); + void (*free_db_resources)(struct gve_priv *priv); int (*get_ptype_map)(struct gve_priv *priv); int (*configure_rss)(struct gve_priv *priv, struct ethtool_rxfh_param *param); @@ -1172,6 +1176,11 @@ static inline u32 gve_rx_idx_to_ntfy(struct gve_priv *priv, u32 queue_idx) return (priv->num_ntfy_blks / 2) + queue_idx; } +static inline u32 gve_ntfy_to_msix_idx(struct gve_priv *priv, u32 ntfy_blk_idx) +{ + return ntfy_blk_idx; +} + static inline bool gve_is_qpl(struct gve_priv *priv) { return priv->queue_format == GVE_GQI_QPL_FORMAT || @@ -1384,6 +1393,9 @@ int gve_adjust_queues(struct gve_priv *priv, struct gve_rx_queue_config new_rx_config, struct gve_tx_queue_config new_tx_config, bool reset_rss); +/* Initialization sequence */ +int gve_alloc_counter_array(struct gve_priv *priv); +void gve_free_counter_array(struct gve_priv *priv); /* flow steering rule */ int gve_get_flow_rule_entry(struct gve_priv *priv, struct ethtool_rxnfc *cmd); int gve_get_flow_rule_ids(struct gve_priv *priv, struct ethtool_rxnfc *cmd, u32 *rule_locs); diff --git a/drivers/net/ethernet/google/gve/gve_adminq.c b/drivers/net/ethernet/google/gve/gve_adminq.c index a9752525d974..966b1776ae44 100644 --- a/drivers/net/ethernet/google/gve/gve_adminq.c +++ b/drivers/net/ethernet/google/gve/gve_adminq.c @@ -1744,3 +1744,74 @@ void gve_adminq_set_num_queues(struct gve_priv *priv) device_info->default_rx_queues, priv->rx_cfg.num_queues); } + +int gve_adminq_request_db_info(struct gve_priv *priv) +{ + int err; + int i; + + /* Alloc dma addrs needed for shm regions */ + err = gve_alloc_counter_array(priv); + if (err) { + dev_err(&priv->pdev->dev, + "Failed to alloc db counter array."); + return err; + } + + priv->irq_db_indices = + dma_alloc_coherent(&priv->pdev->dev, + priv->num_ntfy_blks * + sizeof(*priv->irq_db_indices), + &priv->irq_db_indices_bus, GFP_KERNEL); + if (!priv->irq_db_indices) { + err = -ENOMEM; + goto abort_with_counter_array; + } + + err = gve_adminq_configure_device_resources(priv, + priv->counter_array_bus, + priv->num_event_counters, + priv->irq_db_indices_bus, + priv->num_ntfy_blks); + if (unlikely(err)) { + dev_err(&priv->pdev->dev, + "could not setup device_resources: err=%d\n", err); + err = -ENXIO; + goto abort_with_irq_db_indices; + } + + for (i = 0; i < priv->num_ntfy_blks; i++) + priv->ntfy_blocks[i].irq_db_index = + &priv->irq_db_indices[i].index; + return 0; + +abort_with_irq_db_indices: + dma_free_coherent(&priv->pdev->dev, priv->num_ntfy_blks * + sizeof(*priv->irq_db_indices), + priv->irq_db_indices, priv->irq_db_indices_bus); + priv->irq_db_indices = NULL; +abort_with_counter_array: + gve_free_counter_array(priv); + return err; +} + +void gve_adminq_free_db_resources(struct gve_priv *priv) +{ + int err; + + /* Log error in deconfigure device, but don't fail. This is only ever + * called as a reset is about to be triggered, so it would be redundant + * to trigger a reset. + */ + err = gve_adminq_deconfigure_device_resources(priv); + if (err) + dev_err(&priv->pdev->dev, + "Could not deconfigure device resources: err=%d\n", + err); + + dma_free_coherent(&priv->pdev->dev, priv->num_ntfy_blks * + sizeof(*priv->irq_db_indices), + priv->irq_db_indices, priv->irq_db_indices_bus); + priv->irq_db_indices = NULL; + gve_free_counter_array(priv); +} diff --git a/drivers/net/ethernet/google/gve/gve_adminq.h b/drivers/net/ethernet/google/gve/gve_adminq.h index e783589c5ed6..948bd3b17496 100644 --- a/drivers/net/ethernet/google/gve/gve_adminq.h +++ b/drivers/net/ethernet/google/gve/gve_adminq.h @@ -657,5 +657,7 @@ int gve_adminq_map_db_bar(struct gve_priv *priv); void gve_adminq_unmap_db_bar(struct gve_priv *priv); int gve_adminq_set_num_ntfy_blks(struct gve_priv *priv); void gve_adminq_set_num_queues(struct gve_priv *priv); +int gve_adminq_request_db_info(struct gve_priv *priv); +void gve_adminq_free_db_resources(struct gve_priv *priv); #endif /* _GVE_ADMINQ_H */ diff --git a/drivers/net/ethernet/google/gve/gve_main.c b/drivers/net/ethernet/google/gve/gve_main.c index 98970508ae54..55f48aee125e 100644 --- a/drivers/net/ethernet/google/gve/gve_main.c +++ b/drivers/net/ethernet/google/gve/gve_main.c @@ -179,7 +179,7 @@ static void gve_free_rss_config_cache(struct gve_priv *priv) memset(rss_config, 0, sizeof(*rss_config)); } -static int gve_alloc_counter_array(struct gve_priv *priv) +int gve_alloc_counter_array(struct gve_priv *priv) { priv->counter_array = dma_alloc_coherent(&priv->pdev->dev, @@ -192,7 +192,7 @@ static int gve_alloc_counter_array(struct gve_priv *priv) return 0; } -static void gve_free_counter_array(struct gve_priv *priv) +void gve_free_counter_array(struct gve_priv *priv) { if (!priv->counter_array) return; @@ -428,15 +428,6 @@ int gve_napi_poll_dqo(struct napi_struct *napi, int budget) static void gve_free_notify_blocks(struct gve_priv *priv) { pci_disable_msix(priv->pdev); - if (priv->irq_db_indices) { - dma_free_coherent(&priv->pdev->dev, - priv->num_ntfy_blks * - sizeof(*priv->irq_db_indices), - priv->irq_db_indices, - priv->irq_db_indices_bus); - priv->irq_db_indices = NULL; - } - kvfree(priv->ntfy_blocks); priv->ntfy_blocks = NULL; kvfree(priv->msix_vectors); @@ -493,24 +484,14 @@ static int gve_alloc_notify_blocks(struct gve_priv *priv) priv->rx_cfg.num_queues = priv->rx_cfg.max_queues; } - priv->irq_db_indices = - dma_alloc_coherent(&priv->pdev->dev, - priv->num_ntfy_blks * - sizeof(*priv->irq_db_indices), - &priv->irq_db_indices_bus, GFP_KERNEL); - if (!priv->irq_db_indices) { - err = -ENOMEM; - goto abort; - } - priv->ntfy_blocks = kvzalloc(priv->num_ntfy_blks * sizeof(*priv->ntfy_blocks), GFP_KERNEL); if (!priv->ntfy_blocks) { err = -ENOMEM; goto abort; } - return 0; + return 0; abort: gve_free_notify_blocks(priv); return err; @@ -525,13 +506,14 @@ static void gve_teardown_notify_blocks(struct gve_priv *priv) for (i = 0; i < priv->num_ntfy_blks; i++) { struct gve_notify_block *block = &priv->ntfy_blocks[i]; + int msix_idx = gve_ntfy_to_msix_idx(priv, i); if (!block->irq_requested) continue; - irq_set_affinity_hint(priv->msix_vectors[i].vector, + irq_set_affinity_hint(priv->msix_vectors[msix_idx].vector, NULL); - free_irq(priv->msix_vectors[i].vector, block); + free_irq(priv->msix_vectors[msix_idx].vector, block); block->irq = 0; block->irq_requested = false; } @@ -560,12 +542,11 @@ static int gve_setup_notify_blocks(struct gve_priv *priv) } priv->mgmt_irq_requested = true; - /* Setup the other blocks - the first n-1 vectors */ node_mask = gve_get_node_mask(priv); cur_cpu = cpumask_first(node_mask); for (i = 0; i < priv->num_ntfy_blks; i++) { struct gve_notify_block *block = &priv->ntfy_blocks[i]; - int msix_idx = i; + int msix_idx = gve_ntfy_to_msix_idx(priv, i); snprintf(block->name, sizeof(block->name), "gve-ntfy-blk%d@pci:%s", i, pci_name(priv->pdev)); @@ -575,14 +556,13 @@ static int gve_setup_notify_blocks(struct gve_priv *priv) IRQF_NO_AUTOEN, block->name, block); if (err) { dev_err(&priv->pdev->dev, - "Failed to receive msix vector %d\n", i); + "Failed to receive msix vector %d\n", msix_idx); goto abort; } block->irq = priv->msix_vectors[msix_idx].vector; block->irq_requested = true; irq_set_affinity_and_hint(block->irq, cpumask_of(cur_cpu)); - block->irq_db_index = &priv->irq_db_indices[i].index; cur_cpu = cpumask_next(cur_cpu, node_mask); /* Wrap once CPUs in the node have been exhausted, or when @@ -599,7 +579,6 @@ static int gve_setup_notify_blocks(struct gve_priv *priv) return err; } - static void gve_free_control_plane_resources(struct gve_priv *priv) { bitmap_free(priv->xsk_pools); @@ -608,9 +587,8 @@ static void gve_free_control_plane_resources(struct gve_priv *priv) kvfree(priv->ptype_lut_dqo); priv->ptype_lut_dqo = NULL; - gve_free_stats_report(priv); gve_free_notify_blocks(priv); - gve_free_counter_array(priv); + gve_free_stats_report(priv); gve_free_rss_config_cache(priv); gve_free_flow_rule_caches(priv); } @@ -623,9 +601,6 @@ static int gve_alloc_control_plane_resources(struct gve_priv *priv) if (err) return err; err = gve_alloc_rss_config_cache(priv); - if (err) - goto abort; - err = gve_alloc_counter_array(priv); if (err) goto abort; err = gve_alloc_notify_blocks(priv); @@ -661,15 +636,9 @@ static int gve_setup_control_plane_resources(struct gve_priv *priv) const struct gve_ctrl_ops *ops = priv->ctrl_ops; int err; - err = gve_adminq_configure_device_resources(priv, - priv->counter_array_bus, - priv->num_event_counters, - priv->irq_db_indices_bus, - priv->num_ntfy_blks); - if (unlikely(err)) { - dev_err(&priv->pdev->dev, - "could not setup device_resources: err=%d\n", err); - err = -ENXIO; + err = ops->request_db_info(priv); + if (err) { + dev_err(&priv->pdev->dev, "Failed to get db info"); return err; } @@ -678,7 +647,7 @@ static int gve_setup_control_plane_resources(struct gve_priv *priv) if (err) { dev_err(&priv->pdev->dev, "Failed to get ptype map: err=%d\n", err); - goto deconfigure_device; + goto free_db_resources; } } @@ -708,8 +677,8 @@ static int gve_setup_control_plane_resources(struct gve_priv *priv) teardown_clock: gve_teardown_clock(priv); -deconfigure_device: - gve_adminq_deconfigure_device_resources(priv); +free_db_resources: + ops->free_db_resources(priv); return err; } @@ -739,12 +708,7 @@ static void gve_teardown_control_plane_resources(struct gve_priv *priv) dev_err(&priv->pdev->dev, "Failed to detach stats report: err=%d\n", err); gve_teardown_clock(priv); - - err = gve_adminq_deconfigure_device_resources(priv); - if (err) - dev_err(&priv->pdev->dev, - "Could not deconfigure device resources: err=%d\n", - err); + ops->free_db_resources(priv); } gve_clear_device_resources_ok(priv); @@ -2494,6 +2458,8 @@ static const struct gve_ctrl_ops gve_adminq_ops = { .reset_flow_rules = gve_adminq_reset_flow_rules, .setup_stats_report = gve_adminq_report_stats, .configure_rss = gve_adminq_configure_rss, + .request_db_info = gve_adminq_request_db_info, + .free_db_resources = gve_adminq_free_db_resources, }; static int gve_init_priv(struct gve_priv *priv) -- 2.54.0.669.g59709faab0-goog