From: Saeed Mahameed <saeedm@nvidia.com>
To: Mitchell Augustin <mitchell.augustin@canonical.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>,
leon@kernel.org, tariqt@nvidia.com, andrew+netdev@lunn.ch,
"David S. Miller" <davem@davemloft.net>,
Eric Dumazet <edumazet@google.com>,
Jakub Kicinski <kuba@kernel.org>, Paolo Abeni <pabeni@redhat.com>,
netdev@vger.kernel.org, linux-rdma@vger.kernel.org,
linux-kernel@vger.kernel.org, Talat Batheesh <talatb@nvidia.com>,
Feras Daoud <ferasda@nvidia.com>
Subject: Re: modprobe mlx5_core on OCI bare-metal instance causes unrecoverable hang and I/O error
Date: Fri, 7 Feb 2025 11:33:21 -0800 [thread overview]
Message-ID: <Z6ZgAbyCsJzoqbZc@x130> (raw)
In-Reply-To: <CAHTA-uZMZ6qQZf_n55gNaTjQQ0j8nXdt1Yi_+8+-YUNhxcrs_A@mail.gmail.com>
On 07 Feb 13:24, Mitchell Augustin wrote:
>*facepalm*
>
>Thanks, I can't believe that wasn't my first thought as soon as I
>learned these instances were using iSCSI. That's almost certainly what
>is happening on this OCI instance, since the host adapter for its
>iSCSI transport is a ConnectX card.
>
>The fact that I was able to see similar behavior once on a machine
>booted from a local disk (in the A100 test I mentioned) is still
>confusing though. I'll update this thread if I can figure out a
>reliable way to reproduce that behavior.
>
BTW I saw this happening in few instances of virtualized environments as
well, where the VM storage is network/RDMA packed by the host driver/network,
when the host driver restarts (which is a normal behavior for a PV setup),
the VM also get storage related timeouts and soft lockups. Graceful
shutdown needs to be handled inside of the network backed block devices
IMHO.
-Saeed.
prev parent reply other threads:[~2025-02-07 19:33 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-02-05 23:09 modprobe mlx5_core on OCI bare-metal instance causes unrecoverable hang and I/O error Mitchell Augustin
2025-02-07 15:54 ` Jason Gunthorpe
2025-02-07 16:02 ` Mitchell Augustin
2025-02-07 19:01 ` Jason Gunthorpe
2025-02-07 19:24 ` Mitchell Augustin
2025-02-07 19:33 ` Saeed Mahameed [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Z6ZgAbyCsJzoqbZc@x130 \
--to=saeedm@nvidia.com \
--cc=andrew+netdev@lunn.ch \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=ferasda@nvidia.com \
--cc=jgg@ziepe.ca \
--cc=kuba@kernel.org \
--cc=leon@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=mitchell.augustin@canonical.com \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=talatb@nvidia.com \
--cc=tariqt@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.