From: Saeed Mahameed <saeedm@nvidia.com>
To: Mitchell Augustin <mitchell.augustin@canonical.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>,
leon@kernel.org, tariqt@nvidia.com, andrew+netdev@lunn.ch,
"David S. Miller" <davem@davemloft.net>,
Eric Dumazet <edumazet@google.com>,
Jakub Kicinski <kuba@kernel.org>, Paolo Abeni <pabeni@redhat.com>,
netdev@vger.kernel.org, linux-rdma@vger.kernel.org,
linux-kernel@vger.kernel.org, Talat Batheesh <talatb@nvidia.com>,
Feras Daoud <ferasda@nvidia.com>
Subject: Re: modprobe mlx5_core on OCI bare-metal instance causes unrecoverable hang and I/O error
Date: Fri, 7 Feb 2025 11:33:21 -0800 [thread overview]
Message-ID: <Z6ZgAbyCsJzoqbZc@x130> (raw)
In-Reply-To: <CAHTA-uZMZ6qQZf_n55gNaTjQQ0j8nXdt1Yi_+8+-YUNhxcrs_A@mail.gmail.com>
On 07 Feb 13:24, Mitchell Augustin wrote:
>*facepalm*
>
>Thanks, I can't believe that wasn't my first thought as soon as I
>learned these instances were using iSCSI. That's almost certainly what
>is happening on this OCI instance, since the host adapter for its
>iSCSI transport is a ConnectX card.
>
>The fact that I was able to see similar behavior once on a machine
>booted from a local disk (in the A100 test I mentioned) is still
>confusing though. I'll update this thread if I can figure out a
>reliable way to reproduce that behavior.
>
BTW I saw this happening in few instances of virtualized environments as
well, where the VM storage is network/RDMA packed by the host driver/network,
when the host driver restarts (which is a normal behavior for a PV setup),
the VM also get storage related timeouts and soft lockups. Graceful
shutdown needs to be handled inside of the network backed block devices
IMHO.
-Saeed.
prev parent reply other threads:[~2025-02-07 19:33 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-02-05 23:09 modprobe mlx5_core on OCI bare-metal instance causes unrecoverable hang and I/O error Mitchell Augustin
2025-02-07 15:54 ` Jason Gunthorpe
2025-02-07 16:02 ` Mitchell Augustin
2025-02-07 19:01 ` Jason Gunthorpe
2025-02-07 19:24 ` Mitchell Augustin
2025-02-07 19:33 ` Saeed Mahameed [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Z6ZgAbyCsJzoqbZc@x130 \
--to=saeedm@nvidia.com \
--cc=andrew+netdev@lunn.ch \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=ferasda@nvidia.com \
--cc=jgg@ziepe.ca \
--cc=kuba@kernel.org \
--cc=leon@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=mitchell.augustin@canonical.com \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=talatb@nvidia.com \
--cc=tariqt@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).