From: Simon Horman <horms@kernel.org>
To: Alexandra Winter <wintera@linux.ibm.com>
Cc: David Miller <davem@davemloft.net>,
Jakub Kicinski <kuba@kernel.org>, Paolo Abeni <pabeni@redhat.com>,
Eric Dumazet <edumazet@google.com>,
Andrew Lunn <andrew+netdev@lunn.ch>,
netdev@vger.kernel.org, linux-s390@vger.kernel.org,
Heiko Carstens <hca@linux.ibm.com>,
Vasily Gorbik <gor@linux.ibm.com>,
Alexander Gordeev <agordeev@linux.ibm.com>,
Christian Borntraeger <borntraeger@linux.ibm.com>,
Sven Schnelle <svens@linux.ibm.com>,
Thorsten Winkler <twinkler@linux.ibm.com>,
Halil Pasic <pasic@linux.ibm.com>,
Aliaksei Makarau <Aliaksei.Makarau@ibm.com>,
Mahanta Jambigi <mjambigi@linux.ibm.com>
Subject: Re: [PATCH net v2] s390/ism: fix concurrency management in ism_cmd()
Date: Wed, 23 Jul 2025 15:10:11 +0100 [thread overview]
Message-ID: <20250723141011.GB2459@horms.kernel.org> (raw)
In-Reply-To: <20250722161817.1298473-1-wintera@linux.ibm.com>
On Tue, Jul 22, 2025 at 06:18:17PM +0200, Alexandra Winter wrote:
> From: Halil Pasic <pasic@linux.ibm.com>
>
> The s390x ISM device data sheet clearly states that only one
> request-response sequence is allowable per ISM function at any point in
> time. Unfortunately as of today the s390/ism driver in Linux does not
> honor that requirement. This patch aims to rectify that.
>
> This problem was discovered based on Aliaksei's bug report which states
> that for certain workloads the ISM functions end up entering error state
> (with PEC 2 as seen from the logs) after a while and as a consequence
> connections handled by the respective function break, and for future
> connection requests the ISM device is not considered -- given it is in a
> dysfunctional state. During further debugging PEC 3A was observed as
> well.
>
> A kernel message like
> [ 1211.244319] zpci: 061a:00:00.0: Event 0x2 reports an error for PCI function 0x61a
> is a reliable indicator of the stated function entering error state
> with PEC 2. Let me also point out that a kernel message like
> [ 1211.244325] zpci: 061a:00:00.0: The ism driver bound to the device does not support error recovery
> is a reliable indicator that the ISM function won't be auto-recovered
> because the ISM driver currently lacks support for it.
>
> On a technical level, without this synchronization, commands (inputs to
> the FW) may be partially or fully overwritten (corrupted) by another CPU
> trying to issue commands on the same function. There is hard evidence that
> this can lead to DMB token values being used as DMB IOVAs, leading to
> PEC 2 PCI events indicating invalid DMA. But this is only one of the
> failure modes imaginable. In theory even completely losing one command
> and executing another one twice and then trying to interpret the outputs
> as if the command we intended to execute was actually executed and not
> the other one is also possible. Frankly, I don't feel confident about
> providing an exhaustive list of possible consequences.
>
> Fixes: 684b89bc39ce ("s390/ism: add device driver for internal shared memory")
> Reported-by: Aliaksei Makarau <Aliaksei.Makarau@ibm.com>
> Tested-by: Mahanta Jambigi <mjambigi@linux.ibm.com>
> Tested-by: Aliaksei Makarau <Aliaksei.Makarau@ibm.com>
> Signed-off-by: Halil Pasic <pasic@linux.ibm.com>
> Reviewed-by: Alexandra Winter <wintera@linux.ibm.com>
> Signed-off-by: Alexandra Winter <wintera@linux.ibm.com>
Thanks for the update.
Reviewed-by: Simon Horman <horms@kernel.org>
next prev parent reply other threads:[~2025-07-23 14:10 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-07-22 16:18 [PATCH net v2] s390/ism: fix concurrency management in ism_cmd() Alexandra Winter
2025-07-23 14:10 ` Simon Horman [this message]
2025-07-24 9:30 ` patchwork-bot+netdevbpf
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20250723141011.GB2459@horms.kernel.org \
--to=horms@kernel.org \
--cc=Aliaksei.Makarau@ibm.com \
--cc=agordeev@linux.ibm.com \
--cc=andrew+netdev@lunn.ch \
--cc=borntraeger@linux.ibm.com \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=gor@linux.ibm.com \
--cc=hca@linux.ibm.com \
--cc=kuba@kernel.org \
--cc=linux-s390@vger.kernel.org \
--cc=mjambigi@linux.ibm.com \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=pasic@linux.ibm.com \
--cc=svens@linux.ibm.com \
--cc=twinkler@linux.ibm.com \
--cc=wintera@linux.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox