Intel-XE Archive on lore.kernel.org
 help / color / mirror / Atom feed
From: Raag Jadav <raag.jadav@intel.com>
To: Rodrigo Vivi <rodrigo.vivi@intel.com>
Cc: "Jani Nikula" <jani.nikula@linux.intel.com>,
	"Michal Wajdeczko" <michal.wajdeczko@intel.com>,
	intel-xe@lists.freedesktop.org,
	"Mallesh Koujalagi" <mallesh.koujalagi@intel.com>,
	"Thomas Hellström" <thomas.hellstrom@linux.intel.com>,
	"Matthew Brost" <matthew.brost@intel.com>,
	"Aravind Iddamsetty" <aravind.iddamsetty@intel.com>,
	"Riana Tauro" <riana.tauro@intel.com>,
	"Badal Nilawar" <badal.nilawar@intel.com>
Subject: Re: [PATCH v2 00/22] drm/xe: Add structured SIGID error logging infrastructure
Date: Thu, 30 Jul 2026 15:50:41 +0200	[thread overview]
Message-ID: <amtWsQMoiyxk6H4k@black.igk.intel.com> (raw)
In-Reply-To: <amqhoFzQaf1HsuFq@intel.com>

On Wed, Jul 29, 2026 at 08:58:08PM -0400, Rodrigo Vivi wrote:
> On Wed, Jul 29, 2026 at 11:31:46AM +0300, Jani Nikula wrote:
> > On Tue, 28 Jul 2026, Michal Wajdeczko <michal.wajdeczko@intel.com> wrote:
> > > Today the driver reports faults with ad-hoc drm_err()/xe_gt_err()
> > > strings that have no stable shape. That is readable for a human, but it
> > > gives fleet tooling nothing durable to match on: the wording changes
> > > between releases, lines can be rate-limited or dropped under an error
> > > storm, and there is no consistent way to ask "which recognised fault
> > > just happened?".
> > >
> > > Introduce a signature identifier (SIGID): a small, stable integer that
> > > names one recognised Xe fault situation and serves as the primary handle
> > > for triage. A SIGID maps, through published end-user documentation, to a
> > > description and a recommended action; the driver only has to emit the
> > > right SIGID next to the usual human-readable text.
> > >
> > > Design decisions:
> > >
> > >  - Software-emitted signatures only. This header enumerates just the
> > >    situations the driver detects and reports itself. Signatures that
> > >    originate in firmware or hardware are identified by those layers (via
> > >    their own records/counters) and are logged as received -- minting a
> > >    driver-side id for them would duplicate an id the reporting layer
> > >    already owns.
> > >
> > >  - Flat catalogue, chosen per report site. Each site emits the single
> > >    most specific situation for that site, so a multi-layer failure
> > >    produces a chain of reports rather than one ambiguous classification
> > >    (e.g. a failed GT reset reports GT_TDR and then WEDGED). A site that
> > >    matches no defined situation keeps using ordinary xe_err() /
> > >    xe_gt_err() rather than forcing a wrong id.
> > >
> > >  - Stable numbering. A single flat list numbered sequentially from 1, in
> > >    introduction order. Values are only ever appended, never renumbered
> > >    or reused.
> > >
> > >  - First-order action. Each SIGID carries a coarse, in-tree resolution
> > >    bucket (COLLECT / RETRY / UPDATE / RECOVER) so it is actionable
> > >    without an external reference, and so every new id must declare what
> > >    to do about it.
> > >
> > >  - Severity is decoupled from the SIGID and chosen at the call site via
> > >    xe_ras_log_fatal() / _recoverable() / _info(); the same situation can
> > >    be reported at different severities depending on the instance.
> > >
> > >  - dmesg stays close to a normal xe error line by reusing xe_err() /
> > >    xe_gt_err() (and their Tile/GT decoration); the only stable,
> > >    machine-matchable token added is SIGID=<n>. dmesg is not an ABI --
> > >    the durable machine record is the CPER carrying the same SIGID (a
> > >    planned follow-up, left as a TODO).
> > >
> > > Wire up a representative site for each software signature so the set is
> > > exercised rather than merely declared:
> > >
> > >  - PROBE: xe workqueue allocation failure during early init
> > >  - WEDGED: xe_device_declare_wedged() (drop redundant "CRITICAL" + BDF)
> > >  - SURVIVABILITY: entering boot survivability mode
> > >  - RUNTIME_FW: GuC mmio request failure
> > >  - DEVICE_FW: PCODE mailbox failure
> > >  - GT_TDR: GT reset failure (which then chains into a WEDGED report)
> > >  - MEM_FAULT: page-fault queue overflow
> > >  - IO_BUS: PCI re-enable failure after a bus reset
> > 
> > The idea is not new, see for example [1] and [2]. I'm not sure if that
> > was ever merged, though. Would be good to know what came of it, and why.
> > 
> > There's also the printk index support already in the kernel [3]. Did you
> > look into that? If not, please do. If yes, please iterate why that can't
> > be used, or extended, and a local mechanism is required.
> 
> Well, I do agree with your feeling here. When this request first came to us,
> it was framed similarly to the two older proposals you referenced.
> I pushed back making it very clear that our kernel log is not an ABI and
> it will never be... that our debug messages are for our own developers to
> consume and edit as the code evolves. So, I refused and blocked any
> attempt to modify all the messages and force them to follow this format.
> 
> So, this evolved a bit since. I hope. We have agreed that the ABI itself
> is not the dmesg, but the CPER that is emitted on tracefs... that's the
> ultimate goal. The message here is just an extra helper and it brings
> both our CPER/tracefs and the dmesg helper. So, the printk index cannot
> be used here. It has no mechanism to influence or annotate what goes
> into CPER records. And even if we modified it, its volatile number
> doesn't work with the CPER goals.
> 
> The goal is to have this SIGID mostly identifying FW/HW stuff and a few
> of the key points that could help providing a guidance/map to a recovery
> path. With an intentionally narrow scope.

So why not stick to a specific interface instead kernel logs having to
bare the burden here? CI results are already full of noise, so it's a
bit unclear how this makes developers' life any easier. I now have to
figure out all the macros before reaching to the point of failure :)

Raag

  reply	other threads:[~2026-07-30 13:50 UTC|newest]

Thread overview: 39+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-07-28 16:10 [PATCH v2 00/22] drm/xe: Add structured SIGID error logging infrastructure Michal Wajdeczko
2026-07-28 16:10 ` [PATCH v2 01/22] drm/xe: Introduce xe_any helpers Michal Wajdeczko
2026-07-28 19:11   ` Rodrigo Vivi
2026-07-29  7:23   ` Mallesh, Koujalagi
2026-07-29  7:58   ` Jani Nikula
2026-07-28 16:10 ` [PATCH v2 02/22] drm/xe/log: Add structured SIGID error logging infrastructure Michal Wajdeczko
2026-07-28 16:10 ` [PATCH v2 03/22] drm/xe/log: Introduce structured component/location identifiers Michal Wajdeczko
2026-07-28 19:13   ` Rodrigo Vivi
2026-07-28 16:10 ` [PATCH v2 04/22] drm/xe/log: Add component/location decorations to dmesg Michal Wajdeczko
2026-07-28 19:16   ` Rodrigo Vivi
2026-07-28 16:10 ` [PATCH v2 05/22] drm/xe/log: Add SIGID log helpers for severity Michal Wajdeczko
2026-07-28 16:10 ` [PATCH v2 06/22] drm/xe/log: Add SIGID log helpers for location Michal Wajdeczko
2026-07-28 16:10 ` [PATCH v2 07/22] drm/xe/log: Add SIGID log helpers for location & severity Michal Wajdeczko
2026-07-28 16:10 ` [PATCH v2 08/22] drm/xe/log: Add SIGID log helpers for components Michal Wajdeczko
2026-07-29  8:06   ` Jani Nikula
2026-07-29 13:32     ` Michal Wajdeczko
2026-07-29 15:02       ` Jani Nikula
2026-07-28 16:10 ` [PATCH v2 09/22] drm/xe/log: Add SIGID log helpers for errno-only Michal Wajdeczko
2026-07-28 16:10 ` [PATCH v2 10/22] drm/xe/log: Add hardware error signatures Michal Wajdeczko
2026-07-28 16:10 ` [PATCH v2 11/22] drm/xe/log: Extend components list with hardware items Michal Wajdeczko
2026-07-28 16:10 ` [PATCH v2 12/22] drm/xe/ras: Check RAS and LOG component definitions Michal Wajdeczko
2026-07-28 16:10 ` [PATCH v2 13/22] drm/xe/kunit: Setup driver data in the test device Michal Wajdeczko
2026-07-28 16:10 ` [PATCH v2 14/22] drm/xe/tests: Add Kunit tests for xe_log Michal Wajdeczko
2026-07-28 16:10 ` [PATCH v2 15/22] drm/xe: Report 'probe blocked' error using SIGID Michal Wajdeczko
2026-07-28 16:10 ` [PATCH v2 16/22] drm/xe: Report 'device wedged' errors " Michal Wajdeczko
2026-07-28 16:10 ` [PATCH v2 17/22] drm/xe: Report 'Survivability Mode' " Michal Wajdeczko
2026-07-28 16:10 ` [PATCH v2 18/22] drm/xe/guc: Report 'GuC mmio' " Michal Wajdeczko
2026-07-28 16:10 ` [PATCH v2 19/22] drm/xe/pcode: Report 'Mailbox failed' error " Michal Wajdeczko
2026-07-28 16:10 ` [PATCH v2 20/22] drm/xe/gt: Report 'reset failed' errors " Michal Wajdeczko
2026-07-28 16:10 ` [PATCH v2 21/22] drm/xe/gt: Report 'pagefault' " Michal Wajdeczko
2026-07-28 16:10 ` [PATCH v2 22/22] drm/xe/pci: Report 'cannot re-enable' error " Michal Wajdeczko
2026-07-28 16:17 ` ✗ CI.checkpatch: warning for drm/xe: Add structured SIGID error logging infrastructure (rev2) Patchwork
2026-07-28 16:18 ` ✓ CI.KUnit: success " Patchwork
2026-07-28 17:01 ` ✗ Xe.CI.BAT: failure " Patchwork
2026-07-28 20:57 ` ✗ Xe.CI.FULL: " Patchwork
2026-07-29  8:31 ` [PATCH v2 00/22] drm/xe: Add structured SIGID error logging infrastructure Jani Nikula
2026-07-30  0:58   ` Rodrigo Vivi
2026-07-30 13:50     ` Raag Jadav [this message]
2026-07-30 16:27       ` Rodrigo Vivi

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=amtWsQMoiyxk6H4k@black.igk.intel.com \
    --to=raag.jadav@intel.com \
    --cc=aravind.iddamsetty@intel.com \
    --cc=badal.nilawar@intel.com \
    --cc=intel-xe@lists.freedesktop.org \
    --cc=jani.nikula@linux.intel.com \
    --cc=mallesh.koujalagi@intel.com \
    --cc=matthew.brost@intel.com \
    --cc=michal.wajdeczko@intel.com \
    --cc=riana.tauro@intel.com \
    --cc=rodrigo.vivi@intel.com \
    --cc=thomas.hellstrom@linux.intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox