Re: [PATCH v4 00/16] Enable DSA 2.0 Event Log and completion record faulting features

[Vinod Koul <vkoul@kernel.org>]

On 07-04-23, 13:31, Fenghua Yu wrote:
> Applications can send 64B descriptors to the DSA device via CPU
> instructions MOVDIR64B or ENQCMD. The application can choose to have
> the device write back a completion record (CR) in system memory to
> indicate the status of the descriptor submitted on completion.
> 
> With the DSA hardware, the device is able to do on demand paging through
> the hardware by faulting in the user pages that do not have physical memory
> page backing with assistance from IOMMU. In the spec this was designated as
> the block on fault feature. While this hardware feature made operation
> simpler, it also stalls the device engines while the memory pages are being
> faulted in through Page Request Service (PRS). For applications sharing the
> same workqueue (wq) or wqs in the same group, operations are stalled if
> there are no free engines. To avoid slowing the performance of all other
> running applications sharing the same device engine(s), PRS can to be
> disabled and software can deal with partial completion.
> 
> The block on fault feature on DSA 1.0 can be disabled for the wq. However,
> PRS is not completely disabled for the whole path. It is not disabled for
> CRs or batch list for a batch operation.
> 
> The other issue is the DSA 1.0 error reporting mechanism, SWERROR register.
> The SWERROR register can only report a single error at a time until the
> driver reads and acknowledges the error. The follow on errors cannot be
> reported until the current error is "cleared" by the software by writing
> a bit to the SWERR register. If a large number of faults arrive and the
> software cannot clear them fast enough, overflowed errors will be dropped
> by the device.
> 
> A CR is the optional 32 bytes (DSA) or 64 bytes (IAA) status that is
> written back for a submitted descriptor. If the address for the CR faults,
> the error is reported to the SWERROR register instead.
> 
> With DSA 2.0 hardware [1], the event log feature is added. All errors are
> reported as an entry in a circular buffer reside in the system memory.
> The system admin is responsible to configure the size of the circular
> buffer large enough per device to handle the potential errors that may be
> reported. If the buffer is full and another error needs to be reported,
> the device engine will block until there's a free slot in the buffer.
> An event log entry for a faulted CR will contain the error information,
> the CR address that faulted, and the expected CR content the device had
> originally intended to write.
> 
> DSA 2.0 also introduces per wq PRS disable knob. This will disable all PRS
> operations for the specific wq. The device will still have Address
> Translation Service (ATS) on. When ATS fails on a memory address for a CR,
> an eventlog entry will be written by the hardware into the event log
> ring buffer. The driver software is expected to parse the event log entry,
> fault in the address of the CR, and the write the content of the CR to
> the memory address.
> 
> This patch series will implement the DSA 2 event log support. The support
> for the handling of the faulted user CR is added. The driver is also
> adding the same support for batch operation descriptors. With a batch
> operation the handling of the event log entry is a bit more complex.
> The faulting CR could be for the batch descriptor or any of the operation
> descriptors within the batch. The hardware generates a batch identifier
> that is used by the driver software to correlate the event log entries for
> the relevant descriptors of that batch.
> 
> The faulting of source and destination addresses for the operation is not
> handled by the driver. That is left to be handled by the user application
> by faulting in the memory and re-submit the remaining operation.

Applied, thanks

-- 
~Vinod