linux-fsdevel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Chaitanya Kulkarni <chaitanyak@nvidia.com>
To: Christoph Hellwig <hch@lst.de>,
	Chaitanya Kulkarni <chaitanyak@nvidia.com>
Cc: "linux-block@vger.kernel.org" <linux-block@vger.kernel.org>,
	"linux-raid@vger.kernel.org" <linux-raid@vger.kernel.org>,
	"linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>,
	"linux-fsdevel@vger.kernel.org" <linux-fsdevel@vger.kernel.org>,
	"axboe@kernel.dk" <axboe@kernel.dk>,
	"agk@redhat.com" <agk@redhat.com>,
	"song@kernel.org" <song@kernel.org>,
	"djwong@kernel.org" <djwong@kernel.org>,
	"kbusch@kernel.org" <kbusch@kernel.org>,
	"sagi@grimberg.me" <sagi@grimberg.me>,
	"jejb@linux.ibm.com" <jejb@linux.ibm.com>,
	"martin.petersen@oracle.com" <martin.petersen@oracle.com>,
	"viro@zeniv.linux.org.uk" <viro@zeniv.linux.org.uk>,
	"javier@javigon.com" <javier@javigon.com>,
	"johannes.thumshirn@wdc.com" <johannes.thumshirn@wdc.com>,
	"bvanassche@acm.org" <bvanassche@acm.org>,
	"dongli.zhang@oracle.com" <dongli.zhang@oracle.com>,
	"ming.lei@redhat.com" <ming.lei@redhat.com>,
	"willy@infradead.org" <willy@infradead.org>,
	"jefflexu@linux.alibaba.com" <jefflexu@linux.alibaba.com>,
	"josef@toxicpanda.com" <josef@toxicpanda.com>,
	"clm@fb.com" <clm@fb.com>, "dsterba@suse.com" <dsterba@suse.com>,
	"jack@suse.com" <jack@suse.com>, "tytso@mit.edu" <tytso@mit.edu>,
	"adilger.kernel@dilger.ca" <adilger.kernel@dilger.ca>,
	"jlayton@kernel.org" <jlayton@kernel.org>,
	"idryomov@gmail.com" <idryomov@gmail.com>,
	"danil.kipnis@cloud.ionos.com" <danil.kipnis@cloud.ionos.com>,
	"ebiggers@google.com" <ebiggers@google.com>,
	"jinpu.wang@cloud.ionos.com" <jinpu.wang@cloud.ionos.com>
Subject: Re: [PATCH 1/6] block: add support for REQ_OP_VERIFY
Date: Tue, 5 Jul 2022 23:55:09 +0000	[thread overview]
Message-ID: <dc23bece-359e-0da8-c7fc-4e632169b22f@nvidia.com> (raw)
In-Reply-To: <20220705083416.GA19123@lst.de>

On 7/5/22 01:34, Christoph Hellwig wrote:
> On Thu, Jun 30, 2022 at 02:14:01AM -0700, Chaitanya Kulkarni wrote:
>> This adds a new block layer operation to offload verifying a range of
>> LBAs. This support is needed in order to provide file systems and
>> fabrics, kernel components to offload LBA verification when it is
>> supported by the hardware controller. In case hardware offloading is
>> not supported then we provide API to emulate the same. The prominent
>> example of that is SCSI and NVMe Verify command. We also provide
>> an emulation of the same operation that can be used in case H/W does
>> not support verify. This is still useful when block device is remotely
>> attached e.g. using NVMeOF.
> 
> What is the point of providing the offload?

Data block verification is done at the time of file-scrubbing
e.g. see [1], having support to offload the verify command will :-

1. Reduce the DMA transfer at the time of scrubbing :-

In the absense of verify command user has to send the read
command that will trigger same behaviour as verify command, but
with the DMA traffic with operating system storage stack overhead
of REQ_OP_READ. This overhead gets duplicated for the fabrics
controller where host and target now has to issue REQ_OP_READ
leading to significant DMA transfer as compare to REQ_OP_VERIFY
for each protocol SCSI/NVMe etc. This makes it possible to do
a low-level scrub of the stored data without being bottlenecked
by the host interface bandwidth.

2. Allow us to use to unify interface for applications :-

Currently in linux there is no unified interface to issue
verify command so each application will have to duplicate the
code for the discovering controllers protocol type, opencoding
device passthru ioctl for protocol spefcific verify command and
issuing verify read emulation if it is not supported, see [1].

3. Allow us to use controller's internal bandwidth :-

For some controllers offloading the verify command can reault in
decrease in data block verification time, since their internal
bandwidth can be higher than host DMA transfer + OS storage stack
overhead of the read command.

4. Pro-actively avoiding unrecoverable read errors:-

Verify command does everything a normal read command does, except
for returning the data to the host system. This makes it possible
to do a low-level scrub of the stored data without being
bottlenecked by the host interface bandwidth.

Please note that analyzing controller verify command performance
for common protocols (SCSI/NVMe) is out of scope for REQ_OP_VERIFY.

-ck

[1] xfs_scrub issueing the verify command :-
xfs-progs/scrub/disk.c 340: disk_read_verify()->disk_scsi_verify()



  reply	other threads:[~2022-07-05 23:55 UTC|newest]

Thread overview: 41+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-06-30  9:14 [PATCH 0/6] block: add support for REQ_OP_VERIFY Chaitanya Kulkarni
2022-06-30  9:14 ` [PATCH 1/6] " Chaitanya Kulkarni
2022-06-30 16:18   ` Darrick J. Wong
2022-07-05 16:50     ` Chaitanya Kulkarni
2022-07-05 17:57       ` Darrick J. Wong
2022-07-06  1:32         ` Chaitanya Kulkarni
2022-07-05  8:34   ` Christoph Hellwig
2022-07-05 23:55     ` Chaitanya Kulkarni [this message]
2022-06-30  9:14 ` [PATCH 2/6] nvme: add support for the Verify command Chaitanya Kulkarni
2022-06-30 16:24   ` Keith Busch
2022-07-05  8:34     ` Christoph Hellwig
2022-07-05 23:56       ` Chaitanya Kulkarni
2022-06-30  9:14 ` [PATCH 3/6] nvmet: add Verify command support for bdev-ns Chaitanya Kulkarni
2022-06-30  9:14 ` [PATCH 4/6] nvmet: add Verify emulation " Chaitanya Kulkarni
2022-07-05  8:35   ` Christoph Hellwig
2022-07-06  0:00     ` Chaitanya Kulkarni
2022-06-30  9:14 ` [PATCH 5/6] nvmet: add verify emulation support for file-ns Chaitanya Kulkarni
2022-07-05  8:36   ` Christoph Hellwig
2022-07-06  0:06     ` Chaitanya Kulkarni
2022-06-30  9:14 ` [PATCH 6/6] null_blk: add REQ_OP_VERIFY support Chaitanya Kulkarni
2022-07-05  8:38 ` [PATCH 0/6] block: add support for REQ_OP_VERIFY Christoph Hellwig
2022-07-05 16:47   ` Chaitanya Kulkarni
2022-07-06 17:42 ` Matthew Wilcox
2022-07-13  9:14   ` Chaitanya Kulkarni
2022-07-13  9:36     ` Johannes Thumshirn
2022-07-13 12:17     ` Matthew Wilcox
2022-07-13 14:04       ` Keith Busch
2022-12-01 18:12   ` Chaitanya Kulkarni
2022-12-01 19:39     ` Matthew Wilcox
2022-12-02  7:16       ` Hannes Reinecke
2022-12-02 14:58         ` Keith Busch
2022-12-02 17:33           ` Clay Mayers
2022-12-02 18:58             ` Keith Busch
2022-12-09  4:52             ` Martin K. Petersen
2022-12-10 13:06               ` Carlos Carvalho
2022-12-12  6:30                 ` hch
2022-12-03  4:19           ` Javier González
2022-12-04 20:29             ` Keith Busch
2022-12-08 20:06               ` Javier González
2022-12-08 20:02             ` Matthew Wilcox
2022-12-02  7:13     ` Hannes Reinecke

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=dc23bece-359e-0da8-c7fc-4e632169b22f@nvidia.com \
    --to=chaitanyak@nvidia.com \
    --cc=adilger.kernel@dilger.ca \
    --cc=agk@redhat.com \
    --cc=axboe@kernel.dk \
    --cc=bvanassche@acm.org \
    --cc=clm@fb.com \
    --cc=danil.kipnis@cloud.ionos.com \
    --cc=djwong@kernel.org \
    --cc=dongli.zhang@oracle.com \
    --cc=dsterba@suse.com \
    --cc=ebiggers@google.com \
    --cc=hch@lst.de \
    --cc=idryomov@gmail.com \
    --cc=jack@suse.com \
    --cc=javier@javigon.com \
    --cc=jefflexu@linux.alibaba.com \
    --cc=jejb@linux.ibm.com \
    --cc=jinpu.wang@cloud.ionos.com \
    --cc=jlayton@kernel.org \
    --cc=johannes.thumshirn@wdc.com \
    --cc=josef@toxicpanda.com \
    --cc=kbusch@kernel.org \
    --cc=linux-block@vger.kernel.org \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-nvme@lists.infradead.org \
    --cc=linux-raid@vger.kernel.org \
    --cc=martin.petersen@oracle.com \
    --cc=ming.lei@redhat.com \
    --cc=sagi@grimberg.me \
    --cc=song@kernel.org \
    --cc=tytso@mit.edu \
    --cc=viro@zeniv.linux.org.uk \
    --cc=willy@infradead.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).