From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jeff Moyer Subject: Re: [PATCH v2 5/5] dax: handle media errors in dax_do_io Date: Mon, 02 May 2016 11:18:36 -0400 Message-ID: References: <1459303190-20072-1-git-send-email-vishal.l.verma@intel.com> <1459303190-20072-6-git-send-email-vishal.l.verma@intel.com> <20160420205923.GA24797@infradead.org> <1461434916.3695.7.camel@intel.com> <20160425083114.GA27556@infradead.org> <1461604476.3106.12.camel@intel.com> <20160425232552.GD18496@dastard> <1461628381.1421.24.camel@intel.com> <20160426004155.GF18496@dastard> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: "Verma\, Vishal L" , "linux-kernel\@vger.kernel.org" , "linux-block\@vger.kernel.org" , "hch\@infradead.org" , "xfs\@oss.sgi.com" , "linux-nvdimm\@ml01.01.org" , "linux-mm\@kvack.org" , "viro\@zeniv.linux.org.uk" , "axboe\@fb.com" , "akpm\@linux-foundation.org" , "linux-fsdevel\@vger.kernel.org" , "linux-ext4\@vger.kernel.org" , "Wilcox\, Matthew R" , "jack\@suse.cz" To: Dave Chinner , "Dan J. Williams" Return-path: In-Reply-To: <20160426004155.GF18496@dastard> (Dave Chinner's message of "Tue, 26 Apr 2016 10:41:55 +1000") Sender: owner-linux-mm@kvack.org List-Id: linux-ext4.vger.kernel.org Dave Chinner writes: > On Mon, Apr 25, 2016 at 11:53:13PM +0000, Verma, Vishal L wrote: >> On Tue, 2016-04-26 at 09:25 +1000, Dave Chinner wrote: > You're assuming that only the DAX aware application accesses it's > files. users, backup programs, data replicators, fileystem > re-organisers (e.g. defragmenters) etc all may access the files and > they may throw errors. What then? I'm not sure how this is any different from regular storage. If an application gets EIO, it's up to the app to decide what to do with that. >> > Where does the application find the data that was lost to be able to >> > rewrite it? >>=20 >> The data that was lost is gone -- this assumes the application has some >> ability to recover using a journal/log or other redundancy - yes, at the >> application layer. If it doesn't have this sort of capability, the only >> option is to restore files from a backup/mirror. > > So the architecture has a built in assumption that only userspace > can handle data loss? Remember that the proposed programming model completely bypasses the kernel, so yes, it is expected that user-space will have to deal with the problem. > What about filesytsems like NOVA, that use log structured design to > provide DAX w/ update atomicity and can potentially also provide > redundancy/repair through the same mechanisms? Won't pmem native > filesystems with built in data protection features like this remove > the need for adding all this to userspace applications? I don't think we'll /only/ support NOVA for pmem. So we'll have to deal with this for existing file systems, right? > If so, shouldn't that be the focus of development rahter than > placing the burden on userspace apps to handle storage repair > situations? It really depends on the programming model. In the model Vishal is talking about, either the applications themselves or the libraries they link to are expected to implement the redundancies where necessary. >> > There's an implicit assumption that applications will keep redundant >> > copies of their data at the /application layer/ and be able to >> > automatically repair it? That's one way to do things. It really depends on the application what it will do for recovery. >> > And then there's the implicit assumption that it will unlink and >> > free the entire file before writing a new copy I think Vishal was referring to restoring from backup. cp itself will truncate the file before overwriting, iirc. >> To summarize, the two cases we want to handle are: >> 1. Application has inbuilt recovery: >> =C2=A0 - hits badblock >> =C2=A0 - figures out it is able to recover the data >> =C2=A0 - handles SIGBUS or EIO >> =C2=A0 - does a (sector aligned) write() to restore the data > > The "figures out" step here is where >95% of the work we'd have to > do is. And that's in filesystem and block layer code, not > userspace, and userspace can't do that work in a signal handler. > And it can still fall down to the second case when the application > doesn't have another copy of the data somewhere. I read that "figures out" step as the application determining whether or not it had a redundant copy. > FWIW, we don't have a DAX enabled filesystem that can do > reverse block mapping, so we're a year or two away from this being a > workable production solution from the filesystem perspective. And > AFAICT, it's not even on the roadmap for dm/md layers. Do we even need that? What if we added an FIEMAP flag for determining bad blocks. The file system could simply walk the list of extents for the file and check the corresponding disk blocks. No reverse mapping required. Also note that DM/MD don't support direct_access(), either, so I don't think they're relevant for this discussion. >> 2. Application doesn't have any inbuilt recovery mechanism >> =C2=A0 - hits badblock >> =C2=A0 - gets SIGBUS (or EIO) and crashes >> =C2=A0 - Sysadmin restores file from backup > > Which is no different to an existing non-DAX application getting an > EIO/sigbus from current storage technologies. > > Except: in the existing storage stack, redundancy and correction has > already had to have failed for the application to see such an error. > Hence this is normally considered a DR case as there's had to be > cascading failures (e.g. multiple disk failures in a RAID) to get > to this stage, not a single error in a single sector in > non-redundant storage. > > We need some form of redundancy and correction in the PMEM stack to > prevent single sector errors from taking down services until an > administrator can correct the problem. I'm trying to understand > where this is supposed to fit into the picture - at this point I > really don't think userspace applications are going to be able to do > this reliably.... Not all storage is configured into a RAID volume, and in some instances, the application is better positioned to recover the data (gluster/ceph, for example). It really comes down to whether applications or libraries will want to implement redundancy themselves in order to get a bump in performance by not going through the kernel. And I think I know what your opinion is on that front. :-) Speaking of which, did you see the numbers Dan shared at LSF on how much overhead there is in calling into the kernel for syncing? Dan, can/did you publish that spreadsheet somewhere? Cheers, Jeff -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 876937CB8 for ; Mon, 2 May 2016 10:18:44 -0500 (CDT) Received: from cuda.sgi.com (cuda3.sgi.com [192.48.176.15]) by relay2.corp.sgi.com (Postfix) with ESMTP id 5855D304066 for ; Mon, 2 May 2016 08:18:41 -0700 (PDT) Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by cuda.sgi.com with ESMTP id xOxgwA8ZkXikWs8J (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 02 May 2016 08:18:40 -0700 (PDT) From: Jeff Moyer Subject: Re: [PATCH v2 5/5] dax: handle media errors in dax_do_io References: <1459303190-20072-1-git-send-email-vishal.l.verma@intel.com> <1459303190-20072-6-git-send-email-vishal.l.verma@intel.com> <20160420205923.GA24797@infradead.org> <1461434916.3695.7.camel@intel.com> <20160425083114.GA27556@infradead.org> <1461604476.3106.12.camel@intel.com> <20160425232552.GD18496@dastard> <1461628381.1421.24.camel@intel.com> <20160426004155.GF18496@dastard> Date: Mon, 02 May 2016 11:18:36 -0400 In-Reply-To: <20160426004155.GF18496@dastard> (Dave Chinner's message of "Tue, 26 Apr 2016 10:41:55 +1000") Message-ID: MIME-Version: 1.0 List-Id: XFS Filesystem from SGI List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: xfs-bounces@oss.sgi.com Sender: xfs-bounces@oss.sgi.com To: Dave Chinner , "Dan J. Williams" Cc: "hch@infradead.org" , "jack@suse.cz" , "axboe@fb.com" , "Verma, Vishal L" , "linux-kernel@vger.kernel.org" , "xfs@oss.sgi.com" , "linux-block@vger.kernel.org" , "linux-mm@kvack.org" , "viro@zeniv.linux.org.uk" , "linux-nvdimm@ml01.01.org" , "linux-fsdevel@vger.kernel.org" , "akpm@linux-foundation.org" , "linux-ext4@vger.kernel.org" , "Wilcox, Matthew R" RGF2ZSBDaGlubmVyIDxkYXZpZEBmcm9tb3JiaXQuY29tPiB3cml0ZXM6Cgo+IE9uIE1vbiwgQXBy IDI1LCAyMDE2IGF0IDExOjUzOjEzUE0gKzAwMDAsIFZlcm1hLCBWaXNoYWwgTCB3cm90ZToKPj4g T24gVHVlLCAyMDE2LTA0LTI2IGF0IDA5OjI1ICsxMDAwLCBEYXZlIENoaW5uZXIgd3JvdGU6Cj4g WW91J3JlIGFzc3VtaW5nIHRoYXQgb25seSB0aGUgREFYIGF3YXJlIGFwcGxpY2F0aW9uIGFjY2Vz c2VzIGl0J3MKPiBmaWxlcy4gIHVzZXJzLCBiYWNrdXAgcHJvZ3JhbXMsIGRhdGEgcmVwbGljYXRv cnMsIGZpbGV5c3RlbQo+IHJlLW9yZ2FuaXNlcnMgKGUuZy4gIGRlZnJhZ21lbnRlcnMpIGV0YyBh bGwgbWF5IGFjY2VzcyB0aGUgZmlsZXMgYW5kCj4gdGhleSBtYXkgdGhyb3cgZXJyb3JzLiBXaGF0 IHRoZW4/CgpJJ20gbm90IHN1cmUgaG93IHRoaXMgaXMgYW55IGRpZmZlcmVudCBmcm9tIHJlZ3Vs YXIgc3RvcmFnZS4gIElmIGFuCmFwcGxpY2F0aW9uIGdldHMgRUlPLCBpdCdzIHVwIHRvIHRoZSBh cHAgdG8gZGVjaWRlIHdoYXQgdG8gZG8gd2l0aCB0aGF0LgoKPj4gPiBXaGVyZSBkb2VzIHRoZSBh cHBsaWNhdGlvbiBmaW5kIHRoZSBkYXRhIHRoYXQgd2FzIGxvc3QgdG8gYmUgYWJsZSB0bwo+PiA+ IHJld3JpdGUgaXQ/Cj4+IAo+PiBUaGUgZGF0YSB0aGF0IHdhcyBsb3N0IGlzIGdvbmUgLS0gdGhp cyBhc3N1bWVzIHRoZSBhcHBsaWNhdGlvbiBoYXMgc29tZQo+PiBhYmlsaXR5IHRvIHJlY292ZXIg dXNpbmcgYSBqb3VybmFsL2xvZyBvciBvdGhlciByZWR1bmRhbmN5IC0geWVzLCBhdCB0aGUKPj4g YXBwbGljYXRpb24gbGF5ZXIuIElmIGl0IGRvZXNuJ3QgaGF2ZSB0aGlzIHNvcnQgb2YgY2FwYWJp bGl0eSwgdGhlIG9ubHkKPj4gb3B0aW9uIGlzIHRvIHJlc3RvcmUgZmlsZXMgZnJvbSBhIGJhY2t1 cC9taXJyb3IuCj4KPiBTbyB0aGUgYXJjaGl0ZWN0dXJlIGhhcyBhIGJ1aWx0IGluIGFzc3VtcHRp b24gdGhhdCBvbmx5IHVzZXJzcGFjZQo+IGNhbiBoYW5kbGUgZGF0YSBsb3NzPwoKUmVtZW1iZXIg dGhhdCB0aGUgcHJvcG9zZWQgcHJvZ3JhbW1pbmcgbW9kZWwgY29tcGxldGVseSBieXBhc3NlcyB0 aGUKa2VybmVsLCBzbyB5ZXMsIGl0IGlzIGV4cGVjdGVkIHRoYXQgdXNlci1zcGFjZSB3aWxsIGhh dmUgdG8gZGVhbCB3aXRoCnRoZSBwcm9ibGVtLgoKPiBXaGF0IGFib3V0IGZpbGVzeXRzZW1zIGxp a2UgTk9WQSwgdGhhdCB1c2UgbG9nIHN0cnVjdHVyZWQgZGVzaWduIHRvCj4gcHJvdmlkZSBEQVgg dy8gdXBkYXRlIGF0b21pY2l0eSBhbmQgY2FuIHBvdGVudGlhbGx5IGFsc28gcHJvdmlkZQo+IHJl ZHVuZGFuY3kvcmVwYWlyIHRocm91Z2ggdGhlIHNhbWUgbWVjaGFuaXNtcz8gV29uJ3QgcG1lbSBu YXRpdmUKPiBmaWxlc3lzdGVtcyB3aXRoIGJ1aWx0IGluIGRhdGEgcHJvdGVjdGlvbiBmZWF0dXJl cyBsaWtlIHRoaXMgcmVtb3ZlCj4gdGhlIG5lZWQgZm9yIGFkZGluZyBhbGwgdGhpcyB0byB1c2Vy c3BhY2UgYXBwbGljYXRpb25zPwoKSSBkb24ndCB0aGluayB3ZSdsbCAvb25seS8gc3VwcG9ydCBO T1ZBIGZvciBwbWVtLiAgU28gd2UnbGwgaGF2ZSB0byBkZWFsCndpdGggdGhpcyBmb3IgZXhpc3Rp bmcgZmlsZSBzeXN0ZW1zLCByaWdodD8KCj4gSWYgc28sIHNob3VsZG4ndCB0aGF0IGJlIHRoZSBm b2N1cyBvZiBkZXZlbG9wbWVudCByYWh0ZXIgdGhhbgo+IHBsYWNpbmcgdGhlIGJ1cmRlbiBvbiB1 c2Vyc3BhY2UgYXBwcyB0byBoYW5kbGUgc3RvcmFnZSByZXBhaXIKPiBzaXR1YXRpb25zPwoKSXQg cmVhbGx5IGRlcGVuZHMgb24gdGhlIHByb2dyYW1taW5nIG1vZGVsLiAgSW4gdGhlIG1vZGVsIFZp c2hhbCBpcwp0YWxraW5nIGFib3V0LCBlaXRoZXIgdGhlIGFwcGxpY2F0aW9ucyB0aGVtc2VsdmVz IG9yIHRoZSBsaWJyYXJpZXMgdGhleQpsaW5rIHRvIGFyZSBleHBlY3RlZCB0byBpbXBsZW1lbnQg dGhlIHJlZHVuZGFuY2llcyB3aGVyZSBuZWNlc3NhcnkuCgo+PiA+IFRoZXJlJ3MgYW4gaW1wbGlj aXQgYXNzdW1wdGlvbiB0aGF0IGFwcGxpY2F0aW9ucyB3aWxsIGtlZXAgcmVkdW5kYW50Cj4+ID4g Y29waWVzIG9mIHRoZWlyIGRhdGEgYXQgdGhlIC9hcHBsaWNhdGlvbiBsYXllci8gYW5kIGJlIGFi bGUgdG8KPj4gPiBhdXRvbWF0aWNhbGx5IHJlcGFpciBpdD8KClRoYXQncyBvbmUgd2F5IHRvIGRv IHRoaW5ncy4gIEl0IHJlYWxseSBkZXBlbmRzIG9uIHRoZSBhcHBsaWNhdGlvbiB3aGF0Cml0IHdp bGwgZG8gZm9yIHJlY292ZXJ5LgoKPj4gPiBBbmQgdGhlbiB0aGVyZSdzIHRoZSBpbXBsaWNpdCBh c3N1bXB0aW9uIHRoYXQgaXQgd2lsbCB1bmxpbmsgYW5kCj4+ID4gZnJlZSB0aGUgZW50aXJlIGZp bGUgYmVmb3JlIHdyaXRpbmcgYSBuZXcgY29weQoKSSB0aGluayBWaXNoYWwgd2FzIHJlZmVycmlu ZyB0byByZXN0b3JpbmcgZnJvbSBiYWNrdXAuICBjcCBpdHNlbGYgd2lsbAp0cnVuY2F0ZSB0aGUg ZmlsZSBiZWZvcmUgb3ZlcndyaXRpbmcsIGlpcmMuCgo+PiBUbyBzdW1tYXJpemUsIHRoZSB0d28g Y2FzZXMgd2Ugd2FudCB0byBoYW5kbGUgYXJlOgo+PiAxLiBBcHBsaWNhdGlvbiBoYXMgaW5idWls dCByZWNvdmVyeToKPj4gwqAgLSBoaXRzIGJhZGJsb2NrCj4+IMKgIC0gZmlndXJlcyBvdXQgaXQg aXMgYWJsZSB0byByZWNvdmVyIHRoZSBkYXRhCj4+IMKgIC0gaGFuZGxlcyBTSUdCVVMgb3IgRUlP Cj4+IMKgIC0gZG9lcyBhIChzZWN0b3IgYWxpZ25lZCkgd3JpdGUoKSB0byByZXN0b3JlIHRoZSBk YXRhCj4KPiBUaGUgImZpZ3VyZXMgb3V0IiBzdGVwIGhlcmUgaXMgd2hlcmUgPjk1JSBvZiB0aGUg d29yayB3ZSdkIGhhdmUgdG8KPiBkbyBpcy4gQW5kIHRoYXQncyBpbiBmaWxlc3lzdGVtIGFuZCBi bG9jayBsYXllciBjb2RlLCBub3QKPiB1c2Vyc3BhY2UsIGFuZCB1c2Vyc3BhY2UgY2FuJ3QgZG8g dGhhdCB3b3JrIGluIGEgc2lnbmFsIGhhbmRsZXIuCj4gQW5kIGl0ICBjYW4gc3RpbGwgZmFsbCBk b3duIHRvIHRoZSBzZWNvbmQgY2FzZSB3aGVuIHRoZSBhcHBsaWNhdGlvbgo+IGRvZXNuJ3QgaGF2 ZSBhbm90aGVyIGNvcHkgb2YgdGhlIGRhdGEgc29tZXdoZXJlLgoKSSByZWFkIHRoYXQgImZpZ3Vy ZXMgb3V0IiBzdGVwIGFzIHRoZSBhcHBsaWNhdGlvbiBkZXRlcm1pbmluZyB3aGV0aGVyIG9yCm5v dCBpdCBoYWQgYSByZWR1bmRhbnQgY29weS4KCj4gRldJVywgd2UgZG9uJ3QgaGF2ZSBhIERBWCBl bmFibGVkIGZpbGVzeXN0ZW0gdGhhdCBjYW4gZG8KPiByZXZlcnNlIGJsb2NrIG1hcHBpbmcsIHNv IHdlJ3JlIGEgeWVhciBvciB0d28gYXdheSBmcm9tIHRoaXMgYmVpbmcgYQo+IHdvcmthYmxlIHBy b2R1Y3Rpb24gc29sdXRpb24gZnJvbSB0aGUgZmlsZXN5c3RlbSBwZXJzcGVjdGl2ZS4gQW5kCj4g QUZBSUNULCBpdCdzIG5vdCBldmVuIG9uIHRoZSByb2FkbWFwIGZvciBkbS9tZCBsYXllcnMuCgpE byB3ZSBldmVuIG5lZWQgdGhhdD8gIFdoYXQgaWYgd2UgYWRkZWQgYW4gRklFTUFQIGZsYWcgZm9y IGRldGVybWluaW5nCmJhZCBibG9ja3MuICBUaGUgZmlsZSBzeXN0ZW0gY291bGQgc2ltcGx5IHdh bGsgdGhlIGxpc3Qgb2YgZXh0ZW50cyBmb3IKdGhlIGZpbGUgYW5kIGNoZWNrIHRoZSBjb3JyZXNw b25kaW5nIGRpc2sgYmxvY2tzLiAgTm8gcmV2ZXJzZSBtYXBwaW5nCnJlcXVpcmVkLiAgQWxzbyBu b3RlIHRoYXQgRE0vTUQgZG9uJ3Qgc3VwcG9ydCBkaXJlY3RfYWNjZXNzKCksIGVpdGhlciwKc28g SSBkb24ndCB0aGluayB0aGV5J3JlIHJlbGV2YW50IGZvciB0aGlzIGRpc2N1c3Npb24uCgo+PiAy LiBBcHBsaWNhdGlvbiBkb2Vzbid0IGhhdmUgYW55IGluYnVpbHQgcmVjb3ZlcnkgbWVjaGFuaXNt Cj4+IMKgIC0gaGl0cyBiYWRibG9jawo+PiDCoCAtIGdldHMgU0lHQlVTIChvciBFSU8pIGFuZCBj cmFzaGVzCj4+IMKgIC0gU3lzYWRtaW4gcmVzdG9yZXMgZmlsZSBmcm9tIGJhY2t1cAo+Cj4gV2hp Y2ggaXMgbm8gZGlmZmVyZW50IHRvIGFuIGV4aXN0aW5nIG5vbi1EQVggYXBwbGljYXRpb24gZ2V0 dGluZyBhbgo+IEVJTy9zaWdidXMgZnJvbSBjdXJyZW50IHN0b3JhZ2UgdGVjaG5vbG9naWVzLgo+ Cj4gRXhjZXB0OiBpbiB0aGUgZXhpc3Rpbmcgc3RvcmFnZSBzdGFjaywgcmVkdW5kYW5jeSBhbmQg Y29ycmVjdGlvbiBoYXMKPiBhbHJlYWR5IGhhZCB0byBoYXZlIGZhaWxlZCBmb3IgdGhlIGFwcGxp Y2F0aW9uIHRvIHNlZSBzdWNoIGFuIGVycm9yLgo+IEhlbmNlIHRoaXMgaXMgbm9ybWFsbHkgY29u c2lkZXJlZCBhIERSIGNhc2UgYXMgdGhlcmUncyBoYWQgdG8gYmUKPiBjYXNjYWRpbmcgZmFpbHVy ZXMgKGUuZy4gIG11bHRpcGxlIGRpc2sgZmFpbHVyZXMgaW4gYSBSQUlEKSB0byBnZXQKPiB0byB0 aGlzIHN0YWdlLCBub3QgYSBzaW5nbGUgZXJyb3IgaW4gYSBzaW5nbGUgc2VjdG9yIGluCj4gbm9u LXJlZHVuZGFudCBzdG9yYWdlLgo+Cj4gV2UgbmVlZCBzb21lIGZvcm0gb2YgcmVkdW5kYW5jeSBh bmQgY29ycmVjdGlvbiBpbiB0aGUgUE1FTSBzdGFjayB0bwo+IHByZXZlbnQgc2luZ2xlIHNlY3Rv ciBlcnJvcnMgZnJvbSB0YWtpbmcgZG93biBzZXJ2aWNlcyB1bnRpbCBhbgo+IGFkbWluaXN0cmF0 b3IgY2FuIGNvcnJlY3QgdGhlIHByb2JsZW0uIEknbSB0cnlpbmcgdG8gdW5kZXJzdGFuZAo+IHdo ZXJlIHRoaXMgaXMgc3VwcG9zZWQgdG8gZml0IGludG8gdGhlIHBpY3R1cmUgLSBhdCB0aGlzIHBv aW50IEkKPiByZWFsbHkgZG9uJ3QgdGhpbmsgdXNlcnNwYWNlIGFwcGxpY2F0aW9ucyBhcmUgZ29p bmcgdG8gYmUgYWJsZSB0byBkbwo+IHRoaXMgcmVsaWFibHkuLi4uCgpOb3QgYWxsIHN0b3JhZ2Ug aXMgY29uZmlndXJlZCBpbnRvIGEgUkFJRCB2b2x1bWUsIGFuZCBpbiBzb21lIGluc3RhbmNlcywK dGhlIGFwcGxpY2F0aW9uIGlzIGJldHRlciBwb3NpdGlvbmVkIHRvIHJlY292ZXIgdGhlIGRhdGEg KGdsdXN0ZXIvY2VwaCwKZm9yIGV4YW1wbGUpLiAgSXQgcmVhbGx5IGNvbWVzIGRvd24gdG8gd2hl dGhlciBhcHBsaWNhdGlvbnMgb3IgbGlicmFyaWVzCndpbGwgd2FudCB0byBpbXBsZW1lbnQgcmVk dW5kYW5jeSB0aGVtc2VsdmVzIGluIG9yZGVyIHRvIGdldCBhIGJ1bXAgaW4KcGVyZm9ybWFuY2Ug Ynkgbm90IGdvaW5nIHRocm91Z2ggdGhlIGtlcm5lbC4gIEFuZCBJIHRoaW5rIEkga25vdyB3aGF0 CnlvdXIgb3BpbmlvbiBpcyBvbiB0aGF0IGZyb250LiAgOi0pCgpTcGVha2luZyBvZiB3aGljaCwg ZGlkIHlvdSBzZWUgdGhlIG51bWJlcnMgRGFuIHNoYXJlZCBhdCBMU0Ygb24gaG93IG11Y2gKb3Zl cmhlYWQgdGhlcmUgaXMgaW4gY2FsbGluZyBpbnRvIHRoZSBrZXJuZWwgZm9yIHN5bmNpbmc/ICBE YW4sIGNhbi9kaWQKeW91IHB1Ymxpc2ggdGhhdCBzcHJlYWRzaGVldCBzb21ld2hlcmU/CgpDaGVl cnMsCkplZmYKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f CnhmcyBtYWlsaW5nIGxpc3QKeGZzQG9zcy5zZ2kuY29tCmh0dHA6Ly9vc3Muc2dpLmNvbS9tYWls bWFuL2xpc3RpbmZvL3hmcwo= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yw0-f197.google.com (mail-yw0-f197.google.com [209.85.161.197]) by kanga.kvack.org (Postfix) with ESMTP id 20B3D6B007E for ; Mon, 2 May 2016 11:18:41 -0400 (EDT) Received: by mail-yw0-f197.google.com with SMTP id x189so196114714ywe.2 for ; Mon, 02 May 2016 08:18:41 -0700 (PDT) Received: from mx1.redhat.com (mx1.redhat.com. [209.132.183.28]) by mx.google.com with ESMTPS id 199si15014837qhk.92.2016.05.02.08.18.39 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 02 May 2016 08:18:40 -0700 (PDT) From: Jeff Moyer Subject: Re: [PATCH v2 5/5] dax: handle media errors in dax_do_io References: <1459303190-20072-1-git-send-email-vishal.l.verma@intel.com> <1459303190-20072-6-git-send-email-vishal.l.verma@intel.com> <20160420205923.GA24797@infradead.org> <1461434916.3695.7.camel@intel.com> <20160425083114.GA27556@infradead.org> <1461604476.3106.12.camel@intel.com> <20160425232552.GD18496@dastard> <1461628381.1421.24.camel@intel.com> <20160426004155.GF18496@dastard> Date: Mon, 02 May 2016 11:18:36 -0400 In-Reply-To: <20160426004155.GF18496@dastard> (Dave Chinner's message of "Tue, 26 Apr 2016 10:41:55 +1000") Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Sender: owner-linux-mm@kvack.org List-ID: To: Dave Chinner , "Dan J. Williams" Cc: "Verma, Vishal L" , "linux-kernel@vger.kernel.org" , "linux-block@vger.kernel.org" , "hch@infradead.org" , "xfs@oss.sgi.com" , "linux-nvdimm@ml01.01.org" , "linux-mm@kvack.org" , "viro@zeniv.linux.org.uk" , "axboe@fb.com" , "akpm@linux-foundation.org" , "linux-fsdevel@vger.kernel.org" , "linux-ext4@vger.kernel.org" , "Wilcox, Matthew R" , "jack@suse.cz" Dave Chinner writes: > On Mon, Apr 25, 2016 at 11:53:13PM +0000, Verma, Vishal L wrote: >> On Tue, 2016-04-26 at 09:25 +1000, Dave Chinner wrote: > You're assuming that only the DAX aware application accesses it's > files. users, backup programs, data replicators, fileystem > re-organisers (e.g. defragmenters) etc all may access the files and > they may throw errors. What then? I'm not sure how this is any different from regular storage. If an application gets EIO, it's up to the app to decide what to do with that. >> > Where does the application find the data that was lost to be able to >> > rewrite it? >>=20 >> The data that was lost is gone -- this assumes the application has some >> ability to recover using a journal/log or other redundancy - yes, at the >> application layer. If it doesn't have this sort of capability, the only >> option is to restore files from a backup/mirror. > > So the architecture has a built in assumption that only userspace > can handle data loss? Remember that the proposed programming model completely bypasses the kernel, so yes, it is expected that user-space will have to deal with the problem. > What about filesytsems like NOVA, that use log structured design to > provide DAX w/ update atomicity and can potentially also provide > redundancy/repair through the same mechanisms? Won't pmem native > filesystems with built in data protection features like this remove > the need for adding all this to userspace applications? I don't think we'll /only/ support NOVA for pmem. So we'll have to deal with this for existing file systems, right? > If so, shouldn't that be the focus of development rahter than > placing the burden on userspace apps to handle storage repair > situations? It really depends on the programming model. In the model Vishal is talking about, either the applications themselves or the libraries they link to are expected to implement the redundancies where necessary. >> > There's an implicit assumption that applications will keep redundant >> > copies of their data at the /application layer/ and be able to >> > automatically repair it? That's one way to do things. It really depends on the application what it will do for recovery. >> > And then there's the implicit assumption that it will unlink and >> > free the entire file before writing a new copy I think Vishal was referring to restoring from backup. cp itself will truncate the file before overwriting, iirc. >> To summarize, the two cases we want to handle are: >> 1. Application has inbuilt recovery: >> =C2=A0 - hits badblock >> =C2=A0 - figures out it is able to recover the data >> =C2=A0 - handles SIGBUS or EIO >> =C2=A0 - does a (sector aligned) write() to restore the data > > The "figures out" step here is where >95% of the work we'd have to > do is. And that's in filesystem and block layer code, not > userspace, and userspace can't do that work in a signal handler. > And it can still fall down to the second case when the application > doesn't have another copy of the data somewhere. I read that "figures out" step as the application determining whether or not it had a redundant copy. > FWIW, we don't have a DAX enabled filesystem that can do > reverse block mapping, so we're a year or two away from this being a > workable production solution from the filesystem perspective. And > AFAICT, it's not even on the roadmap for dm/md layers. Do we even need that? What if we added an FIEMAP flag for determining bad blocks. The file system could simply walk the list of extents for the file and check the corresponding disk blocks. No reverse mapping required. Also note that DM/MD don't support direct_access(), either, so I don't think they're relevant for this discussion. >> 2. Application doesn't have any inbuilt recovery mechanism >> =C2=A0 - hits badblock >> =C2=A0 - gets SIGBUS (or EIO) and crashes >> =C2=A0 - Sysadmin restores file from backup > > Which is no different to an existing non-DAX application getting an > EIO/sigbus from current storage technologies. > > Except: in the existing storage stack, redundancy and correction has > already had to have failed for the application to see such an error. > Hence this is normally considered a DR case as there's had to be > cascading failures (e.g. multiple disk failures in a RAID) to get > to this stage, not a single error in a single sector in > non-redundant storage. > > We need some form of redundancy and correction in the PMEM stack to > prevent single sector errors from taking down services until an > administrator can correct the problem. I'm trying to understand > where this is supposed to fit into the picture - at this point I > really don't think userspace applications are going to be able to do > this reliably.... Not all storage is configured into a RAID volume, and in some instances, the application is better positioned to recover the data (gluster/ceph, for example). It really comes down to whether applications or libraries will want to implement redundancy themselves in order to get a bump in performance by not going through the kernel. And I think I know what your opinion is on that front. :-) Speaking of which, did you see the numbers Dan shared at LSF on how much overhead there is in calling into the kernel for syncing? Dan, can/did you publish that spreadsheet somewhere? Cheers, Jeff -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754095AbcEBPSs (ORCPT ); Mon, 2 May 2016 11:18:48 -0400 Received: from mx1.redhat.com ([209.132.183.28]:40201 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752211AbcEBPSk convert rfc822-to-8bit (ORCPT ); Mon, 2 May 2016 11:18:40 -0400 From: Jeff Moyer To: Dave Chinner , "Dan J. Williams" Cc: "Verma\, Vishal L" , "linux-kernel\@vger.kernel.org" , "linux-block\@vger.kernel.org" , "hch\@infradead.org" , "xfs\@oss.sgi.com" , "linux-nvdimm\@ml01.01.org" , "linux-mm\@kvack.org" , "viro\@zeniv.linux.org.uk" , "axboe\@fb.com" , "akpm\@linux-foundation.org" , "linux-fsdevel\@vger.kernel.org" , "linux-ext4\@vger.kernel.org" , "Wilcox\, Matthew R" , "jack\@suse.cz" Subject: Re: [PATCH v2 5/5] dax: handle media errors in dax_do_io References: <1459303190-20072-1-git-send-email-vishal.l.verma@intel.com> <1459303190-20072-6-git-send-email-vishal.l.verma@intel.com> <20160420205923.GA24797@infradead.org> <1461434916.3695.7.camel@intel.com> <20160425083114.GA27556@infradead.org> <1461604476.3106.12.camel@intel.com> <20160425232552.GD18496@dastard> <1461628381.1421.24.camel@intel.com> <20160426004155.GF18496@dastard> X-PGP-KeyID: 1F78E1B4 X-PGP-CertKey: F6FE 280D 8293 F72C 65FD 5A58 1FF8 A7CA 1F78 E1B4 X-PCLoadLetter: What the f**k does that mean? Date: Mon, 02 May 2016 11:18:36 -0400 In-Reply-To: <20160426004155.GF18496@dastard> (Dave Chinner's message of "Tue, 26 Apr 2016 10:41:55 +1000") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8BIT X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Mon, 02 May 2016 15:18:39 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Dave Chinner writes: > On Mon, Apr 25, 2016 at 11:53:13PM +0000, Verma, Vishal L wrote: >> On Tue, 2016-04-26 at 09:25 +1000, Dave Chinner wrote: > You're assuming that only the DAX aware application accesses it's > files. users, backup programs, data replicators, fileystem > re-organisers (e.g. defragmenters) etc all may access the files and > they may throw errors. What then? I'm not sure how this is any different from regular storage. If an application gets EIO, it's up to the app to decide what to do with that. >> > Where does the application find the data that was lost to be able to >> > rewrite it? >> >> The data that was lost is gone -- this assumes the application has some >> ability to recover using a journal/log or other redundancy - yes, at the >> application layer. If it doesn't have this sort of capability, the only >> option is to restore files from a backup/mirror. > > So the architecture has a built in assumption that only userspace > can handle data loss? Remember that the proposed programming model completely bypasses the kernel, so yes, it is expected that user-space will have to deal with the problem. > What about filesytsems like NOVA, that use log structured design to > provide DAX w/ update atomicity and can potentially also provide > redundancy/repair through the same mechanisms? Won't pmem native > filesystems with built in data protection features like this remove > the need for adding all this to userspace applications? I don't think we'll /only/ support NOVA for pmem. So we'll have to deal with this for existing file systems, right? > If so, shouldn't that be the focus of development rahter than > placing the burden on userspace apps to handle storage repair > situations? It really depends on the programming model. In the model Vishal is talking about, either the applications themselves or the libraries they link to are expected to implement the redundancies where necessary. >> > There's an implicit assumption that applications will keep redundant >> > copies of their data at the /application layer/ and be able to >> > automatically repair it? That's one way to do things. It really depends on the application what it will do for recovery. >> > And then there's the implicit assumption that it will unlink and >> > free the entire file before writing a new copy I think Vishal was referring to restoring from backup. cp itself will truncate the file before overwriting, iirc. >> To summarize, the two cases we want to handle are: >> 1. Application has inbuilt recovery: >>   - hits badblock >>   - figures out it is able to recover the data >>   - handles SIGBUS or EIO >>   - does a (sector aligned) write() to restore the data > > The "figures out" step here is where >95% of the work we'd have to > do is. And that's in filesystem and block layer code, not > userspace, and userspace can't do that work in a signal handler. > And it can still fall down to the second case when the application > doesn't have another copy of the data somewhere. I read that "figures out" step as the application determining whether or not it had a redundant copy. > FWIW, we don't have a DAX enabled filesystem that can do > reverse block mapping, so we're a year or two away from this being a > workable production solution from the filesystem perspective. And > AFAICT, it's not even on the roadmap for dm/md layers. Do we even need that? What if we added an FIEMAP flag for determining bad blocks. The file system could simply walk the list of extents for the file and check the corresponding disk blocks. No reverse mapping required. Also note that DM/MD don't support direct_access(), either, so I don't think they're relevant for this discussion. >> 2. Application doesn't have any inbuilt recovery mechanism >>   - hits badblock >>   - gets SIGBUS (or EIO) and crashes >>   - Sysadmin restores file from backup > > Which is no different to an existing non-DAX application getting an > EIO/sigbus from current storage technologies. > > Except: in the existing storage stack, redundancy and correction has > already had to have failed for the application to see such an error. > Hence this is normally considered a DR case as there's had to be > cascading failures (e.g. multiple disk failures in a RAID) to get > to this stage, not a single error in a single sector in > non-redundant storage. > > We need some form of redundancy and correction in the PMEM stack to > prevent single sector errors from taking down services until an > administrator can correct the problem. I'm trying to understand > where this is supposed to fit into the picture - at this point I > really don't think userspace applications are going to be able to do > this reliably.... Not all storage is configured into a RAID volume, and in some instances, the application is better positioned to recover the data (gluster/ceph, for example). It really comes down to whether applications or libraries will want to implement redundancy themselves in order to get a bump in performance by not going through the kernel. And I think I know what your opinion is on that front. :-) Speaking of which, did you see the numbers Dan shared at LSF on how much overhead there is in calling into the kernel for syncing? Dan, can/did you publish that spreadsheet somewhere? Cheers, Jeff