From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham
	autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 072E2C64E7B
	for <linux-unionfs@archiver.kernel.org>; Fri, 27 Nov 2020 09:21:16 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id BA119206D9
	for <linux-unionfs@archiver.kernel.org>; Fri, 27 Nov 2020 09:21:15 +0000 (UTC)
Authentication-Results: mail.kernel.org;
	dkim=pass (1024-bit key) header.d=sargun.me header.i=@sargun.me header.b="AF/93sZn"
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1725985AbgK0JVH (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Fri, 27 Nov 2020 04:21:07 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42026 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1726014AbgK0JVH (ORCPT
        <rfc822;linux-unionfs@vger.kernel.org>);
        Fri, 27 Nov 2020 04:21:07 -0500
Received: from mail-pg1-x544.google.com (mail-pg1-x544.google.com [IPv6:2607:f8b0:4864:20::544])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 70BA3C0613D1
        for <linux-unionfs@vger.kernel.org>; Fri, 27 Nov 2020 01:21:07 -0800 (PST)
Received: by mail-pg1-x544.google.com with SMTP id m9so3886551pgb.4
        for <linux-unionfs@vger.kernel.org>; Fri, 27 Nov 2020 01:21:07 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=sargun.me; s=google;
        h=from:to:cc:subject:date:message-id:mime-version
         :content-transfer-encoding;
        bh=vlPx3D5OXHB3+bHYIYOC/2iLJYt2laDE8eD7s5RHs1A=;
        b=AF/93sZng2Jpchw688suTMaOITMbGX2shVh+vKA4ZUXwvo98Mv3TqqtO0ZP5fxrPdz
         dodDdIsrnZ6jlZTj31N4emPWoFqjIYW0/47hCrQkeO4R65ChCzucA5bs6nekcpyzhrkh
         LkzPmr7FIyHy1NknUTUVp0WcMxPk0Y0m1o5ro=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version
         :content-transfer-encoding;
        bh=vlPx3D5OXHB3+bHYIYOC/2iLJYt2laDE8eD7s5RHs1A=;
        b=kNmOs/mViOuU0ZfQO/7LQx9C7jwjMwKM0Cu+OIWd9uvJoCsbM9gcNKMv1h1MD79a/w
         ZcNv8y1bPxM0iGrHMcXKhDpOMAxf1AQArCseeCW1e2v5sSna0E68NgQ7x+xLFHHBrCvW
         40aLl3XMTcwfsXBabOI0ywEr5yyptoiEZKDBD7FH/6JVZpPl70cbH0rgPtJ/rbIMjQiE
         F7N6LURGOcGLLw2C5CLr+/0BrArqLotKPQB1KtVUiddR8QglIuo+w1smFHebrhYEzQn/
         D8a6GaWc0lM4sfJ3NaJzDxB+vnLDScrROu6RjD8P9xx4ZgGG3HOIbMeQkB9wbfs1j6pq
         mKrA==
X-Gm-Message-State: AOAM531fWt+6OX8yxjsd7ia3Gdv/laX6NfcOQOSVTqVRZRAf1SQzrgLg
        OWOgvVWgbefRsOvCu2w9fLo3lsANUYQaXw==
X-Google-Smtp-Source: ABdhPJzb6MrII5EGaD2Ma7r6y4r5a7mmT/w1ly/jearBVl7o4Do185z1fw1HjYPQlowr/CvPvJmvsg==
X-Received: by 2002:aa7:8c16:0:b029:196:33d2:721f with SMTP id c22-20020aa78c160000b029019633d2721fmr6227235pfd.70.1606468866312;
        Fri, 27 Nov 2020 01:21:06 -0800 (PST)
Received: from ubuntu.netflix.com (203.20.25.136.in-addr.arpa. [136.25.20.203])
        by smtp.gmail.com with ESMTPSA id t9sm9938944pjq.46.2020.11.27.01.21.04
        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
        Fri, 27 Nov 2020 01:21:05 -0800 (PST)
From:   Sargun Dhillon <sargun@sargun.me>
To:     linux-unionfs@vger.kernel.org, miklos@szeredi.hu,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Amir Goldstein <amir73il@gmail.com>
Cc:     Sargun Dhillon <sargun@sargun.me>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Vivek Goyal <vgoyal@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel@vger.kernel.org, David Howells <dhowells@redhat.com>
Subject: [PATCH v2 0/4] Make overlayfs volatile mounts reusable
Date:   Fri, 27 Nov 2020 01:20:54 -0800
Message-Id: <20201127092058.15117-1-sargun@sargun.me>
X-Mailer: git-send-email 2.25.1
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org

This adds some documentation on how the current behaviour of overlayfs
works, and adds some non-obvious caveats to the documentation. We may want
to pick those up independently.

The volatile option is great for "ephemeral" containers. Unfortunately,
it doesn't capture all uses. There are two ways to use it safely right now:

1. Throw away the entire upperdir between mounts
2. Manually syncfs between mounts

For certain use-cases like serverless, or short-lived containers, it is
advantageous to be able to stop the container (runtime) and start it up on
demand / invocation of the function. Usually, there is some bootstrap
process which involves downloading some artifacts, or putting secrets on
disk, and then upon invocation of the function, you want to (re)start the
container.

If you have to syncfs every time you do this, it can lead to excess
filesystem overhead for all of the other containers on the machine, and
stall out every container who's upperdir is on the same underlying
filesystem, unless your filesystem offers something like subvolumes,
and if sync can be restricted to a subvolume.

The kernel has information that it can use to determine whether or not this
is safe -- primarily if the underlying FS has had writeback errors or not.
Overlayfs doesn't delay writes, so the consistency of the upperdir is not
contingent on the mount of overlayfs, but rather the mount of the
underlying filesystem. It can also make sure the underlying filesystem
wasn't remounted. Although, it was suggested that we use derive this
information from the upperdir's inode[1], we can checkpoint this data on
disk in an xattr.

Specifically we checkpoint:
  * Superblock "id": This is a new concept introduced in one of the patches
    which keeps track of (re)mounts of filesystems, by having a per boot
    monotonically increasing integer identifying the superblock. This is
    safer than trying to obfuscate the pointer and putting it into an
    xattr (due to leak risk, and address reuse), and during the course
    of a boot, the u64 should not wrap.
  * Overlay "boot id": This is a new UUID that is overlayfs specific,
    as overlayfs is a module that's independent from the rest of the
    system and can be (re)loaded independently -- thus it generates
    a UUID at load time which can be used to uniquely identify it.
  * upperdir / workdir errseq: A sample of the errseq_t on the workdir /
    upperdir's superblock. Since the errseq_t is implemented as a u32
    with errno + error counter, we can safely store it in a checkpoint.

This has to be done in kernelspace because userspace does not have access
to information such as superblock (re)mounts, the writeback errseq value,
and whether the module has been (re)loaded. Although this mechanism only
determines a subset of the error scenarios, it lays out the groundwork
for adding more.

In the future, we may want to make it so overlayfs shuts down on errors,
or have some kind of other behaviour when errors are detected. If we want
a per-file (on the upperdir) piece of metadata can be added indicating
errseq_t for just that file, and we can check it on each close / open,
and then allow the user to make an intelligent decision of how they want
overlayfs to handle the error. This would allow for errors to be captured
during normal operation as opposed to just between remounts.

This also allows for:
volatile -> volatile
non-volatile -> volatile
non-volatile -> non-volatile

But, for now, prevents volatile -> non-volatile. Perhaps with the future
patches around selective inode sync, and tracking errors on every
interaction will make this possible.


Changes since:
V1 [2]:
 * Add documentation commit about current short-comings of the current volatile
   implementation.
 * Do not allow volatile mounts to be mounted as non-volatile
RFC [3]:
 * Changed datastructure names
 * No longer delete the volatile dir / dirty file
 * Moved xattr to volatile directory
 * Split errseq check

[1]: https://lore.kernel.org/linux-unionfs/CAOQ4uxhadzC3-kh-igfxv3pAmC3ocDtAQTxByu4hrn8KtZuieQ@mail.gmail.com/
[2]: https://lore.kernel.org/linux-unionfs/20201125104621.18838-1-sargun@sargun.me/T/#t
[3]: https://lore.kernel.org/linux-unionfs/20201116045758.21774-1-sargun@sargun.me/T/#t

Sargun Dhillon (4):
  fs: Add s_instance_id field to superblock for unique identification
  overlay: Document current outstanding shortcoming of volatile
  overlay: Add the ability to remount volatile directories when safe
  overlay: Add rudimentary checking of writeback errseq on volatile
    remount

 Documentation/filesystems/overlayfs.rst |  24 ++++--
 fs/overlayfs/overlayfs.h                |  38 ++++++++-
 fs/overlayfs/readdir.c                  | 104 +++++++++++++++++++++---
 fs/overlayfs/super.c                    |  74 +++++++++++++----
 fs/overlayfs/util.c                     |   2 +
 fs/super.c                              |   3 +
 include/linux/fs.h                      |   7 ++
 7 files changed, 213 insertions(+), 39 deletions(-)


base-commit: be4df0cea08a8b59eb38d73de988b7ba8022df41
-- 
2.25.1


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,
	INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT
	autolearn=unavailable autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id A22FFC64E7C
	for <linux-unionfs@archiver.kernel.org>; Fri, 27 Nov 2020 09:21:16 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id 35FC221D93
	for <linux-unionfs@archiver.kernel.org>; Fri, 27 Nov 2020 09:21:16 +0000 (UTC)
Authentication-Results: mail.kernel.org;
	dkim=pass (1024-bit key) header.d=sargun.me header.i=@sargun.me header.b="HDolisLK"
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1728033AbgK0JVJ (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Fri, 27 Nov 2020 04:21:09 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42036 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1727841AbgK0JVJ (ORCPT
        <rfc822;linux-unionfs@vger.kernel.org>);
        Fri, 27 Nov 2020 04:21:09 -0500
Received: from mail-pf1-x442.google.com (mail-pf1-x442.google.com [IPv6:2607:f8b0:4864:20::442])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E3DD8C0613D4
        for <linux-unionfs@vger.kernel.org>; Fri, 27 Nov 2020 01:21:08 -0800 (PST)
Received: by mail-pf1-x442.google.com with SMTP id y7so3991380pfq.11
        for <linux-unionfs@vger.kernel.org>; Fri, 27 Nov 2020 01:21:08 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=sargun.me; s=google;
        h=from:to:cc:subject:date:message-id:in-reply-to:references
         :mime-version:content-transfer-encoding;
        bh=363/J7FI+YxpDHZBVwKbULZE4SMrK80CjI/t3VX+JKo=;
        b=HDolisLKeNAEQQTAJLzgVwk1K3d89nsQAXqrAwBs/8B5KxgdeZmaDrkUzA4aXOruDe
         LsnoAJfJxmngvDz/I0XpUrYkmLeVNpBa5iOU+UyoeG3Ex+MSlZsroy/T4Lca+9Z+eFaG
         MGT+lFob3K4ZK1fsisjG4sk2mVzNgNJSy/s90=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to
         :references:mime-version:content-transfer-encoding;
        bh=363/J7FI+YxpDHZBVwKbULZE4SMrK80CjI/t3VX+JKo=;
        b=h4sRvD6Gz/qNkRN3/J0yTfQ0NIqmTHOylsnwIYMFCBSJS2mc2707HPzRJQ2nFHDfIs
         K3/XxXe6KT+3OwHZAePG5urpPFgMwGL3Mu1z9Pp6wsFsXKfN1Q7i5yfSkiG0kwLxUD28
         PY5mBVUk8zk+qEvH5QWf2pI1TFglvj98hipRwcQmVhMFygPPUmYF26ihEBQTD3tcqD9Q
         eMyLEOILSUvejGfjv0GWnm/o0fTVRZtCutSWq2r/yVnqtdJO2rBgROaWNb2y+Ndyj9/q
         Lk7XJ0392TQPU37Yc8DbZDPPePGRXOQUytWf4cMtlvQ8D5vfa9trcBqQn6uyDRflmRhH
         vgug==
X-Gm-Message-State: AOAM531elylMGNHNTe6nwlhgCkfOdSBvrQyAkJZShpZcHEo0/W+eLqoB
        NYLr/O7je6lGY+KB0NC3ZgFmpzlJzPb9uw==
X-Google-Smtp-Source: ABdhPJw0CK18w9KERQ13VRm/od39VBC87CyOcbdF7TfrPfunyw6nFTO727BRKDp9c7TgTCStElzIjQ==
X-Received: by 2002:a17:90a:4283:: with SMTP id p3mr8846906pjg.174.1606468868120;
        Fri, 27 Nov 2020 01:21:08 -0800 (PST)
Received: from ubuntu.netflix.com (203.20.25.136.in-addr.arpa. [136.25.20.203])
        by smtp.gmail.com with ESMTPSA id t9sm9938944pjq.46.2020.11.27.01.21.06
        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
        Fri, 27 Nov 2020 01:21:07 -0800 (PST)
From:   Sargun Dhillon <sargun@sargun.me>
To:     linux-unionfs@vger.kernel.org, miklos@szeredi.hu,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Amir Goldstein <amir73il@gmail.com>
Cc:     Sargun Dhillon <sargun@sargun.me>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Vivek Goyal <vgoyal@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel@vger.kernel.org, David Howells <dhowells@redhat.com>
Subject: [PATCH v2 1/4] fs: Add s_instance_id field to superblock for unique identification
Date:   Fri, 27 Nov 2020 01:20:55 -0800
Message-Id: <20201127092058.15117-2-sargun@sargun.me>
X-Mailer: git-send-email 2.25.1
In-Reply-To: <20201127092058.15117-1-sargun@sargun.me>
References: <20201127092058.15117-1-sargun@sargun.me>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org

This assigns a per-boot unique number to each superblock. This allows
other components to know whether a filesystem has been remounted
since they last interacted with it.

At every boot it is reset to 0. There is no specific reason it is set to 0,
other than repeatability versus using some random starting number. Because
of this, you must store it along some other piece of data which is
initialized at boot time.

This doesn't have any of the overhead of idr, and a u64 wont wrap any time
soon. There is no forward lookup requirement, so an idr is not needed.

In the future, we may want to expose this to userspace. Userspace programs
can benefit from this if they have large chunks of dirty or mmaped memory
that they're interacting with, and they want to see if that volume has been
unmounted, and remounted. Along with this, and a mechanism to inspect the
superblock's errseq a user can determine whether they need to throw away
their cache or similar. This is another benefit in comparison to just
using a pointer to the superblock to uniquely identify it.

Although this doesn't expose an ioctl or similar yet, in the future we
could add an ioctl that allows for fetching the s_instance_id for a given
cache, and inspection of the errseq associated with that.

Signed-off-by: Sargun Dhillon <sargun@sargun.me>
Cc: David Howells <dhowells@redhat.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: linux-fsdevel@vger.kernel.org
Cc: linux-unionfs@vger.kernel.org
---
 fs/super.c         | 3 +++
 include/linux/fs.h | 7 +++++++
 2 files changed, 10 insertions(+)

diff --git a/fs/super.c b/fs/super.c
index 904459b35119..e47ace7f8c3d 100644
--- a/fs/super.c
+++ b/fs/super.c
@@ -42,6 +42,7 @@
 
 static int thaw_super_locked(struct super_block *sb);
 
+static u64 s_instance_id_counter;
 static LIST_HEAD(super_blocks);
 static DEFINE_SPINLOCK(sb_lock);
 
@@ -546,6 +547,7 @@ struct super_block *sget_fc(struct fs_context *fc,
 	s->s_iflags |= fc->s_iflags;
 	strlcpy(s->s_id, s->s_type->name, sizeof(s->s_id));
 	list_add_tail(&s->s_list, &super_blocks);
+	s->s_instance_id = s_instance_id_counter++;
 	hlist_add_head(&s->s_instances, &s->s_type->fs_supers);
 	spin_unlock(&sb_lock);
 	get_filesystem(s->s_type);
@@ -625,6 +627,7 @@ struct super_block *sget(struct file_system_type *type,
 	s->s_type = type;
 	strlcpy(s->s_id, type->name, sizeof(s->s_id));
 	list_add_tail(&s->s_list, &super_blocks);
+	s->s_instance_id = s_instance_id_counter++;
 	hlist_add_head(&s->s_instances, &type->fs_supers);
 	spin_unlock(&sb_lock);
 	get_filesystem(type);
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 7519ae003a08..09bf54382a54 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -1472,6 +1472,13 @@ struct super_block {
 	char			s_id[32];	/* Informational name */
 	uuid_t			s_uuid;		/* UUID */
 
+	/*
+	 * ID identifying this particular instance of the superblock. It can
+	 * be used to determine if a particular filesystem has been remounted.
+	 * It may be exposed to userspace.
+	 */
+	u64			s_instance_id;
+
 	unsigned int		s_max_links;
 	fmode_t			s_mode;
 
-- 
2.25.1


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,
	INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,
	USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 2A17FC64E90
	for <linux-unionfs@archiver.kernel.org>; Fri, 27 Nov 2020 09:21:17 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id C2AFE206D9
	for <linux-unionfs@archiver.kernel.org>; Fri, 27 Nov 2020 09:21:16 +0000 (UTC)
Authentication-Results: mail.kernel.org;
	dkim=pass (1024-bit key) header.d=sargun.me header.i=@sargun.me header.b="XCbkKPcW"
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1728063AbgK0JVM (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Fri, 27 Nov 2020 04:21:12 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42046 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1726014AbgK0JVM (ORCPT
        <rfc822;linux-unionfs@vger.kernel.org>);
        Fri, 27 Nov 2020 04:21:12 -0500
Received: from mail-pg1-x541.google.com (mail-pg1-x541.google.com [IPv6:2607:f8b0:4864:20::541])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B743DC0613D1
        for <linux-unionfs@vger.kernel.org>; Fri, 27 Nov 2020 01:21:10 -0800 (PST)
Received: by mail-pg1-x541.google.com with SMTP id w4so3855133pgg.13
        for <linux-unionfs@vger.kernel.org>; Fri, 27 Nov 2020 01:21:10 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=sargun.me; s=google;
        h=from:to:cc:subject:date:message-id:in-reply-to:references
         :mime-version:content-transfer-encoding;
        bh=N3znUkPKtqbryPODPnE7qItfA1c9fsYLlSw8ARlE2iU=;
        b=XCbkKPcW90nk8ZdKWXtOH5i9ZE+g53qlXZ8cLPT4L1dCcgEvMy8gG5O6KOjkCiRTS6
         T1sT0tOswqbzJT3gDHM0Pr4X0oa1KJkNX6xjd2KuxpINgXtXWLgF9ffxKbDKP1rjpMFz
         nxxOy0yG5pnW+FbC2hlo0yE1i2S7eA9V7xfbA=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to
         :references:mime-version:content-transfer-encoding;
        bh=N3znUkPKtqbryPODPnE7qItfA1c9fsYLlSw8ARlE2iU=;
        b=sHf0mpeA7/gzAFnVdil7mD4IirJ+fDBG78b8BXsAwiCEGmHmeFmFWYZS8v7/TEoxQN
         Ckh2/euyrQIRA0Pb8YWfljFmUURQqiflxJXm0CsgJuqTgueviSRLDfkQqOpmW/D3a8f2
         +xuNCtR75d521HenD+GwEBN6C1JJLVUhbALFnS38kSlKz/F9/8eweWC9rGuWjdw37PC5
         8LXwL/SBQcTLlW8vne27VL67y4NLVNjbM7titxFHOYq/xc7dDpf9Rx0a9d4PIQRDv5/1
         MO7oESU2F5YEZdFmcI9qQ6qqP4yBcGr20qDWNBzSb+vjumDbtv9oFUDrK4kcYvgef6Xk
         nQ+A==
X-Gm-Message-State: AOAM530eyATABOiXEKZNDEGBomOFk27hmxaOo8wkGvdNZ/ciGMS6qlum
        xT+21MnOfpEv2o0/pUsQ+R8Y77CrlDYrgQ==
X-Google-Smtp-Source: ABdhPJy+Nm0BqxYRvhxozEhyB0RB1iAA9/rcKtYRc0oHpzl11BL3ij3EUW9jZDE3KklG4mSjjnCSJA==
X-Received: by 2002:a17:90a:2a83:: with SMTP id j3mr9035135pjd.84.1606468869856;
        Fri, 27 Nov 2020 01:21:09 -0800 (PST)
Received: from ubuntu.netflix.com (203.20.25.136.in-addr.arpa. [136.25.20.203])
        by smtp.gmail.com with ESMTPSA id t9sm9938944pjq.46.2020.11.27.01.21.08
        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
        Fri, 27 Nov 2020 01:21:09 -0800 (PST)
From:   Sargun Dhillon <sargun@sargun.me>
To:     linux-unionfs@vger.kernel.org, miklos@szeredi.hu,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Amir Goldstein <amir73il@gmail.com>
Cc:     Sargun Dhillon <sargun@sargun.me>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Vivek Goyal <vgoyal@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel@vger.kernel.org, David Howells <dhowells@redhat.com>
Subject: [PATCH v2 2/4] overlay: Document current outstanding shortcoming of volatile
Date:   Fri, 27 Nov 2020 01:20:56 -0800
Message-Id: <20201127092058.15117-3-sargun@sargun.me>
X-Mailer: git-send-email 2.25.1
In-Reply-To: <20201127092058.15117-1-sargun@sargun.me>
References: <20201127092058.15117-1-sargun@sargun.me>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org

This documents behaviour that was discussed in a thread about the volatile
feature. Specifically, how failures can go hidden from asynchronous writes
(such as from mmap, or writes that are not immediately flushed to the
filesystem). Although we pass through calls like msync, fallocate, and
write, and will still return errors on those, it doesn't guarantee all
kinds of errors will happen at those times, and thus may hide errors.

In the future, we can add error checking to all interactions with the
upperdir, and pass through errseq_t from the upperdir on mappings,
and other interactions with the filesystem[1].

[1]: https://lore.kernel.org/linux-unionfs/20201116045758.21774-1-sargun@sargun.me/T/#m7d501f375e031056efad626e471a1392dd3aad33

Signed-off-by: Sargun Dhillon <sargun@sargun.me>
Cc: linux-fsdevel@vger.kernel.org
Cc: linux-unionfs@vger.kernel.org
Cc: Miklos Szeredi <miklos@szeredi.hu>
Cc: Amir Goldstein <amir73il@gmail.com>
Cc: Vivek Goyal <vgoyal@redhat.com>
---
 Documentation/filesystems/overlayfs.rst | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/Documentation/filesystems/overlayfs.rst b/Documentation/filesystems/overlayfs.rst
index 580ab9a0fe31..c6e30c1bc2f2 100644
--- a/Documentation/filesystems/overlayfs.rst
+++ b/Documentation/filesystems/overlayfs.rst
@@ -570,7 +570,11 @@ Volatile mount
 This is enabled with the "volatile" mount option.  Volatile mounts are not
 guaranteed to survive a crash.  It is strongly recommended that volatile
 mounts are only used if data written to the overlay can be recreated
-without significant effort.
+without significant effort.  In addition to this, the sync family of syscalls
+are not sufficient to determine whether a write failed as sync calls are
+omitted.  For this reason, it is important that the filesystem used by the
+upperdir handles failure in a fashion that's suitable for the user.  For
+example, upon detecting a fault, ext4 can be configured to panic.
 
 The advantage of mounting with the "volatile" option is that all forms of
 sync calls to the upper filesystem are omitted.
-- 
2.25.1


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,
	INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT
	autolearn=ham autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id E165EC2D0E4
	for <linux-unionfs@archiver.kernel.org>; Fri, 27 Nov 2020 09:21:17 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id 954C5206D9
	for <linux-unionfs@archiver.kernel.org>; Fri, 27 Nov 2020 09:21:17 +0000 (UTC)
Authentication-Results: mail.kernel.org;
	dkim=pass (1024-bit key) header.d=sargun.me header.i=@sargun.me header.b="BvVHKFHm"
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1728114AbgK0JVP (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Fri, 27 Nov 2020 04:21:15 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42064 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1728078AbgK0JVO (ORCPT
        <rfc822;linux-unionfs@vger.kernel.org>);
        Fri, 27 Nov 2020 04:21:14 -0500
Received: from mail-pl1-x642.google.com (mail-pl1-x642.google.com [IPv6:2607:f8b0:4864:20::642])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CAF2CC0613D4
        for <linux-unionfs@vger.kernel.org>; Fri, 27 Nov 2020 01:21:14 -0800 (PST)
Received: by mail-pl1-x642.google.com with SMTP id v21so2389212plo.12
        for <linux-unionfs@vger.kernel.org>; Fri, 27 Nov 2020 01:21:14 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=sargun.me; s=google;
        h=from:to:cc:subject:date:message-id:in-reply-to:references
         :mime-version:content-transfer-encoding;
        bh=okmQhfPLGgBjgjZXbAm+aIoWMwXVsIOZ9BcK51UFRVI=;
        b=BvVHKFHm+5yavD2kii5ezOno5mhVTxOxRqvPhwq2ZnepF4QcmilndtgKa4E4XKjivE
         QKdejxSbJlfZ/cy5wfV7pZTIzrVmELPefTzuRmRzPrquEAGqCyBYaDRE9+vLPHnPRa5h
         SCjMvFlBWRuy4HEG7ld/MUhxYm0Ai0wz7nBok=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to
         :references:mime-version:content-transfer-encoding;
        bh=okmQhfPLGgBjgjZXbAm+aIoWMwXVsIOZ9BcK51UFRVI=;
        b=pfkRIq2idcKtpv68IBZ3Bc4ZLXiSHx0Vx8tMk8tJ3jJ9onLfH096+Gyy6bS0U10SUW
         WjnZcZyr6bF8W9IuOAiMd0yuYJUr0lqIYD9MPX893Z9LSQMFQqE1jOWSaxWeFswHn9KX
         GmmU0BjDHvl9h88We5k7DzZZ2wCeDLjG3tjf1JrIlQso+3Vc8G5S6ZpeUm9kT2mBRP58
         xrADQVtaNVic2RvBzNRO8oMn9KGDDiYf8gELQ7Rk/RZ5CAcJeRVahH5bqOZdz3yvbTrm
         MPWNK7gJxNFbUxESHhJ8MuowxJvTcK94WVCfqMUmgMusF5HxR6kqLZx8Qg3opek+skKo
         AlsA==
X-Gm-Message-State: AOAM533HMk6dq00N8aK4wUdLZ8GxEeiD3FjkgYDkFNzuk6N+78SIFBSv
        i5NwYaMioHoq8wZsv75AoayzSgjtbX+Bag==
X-Google-Smtp-Source: ABdhPJxQVST1I6ABrDukfZAWfDWDJtHcCtYIHhwxiksn57cMXHQkRbzCSL6dkDHbM45XS3PnXYrN7A==
X-Received: by 2002:a17:902:8f82:b029:da:23e0:17d7 with SMTP id z2-20020a1709028f82b02900da23e017d7mr6225058plo.37.1606468873951;
        Fri, 27 Nov 2020 01:21:13 -0800 (PST)
Received: from ubuntu.netflix.com (203.20.25.136.in-addr.arpa. [136.25.20.203])
        by smtp.gmail.com with ESMTPSA id t9sm9938944pjq.46.2020.11.27.01.21.12
        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
        Fri, 27 Nov 2020 01:21:13 -0800 (PST)
From:   Sargun Dhillon <sargun@sargun.me>
To:     linux-unionfs@vger.kernel.org, miklos@szeredi.hu,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Amir Goldstein <amir73il@gmail.com>
Cc:     Sargun Dhillon <sargun@sargun.me>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Vivek Goyal <vgoyal@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel@vger.kernel.org, David Howells <dhowells@redhat.com>
Subject: [PATCH v2 4/4] overlay: Add rudimentary checking of writeback errseq on volatile remount
Date:   Fri, 27 Nov 2020 01:20:58 -0800
Message-Id: <20201127092058.15117-5-sargun@sargun.me>
X-Mailer: git-send-email 2.25.1
In-Reply-To: <20201127092058.15117-1-sargun@sargun.me>
References: <20201127092058.15117-1-sargun@sargun.me>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org

Volatile remounts validate the following at the moment:
 * Has the module been reloaded / the system rebooted
 * Has the workdir been remounted

This adds a new check for errors detected via the superblock's
errseq_t. At mount time, the errseq_t is snapshotted to disk,
and upon remount it's re-verified. This allows for kernel-level
detection of errors without forcing userspace to perform a
sync and allows for the hidden detection of writeback errors.

Signed-off-by: Sargun Dhillon <sargun@sargun.me>
Cc: linux-fsdevel@vger.kernel.org
Cc: linux-unionfs@vger.kernel.org
Cc: Miklos Szeredi <miklos@szeredi.hu>
Cc: Amir Goldstein <amir73il@gmail.com>
Cc: Vivek Goyal <vgoyal@redhat.com>
---
 fs/overlayfs/overlayfs.h | 1 +
 fs/overlayfs/readdir.c   | 6 ++++++
 fs/overlayfs/super.c     | 1 +
 3 files changed, 8 insertions(+)

diff --git a/fs/overlayfs/overlayfs.h b/fs/overlayfs/overlayfs.h
index de694ee99d7c..e8a711953b64 100644
--- a/fs/overlayfs/overlayfs.h
+++ b/fs/overlayfs/overlayfs.h
@@ -85,6 +85,7 @@ struct ovl_volatile_info {
 	 */
 	uuid_t		ovl_boot_id;	/* Must stay first member */
 	u64		s_instance_id;
+	errseq_t	errseq;	/* Implemented as a u32 */
 } __packed;
 
 /*
diff --git a/fs/overlayfs/readdir.c b/fs/overlayfs/readdir.c
index 7b66fbb20261..5795b28bb4cf 100644
--- a/fs/overlayfs/readdir.c
+++ b/fs/overlayfs/readdir.c
@@ -1117,6 +1117,12 @@ static int ovl_verify_volatile_info(struct ovl_fs *ofs,
 		return -EINVAL;
 	}
 
+	err = errseq_check(&volatiledir->d_sb->s_wb_err, info.errseq);
+	if (err) {
+		pr_debug("Workdir filesystem reports errors: %d\n", err);
+		return -EINVAL;
+	}
+
 	return 1;
 }
 
diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
index a8ee3ba4ebbd..2e473f8c75dd 100644
--- a/fs/overlayfs/super.c
+++ b/fs/overlayfs/super.c
@@ -1248,6 +1248,7 @@ static int ovl_set_volatile_info(struct ovl_fs *ofs, struct dentry *volatiledir)
 	int err;
 	struct ovl_volatile_info info = {
 		.s_instance_id = volatiledir->d_sb->s_instance_id,
+		.errseq = errseq_sample(&volatiledir->d_sb->s_wb_err),
 	};
 
 	uuid_copy(&info.ovl_boot_id, &ovl_boot_id);
-- 
2.25.1


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,
	INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,
	USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id B67E7C64E8A
	for <linux-unionfs@archiver.kernel.org>; Fri, 27 Nov 2020 09:21:17 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id 568D62065E
	for <linux-unionfs@archiver.kernel.org>; Fri, 27 Nov 2020 09:21:17 +0000 (UTC)
Authentication-Results: mail.kernel.org;
	dkim=pass (1024-bit key) header.d=sargun.me header.i=@sargun.me header.b="yK1PbGT9"
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1728056AbgK0JVO (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Fri, 27 Nov 2020 04:21:14 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42054 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1726014AbgK0JVM (ORCPT
        <rfc822;linux-unionfs@vger.kernel.org>);
        Fri, 27 Nov 2020 04:21:12 -0500
Received: from mail-pg1-x541.google.com (mail-pg1-x541.google.com [IPv6:2607:f8b0:4864:20::541])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B9C93C0613D4
        for <linux-unionfs@vger.kernel.org>; Fri, 27 Nov 2020 01:21:12 -0800 (PST)
Received: by mail-pg1-x541.google.com with SMTP id o4so1607657pgj.0
        for <linux-unionfs@vger.kernel.org>; Fri, 27 Nov 2020 01:21:12 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=sargun.me; s=google;
        h=from:to:cc:subject:date:message-id:in-reply-to:references
         :mime-version:content-transfer-encoding;
        bh=CtvcgZK9iYPDW2XGOFNWnwy2h9J2gIbXQeuOPZlJjW0=;
        b=yK1PbGT9kHNND4Ld+pGseZLGkvfFlYz7M8+NVdsqpuwYBbFPyWdacCsXIOzISCodi7
         Y1pAYcEMZPPvCSoVqVGsp8hxa1VCy9wN3DXnxMPm6NLS9n3IZao3O3JcfAEuRlFpEr3c
         JgokqPECKGmBi7gPeK4i2w7b4qGBvo389gDBo=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to
         :references:mime-version:content-transfer-encoding;
        bh=CtvcgZK9iYPDW2XGOFNWnwy2h9J2gIbXQeuOPZlJjW0=;
        b=O7Bb38gZaKfmnVOWg9T+67scoqz8iCMCzvjBqL6utcnnwtND2qxppVqnnQ15K/+/U9
         kLVq0+CobECgoJxmI9LM5IP196AdgU2ye+OSnrFytLpj9ETmYw4lVoc7/NXxfDSKnv9s
         g9ib4kDChQURy1cVclotefNyOGOL4K6A3ZrcF2yJf+Rq5RBdgYW8pbz1E4iFORMs80wZ
         BrBtlOPsACN6+2nqDry+fhvhUZfZ0r34XLWLfsk5VEcEb8sTab3UGXGvYx/3+K7Wc8fn
         iQDv98sSIKx4m4PdxISFQivXfugqJaS13Cg9nIWLlxvfrUb03/otDsZA0Ao/bbMZV+1R
         cPcA==
X-Gm-Message-State: AOAM531+KOpoyo+QwR/9S2WaHyeOBqTPWxvR0JhLTp8Bqr40VX+IgOlX
        NN0C3kAippYG58gD3SJPR5b+MOjyokSCjA==
X-Google-Smtp-Source: ABdhPJxPrOjTKbpDK5Tsejg5kFvDpC1doqjiJ3Nm3sMmUS0iPNPo6h4eO7X/g9KRl3j0ddnWF8G8jg==
X-Received: by 2002:a17:90a:8402:: with SMTP id j2mr8837033pjn.120.1606468871882;
        Fri, 27 Nov 2020 01:21:11 -0800 (PST)
Received: from ubuntu.netflix.com (203.20.25.136.in-addr.arpa. [136.25.20.203])
        by smtp.gmail.com with ESMTPSA id t9sm9938944pjq.46.2020.11.27.01.21.10
        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
        Fri, 27 Nov 2020 01:21:11 -0800 (PST)
From:   Sargun Dhillon <sargun@sargun.me>
To:     linux-unionfs@vger.kernel.org, miklos@szeredi.hu,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Amir Goldstein <amir73il@gmail.com>
Cc:     Sargun Dhillon <sargun@sargun.me>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Vivek Goyal <vgoyal@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel@vger.kernel.org, David Howells <dhowells@redhat.com>
Subject: [PATCH v2 3/4] overlay: Add the ability to remount volatile directories when safe
Date:   Fri, 27 Nov 2020 01:20:57 -0800
Message-Id: <20201127092058.15117-4-sargun@sargun.me>
X-Mailer: git-send-email 2.25.1
In-Reply-To: <20201127092058.15117-1-sargun@sargun.me>
References: <20201127092058.15117-1-sargun@sargun.me>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org

Overlayfs added the ability to setup mounts where all syncs could be
short-circuted in (2a99ddacee43: ovl: provide a mount option "volatile").

A user might want to remount this fs, but we do not let the user because
of the "incompat" detection feature. In the case of volatile, it is safe
to do something like[1]:

$ sync -f /root/upperdir
$ rm -rf /root/workdir/incompat/volatile

There are two ways to go about this. You can call sync on the underlying
filesystem, check the error code, and delete the dirty file if everything
is clean. If you're running lots of containers on the same filesystem, or
you want to avoid all unnecessary I/O, this may be suboptimal.

Alternatively, you can blindly delete the dirty file, and "hope for the
best".

This patch introduces transparent functionality to check if it is
(relatively) safe to reuse the upperdir. It ensures that the filesystem
hasn't been remounted, the system hasn't been rebooted, nor has the
overlayfs code changed. Since the structure is explicitly not meant to be
used between different versions of the code, its stability does not matter
so much.

[1]: https://lore.kernel.org/linux-unionfs/CAOQ4uxhKr+j5jFyEC2gJX8E8M19mQ3CqdTYaPZOvDQ9c0qLEzw@mail.gmail.com/T/#m6abe713e4318202ad57f301bf28a414e1d824f9c

Signed-off-by: Sargun Dhillon <sargun@sargun.me>
Cc: linux-fsdevel@vger.kernel.org
Cc: linux-unionfs@vger.kernel.org
Cc: Miklos Szeredi <miklos@szeredi.hu>
Cc: Amir Goldstein <amir73il@gmail.com>
Cc: Vivek Goyal <vgoyal@redhat.com>
---
 Documentation/filesystems/overlayfs.rst | 18 +++--
 fs/overlayfs/overlayfs.h                | 37 +++++++++-
 fs/overlayfs/readdir.c                  | 98 ++++++++++++++++++++++---
 fs/overlayfs/super.c                    | 73 +++++++++++++-----
 fs/overlayfs/util.c                     |  2 +
 5 files changed, 190 insertions(+), 38 deletions(-)

diff --git a/Documentation/filesystems/overlayfs.rst b/Documentation/filesystems/overlayfs.rst
index c6e30c1bc2f2..b485fdb65b85 100644
--- a/Documentation/filesystems/overlayfs.rst
+++ b/Documentation/filesystems/overlayfs.rst
@@ -579,13 +579,17 @@ example, upon detecting a fault, ext4 can be configured to panic.
 The advantage of mounting with the "volatile" option is that all forms of
 sync calls to the upper filesystem are omitted.
 
-When overlay is mounted with "volatile" option, the directory
-"$workdir/work/incompat/volatile" is created.  During next mount, overlay
-checks for this directory and refuses to mount if present. This is a strong
-indicator that user should throw away upper and work directories and create
-fresh one. In very limited cases where the user knows that the system has
-not crashed and contents of upperdir are intact, The "volatile" directory
-can be removed.
+When overlay is mounted with the "volatile" option, the directory
+"$workdir/work/incompat/volatile" is created.  This acts as a indicator
+that the user should throw away upper and work directories and create fresh
+ones.  In some cases, the overlayfs can detect if the upperdir can be
+reused safely in a subsequent volatile mounts, and mounting will proceed as
+normal.  If the filesystem is unable to determine if this is safe (due to a
+reboot, upgraded kernel code, or loss of checkpoint, etc...), the user may
+bypass these safety checks and remove the "volatile" directory if they know
+the system did not encounter a fault and the contents of the upperdir are
+intact. Then, the user can remount the filesystem as normal.
+
 
 Testsuite
 ---------
diff --git a/fs/overlayfs/overlayfs.h b/fs/overlayfs/overlayfs.h
index f8880aa2ba0e..de694ee99d7c 100644
--- a/fs/overlayfs/overlayfs.h
+++ b/fs/overlayfs/overlayfs.h
@@ -32,8 +32,13 @@ enum ovl_xattr {
 	OVL_XATTR_NLINK,
 	OVL_XATTR_UPPER,
 	OVL_XATTR_METACOPY,
+	OVL_XATTR_VOLATILE,
 };
 
+#define OVL_INCOMPATDIR_NAME "incompat"
+#define OVL_VOLATILEDIR_NAME "volatile"
+#define OVL_VOLATILE_DIRTY_NAME "dirty"
+
 enum ovl_inode_flag {
 	/* Pure upper dir that may contain non pure upper entries */
 	OVL_IMPURE,
@@ -57,6 +62,31 @@ enum {
 	OVL_XINO_ON,
 };
 
+/*
+ * This is copied into the volatile xattr, and the user does not interact with
+ * it. There is no stability requirement, as a reboot explicitly invalidates
+ * a volatile workdir. It is explicitly meant not to be a stable api.
+ *
+ * Although this structure isn't meant to be stable it is exposed to potentially
+ * unprivileged users. We don't do any kind of cryptographic operations with
+ * the structure, so it could be tampered with, or inspected. Don't put
+ * kernel memory pointers in it, or anything else that could cause problems,
+ * or information disclosure.
+ */
+struct ovl_volatile_info {
+	/*
+	 * This uniquely identifies a boot, and is reset if overlayfs itself
+	 * is reloaded. Therefore we check our current / known boot_id
+	 * against this before looking at any other fields to validate:
+	 * 1. Is this datastructure laid out in the way we expect? (Overlayfs
+	 *    module, reboot, etc...)
+	 * 2. Could something have changed (like the s_instance_id counter
+	 *    resetting)
+	 */
+	uuid_t		ovl_boot_id;	/* Must stay first member */
+	u64		s_instance_id;
+} __packed;
+
 /*
  * The tuple (fh,uuid) is a universal unique identifier for a copy up origin,
  * where:
@@ -422,8 +452,8 @@ void ovl_cleanup_whiteouts(struct dentry *upper, struct list_head *list);
 void ovl_cache_free(struct list_head *list);
 void ovl_dir_cache_free(struct inode *inode);
 int ovl_check_d_type_supported(struct path *realpath);
-int ovl_workdir_cleanup(struct inode *dir, struct vfsmount *mnt,
-			struct dentry *dentry, int level);
+int ovl_workdir_cleanup(struct ovl_fs *ofs, struct inode *dir,
+			struct vfsmount *mnt, struct dentry *dentry, int level);
 int ovl_indexdir_cleanup(struct ovl_fs *ofs);
 
 /* inode.c */
@@ -520,3 +550,6 @@ int ovl_set_origin(struct dentry *dentry, struct dentry *lower,
 
 /* export.c */
 extern const struct export_operations ovl_export_operations;
+
+/* super.c */
+extern uuid_t ovl_boot_id;
diff --git a/fs/overlayfs/readdir.c b/fs/overlayfs/readdir.c
index 01620ebae1bd..7b66fbb20261 100644
--- a/fs/overlayfs/readdir.c
+++ b/fs/overlayfs/readdir.c
@@ -1080,10 +1080,78 @@ int ovl_check_d_type_supported(struct path *realpath)
 
 	return rdd.d_type_supported;
 }
+static int ovl_verify_volatile_info(struct ovl_fs *ofs,
+				    struct dentry *volatiledir)
+{
+	int err;
+	struct ovl_volatile_info info;
+
+	if (!volatiledir->d_inode)
+		return 0;
+
+	if (!ofs->config.ovl_volatile) {
+		pr_debug("Mount is not volatile; upperdir is marked volatile\n");
+		return -EINVAL;
+	}
+
+	err = ovl_do_getxattr(ofs, volatiledir, OVL_XATTR_VOLATILE, &info,
+			      sizeof(info));
+	if (err < 0) {
+		pr_debug("Unable to read volatile xattr: %d\n", err);
+		return -EINVAL;
+	}
+
+	if (err != sizeof(info)) {
+		pr_debug("%s xattr on-disk size is %d expected to read %zd\n",
+			 ovl_xattr(ofs, OVL_XATTR_VOLATILE), err, sizeof(info));
+		return -EINVAL;
+	}
+
+	if (!uuid_equal(&ovl_boot_id, &info.ovl_boot_id)) {
+		pr_debug("boot id has changed (reboot or module reloaded)\n");
+		return -EINVAL;
+	}
+
+	if (volatiledir->d_sb->s_instance_id != info.s_instance_id) {
+		pr_debug("workdir has been unmounted and remounted\n");
+		return -EINVAL;
+	}
+
+	return 1;
+}
 
-#define OVL_INCOMPATDIR_NAME "incompat"
+/*
+ * ovl_check_incompat checks this specific incompat entry for incompatibility.
+ * If it is found to be incompatible -EINVAL will be returned.
+ *
+ * If the directory should be preserved, then this function returns 1.
+ */
+static int ovl_check_incompat(struct ovl_fs *ofs, struct ovl_cache_entry *p,
+			      struct path *path)
+{
+	int err = -EINVAL;
+	struct dentry *d;
+
+	if (!strcmp(p->name, OVL_VOLATILEDIR_NAME)) {
+		d = lookup_one_len(p->name, path->dentry, p->len);
+		if (IS_ERR(d))
+			return PTR_ERR(d);
+
+		err = ovl_verify_volatile_info(ofs, d);
+		dput(d);
+	}
+
+	if (err == -EINVAL)
+		pr_err("incompat feature '%s' cannot be mounted\n", p->name);
+	else
+		pr_debug("incompat '%s' handled: %d\n", p->name, err);
+
+	dput(d);
+	return err;
+}
 
-static int ovl_workdir_cleanup_recurse(struct path *path, int level)
+static int ovl_workdir_cleanup_recurse(struct ovl_fs *ofs, struct path *path,
+				       int level)
 {
 	int err;
 	struct inode *dir = path->dentry->d_inode;
@@ -1125,16 +1193,19 @@ static int ovl_workdir_cleanup_recurse(struct path *path, int level)
 			if (p->len == 2 && p->name[1] == '.')
 				continue;
 		} else if (incompat) {
-			pr_err("overlay with incompat feature '%s' cannot be mounted\n",
-				p->name);
-			err = -EINVAL;
-			break;
+			err = ovl_check_incompat(ofs, p, path);
+			if (err < 0)
+				break;
+			/* Skip cleaning this */
+			if (err == 1)
+				continue;
 		}
 		dentry = lookup_one_len(p->name, path->dentry, p->len);
 		if (IS_ERR(dentry))
 			continue;
 		if (dentry->d_inode)
-			err = ovl_workdir_cleanup(dir, path->mnt, dentry, level);
+			err = ovl_workdir_cleanup(ofs, dir, path->mnt, dentry,
+						  level);
 		dput(dentry);
 		if (err)
 			break;
@@ -1142,11 +1213,13 @@ static int ovl_workdir_cleanup_recurse(struct path *path, int level)
 	inode_unlock(dir);
 out:
 	ovl_cache_free(&list);
+	if (incompat && err >= 0)
+		return 1;
 	return err;
 }
 
-int ovl_workdir_cleanup(struct inode *dir, struct vfsmount *mnt,
-			 struct dentry *dentry, int level)
+int ovl_workdir_cleanup(struct ovl_fs *ofs, struct inode *dir,
+			struct vfsmount *mnt, struct dentry *dentry, int level)
 {
 	int err;
 
@@ -1159,7 +1232,7 @@ int ovl_workdir_cleanup(struct inode *dir, struct vfsmount *mnt,
 		struct path path = { .mnt = mnt, .dentry = dentry };
 
 		inode_unlock(dir);
-		err = ovl_workdir_cleanup_recurse(&path, level + 1);
+		err = ovl_workdir_cleanup_recurse(ofs, &path, level + 1);
 		inode_lock_nested(dir, I_MUTEX_PARENT);
 		if (!err)
 			err = ovl_cleanup(dir, dentry);
@@ -1206,9 +1279,10 @@ int ovl_indexdir_cleanup(struct ovl_fs *ofs)
 		}
 		/* Cleanup leftover from index create/cleanup attempt */
 		if (index->d_name.name[0] == '#') {
-			err = ovl_workdir_cleanup(dir, path.mnt, index, 1);
-			if (err)
+			err = ovl_workdir_cleanup(ofs, dir, path.mnt, index, 1);
+			if (err < 0)
 				break;
+			err = 0;
 			goto next;
 		}
 		err = ovl_verify_index(ofs, index);
diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
index 290983bcfbb3..a8ee3ba4ebbd 100644
--- a/fs/overlayfs/super.c
+++ b/fs/overlayfs/super.c
@@ -15,6 +15,7 @@
 #include <linux/seq_file.h>
 #include <linux/posix_acl_xattr.h>
 #include <linux/exportfs.h>
+#include <linux/uuid.h>
 #include "overlayfs.h"
 
 MODULE_AUTHOR("Miklos Szeredi <miklos@szeredi.hu>");
@@ -23,6 +24,7 @@ MODULE_LICENSE("GPL");
 
 
 struct ovl_dir_cache;
+uuid_t ovl_boot_id;
 
 #define OVL_MAX_STACK 500
 
@@ -722,20 +724,24 @@ static struct dentry *ovl_workdir_create(struct ovl_fs *ofs,
 				goto out_unlock;
 
 			retried = true;
-			err = ovl_workdir_cleanup(dir, mnt, work, 0);
-			dput(work);
-			if (err == -EINVAL) {
-				work = ERR_PTR(err);
-				goto out_unlock;
+			err = ovl_workdir_cleanup(ofs, dir, mnt, work, 0);
+			/* check if we should reuse the workdir */
+			if (err != 1) {
+				dput(work);
+				if (err == -EINVAL) {
+					work = ERR_PTR(err);
+					goto out_unlock;
+				}
+				goto retry;
 			}
-			goto retry;
+		} else {
+			work = ovl_create_real(dir, work,
+					       OVL_CATTR(attr.ia_mode));
+			err = PTR_ERR(work);
+			if (IS_ERR(work))
+				goto out_err;
 		}
 
-		work = ovl_create_real(dir, work, OVL_CATTR(attr.ia_mode));
-		err = PTR_ERR(work);
-		if (IS_ERR(work))
-			goto out_err;
-
 		/*
 		 * Try to remove POSIX ACL xattrs from workdir.  We are good if:
 		 *
@@ -1237,26 +1243,58 @@ static struct dentry *ovl_lookup_or_create(struct dentry *parent,
 	return child;
 }
 
+static int ovl_set_volatile_info(struct ovl_fs *ofs, struct dentry *volatiledir)
+{
+	int err;
+	struct ovl_volatile_info info = {
+		.s_instance_id = volatiledir->d_sb->s_instance_id,
+	};
+
+	uuid_copy(&info.ovl_boot_id, &ovl_boot_id);
+	err = ovl_do_setxattr(ofs, volatiledir, OVL_XATTR_VOLATILE, &info,
+			      sizeof(info));
+
+	if (err == -EOPNOTSUPP)
+		return 0;
+
+	return err;
+}
+
 /*
  * Creates $workdir/work/incompat/volatile/dirty file if it is not already
  * present.
  */
 static int ovl_create_volatile_dirty(struct ovl_fs *ofs)
 {
+	int err;
 	unsigned int ctr;
-	struct dentry *d = dget(ofs->workbasedir);
+	struct dentry *volatiledir, *d = dget(ofs->workbasedir);
 	static const char *const volatile_path[] = {
-		OVL_WORKDIR_NAME, "incompat", "volatile", "dirty"
+		OVL_WORKDIR_NAME,
+		OVL_INCOMPATDIR_NAME,
+		OVL_VOLATILEDIR_NAME,
 	};
 	const char *const *name = volatile_path;
 
-	for (ctr = ARRAY_SIZE(volatile_path); ctr; ctr--, name++) {
-		d = ovl_lookup_or_create(d, *name, ctr > 1 ? S_IFDIR : S_IFREG);
+	/* Create the volatile subdirectory that we put the xattr on */
+	for (ctr = 0; ctr < ARRAY_SIZE(volatile_path); ctr++, name++) {
+		d = ovl_lookup_or_create(d, *name, S_IFDIR);
 		if (IS_ERR(d))
 			return PTR_ERR(d);
 	}
-	dput(d);
-	return 0;
+	volatiledir = dget(d);
+
+	/* Create the dirty file exists before we set the xattr */
+	d = ovl_lookup_or_create(d, OVL_VOLATILE_DIRTY_NAME, S_IFREG);
+	if (!IS_ERR(d)) {
+		dput(d);
+		err = ovl_set_volatile_info(ofs, volatiledir);
+	} else {
+		err = PTR_ERR(d);
+	}
+
+	dput(volatiledir);
+	return err;
 }
 
 static int ovl_make_workdir(struct super_block *sb, struct ovl_fs *ofs,
@@ -2044,6 +2082,7 @@ static int __init ovl_init(void)
 {
 	int err;
 
+	uuid_gen(&ovl_boot_id);
 	ovl_inode_cachep = kmem_cache_create("ovl_inode",
 					     sizeof(struct ovl_inode), 0,
 					     (SLAB_RECLAIM_ACCOUNT|
diff --git a/fs/overlayfs/util.c b/fs/overlayfs/util.c
index 23f475627d07..87c9f5a063ed 100644
--- a/fs/overlayfs/util.c
+++ b/fs/overlayfs/util.c
@@ -580,6 +580,7 @@ bool ovl_check_dir_xattr(struct super_block *sb, struct dentry *dentry,
 #define OVL_XATTR_NLINK_POSTFIX		"nlink"
 #define OVL_XATTR_UPPER_POSTFIX		"upper"
 #define OVL_XATTR_METACOPY_POSTFIX	"metacopy"
+#define OVL_XATTR_VOLATILE_POSTFIX	"volatile"
 
 #define OVL_XATTR_TAB_ENTRY(x) \
 	[x] = OVL_XATTR_PREFIX x ## _POSTFIX
@@ -592,6 +593,7 @@ const char *ovl_xattr_table[] = {
 	OVL_XATTR_TAB_ENTRY(OVL_XATTR_NLINK),
 	OVL_XATTR_TAB_ENTRY(OVL_XATTR_UPPER),
 	OVL_XATTR_TAB_ENTRY(OVL_XATTR_METACOPY),
+	OVL_XATTR_TAB_ENTRY(OVL_XATTR_VOLATILE),
 };
 
 int ovl_check_setxattr(struct dentry *dentry, struct dentry *upperdentry,
-- 
2.25.1


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-10.2 required=3.0 tests=BAYES_00,
	HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,
	SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=ham
	autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 61437C2D0E4
	for <linux-unionfs@archiver.kernel.org>; Fri, 27 Nov 2020 11:10:49 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id 097D4208B3
	for <linux-unionfs@archiver.kernel.org>; Fri, 27 Nov 2020 11:10:48 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1726178AbgK0LKd (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Fri, 27 Nov 2020 06:10:33 -0500
Received: from mga03.intel.com ([134.134.136.65]:28091 "EHLO mga03.intel.com"
        rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
        id S1725616AbgK0LKd (ORCPT <rfc822;linux-unionfs@vger.kernel.org>);
        Fri, 27 Nov 2020 06:10:33 -0500
IronPort-SDR: IGbs0bcbiDmSpJvGzAkPS4MlAvhlzUel8fwYBtuNLydGf3G4kpGhFIFf8w3uBFk7wJ1sSHxRsv
 QLnJk5lIAHQw==
X-IronPort-AV: E=McAfee;i="6000,8403,9817"; a="172477415"
X-IronPort-AV: E=Sophos;i="5.78,374,1599548400"; 
   d="gz'50?scan'50,208,50";a="172477415"
X-Amp-Result: UNKNOWN
X-Amp-Original-Verdict: FILE UNKNOWN
X-Amp-File-Uploaded: False
Received: from fmsmga002.fm.intel.com ([10.253.24.26])
  by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Nov 2020 03:10:31 -0800
IronPort-SDR: PNmrhZU2ZbzAWkwQkI3kHqCP+R7Ugxg8CAx6h8QGzJNvSMmYtuZzCAI45DimVCWb/y4IqzKfFL
 J+5IWxjPwlEg==
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.78,374,1599548400"; 
   d="gz'50?scan'50,208,50";a="366087134"
Received: from lkp-server01.sh.intel.com (HELO b5888d13d5a5) ([10.239.97.150])
  by fmsmga002.fm.intel.com with ESMTP; 27 Nov 2020 03:10:28 -0800
Received: from kbuild by b5888d13d5a5 with local (Exim 4.92)
        (envelope-from <lkp@intel.com>)
        id 1kibdz-00001K-Rp; Fri, 27 Nov 2020 11:10:27 +0000
Date:   Fri, 27 Nov 2020 19:09:40 +0800
From:   kernel test robot <lkp@intel.com>
To:     Sargun Dhillon <sargun@sargun.me>, linux-unionfs@vger.kernel.org,
        miklos@szeredi.hu, Alexander Viro <viro@zeniv.linux.org.uk>,
        Amir Goldstein <amir73il@gmail.com>
Cc:     kbuild-all@lists.01.org, clang-built-linux@googlegroups.com,
        Sargun Dhillon <sargun@sargun.me>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Vivek Goyal <vgoyal@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel@vger.kernel.org, David Howells <dhowells@redhat.com>
Subject: Re: [PATCH v2 3/4] overlay: Add the ability to remount volatile
 directories when safe
Message-ID: <202011271923.uGQVdM7B-lkp@intel.com>
References: <20201127092058.15117-4-sargun@sargun.me>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="gKMricLos+KVdGMg"
Content-Disposition: inline
In-Reply-To: <20201127092058.15117-4-sargun@sargun.me>
User-Agent: Mutt/1.10.1 (2018-07-13)
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org


--gKMricLos+KVdGMg
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

Hi Sargun,

Thank you for the patch! Perhaps something to improve:

[auto build test WARNING on be4df0cea08a8b59eb38d73de988b7ba8022df41]

url:    https://github.com/0day-ci/linux/commits/Sargun-Dhillon/Make-overlayfs-volatile-mounts-reusable/20201127-172416
base:    be4df0cea08a8b59eb38d73de988b7ba8022df41
config: powerpc-randconfig-r034-20201127 (attached as .config)
compiler: clang version 12.0.0 (https://github.com/llvm/llvm-project f095ac11a9550530a4a54298debb8b04b36422be)
reproduce (this is a W=1 build):
        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # install powerpc cross compiling tool for clang build
        # apt-get install binutils-powerpc-linux-gnu
        # https://github.com/0day-ci/linux/commit/a9871ab1de6c660e6a4c49fcffdb666851003b35
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Sargun-Dhillon/Make-overlayfs-volatile-mounts-reusable/20201127-172416
        git checkout a9871ab1de6c660e6a4c49fcffdb666851003b35
        # save the attached .config to linux build tree
        COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=powerpc 

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>

All warnings (new ones prefixed by >>):

>> fs/overlayfs/readdir.c:1135:6: warning: variable 'd' is used uninitialized whenever 'if' condition is false [-Wsometimes-uninitialized]
           if (!strcmp(p->name, OVL_VOLATILEDIR_NAME)) {
               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   fs/overlayfs/readdir.c:1149:7: note: uninitialized use occurs here
           dput(d);
                ^
   fs/overlayfs/readdir.c:1135:2: note: remove the 'if' if its condition is always true
           if (!strcmp(p->name, OVL_VOLATILEDIR_NAME)) {
           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   fs/overlayfs/readdir.c:1133:18: note: initialize the variable 'd' to silence this warning
           struct dentry *d;
                           ^
                            = NULL
   1 warning generated.

vim +1135 fs/overlayfs/readdir.c

  1122	
  1123	/*
  1124	 * ovl_check_incompat checks this specific incompat entry for incompatibility.
  1125	 * If it is found to be incompatible -EINVAL will be returned.
  1126	 *
  1127	 * If the directory should be preserved, then this function returns 1.
  1128	 */
  1129	static int ovl_check_incompat(struct ovl_fs *ofs, struct ovl_cache_entry *p,
  1130				      struct path *path)
  1131	{
  1132		int err = -EINVAL;
  1133		struct dentry *d;
  1134	
> 1135		if (!strcmp(p->name, OVL_VOLATILEDIR_NAME)) {
  1136			d = lookup_one_len(p->name, path->dentry, p->len);
  1137			if (IS_ERR(d))
  1138				return PTR_ERR(d);
  1139	
  1140			err = ovl_verify_volatile_info(ofs, d);
  1141			dput(d);
  1142		}
  1143	
  1144		if (err == -EINVAL)
  1145			pr_err("incompat feature '%s' cannot be mounted\n", p->name);
  1146		else
  1147			pr_debug("incompat '%s' handled: %d\n", p->name, err);
  1148	
  1149		dput(d);
  1150		return err;
  1151	}
  1152	

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org

--gKMricLos+KVdGMg
Content-Type: application/gzip
Content-Disposition: attachment; filename=".config.gz"
Content-Transfer-Encoding: base64

H4sICCfawF8AAy5jb25maWcAjFxbc9y2kn4/v2LKqdo6++B4LpIi75YeQBCcQYYkKACckfSC
Go/Gjja6nZGU2P9+u8EbQIJyUpXE0924N7q/bjT9y79+mZC316eH3evdfnd//2Py7fB4OO5e
D7eTr3f3h/+dxGKSCz1hMde/gnB69/j2/dPz09+H4/N+cvrr51+nH4/7xWR9OD4e7if06fHr
3bc36ODu6fFfv/yLijzhS0Op2TCpuMiNZlf64sP+fvf4bfLX4fgCcpPZ/Nfpr9PJv7/dvf7P
p0/w34e74/Hp+On+/q8H83x8+r/D/nXydfr5dLefzXafT0+np4vp7mR3ejL/fH57+PLl/Mv0
5Mvi7GQ+/3L47w/NqMtu2ItpQ0zjIQ3kuDI0Jfny4ocjCMQ0jTuSlWibz+ZT+MfpY0WUISoz
S6GF08hnGFHqotRBPs9TnjOHJXKlZUm1kKqjcnlptkKuO0pU8jTWPGNGkyhlRgnpDKBXkhFY
TJ4I+A+IKGwKh/PLZGkP+37ycnh9e+6OK5JizXIDp6Wywhk459qwfGOIhO3hGdcXizn00s42
KziMrpnSk7uXyePTK3bc7qegJG327sOHrp3LMKTUItDYrtAokmpsWhNXZMPMmsmcpWZ5w52Z
upz0JiNjHOeY/H7ayTmduNMa8kOzjllCylTbjXPm3ZBXQumcZOziw78fnx5Rcdtu1ZYUwfHU
tdrwggYG2xJNV+ayZKWrQVIoZTKWCXltiNaErjpmqVjKo97eEAm9kBKuOwwFB5M2ugJqN3l5
+/Ly4+X18NDpypLlTHJqtVKtxLbrrs8xKduwNMynK/f4kBKLjPDcpyme+YRESMriWsO5e3tV
QaRiKOSepjtkzKJymSh/lw+Pt5Onr7219idsb9qm254em4JCr2GpuVYBZiaUKYuYaNZsrL57
ADsY2lvN6RpuIYPdc+5zLszqBm9bJnJ3cUAsYAwR85B6VK14nLJeT14XfLkykim7RBnem8F0
u+aFZCwrNPSbs6D2NgIbkZa5JvI6MNFapptl04gKaDMgc7sJdiNpUX7Su5c/J68wxckOpvvy
unt9mez2+6e3x9e7x2/d1m64hB6L0hBq+62Up52o3XmfHZhqoBOTE803zh5HKoapCsrgHoKY
dkfp88xmEdoQxZ1VK97aj5grNPex7bI+nX+wB3avJC0nKqBxsKkGeMPd94jww7Ar0DbnPJQn
YTvqkcDxKNu0vgwB1oBUxixE15JQNpyT0nAju6vhcHIGdkKxJY1SrrTPS0gOThl92YAINosk
Fx4jEqLfgSXBsaTk+uK0gwR2XEEjVJHRBRjrnLPIPUT/cFqDt67+4JjAdXtIgrrkFfQJ9/fi
oTpstf/jcPt2fzhOvh52r2/Hw4sl16MFuK37WEpRFsrVWfAlNHQXonRdizvex/42iq5qJa3p
CeHSOLygrYCr9TORuv+Cxyowo5orY9f/18QE1POGyQE9ZhtO2YAMB9i/uzUnKpLxoa2Lccwt
+HrwS3Dd3Y5KrUwemj+4ZwkcVxZWGpaFLaLrQvBco/kGvMjcZnYHLbSy8xoDFomCGcN9p+Cc
4sAg0ir4D/fEYb8supEOSra/SQa9KVGCf/bAnowtxAqfeGwi4M3HmH0Q1nGuPC9mRUN4zDJO
nLsbmxulnanDRUY/498yAOgC/EzGbxgCDnSy8L+M5NTb5L6Ygj8E5mBBFmDZGK89FWDeAAsQ
wxBPo+/wffq7goHeEZRoB5NUv8FWU2ZdZWV3nAUXSfejb9EzcDIctdDTpSXTGZgwU+Ofd5Tp
PYlkRXLAImGUIBS/CkKQ1tyBoq+DLLhwQTpLE9hEGTqRiABQTEoL5br5lRCoBoRZIXxBxZc5
SZOwgbJrSEJ3yeLDxFE9tQLL6nZMeEiFuTCl7IEVEm84LKHe7pB5gK4jIiV3Ld4aZa8zNaQY
D9a2VLtPaAN6AKdITIeFO7MGemNhSnD5NsrAsLebmcEeIkLXzoxCYuo6p/YovSuv2GX43LOI
xXHQnNmbiJfZtGC90zA6m564bazDrJMcxeH49en4sHvcHybsr8MjwCsCrpQiwAJoXGHRup+u
+yCY/oc9toAzqzqrsDBzswIqLaPW4XgxOdEQ0Idvi0pJFLIi0Jen46kIi5EITkYuWQNJ/UbA
RT+LgMtIuO4iG51EJ7giMgZ4GDowtSqTJIUzIzAiKIsARyWkZ+w0yyozuQGAmXDaGNQOJyU8
9QJFaxCt51QuBPMzI237gi7mnp4U9GyoJ8XxaX94eXk6QpT0/Px0fPVUoqDoZ9YLZc6+fw/b
N0dkOguFBCBwfvr9e28q5nykw5PpCP0kTGfz6TQwbBvEWjzbSZ8C6EViuK+zALdtOZ06pwMr
QEpGfVriwudEpXip7c3yUD5uGPPleiTbV01euHTNzNlJxN3gcnWtBjRonmUI6cEircboxipI
/yAZSjh0ELc+0rswWTFy1LEQMmK1fa0VdKhjrZGIlXAngQFqhJudx5w4G7aYe8vzJmgNb5YR
gM85oDIOGDUjVxez394T4PnFbBYWaOzQzzry5Lz+colZG3VxOmujMYj26LoKpFRZFH7S05Kh
RZKSpRryMUcDYHfIaHR8tWV8ufKPv3ewtUPKhSqYw2NEptc1RnFakLxOHWGsOTtvo8QKnIuM
azCCECEYa4xcT42DlXG0NLMzuGnOIWGWzm5gR2tyAyXPwEr2PSmPmKywI4IsxaO0L1Lvh6pc
iTW11tKOiZVgUiOm/MkCIKnbS7Yc5XFCVRda+7z4Pd4GeO32Fcsq522Ti04bcBFJsSS+QUDY
kDNMZhRZkzUq7nev6HxDtlqBPoYShU6XStC+nYkU/TxbzCC8cJKrdP55AbCCD+Tm5jJ29igr
6PnpyRTW2yeehYjfvxsa+wF6Tc6CQbHlLs5MqKNwP7CKsJOaTWfTPrfjzefeGBVFxmuftKg3
qUPMW2kZYb8Emy0hOA0tbK04blHX/RVTzRociKCvzLJYOHbiMsPt9qZgSSG3W3HOB8Knp2H0
j7yzaXglER7zedDgk3qQTpN5ViwcVc42GZsPXOclA69Se1QHAFYaX5u1sUX5UGINOHNZ9p5x
OodekAIOiUiC+c8RlwXODOLHK7BsvOCuqrlpP/xVxUg9WqaWPfvHz+enn11nBYFQvgF7edEl
fyfJ8fCft8Pj/sfkZb+79/K96AQBY176bhEpZik2+DQCkQXTI+x+RrFlggsKtWmsMLYdSxcE
ZcUWQD3EJeF0V6gJxlk2qfTPm4g8ZjCfkZxaqAXwYJiNjfxC2Sd3r/z1BiWaVQb3Y3RRIcFm
KRcPI+fWzbtJhqKifO0ryuT2ePdXFbt1/VTb4J9vTTMFhMEx23S8SyH5ZSPhArWwYjZT4bf3
h3pwILUTRLKvvP7LTUOxq0wJRLhyhJmxvOzvc8vUTAyDFzydgrbTmMTt1nTgc1TGXXW1CIfi
Lrab0SXMpQgmO8BLezg2c41ibqSmHiLuO3I3Zn96xooE53zxqauXcVndmNk0bK6BBQY3lES4
MQvf4la9hGUvFt3rQEb0CiKZMu3FqD59kJjqsVdbU+bgHlIGBx1O21rAxnKLkuqX4JXQReqm
psMyEv7k39I1u2IhJyKZjcZJBSi7tIyN+jEjismjsL2RGDTFZTD4sdMCBK9hTvX0vHC/w6Gr
csl0GiVunUSasiVJGzxtNiQtmQMcwaucrC3C7UFUm19QK54ATm/xZF1FUZNPWryNyYa+rH2W
xjjH3IicCYm3s4tkaBbbMo+uIIBdQbhiNJGwBuWly4sssCuKUYyTXN3vabnV/OjtZaj2NWT1
Up1ASiMaPB3kSf9utmO6/bdRGlasgGbSujCi8c7k9i9Mbt22lSZu8hJdRWyT58J/3LBy8eHr
7u3eEvD58mUCd3yya/rbuwVHzXQmu+Nh8vZyuO3WnYot6hAm5S+m3xfT6p8el5LM5GV2seiC
NtAEkSSIDKbf971Wde0HLFiG2JhD4JR0AtMxAZLyZY4CJ76Atnn2aspt7+3+9/bU1eCUR0au
GHHe4hCrlTDQzeCNAdFelXYZ7H2TBdsd93/cvR72+Dr48fbwDDM4PL4OtUtUCTbWuWObZG3J
XVK5Hzb+DgYAHFnkFoi0moQXnaUJFhZ1XNszSxIOMVUOAW9udxHfuSg+qPdsGyAKW0SkeQ5x
19bdmbVkuj8d2zkH7IyJCWD2xx3Mv6KO9VSLY0VV0nuBsfykzKk150xKIQ3Pf2fU9wpdbY5t
v4KIc5izQNBtfXtlrAJ5dLBMmifXzcucL2ATQqjupr9cLE7LRFwXafVXh0G+AUtfJWTq/TfE
xf2VnHLxd5d+95MEHd0+gFZ9on8IbUZIbTAXtwQvCY2r1ADmloNsfH7/iUjlC/AK9jcTppBn
3CiSMDDpxRVd9f3plpE1ekaGDyeEXpZc9rvZElBdbn0Olis15XGBldZG38A98pJDViJczeLp
srys6pZGJeB865EKRjF37uyHiMsU1BcvET6k4SNQoH92heqTV6VaeGwBBbTNG1QQWqeXU3wv
Iel41K61DQnB3jgtaQo+2ODD0pZINyUhsKKQL1UJC87jAZ30bmCdmqwuCW5lb3KV/wJrXrsF
ub0KrF9puKjal+mSAn3mey9nNkWthanKG9xqyrX7SDR0qEsqNh+/7MA9Tv6sYMPz8enrXR0v
d7VyIFZP5b1pWLHavhvi56jfHan/0vITH9MiTW0yfJt1bax9mVQZjj71dRYP0thwVA/UuU+o
YVsqiFesUjPLHBlBkORYxlDGrZ6IpG2Nq/u82s0zMKhqwOTYuI3Q2DO7I6JWZPYPZObzk3cW
Ucucno1MFpiL85N/MMzpbP7+MPaN5cPLHzsY7MOgF7yMErzMeyNh3nprMq5UVZdXF99gMg1D
hlDFTQ4WDi7/dRaJdKAfqipqS8H1ukVOEd449ydANqo4GMnL0gMsXYUUXG0/cdSU0kRqGSRW
ZbpdkUNbeaPZUnJ9HdyGRgpjkFBc2PABLAit015RwZCLKHl0oDqoqfyYHBltG+n+EPWGcAFB
Hsvp+EpaQSpG8pL1CCYLVwJUC8IHhGAu356bDUNI2p9kVZcPASiV10W/8qZKmuyOr3c2BNE/
ng9+CQCAf27BXRPqhBRfxUJ1op0SsIR75C7b0RvRXUd2aREBF034BRFcW+XnwHWQ46J6f8Iq
I//jA4e5vo78IqCGESWXwdDQH6/1GSqfdf2Xeb2tqgBMjuZ1ABja6J5owAvUyGwbcKkQcxoB
tjUlRYFqSuIYTYOxt72T70r+7Jaw74f92+vuy/3BfugysfUXr87mRDxPMo2AxTmNNKkrT3wh
RSUvdBf11GSwPdTJTELLGse2GzU2CzvF7PDwdPwxyXaPu2+Hh2DAVedGnFUCAXYktmkck5E+
ak6I0mbp2i+7j2vGCltk459CnQriSvSzVKpIAQYV2sINQJ7q4qRXdUJHStQsopYMD9YDoRlf
StKHXABUl6b3AGmBDx6y0f0n+ggQj3vmFhcDRopK72lrrUJplSbhbVFmxnM7xsXJ9PNZm93A
IuICS5UAaq+dXaeA8XNK4E44NK/sNCOB4pyGGDRIyCUQxqmL9q38phDCgQ43UenA1ptFAsi1
U7cbC4gEKmA7YkOzofU7BR7VE3Yd/3oXP27qbzAMXffK4t2yJ9wiHCXsopdYOgv2dJURvzRp
gHI1qwIS4iHL8avRjZGz0BLtLaBYtPa71Zw6x/TX3d7NdHuxKOXuFsDPkFZTStwi2IJmMGd3
7yuKhZeG8iEsL+jH/e54O/lyvLv9duhPQkI4HXPne6UxAn4V0NUZLKZuMrYSqNO88sroKzOA
QgNxiC+gyZKPfNLRirGx0stu3DJDqB58hWyEUCNyd9sahgVthgIkGeyc3D3f3aLL+fvudf+H
85Qz6EQrfvpbKKpqhy+UubrqLpHb8Ox8SEd5uDTzIUdeWc7CrqWp6w9PtEu13e1rHZyI1tJ3
FZVVVLdiaRFEWLA1Oiv8mqKGBhAWYsFQLllDME4wRHZ0V1YjJRwcLpig6uu75q4kd8eHvzHB
ev+0u7WvVo3r2VrldpMTLcnaDMALpVdujFnvZhAv7d21szmg0SV3ck1A4BqJ/kybVjbpgtjW
8ciN3bExQ5g3RrU1KZJv3IXXVLaRrHceSMdEZt3EVG8sYSuamUtAhesSv7sc+arSdkVsTWzd
oU01dvpYtW541deZw8yhzR2VWlSNg+xNmcIPEnFw/NyNniVbejij+m34nA5oys0JtrSMO9en
Im5nAzl88h8OIr231pFL1D6H3Fo7792qTFzpoKPIVtziBuf+ul04vixXwbhfO84Yftidbz/J
6eD78+744qNyjYm43yzsV64pRIYTZ43YbZQSyVDAYYMO2y8dmhECrJhLht8dX9fx7ceZP4LX
hU2+2wfAkQ91hi0wPy3y9DoYPgw3x+5ZCX+cZE8YVVT10fq4e3y5r55+0t2PwS5CLARXsLfC
aj29bbVEI0Vw9okOZcByIHc94y8I6z2slPcadu4oiUc6VSqJnZBBZaYaxT9dUYydbBsqgrHN
AOp38Y4k2Scpsk/J/e7lj8n+j7vnYc2DVbCE98f7ncWMWsswMio+xjdmx9dWiF6xcC/wtudI
4UWOSA7RO4/1yjhXP8Cdv8s98bk4Pp8FaPMALdcsxb+c4KHPIVms+pcZ6eA7yZBaap727hTJ
+hsjRSgCsTc/UuB7PdQwfnJVmLh7fnZKWOz7qJXa7bFCuHe8AmsFrnDfMFhW/j5gaIW2vDfb
mlynwMdUrxYSiZcbdDiY/4QILx1To0ZuySD64sGZ2UqRKjDrD2Lh9VjPEL42p9BEED/ZterD
ycP914/7p8fX3d3j4XYCXdX2P3x3sBgAAnC18s+/JZut5JpV3yZc9xfQSQkdqoWw2k5XxXyx
xvxr74iU0vPTkEWxzLRavLefAa2Ef4E6asGtkZzjJgze6O9e/vwoHj9S3MCxYMouUtDlwsmT
QJRHsWTAZE5NRUfVNqvQfKf608Owc8kB0PqDIqX6YsjbAjCSuVc65hDrM6oOzNfERqL5iDnY
HI4w3Gp+hZZyWRUu9ufIKETmWwj4AOvkS7/ngAC4B9rXIkm2VnRMEwpu6kW3+2o3zG5dWsDF
mvxX9f85RKTZ5KEKtIPqbsX8WV7av2Wk8wX1ED/veDBJ0eu5Jto3yBNM1fb+VhSUKSPuNwKC
2ab2OVytMDviZnMagYhF9d9mMp/2eZgu88Btw1imJQuN1k+gAXl1DcFLhSSbKEA70Fgk7p8x
Lap9jA5EzOpp74kaiFWSJshai+h3VzOAFF/nJAuG3sCsU6ZeHxW07n6DAJMbxBVuvrFiiNQv
EwNqlZEN/QUPOBmb+arSbVSsmGTu3+xQvfXi1xdNAgZxjf+ZRkN46BFAOESDMDMRg9bIUKX9
SzOCvDaJ02ORq/Pz3z6fDRmz+fnJsKdc2GkNH1cGdjTfZGyi2u8QGmwJ1MZ+ddktJNovdMEl
r0L4FAVWW69u2dISEoHdctSxotJB71VBWhChexOtUMjdy96Jr5qjjE/np1cmLoRzVA7RRolB
BoaKzoQg6M6uUSeD3gmW83kxVyfT8Asny2kqVCmZQRX+f86ebTluI9df0dOp3arNmvfLQx44
JGeGa3JIszkaKi9TiqxsXJEllyXvJn9/Gmhe+oKmTp1UxfYAYN8bDaABdJWTAQRZV7A0cbys
VpSuitVe6jhUYg6B8hTvTy68Mc6XrgPHhaHF93+i2R3dON4mwUalDmWyOjZ55IeSIFswN0qU
6ERgDby3/Njo/CmsndI5lMNohPjI8cqKfSnvpFuMX5IHJvdgcRtLuCw7EG5f9UUs4Nds8IK1
2BUYGkBw3swVKWlCNNkYJXFI9GQiSP18jCQ70wwdxyAiyuOKwzVJj13JyIBvQVSWruMEytWN
2lGRyufxz/vXm+r59e37j68Yxvv6+/13Lqy8gaYKdDdPXHi5+cw3y5dv8E/Zq/r/8bW5YuqK
+bB5NpcVEvGtR20DcC/KQLfp6tlYUT2/PT7dwOnxPzffH58wOxwRLXXbdmAxoW8EN4qQZiQ/
0r6mCndZdgA6HhXS0Qc/ZgPL0+P96yMvhUuiLw84nmgo+PDl8yP8/8/vr28o9P/++PTtw5fn
315uXp5veAFCJJF4GIfxFSrc2cwreI7kSg2leADqIImX4jcUpTR4gsmWMalwmU9LYE6vnAZQ
DnqLVm1uMToACeSJ0a6/xQTzboPywwHzrHz49ce/f/vypzwQc/2gTYFPzZJeJmfVLIob+x59
IZpWuhrps6rAgBs5ej2XkxzhN0raFoRMrZ8tGljtVN/N21/fHm/+xnfGH/+4ebv/9viPm7z4
ie/Pv5vNZ3JGv2MvYNLxtMAOBCyXdDtsFCoqmZLrC+F1ezioGckAyvLsJMy1Si+GeYO/agMH
Uq8YKrVWBhkDJ7gUrThh6mrH/6I5ANL0nfia3Gt6i7SP6/aCsZbUSYLzdtQn8njtiyzXesCh
x44rMyZxKcd/z8CsPmcy96XW3MLCBklmg82JmfYk+VLkB9i14M4IDrcqCh2upBYArEOTv2B2
XAP9/vIEng03//3y9jsfhOef2H5/88z5y38eb75Ayobf7h8elUBSKCQ75tWyeSjWC/i8vJVa
jyAMLdJ6tGeSkQljEPb7ZUnxxjzorXz48fr28vWmgBQ6UgulEnaN2HKiDOBpZEFIJi1T+LS4
KKLBDENXyoJM2rOQaDmSFvjtjBDc/N3mCLKq/enl+ekvnXRtboezaV7OIpiYGuVu7rf7p6df
7x/+uPlw8/T47/sHSi8uTDYuwxqRh4lz8FLNJ8URYNjPqGuupkCm6SjFAMQ1IY6kkwpQoHnr
FZtqA0ejXnYnD88OfRs2vcwKakULDcLI1jLkXOpCTZn6hiPBmRR1MuWTDje9+UXdth34907V
SQOQ5dVpaCG+CO+/ZHNBlkMUVdOeWbkblBvnS3Uq9llP28G44mOens/ffrxZD8DqBOle5bMa
AFy8JeOzBXK/By27BpX8q4qB9EMQd6iBRbrLj8J2q2CabOirccIs1yhPkBlw4QOvWmuvOCxE
NTOcT0V2ltybNSzL+7I8XcefXccLtmnufo6jRCX5V3tHVF3eCqA2jOWt5nYgzYhhiNS+/Vje
7dqsp2xlUmOVvQMA3nk6RZnAchWzymzeuECQ32UdnUxW4MuaywiacK6R3LJxHLOtQvSISr2R
d6es47ohs2gBy0wwyIQpD8EMu2anjMs4xLcrhV/IE7bCC1o7WQjydtfTnVtIDnuPTmm0UvSW
NHMKxbV5j+gMcYRNS117LEQYYJjlEutZUIzL1sBRyp5ADo1827cWJ0QQG+Lq+R45sBfISNJS
J8hCAllB6hpdbIyWgr9k2+9sqB34rVPVMnDkU/0zTKLhUhX8x1bbfjmWp+M5I+ovdilZ8yFr
ylz1MiRqPnNR79Bne0q9XxclCx3XJWsBhnR+b510bOyyQte8dao9q7Jop/M2jEGSk9fg74mT
8FnN2yYwWd/QnvOjYKN2DgYeqBqbTpKuSZzx2p44AzSLzYrYDUaysxPB0JT1Nef8AxpgrXrX
ZG7oGGzcH53r7jwM7ck8qcY4jkJnaRiBTX2u9QPjMtvNCZLUC8XXG81vctePE//aXXrRji3a
JksCMu5b4PE2cleWnby7JRTX39vCgrutOJPTMR/H4V+pDuzLA4R7t/3adw0/nNfuGIurY1Ho
uYlCoU/p2Hl8RXQlJZAJkrNFjumyugE/TWo0VcJ8HzqR70PyK6KYfJ+EMR05MlFcmmmo7VVw
knlYta9xyPsWck6DORhmxVpMkaVO6Nn2B2Aj31xmCtGlSXwXthi1v8ba39xg1SfmRSlpUkJ8
3mQQwKxP9ARWTepTiUWZIXeq+b92mbEgi/7Wi/gCEOvLEDoRHYXb6NiG7sFMxzr7CmVD11S5
q+/7vqkC7doWQaJ/qzMNwGwGT4FsqBxFiNo7Ui6gGYKMt9Vq9YrJeqrTu64B8XSIr9wRTDAq
zkqgwnDWe4/33z+jC2P1ob3RDURqK4lbM6bebOHPa5U4gSI+CDBXID7uaO8tQVBXO03wVdB9
dpFMFAiarhLGjoHIrGMnQ7PA6I1hHuioG63hvdXlcBXf7ciSkQ/QH561AQXhYrpjXH39Jtj1
xMKQyt60ENSBfAFOzeNqGifUR6Gt/H7//f6BK2mU4+IwUOxnkhkwbuysWAcxIR3J9bpO1bf4
bryK9Ly9BgULCTqqSDIFwjG5HXoNkxgIbJVtoYgSubaF7LzPZHkX0azSAazaayB886JoD3oz
IT1Nu98rzLdrdkaV1LXaZYpPXstcQCJXVdXCDbhU9IrfZYFPX0GuNCIy6x2iPB/6k6nXTk5O
D8S6mIqAKISG64+BOCDW27oFHlhuHPPe00+l2R3TVutcKR9WxSmA//6oAOZMXTOXyy6zx7Rk
qxkFHBw2vTCSTT/8fzIRCmcw9Z3iWTFDwKtC2n7mRlqbIia2P7MB038vLuzCkMC1Y9OiI3s2
8x9XtCKA54vi/unldoc5RGKq21u1qOY8znaa5sfT25dvT49/8mZDO9C5irh2g8+yfidYGy+U
q6tcHbNUystHQmVrLHD+58Z39ZAHvhNRn3Z5lmo58yw0f25U0FUnvu5rqgIuA1sLL0rp480m
NPWYd3VBrvLN4ZZbKqIR0Iyrzl1WH1oIR/u6rp2F74Pr+Dp369rCJx5ufgXH8skR8W9fX17f
nv66efz66+Pnz4+fbz5MVD+9PP8El3R/V1biNYflTk1pTu4ZMWCQtRxDO3TTrIZmdUZmfNPI
lhtBZTzKprz19MItqb4A9bFsOvm5LZy1TttrLbSY6aXytUWa7xWi/qNvW96sagbZ3wFgQnz5
eYkX5ezj+f4Jpu0Dlyj5jN1/vv+GPIUwMHqLz5LBxNu338USm4qRJl8vYs8qcq1al5b6ORvO
lOSLqFrLv7cAJ48H66RDAAu9agADe2Dz0zkUU+qJscl8ac7z4sQAsrqzz4fMRQavwqV8kQ7X
p/oFEwfpRSGsXFJJgjLR3L/C1K6XfaY/K97NohCr1M6VaXFvK4LmVBxnD7tMdu1EINeKuDxS
36ngnB9Ep1wrYN10itcRYC62qzaBxPiZr+o3xvqScKexu+7rcqQtWkAxMR0JUjexc63rzqyn
tljGZmwp/L0kaAtRjCdtTPo2/zi9FaZUwHI3qVjkkAoB4Ku9SAEpz/gom1IAMkKeT73tgg1Y
yv3l7vSp6a6HT0bzs6ZQ1pN0upg+EtCa8yjTd99f3l4eXp6mhahwBuxQV2kOhPI0LBdiJRv0
/gx1GXkjZdPCcie+oINExhutKIGZ0u5z+NC3pFsALEDhfiqX3Mj5oWRpn/9QRCqhCfOlqt7o
ruCnL+AtJUXMgl/GEc0cq/rTESG3Q8c/fnn4Qz+ay2cMye+Od1zrxVe0TuUAby5CHAIOBRuy
BgIobt5e0MeI83R+HnzGqCV+SGCpr/9UcmYalS361yT2GGF3E+K6vMK0ftDI128SPUg5czYx
9Qv4F12FgpjyfutNmpuClkLFGD9j0C5G7b+ZoMk7z2dOohqoDKyyk3SsiYHkLnVJNYiNbujQ
draFZGj22xTC7OxR22UmERZQc6javKzlxHEzvG7z4yk7yFa4tbYCjhPjk5wFce2GFoRvQyQ2
RCq5EgAXV56BnADo+w5OApNzvJR4v90L3m98UvWf1JccxWqaiFcrCmgq+P4BZUEBpPFCHEJh
Mnxn4ZNTEoCv99++cTEZBUtCjsIv42Ac8QQkJ1sEFeFRbmuQcRgL+/4l63YabE08r5a/H+Av
xyUf1pD6LLvXKejeHPPrsb4od6wIrNtDld/S98diHHdJxGJ64QuCLk/GcYOAZU0WFh5fVe2O
ethjnuFc9j5H4CJVK2MPXif4jIGe4YGa3EV7Qujjn984g6YmPSu6MExIK51AnzqtHYfLVdFA
pFXnUFBP78gENd4dRasj6L+kBrKiY70acS8yGoUNXZV7ietY1QNtbMSG2RfmmClD0le/tCdz
7e6K2Ak960juCt5yt7ncao0XFynaOl60PG3Rdn4aUF72EzaJ/VErSeeWywwAwybBoWPOCnJv
W719Hg5h4muF8WOjK4kpYbyCJLKVhfjU9Yz9OnxqRvtn4irJqA3AenzBvHXMaV7kys3p51zS
jQJz5Hw3dc3Bh13h6tDc95PE0aBdxVrWa8Cxz9wAr19Wp06zgeryPBz68qA++iSq5WqBnNMI
0wdgn92f/vtl0pEJSfriTrrgtWBekFDSi0ziXhTtYEVZjBorATtUsj2SaJXcWvZ0r3ig83Im
IfxYykaWBc6ERVoHQ6ecUF46Kora0AqF69tKjSwIz/JF4oSWL3zHhnCJniLC12ZBRl3znnwd
W6FKbGMSkqE2MkWcOHSz4sTS3qR0AluDk9KNyT2srgRJLoWrjWt2S3EsgYM4PcWzVALbFqpO
go+IZkq8nURRD7mXhp6tjmaIfM+npWqJbKrindYsMgRZhsAu9z3UhWqJSSeatpC1f/EZiYPI
toZGiZohKV19p4+MgOrGpq7IBF7ifZOUlxXwghQYfiQTx+zDon0zeROASq9wOQEmiENHh2Ke
GA0GKvIBVhOXk5xIYuVTu65ZPiRpECrywIzLL57jUkFYMwFsicgxC9X3kAJ3LXDl0JwxdXlo
ryX5ZvZMwnaKqXjuMQfTzpEZV8xMvFbo7pMXj3KWKA2hKrg68lh8ojozo4theh6e6b7GxidC
vCIaOneTEwgXLOpTl/RrWtYEegQRS0KDz55D6soCaJJc9+eyvh6y86E0C+JL3Y2dgFggE8az
YDxVGJobPLsXbfQJd5fjmxsApEsvpkq1PUyylIjrhShx8KPQNeHC77/FtrhBpOZxkNqJsio5
+SpR+m5/08RsRdN5kZeacL4GAzeUdBoFkTpUYwHlhfFGM4Ai9kPLxyGvcPvjMEkdchM3Oz/Y
qlgI5KljdnTyiYtNboOrVZxwAcERD21d7Cs5wceM6YfQ8Ym11Q+cg4Ym/Jwz13E8YhKErkX1
mOtZaRpSfkMizvur8vN6WxU6aLq+ETYaEXIuopaMQJYlArmIfVeRXyRM4NIeewoJJWWuBI3r
eNJAq4jQhohsiNSC8C11uHFMIlJPZk0rYohH14LwXYcepoEPAmlvVyjIBnJE5FlLfS94HGno
t+kWGua/VwrLuT5NX+0vNCOkyjhRdwBEeV1pSRO2kAxjt11hzv/IKkjE2lP8eSZDJ5OhbDpq
CAum2XUJClfruU4gPEUzJWfXhNvHoR+HjKr5wCgVZcbODspkqYc6dBPWkAjPYQ1ZHZfDyIvB
Fe8RBaLRUo4UmDHH6hi5PrnYq12TlZZnnleSjnxafSEAu6bKz2bUv/KAaCkXPnrX88gGYUzd
gQ5dW2hmg/xGo8SJQPAjgSCYyIRQhUEdqV4bysiU7s6Q8xN6a0kChefSDQ08jxg+RFi6FniR
tR1etL1HQcTh/22tdE4RORFRNWJcgpkjAjOxkvWl8XtN8rkESbuNyiQ+weMh70REnVWI8FNL
m6IoeKe+KArJQUZUSkk4amNTqrF555MH65BHIXmcN+Vp77m7Jhebb/O8ymXtZ1kTTeRT0Jho
HofStCG52pp4axQ4OqEKS8iKE7LihFr/TULt68ayNbnMsM1mmpS2ikgEoUe6hSsUgWupnqO2
D/vTkAsTYsWGlo7XWkjzgWvpWwv31OVNjOvA+BgvLVLy4fpGcddcPmg0Z2VZRPPeEWF2XMns
9pSXmnTkXPP9viNqrk6sO/fXqmMd2YCq90NvUwbgFIkTkVuq6jsWakmJdBJWRwk/76l15nGF
lhB08XiISRY4odaIoe2Dwk+og2LiyoGN+Trk+uM4z+F88z3+y4nCd88NztQSyrYkkwQBJaGD
zhsl5OB0Y8nPlC1BnKuFgRNQZyTHhH4UEwfSOS9SJRhHRngUYiy60qUq+aXmzSM+gLAmUhBj
x4GaQA72yDniCJ/yv5XwOXFklFwiDRyfKpGjPNfZZmucJgJj4VbFDcuDuHFTYlTYMLA4pPvT
NPzs3D6pXC8pEpc4H7KCxYlnQ8RkhRnvSrLNDU6Z8I8h4NShyeG+R5/TMclThmOTv5NXbGjg
BfWtdQ4E5HwiJnmvdFueNZlkc5Q4QegS5/Al8ePYP9CIxC1oRGpFeDaEpe6U2E0CDhtw8ogy
usspas6vLPm4VaroRKZgm2nEXa/ZBrTpr3A8xNV3iiaQeDQTgo0pC/ZMVDZlf4Dnlpark2tR
1tndtWHrC2ozsZygcoZBXlR8RWLoK9kzbcbLj4ezoeyul0rNpEAR7kGjZ8esp9U16hPxRKTl
pfD5A7Vss7F6Iwk0OO3iHzR6bYZituvOMxXZoaK8xefEt2jWGZtefN7op5ohf3aEoBaLeKyc
qHduGfryLV+Sl1crcr3ZnMKxqBXOdnytMVapTzgzyXsKSBi4niv4KVkPZHCiC5AIVPj05oHq
NbWDZ3XMcnbKaz1IhDUy+eEhBDOR4FkFzlVBVpm8Ua4rFTx9m7CTXwdaQ39++/H8gKntrSmn
98VVfY4KINKlnQxlfoyRqutd1wT1aK0FQnKFB5XFSIbfZ4OXxA62gugYkmC2AHAkz1s14fSC
PNZ5Ycl2wmn46ISpM1IWI0SbvkdYsrinImBq+BbAFz8ipV4BteRHkQiUgAOcEt1ZcwH6FDCh
gOqNxwremCvYs6R32YKVk4VCkZP10ujABBfjpDQCMZTANSNVY/UCpW6pJqSSIwJgh2wowdUa
zaTaROWur1y8SkDVlCYjFAMcIsT1l1L0sYq4yIJjJXmlDxDFw6pcEhcAxkvUXNigiOoTizzb
MtVjmwAmEnEYMy3AtM674COLZ7NYmKMbhKTJZEJrHnIrNCShSURBU9/cMhyekF58EzpJnVib
DOHyQBSVpKTta8UmWquGyI/09nNYGmuw2dAl11n+ggGKdJIX3H86VsJBAg61jvk2eu3rkv5C
MesvUDWWBgsVF4pKsas7otK2/mNC+nIh7hQOkasNFStzPSE+QKsgjkYj0zSimtAi8yP2413C
VxzNnLLdGDqbR8TsTimcFIfmy8P3l8enx4e37y/PXx5ebxCPGWQxWxkRiQUEOr8SQCMxxOxq
+H+vRmmq5tIMsAGCfXw/HK8Dy2F2Fazwa9XHEzwOSM/kqcAac6TIC0oLMoAbbNcJFfObuO8m
LRwCFWusc/ZY1fozX5ubUOXKfG7q7JqrdHBChJGdj82JabYJksjO6CaPWqrDEto4kmb4xtm+
kGhpyCcc59qWOP/hUgeOby53mSBygs39cKldL/YJ0a5u/NA3dv+Q+2GS2g4e4VyslbNEoKjy
ken9LYE3RmumoEUhL9BLvDShZqAw0NZJ5apwqnN0hCVmLUngbBQjvKoNmCkfTnCjd5NWTsAo
2QkbSb6tDVy6PTbg+eEmungzY3SfEPUr0jdf8FbMraOxfwh7WkFzgp6F8csR8TYtZPl4NjXL
jVszVhnPQhoU+2osi+ttWw+Z7Ce2EkDCjDNXbzmCnZtSiepbqUAFF+97z3SbtXIh6sBZC13W
JJiRC3SlAkUrIc2PKo2qjEm4IvRl7ywJI5Q1CjOrNUR77EEaCo26xGSUoUhJ0zhHclBTbAT+
0UQeuac1EpeqfZ+dQj8MQ7p+xCbJe9NlUb6l5GqoAlBDIzC3oXwrvGIrVqe+7N6uoCIvdjMK
x5l55JMzIQcIEh0BiSKmrKsaiUcvE3R2JN9AVUgsTYMznexqLY4havoAFcUR3RtQVkIy+kWh
ERqLrQTDXZIiSqKAbB6iZMVBRQmNha4XNJd3q9Vc5TVkTGlLGo2s6ej9TiJr4Tb/UI0ocSwb
W2C9d+Zm0rJ1rUGliJN3GsJpEvn2R0Z1Lp9dGteFgRvRmCQJU8vQ/C9l19LcKLKs/4rirGYW
c4eXEFqcBQIk0QZBU0iWe6PwuNVuxdiWry1HTN9ffyureGQWiXzOwhFWfklR1DMrKx8SG5Eq
MdPX2XxE9YW45PmSNXGhLI7LdqBEpnzXGsfWHmnOAcwXg/+gN2VXqOFpFGHL7beEXDsibCeX
VX8cCthJo6D5yGQtb7nIIj3+NSryNiUs87iCt2Jx2BmJRgaczFEZgfpQfL0AKVxxX24a+vaI
cPIypLfjFBSfDBUxzYOZP7LciGwlpWZWoEVMpriHIHlGt/xwpPC7IDDCf/E8sw1XNlyP23KY
j2CDkypFHdf/bJ7pM+mIV5HJxmf4NpgCdtkYnn0NzHZHFvP2fPz5q2U7j7cFHDqvFrEzbyF7
SJ9TPmmiod8oP8mycJEuUFSAytQYVRB7B4Wbz9KKnHmqqA1/O5JfPmrC8XHzOGoVVL8wZVPU
6TLFkXuAWqabAUHn9Tikmy/o5JPEaagYwFlLZ5Pr7+rgheuZy9oGAmjGMNClNSXhZ3TGiLf7
15+gVGLitcVMhglIN4BjPDYnMEzWydff7p+Pk78+fvyAiERm8L/los0P3X+1pKlmu8MkPP66
LO/yg7hQ+FCo/FumWQYpofsOb4CoKO/k4+EASCHW+CJL6SPiTvBlAcCWBQBf1lKOkHQlO3sj
+4LcuklwUdTrBuG/apGu+Cfla+osufqs+ooCG5RJIngLNtHhhFEkZL2F6kOk9EHnk0792Yb1
Glz3yWJ2q9AmcrSkFSXEM67YaQQ1tWPj0gQaTmSHON0SGhjJrfa1FCEsQh+6vEBF9JncqEqe
1FWxKXJOsQXwIvAdoyYNDWKMJEabtdBYKgTJUlYQzDOt5f9jLELI77d4J1t2Nulk8fcPfz+d
Hn9eIGlpFI8m95DYIcpCIZq1DDcIYJm3tCzHc2o2gaDiyIUTuKslPkAqer2T6/TXnVlimqVz
hz2/tajrWLSoOi4cLzcL2q1Wjuc6IaeSAnwYhgSoYS5cf75cYQfw5jOmln2zxO51QF/v5XYz
o7SizuU6O0WzHDxbMxW6mDTmryHeqECZJ0vqnN8Do26SPUt7CcUUqzbD2yyJOdC01+mRMIZD
BrnbM8AZtwmjz+lldqYEpTyw+BwZBtf86mvkgX463fPV5EQa7muUtuoTptF8JKgqu6ljzTLu
kq1nWsRSmJxxIyOson202fDt1ag4rzdFQtIif7IEtM/LJRlMrpBglBUrZHACv8ALBaKPyrUR
tzSC1MLOtg9iirJt7Tgeu5QNBA50EVZsN8OU3Wu52Q+WszXxGEzj3o+6rpLNql7j6ku8Cm/Z
Sm/XKe/hBWU2RkSDGonX48Pp/knVjInjBI+GXp2wKScUGFXbPa2+Ih2WyIBNUcuSRgxTxK2U
OzhbKNUMSXaTbmjZ0VrKmHcmLZW/7syyo2LLexcBmIdRmGVmQUq8NGh3ZaUzQpHCZSesik2V
CjaatWRIcnFYLmlZSZZEJMQ10L4Z+Q50Z+WLlE2RpNBlZRSyyooqLbaDWsqix9KFKPhu0CG3
YVYX/H08wLs0uRXFJuXupFRF7iplL0erl0IcL4NUG4QvIUnMAaT6Nt2sQ6Osm2QDweeMhBqA
ZJEyqxyte5aMNWiWbIpdMSivWKVXBr6UjdNI56oyHsxlG1aszaBG77QxG/msKtEjalBWGlWF
KJZ8VH7FUUBcdTYrhoIh65waBPSFmzqlBHmgSm7M15fhBiwO5fAaa7syqcPsbmMsAaWckXL5
HhSnyYbgyDB0+wNdQ1oYiuaBJBbGatlhUTo+NqTYBT60G96AV3FU8jRkfKUIU91mhJaLLY41
q4jgfCu3E5O3TsJ8QEoySCeQDCazLLbMWP2bGkC50Z8rSFEUCryedaTB4izysKq/FHfwgp4f
UweP1OmuMOarPJolyaDT67Wcr5z+UYMQA94MJ4ypgxdvYQM8lMIdrF1pmhf12IK8Tzf5YI5/
S6riSqN+u4vlZmYuZ9o+/LDeLlh6JKsO6lT1a7DhZeVI/ldmH+4DmnNig4qKnhLpacDbZcJA
xPZ5UOgWkOmUnLzx7AEORtfSa2Ny1jZBbn1m/rqWNryZRjElxeX08DcnhHRPbzciXCYQhmjL
HndzsGQ+qByX5O3iMMh7OXjv+vx+uRp5epPctutLKyXKX/rsxNEOhs0yQtSiLFclHFJNwYsK
Fr2NFDggLUUkd79V0kU0hg1soJ9Qj4VhbTtU+a/pG9dypmxSI43LY6URaUjTwXOIPTurSka5
7zqBWXWgToNBHaLKsmzPHolZoViSzJaHEJdXsysOdQAdfp8ic4fMHnWNasLpDLuzd8S5s2eo
lm1SzZslRdRRJs1iG6phDK8ganmoXwc2ax5DpDeZDVmeH+H6L+edhhsm8zDc14oN/9LBcEVu
Ptba48hjF7tedkzTYT+NKgMaNLIdT1jBdFhX9uJKQb0JjPm2RewEI4ZOul1qdzriFKx7ePTq
VMF1FMJ1h9FRdRZN5/Z+MFpaA1+OjB2iuhE7/ccYKr3lrPmhN3Xs+CPW6YohFa69zFybtVPD
HFpBaKwykx/nt8lfT6eXv3+zf5/IzWBSrRaTRoz+gEiR3K41+a3f9H831qkFyED5oJe1feeV
Dsn2Y5lIFA5WSOOoCvZ/x8oFuueU4Wczl7iFwexqIDozc6aiGzfy9lXu2spntmvd+u30+Dhc
xGu59q8MJTYGdB6k0a9omAq5eayL2qxcg8apuBktP685IZ+wrBMpES6SsB4tpJPcxzukZY1K
LogxYQkjKWamNC02YRixL6If3biuqd5VvXB6vUCo+ffJRXdFP6A3x8uP0xMkoXg4v/w4PU5+
gx673L89Hi+/Y5mE9kwVylNpsuHiKtJPDnPD6YXAZWgcrcfYQD90ZcJ0zbeN2eYJoygBZ6w0
g7b91WmC7v/+eIUPfz8/HSfvr8fjw098MTbC0ZaayIX8IFdk8HwRUbVF95gK6i8AG2pVRwcS
BB0I4PbsB3YwRAxJC0jrqC7k6sESW332v94uD9a/0BWoZIHsCVL25a9I62jcgFJimx3KWiIJ
bEZvYEw39VIHXKf1U3QpqEYMWaemInVp6YdtmiiPxJF6xdVOp55AWbSgegORsWVuUtLSWgAQ
LhbTb4lwOSQpviFH656+D3CAwpYeC7gGGqMfIjllttWd+cUtx4xN1tgz+DNnWMX1XR5McUyU
FoDALnOanw1BYHF25W2MwwyB5qxZS8+hbcaYV1diGrmzEUPOhicVme2MOIhTHvYi3WDxh222
l/TpkKwiijguV20F8S5mhMX1Xa7NFPb50wEzBHPPrgO+FxVyuI25Zbhl6o26TeCr69ywnzoW
q7zrwsboaTDiACDxlFpAyIPJ3AqHwFIKC8TetS1JTi+b/WiJTAPWwAk9ii/ZWnqSy7MhMzWr
naQHHD0ILHYsiCkfe6zDYznXyejtAqPT9YntUd6YEjN4w85U64szsu4wIx3oHtODij7jPhqQ
+YgBFV5wbNaQs23T+cyyuflR7b3PetW3bWv4IWo58ZjhrVc/pknkRHNsh1svo5KEZqi0D+Yh
3MSNV0bXjZCrZ7jdMG0mT87Xlzpdm2srsRqf88hhJ4PChqG0dN6hp/uLPNE8X98Uo7wQ7Dhw
iO1aT58SnwFEn/LjyQ8grEWeZmPbnh/w3luEZf4Zy8xhY/lgDi9g9zOAgs8fdvhnHc/iNT0d
y5guADP4/EJT39izOry61XpBHTA7HNBdZhUE+nTO8Ivcdzxmuiy+eqbpdjv4ymnExttpGWBw
Mmu7ac6A6VN2zb8Sdbll0SnS2vPO+eUPOGxdHfh98p/B+5o0PFdet6zlfxa3IA28xztAW1sP
95kmwUV3Xy6O8rTx9knlW3Op7v0xBApQho4czQwZj5AdiQkByf0GFn+huNtEh3p/SDYqHAvo
aTdJdhC3aR3ROrT5CAmt863Sz9Ea6sgvXR80CbtzsTLyDPaWjbcqEqyEOW28MjrLsW0f5P3T
lK4ISM2cHVJJ9fnJ2yQVbxLvxeVYXZTdzhoKOuSrnJPDeg78lTqNIhufVR6VNHPXH1GXea5v
pbZH+GSMktok5Rv0zKEK0xiVvtguJ+dX8L2j2f+g/GWa8e6l+rlDXuySxu7zGts6CUcugYz3
d+Ntu49TUWYhMiddx56nI/p3xd8Iy2Y90tMcWihK04N5J1vb/g27jjRpVEEjkaBQNOpnC/ZB
ixpyVUAL/Xvav0AD+h7jkCdCjEWFhWDb6uIXIh9xKSUwA7mkQoC6cGGLV/UYL7Vv1i1Wy28h
i06zNkJuKdRyAMWQs0JDTNHq4WorkDQB876JSkNNVeWUXO/auPZMWTp7OX5EU0A3zOnQdnGJ
nB/hF+TaRRQVZict6mxhEmnKeEVDH6BY4KXmYyISxHlEU+HNbH9oGEwbRHPvCHklwuhuILWp
gAHv5x+XyfrX6/Htj93k8eP4fiEm3q3n7CesbY1XVXLXJsRtp3cdrlI2SNhwZ2kphzItsREL
GLtF2Q2yis9uQG2TFQVJINIygh2tnEnY2l6poI1COhqTkwqoaxHfcCO7fw6pH1hw7uFoOAgT
6dTF8fgNaGqPPWV7Yw95JLwexVjbTMQSxVEys3z2pYBpW1Ku8Eg4FkTD4EwRcR20txCquySq
qF3sO7FDP/fWUcUB5rnN2bJ3Ed8lrSsxh2mf7iYeWb/G38oT2oa9/o5UYlRx/njjAl2p6wYa
h05R5DK/SMgoF+CVQsOggdkSJDyQk6T2vcW/Ub4x9q3osj5Ms0XBpgyXn7pFWmTt83F8Ob6d
HiYKnJT3j0el2p+I4frwGSuSh9SblCBIs2Vqne/x+Xw5vr6dHxiBNAETlEa52wvSHVUOVJpA
uKsdU6p+2+vz+yPzolKKhEh6hp/tPoEsQujTnVAFxqm3adXHfzl/vHy/hdxXvbCrgSKa/CZ+
vV+Oz5PiZRL9PL3+Dnr/h9MP2ZK9jYT2p3l+Oj9KsjhTNUDrV8PA+jm4SPg++tgQ1d4Cb+f7
7w/n57HnWFxn39iXfy7fjsf3h3vZ/V/Pb+nXQSFYfC5DTqj8uk2jqJfvu/d+Vrq+gfqffD9W
8QGmwK8f90/ya0Y/l8XRDqcyew/G8v70dHr5xyiTCv27aIs/j3uiuyD6j0ZLJzXmbTDGTv7W
Pyers2R8OdOuaAM3qsCRaV7Kg1exiRMpNfFWz5i/TCrwqgj5UyzhBBtSQdJxY7gLeIF3GvJ8
KIQU8AYN3X5aPBxjfTsckh1/iZjs66i/vkz+uTycX5qpOjRU0swqXOSXMEI2hw2wFKHc8olm
oUFGblMbFOI/uzgOQU83jBkawHSbbsn1ZmpkyGqQqgbXfF5kbFhEPp2yYXYbHGzWGuMabD9W
VJyBbIqlfciAIU9fS+zp2NMO0YJjpWdrSm+WBg4F464+sArCb5bpUnFRcnOhK7f3poYE1f8u
BfsM/Zj2rQKmRMfiYBZx27tVUXLLPlI1NXjbMRo+PByfjm/n56OZ/jOUZ1nb50NTtxi6Mwnj
febiRBkNgUYYbIkkHqEizpwBgeUy8sfloYOVdfK3Zw1+D54BGil8kUdyqKsL94ynmmUgxAiD
FYdOwInIcejaxP9djr0qtriLB42g1lUEnFsJGbLqSrixMQ7qFgj3qRjB4J7gGg42OQZ+sxfx
3PhJ21KTdHP1So999OXG5q0V88h18G1anoczkhasIRjBKhsiDW0pib5Pywo8nGZZEubTqT0M
5KfpXPUUgpbNfB/JATQlBN+hEXhEFJqGkugUeyNPiGwMcokswmbNbeUqOkP1rH25lzLa5HKe
fD89ni73T2DrIXeXC9lgQohauFLhc7OaKBLDeGbN7Yq/wJCg7XBX+gDMHaMcx+cGMABzG09e
+XvwKHsbD2nRZj551LcGvw/pEsJZyYN5mGV4zhLYWEDk9ucbv4MDreUM50WB38ZXkOjo8ncQ
zIyvmjvcGAfAI8vlbD7f00fnns/dqMnFVmmOSWzMJlzvgBYElBZFEBbDpkQdWVbuv4SabHZJ
VpRJl4cRrzaBh69l1nsSgkvH72/e0Zs41ZHjsbGYFIKTyijC3B88zcc4Dfe2vhJHh1HIpjQy
3TTIxoCTiIM1J0BwsVEKqF18/KmQOojk+ASCzorRv1CS5mw0Ex1c12yoTbgdSSejJTOzo5Ru
fgdyY2cMSTX7ECLmkIYxp6jvGXZGNXpEAuy9og4MY+Rei5UEmxdxZ+bbLWYQY5Qw16psK7DJ
i1sqayLegp4gGZs02XZsNxgWZVuB4JM8tI8FwsKbQkP2beHj7ImKLEvCeUQ0bTangrGmBq7H
38s0sB/wxkHNe5Rx9SiDjqLKd6rE6yzypngs75a+bdHmb06K+7bj2x3m2m6C95vl2/nlMkle
vqNNJtUJsaMwS5gy0RON6uH1SR4yjW0qcPG6vM4jr1ERdhqJ7iktof48Pp/ksb25c6Ria53J
GVOuG+mIW08VR/Kt6H1/Oqku8QPL/G1KfopGdpYoEgFZD8OvNBxMmYuZhb3/RRQ3cVBNmiE3
aSJ4mrH+t/AFaZXCgWtVkrxspcA/d9+CZrdptXtmE+p73NP39h5X9uskOj8/n1+IB3QrdOpD
CjXFNuD+GNL7N7Hl46GUiz77i2oJrd4SZfucWSclr4qye0pXyhRoOwbtDdZrQgYFG3IwrQyP
kbFgYE0Xa91CM7vkRLvX04NIbUgSmFojF7wQFnMkHhRA7LlDAh5NeQQUj/eoVxAXFUEC07kD
huMCjdmGahQ+nbvsxJOI5Rm8vuNVw0DUCA/8q/DcH4m9K0ESjVH9Duhv32yW2Xi7z2YW770A
GJvPDbIDW0RcDAILLRNxWUAADjLfY+F5rPAt5SGbnGxAQPKxL1PuOy75He6n9oz+DuhAkJKK
N2OjNAIyp2KN3GRkZa3AARegsW1KckynrNCnwZlr010caL6Nqq33rbZZ2hv3a1NHm1HL9eT7
x/Pzr0bdaawQWhEZb/OcOCmYmFaO8DG5Brxax8PeEgxq08QyOv7vx/Hl4ddE/Hq5/Dy+n/4P
PHHiWPxZZlmrntcXL+oW5P5yfvszPr1f3k5/fYCxAV0mICMi+/qrRWgru5/378c/Msl2/D7J
zufXyW+yCr9PfnRVfEdVxJv10iPZlhShSQjWvP2/LbuPOHS1ecga+vjr7fz+cH49yg8fygBK
PWWNRL3VqO3ya6XGfLouKG2Xzz+wrwSJLLnIV7Y/+G0KEYpmKI6W+1A4kPd8LGxDs7mu7qqC
6HvycutaJFWIJrC7ln6aVfooaFwnpGBGJZTWK9exLG6+DrtJyxnH+6fLTyS+tdS3y6S6vxwn
+fnldDmTgbdMPI+spopAthPQfFs2H9ZcQw6RRrj3IRBXUVfw4/n0/XT5hcZcW5nccfEpIV7X
eJ1bw5nEMsMStP7peRobvlPrWjgjWcXX9ZbPHpfOiE4KfjukTwa11+umXCMu4Bv4fLx//3g7
Ph+lwP4hW4NMepgARLHakPwhaTYdkKhMndo0bbKmjOziDUhErOW+EMGMRGJrKHTAd1Sqnsz3
PtFd7A5plEMuZ4unGrMII8YEBkxOPb+ZevwugnjG5JpmvmUi92Ox5zeY8W7DsxY6QHloPXPU
/h5FOz2qgE7DsR1/kYPV0F2H8Ra0MOySmLkWVlXL33J5oPrHMhZzl52oCpobI2Rtz9iwnADg
wRXlrmMHpKJAYhULEnCp00wEjuScMASAP0WDZlU6YWlhM31NkZ9pWehOCCWydeaWzWZe1ZjD
H/wVaLMSGr4ByIZRSDRSVqyJxhcR2g4WuqqysqZcps0upBxSzFVj6W6ynex4j43NItdfuVpT
t66Gxp02NkVok7DxRVnL8YIqWIaQ1b6h9Y2a2jZrKAkASeNe37guHqVyvm13qXCmDMlIU9+R
ybpSR8L1sEWVIswcrs9r2a2GX1UvRAPGurUDMqMFSpI3dbn9YCumduAQM9JdtMk8PnqDhrCK
d5fkmW8RHYKi4ITlu8y38fT7JntJdgoRBumiom0F7x9fjhd9ocEsNzfBHHttq994T7mx5nO8
uTZ3cXm42rBEU6WCIWMF7gbmyrVHbtvgsaQu8gRi3hAhLI/cqYPTLjfLuHoRL1G11bsGMwJX
O4wg222A/aEMwBi3BkhGbwtWucumWdb0gXaKomP5pNgO73M5vj4d/yGHDKUI2hJ9FWFsBJeH
p9PL2CjCuqhNlKUbpscQj75JP1RF3cZFQzst8x5VgzZAwOSPyfvl/uW7PJe+HOlXrCttjMde
yYM5f1Vty5qoyhBDDQ76WVGULcO4tADuyzxX8xl8ZZt9/0VKwso77P7l8eNJ/v96fj+pxDb4
dNXN6M/ZyVnt9XyR0smptzHoZIipg6/9Y2EH/1/Zsyy3keu6P1/hyureqszEkmXHXmRBdVNS
R/0yu9uSvelybE2imvhRll0zOV9/AbIfIAl65s5iHAFovgkCIAgc25cvp7MT+2oFQeyLO4Oh
5o4IUzGf24DJiX2Xg8zToXAecNZleuzlY3K0HKeDbOdh0O23EGlWXrgh64Mlm6+N5v6yO6DE
x3DOeXl8dpwtbWZXTgN6cJyugJ8HnLJKEPi4g8KSLdyAiyUbyDeJShxUSz1NJ1RbMr9d5tJB
Ayy6TE9MGSN9dXrG37oB4uSzx1379jNQVnE2GPvMP7VU0lU5PT6z+nBTCpA+z9g59iZyFMIf
94/fWcNGdXJxcsqW5n/XrZanv/cPqPXhfr3f496/Y40mWtQMSnZJLBRGIJPtVSA32XwSejRa
8i77ahF//jyjQnSlFrZCX20vnGVIUach/woohhenUcw5OQ4kLrlKT0/SYyak9jBf7w5l53N8
ePqJMXn+0TVjWtmBwBAymQbYwT8Ua06j3cMzmvtY1oDG4otzm7cmmUlTXURFY2K++nu8lhnJ
YpCl24vjs4ltc9Ew/uo2Ax2JXqfib7IPazi1qEivf9Os9Gi0mZyfntHjmOvmoBNQL3344Ub1
QJDzohBB6Dy8qEmERwQOqSkJTMfcOreYjq5F58Z0PUnxudHdj/0zExFQXaLPvf18sF0kbJS+
LtGyuqTOvV7ZQ9GliNat82oG+Jas0aWwVkWa2vKBYRKr66Pq7dtBewSP7ewTmgOaPKsZgW2W
gGwaW+h5lLVrTHfYVPOp/nIcQfiie6La1oVS6IX4i0PGwc8qkdJolojCyUuy7Xl2iVXauCzZ
ypRrJSLLrWin53nWrqoksr8bUNgJa7YBWUQyLfCCT8WSfx5oD+dQMHosR4K8MEriFCP0f8Vk
DzQMYsTH61fCf+MgHu9fnvb3hLHksSoSK/1yB2rnSQ5rCZYIz976oogQIDjbQR9Xh/4cNtro
02LA6E1SxSIQeKJLrSjxyYWf9mO1OXp9ub3TR5qfKAQ2HSdr6Bey9cq2yhlYu6y5IMEDOqtI
zocBWtIAvAO05yKjcdVvLDHsl0vuZcLCfosHP3XwSnyMlhcxmycCSLpwq3ZkQoLob7h9jNDx
bQPFVlbIaw2ZS/RytoFFRF0wMT4mHB3b0YZI9DQ2RGiDLizLzxdT3nu8w1eTWeAER4JguHxE
4gMnXgllWjbszawtSoshV0nBJwuq0iRz0mtZK03Bv3PY0SwBnLRIwvH5oqrpenIeDZjrwz0G
89JcxRrUK4HiGYhmoAKWQlVsBYBLikxYnZTbetouOHMdYE5aenR2AOBiVQJTFKU+qpJRo0yg
shEzc0uZ4dsMTE6ja/doAxXMnApoF2bBMGAauW4wBwq+KyfC/td5TOrGX65gAPVl80hEK/LC
RMkEBhcwVqiwHgik0ZqB6/dkSb4o2ILarahrxaPoWJDTfCToR4Qz8fbNHD78SksMfOHPIUKd
odGEaCjBaL9WFVtdKbv0l4vKXWujklor78P+HElS86HF06bherC57LkVWlv4StFeowZiYhG3
mMdoXBQJnNgINq+viVkij9H58dqiCLUPRBl1XaKdiW/mlbSnYACRY8dDzZsEmHGOHt+5qJtA
yqNqSHc1nvJ+JISBnWmMfkxFWiP8MnpYF7EQH61kSQU8NOdacdkUtRVWolTAOQ243QiVh8bO
UIR2u8HWShJV5nKR1e0VeQBtAETn1F9FNVkSoqmLRTVr6UMdA3OXoeZkXA8LmJRUXFt8YoRh
FoEEU3y18Od9ApFuhM6mlabFhg4ZIUaxjlvwhCST0MeiHGJIRrd3P3bOA0rN63hvEkNtyOPf
QFb7FF/F+jhiTqOkKi7Ozo75gWniRT+IfeF8gcaGVlSfFqL+JLf4/7x2qhwWX+1MTFbBl3wD
rgZq8nUfeRQT8pWYaHt28pnDJwUGogBl6suH/eEJc6j+NvlAd8FI2tQLzuNc98Sq30CYGt5e
/zj/MLDB2lt9GhTaCxqpNpZE8d5gGjXwsHu7fzr6gxtkfYjZ54kGrdHBkWmARmLAYLqzNBAH
GLNhJDUNq24eXa+SNAa9cASvpcrpJnIUelC87TZpAH/KOTT62OU9LJqlrNM5u3xAqVnEbaQk
JjUij/vxz3jg9nqgP55kjSaViVSDMU5lxp59NOQQ/OhXibX6CLpfvi0sX0sXo7jPJ3zKKpvo
M+/gZxGds1fyDsk02JDz039VB/cCxCah2akdDDEuORhLrXdwnBHbIZkFCz4NNuYsiLkINubi
hHfTtYlOeYOmUxJvnLWJZnzAOru9bLxXJAHWj8uyPQ/0dDK1X024SM54jzQ6KJJdZl/VxF1f
PYIzSFL8CV/ezG1fj+BcMCj+LNSQ0ALu8RehDye8f4BFwnsrWyShhq+L5LxV9jrWsMYemUxE
LZzQNMFTD44k5sdw228woOc2inulPpCoAlQIkfu1RdcqSdMk8jFLIVO+QsyZsw4OBlIk0Fon
BoFLkTdJHeh8wvUfxOy1FXMIEXjyk0BVeYIr3AO0OcY5SJMbfd8Mele66IJC92JU0W4sm6+l
/ZtXEbu7txe8jxgDoQ3HJg22jb9ApLxsJAZzslVaTMcKehxMF5JhVCmqChlFRcZ+gW28wuSq
Jo2YdQQj0uQijgySN6N0iiZG56q0ibpWScTFVOC0/h7GntFD0bmsN4UiKvmAKUVNZk1HegKx
K5Y5dBW1HZSVQfYGPU4YKWUUDlwyTssC/RGVoapoFE3UqXXmSH+J2XBNMtx/QJumfvh0+LZ/
/PR22L08PN3vfvux+/m8e/nADGdaiLhM+JjzA9G1CMQFHCgwfw9IoWzOY1JXtI6LTY6Oiswg
U3QrhbIjqWrlWqNRxpMpjlqE2yLnI88F6NHktgyYQAKfaCxMIjCf1NKzh7JoOwfgqFnztj82
oqG8stJowM8W5U4QFJuGHVxNEcdGPKURJzsFYdw3gvBHnIAP6PN///TX48dftw+3H38+3d4/
7x8/Hm7/2EH5+/uPGP/+O/KLj9+e//hgWMh69/K4+6nzKu/05fHISv4zZl462j/u0dl0/9/b
7hFCL/hGsCEqrR23V0KZdMNd2gMiHnNUmNKLDj2AYAPAhOEKsCdgQMGOfDepgkOKVbBmjQST
TJj9TbNOuJViDBY4VggJb1jmx6hHh4d4eITm8vG+pdtCGSsUWQgmhKad/t3AQDeJymsXuqVa
lgGVly4Eg2ueAROOiisybcj7i8Fk8PLr+fXp6O7pZXf09HJkOBBZCZoYFBpqLuuAIl2KMgmA
pz5cipgF+qTVOkrKlZXz3kb4n6ysZF8E6JMqGmJxhLGEg2724DY82BIRavy6LH3qdVn6JURF
xpB2ASpDcEsls1GYeEaHytXmcs4T2iaX21qJzrbu1rZcTKbnWZN6iLxJeSDXMP2HY5T9EDT1
CsQUb9i7OLLGrvH27ef+7rc/d7+O7vQq/o75gH95i1dVwmtX7C8WGfnVySheMUAVV5bfe9+p
Rl3J6enpxNK2zIXu2+sP9Pi6u33d3R/JR91g9Kz7a//640gcDk93e42Kb19vvR5EUebPAwOL
ViAFiulxWaTXXaR9t41CLhOMps7fY3W7TF4mV+G5kVAHMNKrfh7m+lUaCjAHv+XzyG/lYu6N
aVT76zqiNztD3XOPLtUGMRtWLObMBJXQnHC/tkx9IPVulH271g8khk2tG+62um8rhiLrB2l1
e/gRGqNM+IO04oBbbjivDGXvnbg7vPo1qOhkykwEgplh2m5XoUi+HcU8FWs55dLIWgT+eEKV
9eQ4Thb+emY5+DsrOYs5e8WA5D85bUvXT8IhSWB1a++Sd1aKyuIJTdJBwPYLlxExPeUtPiPF
Cfvqpt+SKzHxBgeB2B8OAfVx4NMJc9quxIkPzBgYXsLMi6XPlZcKIwj5A74pT+2ASEbm2D//
sNzmBt5UsRyrckInehR5M094fbSnUNE7iwVksY0dZNhB9CERvA0kMpmmNMfvgECV3IviQrDs
e6QRfeYVGdspiXvBTP99r+vrlbgR75y0lUgrODKCZwkzI5XjbuLjVckHUhwW14wptpacqtUj
NwU7RR18nCGzwp4entEN19Zq+oFcpILmP++PkZvCg53P/O2S3syYb2crX3q4qeq45//q9vH+
6eEof3v4tnvpn3A7L7+H5VwlbVQq1ru274SaL00kcG+dIIY9PQyG47Iawx3BiPBq+JqgqibR
7ZDqJUTm7t0iqIbxc//t5Rb0qZent9f9I3MM4sNFIX2BUz9oNMfJkCXD3wiEKjxqSGTWNcm3
ESIJVMILeT6d2a8+vD/RQCxNbuSXyXsk7zWSnIyhHhBBkGvJcEq43VxtmK6BRpllEq152hRY
X5e2ptojy2aedjRVM7fJtqfHF20kVWdFlJ1bEzFbrqPqHP0GrhCLZXAUn/tcBgEsKgn4MfFA
T5Zo3CulceRA34rejjksU3x4+4cW0A869eph//3R+CPf/djd/Qkq/rhkdVgsicXqej7cwceH
T/gFkLWgkfz+vHsYLvHMVWBbY1Z1Y3S1w/z7+OrLB/dro5ORwfO+9yhavcJmxxdnlimwyGOh
rt3mcBZBUy5sK0wqWtXBlo8Uevfjv7ADozPAvxjbvsh5kmPrtO/I4svwhDnEPIyJo7yky7iH
tXNQIYEVKi5Cf5rkUiigzZeU6aCjtdXFeQJSD6aMIAPeO0uDQJRH5XW7UEXmuNNQklTmDjYq
VEz3NfQ2k6AsZ3NJrYPGDC8stToCbRAYsAWanNkUvogdtUndtJYZA2V/a+8DYLiwCBzvmgS2
t5xfhzRIQsKLXJpAqI1Zws6XMNr8R2fWoRvZvz4Tk2ky97WdiMjqRrmhY5zHRUa6PqLg9Ee5
wjwosqCx9OE3yFPh4LOFixtzFjhQkDXGki0oKZnAZyw1yBw8nG8fSCMMuQZz9NsbBLu/2+25
dWR0UO0FH9CsOpLEyW5kY4XKvKoAVq9gTzD1VcD5OQWtQ8+jr15p9tyOPQY8C0Y5j4Nrqc7b
5/RWqeccEbEdwQ8d/bnWsU0zy/JbFVECO/1KQo+VlWVIaO9e6pdvQDo5UCZKG24SNfUtxlxW
1Gqbg9TeVgYBHGlJr8o0TqeNEqW+sKJnu4KmI07EsWrr9mw2p9epsQ5EG6VC4RXMStrvP6pN
n+ZmdEzEovDpRsDHqVqmZjTJIKfF3P7F7NZhJnQK5DMyeVF609aClJCoS5SfCF/NysRKeAw/
FjEpHB9aKDSd1YrMT4XPP4rUGSucihKfSVjW8QEFGCVxnHFFCfQFBOGEoWs6p8tF2lQr53pX
X4bEsizoUMO0OI7gePmaLwNMfXhg6Byt9gVSL/ho6PPL/vH1T/NO7mF3+O7fUOMSL7Sr7DKF
UzMd7OefgxSXTSLrL7NhJjrRzithRvp1nc0LlDKlUjno4GzHgo0dNMT9z91vr/uHTgg5aNI7
A3/hspDpFat1G+6GzySjyxrU/G2X8QVseKk9YL+cTy6m9vyUwADwoU3GGzCUFLEuGKiYWlcS
H5KhsygsCbqgTVtB2tN+CFlSZcJKjudidPPaIk+v3TLM7eqiyc0Herm2J9M5T7eRYq2D20el
lbfiX4/3f2hamW4Zxrtvb9+/4yVb8nh4fXl7cBPQZWKZaL9ENhdY1z7LuNTDNKvZ4P/f+VDf
n2i6DF96vFOOe889cGfN3GGY18vY4ob4m9O2Bi4wr0QO8lKe1CDMt2aOh681ll39/2oE7X6a
23R3WtGhsRfDuyvRoTDiR4pbFhQQjFtKDWWmDMT2LN0ZuQHVb6Buu3O+p1hHscml5bmhoWWR
VIXrX+7VA5tp4begmONLPdbjJG3mPZF1jawReExyX+m57kYUeH0KG8KvtMewe77bu/rCu3Fz
9vVNAB4TdzQyj12WM6wfU9ZV1pbLGsfXb8oV/4rP/fC9RdrRmrya7uQHwCYriL6Rd1GrZLly
hJ5hSHWP0QV+4XjOM+j3N6KoqJOag8BbFVsG6bwdDNY3zBgs+iXhiZsX444FqcmSq52KAwUa
cNHUaWL7TRhEkiOcnThD0K+MwNQRojGPpNvz0bVaG1REiNd4bMGblZWTpNFcViH9UfH0fPh4
hFEl357NkbC6ffxuZ+iBsYzQbaMoSm7HWXh8NtdI6NS4W4tFjd4WTTmEwGfZBKLaVQPTV4tq
TRel8RMZUDoTG0zNl8l0GLt5UdRasidkul1EHwiRdE2ejE3eXMJ5DKdyXPAvZvSEmD6xM/L+
0BovQziE79/w5KXsfBw0zV88Gd3Geyxw9H1hSrc3OY7hWsrSeXLVcWslZWbfaxg7EN40j2fZ
/xye9494+wzdfHh73f29g3/sXu9+//33/yVBErS/GJa71NLwkMGuXz4KE8eOz78ssBIbU0AO
Q+60VcNxDIKnDmppTS23VhpksyW6rIAeW+TJNxuDgaOn2Nhujl1Nm0pm3me6hQ4j0/57svQH
vUO8cxoY1QraICWXy3EsBscZdbxe86js6jFOAHrctXZa4LGTo3I36in/j6nvC6wVZgsB3rNI
BXWA1XxPI+koaAkYHcyaHG+4YPkb21Cwp2sjOvTSkdl0fxqB6/729fYIJa07NHMSDakbpKTy
jr2yA7rMk9+ABmk8ckEDYBpp5JY2FrVAgyNGpEo6OcZiE4EWu1VFSnbelUxOxqhhpUKzhaLG
21VR4wyBsyJG/QcoMX1cG7RKIgX9mhkJJAHZr9XpDJxFhzjQVVqtPw2cfTpxKsDFEqxeXjJP
tsYoL9bgOPv6stOb1KgxWQTmtSpI1hg/gB8ANC3m0XVdcHsy11HGoPHKkXUGhe597FKJcsXT
xNegewM7WPQ7ySrA7L1MP43XLogqdkjwGZkec6QECT6nbMKk6e0+NKWQxaLLjmz+qU0dbgY5
Hfdb01u8Hf7UOKYmZbzXPVJUp9dVG2ps644nNCOxjffq6wDk7Blmb+EtrZ5JCYxJTrmWBnSd
NwHNyQ0XRerLk3UAaWyMLq7nZR5c55v1K1orWYdQXXiJUVYwcJMkNUoTxzfApTO/Au/AO5qr
BcYSxJAaWYwXMJwOvY5QnF1HxVU7V8VackY2KAEtdUmnv8uYHhL4cqGjoDOmw4YRnMcNn5/+
2r083/GCVVlGgzvmRirFCqNIZJDWHVHeu0HDMQ0iwNmM0ssMM/UYpdDy28RnKSVKk9QWOOK/
NlkJPGSOjvRSO8IbZbL6ZxIvuAK6wCfbpmSqyaqkNRZKBontx92EojqGNfFK3lruN/hrsAs6
UBjaCkShuf0sgX7RqgIvcPlUpCglUlYwj5OOS1hGBxxvfPoQtGIvRJKaRMbud2Udw3CGdbKV
iIHhFIsFPhc+/nt3bP4jhk1veVGTbb07vKKAhOJ+hIlcb7+T4JU6aoalTuowGkyuZQvvnsoG
KrcdX+FPXUOkObwdpaMXWlq9wrtQSZbZaDh+whTmXTVBjBwVhr6C1cryDz0v2hikxeIwTSbW
sn9dFZ7fQWCw6reaNpgI37OgrG0XfWNNgOWAzKvj6dQ4YFFrDtdZz/SdgkJjmP14C0nQRqya
DA9E3txpqGAbCiWFcR44/numF19/MMIxjDfVtVGGjAfSKNqvYx1uzVFT0S+gglOPqVETZEmO
dmwSkE6D8RNrzZnjxthCr0OLbt5L9prLuLLeHD0YfRlT34wVaYFMIihn6ihAoKm3Qxkc45YK
hUm72v5GimF89J2GjdFjsJJbZBbuyJhrHfOSrPKRlfVexBhvAFwXWwfaOVnYBfjXSD0Y9mzK
eRJqPD588uZ+q4WN0CfEjEfBCi/ojbHSbpl9ca9BSSwcSLrO/A7hRajbts7OFbavoMtG674F
HHd5kmPstfpdJxNdziJRGeiW0mloF0VhlJ70b5vh9htLO9+wCOLE4q1raF0VWqimh7FMhbtU
uveK2gvJHTPLohseOXyyJGB5herVQoxt+e2/09AxEpnMXDvAu8ec9zjLXF/+H06ZP0/yaQEA

--gKMricLos+KVdGMg--

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,
	HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham
	autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id D3F9CC64E7A
	for <linux-unionfs@archiver.kernel.org>; Fri, 27 Nov 2020 12:53:05 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id 5A09822252
	for <linux-unionfs@archiver.kernel.org>; Fri, 27 Nov 2020 12:53:05 +0000 (UTC)
Authentication-Results: mail.kernel.org;
	dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="svAdVyd5"
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1726885AbgK0MxF (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Fri, 27 Nov 2020 07:53:05 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46520 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1726477AbgK0MxE (ORCPT
        <rfc822;linux-unionfs@vger.kernel.org>);
        Fri, 27 Nov 2020 07:53:04 -0500
Received: from mail-io1-xd41.google.com (mail-io1-xd41.google.com [IPv6:2607:f8b0:4864:20::d41])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8C55DC0613D1;
        Fri, 27 Nov 2020 04:53:04 -0800 (PST)
Received: by mail-io1-xd41.google.com with SMTP id t8so4686323iov.8;
        Fri, 27 Nov 2020 04:53:04 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20161025;
        h=mime-version:references:in-reply-to:from:date:message-id:subject:to
         :cc;
        bh=7pN1PyUJpffxldlksYc4OhYw5FcKBSSXURbDzucU5uk=;
        b=svAdVyd5ghUWu3rmsV3Dbf/5zRFcGx9Oj6RK3UPz2tstDMGvBPcJ4K6U0VfonvEXJx
         og+zp2TBHr+cTUaGjUp6ym7GYsBQmT0ia8mApyh3RtE4pbfIkZqo6ULJ5l1kP4KZrIWS
         PjCdXpyheToq0tnzTfwg5NoFAUXZDyLnql21M+evpvCZrSWR9YhJf1OOhOAth8MtORhk
         KdyvOU845m3KdsiH9xGWIi0QpxKQygRUjosQ6Gtnl5SpZUT9yH3JkqaXCCNGN5I77xnm
         4QVoy5DcFwIA5/yUGFKrk1kZsYOl7kIRha0UOuGfvVNSotXrkQurLJdYRTXhI4hzeEem
         BpEw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:mime-version:references:in-reply-to:from:date
         :message-id:subject:to:cc;
        bh=7pN1PyUJpffxldlksYc4OhYw5FcKBSSXURbDzucU5uk=;
        b=gNZ+w+/ksfQYmUpW81PQ+x1jF1d6To38k3f2QlwRo+BCUAP7+IsmlvsKIfOZVB9thm
         KixNLtdm4UFpoW2OfZzd5tkrSDTJmzVCxHfsGfByttHPkoUtHrJIP17CPHYcH2pUKfpe
         yiTkjpAZ4TT73b7S5N24jS54PN2AHg8ja6Kw8AXOeUYlQayKUdKhpokSa93h2deofNMg
         +aIUJd+HDS7rf5G/7QrCD1aLouCQjbvefikkyN5ecBmfO09A9EzWnt5mOVo7NhoZYDPT
         iDelOfIFobk6UvvvcYp05EyZmXVhzsYSDCOGlmQ01cQE6FPJQit8kLoAgQjk6yPaqUac
         VC0w==
X-Gm-Message-State: AOAM5309+9SONE7VmeM793qBG5tGqfXHDWkkuQK97BX2Tuk20swgajxb
        HhuhR9w9e8GoF/p2a+xeY79DIaDNzA0cPPER8ZKNeF/j
X-Google-Smtp-Source: ABdhPJzbpnadQMzJ6hpcInkVq6AeU2hAOLxqtVIYwPY7R/mXI4GgZp+HWcx1wL93Un2kltfYhX9YWNGilJGHgv1VILo=
X-Received: by 2002:a02:ccd6:: with SMTP id k22mr7324899jaq.93.1606481583344;
 Fri, 27 Nov 2020 04:53:03 -0800 (PST)
MIME-Version: 1.0
References: <20201127092058.15117-1-sargun@sargun.me> <20201127092058.15117-3-sargun@sargun.me>
In-Reply-To: <20201127092058.15117-3-sargun@sargun.me>
From:   Amir Goldstein <amir73il@gmail.com>
Date:   Fri, 27 Nov 2020 14:52:52 +0200
Message-ID: <CAOQ4uxgaLuLb+f6WCMvmKHNTELvcvN8C5_u=t5hhoGT8Op7QuQ@mail.gmail.com>
Subject: Re: [PATCH v2 2/4] overlay: Document current outstanding shortcoming
 of volatile
To:     Sargun Dhillon <sargun@sargun.me>
Cc:     overlayfs <linux-unionfs@vger.kernel.org>,
        Miklos Szeredi <miklos@szeredi.hu>,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Vivek Goyal <vgoyal@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel <linux-fsdevel@vger.kernel.org>,
        David Howells <dhowells@redhat.com>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org

On Fri, Nov 27, 2020 at 11:21 AM Sargun Dhillon <sargun@sargun.me> wrote:
>
> This documents behaviour that was discussed in a thread about the volatile
> feature. Specifically, how failures can go hidden from asynchronous writes
> (such as from mmap, or writes that are not immediately flushed to the
> filesystem). Although we pass through calls like msync, fallocate, and
> write, and will still return errors on those, it doesn't guarantee all
> kinds of errors will happen at those times, and thus may hide errors.
>
> In the future, we can add error checking to all interactions with the
> upperdir, and pass through errseq_t from the upperdir on mappings,
> and other interactions with the filesystem[1].
>
> [1]: https://lore.kernel.org/linux-unionfs/20201116045758.21774-1-sargun@sargun.me/T/#m7d501f375e031056efad626e471a1392dd3aad33
>
> Signed-off-by: Sargun Dhillon <sargun@sargun.me>
> Cc: linux-fsdevel@vger.kernel.org
> Cc: linux-unionfs@vger.kernel.org
> Cc: Miklos Szeredi <miklos@szeredi.hu>
> Cc: Amir Goldstein <amir73il@gmail.com>
> Cc: Vivek Goyal <vgoyal@redhat.com>
> ---
>  Documentation/filesystems/overlayfs.rst | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/Documentation/filesystems/overlayfs.rst b/Documentation/filesystems/overlayfs.rst
> index 580ab9a0fe31..c6e30c1bc2f2 100644
> --- a/Documentation/filesystems/overlayfs.rst
> +++ b/Documentation/filesystems/overlayfs.rst
> @@ -570,7 +570,11 @@ Volatile mount
>  This is enabled with the "volatile" mount option.  Volatile mounts are not
>  guaranteed to survive a crash.  It is strongly recommended that volatile
>  mounts are only used if data written to the overlay can be recreated
> -without significant effort.
> +without significant effort.  In addition to this, the sync family of syscalls
> +are not sufficient to determine whether a write failed as sync calls are
> +omitted.  For this reason, it is important that the filesystem used by the
> +upperdir handles failure in a fashion that's suitable for the user.  For
> +example, upon detecting a fault, ext4 can be configured to panic.
>

Reading this now, I think I may have wrongly analysed the issue.
Specifically, when I wrote that the very minimum is to document the
issue, it was under the assumption that a proper fix is hard.
I think I was wrong and that the very minimum is to check for errseq
since mount on the fsync and syncfs calls.

Why? first of all because it is very very simple and goes a long way to
fix the broken contract with applications, not the contract about durability
obviously, but the contract about write+fsync+read expects to find the written
data (during the same mount era).

Second, because the sentence you added above is hard for users to understand
and out of context. If we properly handle the writeback error in fsync/syncfs,
then this sentence becomes irrelevant.
The fact that ext4 can lose data if application did not fsync after
write is true
for volatile as well as non-volatile and it is therefore not relevant
in the context
of overlayfs documentation at all.

Am I wrong saying that it is very very simple to fix?
Would you mind making that fix at the bottom of the patch series, so it can
be easily applied to stable kernels?

Thanks,
Amir.

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,
	HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham
	autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 7A776C2D0E4
	for <linux-unionfs@archiver.kernel.org>; Fri, 27 Nov 2020 13:05:13 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id 05BBC206D9
	for <linux-unionfs@archiver.kernel.org>; Fri, 27 Nov 2020 13:05:12 +0000 (UTC)
Authentication-Results: mail.kernel.org;
	dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="QUyskmOA"
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1729402AbgK0NFM (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Fri, 27 Nov 2020 08:05:12 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48410 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1725865AbgK0NFM (ORCPT
        <rfc822;linux-unionfs@vger.kernel.org>);
        Fri, 27 Nov 2020 08:05:12 -0500
Received: from mail-io1-xd44.google.com (mail-io1-xd44.google.com [IPv6:2607:f8b0:4864:20::d44])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E1EBDC0613D1;
        Fri, 27 Nov 2020 05:05:11 -0800 (PST)
Received: by mail-io1-xd44.google.com with SMTP id s10so4759585ioe.1;
        Fri, 27 Nov 2020 05:05:11 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20161025;
        h=mime-version:references:in-reply-to:from:date:message-id:subject:to
         :cc;
        bh=/aCfQfU+2NxErOuM6SBa5QT3V1nalBKQFOreo5Hzh9M=;
        b=QUyskmOA4jrVRCDiEEaw/xep3qOSEwiJOzF5wkZqQJ5/IU5vX86Gtg8l6TJXWoYt6I
         5Y5AHGhR2V6+jQX/lmtsWDKiGkynWmimvQzbqKDUD4d2msWTYRpi+89ahnASVLVYJilp
         hWOhwA8q0uFDT8pnSz4pfTaMupMG7T25comj3rVGKw58aPWh64PZeUPqYak4ZFkIwv6c
         IwxnruDEjL/7MJ0DfTCNofqZNpcqhttmRodC9B8gnLUBBrlENU2JSbAKmjn0HharHtjV
         auK8Hhn5NEsfj9I+ZAynv/SWJjPoY7eMT+XbUdUGXL6lvvf7sBiWmP/A93ZOuFO6ETuZ
         sxTg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:mime-version:references:in-reply-to:from:date
         :message-id:subject:to:cc;
        bh=/aCfQfU+2NxErOuM6SBa5QT3V1nalBKQFOreo5Hzh9M=;
        b=rLSynWAvFZZRt3Ht/nbXoIDxad0AprtICQxE8QuSNCgEarbuvdR6Kacg6skftnMQUP
         2R9qbnMgIr7koGGUimJbcS7Duhr/h2IkIfVXVzJH3T+UI0PVFB3x4Am4rp5Sg8kofKZt
         9rR0Jzd3Y+cWA5IpkhAcfFo/DRVsqWX9FOl7YolToX7b7dDYqu5YBSbBSiDLRwXoqnz3
         U6ks5hNGs7KnU0DGkvsiCjEGKQEYw+YtGzMPiyp/mobc8DdI7hwrXC6sQY4znmU9Mghk
         Xg40iYpuWKMC9UMlF0QgP9BpCajwg6rx2DZ4FNRRwYcEY/wtARK5cJ2YvnIesJgvcH2b
         OFlA==
X-Gm-Message-State: AOAM532eKLfxNS0lYG/8C9uSJ4Z7vyeSZeMOKFy4MYF59AX6tLE1E27C
        Ch2r0CKkuKkjGqJRiNJnk9uIne4ytVrsWSmcxT4=
X-Google-Smtp-Source: ABdhPJyOqC/zPrxlOuXVupO3nkYt6H58mtxUArgl0z5u7cunUAUWf6a9Q4tAIzMFNEp8f8LoszQYvTaQD/LB/ejt5d0=
X-Received: by 2002:a02:ccd6:: with SMTP id k22mr7375006jaq.93.1606482311192;
 Fri, 27 Nov 2020 05:05:11 -0800 (PST)
MIME-Version: 1.0
References: <20201127092058.15117-1-sargun@sargun.me> <20201127092058.15117-4-sargun@sargun.me>
In-Reply-To: <20201127092058.15117-4-sargun@sargun.me>
From:   Amir Goldstein <amir73il@gmail.com>
Date:   Fri, 27 Nov 2020 15:04:59 +0200
Message-ID: <CAOQ4uxhE4qQrRn2gVs+EoyyCQdGCBZLHdfKf+oDBvii0m3aX4Q@mail.gmail.com>
Subject: Re: [PATCH v2 3/4] overlay: Add the ability to remount volatile
 directories when safe
To:     Sargun Dhillon <sargun@sargun.me>
Cc:     overlayfs <linux-unionfs@vger.kernel.org>,
        Miklos Szeredi <miklos@szeredi.hu>,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Vivek Goyal <vgoyal@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel <linux-fsdevel@vger.kernel.org>,
        David Howells <dhowells@redhat.com>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org

On Fri, Nov 27, 2020 at 11:21 AM Sargun Dhillon <sargun@sargun.me> wrote:
>
> Overlayfs added the ability to setup mounts where all syncs could be
> short-circuted in (2a99ddacee43: ovl: provide a mount option "volatile").
>
> A user might want to remount this fs, but we do not let the user because
> of the "incompat" detection feature. In the case of volatile, it is safe
> to do something like[1]:
>
> $ sync -f /root/upperdir
> $ rm -rf /root/workdir/incompat/volatile
>
> There are two ways to go about this. You can call sync on the underlying
> filesystem, check the error code, and delete the dirty file if everything
> is clean. If you're running lots of containers on the same filesystem, or
> you want to avoid all unnecessary I/O, this may be suboptimal.
>
> Alternatively, you can blindly delete the dirty file, and "hope for the
> best".
>
> This patch introduces transparent functionality to check if it is
> (relatively) safe to reuse the upperdir. It ensures that the filesystem
> hasn't been remounted, the system hasn't been rebooted, nor has the
> overlayfs code changed. Since the structure is explicitly not meant to be
> used between different versions of the code, its stability does not matter
> so much.
>
> [1]: https://lore.kernel.org/linux-unionfs/CAOQ4uxhKr+j5jFyEC2gJX8E8M19mQ3CqdTYaPZOvDQ9c0qLEzw@mail.gmail.com/T/#m6abe713e4318202ad57f301bf28a414e1d824f9c
>
> Signed-off-by: Sargun Dhillon <sargun@sargun.me>
> Cc: linux-fsdevel@vger.kernel.org
> Cc: linux-unionfs@vger.kernel.org
> Cc: Miklos Szeredi <miklos@szeredi.hu>
> Cc: Amir Goldstein <amir73il@gmail.com>
> Cc: Vivek Goyal <vgoyal@redhat.com>

You may add:
Reviewed-by: Amir Goldstein <amir73il@gmail.com>

> ---
>  Documentation/filesystems/overlayfs.rst | 18 +++--
>  fs/overlayfs/overlayfs.h                | 37 +++++++++-
>  fs/overlayfs/readdir.c                  | 98 ++++++++++++++++++++++---
>  fs/overlayfs/super.c                    | 73 +++++++++++++-----
>  fs/overlayfs/util.c                     |  2 +
>  5 files changed, 190 insertions(+), 38 deletions(-)
>
> diff --git a/Documentation/filesystems/overlayfs.rst b/Documentation/filesystems/overlayfs.rst
> index c6e30c1bc2f2..b485fdb65b85 100644
> --- a/Documentation/filesystems/overlayfs.rst
> +++ b/Documentation/filesystems/overlayfs.rst
> @@ -579,13 +579,17 @@ example, upon detecting a fault, ext4 can be configured to panic.
>  The advantage of mounting with the "volatile" option is that all forms of
>  sync calls to the upper filesystem are omitted.
>
> -When overlay is mounted with "volatile" option, the directory
> -"$workdir/work/incompat/volatile" is created.  During next mount, overlay
> -checks for this directory and refuses to mount if present. This is a strong
> -indicator that user should throw away upper and work directories and create
> -fresh one. In very limited cases where the user knows that the system has
> -not crashed and contents of upperdir are intact, The "volatile" directory
> -can be removed.
> +When overlay is mounted with the "volatile" option, the directory
> +"$workdir/work/incompat/volatile" is created.  This acts as a indicator
> +that the user should throw away upper and work directories and create fresh
> +ones.  In some cases, the overlayfs can detect if the upperdir can be
> +reused safely in a subsequent volatile mounts, and mounting will proceed as
> +normal.  If the filesystem is unable to determine if this is safe (due to a
> +reboot, upgraded kernel code, or loss of checkpoint, etc...), the user may
> +bypass these safety checks and remove the "volatile" directory if they know
> +the system did not encounter a fault and the contents of the upperdir are
> +intact. Then, the user can remount the filesystem as normal.
> +
>
>  Testsuite
>  ---------
> diff --git a/fs/overlayfs/overlayfs.h b/fs/overlayfs/overlayfs.h
> index f8880aa2ba0e..de694ee99d7c 100644
> --- a/fs/overlayfs/overlayfs.h
> +++ b/fs/overlayfs/overlayfs.h
> @@ -32,8 +32,13 @@ enum ovl_xattr {
>         OVL_XATTR_NLINK,
>         OVL_XATTR_UPPER,
>         OVL_XATTR_METACOPY,
> +       OVL_XATTR_VOLATILE,
>  };
>
> +#define OVL_INCOMPATDIR_NAME "incompat"
> +#define OVL_VOLATILEDIR_NAME "volatile"
> +#define OVL_VOLATILE_DIRTY_NAME "dirty"
> +
>  enum ovl_inode_flag {
>         /* Pure upper dir that may contain non pure upper entries */
>         OVL_IMPURE,
> @@ -57,6 +62,31 @@ enum {
>         OVL_XINO_ON,
>  };
>
> +/*
> + * This is copied into the volatile xattr, and the user does not interact with
> + * it. There is no stability requirement, as a reboot explicitly invalidates
> + * a volatile workdir. It is explicitly meant not to be a stable api.
> + *
> + * Although this structure isn't meant to be stable it is exposed to potentially
> + * unprivileged users. We don't do any kind of cryptographic operations with
> + * the structure, so it could be tampered with, or inspected. Don't put
> + * kernel memory pointers in it, or anything else that could cause problems,
> + * or information disclosure.
> + */
> +struct ovl_volatile_info {
> +       /*
> +        * This uniquely identifies a boot, and is reset if overlayfs itself
> +        * is reloaded. Therefore we check our current / known boot_id
> +        * against this before looking at any other fields to validate:
> +        * 1. Is this datastructure laid out in the way we expect? (Overlayfs
> +        *    module, reboot, etc...)
> +        * 2. Could something have changed (like the s_instance_id counter
> +        *    resetting)
> +        */
> +       uuid_t          ovl_boot_id;    /* Must stay first member */
> +       u64             s_instance_id;
> +} __packed;
> +
>  /*
>   * The tuple (fh,uuid) is a universal unique identifier for a copy up origin,
>   * where:
> @@ -422,8 +452,8 @@ void ovl_cleanup_whiteouts(struct dentry *upper, struct list_head *list);
>  void ovl_cache_free(struct list_head *list);
>  void ovl_dir_cache_free(struct inode *inode);
>  int ovl_check_d_type_supported(struct path *realpath);
> -int ovl_workdir_cleanup(struct inode *dir, struct vfsmount *mnt,
> -                       struct dentry *dentry, int level);
> +int ovl_workdir_cleanup(struct ovl_fs *ofs, struct inode *dir,
> +                       struct vfsmount *mnt, struct dentry *dentry, int level);
>  int ovl_indexdir_cleanup(struct ovl_fs *ofs);
>
>  /* inode.c */
> @@ -520,3 +550,6 @@ int ovl_set_origin(struct dentry *dentry, struct dentry *lower,
>
>  /* export.c */
>  extern const struct export_operations ovl_export_operations;
> +
> +/* super.c */
> +extern uuid_t ovl_boot_id;
> diff --git a/fs/overlayfs/readdir.c b/fs/overlayfs/readdir.c
> index 01620ebae1bd..7b66fbb20261 100644
> --- a/fs/overlayfs/readdir.c
> +++ b/fs/overlayfs/readdir.c
> @@ -1080,10 +1080,78 @@ int ovl_check_d_type_supported(struct path *realpath)
>
>         return rdd.d_type_supported;
>  }
> +static int ovl_verify_volatile_info(struct ovl_fs *ofs,
> +                                   struct dentry *volatiledir)
> +{
> +       int err;
> +       struct ovl_volatile_info info;
> +
> +       if (!volatiledir->d_inode)
> +               return 0;
> +
> +       if (!ofs->config.ovl_volatile) {
> +               pr_debug("Mount is not volatile; upperdir is marked volatile\n");
> +               return -EINVAL;
> +       }
> +
> +       err = ovl_do_getxattr(ofs, volatiledir, OVL_XATTR_VOLATILE, &info,
> +                             sizeof(info));
> +       if (err < 0) {
> +               pr_debug("Unable to read volatile xattr: %d\n", err);
> +               return -EINVAL;
> +       }
> +
> +       if (err != sizeof(info)) {
> +               pr_debug("%s xattr on-disk size is %d expected to read %zd\n",
> +                        ovl_xattr(ofs, OVL_XATTR_VOLATILE), err, sizeof(info));
> +               return -EINVAL;
> +       }
> +
> +       if (!uuid_equal(&ovl_boot_id, &info.ovl_boot_id)) {
> +               pr_debug("boot id has changed (reboot or module reloaded)\n");
> +               return -EINVAL;
> +       }
> +
> +       if (volatiledir->d_sb->s_instance_id != info.s_instance_id) {
> +               pr_debug("workdir has been unmounted and remounted\n");
> +               return -EINVAL;
> +       }
> +
> +       return 1;
> +}
>
> -#define OVL_INCOMPATDIR_NAME "incompat"
> +/*
> + * ovl_check_incompat checks this specific incompat entry for incompatibility.
> + * If it is found to be incompatible -EINVAL will be returned.
> + *
> + * If the directory should be preserved, then this function returns 1.
> + */
> +static int ovl_check_incompat(struct ovl_fs *ofs, struct ovl_cache_entry *p,
> +                             struct path *path)
> +{
> +       int err = -EINVAL;
> +       struct dentry *d;
> +
> +       if (!strcmp(p->name, OVL_VOLATILEDIR_NAME)) {
> +               d = lookup_one_len(p->name, path->dentry, p->len);
> +               if (IS_ERR(d))
> +                       return PTR_ERR(d);
> +
> +               err = ovl_verify_volatile_info(ofs, d);
> +               dput(d);
> +       }
> +
> +       if (err == -EINVAL)
> +               pr_err("incompat feature '%s' cannot be mounted\n", p->name);
> +       else
> +               pr_debug("incompat '%s' handled: %d\n", p->name, err);
> +
> +       dput(d);

Letfover.

Thanks,
Amir.

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-17.2 required=3.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,
	INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,
	USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 9FB63C3E8C5
	for <linux-unionfs@archiver.kernel.org>; Fri, 27 Nov 2020 22:13:12 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id 4951D22210
	for <linux-unionfs@archiver.kernel.org>; Fri, 27 Nov 2020 22:13:12 +0000 (UTC)
Authentication-Results: mail.kernel.org;
	dkim=pass (1024-bit key) header.d=sargun.me header.i=@sargun.me header.b="p+NwQMlz"
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1731830AbgK0WMv (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Fri, 27 Nov 2020 17:12:51 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48880 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1732277AbgK0WMD (ORCPT
        <rfc822;linux-unionfs@vger.kernel.org>);
        Fri, 27 Nov 2020 17:12:03 -0500
Received: from mail-io1-xd41.google.com (mail-io1-xd41.google.com [IPv6:2607:f8b0:4864:20::d41])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7BC7FC0613D1
        for <linux-unionfs@vger.kernel.org>; Fri, 27 Nov 2020 14:11:58 -0800 (PST)
Received: by mail-io1-xd41.google.com with SMTP id y5so4996580iow.5
        for <linux-unionfs@vger.kernel.org>; Fri, 27 Nov 2020 14:11:58 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=sargun.me; s=google;
        h=date:from:to:cc:subject:message-id:references:mime-version
         :content-disposition:in-reply-to:user-agent;
        bh=8Tzy7SZbbwW8nqIlM9ULMcC6IhFFL2USjTf7kSV1L4w=;
        b=p+NwQMlzpXG75n2TL8qbRwFJghPvivJkvvimq+bxms0kZhtK2eh3qxcXEZZEqf95b4
         V2AG1HX4Bsy3Lb0jC8zyxetSohsDW24n9dgbcGB/88t+e2XH4MokECq5EKjPSjQIei9q
         lwJUNHJFho2Ac0XVmw6VP7gLaHXNR9i4h+iKE=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:date:from:to:cc:subject:message-id:references
         :mime-version:content-disposition:in-reply-to:user-agent;
        bh=8Tzy7SZbbwW8nqIlM9ULMcC6IhFFL2USjTf7kSV1L4w=;
        b=LZ9z3+0jnWnxO/bYgYQJMC/3G4wBwTdEmHC+KpI+5lXVHWFo8O6lXJ0cX3F9+A701v
         ILuwNw6MutQ6f1MpRzOgaGhWb7/bdjeqR++jfbCgBl5LAGYjyLo1gy+SKWasMfq47pll
         147q2pmbD008Vj2F+kYi4PlEx3G3iqWKvcOW5Gqbp6TQg/VgbeMmaKXheP0ABDiSes9R
         8n5TdvnLETJHoDvK5HPmtpc/U/9SFfuXgStNm46EPwZ/p3hZlgjDfGgQik/c29sDYZ+t
         nxVYJq1YTiPzpgxV9GlnW2kfYMKCKeT6gPmQTUI8Bg4nMdc/N41XFFZ43sIC1aYCck49
         qDTQ==
X-Gm-Message-State: AOAM5305burTSrJEVGd3euQKlof931ADK/kKo8LvblNRSZlAde2BhgtE
        EdZXZ8mDxlyLOMmjTbv2Khl98A==
X-Google-Smtp-Source: ABdhPJyYUXypKceUkeItGokO2dAqrfJEhphXr6i6WKn0Sg4j0XCBKAfEOf7JCVaPhZvizRIWNtY7sQ==
X-Received: by 2002:a6b:6217:: with SMTP id f23mr7558740iog.201.1606515117668;
        Fri, 27 Nov 2020 14:11:57 -0800 (PST)
Received: from ircssh-2.c.rugged-nimbus-611.internal (80.60.198.104.bc.googleusercontent.com. [104.198.60.80])
        by smtp.gmail.com with ESMTPSA id r3sm6155572ilt.76.2020.11.27.14.11.56
        (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256);
        Fri, 27 Nov 2020 14:11:57 -0800 (PST)
Date:   Fri, 27 Nov 2020 22:11:55 +0000
From:   Sargun Dhillon <sargun@sargun.me>
To:     Amir Goldstein <amir73il@gmail.com>
Cc:     overlayfs <linux-unionfs@vger.kernel.org>,
        Miklos Szeredi <miklos@szeredi.hu>,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Vivek Goyal <vgoyal@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel <linux-fsdevel@vger.kernel.org>,
        David Howells <dhowells@redhat.com>,
        Jeff Layton <jlayton@redhat.com>
Subject: Re: [PATCH v2 2/4] overlay: Document current outstanding shortcoming
 of volatile
Message-ID: <20201127221154.GA23383@ircssh-2.c.rugged-nimbus-611.internal>
References: <20201127092058.15117-1-sargun@sargun.me>
 <20201127092058.15117-3-sargun@sargun.me>
 <CAOQ4uxgaLuLb+f6WCMvmKHNTELvcvN8C5_u=t5hhoGT8Op7QuQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAOQ4uxgaLuLb+f6WCMvmKHNTELvcvN8C5_u=t5hhoGT8Op7QuQ@mail.gmail.com>
User-Agent: Mutt/1.9.4 (2018-02-28)
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org

On Fri, Nov 27, 2020 at 02:52:52PM +0200, Amir Goldstein wrote:
> On Fri, Nov 27, 2020 at 11:21 AM Sargun Dhillon <sargun@sargun.me> wrote:
> >
> > This documents behaviour that was discussed in a thread about the volatile
> > feature. Specifically, how failures can go hidden from asynchronous writes
> > (such as from mmap, or writes that are not immediately flushed to the
> > filesystem). Although we pass through calls like msync, fallocate, and
> > write, and will still return errors on those, it doesn't guarantee all
> > kinds of errors will happen at those times, and thus may hide errors.
> >
> > In the future, we can add error checking to all interactions with the
> > upperdir, and pass through errseq_t from the upperdir on mappings,
> > and other interactions with the filesystem[1].
> >
> > [1]: https://lore.kernel.org/linux-unionfs/20201116045758.21774-1-sargun@sargun.me/T/#m7d501f375e031056efad626e471a1392dd3aad33
> >
> > Signed-off-by: Sargun Dhillon <sargun@sargun.me>
> > Cc: linux-fsdevel@vger.kernel.org
> > Cc: linux-unionfs@vger.kernel.org
> > Cc: Miklos Szeredi <miklos@szeredi.hu>
> > Cc: Amir Goldstein <amir73il@gmail.com>
> > Cc: Vivek Goyal <vgoyal@redhat.com>
> > ---
> >  Documentation/filesystems/overlayfs.rst | 6 +++++-
> >  1 file changed, 5 insertions(+), 1 deletion(-)
> >
> > diff --git a/Documentation/filesystems/overlayfs.rst b/Documentation/filesystems/overlayfs.rst
> > index 580ab9a0fe31..c6e30c1bc2f2 100644
> > --- a/Documentation/filesystems/overlayfs.rst
> > +++ b/Documentation/filesystems/overlayfs.rst
> > @@ -570,7 +570,11 @@ Volatile mount
> >  This is enabled with the "volatile" mount option.  Volatile mounts are not
> >  guaranteed to survive a crash.  It is strongly recommended that volatile
> >  mounts are only used if data written to the overlay can be recreated
> > -without significant effort.
> > +without significant effort.  In addition to this, the sync family of syscalls
> > +are not sufficient to determine whether a write failed as sync calls are
> > +omitted.  For this reason, it is important that the filesystem used by the
> > +upperdir handles failure in a fashion that's suitable for the user.  For
> > +example, upon detecting a fault, ext4 can be configured to panic.
> >
> 
> Reading this now, I think I may have wrongly analysed the issue.
> Specifically, when I wrote that the very minimum is to document the
> issue, it was under the assumption that a proper fix is hard.
> I think I was wrong and that the very minimum is to check for errseq
> since mount on the fsync and syncfs calls.
> 
> Why? first of all because it is very very simple and goes a long way to
> fix the broken contract with applications, not the contract about durability
> obviously, but the contract about write+fsync+read expects to find the written
> data (during the same mount era).
> 
> Second, because the sentence you added above is hard for users to understand
> and out of context. If we properly handle the writeback error in fsync/syncfs,
> then this sentence becomes irrelevant.
> The fact that ext4 can lose data if application did not fsync after
> write is true
> for volatile as well as non-volatile and it is therefore not relevant
> in the context
> of overlayfs documentation at all.
> 
> Am I wrong saying that it is very very simple to fix?
> Would you mind making that fix at the bottom of the patch series, so it can
> be easily applied to stable kernels?
> 
> Thanks,
> Amir.

I'm not sure it's so easy. In VFS, there are places where the superblock's 
errseq is checked[1]. AFAIK, that's going to check "our" errseq, and not the 
errseq of the real corresponding real file's superblock. One solution might be 
as part of all these callbacks we set our errseq to the errseq of the filesystem 
that the upperdir, and then rely on VFS's checking.

I'm having a hard time figuring out how to deal with the per-mapping based
error tracking. It seems like this infrastructure was only partially completed
by Jeff Layton[2]. I don't now how it's actually supposed to work right now,
as not all of his patches landed.

How about I split this into two patchsets? One, where I add the logic to copy
the errseq on callbacks to fsync from the upperdir to the ovl fs superblock,
and thus allowing VFS to bubble up errors, and the documentation. We can CC
stable on those because I think it has an effect that's universal across
all filesystems.

P.S. 
I notice you maintain overlay tests outside of the kernel. Unfortunately, I 
think for this kind of test, it requires in kernel code to artificially bump the 
writeback error count on the upperdir, or it requires the failure injection 
infrastructure. 

Simulating this behaviour is non-trivial without in-kernel support:

P1: Open(f) -> p1.fd
P2: Open(f) -> p2.fd
P1: syncfs(p1.fd) -> errrno
P2: syncfs(p1.fd) -> 0 # This should return an error


[1]: https://elixir.bootlin.com/linux/latest/source/fs/sync.c#L175
[2]: https://lwn.net/Articles/722250/

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,
	URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 48986C3E8C5
	for <linux-unionfs@archiver.kernel.org>; Sat, 28 Nov 2020 02:17:37 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id ED7F5207CD
	for <linux-unionfs@archiver.kernel.org>; Sat, 28 Nov 2020 02:17:36 +0000 (UTC)
Authentication-Results: mail.kernel.org;
	dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="MBZfPJFd"
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1730301AbgK1CPk (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Fri, 27 Nov 2020 21:15:40 -0500
Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:50217 "EHLO
        us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK)
        by vger.kernel.org with ESMTP id S1731493AbgK1CBV (ORCPT
        <rfc822;linux-unionfs@vger.kernel.org>);
        Fri, 27 Nov 2020 21:01:21 -0500
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;
        s=mimecast20190719; t=1606528872;
        h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
         to:to:cc:cc:mime-version:mime-version:content-type:content-type:
         content-transfer-encoding:content-transfer-encoding:
         in-reply-to:in-reply-to:references:references;
        bh=R4Zi8l3ZKIxJeXLSAM0McyGUGdr5i/PPqj1bQcvcbGk=;
        b=MBZfPJFdg4hUyT3LxdlMjtmUNukJBpsIrpMdKMP5TjrC/n8QJPlPtwk7lyIHTpLlJLm77r
        YB1xlncUi3Mi9hBozHy+7vOvNeFUeUj+4LGVYd8HmBsJPBFWCRunPcyjmaNrf4zrlrlacE
        RILkEE9J9WRnvsXjSe20Ui9WoZpV9pw=
Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com
 [209.85.222.200]) (Using TLS) by relay.mimecast.com with ESMTP id
 us-mta-135-6a9UTadEPfa5N8IGBVW4GQ-1; Fri, 27 Nov 2020 21:01:10 -0500
X-MC-Unique: 6a9UTadEPfa5N8IGBVW4GQ-1
Received: by mail-qk1-f200.google.com with SMTP id r124so4757542qkd.8
        for <linux-unionfs@vger.kernel.org>; Fri, 27 Nov 2020 18:01:10 -0800 (PST)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to
         :references:user-agent:mime-version:content-transfer-encoding;
        bh=R4Zi8l3ZKIxJeXLSAM0McyGUGdr5i/PPqj1bQcvcbGk=;
        b=gO9KWwvP8wozSwAL5PIMFToa3TjGs44CUTRsLjXRFX0nW8iK6eX9Y4Zuw9e5gULeTR
         HTFRC64IXueh9Zyj5yvqtrwxEVtKLjrn+Y0jDk11zePuijMOVcIvAVcnk5Ee6moAR3Nq
         D5f5Rx/rQOGvrJpEe+dZ1RJ4DCPpu/WFtlxi+FSgP7H812lWqWyk60FZwTH3m5sLuA5d
         SLQlDsouIGrY/sODtC593YrckvoPAUItESCAVlOwOCdNnT91PwiHpi6m8+cRzrM7VxYd
         u0ETEnPWXCWKfjgTgH2rXch49v7GsRMxawh7g5Onjqzhi3z8Ju9sYM8xq3botAN7Cmf1
         hznQ==
X-Gm-Message-State: AOAM5301uPQ4iGzAYMc4xXfpqdDJAq8DI0zVqP74GJdoWITUWIO2ctpO
        Ncfu0LnwOAsGjCWO/tVAhdxriiDFyl9FnSN5R7s+Syzx8sRY9F4G5FeUEZ33ry97+vE84K9R2M8
        MERAfhYYsfZFoVpqz2yDDUYNs0A==
X-Received: by 2002:a0c:eb91:: with SMTP id x17mr11308170qvo.36.1606528869485;
        Fri, 27 Nov 2020 18:01:09 -0800 (PST)
X-Google-Smtp-Source: ABdhPJzKIQmG/r5SsENO5IxIz0Lu6BYtU25slmJWSehKcRJTMAzc06HGrwLCuYsD1mZENcrHTCLs3Q==
X-Received: by 2002:a0c:eb91:: with SMTP id x17mr11308150qvo.36.1606528869253;
        Fri, 27 Nov 2020 18:01:09 -0800 (PST)
Received: from tleilax.poochiereds.net (68-20-15-154.lightspeed.rlghnc.sbcglobal.net. [68.20.15.154])
        by smtp.gmail.com with ESMTPSA id n4sm7747503qkf.42.2020.11.27.18.01.08
        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
        Fri, 27 Nov 2020 18:01:08 -0800 (PST)
Message-ID: <1338a059d03db0e85cf3f3234fd33434a45606c6.camel@redhat.com>
Subject: Re: [PATCH v2 2/4] overlay: Document current outstanding
 shortcoming of volatile
From:   Jeff Layton <jlayton@redhat.com>
To:     Sargun Dhillon <sargun@sargun.me>,
        Amir Goldstein <amir73il@gmail.com>
Cc:     overlayfs <linux-unionfs@vger.kernel.org>,
        Miklos Szeredi <miklos@szeredi.hu>,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Vivek Goyal <vgoyal@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel <linux-fsdevel@vger.kernel.org>,
        David Howells <dhowells@redhat.com>
Date:   Fri, 27 Nov 2020 21:01:07 -0500
In-Reply-To: <20201127221154.GA23383@ircssh-2.c.rugged-nimbus-611.internal>
References: <20201127092058.15117-1-sargun@sargun.me>
         <20201127092058.15117-3-sargun@sargun.me>
         <CAOQ4uxgaLuLb+f6WCMvmKHNTELvcvN8C5_u=t5hhoGT8Op7QuQ@mail.gmail.com>
         <20201127221154.GA23383@ircssh-2.c.rugged-nimbus-611.internal>
Content-Type: text/plain; charset="UTF-8"
User-Agent: Evolution 3.38.1 (3.38.1-1.fc33) 
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org

On Fri, 2020-11-27 at 22:11 +0000, Sargun Dhillon wrote:
> On Fri, Nov 27, 2020 at 02:52:52PM +0200, Amir Goldstein wrote:
> > On Fri, Nov 27, 2020 at 11:21 AM Sargun Dhillon <sargun@sargun.me> wrote:
> > > 
> > > This documents behaviour that was discussed in a thread about the volatile
> > > feature. Specifically, how failures can go hidden from asynchronous writes
> > > (such as from mmap, or writes that are not immediately flushed to the
> > > filesystem). Although we pass through calls like msync, fallocate, and
> > > write, and will still return errors on those, it doesn't guarantee all
> > > kinds of errors will happen at those times, and thus may hide errors.
> > > 
> > > In the future, we can add error checking to all interactions with the
> > > upperdir, and pass through errseq_t from the upperdir on mappings,
> > > and other interactions with the filesystem[1].
> > > 
> > > [1]: https://lore.kernel.org/linux-unionfs/20201116045758.21774-1-sargun@sargun.me/T/#m7d501f375e031056efad626e471a1392dd3aad33
> > > 
> > > Signed-off-by: Sargun Dhillon <sargun@sargun.me>
> > > Cc: linux-fsdevel@vger.kernel.org
> > > Cc: linux-unionfs@vger.kernel.org
> > > Cc: Miklos Szeredi <miklos@szeredi.hu>
> > > Cc: Amir Goldstein <amir73il@gmail.com>
> > > Cc: Vivek Goyal <vgoyal@redhat.com>
> > > ---
> > > Â Documentation/filesystems/overlayfs.rst | 6 +++++-
> > > Â 1 file changed, 5 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/Documentation/filesystems/overlayfs.rst b/Documentation/filesystems/overlayfs.rst
> > > index 580ab9a0fe31..c6e30c1bc2f2 100644
> > > --- a/Documentation/filesystems/overlayfs.rst
> > > +++ b/Documentation/filesystems/overlayfs.rst
> > > @@ -570,7 +570,11 @@ Volatile mount
> > > Â This is enabled with the "volatile" mount option.  Volatile mounts are not
> > > Â guaranteed to survive a crash.  It is strongly recommended that volatile
> > > Â mounts are only used if data written to the overlay can be recreated
> > > -without significant effort.
> > > +without significant effort.  In addition to this, the sync family of syscalls
> > > +are not sufficient to determine whether a write failed as sync calls are
> > > +omitted.  For this reason, it is important that the filesystem used by the
> > > +upperdir handles failure in a fashion that's suitable for the user.  For
> > > +example, upon detecting a fault, ext4 can be configured to panic.
> > > 
> > 
> > Reading this now, I think I may have wrongly analysed the issue.
> > Specifically, when I wrote that the very minimum is to document the
> > issue, it was under the assumption that a proper fix is hard.
> > I think I was wrong and that the very minimum is to check for errseq
> > since mount on the fsync and syncfs calls.
> > 
> > Why? first of all because it is very very simple and goes a long way to
> > fix the broken contract with applications, not the contract about durability
> > obviously, but the contract about write+fsync+read expects to find the written
> > data (during the same mount era).
> > 
> > Second, because the sentence you added above is hard for users to understand
> > and out of context. If we properly handle the writeback error in fsync/syncfs,
> > then this sentence becomes irrelevant.
> > The fact that ext4 can lose data if application did not fsync after
> > write is true
> > for volatile as well as non-volatile and it is therefore not relevant
> > in the context
> > of overlayfs documentation at all.
> > 
> > Am I wrong saying that it is very very simple to fix?
> > Would you mind making that fix at the bottom of the patch series, so it can
> > be easily applied to stable kernels?
> > 
> > Thanks,
> > Amir.
> 
> I'm not sure it's so easy. In VFS, there are places where the superblock's 
> errseq is checked[1]. AFAIK, that's going to check "our" errseq, and not the 
> errseq of the real corresponding real file's superblock. One solution might be 
> as part of all these callbacks we set our errseq to the errseq of the filesystem 
> that the upperdir, and then rely on VFS's checking.
> 
> I'm having a hard time figuring out how to deal with the per-mapping based
> error tracking. It seems like this infrastructure was only partially completed
> by Jeff Layton[2]. I don't now how it's actually supposed to work right now,
> as not all of his patches landed.
> 

The patches in the series were all merged, but we ended up going with a
simpler solution [1] than the first series I posted. Instead of plumbing
the errseq_t handling down into sync_fs, we did it in the syscall
wrapper.

I think the tricky part here is that there is no struct file plumbed
into ->sync_fs, so you don't have an errseq_t cursor to work with by the
time that gets called.

What may be easiest is to just propagate the s_wb_err value from the
upper_sb to the overlayfs superblock in ovl_sync_fs(). That should get
called before the errseq_check_and_advance in the syncfs syscall wrapper
and should ensure that syncfs() calls against the overlayfs sb see any
errors that occurred on the upper_sb.

Something like this maybe? Totally untested of course. May also need to
think about how to ensure ordering between racing syncfs calls too
(don't really want the s_wb_err going "backward"):

----------------------------8<---------------------------------
$ git diff
diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
index 290983bcfbb3..d725705abdac 100644
--- a/fs/overlayfs/super.c
+++ b/fs/overlayfs/super.c
@@ -283,6 +283,9 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
        ret = sync_filesystem(upper_sb);
        up_read(&upper_sb->s_umount);
 
+       /* Propagate s_wb_err from upper_sb to overlayfs sb */
+       WRITE_ONCE(sb->s_wb_err, READ_ONCE(upper_sb->s_wb_err));
+
        return ret;
 }
----------------------------8<---------------------------------

[1]: https://www.spinics.net/lists/linux-api/msg41276.html

How about I split this into two patchsets? One, where I add the logic to copy
the errseq on callbacks to fsync from the upperdir to the ovl fs superblock,
and thus allowing VFS to bubble up errors, and the documentation. We can CC
stable on those because I think it has an effect that's universal across
all filesystems.

P.S. 
I notice you maintain overlay tests outside of the kernel. Unfortunately, I 
think for this kind of test, it requires in kernel code to artificially bump the 
writeback error count on the upperdir, or it requires the failure injection 
infrastructure. 

Simulating this behaviour is non-trivial without in-kernel support:

P1: Open(f) -> p1.fd
P2: Open(f) -> p2.fd
P1: syncfs(p1.fd) -> errrno
P2: syncfs(p1.fd) -> 0 # This should return an error


[1]: https://elixir.bootlin.com/linux/latest/source/fs/sync.c#L175
[2]: https://lwn.net/Articles/722250/


-- 
Jeff Layton <jlayton@redhat.com>


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-17.2 required=3.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,
	INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,
	USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 42952C64E90
	for <linux-unionfs@archiver.kernel.org>; Sat, 28 Nov 2020 17:57:38 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id EBD33246E7
	for <linux-unionfs@archiver.kernel.org>; Sat, 28 Nov 2020 17:57:37 +0000 (UTC)
Authentication-Results: mail.kernel.org;
	dkim=pass (1024-bit key) header.d=sargun.me header.i=@sargun.me header.b="YY6/W8Rn"
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1731388AbgK1Rzz (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Sat, 28 Nov 2020 12:55:55 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53486 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1732514AbgK1EwS (ORCPT
        <rfc822;linux-unionfs@vger.kernel.org>);
        Fri, 27 Nov 2020 23:52:18 -0500
Received: from mail-io1-xd42.google.com (mail-io1-xd42.google.com [IPv6:2607:f8b0:4864:20::d42])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B78B5C0613D2
        for <linux-unionfs@vger.kernel.org>; Fri, 27 Nov 2020 20:45:34 -0800 (PST)
Received: by mail-io1-xd42.google.com with SMTP id u21so6557351iol.12
        for <linux-unionfs@vger.kernel.org>; Fri, 27 Nov 2020 20:45:34 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=sargun.me; s=google;
        h=date:from:to:cc:subject:message-id:references:mime-version
         :content-disposition:content-transfer-encoding:in-reply-to
         :user-agent;
        bh=ddBc/M8WFLLO8uTguRig7EUKcBKEUJZOaUkItrZRsxE=;
        b=YY6/W8RniG3sJcPruk+A7qMyMEN1CYCA9aHpMoUnXOD/C3co0iD5R02PPEALa051yk
         LD+nJV1XsPwQJUO6RwynxFkxlv1zxDEqVS9qMYmIrLRNdtzKC2sIMUwc9Q5xTGNdDGZT
         nlYe7bbRqDydzHQYV/aSa84ELDoUp3eyps/NY=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:date:from:to:cc:subject:message-id:references
         :mime-version:content-disposition:content-transfer-encoding
         :in-reply-to:user-agent;
        bh=ddBc/M8WFLLO8uTguRig7EUKcBKEUJZOaUkItrZRsxE=;
        b=BfoioQ3WzQYrl/ztL6GMh8TN1EjWL9gmi5FsrZd4RsFcJ82sQeJe1r61Ms8PVLEOvt
         LvlwSra9VYjtHxJX6mqxSRl5dJ39ps0w1YC6IHtEzxKOIDVANbx2F8Xq1WxdjOipxnBF
         F/Z87dmUMujYe6aToHcuKhUsB9hI1TXk9e1jjsxxmOswI9DxX/Swk5I5l2ksLkeeiaMY
         PVYp2ZXsuqntDKslP1SHTZG0e7UsdH/ZlgaF54QLfpD7vSjufwPFiyEQ7n2/wlGoXoMb
         S3qp8ACmxWhS6Tm90WdnExdxNZ6xSJ4kgbqvxevHjXtuV3yivf7E4QQdExBPKDtTG1k/
         uzQg==
X-Gm-Message-State: AOAM533RucjFzDaQ+Av9vLY8AtGt5NOxzBha6MM2JdxW4O4s8Z2jm23K
        fJorwOJceAHk59dapA9H6VIk6g==
X-Google-Smtp-Source: ABdhPJwLJXa7sAsZwe0onSLDOFf/awoe2qHJF9iIQNxEmELEgS363ybLrdMZwuBP3XcL2A4uU0t4Cg==
X-Received: by 2002:a02:4:: with SMTP id 4mr10647298jaa.121.1606538733902;
        Fri, 27 Nov 2020 20:45:33 -0800 (PST)
Received: from ircssh-2.c.rugged-nimbus-611.internal (80.60.198.104.bc.googleusercontent.com. [104.198.60.80])
        by smtp.gmail.com with ESMTPSA id d5sm4671748ios.25.2020.11.27.20.45.33
        (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256);
        Fri, 27 Nov 2020 20:45:33 -0800 (PST)
Date:   Sat, 28 Nov 2020 04:45:31 +0000
From:   Sargun Dhillon <sargun@sargun.me>
To:     Jeff Layton <jlayton@redhat.com>
Cc:     Amir Goldstein <amir73il@gmail.com>,
        overlayfs <linux-unionfs@vger.kernel.org>,
        Miklos Szeredi <miklos@szeredi.hu>,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Vivek Goyal <vgoyal@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel <linux-fsdevel@vger.kernel.org>,
        David Howells <dhowells@redhat.com>
Subject: Re: [PATCH v2 2/4] overlay: Document current outstanding shortcoming
 of volatile
Message-ID: <20201128044530.GA28230@ircssh-2.c.rugged-nimbus-611.internal>
References: <20201127092058.15117-1-sargun@sargun.me>
 <20201127092058.15117-3-sargun@sargun.me>
 <CAOQ4uxgaLuLb+f6WCMvmKHNTELvcvN8C5_u=t5hhoGT8Op7QuQ@mail.gmail.com>
 <20201127221154.GA23383@ircssh-2.c.rugged-nimbus-611.internal>
 <1338a059d03db0e85cf3f3234fd33434a45606c6.camel@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <1338a059d03db0e85cf3f3234fd33434a45606c6.camel@redhat.com>
User-Agent: Mutt/1.9.4 (2018-02-28)
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org

On Fri, Nov 27, 2020 at 09:01:07PM -0500, Jeff Layton wrote:
> On Fri, 2020-11-27 at 22:11 +0000, Sargun Dhillon wrote:
> > On Fri, Nov 27, 2020 at 02:52:52PM +0200, Amir Goldstein wrote:
> > > On Fri, Nov 27, 2020 at 11:21 AM Sargun Dhillon <sargun@sargun.me> wrote:
> > > > 
> > > > This documents behaviour that was discussed in a thread about the volatile
> > > > feature. Specifically, how failures can go hidden from asynchronous writes
> > > > (such as from mmap, or writes that are not immediately flushed to the
> > > > filesystem). Although we pass through calls like msync, fallocate, and
> > > > write, and will still return errors on those, it doesn't guarantee all
> > > > kinds of errors will happen at those times, and thus may hide errors.
> > > > 
> > > > In the future, we can add error checking to all interactions with the
> > > > upperdir, and pass through errseq_t from the upperdir on mappings,
> > > > and other interactions with the filesystem[1].
> > > > 
> > > > [1]: https://lore.kernel.org/linux-unionfs/20201116045758.21774-1-sargun@sargun.me/T/#m7d501f375e031056efad626e471a1392dd3aad33
> > > > 
> > > > Signed-off-by: Sargun Dhillon <sargun@sargun.me>
> > > > Cc: linux-fsdevel@vger.kernel.org
> > > > Cc: linux-unionfs@vger.kernel.org
> > > > Cc: Miklos Szeredi <miklos@szeredi.hu>
> > > > Cc: Amir Goldstein <amir73il@gmail.com>
> > > > Cc: Vivek Goyal <vgoyal@redhat.com>
> > > > ---
> > > >  Documentation/filesystems/overlayfs.rst | 6 +++++-
> > > >  1 file changed, 5 insertions(+), 1 deletion(-)
> > > > 
> > > > diff --git a/Documentation/filesystems/overlayfs.rst b/Documentation/filesystems/overlayfs.rst
> > > > index 580ab9a0fe31..c6e30c1bc2f2 100644
> > > > --- a/Documentation/filesystems/overlayfs.rst
> > > > +++ b/Documentation/filesystems/overlayfs.rst
> > > > @@ -570,7 +570,11 @@ Volatile mount
> > > >  This is enabled with the "volatile" mount option.  Volatile mounts are not
> > > >  guaranteed to survive a crash.  It is strongly recommended that volatile
> > > >  mounts are only used if data written to the overlay can be recreated
> > > > -without significant effort.
> > > > +without significant effort.  In addition to this, the sync family of syscalls
> > > > +are not sufficient to determine whether a write failed as sync calls are
> > > > +omitted.  For this reason, it is important that the filesystem used by the
> > > > +upperdir handles failure in a fashion that's suitable for the user.  For
> > > > +example, upon detecting a fault, ext4 can be configured to panic.
> > > > 
> > > 
> > > Reading this now, I think I may have wrongly analysed the issue.
> > > Specifically, when I wrote that the very minimum is to document the
> > > issue, it was under the assumption that a proper fix is hard.
> > > I think I was wrong and that the very minimum is to check for errseq
> > > since mount on the fsync and syncfs calls.
> > > 
> > > Why? first of all because it is very very simple and goes a long way to
> > > fix the broken contract with applications, not the contract about durability
> > > obviously, but the contract about write+fsync+read expects to find the written
> > > data (during the same mount era).
> > > 
> > > Second, because the sentence you added above is hard for users to understand
> > > and out of context. If we properly handle the writeback error in fsync/syncfs,
> > > then this sentence becomes irrelevant.
> > > The fact that ext4 can lose data if application did not fsync after
> > > write is true
> > > for volatile as well as non-volatile and it is therefore not relevant
> > > in the context
> > > of overlayfs documentation at all.
> > > 
> > > Am I wrong saying that it is very very simple to fix?
> > > Would you mind making that fix at the bottom of the patch series, so it can
> > > be easily applied to stable kernels?
> > > 
> > > Thanks,
> > > Amir.
> > 
> > I'm not sure it's so easy. In VFS, there are places where the superblock's 
> > errseq is checked[1]. AFAIK, that's going to check "our" errseq, and not the 
> > errseq of the real corresponding real file's superblock. One solution might be 
> > as part of all these callbacks we set our errseq to the errseq of the filesystem 
> > that the upperdir, and then rely on VFS's checking.
> > 
> > I'm having a hard time figuring out how to deal with the per-mapping based
> > error tracking. It seems like this infrastructure was only partially completed
> > by Jeff Layton[2]. I don't now how it's actually supposed to work right now,
> > as not all of his patches landed.
> > 
> 
> The patches in the series were all merged, but we ended up going with a
> simpler solution [1] than the first series I posted. Instead of plumbing
> the errseq_t handling down into sync_fs, we did it in the syscall
> wrapper.
Jeff,

Thanks for replying. I'm still a little confused as to what the 
per-address_space wb_err. It seems like we should implement the
flush method, like so:
diff --git a/fs/overlayfs/file.c b/fs/overlayfs/file.c
index efccb7c1f9bc..32e5bc0aacd6 100644
--- a/fs/overlayfs/file.c
+++ b/fs/overlayfs/file.c
@@ -787,6 +787,16 @@ static loff_t ovl_remap_file_range(struct file *file_in, loff_t pos_in,
                            remap_flags, op);
 }
 
+static int ovl_flush(struct file *file, fl_owner_t id)
+{
+       struct file *realfile = file->private_data;
+
+       if (real_file->f_op->flush)
+               return real_file->f_op->flush(filp, id);
+
+       return 0;
+}
+
 const struct file_operations ovl_file_operations = {
        .open           = ovl_open,
        .release        = ovl_release,
@@ -798,6 +808,7 @@ const struct file_operations ovl_file_operations = {
        .fallocate      = ovl_fallocate,
        .fadvise        = ovl_fadvise,
        .unlocked_ioctl = ovl_ioctl,
+       .flush          = ovl_flush,
 #ifdef CONFIG_COMPAT
        .compat_ioctl   = ovl_compat_ioctl,
 #endif


> 
> I think the tricky part here is that there is no struct file plumbed
> into ->sync_fs, so you don't have an errseq_t cursor to work with by the
> time that gets called.
> 
> What may be easiest is to just propagate the s_wb_err value from the
> upper_sb to the overlayfs superblock in ovl_sync_fs(). That should get
> called before the errseq_check_and_advance in the syncfs syscall wrapper
> and should ensure that syncfs() calls against the overlayfs sb see any
> errors that occurred on the upper_sb.
> 
> Something like this maybe? Totally untested of course. May also need to
> think about how to ensure ordering between racing syncfs calls too
> (don't really want the s_wb_err going "backward"):
> 
> ----------------------------8<---------------------------------
> $ git diff
> diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
> index 290983bcfbb3..d725705abdac 100644
> --- a/fs/overlayfs/super.c
> +++ b/fs/overlayfs/super.c
> @@ -283,6 +283,9 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
>         ret = sync_filesystem(upper_sb);
>         up_read(&upper_sb->s_umount);
>  
> +       /* Propagate s_wb_err from upper_sb to overlayfs sb */
> +       WRITE_ONCE(sb->s_wb_err, READ_ONCE(upper_sb->s_wb_err));
> +
>         return ret;
>  }
> ----------------------------8<---------------------------------
> 
> [1]: https://www.spinics.net/lists/linux-api/msg41276.html

So, 
I think this will have bad behaviour because syncfs() twice in a row will 
return the error twice. The reason is that normally in syncfs it calls 
errseq_check_and_advance marking the error on the super block as seen. If we 
copy-up the error value each time, it will break this semantic, as we do not set 
seen on the upperdir.

Either we need to set the seen flag on the upperdir's errseq_t, or have a sync 
method, like:
diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
index 290983bcfbb3..4931d1797c03 100644
--- a/fs/overlayfs/super.c
+++ b/fs/overlayfs/super.c
@@ -259,6 +259,7 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
 {
        struct ovl_fs *ofs = sb->s_fs_info;
        struct super_block *upper_sb;
+       errseq_t src;
        int ret;
 
        if (!ovl_upper_mnt(ofs))
@@ -283,6 +284,11 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
        ret = sync_filesystem(upper_sb);
        up_read(&upper_sb->s_umount);
 
+       /* Propagate the error up from the upper_sb once */
+       src = READ_ONCE(upper_sb->s_wb_err);
+       if (errseq_counter(src) != errseq_counter(sb->s_wb_err))
+               WRITE_ONCE(sb->s_wb_err, src & ~ERRSEQ_SEEN);
+
        return ret;
 }
 
@@ -1945,6 +1951,7 @@ static int ovl_fill_super(struct super_block *sb, void *data, int silent)
 
                sb->s_stack_depth = ovl_upper_mnt(ofs)->mnt_sb->s_stack_depth;
                sb->s_time_gran = ovl_upper_mnt(ofs)->mnt_sb->s_time_gran;
+               sb->s_wb_err = READ_ONCE(ovl_upper_mnt(ofs)->mnt_sb->s_wb_err);
 
diff --git a/lib/errseq.c b/lib/errseq.c
index 81f9e33aa7e7..53275c168265 100644
--- a/lib/errseq.c
+++ b/lib/errseq.c
@@ -204,3 +204,18 @@ int errseq_check_and_advance(errseq_t *eseq, errseq_t *since)
        return err;
 }
 EXPORT_SYMBOL(errseq_check_and_advance);
+
+/**
+ * errseq_counter() - Get the value of the errseq counter
+ * @eseq: Value being checked
+ *
+ * This returns the errseq_t counter value. Reminder, it can wrap because
+ * there are only a limited number of counter bits.
+ *
+ * Return: The counter portion of the errseq_t.
+ */
+int errseq_counter(errseq_t eseq)
+{
+       return eseq >> (ERRSEQ_SHIFT + 1);
+}

This also needs some locking sprinkled on it because racing can occur with 
sync_fs. This would be much easier if the errseq_t was 64-bits, and didn't go 
backwards, because you could just use a simple cmpxchg64. I know that would make 
a bunch of structures larger, and if it's just overlayfs that has to pay the tax
we can just sprinkle a mutex on it.

We can always "copy-up" the errseq_t because if the upperdir's errseq_t is read, 
and we haven't done a copy yet, we'll get it. Since this will be strictly 
serialized a simple equivalence check will work versus actually having to deal 
with happens before behaviour. There is still a correctness flaw here though, 
which is exactly enough errors occur to result in it wrapping back to the same 
value, it will break.

By the way, how do y'all test this error handling behaviour? I didn't
find any automated testing for what currently exists.
> 
> How about I split this into two patchsets? One, where I add the logic to copy
> the errseq on callbacks to fsync from the upperdir to the ovl fs superblock,
> and thus allowing VFS to bubble up errors, and the documentation. We can CC
> stable on those because I think it has an effect that's universal across
> all filesystems.
> 
> P.S. 
> I notice you maintain overlay tests outside of the kernel. Unfortunately, I 
> think for this kind of test, it requires in kernel code to artificially bump the 
> writeback error count on the upperdir, or it requires the failure injection 
> infrastructure. 
> 
> Simulating this behaviour is non-trivial without in-kernel support:
> 
> P1: Open(f) -> p1.fd
> P2: Open(f) -> p2.fd
> P1: syncfs(p1.fd) -> errrno
> P2: syncfs(p1.fd) -> 0 # This should return an error
> 
> 
> [1]: https://elixir.bootlin.com/linux/latest/source/fs/sync.c#L175
> [2]: https://lwn.net/Articles/722250/
> 
> 
> -- 
> Jeff Layton <jlayton@redhat.com>
> 

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-2.7 required=3.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,
	HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS
	autolearn=no autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 26243C83025
	for <linux-unionfs@archiver.kernel.org>; Sat, 28 Nov 2020 22:06:04 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id DAAF522470
	for <linux-unionfs@archiver.kernel.org>; Sat, 28 Nov 2020 22:06:03 +0000 (UTC)
Authentication-Results: mail.kernel.org;
	dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="px9OARAc"
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1732303AbgK1VuV (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Sat, 28 Nov 2020 16:50:21 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43526 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1732431AbgK1TAU (ORCPT
        <rfc822;linux-unionfs@vger.kernel.org>);
        Sat, 28 Nov 2020 14:00:20 -0500
Received: from mail-io1-xd41.google.com (mail-io1-xd41.google.com [IPv6:2607:f8b0:4864:20::d41])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 38340C02B8F1;
        Sat, 28 Nov 2020 01:06:38 -0800 (PST)
Received: by mail-io1-xd41.google.com with SMTP id i9so6921276ioo.2;
        Sat, 28 Nov 2020 01:06:38 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20161025;
        h=mime-version:references:in-reply-to:from:date:message-id:subject:to
         :cc;
        bh=MIIZAXDzKwop2fyL+ugRMktDfmpGkDPbQUZJlWUas5k=;
        b=px9OARAcD/eF7z6PpWzwqPpkkPggWWfPEfc72C7hyzAFdeQw4CeKWDVVmrtKvqEJai
         5ZbNKSNgNutfj3oh5/ACoqmW/Gfoi7hnUTDennY1TCOWVlDHplv/fP56s/duH9SzJ5k1
         hnFmYEUGw805ay4cdwO77s0yTMXgmYQzsdMHA/IXMLHI5sz2K2SsaNClsKRNXWXHhmtz
         q/m4X+1dBfXFyPgtrfqQe+/fSs2UrrPTMXposgGaW5MRSgBl9zjq+sDfCIr2Jkahy9Ba
         Qs4rfnpDMGZ1FOgqqTARI59GihW73OosE+45EfmbzcdQYzxRhDQD3ZQagMyaF8LxXhSN
         lu7A==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:mime-version:references:in-reply-to:from:date
         :message-id:subject:to:cc;
        bh=MIIZAXDzKwop2fyL+ugRMktDfmpGkDPbQUZJlWUas5k=;
        b=I1vOzydsny50wnBlWs89bP/mVA3icGRQWFZtG3Ai+P9MGYei0iay/sJRTFW7iWsEhP
         Pl07F0qbzXJYISyfeUZtlkBR/xybK6xr2KqtXPFKeCBqgpGnvLEeZs7qaZ5VStc1hEgZ
         t90aI3y9L2AYJ6eo3zqEctxSIlCPzDk/ezvkEjfPuiDnGyEs/Ln82GvL5ox3qJTouJLf
         te61DKWMSif4G0swWo1/dR4DnNr8naGh7xqdinwk/oaQZvw/r/3zG8ZiuTTjhzpnc8KK
         Zp1qxWaQ0d+8T4z/9OgPvnbJ4yvgpOPDLtAWXyGeeKE4CE85i623XdMV2EEn5T5kfDUb
         Or/g==
X-Gm-Message-State: AOAM533xE9m23+lWJEwai5BFTv/X5Tja5UFG+/Mc7PnrwE5FbVDsZhmR
        aN2jci4sCc80oR2P43Imp+lbbMlujiJ0jD30570=
X-Google-Smtp-Source: ABdhPJwzR7VXTRrZRR5BOdY7EvDKmkHZ5b/9fOg5BD5byoj8XxyWRZnQeAnjuuso57zt/vXIZPKgHBmMaylTyhaVFNQ=
X-Received: by 2002:a02:cc89:: with SMTP id s9mr4242657jap.81.1606554397527;
 Sat, 28 Nov 2020 01:06:37 -0800 (PST)
MIME-Version: 1.0
References: <20201127092058.15117-1-sargun@sargun.me> <20201127092058.15117-3-sargun@sargun.me>
 <CAOQ4uxgaLuLb+f6WCMvmKHNTELvcvN8C5_u=t5hhoGT8Op7QuQ@mail.gmail.com>
 <20201127221154.GA23383@ircssh-2.c.rugged-nimbus-611.internal> <CAOQ4uxiLRy9ioqaqtOp7P6hLy8Gx5vRO86mie7FAdTu2OfnGrw@mail.gmail.com>
In-Reply-To: <CAOQ4uxiLRy9ioqaqtOp7P6hLy8Gx5vRO86mie7FAdTu2OfnGrw@mail.gmail.com>
From:   Amir Goldstein <amir73il@gmail.com>
Date:   Sat, 28 Nov 2020 11:06:26 +0200
Message-ID: <CAOQ4uxhra_RB98gJ7ovGhbUV1atCR1rMPnf63tT37WtrNC0asg@mail.gmail.com>
Subject: Re: [PATCH v2 2/4] overlay: Document current outstanding shortcoming
 of volatile
To:     Sargun Dhillon <sargun@sargun.me>
Cc:     overlayfs <linux-unionfs@vger.kernel.org>,
        Miklos Szeredi <miklos@szeredi.hu>,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Vivek Goyal <vgoyal@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel <linux-fsdevel@vger.kernel.org>,
        David Howells <dhowells@redhat.com>,
        Jeff Layton <jlayton@redhat.com>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org

On Sat, Nov 28, 2020 at 10:56 AM Amir Goldstein <amir73il@gmail.com> wrote:
>
> > I notice you maintain overlay tests outside of the kernel. Unfortunately, I
> > think for this kind of test, it requires in kernel code to artificially bump the
> > writeback error count on the upperdir, or it requires the failure injection
> > infrastructure.
> >
> > Simulating this behaviour is non-trivial without in-kernel support:
> >
> > P1: Open(f) -> p1.fd
> > P2: Open(f) -> p2.fd
> > P1: syncfs(p1.fd) -> errrno
> > P2: syncfs(p1.fd) -> 0 # This should return an error
> >
>
> failure injection is an option. xfstest generic/019 is an example.
> generic/108 uses a different method and generic/361 uses a plain
> loop image over commit without any fault injection to trigger writeback
> errors.
>
> With current xfstests, check -overlay run (runs all generic tests with
> overlayfs over the configured base fs) all the 3 tests mentioned above
> will be skipped because of _require_block_device, but the requirement
> is there for a different reason for each of them.
>
> At first look, the loop device approach is the most generic one and could
> easily work also with overlayfs, so you could create an overlay specific
> test (non generic) based on generic/361, but it is not easy to use the
> helper _scratch_mkfs_sized, because check -overlay runs do not mkfs
> the base scratch fs.
>
> My recommendation would be to fix generic/019 in a similar manner
> to the way that tests that _require_scratch_shutdown were fixed to run
> with check -overlay:
>
> * Instead of _require_block_device, add a specific helper
>    _require_scratch_fail_make_request, which like _require_scratch_shutdown
>    knows how to deal with overlay FSTYP correctly
>
> * Instead of `_sysfs_dev $SCRATCH_DEV` use a helper _scratch_sysfs_dev
>    that knows how to deal with overlay FSTYP correctly
>

I missed:

* Instead of `blockdev --getsz $SCRATCH_DEV` use helper _scratch_blockdev_getsz

> This will add test coverage to overlayfs fsync/syncfs and then you can
> do one or both of:
> 1. Run 'check -overlay generic/019' with  OVERLAY_MOUNT_OPTIONS="volatile"
> 2. Fork an overlay specific test from the generic test that will test the
>     volatile error handling on every 'check -overlay -g quick' run
>
> #2 will provide better coverage against regressions in volatile writeback error
> handling and will be a good start for a test to reuse a volatile mount after
> writeback errors.
>
> Thanks,
> Amir.

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,
	HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham
	autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 079AAC83024
	for <linux-unionfs@archiver.kernel.org>; Sat, 28 Nov 2020 22:06:04 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id A3FD12344C
	for <linux-unionfs@archiver.kernel.org>; Sat, 28 Nov 2020 22:06:03 +0000 (UTC)
Authentication-Results: mail.kernel.org;
	dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="RZ4fSovN"
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1732431AbgK1VuV (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Sat, 28 Nov 2020 16:50:21 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44198 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1732864AbgK1TEa (ORCPT
        <rfc822;linux-unionfs@vger.kernel.org>);
        Sat, 28 Nov 2020 14:04:30 -0500
Received: from mail-io1-xd42.google.com (mail-io1-xd42.google.com [IPv6:2607:f8b0:4864:20::d42])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5AEBAC094257;
        Fri, 27 Nov 2020 23:12:15 -0800 (PST)
Received: by mail-io1-xd42.google.com with SMTP id n14so1954568iom.10;
        Fri, 27 Nov 2020 23:12:15 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20161025;
        h=mime-version:references:in-reply-to:from:date:message-id:subject:to
         :cc;
        bh=ceCBPQVXMQ3ohtM1wHnupz/6cAbrRR+fiYn+1MH3RLY=;
        b=RZ4fSovNSoHE9o69TixkeyVutYn0W93XdmQuuA2W7QGQk5IU08pUrNHnA+fz+p06/H
         5+jcBIr421n1PStT4ZHAO5y+AIEKTJsAsXwSY1m1spKBsq/N6EkWJJWlS/U0tfXXFEs+
         MMXROhsXdu3kjd3pMOcCZReg0HqiuefQBRNwYd5NNUqQ24l2WZ9fIN4rVBi94ncmG5D/
         ++9DxPISLTK7t+vao8+2R1TPpQnrEp2RXYNA5OyAJE2S6dGqe4lbrD6odcqO/5qIIIFP
         WyUzB2ZyEVi1HL/zmR+9mPDKrRc49A43k+usVl5eUqjMv4QjHu53kHub1QQFNz/Et39U
         iSPQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:mime-version:references:in-reply-to:from:date
         :message-id:subject:to:cc;
        bh=ceCBPQVXMQ3ohtM1wHnupz/6cAbrRR+fiYn+1MH3RLY=;
        b=O1v2oU7LN5RFpvflkHQR9rdVeLGX56nsSHkpeqOumHk2+i1Rgyl/u9wTeMe91TP4jh
         wK1r3EZ0srbKrOYvKLtn2gs60+gSz4eC8H/ggBjFZSDkW3Zd2lRXAvobGs8DAEFDQF9a
         IhyNjuWjZwgxm6U1ZijRTXNDZ5IEwx0B8EsiZRwvl7/PwzuTZgLQVXXzf2ZWLKI/l4P1
         mjWQUlO32+pfHr/LnYTZmJaISYUOWI7sLSFxnGVfByl23841MGE6G+E/ZDZlbqSq3KQd
         GuWeXM4JvFCM9VBwnTmYVDlTViGNZinekDI6QS3sZIHdcUY5JT2NDWkApcUwdyv18orK
         sBig==
X-Gm-Message-State: AOAM530etzJ1PIai7QFFV0la441EIbWRTSxaZNDKSnbpeCkyZz0ErFpZ
        qcVNj3TKFgBntUPgazyx6Dtctv5Nn4Vec8Zf/BA=
X-Google-Smtp-Source: ABdhPJyisuw334cTjrWuNl7XexclzbFprBgSlUC2xTeYVPUdVrLzwmtUGy26NGnT/wo31LH7BoGIW9v9yRqWgLONzO0=
X-Received: by 2002:a5d:964a:: with SMTP id d10mr8901322ios.5.1606547534373;
 Fri, 27 Nov 2020 23:12:14 -0800 (PST)
MIME-Version: 1.0
References: <20201127092058.15117-1-sargun@sargun.me> <20201127092058.15117-3-sargun@sargun.me>
 <CAOQ4uxgaLuLb+f6WCMvmKHNTELvcvN8C5_u=t5hhoGT8Op7QuQ@mail.gmail.com>
 <20201127221154.GA23383@ircssh-2.c.rugged-nimbus-611.internal>
 <1338a059d03db0e85cf3f3234fd33434a45606c6.camel@redhat.com> <20201128044530.GA28230@ircssh-2.c.rugged-nimbus-611.internal>
In-Reply-To: <20201128044530.GA28230@ircssh-2.c.rugged-nimbus-611.internal>
From:   Amir Goldstein <amir73il@gmail.com>
Date:   Sat, 28 Nov 2020 09:12:03 +0200
Message-ID: <CAOQ4uxjT6FF03Sq3qXuqDcqJQnzQq2dD_XVbuj_Fb9A2Ag585w@mail.gmail.com>
Subject: Re: [PATCH v2 2/4] overlay: Document current outstanding shortcoming
 of volatile
To:     Sargun Dhillon <sargun@sargun.me>
Cc:     Jeff Layton <jlayton@redhat.com>,
        overlayfs <linux-unionfs@vger.kernel.org>,
        Miklos Szeredi <miklos@szeredi.hu>,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Vivek Goyal <vgoyal@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel <linux-fsdevel@vger.kernel.org>,
        David Howells <dhowells@redhat.com>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org

On Sat, Nov 28, 2020 at 6:45 AM Sargun Dhillon <sargun@sargun.me> wrote:
>
> On Fri, Nov 27, 2020 at 09:01:07PM -0500, Jeff Layton wrote:
> > On Fri, 2020-11-27 at 22:11 +0000, Sargun Dhillon wrote:
> > > On Fri, Nov 27, 2020 at 02:52:52PM +0200, Amir Goldstein wrote:
> > > > On Fri, Nov 27, 2020 at 11:21 AM Sargun Dhillon <sargun@sargun.me> wrote:
> > > > >
> > > > > This documents behaviour that was discussed in a thread about the volatile
> > > > > feature. Specifically, how failures can go hidden from asynchronous writes
> > > > > (such as from mmap, or writes that are not immediately flushed to the
> > > > > filesystem). Although we pass through calls like msync, fallocate, and
> > > > > write, and will still return errors on those, it doesn't guarantee all
> > > > > kinds of errors will happen at those times, and thus may hide errors.
> > > > >
> > > > > In the future, we can add error checking to all interactions with the
> > > > > upperdir, and pass through errseq_t from the upperdir on mappings,
> > > > > and other interactions with the filesystem[1].
> > > > >
> > > > > [1]: https://lore.kernel.org/linux-unionfs/20201116045758.21774-1-sargun@sargun.me/T/#m7d501f375e031056efad626e471a1392dd3aad33
> > > > >
> > > > > Signed-off-by: Sargun Dhillon <sargun@sargun.me>
> > > > > Cc: linux-fsdevel@vger.kernel.org
> > > > > Cc: linux-unionfs@vger.kernel.org
> > > > > Cc: Miklos Szeredi <miklos@szeredi.hu>
> > > > > Cc: Amir Goldstein <amir73il@gmail.com>
> > > > > Cc: Vivek Goyal <vgoyal@redhat.com>
> > > > > ---
> > > > >  Documentation/filesystems/overlayfs.rst | 6 +++++-
> > > > >  1 file changed, 5 insertions(+), 1 deletion(-)
> > > > >
> > > > > diff --git a/Documentation/filesystems/overlayfs.rst b/Documentation/filesystems/overlayfs.rst
> > > > > index 580ab9a0fe31..c6e30c1bc2f2 100644
> > > > > --- a/Documentation/filesystems/overlayfs.rst
> > > > > +++ b/Documentation/filesystems/overlayfs.rst
> > > > > @@ -570,7 +570,11 @@ Volatile mount
> > > > >  This is enabled with the "volatile" mount option.  Volatile mounts are not
> > > > >  guaranteed to survive a crash.  It is strongly recommended that volatile
> > > > >  mounts are only used if data written to the overlay can be recreated
> > > > > -without significant effort.
> > > > > +without significant effort.  In addition to this, the sync family of syscalls
> > > > > +are not sufficient to determine whether a write failed as sync calls are
> > > > > +omitted.  For this reason, it is important that the filesystem used by the
> > > > > +upperdir handles failure in a fashion that's suitable for the user.  For
> > > > > +example, upon detecting a fault, ext4 can be configured to panic.
> > > > >
> > > >
> > > > Reading this now, I think I may have wrongly analysed the issue.
> > > > Specifically, when I wrote that the very minimum is to document the
> > > > issue, it was under the assumption that a proper fix is hard.
> > > > I think I was wrong and that the very minimum is to check for errseq
> > > > since mount on the fsync and syncfs calls.
> > > >
> > > > Why? first of all because it is very very simple and goes a long way to
> > > > fix the broken contract with applications, not the contract about durability
> > > > obviously, but the contract about write+fsync+read expects to find the written
> > > > data (during the same mount era).
> > > >
> > > > Second, because the sentence you added above is hard for users to understand
> > > > and out of context. If we properly handle the writeback error in fsync/syncfs,
> > > > then this sentence becomes irrelevant.
> > > > The fact that ext4 can lose data if application did not fsync after
> > > > write is true
> > > > for volatile as well as non-volatile and it is therefore not relevant
> > > > in the context
> > > > of overlayfs documentation at all.
> > > >
> > > > Am I wrong saying that it is very very simple to fix?
> > > > Would you mind making that fix at the bottom of the patch series, so it can
> > > > be easily applied to stable kernels?
> > > >
> > > > Thanks,
> > > > Amir.
> > >
> > > I'm not sure it's so easy. In VFS, there are places where the superblock's
> > > errseq is checked[1]. AFAIK, that's going to check "our" errseq, and not the
> > > errseq of the real corresponding real file's superblock. One solution might be
> > > as part of all these callbacks we set our errseq to the errseq of the filesystem
> > > that the upperdir, and then rely on VFS's checking.
> > >
> > > I'm having a hard time figuring out how to deal with the per-mapping based
> > > error tracking. It seems like this infrastructure was only partially completed
> > > by Jeff Layton[2]. I don't now how it's actually supposed to work right now,
> > > as not all of his patches landed.
> > >
> >
> > The patches in the series were all merged, but we ended up going with a
> > simpler solution [1] than the first series I posted. Instead of plumbing
> > the errseq_t handling down into sync_fs, we did it in the syscall
> > wrapper.
> Jeff,
>
> Thanks for replying. I'm still a little confused as to what the
> per-address_space wb_err. It seems like we should implement the
> flush method, like so:
> diff --git a/fs/overlayfs/file.c b/fs/overlayfs/file.c
> index efccb7c1f9bc..32e5bc0aacd6 100644
> --- a/fs/overlayfs/file.c
> +++ b/fs/overlayfs/file.c
> @@ -787,6 +787,16 @@ static loff_t ovl_remap_file_range(struct file *file_in, loff_t pos_in,
>                             remap_flags, op);
>  }
>
> +static int ovl_flush(struct file *file, fl_owner_t id)
> +{
> +       struct file *realfile = file->private_data;
> +
> +       if (real_file->f_op->flush)
> +               return real_file->f_op->flush(filp, id);
> +
> +       return 0;
> +}
> +
>  const struct file_operations ovl_file_operations = {
>         .open           = ovl_open,
>         .release        = ovl_release,
> @@ -798,6 +808,7 @@ const struct file_operations ovl_file_operations = {
>         .fallocate      = ovl_fallocate,
>         .fadvise        = ovl_fadvise,
>         .unlocked_ioctl = ovl_ioctl,
> +       .flush          = ovl_flush,
>  #ifdef CONFIG_COMPAT
>         .compat_ioctl   = ovl_compat_ioctl,
>  #endif
>
>
> >
> > I think the tricky part here is that there is no struct file plumbed
> > into ->sync_fs, so you don't have an errseq_t cursor to work with by the
> > time that gets called.
> >
> > What may be easiest is to just propagate the s_wb_err value from the
> > upper_sb to the overlayfs superblock in ovl_sync_fs(). That should get
> > called before the errseq_check_and_advance in the syncfs syscall wrapper
> > and should ensure that syncfs() calls against the overlayfs sb see any
> > errors that occurred on the upper_sb.
> >
> > Something like this maybe? Totally untested of course. May also need to
> > think about how to ensure ordering between racing syncfs calls too
> > (don't really want the s_wb_err going "backward"):
> >
> > ----------------------------8<---------------------------------
> > $ git diff
> > diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
> > index 290983bcfbb3..d725705abdac 100644
> > --- a/fs/overlayfs/super.c
> > +++ b/fs/overlayfs/super.c
> > @@ -283,6 +283,9 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
> >         ret = sync_filesystem(upper_sb);
> >         up_read(&upper_sb->s_umount);
> >
> > +       /* Propagate s_wb_err from upper_sb to overlayfs sb */
> > +       WRITE_ONCE(sb->s_wb_err, READ_ONCE(upper_sb->s_wb_err));
> > +
> >         return ret;
> >  }
> > ----------------------------8<---------------------------------
> >
> > [1]: https://www.spinics.net/lists/linux-api/msg41276.html
>
> So,
> I think this will have bad behaviour because syncfs() twice in a row will
> return the error twice. The reason is that normally in syncfs it calls
> errseq_check_and_advance marking the error on the super block as seen. If we
> copy-up the error value each time, it will break this semantic, as we do not set
> seen on the upperdir.
>
> Either we need to set the seen flag on the upperdir's errseq_t, or have a sync
> method, like:
> diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
> index 290983bcfbb3..4931d1797c03 100644
> --- a/fs/overlayfs/super.c
> +++ b/fs/overlayfs/super.c
> @@ -259,6 +259,7 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
>  {
>         struct ovl_fs *ofs = sb->s_fs_info;
>         struct super_block *upper_sb;
> +       errseq_t src;
>         int ret;
>
>         if (!ovl_upper_mnt(ofs))
> @@ -283,6 +284,11 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
>         ret = sync_filesystem(upper_sb);
>         up_read(&upper_sb->s_umount);
>
> +       /* Propagate the error up from the upper_sb once */
> +       src = READ_ONCE(upper_sb->s_wb_err);
> +       if (errseq_counter(src) != errseq_counter(sb->s_wb_err))
> +               WRITE_ONCE(sb->s_wb_err, src & ~ERRSEQ_SEEN);
> +
>         return ret;
>  }
>
> @@ -1945,6 +1951,7 @@ static int ovl_fill_super(struct super_block *sb, void *data, int silent)
>
>                 sb->s_stack_depth = ovl_upper_mnt(ofs)->mnt_sb->s_stack_depth;
>                 sb->s_time_gran = ovl_upper_mnt(ofs)->mnt_sb->s_time_gran;
> +               sb->s_wb_err = READ_ONCE(ovl_upper_mnt(ofs)->mnt_sb->s_wb_err);
>
> diff --git a/lib/errseq.c b/lib/errseq.c
> index 81f9e33aa7e7..53275c168265 100644
> --- a/lib/errseq.c
> +++ b/lib/errseq.c
> @@ -204,3 +204,18 @@ int errseq_check_and_advance(errseq_t *eseq, errseq_t *since)
>         return err;
>  }
>  EXPORT_SYMBOL(errseq_check_and_advance);
> +
> +/**
> + * errseq_counter() - Get the value of the errseq counter
> + * @eseq: Value being checked
> + *
> + * This returns the errseq_t counter value. Reminder, it can wrap because
> + * there are only a limited number of counter bits.
> + *
> + * Return: The counter portion of the errseq_t.
> + */
> +int errseq_counter(errseq_t eseq)
> +{
> +       return eseq >> (ERRSEQ_SHIFT + 1);
> +}
>
> This also needs some locking sprinkled on it because racing can occur with
> sync_fs. This would be much easier if the errseq_t was 64-bits, and didn't go
> backwards, because you could just use a simple cmpxchg64. I know that would make
> a bunch of structures larger, and if it's just overlayfs that has to pay the tax
> we can just sprinkle a mutex on it.
>
> We can always "copy-up" the errseq_t because if the upperdir's errseq_t is read,
> and we haven't done a copy yet, we'll get it. Since this will be strictly
> serialized a simple equivalence check will work versus actually having to deal
> with happens before behaviour. There is still a correctness flaw here though,
> which is exactly enough errors occur to result in it wrapping back to the same
> value, it will break.
>
> By the way, how do y'all test this error handling behaviour? I didn't
> find any automated testing for what currently exists.
> >
> > How about I split this into two patchsets? One, where I add the logic to copy
> > the errseq on callbacks to fsync from the upperdir to the ovl fs superblock,
> > and thus allowing VFS to bubble up errors, and the documentation. We can CC
> > stable on those because I think it has an effect that's universal across
> > all filesystems.
> >
> > P.S.
> > I notice you maintain overlay tests outside of the kernel. Unfortunately, I
> > think for this kind of test, it requires in kernel code to artificially bump the
> > writeback error count on the upperdir, or it requires the failure injection
> > infrastructure.
> >
> > Simulating this behaviour is non-trivial without in-kernel support:
> >
> > P1: Open(f) -> p1.fd
> > P2: Open(f) -> p2.fd
> > P1: syncfs(p1.fd) -> errrno
> > P2: syncfs(p1.fd) -> 0 # This should return an error
> >
> >
> > [1]: https://elixir.bootlin.com/linux/latest/source/fs/sync.c#L175
> > [2]: https://lwn.net/Articles/722250/
> >
> >

Sargun (and Jeff),

Thank you for this discussion. It would be very nice to work on testing
and fixing errseq propagation is correct on overlayfs.
Alas, this is not what I suggested.

What I suggested was a solution only for the volatile overlay issue
where data can vaporise without applications noticing:
"...the very minimum is to check for errseq since mount on the fsync
 and syncfs calls."

Do you get it? there is no pre-file state in the game, not for fsync and
not for syncfs.

Any single error, no matter how temporary it is and what damage it may
or may not have caused to upper layer consistency, permanently
invalidates the reliability of the volatile overlay, resulting in:
Effective immediately: every fsync/syncfs returns EIO.
Going forward: maybe implement overlay shutdown, so every access
returns EIO.

So now that I hopefully explained myself better, I'll ask again:
Am I wrong saying that it is very very simple to fix?
Would you mind making that fix at the bottom of the patch series, so it can
be easily applied to stable kernels?

Thanks,
Amir.

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,
	URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 8B640C63777
	for <linux-unionfs@archiver.kernel.org>; Sat, 28 Nov 2020 22:06:36 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id 3BDC6208D5
	for <linux-unionfs@archiver.kernel.org>; Sat, 28 Nov 2020 22:06:36 +0000 (UTC)
Authentication-Results: mail.kernel.org;
	dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="DZ3ZYQLJ"
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1727175AbgK1VuS (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Sat, 28 Nov 2020 16:50:18 -0500
Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:44340 "EHLO
        us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK)
        by vger.kernel.org with ESMTP id S1732866AbgK1SJi (ORCPT
        <rfc822;linux-unionfs@vger.kernel.org>);
        Sat, 28 Nov 2020 13:09:38 -0500
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;
        s=mimecast20190719; t=1606586890;
        h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
         to:to:cc:cc:mime-version:mime-version:content-type:content-type:
         content-transfer-encoding:content-transfer-encoding:
         in-reply-to:in-reply-to:references:references;
        bh=Brfam21Lz9yLQxtBJnvI7hi1Tk6QAjodKT7WugEuJRA=;
        b=DZ3ZYQLJS7/va4dJ4/PfQ+ONtxlKUsg3IJik1LsLwQB8beR/aiPrhCUJ5evYTaZR/kZCOF
        FHd4YqPn9BufLHXLaftADYdu5ZdD5i/fF1vbZlMJrMkysHBysptPLXT5+LQH34GCGzZmuf
        pWRHGJrOK7JT1YsUjKDIHJGLFN1V6pU=
Received: from mail-il1-f198.google.com (mail-il1-f198.google.com
 [209.85.166.198]) (Using TLS) by relay.mimecast.com with ESMTP id
 us-mta-460-cdCMFsxmP2-AJ_8XIdGxYA-1; Sat, 28 Nov 2020 07:04:59 -0500
X-MC-Unique: cdCMFsxmP2-AJ_8XIdGxYA-1
Received: by mail-il1-f198.google.com with SMTP id y6so2194941ilu.14
        for <linux-unionfs@vger.kernel.org>; Sat, 28 Nov 2020 04:04:59 -0800 (PST)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to
         :references:user-agent:mime-version:content-transfer-encoding;
        bh=Brfam21Lz9yLQxtBJnvI7hi1Tk6QAjodKT7WugEuJRA=;
        b=WCLVpQs10ln69SdRvfqsWCaSKayJEyTaOSREo+zB8ZE1JoJJVCmXZJj2rjwFJCHmpe
         RIejHjiVMLn+/+PK/lgtfkXIpmIHlyNmgZe5/DlhrSmuUkaFoC/eReg40vR7OgkBXI7N
         zhAsEhyJCUH8TRInZKAcNNgy6XzIy5MBIgwufieb4MZCaXGUANq9bnioYQb7vlJb7qf5
         mArJXo2qdmzqwnfbZ1ST4MVs6+v8Anb0YwNjWa0MZ4JARd02Ba8ieF6nMMoKdbrOFzZO
         0uUhHxGY1Qj+ofohOrDiyPZGy2uQQxHoGVw8kiS8QQAnnJ6c91KYC6sOR5TXtFvvmdzE
         /M+A==
X-Gm-Message-State: AOAM533YKL+CQUmj2YVHTFVy1ysNb/gcSMTfTBhWXAeD8vnHBYaY2Frv
        +Vhl6fuoJmg+JXsbGRAgm1MGlPc2MGuRFJC60oDjOWCgvLqCTDEM1ZMUQSf94Z8lkICgsLdOu2/
        b+IKlhWKOVUfcw48uiFJFa5TNcw==
X-Received: by 2002:a5d:9049:: with SMTP id v9mr9591129ioq.199.1606565098976;
        Sat, 28 Nov 2020 04:04:58 -0800 (PST)
X-Google-Smtp-Source: ABdhPJxHhR54EsXGeE7ROJ0eQhVLnoTIY9okaZ9rbcQg5+gtLhu3SHgmOO6xB/qgodjvE121S0GA6A==
X-Received: by 2002:a5d:9049:: with SMTP id v9mr9591115ioq.199.1606565098738;
        Sat, 28 Nov 2020 04:04:58 -0800 (PST)
Received: from tleilax.poochiereds.net (68-20-15-154.lightspeed.rlghnc.sbcglobal.net. [68.20.15.154])
        by smtp.gmail.com with ESMTPSA id s134sm7357755ilc.64.2020.11.28.04.04.57
        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
        Sat, 28 Nov 2020 04:04:58 -0800 (PST)
Message-ID: <639dbfa8ee72a058e4e4289860c2eae8c19746d2.camel@redhat.com>
Subject: Re: [PATCH v2 2/4] overlay: Document current outstanding
 shortcoming of volatile
From:   Jeff Layton <jlayton@redhat.com>
To:     Sargun Dhillon <sargun@sargun.me>
Cc:     Amir Goldstein <amir73il@gmail.com>,
        overlayfs <linux-unionfs@vger.kernel.org>,
        Miklos Szeredi <miklos@szeredi.hu>,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Vivek Goyal <vgoyal@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel <linux-fsdevel@vger.kernel.org>,
        David Howells <dhowells@redhat.com>
Date:   Sat, 28 Nov 2020 07:04:56 -0500
In-Reply-To: <20201128044530.GA28230@ircssh-2.c.rugged-nimbus-611.internal>
References: <20201127092058.15117-1-sargun@sargun.me>
         <20201127092058.15117-3-sargun@sargun.me>
         <CAOQ4uxgaLuLb+f6WCMvmKHNTELvcvN8C5_u=t5hhoGT8Op7QuQ@mail.gmail.com>
         <20201127221154.GA23383@ircssh-2.c.rugged-nimbus-611.internal>
         <1338a059d03db0e85cf3f3234fd33434a45606c6.camel@redhat.com>
         <20201128044530.GA28230@ircssh-2.c.rugged-nimbus-611.internal>
Content-Type: text/plain; charset="UTF-8"
User-Agent: Evolution 3.38.1 (3.38.1-1.fc33) 
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org

On Sat, 2020-11-28 at 04:45 +0000, Sargun Dhillon wrote:
> On Fri, Nov 27, 2020 at 09:01:07PM -0500, Jeff Layton wrote:
> > On Fri, 2020-11-27 at 22:11 +0000, Sargun Dhillon wrote:
> > > On Fri, Nov 27, 2020 at 02:52:52PM +0200, Amir Goldstein wrote:
> > > > On Fri, Nov 27, 2020 at 11:21 AM Sargun Dhillon <sargun@sargun.me> wrote:
> > > > > 
> > > > > This documents behaviour that was discussed in a thread about the volatile
> > > > > feature. Specifically, how failures can go hidden from asynchronous writes
> > > > > (such as from mmap, or writes that are not immediately flushed to the
> > > > > filesystem). Although we pass through calls like msync, fallocate, and
> > > > > write, and will still return errors on those, it doesn't guarantee all
> > > > > kinds of errors will happen at those times, and thus may hide errors.
> > > > > 
> > > > > In the future, we can add error checking to all interactions with the
> > > > > upperdir, and pass through errseq_t from the upperdir on mappings,
> > > > > and other interactions with the filesystem[1].
> > > > > 
> > > > > [1]: https://lore.kernel.org/linux-unionfs/20201116045758.21774-1-sargun@sargun.me/T/#m7d501f375e031056efad626e471a1392dd3aad33
> > > > > 
> > > > > Signed-off-by: Sargun Dhillon <sargun@sargun.me>
> > > > > Cc: linux-fsdevel@vger.kernel.org
> > > > > Cc: linux-unionfs@vger.kernel.org
> > > > > Cc: Miklos Szeredi <miklos@szeredi.hu>
> > > > > Cc: Amir Goldstein <amir73il@gmail.com>
> > > > > Cc: Vivek Goyal <vgoyal@redhat.com>
> > > > > ---
> > > > > Ãƒâ€šÃ‚Â Documentation/filesystems/overlayfs.rst | 6 +++++-
> > > > > Ãƒâ€šÃ‚Â 1 file changed, 5 insertions(+), 1 deletion(-)
> > > > > 
> > > > > diff --git a/Documentation/filesystems/overlayfs.rst b/Documentation/filesystems/overlayfs.rst
> > > > > index 580ab9a0fe31..c6e30c1bc2f2 100644
> > > > > --- a/Documentation/filesystems/overlayfs.rst
> > > > > +++ b/Documentation/filesystems/overlayfs.rst
> > > > > @@ -570,7 +570,11 @@ Volatile mount
> > > > > Ãƒâ€šÃ‚Â This is enabled with the "volatile" mount option.  Volatile mounts are not
> > > > > Ãƒâ€šÃ‚Â guaranteed to survive a crash.  It is strongly recommended that volatile
> > > > > Ãƒâ€šÃ‚Â mounts are only used if data written to the overlay can be recreated
> > > > > -without significant effort.
> > > > > +without significant effort.  In addition to this, the sync family of syscalls
> > > > > +are not sufficient to determine whether a write failed as sync calls are
> > > > > +omitted.  For this reason, it is important that the filesystem used by the
> > > > > +upperdir handles failure in a fashion that's suitable for the user.  For
> > > > > +example, upon detecting a fault, ext4 can be configured to panic.
> > > > > 
> > > > 
> > > > Reading this now, I think I may have wrongly analysed the issue.
> > > > Specifically, when I wrote that the very minimum is to document the
> > > > issue, it was under the assumption that a proper fix is hard.
> > > > I think I was wrong and that the very minimum is to check for errseq
> > > > since mount on the fsync and syncfs calls.
> > > > 
> > > > Why? first of all because it is very very simple and goes a long way to
> > > > fix the broken contract with applications, not the contract about durability
> > > > obviously, but the contract about write+fsync+read expects to find the written
> > > > data (during the same mount era).
> > > > 
> > > > Second, because the sentence you added above is hard for users to understand
> > > > and out of context. If we properly handle the writeback error in fsync/syncfs,
> > > > then this sentence becomes irrelevant.
> > > > The fact that ext4 can lose data if application did not fsync after
> > > > write is true
> > > > for volatile as well as non-volatile and it is therefore not relevant
> > > > in the context
> > > > of overlayfs documentation at all.
> > > > 
> > > > Am I wrong saying that it is very very simple to fix?
> > > > Would you mind making that fix at the bottom of the patch series, so it can
> > > > be easily applied to stable kernels?
> > > > 
> > > > Thanks,
> > > > Amir.
> > > 
> > > I'm not sure it's so easy. In VFS, there are places where the superblock's 
> > > errseq is checked[1]. AFAIK, that's going to check "our" errseq, and not the 
> > > errseq of the real corresponding real file's superblock. One solution might be 
> > > as part of all these callbacks we set our errseq to the errseq of the filesystem 
> > > that the upperdir, and then rely on VFS's checking.
> > > 
> > > I'm having a hard time figuring out how to deal with the per-mapping based
> > > error tracking. It seems like this infrastructure was only partially completed
> > > by Jeff Layton[2]. I don't now how it's actually supposed to work right now,
> > > as not all of his patches landed.
> > > 
> > 
> > The patches in the series were all merged, but we ended up going with a
> > simpler solution [1] than the first series I posted. Instead of plumbing
> > the errseq_t handling down into sync_fs, we did it in the syscall
> > wrapper.
> Jeff,
> 
> Thanks for replying. I'm still a little confused as to what the 
> per-address_space wb_err. It seems like we should implement the
> flush method, like so:
> diff --git a/fs/overlayfs/file.c b/fs/overlayfs/file.c
> index efccb7c1f9bc..32e5bc0aacd6 100644
> --- a/fs/overlayfs/file.c
> +++ b/fs/overlayfs/file.c
> @@ -787,6 +787,16 @@ static loff_t ovl_remap_file_range(struct file *file_in, loff_t pos_in,
> Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â remap_flags, op);
> Â }
> Â 
> 
> 
> 
> +static int ovl_flush(struct file *file, fl_owner_t id)
> +{
> +       struct file *realfile = file->private_data;
> +
> +       if (real_file->f_op->flush)
> +               return real_file->f_op->flush(filp, id);
> +
> +       return 0;
> +}
> +
> Â const struct file_operations ovl_file_operations = {
> Â Â Â Â Â Â Â Â .open           = ovl_open,
> Â Â Â Â Â Â Â Â .release        = ovl_release,
> @@ -798,6 +808,7 @@ const struct file_operations ovl_file_operations = {
> Â Â Â Â Â Â Â Â .fallocate      = ovl_fallocate,
> Â Â Â Â Â Â Â Â .fadvise        = ovl_fadvise,
> Â Â Â Â Â Â Â Â .unlocked_ioctl = ovl_ioctl,
> +       .flush          = ovl_flush,
> Â #ifdef CONFIG_COMPAT
> Â Â Â Â Â Â Â Â .compat_ioctl   = ovl_compat_ioctl,
> Â #endif
> 
> 
> > 
> > I think the tricky part here is that there is no struct file plumbed
> > into ->sync_fs, so you don't have an errseq_t cursor to work with by the
> > time that gets called.
> > 
> > What may be easiest is to just propagate the s_wb_err value from the
> > upper_sb to the overlayfs superblock in ovl_sync_fs(). That should get
> > called before the errseq_check_and_advance in the syncfs syscall wrapper
> > and should ensure that syncfs() calls against the overlayfs sb see any
> > errors that occurred on the upper_sb.
> > 
> > Something like this maybe? Totally untested of course. May also need to
> > think about how to ensure ordering between racing syncfs calls too
> > (don't really want the s_wb_err going "backward"):
> > 
> > ----------------------------8<---------------------------------
> > $ git diff
> > diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
> > index 290983bcfbb3..d725705abdac 100644
> > --- a/fs/overlayfs/super.c
> > +++ b/fs/overlayfs/super.c
> > @@ -283,6 +283,9 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
> > Â Â Â Â Â Â Â Â ret = sync_filesystem(upper_sb);
> > Â Â Â Â Â Â Â Â up_read(&upper_sb->s_umount);
> > Â 
> > 
> > 
> > 
> > +       /* Propagate s_wb_err from upper_sb to overlayfs sb */
> > +       WRITE_ONCE(sb->s_wb_err, READ_ONCE(upper_sb->s_wb_err));
> > +
> > Â Â Â Â Â Â Â Â return ret;
> > Â }
> > ----------------------------8<---------------------------------
> > 
> > [1]: https://www.spinics.net/lists/linux-api/msg41276.html
> 
> So, 
> I think this will have bad behaviour because syncfs() twice in a row will 
> return the error twice. The reason is that normally in syncfs it calls 
> errseq_check_and_advance marking the error on the super block as seen. If we 
> copy-up the error value each time, it will break this semantic, as we do not set 
> seen on the upperdir.
> 

You're absolutely right. I thought about this after I had sent the mail
last night.

> Either we need to set the seen flag on the upperdir's errseq_t, or have a sync 
> method, like:
> diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
> index 290983bcfbb3..4931d1797c03 100644
> --- a/fs/overlayfs/super.c
> +++ b/fs/overlayfs/super.c
> @@ -259,6 +259,7 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
> Â {
> Â Â Â Â Â Â Â Â struct ovl_fs *ofs = sb->s_fs_info;
> Â Â Â Â Â Â Â Â struct super_block *upper_sb;
> +       errseq_t src;
> Â Â Â Â Â Â Â Â int ret;
> Â 
> 
> 
> 
> 
> 
> 
> 
> Â Â Â Â Â Â Â Â if (!ovl_upper_mnt(ofs))
> @@ -283,6 +284,11 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
> Â Â Â Â Â Â Â Â ret = sync_filesystem(upper_sb);
> Â Â Â Â Â Â Â Â up_read(&upper_sb->s_umount);
> Â 
> 
> 
> 
> 
> 
> 
> 
> +       /* Propagate the error up from the upper_sb once */
> +       src = READ_ONCE(upper_sb->s_wb_err);
> +       if (errseq_counter(src) != errseq_counter(sb->s_wb_err))
> +               WRITE_ONCE(sb->s_wb_err, src & ~ERRSEQ_SEEN);
> +
> Â Â Â Â Â Â Â Â return ret;
> Â }
> Â 
> 
> 
> 
> 
> 
> 
> 
> @@ -1945,6 +1951,7 @@ static int ovl_fill_super(struct super_block *sb, void *data, int silent)
> Â 
> 
> 
> 
> 
> 
> 
> 
> Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â sb->s_stack_depth = ovl_upper_mnt(ofs)->mnt_sb->s_stack_depth;
> Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â sb->s_time_gran = ovl_upper_mnt(ofs)->mnt_sb->s_time_gran;
> +               sb->s_wb_err = READ_ONCE(ovl_upper_mnt(ofs)->mnt_sb->s_wb_err);
> Â 
> 
> 
> 
> 
> 
> 
> 
> diff --git a/lib/errseq.c b/lib/errseq.c
> index 81f9e33aa7e7..53275c168265 100644
> --- a/lib/errseq.c
> +++ b/lib/errseq.c
> @@ -204,3 +204,18 @@ int errseq_check_and_advance(errseq_t *eseq, errseq_t *since)
> Â Â Â Â Â Â Â Â return err;
> Â }
> Â EXPORT_SYMBOL(errseq_check_and_advance);
> +
> +/**
> + * errseq_counter() - Get the value of the errseq counter
> + * @eseq: Value being checked
> + *
> + * This returns the errseq_t counter value. Reminder, it can wrap because
> + * there are only a limited number of counter bits.
> + *
> + * Return: The counter portion of the errseq_t.
> + */
> +int errseq_counter(errseq_t eseq)
> +{
> +       return eseq >> (ERRSEQ_SHIFT + 1);
> +}
> 
> This also needs some locking sprinkled on it because racing can occur with 
> sync_fs. This would be much easier if the errseq_t was 64-bits, and didn't go 
> backwards, because you could just use a simple cmpxchg64. I know that would make 
> a bunch of structures larger, and if it's just overlayfs that has to pay the tax
> we can just sprinkle a mutex on it.
> 

A spinlock would work here too. None of these are blocking operations.

> We can always "copy-up" the errseq_t because if the upperdir's errseq_t is read, 
> and we haven't done a copy yet, we'll get it. Since this will be strictly 
> serialized a simple equivalence check will work versus actually having to deal 
> with happens before behaviour. There is still a correctness flaw here though, 
> which is exactly enough errors occur to result in it wrapping back to the same 
> value, it will break.
> 
> By the way, how do y'all test this error handling behaviour? I didn't
> find any automated testing for what currently exists.

There are a couple of xfstests that you may be able to use as a starting
point. See:

generic/441
generic/442


> > 
> > How about I split this into two patchsets? One, where I add the logic to copy
> > the errseq on callbacks to fsync from the upperdir to the ovl fs superblock,
> > and thus allowing VFS to bubble up errors, and the documentation. We can CC
> > stable on those because I think it has an effect that's universal across
> > all filesystems.
> > 
> > P.S. 
> > I notice you maintain overlay tests outside of the kernel. Unfortunately, I 
> > think for this kind of test, it requires in kernel code to artificially bump the 
> > writeback error count on the upperdir, or it requires the failure injection 
> > infrastructure. 
> > 
> > Simulating this behaviour is non-trivial without in-kernel support:
> > 
> > P1: Open(f) -> p1.fd
> > P2: Open(f) -> p2.fd
> > P1: syncfs(p1.fd) -> errrno
> > P2: syncfs(p1.fd) -> 0 # This should return an error
> > 
> > 
> > [1]: https://elixir.bootlin.com/linux/latest/source/fs/sync.c#L175
> > [2]: https://lwn.net/Articles/722250/
> > 
> > 
> > -- 
> > Jeff Layton <jlayton@redhat.com>
> > 
> 

-- 
Jeff Layton <jlayton@redhat.com>


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-2.7 required=3.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,
	HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS
	autolearn=no autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 6500CC64E8A
	for <linux-unionfs@archiver.kernel.org>; Sat, 28 Nov 2020 22:06:37 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id 33C2621534
	for <linux-unionfs@archiver.kernel.org>; Sat, 28 Nov 2020 22:06:37 +0000 (UTC)
Authentication-Results: mail.kernel.org;
	dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="cfrxJQBJ"
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1732195AbgK1VuT (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Sat, 28 Nov 2020 16:50:19 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42368 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1732006AbgK1S52 (ORCPT
        <rfc822;linux-unionfs@vger.kernel.org>);
        Sat, 28 Nov 2020 13:57:28 -0500
Received: from mail-il1-x143.google.com (mail-il1-x143.google.com [IPv6:2607:f8b0:4864:20::143])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C2895C02B8ED;
        Sat, 28 Nov 2020 00:56:42 -0800 (PST)
Received: by mail-il1-x143.google.com with SMTP id a19so6639315ilm.3;
        Sat, 28 Nov 2020 00:56:42 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20161025;
        h=mime-version:references:in-reply-to:from:date:message-id:subject:to
         :cc;
        bh=On7srdy0/TdSxd0+F9jkJfCVKZfYH3vASimHmr7KFgA=;
        b=cfrxJQBJoIZl4NsGfhAzAbbegtPgs53PgmrPipjNQgSty+JEr/Qrfx7kCsUPohRITZ
         ZfDxq67/VDFyg/mCtSaQjm1zckIVgMLooXvbSZ+yWbjjUAi0oNyvvxy+V0PMMB+z0tJP
         JnqQ4cQOUSlrFLRnJThcU2wyJ2Hb8xggwbcxB57CaZChoown0anqdBGBBLI+VgIMhq85
         gfXalMu2jm+ugt7LAM3e58oiLu96Z0pkfl/1GUEb3aXLf7HTX/3ffgc/PxkKxTZ5gw7K
         g60QEPcFi8uZ7k7cVl+Hqkmo2fmdptSeKVPLzfwmomghb2G9RI/CcLAf/Z4zqocupl5y
         GTyw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:mime-version:references:in-reply-to:from:date
         :message-id:subject:to:cc;
        bh=On7srdy0/TdSxd0+F9jkJfCVKZfYH3vASimHmr7KFgA=;
        b=ZZBEDXZibNuWgoYW7F54+HWARSby5z4/VbnH4Dbcd+QYFjZPV0zNe3K6uZZCdKFKrQ
         LM/fAVBVJCF/0Eogn1TXoayg1gRV8eqkw/vp7jcrHerpv8DaaWO395hieKvBNY5D2lGQ
         mby7oW+PkX2f3BGXZWNugOOeP65VKruV9xIVDQfDYPQdd4V998NJQFMzlaaqzWs3L3jI
         GBTdOkgsDL1lK7kpXTfahdZ1mGJALXlh+L8zeG3JkZwrPSyubh4YDevW6gcmPY2NPXA7
         TZEVI2jyOIe9ZfHZqtWPWU/S3qCvC5E/8wEIwJk6G/zMKGZS7bAcNWLjZmMvQ9mg2dxs
         REjA==
X-Gm-Message-State: AOAM533BE4mIbZbr6Ru7tqLajwBrGMMrlgw70g1dHMxURqXTlL3dBfyl
        jHaiKK5uFiWRe2e1oiGeh/Q83aDiwmFcO3pqcfs=
X-Google-Smtp-Source: ABdhPJzP8Sp+e1mwcr1OUIllQ/G6nfnGLb51UAEdFFXi14s8dEVhbFViKPDbjE+9LIIcEB6LnjQRV62CjZyM/kCr//U=
X-Received: by 2002:a92:da82:: with SMTP id u2mr10683732iln.137.1606553801985;
 Sat, 28 Nov 2020 00:56:41 -0800 (PST)
MIME-Version: 1.0
References: <20201127092058.15117-1-sargun@sargun.me> <20201127092058.15117-3-sargun@sargun.me>
 <CAOQ4uxgaLuLb+f6WCMvmKHNTELvcvN8C5_u=t5hhoGT8Op7QuQ@mail.gmail.com> <20201127221154.GA23383@ircssh-2.c.rugged-nimbus-611.internal>
In-Reply-To: <20201127221154.GA23383@ircssh-2.c.rugged-nimbus-611.internal>
From:   Amir Goldstein <amir73il@gmail.com>
Date:   Sat, 28 Nov 2020 10:56:30 +0200
Message-ID: <CAOQ4uxiLRy9ioqaqtOp7P6hLy8Gx5vRO86mie7FAdTu2OfnGrw@mail.gmail.com>
Subject: Re: [PATCH v2 2/4] overlay: Document current outstanding shortcoming
 of volatile
To:     Sargun Dhillon <sargun@sargun.me>
Cc:     overlayfs <linux-unionfs@vger.kernel.org>,
        Miklos Szeredi <miklos@szeredi.hu>,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Vivek Goyal <vgoyal@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel <linux-fsdevel@vger.kernel.org>,
        David Howells <dhowells@redhat.com>,
        Jeff Layton <jlayton@redhat.com>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org

> I notice you maintain overlay tests outside of the kernel. Unfortunately, I
> think for this kind of test, it requires in kernel code to artificially bump the
> writeback error count on the upperdir, or it requires the failure injection
> infrastructure.
>
> Simulating this behaviour is non-trivial without in-kernel support:
>
> P1: Open(f) -> p1.fd
> P2: Open(f) -> p2.fd
> P1: syncfs(p1.fd) -> errrno
> P2: syncfs(p1.fd) -> 0 # This should return an error
>

failure injection is an option. xfstest generic/019 is an example.
generic/108 uses a different method and generic/361 uses a plain
loop image over commit without any fault injection to trigger writeback
errors.

With current xfstests, check -overlay run (runs all generic tests with
overlayfs over the configured base fs) all the 3 tests mentioned above
will be skipped because of _require_block_device, but the requirement
is there for a different reason for each of them.

At first look, the loop device approach is the most generic one and could
easily work also with overlayfs, so you could create an overlay specific
test (non generic) based on generic/361, but it is not easy to use the
helper _scratch_mkfs_sized, because check -overlay runs do not mkfs
the base scratch fs.

My recommendation would be to fix generic/019 in a similar manner
to the way that tests that _require_scratch_shutdown were fixed to run
with check -overlay:

* Instead of _require_block_device, add a specific helper
   _require_scratch_fail_make_request, which like _require_scratch_shutdown
   knows how to deal with overlay FSTYP correctly

* Instead of `_sysfs_dev $SCRATCH_DEV` use a helper _scratch_sysfs_dev
   that knows how to deal with overlay FSTYP correctly

This will add test coverage to overlayfs fsync/syncfs and then you can
do one or both of:
1. Run 'check -overlay generic/019' with  OVERLAY_MOUNT_OPTIONS="volatile"
2. Fork an overlay specific test from the generic test that will test the
    volatile error handling on every 'check -overlay -g quick' run

#2 will provide better coverage against regressions in volatile writeback error
handling and will be a good start for a test to reuse a volatile mount after
writeback errors.

Thanks,
Amir.

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-2.7 required=3.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,
	HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS
	autolearn=no autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 2C147C83011
	for <linux-unionfs@archiver.kernel.org>; Sat, 28 Nov 2020 22:06:38 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id E031421534
	for <linux-unionfs@archiver.kernel.org>; Sat, 28 Nov 2020 22:06:37 +0000 (UTC)
Authentication-Results: mail.kernel.org;
	dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="FPnRfdiL"
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1732207AbgK1VuU (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Sat, 28 Nov 2020 16:50:20 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43160 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1732080AbgK1S57 (ORCPT
        <rfc822;linux-unionfs@vger.kernel.org>);
        Sat, 28 Nov 2020 13:57:59 -0500
Received: from mail-io1-xd32.google.com (mail-io1-xd32.google.com [IPv6:2607:f8b0:4864:20::d32])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4D107C02B8F0;
        Sat, 28 Nov 2020 01:04:49 -0800 (PST)
Received: by mail-io1-xd32.google.com with SMTP id d8so5717552ioc.13;
        Sat, 28 Nov 2020 01:04:49 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20161025;
        h=mime-version:references:in-reply-to:from:date:message-id:subject:to
         :cc;
        bh=8AGnrWYiumdXafgcLifADEbB8pfQT5OzqYJQsv7KOwc=;
        b=FPnRfdiLLw5wVk7CiuBbqzz+/6V7lzE0yO93m1B1TvQv+87eNEOKe/6uLyG2jVq3Vw
         dOyr+feVK1LVbRL57sp1M2iS935WuT8X2BxrMMw8kPHJCqd2atbIeLTCxkBhatzLlvWb
         WgSyNcMeMKT4Z6gzHFxlN+Hk49WquySj+VaFsraciwXqAvFNZ8GLrNyeobOZb6SMrzbz
         aHtltylOiXxdPh5UBibVHJUI8cIZTTUdgsurww2MlQ40zU2NVlaFDsqFBfUPCiJDdOax
         bUqRUpiflFpTW11h0a3xk5D/Mg1QwCGrM3POS458wq0+2WOZhP1eR/7S0rjCTpdyF7G8
         dvnQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:mime-version:references:in-reply-to:from:date
         :message-id:subject:to:cc;
        bh=8AGnrWYiumdXafgcLifADEbB8pfQT5OzqYJQsv7KOwc=;
        b=ltkUZGr2W1nzqWGlIJJGA1Jg0Wl0Kln3nac4WKJQZ5EacgsbzzbfGPXmv7xHOePKjc
         80MHc10kTki8xkBrkTsyRyi+PBUonbpouh6dNnpT/iMvT5dIpBIKK4Ks4mmUgT1y0P2A
         wfscCkX0alCjESBhARFmV/dQywxnNrTJDBzv6oujKZzTXlTdbrhh/UeRGR/lhweAg8/L
         xcAR2Q4vWLOItGXEvTOM/wpt1D9sh7BLBKsWyT1bZ5nIihb4FFo0Iqjbgw5WU1OuMuxy
         LbAWALeJymrGVP7jKRwLNGfAEw1A/rRAIJll38o6tzMoPc6BkCi9EHkM5DyshVGDIzKU
         f3tw==
X-Gm-Message-State: AOAM530rNtS+XIBFwWy25igNSi5mu+aPVc2WGWlqCJHRXLCpYfZNw4Bt
        t01k75aHg7QfGyaaoQfTEXbGtx9AaKICYmge0Qk=
X-Google-Smtp-Source: ABdhPJxEaU7n183i+2OtQoiTo9EP+n6e7u2ckH6pjcpA/PVT0nESydya52Qc10oxkXChJabkUbXMGwmXvJSPZsJjTQw=
X-Received: by 2002:a02:a60a:: with SMTP id c10mr11057629jam.123.1606554288643;
 Sat, 28 Nov 2020 01:04:48 -0800 (PST)
MIME-Version: 1.0
References: <20201127092058.15117-1-sargun@sargun.me> <20201127092058.15117-3-sargun@sargun.me>
 <CAOQ4uxgaLuLb+f6WCMvmKHNTELvcvN8C5_u=t5hhoGT8Op7QuQ@mail.gmail.com>
 <20201127221154.GA23383@ircssh-2.c.rugged-nimbus-611.internal>
 <1338a059d03db0e85cf3f3234fd33434a45606c6.camel@redhat.com>
 <20201128044530.GA28230@ircssh-2.c.rugged-nimbus-611.internal>
 <CAOQ4uxjT6FF03Sq3qXuqDcqJQnzQq2dD_XVbuj_Fb9A2Ag585w@mail.gmail.com> <20201128085227.GB28230@ircssh-2.c.rugged-nimbus-611.internal>
In-Reply-To: <20201128085227.GB28230@ircssh-2.c.rugged-nimbus-611.internal>
From:   Amir Goldstein <amir73il@gmail.com>
Date:   Sat, 28 Nov 2020 11:04:37 +0200
Message-ID: <CAOQ4uxjKRL0Pime7BO9gr_sVfmdhV2XrPhSzOPdDoaCAcRYHBQ@mail.gmail.com>
Subject: Re: [PATCH v2 2/4] overlay: Document current outstanding shortcoming
 of volatile
To:     Sargun Dhillon <sargun@sargun.me>
Cc:     Jeff Layton <jlayton@redhat.com>,
        overlayfs <linux-unionfs@vger.kernel.org>,
        Miklos Szeredi <miklos@szeredi.hu>,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Vivek Goyal <vgoyal@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel <linux-fsdevel@vger.kernel.org>,
        David Howells <dhowells@redhat.com>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org

> > What I suggested was a solution only for the volatile overlay issue
> > where data can vaporise without applications noticing:
> > "...the very minimum is to check for errseq since mount on the fsync
> >  and syncfs calls."
> >
> Yeah, I was confusing the checking that VFS does on our behalf and the checking
> that we can do ourselves in the sync callback. If we return an error prior to
> the vfs checking it short-circuits that entirely.
>
> > Do you get it? there is no pre-file state in the game, not for fsync and not
> > for syncfs.
> >
> > Any single error, no matter how temporary it is and what damage it may
> > or may not have caused to upper layer consistency, permanently
> > invalidates the reliability of the volatile overlay, resulting in:
> > Effective immediately: every fsync/syncfs returns EIO.
> > Going forward: maybe implement overlay shutdown, so every access
> > returns EIO.
> >
> > So now that I hopefully explained myself better, I'll ask again:
> > Am I wrong saying that it is very very simple to fix?
> > Would you mind making that fix at the bottom of the patch series, so it can
> > be easily applied to stable kernels?
> >
> > Thanks,
> > Amir.
>
> I think that this should be easy enough if the semantic is such that volatile
> overlayfs mounts will return EIO on syncfs on every syncfs call if the upperdir's
> super block has experienced errors since the initial mount. I imagine we do not
> want to make it such that if the upperdir has ever experienced errors, return
> EIO on syncfs.
>
> The one caveat that I see is that if the errseq wraps, we can silently begin
> swallowing errors again. Thus, on the first failed syncfs we should just
> store a flag indicating that the volatile fs is bad, and to continue to return
> EIO rather than go through the process of checking errseq_t, but that's easy
> enough to write.

I agree. I sent another reply to your question about testing.
The test I suggested generic/019, only tests that the first fsync
after writeback
error fails and that umount succeeds, so logic is good for volatile overlay.

Thanks,
Amir.

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-17.2 required=3.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,
	INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,
	USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id C94E2C83010
	for <linux-unionfs@archiver.kernel.org>; Sat, 28 Nov 2020 22:06:37 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id 89F4121534
	for <linux-unionfs@archiver.kernel.org>; Sat, 28 Nov 2020 22:06:37 +0000 (UTC)
Authentication-Results: mail.kernel.org;
	dkim=pass (1024-bit key) header.d=sargun.me header.i=@sargun.me header.b="IQradKPz"
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1732167AbgK1VuT (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Sat, 28 Nov 2020 16:50:19 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42392 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1731768AbgK1SzT (ORCPT
        <rfc822;linux-unionfs@vger.kernel.org>);
        Sat, 28 Nov 2020 13:55:19 -0500
Received: from mail-il1-x141.google.com (mail-il1-x141.google.com [IPv6:2607:f8b0:4864:20::141])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 27422C02B8EC
        for <linux-unionfs@vger.kernel.org>; Sat, 28 Nov 2020 00:52:32 -0800 (PST)
Received: by mail-il1-x141.google.com with SMTP id t13so6649796ilp.2
        for <linux-unionfs@vger.kernel.org>; Sat, 28 Nov 2020 00:52:32 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=sargun.me; s=google;
        h=date:from:to:cc:subject:message-id:references:mime-version
         :content-disposition:in-reply-to:user-agent;
        bh=3TFQEugyT3Xmlb0jERCsSaEohC37InbvC2EDez9LZ/I=;
        b=IQradKPzlT/1g9wSEHQ6ET+zTULtLWmwpSi9qaXzGAFz6zHpEy5aKy2cK5PlVNARF3
         LWPMBzRrZECZOVaAIEK2r0GD19BNbLKeNjgM3KVa/eJsHfbb/dOuDeJkJlpn1mY2PxKd
         T4h1vNrGsZKa7c3RWwxBIGEFpXscfpDzWB714=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:date:from:to:cc:subject:message-id:references
         :mime-version:content-disposition:in-reply-to:user-agent;
        bh=3TFQEugyT3Xmlb0jERCsSaEohC37InbvC2EDez9LZ/I=;
        b=m5+tHE8u6XCy4I66A/W5Fe9lX13SsthF0yITrdL4qyCJ5NufZso6fDxQMYTADnxjNN
         AAiV410sa0/XOWc5MXLnbT4T1wJttwWIuICwWz/yODBoqpNZdW6fRIItSmeQGCdvvllk
         FnJ4/11dMQpwBOsJEqCOAv1kW1rtL1ApUuk9b2CyTtHa1NJwGMJNYMSY5Q8beKp+/lZr
         f6bYuX/5QlJeWh5nGQqv2ORicVO72Mm99YNurAm/X6tlqWoOWHknOcQXZXHwxit3ABji
         HgIBLqF9Q+3AVK2pPNyYE8tvEhACitmXoXLKyhZv1fz+vjhDSCjqaNMxB6iH10t5mnOm
         yM8w==
X-Gm-Message-State: AOAM533s1Tg3FWyFFxj9l5k2p/b6OTkWsL3OLvqjgOzoV3Orb/2RPhQZ
        SGUijHjthAQplliSlZpVLs7ieWK4A2x8teUT
X-Google-Smtp-Source: ABdhPJxWZvlO+VbCCCwuVuDzoQqKttwumcFExW9qeNYIpJpwzstLdwm1hvA8Bxj6mGf0VgxNjvegWw==
X-Received: by 2002:a92:85c5:: with SMTP id f188mr10823177ilh.173.1606553550298;
        Sat, 28 Nov 2020 00:52:30 -0800 (PST)
Received: from ircssh-2.c.rugged-nimbus-611.internal (80.60.198.104.bc.googleusercontent.com. [104.198.60.80])
        by smtp.gmail.com with ESMTPSA id q5sm3494958ilg.62.2020.11.28.00.52.29
        (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256);
        Sat, 28 Nov 2020 00:52:29 -0800 (PST)
Date:   Sat, 28 Nov 2020 08:52:28 +0000
From:   Sargun Dhillon <sargun@sargun.me>
To:     Amir Goldstein <amir73il@gmail.com>
Cc:     Jeff Layton <jlayton@redhat.com>,
        overlayfs <linux-unionfs@vger.kernel.org>,
        Miklos Szeredi <miklos@szeredi.hu>,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Vivek Goyal <vgoyal@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel <linux-fsdevel@vger.kernel.org>,
        David Howells <dhowells@redhat.com>
Subject: Re: [PATCH v2 2/4] overlay: Document current outstanding shortcoming
 of volatile
Message-ID: <20201128085227.GB28230@ircssh-2.c.rugged-nimbus-611.internal>
References: <20201127092058.15117-1-sargun@sargun.me>
 <20201127092058.15117-3-sargun@sargun.me>
 <CAOQ4uxgaLuLb+f6WCMvmKHNTELvcvN8C5_u=t5hhoGT8Op7QuQ@mail.gmail.com>
 <20201127221154.GA23383@ircssh-2.c.rugged-nimbus-611.internal>
 <1338a059d03db0e85cf3f3234fd33434a45606c6.camel@redhat.com>
 <20201128044530.GA28230@ircssh-2.c.rugged-nimbus-611.internal>
 <CAOQ4uxjT6FF03Sq3qXuqDcqJQnzQq2dD_XVbuj_Fb9A2Ag585w@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAOQ4uxjT6FF03Sq3qXuqDcqJQnzQq2dD_XVbuj_Fb9A2Ag585w@mail.gmail.com>
User-Agent: Mutt/1.9.4 (2018-02-28)
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org

On Sat, Nov 28, 2020 at 09:12:03AM +0200, Amir Goldstein wrote:
> On Sat, Nov 28, 2020 at 6:45 AM Sargun Dhillon <sargun@sargun.me> wrote:
> >
> > On Fri, Nov 27, 2020 at 09:01:07PM -0500, Jeff Layton wrote:
> > > On Fri, 2020-11-27 at 22:11 +0000, Sargun Dhillon wrote:
> > > > On Fri, Nov 27, 2020 at 02:52:52PM +0200, Amir Goldstein wrote:
> > > > > On Fri, Nov 27, 2020 at 11:21 AM Sargun Dhillon <sargun@sargun.me> wrote:
> > > > > >
> > > > > > This documents behaviour that was discussed in a thread about the volatile
> > > > > > feature. Specifically, how failures can go hidden from asynchronous writes
> > > > > > (such as from mmap, or writes that are not immediately flushed to the
> > > > > > filesystem). Although we pass through calls like msync, fallocate, and
> > > > > > write, and will still return errors on those, it doesn't guarantee all
> > > > > > kinds of errors will happen at those times, and thus may hide errors.
> > > > > >
> > > > > > In the future, we can add error checking to all interactions with the
> > > > > > upperdir, and pass through errseq_t from the upperdir on mappings,
> > > > > > and other interactions with the filesystem[1].
> > > > > >
> > > > > > [1]: https://lore.kernel.org/linux-unionfs/20201116045758.21774-1-sargun@sargun.me/T/#m7d501f375e031056efad626e471a1392dd3aad33
> > > > > >
> > > > > > Signed-off-by: Sargun Dhillon <sargun@sargun.me>
> > > > > > Cc: linux-fsdevel@vger.kernel.org
> > > > > > Cc: linux-unionfs@vger.kernel.org
> > > > > > Cc: Miklos Szeredi <miklos@szeredi.hu>
> > > > > > Cc: Amir Goldstein <amir73il@gmail.com>
> > > > > > Cc: Vivek Goyal <vgoyal@redhat.com>
> > > > > > ---
> > > > > >  Documentation/filesystems/overlayfs.rst | 6 +++++-
> > > > > >  1 file changed, 5 insertions(+), 1 deletion(-)
> > > > > >
> > > > > > diff --git a/Documentation/filesystems/overlayfs.rst b/Documentation/filesystems/overlayfs.rst
> > > > > > index 580ab9a0fe31..c6e30c1bc2f2 100644
> > > > > > --- a/Documentation/filesystems/overlayfs.rst
> > > > > > +++ b/Documentation/filesystems/overlayfs.rst
> > > > > > @@ -570,7 +570,11 @@ Volatile mount
> > > > > >  This is enabled with the "volatile" mount option.  Volatile mounts are not
> > > > > >  guaranteed to survive a crash.  It is strongly recommended that volatile
> > > > > >  mounts are only used if data written to the overlay can be recreated
> > > > > > -without significant effort.
> > > > > > +without significant effort.  In addition to this, the sync family of syscalls
> > > > > > +are not sufficient to determine whether a write failed as sync calls are
> > > > > > +omitted.  For this reason, it is important that the filesystem used by the
> > > > > > +upperdir handles failure in a fashion that's suitable for the user.  For
> > > > > > +example, upon detecting a fault, ext4 can be configured to panic.
> > > > > >
> > > > >
> > > > > Reading this now, I think I may have wrongly analysed the issue.
> > > > > Specifically, when I wrote that the very minimum is to document the
> > > > > issue, it was under the assumption that a proper fix is hard.
> > > > > I think I was wrong and that the very minimum is to check for errseq
> > > > > since mount on the fsync and syncfs calls.
> > > > >
> > > > > Why? first of all because it is very very simple and goes a long way to
> > > > > fix the broken contract with applications, not the contract about durability
> > > > > obviously, but the contract about write+fsync+read expects to find the written
> > > > > data (during the same mount era).
> > > > >
> > > > > Second, because the sentence you added above is hard for users to understand
> > > > > and out of context. If we properly handle the writeback error in fsync/syncfs,
> > > > > then this sentence becomes irrelevant.
> > > > > The fact that ext4 can lose data if application did not fsync after
> > > > > write is true
> > > > > for volatile as well as non-volatile and it is therefore not relevant
> > > > > in the context
> > > > > of overlayfs documentation at all.
> > > > >
> > > > > Am I wrong saying that it is very very simple to fix?
> > > > > Would you mind making that fix at the bottom of the patch series, so it can
> > > > > be easily applied to stable kernels?
> > > > >
> > > > > Thanks,
> > > > > Amir.
> > > >
> > > > I'm not sure it's so easy. In VFS, there are places where the superblock's
> > > > errseq is checked[1]. AFAIK, that's going to check "our" errseq, and not the
> > > > errseq of the real corresponding real file's superblock. One solution might be
> > > > as part of all these callbacks we set our errseq to the errseq of the filesystem
> > > > that the upperdir, and then rely on VFS's checking.
> > > >
> > > > I'm having a hard time figuring out how to deal with the per-mapping based
> > > > error tracking. It seems like this infrastructure was only partially completed
> > > > by Jeff Layton[2]. I don't now how it's actually supposed to work right now,
> > > > as not all of his patches landed.
> > > >
> > >
> > > The patches in the series were all merged, but we ended up going with a
> > > simpler solution [1] than the first series I posted. Instead of plumbing
> > > the errseq_t handling down into sync_fs, we did it in the syscall
> > > wrapper.
> > Jeff,
> >
> > Thanks for replying. I'm still a little confused as to what the
> > per-address_space wb_err. It seems like we should implement the
> > flush method, like so:
> > diff --git a/fs/overlayfs/file.c b/fs/overlayfs/file.c
> > index efccb7c1f9bc..32e5bc0aacd6 100644
> > --- a/fs/overlayfs/file.c
> > +++ b/fs/overlayfs/file.c
> > @@ -787,6 +787,16 @@ static loff_t ovl_remap_file_range(struct file *file_in, loff_t pos_in,
> >                             remap_flags, op);
> >  }
> >
> > +static int ovl_flush(struct file *file, fl_owner_t id)
> > +{
> > +       struct file *realfile = file->private_data;
> > +
> > +       if (real_file->f_op->flush)
> > +               return real_file->f_op->flush(filp, id);
> > +
> > +       return 0;
> > +}
> > +
> >  const struct file_operations ovl_file_operations = {
> >         .open           = ovl_open,
> >         .release        = ovl_release,
> > @@ -798,6 +808,7 @@ const struct file_operations ovl_file_operations = {
> >         .fallocate      = ovl_fallocate,
> >         .fadvise        = ovl_fadvise,
> >         .unlocked_ioctl = ovl_ioctl,
> > +       .flush          = ovl_flush,
> >  #ifdef CONFIG_COMPAT
> >         .compat_ioctl   = ovl_compat_ioctl,
> >  #endif
> >
> >
> > >
> > > I think the tricky part here is that there is no struct file plumbed
> > > into ->sync_fs, so you don't have an errseq_t cursor to work with by the
> > > time that gets called.
> > >
> > > What may be easiest is to just propagate the s_wb_err value from the
> > > upper_sb to the overlayfs superblock in ovl_sync_fs(). That should get
> > > called before the errseq_check_and_advance in the syncfs syscall wrapper
> > > and should ensure that syncfs() calls against the overlayfs sb see any
> > > errors that occurred on the upper_sb.
> > >
> > > Something like this maybe? Totally untested of course. May also need to
> > > think about how to ensure ordering between racing syncfs calls too
> > > (don't really want the s_wb_err going "backward"):
> > >
> > > ----------------------------8<---------------------------------
> > > $ git diff
> > > diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
> > > index 290983bcfbb3..d725705abdac 100644
> > > --- a/fs/overlayfs/super.c
> > > +++ b/fs/overlayfs/super.c
> > > @@ -283,6 +283,9 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
> > >         ret = sync_filesystem(upper_sb);
> > >         up_read(&upper_sb->s_umount);
> > >
> > > +       /* Propagate s_wb_err from upper_sb to overlayfs sb */
> > > +       WRITE_ONCE(sb->s_wb_err, READ_ONCE(upper_sb->s_wb_err));
> > > +
> > >         return ret;
> > >  }
> > > ----------------------------8<---------------------------------
> > >
> > > [1]: https://www.spinics.net/lists/linux-api/msg41276.html
> >
> > So,
> > I think this will have bad behaviour because syncfs() twice in a row will
> > return the error twice. The reason is that normally in syncfs it calls
> > errseq_check_and_advance marking the error on the super block as seen. If we
> > copy-up the error value each time, it will break this semantic, as we do not set
> > seen on the upperdir.
> >
> > Either we need to set the seen flag on the upperdir's errseq_t, or have a sync
> > method, like:
> > diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
> > index 290983bcfbb3..4931d1797c03 100644
> > --- a/fs/overlayfs/super.c
> > +++ b/fs/overlayfs/super.c
> > @@ -259,6 +259,7 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
> >  {
> >         struct ovl_fs *ofs = sb->s_fs_info;
> >         struct super_block *upper_sb;
> > +       errseq_t src;
> >         int ret;
> >
> >         if (!ovl_upper_mnt(ofs))
> > @@ -283,6 +284,11 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
> >         ret = sync_filesystem(upper_sb);
> >         up_read(&upper_sb->s_umount);
> >
> > +       /* Propagate the error up from the upper_sb once */
> > +       src = READ_ONCE(upper_sb->s_wb_err);
> > +       if (errseq_counter(src) != errseq_counter(sb->s_wb_err))
> > +               WRITE_ONCE(sb->s_wb_err, src & ~ERRSEQ_SEEN);
> > +
> >         return ret;
> >  }
> >
> > @@ -1945,6 +1951,7 @@ static int ovl_fill_super(struct super_block *sb, void *data, int silent)
> >
> >                 sb->s_stack_depth = ovl_upper_mnt(ofs)->mnt_sb->s_stack_depth;
> >                 sb->s_time_gran = ovl_upper_mnt(ofs)->mnt_sb->s_time_gran;
> > +               sb->s_wb_err = READ_ONCE(ovl_upper_mnt(ofs)->mnt_sb->s_wb_err);
> >
> > diff --git a/lib/errseq.c b/lib/errseq.c
> > index 81f9e33aa7e7..53275c168265 100644
> > --- a/lib/errseq.c
> > +++ b/lib/errseq.c
> > @@ -204,3 +204,18 @@ int errseq_check_and_advance(errseq_t *eseq, errseq_t *since)
> >         return err;
> >  }
> >  EXPORT_SYMBOL(errseq_check_and_advance);
> > +
> > +/**
> > + * errseq_counter() - Get the value of the errseq counter
> > + * @eseq: Value being checked
> > + *
> > + * This returns the errseq_t counter value. Reminder, it can wrap because
> > + * there are only a limited number of counter bits.
> > + *
> > + * Return: The counter portion of the errseq_t.
> > + */
> > +int errseq_counter(errseq_t eseq)
> > +{
> > +       return eseq >> (ERRSEQ_SHIFT + 1);
> > +}
> >
> > This also needs some locking sprinkled on it because racing can occur with
> > sync_fs. This would be much easier if the errseq_t was 64-bits, and didn't go
> > backwards, because you could just use a simple cmpxchg64. I know that would make
> > a bunch of structures larger, and if it's just overlayfs that has to pay the tax
> > we can just sprinkle a mutex on it.
> >
> > We can always "copy-up" the errseq_t because if the upperdir's errseq_t is read,
> > and we haven't done a copy yet, we'll get it. Since this will be strictly
> > serialized a simple equivalence check will work versus actually having to deal
> > with happens before behaviour. There is still a correctness flaw here though,
> > which is exactly enough errors occur to result in it wrapping back to the same
> > value, it will break.
> >
> > By the way, how do y'all test this error handling behaviour? I didn't
> > find any automated testing for what currently exists.
> > >
> > > How about I split this into two patchsets? One, where I add the logic to copy
> > > the errseq on callbacks to fsync from the upperdir to the ovl fs superblock,
> > > and thus allowing VFS to bubble up errors, and the documentation. We can CC
> > > stable on those because I think it has an effect that's universal across
> > > all filesystems.
> > >
> > > P.S.
> > > I notice you maintain overlay tests outside of the kernel. Unfortunately, I
> > > think for this kind of test, it requires in kernel code to artificially bump the
> > > writeback error count on the upperdir, or it requires the failure injection
> > > infrastructure.
> > >
> > > Simulating this behaviour is non-trivial without in-kernel support:
> > >
> > > P1: Open(f) -> p1.fd
> > > P2: Open(f) -> p2.fd
> > > P1: syncfs(p1.fd) -> errrno
> > > P2: syncfs(p1.fd) -> 0 # This should return an error
> > >
> > >
> > > [1]: https://elixir.bootlin.com/linux/latest/source/fs/sync.c#L175
> > > [2]: https://lwn.net/Articles/722250/
> > >
> > >
> 
> Sargun (and Jeff),
> 
> Thank you for this discussion. It would be very nice to work on testing
> and fixing errseq propagation is correct on overlayfs.
> Alas, this is not what I suggested.
> 
> What I suggested was a solution only for the volatile overlay issue
> where data can vaporise without applications noticing:
> "...the very minimum is to check for errseq since mount on the fsync
>  and syncfs calls."
> 
Yeah, I was confusing the checking that VFS does on our behalf and the checking 
that we can do ourselves in the sync callback. If we return an error prior to 
the vfs checking it short-circuits that entirely.

> Do you get it? there is no pre-file state in the game, not for fsync and not 
> for syncfs.
> 
> Any single error, no matter how temporary it is and what damage it may
> or may not have caused to upper layer consistency, permanently
> invalidates the reliability of the volatile overlay, resulting in:
> Effective immediately: every fsync/syncfs returns EIO.
> Going forward: maybe implement overlay shutdown, so every access
> returns EIO.
> 
> So now that I hopefully explained myself better, I'll ask again:
> Am I wrong saying that it is very very simple to fix?
> Would you mind making that fix at the bottom of the patch series, so it can
> be easily applied to stable kernels?
> 
> Thanks,
> Amir.

I think that this should be easy enough if the semantic is such that volatile
overlayfs mounts will return EIO on syncfs on every syncfs call if the upperdir's
super block has experienced errors since the initial mount. I imagine we do not
want to make it such that if the upperdir has ever experienced errors, return
EIO on syncfs.

The one caveat that I see is that if the errseq wraps, we can silently begin 
swallowing errors again. Thus, on the first failed syncfs we should just
store a flag indicating that the volatile fs is bad, and to continue to return
EIO rather than go through the process of checking errseq_t, but that's easy
enough to write.

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-17.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,
	URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 7ED33C64E8A
	for <linux-unionfs@archiver.kernel.org>; Mon, 30 Nov 2020 18:44:59 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id 2991B20725
	for <linux-unionfs@archiver.kernel.org>; Mon, 30 Nov 2020 18:44:59 +0000 (UTC)
Authentication-Results: mail.kernel.org;
	dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="ibQlIrEf"
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S2388250AbgK3So6 (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Mon, 30 Nov 2020 13:44:58 -0500
Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:27998 "EHLO
        us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK)
        by vger.kernel.org with ESMTP id S2387718AbgK3So6 (ORCPT
        <rfc822;linux-unionfs@vger.kernel.org>);
        Mon, 30 Nov 2020 13:44:58 -0500
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;
        s=mimecast20190719; t=1606761812;
        h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
         to:to:cc:cc:mime-version:mime-version:content-type:content-type:
         in-reply-to:in-reply-to:references:references;
        bh=hkewaRffaH3MK/iLUl4Tk9n8lOvfBWoJswBpVn3Ldx0=;
        b=ibQlIrEfnjdwq2roOCCtQvIA56SMOxt56hvarpJ87AWti0mBxIhAeAux2U/hI3Iu4lr3vQ
        ohDWIya+iv6UN6VkCTk0XQQyd8cp/GSTMGxSe0IdT+BaXFhabcSYk/1uw8O0UQ+abuKxDk
        /0k/m0g/tTBregyKaKAjWM6eUiB57CA=
Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com
 [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id
 us-mta-310-OYBirpIKMo-Yuz5dOED4_A-1; Mon, 30 Nov 2020 13:43:30 -0500
X-MC-Unique: OYBirpIKMo-Yuz5dOED4_A-1
Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16])
        (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
        (No client certificate requested)
        by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 70DEE1007466;
        Mon, 30 Nov 2020 18:43:28 +0000 (UTC)
Received: from horse.redhat.com (ovpn-116-55.rdu2.redhat.com [10.10.116.55])
        by smtp.corp.redhat.com (Postfix) with ESMTP id 5EAA75C1A3;
        Mon, 30 Nov 2020 18:43:27 +0000 (UTC)
Received: by horse.redhat.com (Postfix, from userid 10451)
        id DA7FA22054F; Mon, 30 Nov 2020 13:43:26 -0500 (EST)
Date:   Mon, 30 Nov 2020 13:43:26 -0500
From:   Vivek Goyal <vgoyal@redhat.com>
To:     Sargun Dhillon <sargun@sargun.me>
Cc:     linux-unionfs@vger.kernel.org, miklos@szeredi.hu,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Amir Goldstein <amir73il@gmail.com>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel@vger.kernel.org, David Howells <dhowells@redhat.com>
Subject: Re: [PATCH v2 4/4] overlay: Add rudimentary checking of writeback
 errseq on volatile remount
Message-ID: <20201130184326.GB14328@redhat.com>
References: <20201127092058.15117-1-sargun@sargun.me>
 <20201127092058.15117-5-sargun@sargun.me>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20201127092058.15117-5-sargun@sargun.me>
X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org

On Fri, Nov 27, 2020 at 01:20:58AM -0800, Sargun Dhillon wrote:
> Volatile remounts validate the following at the moment:
>  * Has the module been reloaded / the system rebooted
>  * Has the workdir been remounted
> 
> This adds a new check for errors detected via the superblock's
> errseq_t. At mount time, the errseq_t is snapshotted to disk,
> and upon remount it's re-verified. This allows for kernel-level
> detection of errors without forcing userspace to perform a
> sync and allows for the hidden detection of writeback errors.
> 
> Signed-off-by: Sargun Dhillon <sargun@sargun.me>
> Cc: linux-fsdevel@vger.kernel.org
> Cc: linux-unionfs@vger.kernel.org
> Cc: Miklos Szeredi <miklos@szeredi.hu>
> Cc: Amir Goldstein <amir73il@gmail.com>
> Cc: Vivek Goyal <vgoyal@redhat.com>
> ---
>  fs/overlayfs/overlayfs.h | 1 +
>  fs/overlayfs/readdir.c   | 6 ++++++
>  fs/overlayfs/super.c     | 1 +
>  3 files changed, 8 insertions(+)
> 
> diff --git a/fs/overlayfs/overlayfs.h b/fs/overlayfs/overlayfs.h
> index de694ee99d7c..e8a711953b64 100644
> --- a/fs/overlayfs/overlayfs.h
> +++ b/fs/overlayfs/overlayfs.h
> @@ -85,6 +85,7 @@ struct ovl_volatile_info {
>  	 */
>  	uuid_t		ovl_boot_id;	/* Must stay first member */
>  	u64		s_instance_id;
> +	errseq_t	errseq;	/* Implemented as a u32 */
>  } __packed;
>  
>  /*
> diff --git a/fs/overlayfs/readdir.c b/fs/overlayfs/readdir.c
> index 7b66fbb20261..5795b28bb4cf 100644
> --- a/fs/overlayfs/readdir.c
> +++ b/fs/overlayfs/readdir.c
> @@ -1117,6 +1117,12 @@ static int ovl_verify_volatile_info(struct ovl_fs *ofs,
>  		return -EINVAL;
>  	}
>  
> +	err = errseq_check(&volatiledir->d_sb->s_wb_err, info.errseq);
> +	if (err) {
> +		pr_debug("Workdir filesystem reports errors: %d\n", err);

It probably will make sense to make it pr_err() instead? Will be good
to know if kernel logic detected errors.


Thanks
Vivek


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-17.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,
	URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id D3C0EC71155
	for <linux-unionfs@archiver.kernel.org>; Mon, 30 Nov 2020 19:16:53 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id 87129206ED
	for <linux-unionfs@archiver.kernel.org>; Mon, 30 Nov 2020 19:16:53 +0000 (UTC)
Authentication-Results: mail.kernel.org;
	dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="VVHd3EIg"
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S2388020AbgK3TQp (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Mon, 30 Nov 2020 14:16:45 -0500
Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:36748 "EHLO
        us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK)
        by vger.kernel.org with ESMTP id S2388228AbgK3TQm (ORCPT
        <rfc822;linux-unionfs@vger.kernel.org>);
        Mon, 30 Nov 2020 14:16:42 -0500
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;
        s=mimecast20190719; t=1606763715;
        h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
         to:to:cc:cc:mime-version:mime-version:content-type:content-type:
         in-reply-to:in-reply-to:references:references;
        bh=yG/N1QYbCe2Gv2DEXk2bQ+9lwnJX4uGO5t7uHFM1aMk=;
        b=VVHd3EIg2hQoXsCmsz4TIKrPGHolDCapMPE+pmXkKZ5jJMU4s5UME8ppf6r5MRGC755SNG
        3v7rKlDI4HXIIUoYInl18vktmZsS6OMA4XygTHv1wXAwtf3sHHihr1RjNr9HHGqSUZptmz
        UYH3iO1uEP7InIIKPHXIXG/r5xzxVbY=
Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com
 [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id
 us-mta-584-jGTEpHE7OViaf8TStxy5kg-1; Mon, 30 Nov 2020 14:15:11 -0500
X-MC-Unique: jGTEpHE7OViaf8TStxy5kg-1
Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13])
        (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
        (No client certificate requested)
        by mimecast-mx01.redhat.com (Postfix) with ESMTPS id D7B8E1842146;
        Mon, 30 Nov 2020 19:15:09 +0000 (UTC)
Received: from horse.redhat.com (ovpn-116-55.rdu2.redhat.com [10.10.116.55])
        by smtp.corp.redhat.com (Postfix) with ESMTP id AA4606086F;
        Mon, 30 Nov 2020 19:15:09 +0000 (UTC)
Received: by horse.redhat.com (Postfix, from userid 10451)
        id 2CFDB22054F; Mon, 30 Nov 2020 14:15:09 -0500 (EST)
Date:   Mon, 30 Nov 2020 14:15:09 -0500
From:   Vivek Goyal <vgoyal@redhat.com>
To:     Sargun Dhillon <sargun@sargun.me>
Cc:     linux-unionfs@vger.kernel.org, miklos@szeredi.hu,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Amir Goldstein <amir73il@gmail.com>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel@vger.kernel.org, David Howells <dhowells@redhat.com>
Subject: Re: [PATCH v2 4/4] overlay: Add rudimentary checking of writeback
 errseq on volatile remount
Message-ID: <20201130191509.GC14328@redhat.com>
References: <20201127092058.15117-1-sargun@sargun.me>
 <20201127092058.15117-5-sargun@sargun.me>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20201127092058.15117-5-sargun@sargun.me>
X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org

On Fri, Nov 27, 2020 at 01:20:58AM -0800, Sargun Dhillon wrote:
> Volatile remounts validate the following at the moment:
>  * Has the module been reloaded / the system rebooted
>  * Has the workdir been remounted
> 
> This adds a new check for errors detected via the superblock's
> errseq_t. At mount time, the errseq_t is snapshotted to disk,
> and upon remount it's re-verified. This allows for kernel-level
> detection of errors without forcing userspace to perform a
> sync and allows for the hidden detection of writeback errors.
> 
> Signed-off-by: Sargun Dhillon <sargun@sargun.me>
> Cc: linux-fsdevel@vger.kernel.org
> Cc: linux-unionfs@vger.kernel.org
> Cc: Miklos Szeredi <miklos@szeredi.hu>
> Cc: Amir Goldstein <amir73il@gmail.com>
> Cc: Vivek Goyal <vgoyal@redhat.com>
> ---
>  fs/overlayfs/overlayfs.h | 1 +
>  fs/overlayfs/readdir.c   | 6 ++++++
>  fs/overlayfs/super.c     | 1 +
>  3 files changed, 8 insertions(+)
> 
> diff --git a/fs/overlayfs/overlayfs.h b/fs/overlayfs/overlayfs.h
> index de694ee99d7c..e8a711953b64 100644
> --- a/fs/overlayfs/overlayfs.h
> +++ b/fs/overlayfs/overlayfs.h
> @@ -85,6 +85,7 @@ struct ovl_volatile_info {
>  	 */
>  	uuid_t		ovl_boot_id;	/* Must stay first member */
>  	u64		s_instance_id;
> +	errseq_t	errseq;	/* Implemented as a u32 */
>  } __packed;
>  
>  /*
> diff --git a/fs/overlayfs/readdir.c b/fs/overlayfs/readdir.c
> index 7b66fbb20261..5795b28bb4cf 100644
> --- a/fs/overlayfs/readdir.c
> +++ b/fs/overlayfs/readdir.c
> @@ -1117,6 +1117,12 @@ static int ovl_verify_volatile_info(struct ovl_fs *ofs,
>  		return -EINVAL;
>  	}
>  
> +	err = errseq_check(&volatiledir->d_sb->s_wb_err, info.errseq);
> +	if (err) {
> +		pr_debug("Workdir filesystem reports errors: %d\n", err);
> +		return -EINVAL;
> +	}
> +
>  	return 1;
>  }
>  
> diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
> index a8ee3ba4ebbd..2e473f8c75dd 100644
> --- a/fs/overlayfs/super.c
> +++ b/fs/overlayfs/super.c
> @@ -1248,6 +1248,7 @@ static int ovl_set_volatile_info(struct ovl_fs *ofs, struct dentry *volatiledir)
>  	int err;
>  	struct ovl_volatile_info info = {
>  		.s_instance_id = volatiledir->d_sb->s_instance_id,
> +		.errseq = errseq_sample(&volatiledir->d_sb->s_wb_err),

errse_sample() seems to return 0 if nobody has seen the error yet. That
means on remount we will fail. It is a false failure from our perspective
and we are not interested in knowing if somebody else has seen the
failure or not. 

Maybe we need a flag in errseq_sample() to get us current value
irrespective of the fact whether anybody has seen the error or not?

If we end up making this change, then we probably will have to somehow
mask ERRSEQ_SEEN bit in errseq_check() comparison. Because if we
sampled ->s_wb_err when nobody saw it and later by the remount time
say ERRSEQ_SEEN is set, we don't want remount to fail.

Thanks
Vivek
>  	};
>  
>  	uuid_copy(&info.ovl_boot_id, &ovl_boot_id);
> -- 
> 2.25.1
> 


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-17.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,
	URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id EA710C63777
	for <linux-unionfs@archiver.kernel.org>; Mon, 30 Nov 2020 19:35:31 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id 8D5BD2067C
	for <linux-unionfs@archiver.kernel.org>; Mon, 30 Nov 2020 19:35:31 +0000 (UTC)
Authentication-Results: mail.kernel.org;
	dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="ekk9T4ek"
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1729608AbgK3TfQ (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Mon, 30 Nov 2020 14:35:16 -0500
Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:46624 "EHLO
        us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK)
        by vger.kernel.org with ESMTP id S1729034AbgK3TfQ (ORCPT
        <rfc822;linux-unionfs@vger.kernel.org>);
        Mon, 30 Nov 2020 14:35:16 -0500
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;
        s=mimecast20190719; t=1606764829;
        h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
         to:to:cc:cc:mime-version:mime-version:content-type:content-type:
         in-reply-to:in-reply-to:references:references;
        bh=hZA0rkeTs6RdwOmnFqZ5dPAN2kNIQ0c32hZuL9Bwf40=;
        b=ekk9T4ekTVPUPIJDJyW3z0n62X64NwgaB9Dukz7ztPEBY9d2ySJ7DlEMWgm1dvMy2j3Cuf
        v6PaNRm+QNE8sfgELgbZZwG14Kacb4YUkruxVtvNN1fXkmMZQRnz/dwBUZOWyyXFCfrU0X
        FKCLuGv6Yho67IHnVBOI6UIWo4xyVVw=
Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com
 [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id
 us-mta-446-jh-ZOA9vMl-C2R8bKmsnXg-1; Mon, 30 Nov 2020 14:33:45 -0500
X-MC-Unique: jh-ZOA9vMl-C2R8bKmsnXg-1
Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12])
        (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
        (No client certificate requested)
        by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 10992101AFC9;
        Mon, 30 Nov 2020 19:33:43 +0000 (UTC)
Received: from horse.redhat.com (ovpn-116-55.rdu2.redhat.com [10.10.116.55])
        by smtp.corp.redhat.com (Postfix) with ESMTP id E716960C67;
        Mon, 30 Nov 2020 19:33:42 +0000 (UTC)
Received: by horse.redhat.com (Postfix, from userid 10451)
        id 75C9C22054F; Mon, 30 Nov 2020 14:33:42 -0500 (EST)
Date:   Mon, 30 Nov 2020 14:33:42 -0500
From:   Vivek Goyal <vgoyal@redhat.com>
To:     Sargun Dhillon <sargun@sargun.me>
Cc:     linux-unionfs@vger.kernel.org, miklos@szeredi.hu,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Amir Goldstein <amir73il@gmail.com>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel@vger.kernel.org, David Howells <dhowells@redhat.com>
Subject: Re: [PATCH v2 4/4] overlay: Add rudimentary checking of writeback
 errseq on volatile remount
Message-ID: <20201130193342.GD14328@redhat.com>
References: <20201127092058.15117-1-sargun@sargun.me>
 <20201127092058.15117-5-sargun@sargun.me>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20201127092058.15117-5-sargun@sargun.me>
X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org

On Fri, Nov 27, 2020 at 01:20:58AM -0800, Sargun Dhillon wrote:
> Volatile remounts validate the following at the moment:
>  * Has the module been reloaded / the system rebooted
>  * Has the workdir been remounted
> 
> This adds a new check for errors detected via the superblock's
> errseq_t. At mount time, the errseq_t is snapshotted to disk,
> and upon remount it's re-verified. This allows for kernel-level
> detection of errors without forcing userspace to perform a
> sync and allows for the hidden detection of writeback errors.
> 
> Signed-off-by: Sargun Dhillon <sargun@sargun.me>
> Cc: linux-fsdevel@vger.kernel.org
> Cc: linux-unionfs@vger.kernel.org
> Cc: Miklos Szeredi <miklos@szeredi.hu>
> Cc: Amir Goldstein <amir73il@gmail.com>
> Cc: Vivek Goyal <vgoyal@redhat.com>
> ---
>  fs/overlayfs/overlayfs.h | 1 +
>  fs/overlayfs/readdir.c   | 6 ++++++
>  fs/overlayfs/super.c     | 1 +
>  3 files changed, 8 insertions(+)
> 
> diff --git a/fs/overlayfs/overlayfs.h b/fs/overlayfs/overlayfs.h
> index de694ee99d7c..e8a711953b64 100644
> --- a/fs/overlayfs/overlayfs.h
> +++ b/fs/overlayfs/overlayfs.h
> @@ -85,6 +85,7 @@ struct ovl_volatile_info {
>  	 */
>  	uuid_t		ovl_boot_id;	/* Must stay first member */
>  	u64		s_instance_id;
> +	errseq_t	errseq;	/* Implemented as a u32 */
>  } __packed;
>  
>  /*
> diff --git a/fs/overlayfs/readdir.c b/fs/overlayfs/readdir.c
> index 7b66fbb20261..5795b28bb4cf 100644
> --- a/fs/overlayfs/readdir.c
> +++ b/fs/overlayfs/readdir.c
> @@ -1117,6 +1117,12 @@ static int ovl_verify_volatile_info(struct ovl_fs *ofs,
>  		return -EINVAL;
>  	}
>  
> +	err = errseq_check(&volatiledir->d_sb->s_wb_err, info.errseq);

Might be a stupid question. Will ask anyway.

But what protects against wrapping of counter. IOW, Say we stored info.errseq
value as A. It is possible that bunch of errors occurred and at remount
time ->s_wb_err is back to A and we pass the check. (Despite the fact lots
of errors have occurred since we sampled).

Thanks
Vivek


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-17.2 required=3.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,
	INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,
	USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id C88B8C64E7B
	for <linux-unionfs@archiver.kernel.org>; Tue,  1 Dec 2020 11:10:34 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id 39F64206D8
	for <linux-unionfs@archiver.kernel.org>; Tue,  1 Dec 2020 11:10:34 +0000 (UTC)
Authentication-Results: mail.kernel.org;
	dkim=pass (1024-bit key) header.d=sargun.me header.i=@sargun.me header.b="SyK04Egj"
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1727711AbgLALKS (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Tue, 1 Dec 2020 06:10:18 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42566 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1726689AbgLALKS (ORCPT
        <rfc822;linux-unionfs@vger.kernel.org>);
        Tue, 1 Dec 2020 06:10:18 -0500
Received: from mail-io1-xd41.google.com (mail-io1-xd41.google.com [IPv6:2607:f8b0:4864:20::d41])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B5069C0613D2
        for <linux-unionfs@vger.kernel.org>; Tue,  1 Dec 2020 03:09:32 -0800 (PST)
Received: by mail-io1-xd41.google.com with SMTP id d8so1135847ioc.13
        for <linux-unionfs@vger.kernel.org>; Tue, 01 Dec 2020 03:09:32 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=sargun.me; s=google;
        h=date:from:to:cc:subject:message-id:references:mime-version
         :content-disposition:in-reply-to:user-agent;
        bh=sTzp58gbljNxbSLzTV21VHKsqqlFtduRWMzXhmCTCOY=;
        b=SyK04Egjz+OVF1WLPwHKwhFjVryF6xeuyA+ds/fMne7y/rzLhkAtC7W24lP06UeoaJ
         D7Ner/NnQLJRk3CqG3fBdri+4h+lzNWGHtwcBVtz+0BudVigIYctQATJQbGVEWvsUYDC
         Sv+tYflDvGshKNDTdb/BQz7+FsVANDiStTHss=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:date:from:to:cc:subject:message-id:references
         :mime-version:content-disposition:in-reply-to:user-agent;
        bh=sTzp58gbljNxbSLzTV21VHKsqqlFtduRWMzXhmCTCOY=;
        b=L1KEb/YdkE0Gutwjs7zUAWOYTxabYdrw3AqinTCkMJVDt7gwrqR2jW3BrbyYhnNrMK
         A6Joo2HhtO/6l/ZbIGl/EWdvGJBk3ji3pxHPH4YRFKVZfiGm/IemuY42ozw7vKgSCYiE
         yyAY1Nig2w19rXTG9kRRSRlct6eBO8+0ob5LXD/roQzKQPi4RiFoDcYM9331cWkkkH1C
         VrLZt1cUyh2BoaxZdbPfscKDVKrdl4pnJGpWKfug0oTW6b6902rhYj5XX9hpSHKVfQZW
         t+sG+weW7i+MZolbKJEzkAW05FHz61BSvAD77Pm2UBleF8gD+Y8uWag0jEIhncsfR/y9
         3rBA==
X-Gm-Message-State: AOAM531Ld1MKK6scOC567Pqqt3gMZYwxTyzC/L+DgoQ4ir31USPF68ME
        Eqt1qTgE8K/LaMWDdNY+9Rm8+Q==
X-Google-Smtp-Source: ABdhPJwZwhhH89rhjqazR01KppkOrZzbuuQ5zal66ptOEPyp0qtF2redlBa5ASuHUHINW+so8otyXg==
X-Received: by 2002:a5e:8206:: with SMTP id l6mr1862169iom.126.1606820971810;
        Tue, 01 Dec 2020 03:09:31 -0800 (PST)
Received: from ircssh-2.c.rugged-nimbus-611.internal (80.60.198.104.bc.googleusercontent.com. [104.198.60.80])
        by smtp.gmail.com with ESMTPSA id v62sm640719iod.50.2020.12.01.03.09.30
        (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256);
        Tue, 01 Dec 2020 03:09:31 -0800 (PST)
Date:   Tue, 1 Dec 2020 11:09:29 +0000
From:   Sargun Dhillon <sargun@sargun.me>
To:     Amir Goldstein <amir73il@gmail.com>
Cc:     Jeff Layton <jlayton@redhat.com>,
        overlayfs <linux-unionfs@vger.kernel.org>,
        Miklos Szeredi <miklos@szeredi.hu>,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Vivek Goyal <vgoyal@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel <linux-fsdevel@vger.kernel.org>,
        David Howells <dhowells@redhat.com>
Subject: Re: [PATCH v2 2/4] overlay: Document current outstanding shortcoming
 of volatile
Message-ID: <20201201110928.GA24837@ircssh-2.c.rugged-nimbus-611.internal>
References: <20201127092058.15117-1-sargun@sargun.me>
 <20201127092058.15117-3-sargun@sargun.me>
 <CAOQ4uxgaLuLb+f6WCMvmKHNTELvcvN8C5_u=t5hhoGT8Op7QuQ@mail.gmail.com>
 <20201127221154.GA23383@ircssh-2.c.rugged-nimbus-611.internal>
 <1338a059d03db0e85cf3f3234fd33434a45606c6.camel@redhat.com>
 <20201128044530.GA28230@ircssh-2.c.rugged-nimbus-611.internal>
 <CAOQ4uxjT6FF03Sq3qXuqDcqJQnzQq2dD_XVbuj_Fb9A2Ag585w@mail.gmail.com>
 <20201128085227.GB28230@ircssh-2.c.rugged-nimbus-611.internal>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20201128085227.GB28230@ircssh-2.c.rugged-nimbus-611.internal>
User-Agent: Mutt/1.9.4 (2018-02-28)
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org

On Sat, Nov 28, 2020 at 08:52:27AM +0000, Sargun Dhillon wrote:
> On Sat, Nov 28, 2020 at 09:12:03AM +0200, Amir Goldstein wrote:
> > On Sat, Nov 28, 2020 at 6:45 AM Sargun Dhillon <sargun@sargun.me> wrote:
> > >
> > > On Fri, Nov 27, 2020 at 09:01:07PM -0500, Jeff Layton wrote:
> > > > On Fri, 2020-11-27 at 22:11 +0000, Sargun Dhillon wrote:
> > > > > On Fri, Nov 27, 2020 at 02:52:52PM +0200, Amir Goldstein wrote:
> > > > > > On Fri, Nov 27, 2020 at 11:21 AM Sargun Dhillon <sargun@sargun.me> wrote:
> > > > > > >
> > > > > > > This documents behaviour that was discussed in a thread about the volatile
> > > > > > > feature. Specifically, how failures can go hidden from asynchronous writes
> > > > > > > (such as from mmap, or writes that are not immediately flushed to the
> > > > > > > filesystem). Although we pass through calls like msync, fallocate, and
> > > > > > > write, and will still return errors on those, it doesn't guarantee all
> > > > > > > kinds of errors will happen at those times, and thus may hide errors.
> > > > > > >
> > > > > > > In the future, we can add error checking to all interactions with the
> > > > > > > upperdir, and pass through errseq_t from the upperdir on mappings,
> > > > > > > and other interactions with the filesystem[1].
> > > > > > >
> > > > > > > [1]: https://lore.kernel.org/linux-unionfs/20201116045758.21774-1-sargun@sargun.me/T/#m7d501f375e031056efad626e471a1392dd3aad33
> > > > > > >
> > > > > > > Signed-off-by: Sargun Dhillon <sargun@sargun.me>
> > > > > > > Cc: linux-fsdevel@vger.kernel.org
> > > > > > > Cc: linux-unionfs@vger.kernel.org
> > > > > > > Cc: Miklos Szeredi <miklos@szeredi.hu>
> > > > > > > Cc: Amir Goldstein <amir73il@gmail.com>
> > > > > > > Cc: Vivek Goyal <vgoyal@redhat.com>
> > > > > > > ---
> > > > > > >  Documentation/filesystems/overlayfs.rst | 6 +++++-
> > > > > > >  1 file changed, 5 insertions(+), 1 deletion(-)
> > > > > > >
> > > > > > > diff --git a/Documentation/filesystems/overlayfs.rst b/Documentation/filesystems/overlayfs.rst
> > > > > > > index 580ab9a0fe31..c6e30c1bc2f2 100644
> > > > > > > --- a/Documentation/filesystems/overlayfs.rst
> > > > > > > +++ b/Documentation/filesystems/overlayfs.rst
> > > > > > > @@ -570,7 +570,11 @@ Volatile mount
> > > > > > >  This is enabled with the "volatile" mount option.  Volatile mounts are not
> > > > > > >  guaranteed to survive a crash.  It is strongly recommended that volatile
> > > > > > >  mounts are only used if data written to the overlay can be recreated
> > > > > > > -without significant effort.
> > > > > > > +without significant effort.  In addition to this, the sync family of syscalls
> > > > > > > +are not sufficient to determine whether a write failed as sync calls are
> > > > > > > +omitted.  For this reason, it is important that the filesystem used by the
> > > > > > > +upperdir handles failure in a fashion that's suitable for the user.  For
> > > > > > > +example, upon detecting a fault, ext4 can be configured to panic.
> > > > > > >
> > > > > >
> > > > > > Reading this now, I think I may have wrongly analysed the issue.
> > > > > > Specifically, when I wrote that the very minimum is to document the
> > > > > > issue, it was under the assumption that a proper fix is hard.
> > > > > > I think I was wrong and that the very minimum is to check for errseq
> > > > > > since mount on the fsync and syncfs calls.
> > > > > >
> > > > > > Why? first of all because it is very very simple and goes a long way to
> > > > > > fix the broken contract with applications, not the contract about durability
> > > > > > obviously, but the contract about write+fsync+read expects to find the written
> > > > > > data (during the same mount era).
> > > > > >
> > > > > > Second, because the sentence you added above is hard for users to understand
> > > > > > and out of context. If we properly handle the writeback error in fsync/syncfs,
> > > > > > then this sentence becomes irrelevant.
> > > > > > The fact that ext4 can lose data if application did not fsync after
> > > > > > write is true
> > > > > > for volatile as well as non-volatile and it is therefore not relevant
> > > > > > in the context
> > > > > > of overlayfs documentation at all.
> > > > > >
> > > > > > Am I wrong saying that it is very very simple to fix?
> > > > > > Would you mind making that fix at the bottom of the patch series, so it can
> > > > > > be easily applied to stable kernels?
> > > > > >
> > > > > > Thanks,
> > > > > > Amir.
> > > > >
> > > > > I'm not sure it's so easy. In VFS, there are places where the superblock's
> > > > > errseq is checked[1]. AFAIK, that's going to check "our" errseq, and not the
> > > > > errseq of the real corresponding real file's superblock. One solution might be
> > > > > as part of all these callbacks we set our errseq to the errseq of the filesystem
> > > > > that the upperdir, and then rely on VFS's checking.
> > > > >
> > > > > I'm having a hard time figuring out how to deal with the per-mapping based
> > > > > error tracking. It seems like this infrastructure was only partially completed
> > > > > by Jeff Layton[2]. I don't now how it's actually supposed to work right now,
> > > > > as not all of his patches landed.
> > > > >
> > > >
> > > > The patches in the series were all merged, but we ended up going with a
> > > > simpler solution [1] than the first series I posted. Instead of plumbing
> > > > the errseq_t handling down into sync_fs, we did it in the syscall
> > > > wrapper.
> > > Jeff,
> > >
> > > Thanks for replying. I'm still a little confused as to what the
> > > per-address_space wb_err. It seems like we should implement the
> > > flush method, like so:
> > > diff --git a/fs/overlayfs/file.c b/fs/overlayfs/file.c
> > > index efccb7c1f9bc..32e5bc0aacd6 100644
> > > --- a/fs/overlayfs/file.c
> > > +++ b/fs/overlayfs/file.c
> > > @@ -787,6 +787,16 @@ static loff_t ovl_remap_file_range(struct file *file_in, loff_t pos_in,
> > >                             remap_flags, op);
> > >  }
> > >
> > > +static int ovl_flush(struct file *file, fl_owner_t id)
> > > +{
> > > +       struct file *realfile = file->private_data;
> > > +
> > > +       if (real_file->f_op->flush)
> > > +               return real_file->f_op->flush(filp, id);
> > > +
> > > +       return 0;
> > > +}
> > > +
> > >  const struct file_operations ovl_file_operations = {
> > >         .open           = ovl_open,
> > >         .release        = ovl_release,
> > > @@ -798,6 +808,7 @@ const struct file_operations ovl_file_operations = {
> > >         .fallocate      = ovl_fallocate,
> > >         .fadvise        = ovl_fadvise,
> > >         .unlocked_ioctl = ovl_ioctl,
> > > +       .flush          = ovl_flush,
> > >  #ifdef CONFIG_COMPAT
> > >         .compat_ioctl   = ovl_compat_ioctl,
> > >  #endif
> > >
> > >
> > > >
> > > > I think the tricky part here is that there is no struct file plumbed
> > > > into ->sync_fs, so you don't have an errseq_t cursor to work with by the
> > > > time that gets called.
> > > >
> > > > What may be easiest is to just propagate the s_wb_err value from the
> > > > upper_sb to the overlayfs superblock in ovl_sync_fs(). That should get
> > > > called before the errseq_check_and_advance in the syncfs syscall wrapper
> > > > and should ensure that syncfs() calls against the overlayfs sb see any
> > > > errors that occurred on the upper_sb.
> > > >
> > > > Something like this maybe? Totally untested of course. May also need to
> > > > think about how to ensure ordering between racing syncfs calls too
> > > > (don't really want the s_wb_err going "backward"):
> > > >
> > > > ----------------------------8<---------------------------------
> > > > $ git diff
> > > > diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
> > > > index 290983bcfbb3..d725705abdac 100644
> > > > --- a/fs/overlayfs/super.c
> > > > +++ b/fs/overlayfs/super.c
> > > > @@ -283,6 +283,9 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
> > > >         ret = sync_filesystem(upper_sb);
> > > >         up_read(&upper_sb->s_umount);
> > > >
> > > > +       /* Propagate s_wb_err from upper_sb to overlayfs sb */
> > > > +       WRITE_ONCE(sb->s_wb_err, READ_ONCE(upper_sb->s_wb_err));
> > > > +
> > > >         return ret;
> > > >  }
> > > > ----------------------------8<---------------------------------
> > > >
> > > > [1]: https://www.spinics.net/lists/linux-api/msg41276.html
> > >
> > > So,
> > > I think this will have bad behaviour because syncfs() twice in a row will
> > > return the error twice. The reason is that normally in syncfs it calls
> > > errseq_check_and_advance marking the error on the super block as seen. If we
> > > copy-up the error value each time, it will break this semantic, as we do not set
> > > seen on the upperdir.
> > >
> > > Either we need to set the seen flag on the upperdir's errseq_t, or have a sync
> > > method, like:
> > > diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
> > > index 290983bcfbb3..4931d1797c03 100644
> > > --- a/fs/overlayfs/super.c
> > > +++ b/fs/overlayfs/super.c
> > > @@ -259,6 +259,7 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
> > >  {
> > >         struct ovl_fs *ofs = sb->s_fs_info;
> > >         struct super_block *upper_sb;
> > > +       errseq_t src;
> > >         int ret;
> > >
> > >         if (!ovl_upper_mnt(ofs))
> > > @@ -283,6 +284,11 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
> > >         ret = sync_filesystem(upper_sb);
> > >         up_read(&upper_sb->s_umount);
> > >
> > > +       /* Propagate the error up from the upper_sb once */
> > > +       src = READ_ONCE(upper_sb->s_wb_err);
> > > +       if (errseq_counter(src) != errseq_counter(sb->s_wb_err))
> > > +               WRITE_ONCE(sb->s_wb_err, src & ~ERRSEQ_SEEN);
> > > +
> > >         return ret;
> > >  }
> > >
> > > @@ -1945,6 +1951,7 @@ static int ovl_fill_super(struct super_block *sb, void *data, int silent)
> > >
> > >                 sb->s_stack_depth = ovl_upper_mnt(ofs)->mnt_sb->s_stack_depth;
> > >                 sb->s_time_gran = ovl_upper_mnt(ofs)->mnt_sb->s_time_gran;
> > > +               sb->s_wb_err = READ_ONCE(ovl_upper_mnt(ofs)->mnt_sb->s_wb_err);
> > >
> > > diff --git a/lib/errseq.c b/lib/errseq.c
> > > index 81f9e33aa7e7..53275c168265 100644
> > > --- a/lib/errseq.c
> > > +++ b/lib/errseq.c
> > > @@ -204,3 +204,18 @@ int errseq_check_and_advance(errseq_t *eseq, errseq_t *since)
> > >         return err;
> > >  }
> > >  EXPORT_SYMBOL(errseq_check_and_advance);
> > > +
> > > +/**
> > > + * errseq_counter() - Get the value of the errseq counter
> > > + * @eseq: Value being checked
> > > + *
> > > + * This returns the errseq_t counter value. Reminder, it can wrap because
> > > + * there are only a limited number of counter bits.
> > > + *
> > > + * Return: The counter portion of the errseq_t.
> > > + */
> > > +int errseq_counter(errseq_t eseq)
> > > +{
> > > +       return eseq >> (ERRSEQ_SHIFT + 1);
> > > +}
> > >
> > > This also needs some locking sprinkled on it because racing can occur with
> > > sync_fs. This would be much easier if the errseq_t was 64-bits, and didn't go
> > > backwards, because you could just use a simple cmpxchg64. I know that would make
> > > a bunch of structures larger, and if it's just overlayfs that has to pay the tax
> > > we can just sprinkle a mutex on it.
> > >
> > > We can always "copy-up" the errseq_t because if the upperdir's errseq_t is read,
> > > and we haven't done a copy yet, we'll get it. Since this will be strictly
> > > serialized a simple equivalence check will work versus actually having to deal
> > > with happens before behaviour. There is still a correctness flaw here though,
> > > which is exactly enough errors occur to result in it wrapping back to the same
> > > value, it will break.
> > >
> > > By the way, how do y'all test this error handling behaviour? I didn't
> > > find any automated testing for what currently exists.
> > > >
> > > > How about I split this into two patchsets? One, where I add the logic to copy
> > > > the errseq on callbacks to fsync from the upperdir to the ovl fs superblock,
> > > > and thus allowing VFS to bubble up errors, and the documentation. We can CC
> > > > stable on those because I think it has an effect that's universal across
> > > > all filesystems.
> > > >
> > > > P.S.
> > > > I notice you maintain overlay tests outside of the kernel. Unfortunately, I
> > > > think for this kind of test, it requires in kernel code to artificially bump the
> > > > writeback error count on the upperdir, or it requires the failure injection
> > > > infrastructure.
> > > >
> > > > Simulating this behaviour is non-trivial without in-kernel support:
> > > >
> > > > P1: Open(f) -> p1.fd
> > > > P2: Open(f) -> p2.fd
> > > > P1: syncfs(p1.fd) -> errrno
> > > > P2: syncfs(p1.fd) -> 0 # This should return an error
> > > >
> > > >
> > > > [1]: https://elixir.bootlin.com/linux/latest/source/fs/sync.c#L175
> > > > [2]: https://lwn.net/Articles/722250/
> > > >
> > > >
> > 
> > Sargun (and Jeff),
> > 
> > Thank you for this discussion. It would be very nice to work on testing
> > and fixing errseq propagation is correct on overlayfs.
> > Alas, this is not what I suggested.
> > 
> > What I suggested was a solution only for the volatile overlay issue
> > where data can vaporise without applications noticing:
> > "...the very minimum is to check for errseq since mount on the fsync
> >  and syncfs calls."
> > 
> Yeah, I was confusing the checking that VFS does on our behalf and the checking 
> that we can do ourselves in the sync callback. If we return an error prior to 
> the vfs checking it short-circuits that entirely.
> 
> > Do you get it? there is no pre-file state in the game, not for fsync and not 
> > for syncfs.
> > 
> > Any single error, no matter how temporary it is and what damage it may
> > or may not have caused to upper layer consistency, permanently
> > invalidates the reliability of the volatile overlay, resulting in:
> > Effective immediately: every fsync/syncfs returns EIO.
> > Going forward: maybe implement overlay shutdown, so every access
> > returns EIO.
> > 
> > So now that I hopefully explained myself better, I'll ask again:
> > Am I wrong saying that it is very very simple to fix?
> > Would you mind making that fix at the bottom of the patch series, so it can
> > be easily applied to stable kernels?
> > 
> > Thanks,
> > Amir.
> 
> I think that this should be easy enough if the semantic is such that volatile
> overlayfs mounts will return EIO on syncfs on every syncfs call if the upperdir's
> super block has experienced errors since the initial mount. I imagine we do not
> want to make it such that if the upperdir has ever experienced errors, return
> EIO on syncfs.
> 
> The one caveat that I see is that if the errseq wraps, we can silently begin 
> swallowing errors again. Thus, on the first failed syncfs we should just
> store a flag indicating that the volatile fs is bad, and to continue to return
> EIO rather than go through the process of checking errseq_t, but that's easy
> enough to write.

It turns out this is much more complicated than I initially thought. I'm not 
entirely sure why VFS even defines sync_fs (on the superblock) as returning an 
int. The way that sync_fs works is:

sys_syncfs ->
  sync_filesystem(sb) ->
    __sync_filesystem(sb, N) ->
      sb->s_op->sync_fs
      /* __sync_blockdev has its own writeback error checking */
      __sync_blockdev
  errseq_check_and_advance(sb...)

__sync_filesystem calls the sync_fs callback on the superblock's superblock
operations:

static int __sync_filesystem(struct super_block *sb, int wait)
{
	if (wait)
		sync_inodes_sb(sb);
	else
		writeback_inodes_sb(sb, WB_REASON_SYNC);

	if (sb->s_op->sync_fs)
		sb->s_op->sync_fs(sb, wait);
	return __sync_blockdev(sb->s_bdev, wait);
}

But it completely discards the result. On the other hand, fsync, and fdatasync
exhibit different behaviour:

sys_fdatasync ->
  do_fsync
   (see below)

sys_fsync ->
  do_fsync ->
    vfs_fsync ->
      vfs_fsync_range ->
        return file->f_op->fsync
----


syncfs seems to enforce the semantics laid out by VFS[1]. Specifically the 
statement:
  When there is an error during writeback, they expect that error to be reported 
  when a file sync request is made. After an error has been reported on one 
  request, subsequent requests on the same file descriptor should return 0, unless 
  further writeback errors have occurred since the previous file syncronization.

This is enforced by the errseq_check_and_advance logic. We can hack around this
logic by resetting the errset (setting the error on it) every time we get the
sync_fs callback, but that to me seems wrong. FWIW, implementing this behaviour
for fdatasync, and fsync is easier, because the error is bubbled up from the
filesystem to the VFS. I don't actually think this is a good idea because
it seems like this sync_fs behaviour is a bit...not neccessarily what all
filesystems expect. For example, btrfs_sync_fs returns an error if it
is unable to finish the current transaction. Nowhere in the btrfs code
does it set the errseq on the superblock if this fails.

I think we have a couple paths forward:
1. Change the semantic of sync_fs so the error is always bubbled up if
   it returns a non-zero value. If we do this, we have to decide whether
   or not we would _also_ call errseq_check_and_advance on the SB,
   or leave that to a subsequent call.
2. Have overlayfs forcefully set an error on the superblock on every
   callback to sync_fs. This seems ugly, but I wrote a little patch,
   and it seems to solve the problem for all the fsync / fdatasync /
   sync / syncfs variants without having to do plumbing in VFS.
3. Choose a different set of semantics for how we want to handle
   errors in volatile mounts.

[1]: https://www.kernel.org/doc/html/v5.9/filesystems/vfs.html?highlight=handling%20errors%20during%20writeback#handling-errors-during-writeback

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-2.7 required=3.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,
	HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS
	autolearn=no autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 805FBC64E7B
	for <linux-unionfs@archiver.kernel.org>; Tue,  1 Dec 2020 11:30:15 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id 0AC0C20705
	for <linux-unionfs@archiver.kernel.org>; Tue,  1 Dec 2020 11:30:14 +0000 (UTC)
Authentication-Results: mail.kernel.org;
	dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="bflk46O3"
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1729885AbgLALaO (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Tue, 1 Dec 2020 06:30:14 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45828 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1729654AbgLALaM (ORCPT
        <rfc822;linux-unionfs@vger.kernel.org>);
        Tue, 1 Dec 2020 06:30:12 -0500
Received: from mail-io1-xd41.google.com (mail-io1-xd41.google.com [IPv6:2607:f8b0:4864:20::d41])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B9943C0613CF;
        Tue,  1 Dec 2020 03:29:32 -0800 (PST)
Received: by mail-io1-xd41.google.com with SMTP id z136so1230277iof.3;
        Tue, 01 Dec 2020 03:29:32 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20161025;
        h=mime-version:references:in-reply-to:from:date:message-id:subject:to
         :cc;
        bh=h0FAKq+IIQm8W2FrWhxqoTDybYcuiTh1HtIoPAph5kY=;
        b=bflk46O3dXcllU4nhRwv7/Jue+8DmW9EVwKFxWR99k5mXkqxKMEEYPBphLYYWw0Cvz
         e+hiIkBUuztDHLrWvOUBSR5PoVbUVQQ8lpBYIrCTpXxzN8QLY7nqCkjLFLuP5EqpFymM
         xa73uZvgh8WNtc8QzndfcCChhPxxGTgxLaGkJ+5GpqOekiiNllRGd3dYgbN/Uxe3KydD
         hMi+k2RRRlhlt/PFMsB68pDvqsfalPVYryyFQBQYp4P2wD9N37na6GhRBgmYgKFSpkfU
         nGdI2ShYIJcCUnk0VNJoNe8tw0DBN4fl71h6BzpTN/AokbVi6yLRnE3GwzIGohcMrRtk
         wRpQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:mime-version:references:in-reply-to:from:date
         :message-id:subject:to:cc;
        bh=h0FAKq+IIQm8W2FrWhxqoTDybYcuiTh1HtIoPAph5kY=;
        b=TzQyUqaHRwGFkb8HPERQoRFAIXiSFIiGQqb8S+2ZZlzq0iM1S3RUjypeVOTvJLzeJO
         +ml7DOgKbNLpQKDjnDoN69CKaubQ3PWqR2W5O9kAqkV4z61cYF1jCIeUko2tWIl0aSfD
         L5bmNgHvHdLLRfK+u1ImoSvK4WtFqat/I/12uPWRAvJc+HJLvLyBhnVmd0X5LUO6aJuE
         R4m9lF9SCdBTuKGZModAhropc3PkipTjaWB6DSSuNevY/J7n47mveks+jBDu86LTNwkk
         tYVQmdNxMRg0kYaeB3MVrjETCZYjdFWeBbkpz+h7IAIcQ/xJC3Bwr7yV0+RpCd8T9bCT
         pOlg==
X-Gm-Message-State: AOAM530kp6O1ghLwQKEfNBzpVbDcoSVgTDPWjxKnbh2eobgbaumU6ZXi
        lhEmGw8gV8sKuvSdLMwAHRuc66lp3I9zU6PsmBM=
X-Google-Smtp-Source: ABdhPJwyjIBCMOOKWu0QFSN9gpKyfr16ncY6UqgwDwb4+Zu0lzdpvtkiIjXS8/UlnJ/VHtTegWgvHC93lBCEIEGkjJ4=
X-Received: by 2002:a02:8c2:: with SMTP id 185mr2218799jac.120.1606822172093;
 Tue, 01 Dec 2020 03:29:32 -0800 (PST)
MIME-Version: 1.0
References: <20201127092058.15117-1-sargun@sargun.me> <20201127092058.15117-3-sargun@sargun.me>
 <CAOQ4uxgaLuLb+f6WCMvmKHNTELvcvN8C5_u=t5hhoGT8Op7QuQ@mail.gmail.com>
 <20201127221154.GA23383@ircssh-2.c.rugged-nimbus-611.internal>
 <1338a059d03db0e85cf3f3234fd33434a45606c6.camel@redhat.com>
 <20201128044530.GA28230@ircssh-2.c.rugged-nimbus-611.internal>
 <CAOQ4uxjT6FF03Sq3qXuqDcqJQnzQq2dD_XVbuj_Fb9A2Ag585w@mail.gmail.com>
 <20201128085227.GB28230@ircssh-2.c.rugged-nimbus-611.internal> <20201201110928.GA24837@ircssh-2.c.rugged-nimbus-611.internal>
In-Reply-To: <20201201110928.GA24837@ircssh-2.c.rugged-nimbus-611.internal>
From:   Amir Goldstein <amir73il@gmail.com>
Date:   Tue, 1 Dec 2020 13:29:21 +0200
Message-ID: <CAOQ4uxiVLY14=3zDwcRysw69=EN=2vVq5y8JP0Q72Cz95qJ6ng@mail.gmail.com>
Subject: Re: [PATCH v2 2/4] overlay: Document current outstanding shortcoming
 of volatile
To:     Sargun Dhillon <sargun@sargun.me>
Cc:     Jeff Layton <jlayton@redhat.com>,
        overlayfs <linux-unionfs@vger.kernel.org>,
        Miklos Szeredi <miklos@szeredi.hu>,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Vivek Goyal <vgoyal@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel <linux-fsdevel@vger.kernel.org>,
        David Howells <dhowells@redhat.com>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org

> syncfs seems to enforce the semantics laid out by VFS[1]. Specifically the
> statement:
>   When there is an error during writeback, they expect that error to be reported
>   when a file sync request is made. After an error has been reported on one
>   request, subsequent requests on the same file descriptor should return 0, unless
>   further writeback errors have occurred since the previous file syncronization.
>
> This is enforced by the errseq_check_and_advance logic. We can hack around this
> logic by resetting the errset (setting the error on it) every time we get the
> sync_fs callback, but that to me seems wrong. FWIW, implementing this behaviour
> for fdatasync, and fsync is easier, because the error is bubbled up from the
> filesystem to the VFS. I don't actually think this is a good idea because
> it seems like this sync_fs behaviour is a bit...not neccessarily what all
> filesystems expect. For example, btrfs_sync_fs returns an error if it
> is unable to finish the current transaction. Nowhere in the btrfs code
> does it set the errseq on the superblock if this fails.
>
> I think we have a couple paths forward:
> 1. Change the semantic of sync_fs so the error is always bubbled up if
>    it returns a non-zero value. If we do this, we have to decide whether
>    or not we would _also_ call errseq_check_and_advance on the SB,
>    or leave that to a subsequent call.
> 2. Have overlayfs forcefully set an error on the superblock on every
>    callback to sync_fs. This seems ugly, but I wrote a little patch,
>    and it seems to solve the problem for all the fsync / fdatasync /
>    sync / syncfs variants without having to do plumbing in VFS.
> 3. Choose a different set of semantics for how we want to handle
>    errors in volatile mounts.
>

IMO it is best if you post your patch to fix volatile overlayfs, because
it seems to me that the volatile overlayfs issue is worse than generic
sync_fs issues and it's good if we had a small backportable patch
even if a bit ugly.

Later you can pursue the sync_fs semantic fixes if you wish they
do not contradict the fix in overlayfs, just are just a way to remove
a hack.

Thanks,
Amir.

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,FAKE_REPLY_C,HEADER_FROM_DIFFERENT_DOMAINS,
	INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,
	URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 45A2DC64E7A
	for <linux-unionfs@archiver.kernel.org>; Tue,  1 Dec 2020 11:57:35 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id D14AE2080A
	for <linux-unionfs@archiver.kernel.org>; Tue,  1 Dec 2020 11:57:34 +0000 (UTC)
Authentication-Results: mail.kernel.org;
	dkim=pass (1024-bit key) header.d=sargun.me header.i=@sargun.me header.b="aYyYXHVd"
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S2389958AbgLAL5N (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Tue, 1 Dec 2020 06:57:13 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50432 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S2387433AbgLAL5N (ORCPT
        <rfc822;linux-unionfs@vger.kernel.org>);
        Tue, 1 Dec 2020 06:57:13 -0500
Received: from mail-io1-xd43.google.com (mail-io1-xd43.google.com [IPv6:2607:f8b0:4864:20::d43])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 24E15C0613D2
        for <linux-unionfs@vger.kernel.org>; Tue,  1 Dec 2020 03:56:33 -0800 (PST)
Received: by mail-io1-xd43.google.com with SMTP id n14so1281434iom.10
        for <linux-unionfs@vger.kernel.org>; Tue, 01 Dec 2020 03:56:33 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=sargun.me; s=google;
        h=date:from:to:cc:subject:message-id:mime-version:content-disposition
         :in-reply-to:user-agent;
        bh=z9ULpizLWDQiCJkdcf/66NmmJqFYGorqtuiF70FgIJg=;
        b=aYyYXHVdvVyVPH6ufds7fBBjA5xxE+RIPU0UHwETETBon5X/AxBP3pfU7D6rgLWDxK
         MCPFi3QrsNpvmCT6uMHXOLdMnjdYFbsFK9ztzGqhIRHo5nK6I+1wG60VvNxdgQQB+op+
         wrAG/5RDYf5PphyPYR/ibeHZfeCSXNMaFIKGU=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:date:from:to:cc:subject:message-id:mime-version
         :content-disposition:in-reply-to:user-agent;
        bh=z9ULpizLWDQiCJkdcf/66NmmJqFYGorqtuiF70FgIJg=;
        b=IrhQGNRIFWpAsQ8ieRZkvVJ+VZHYTLVemjRu7zqPF6fdiQTUbrldpesOffGW7OWDkO
         I7QxBqO4xY2TXtGedx36HCIofTAM5FVLCSqtK6V/pX7LYE140CBRTPvfQDygSVJoiH4Y
         omkEbQuSMJtyX9PSJONnYRcgOcnNOLL67UV//p8kk6AgtPPzltDh+oH/WkDFpXfR2jw7
         Qb0K9znkZyqyS70fjIVpq3uFax+zmvnMa0TLE8t9ArAz1ZOXKhS3cFf/HrtAvMSRL+iO
         zmoqKq1p1Rb9c+Y9MIKWTlzT3fXyimAzB/Tpxp0a7XS/oQ36O2F2PzeLEvzA54ofDMYT
         Ao/w==
X-Gm-Message-State: AOAM532rOKfDK6czBnVdF9ABI7ot2EAPg08mx1fK06fj0yUqPKym0A9Q
        hxAjB9zhbrfYckxp5i5ZaRIQ+g==
X-Google-Smtp-Source: ABdhPJz0GL7O0XGhZu0j9b4iHEHojyC5oy0U6R1upsTmsVSxZyQ7aBqTOJ2GyvVcfx+OvymB9iiU2w==
X-Received: by 2002:a02:2e54:: with SMTP id u20mr2267006jae.142.1606823792462;
        Tue, 01 Dec 2020 03:56:32 -0800 (PST)
Received: from ircssh-2.c.rugged-nimbus-611.internal (80.60.198.104.bc.googleusercontent.com. [104.198.60.80])
        by smtp.gmail.com with ESMTPSA id x23sm697798ioh.28.2020.12.01.03.56.32
        (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256);
        Tue, 01 Dec 2020 03:56:32 -0800 (PST)
Date:   Tue, 1 Dec 2020 11:56:30 +0000
From:   Sargun Dhillon <sargun@sargun.me>
To:     Vivek Goyal <vgoyal@redhat.com>
Cc:     linux-unionfs@vger.kernel.org, miklos@szeredi.hu,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Amir Goldstein <amir73il@gmail.com>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel@vger.kernel.org, David Howells <dhowells@redhat.com>,
        Jeff Layton <jlayton@redhat.com>
Subject: Re: [PATCH v2 4/4] overlay: Add rudimentary checking of writeback
 errseq on volatile remount
Message-ID: <20201201115630.GC24837@ircssh-2.c.rugged-nimbus-611.internal>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20201130193342.GD14328@redhat.com>
User-Agent: Mutt/1.9.4 (2018-02-28)
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org

On Mon, Nov 30, 2020 at 02:33:42PM -0500, Vivek Goyal wrote:
> On Fri, Nov 27, 2020 at 01:20:58AM -0800, Sargun Dhillon wrote:
> > Volatile remounts validate the following at the moment:
> >  * Has the module been reloaded / the system rebooted
> >  * Has the workdir been remounted
> > 
> > This adds a new check for errors detected via the superblock's
> > errseq_t. At mount time, the errseq_t is snapshotted to disk,
> > and upon remount it's re-verified. This allows for kernel-level
> > detection of errors without forcing userspace to perform a
> > sync and allows for the hidden detection of writeback errors.
> > 
> > Signed-off-by: Sargun Dhillon <sargun@sargun.me>
> > Cc: linux-fsdevel@vger.kernel.org
> > Cc: linux-unionfs@vger.kernel.org
> > Cc: Miklos Szeredi <miklos@szeredi.hu>
> > Cc: Amir Goldstein <amir73il@gmail.com>
> > Cc: Vivek Goyal <vgoyal@redhat.com>
> > ---
> >  fs/overlayfs/overlayfs.h | 1 +
> >  fs/overlayfs/readdir.c   | 6 ++++++
> >  fs/overlayfs/super.c     | 1 +
> >  3 files changed, 8 insertions(+)
> > 
> > diff --git a/fs/overlayfs/overlayfs.h b/fs/overlayfs/overlayfs.h
> > index de694ee99d7c..e8a711953b64 100644
> > --- a/fs/overlayfs/overlayfs.h
> > +++ b/fs/overlayfs/overlayfs.h
> > @@ -85,6 +85,7 @@ struct ovl_volatile_info {
> >  	 */
> >  	uuid_t		ovl_boot_id;	/* Must stay first member */
> >  	u64		s_instance_id;
> > +	errseq_t	errseq;	/* Implemented as a u32 */
> >  } __packed;
> >  
> >  /*
> > diff --git a/fs/overlayfs/readdir.c b/fs/overlayfs/readdir.c
> > index 7b66fbb20261..5795b28bb4cf 100644
> > --- a/fs/overlayfs/readdir.c
> > +++ b/fs/overlayfs/readdir.c
> > @@ -1117,6 +1117,12 @@ static int ovl_verify_volatile_info(struct ovl_fs *ofs,
> >  		return -EINVAL;
> >  	}
> >  
> > +	err = errseq_check(&volatiledir->d_sb->s_wb_err, info.errseq);
> 
> Might be a stupid question. Will ask anyway.
> 
> But what protects against wrapping of counter. IOW, Say we stored info.errseq
> value as A. It is possible that bunch of errors occurred and at remount
> time ->s_wb_err is back to A and we pass the check. (Despite the fact lots
> of errors have occurred since we sampled).
> 
> Thanks
> Vivek
> 

+Jeff Layton <jlayton@redhat.com>

Nothing. The current errseq API works like this today where if you have 2^20
(1048576) errors, and syncfs (or other calls that mark the errseq as seen), and
the error that occured 1048575 times ago was the same error as you just last
had, and the error on the upperdir has already been marked as seen, the error
will be swallowed up silently.

This exists throughout all of VFS. I think we're potentially making this more
likely by checkpointing to disk. The one aspect which is a little different about
the usecase in the patch is that it relies on this mechanism to determine if
an error has occured after the entire FS was constructed, so it's somewhat
more consequential than the current issue in VFS which will just bubble up
errors in a few files.

On my system syncfs takes about 2 milliseconds, so you have a chance to
experience this every ~30 minutes if the syscalls align in the right way. If
we expanded the errseq_t to u64, we would potentially get a collision
every 4503599627370496 calls, or assuming the 2 millisecond invariant
holds, every 285 years. Now, we probably don't want to make errseq_t into
a u64 because of performance reasons (not all systems have native u64
cmpxchg), and the extra memory it'd take up.

If we really want to avoid this case, I can think of one "simple" solution,
which is something like laying out errseq_t as something like a errseq_t_src
that's 64-bits, and all readers just look at the lower 32-bits. The longer
errseq_t would exist on super_blocks, but files would still get the shorter one.
To potentially avoid the performance penalty of atomic longs, we could also
do something like this:

typedef struct {
    atomic_t overflow;
    u32 errseq;
} errseq_t_big;

And in errseq_set, do:
/* Wraps */
if (new < old)
        atomic_inc(&eseq->overflow);

*shrug*
I don't think that the above scenario is likely though.

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-17.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,
	URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 80421C64E7A
	for <linux-unionfs@archiver.kernel.org>; Tue,  1 Dec 2020 12:47:44 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id 3760520857
	for <linux-unionfs@archiver.kernel.org>; Tue,  1 Dec 2020 12:47:44 +0000 (UTC)
Authentication-Results: mail.kernel.org;
	dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="B5j5x9hc"
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1728675AbgLAMrV (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Tue, 1 Dec 2020 07:47:21 -0500
Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:60859 "EHLO
        us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK)
        by vger.kernel.org with ESMTP id S2390186AbgLAMrU (ORCPT
        <rfc822;linux-unionfs@vger.kernel.org>);
        Tue, 1 Dec 2020 07:47:20 -0500
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;
        s=mimecast20190719; t=1606826753;
        h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
         to:to:cc:cc:mime-version:mime-version:content-type:content-type:
         content-transfer-encoding:content-transfer-encoding:
         in-reply-to:in-reply-to:references:references;
        bh=sfLmN6LKi+/cx/A7X3IVbom0/z1aD5TcebxYU0pmAC0=;
        b=B5j5x9hcvjkRftU5yGXUVaCRm6Hfn+DYXulMUKUPz5IYTt789y8kMSCO83iPYcYhhwlTxd
        eAcALzNqzteMRhjTDWwCnegs+9S9LNWwByXYAIQmFe9TJ6BJlsMnS7XodGHJL6KBhooz3t
        0r/cWb0T5z12XnmQk1VngHogo1TgmOw=
Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com
 [209.85.222.198]) (Using TLS) by relay.mimecast.com with ESMTP id
 us-mta-405-PPwiTOktOCWJOAGYcYIsSg-1; Tue, 01 Dec 2020 07:45:50 -0500
X-MC-Unique: PPwiTOktOCWJOAGYcYIsSg-1
Received: by mail-qk1-f198.google.com with SMTP id t141so1206441qke.22
        for <linux-unionfs@vger.kernel.org>; Tue, 01 Dec 2020 04:45:50 -0800 (PST)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to
         :references:user-agent:mime-version:content-transfer-encoding;
        bh=sfLmN6LKi+/cx/A7X3IVbom0/z1aD5TcebxYU0pmAC0=;
        b=K74Txk1JkwqOuHxc/ttwefskcwYGwNSYxd4Jh1JkJRT9MYDBnZ3ktgHh/+C7bNjNt3
         8UYLA1oT2YSGLxGo+GPtYkSpUappzRVVB6pq7fH8HzfY5HalwfYt7VFO4rQEqjuoTbex
         9rjrUa35GWbeC7AThHcbP7iQEHE8OWohaMn3ABW8lOAtcsGRsj5TsjEggnouB1zhk5NL
         /V2xYsdDw1L8amON2fuJF2pAA7yf4AyKG7MBUtrJWtNo7ibRvV0jQ1Hj54nh82ShXhBH
         hPZSUhpB8GodiGlvZMtvarXOFKtD7bsX36cTkWqnXVLsuTUh/kgwMAygoKh2hrO/dmjl
         s4Bw==
X-Gm-Message-State: AOAM532fTzRbWPp0+CE6fxVI0LpYyvif3pqRSyjYnaLoEpHPaozFo0TB
        fB0626SVPkh3yuGtLdN2SuiKYeX83V2Rp18Se9dN6gxE2Ro4Owhmcy54ADFzO/lCMJJ/eX0h6QX
        bemuXs3B86z4ejq2BJrbCkE/KCQ==
X-Received: by 2002:ac8:6bc6:: with SMTP id b6mr2557525qtt.127.1606826749873;
        Tue, 01 Dec 2020 04:45:49 -0800 (PST)
X-Google-Smtp-Source: ABdhPJwJoHXLOyEJ5cZuEBQHddUI5KwMH+6XW0JGwiKb1k3eN95o9FTB2BM5XulXIXdas6d2+AWE+w==
X-Received: by 2002:ac8:6bc6:: with SMTP id b6mr2557507qtt.127.1606826749648;
        Tue, 01 Dec 2020 04:45:49 -0800 (PST)
Received: from tleilax.poochiereds.net (68-20-15-154.lightspeed.rlghnc.sbcglobal.net. [68.20.15.154])
        by smtp.gmail.com with ESMTPSA id c138sm1365892qke.95.2020.12.01.04.45.48
        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
        Tue, 01 Dec 2020 04:45:48 -0800 (PST)
Message-ID: <49ad785d0862dc49a043ec4f876892df80b51292.camel@redhat.com>
Subject: Re: [PATCH v2 4/4] overlay: Add rudimentary checking of writeback
 errseq on volatile remount
From:   Jeff Layton <jlayton@redhat.com>
To:     Sargun Dhillon <sargun@sargun.me>, Vivek Goyal <vgoyal@redhat.com>
Cc:     linux-unionfs@vger.kernel.org, miklos@szeredi.hu,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Amir Goldstein <amir73il@gmail.com>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel@vger.kernel.org, David Howells <dhowells@redhat.com>
Date:   Tue, 01 Dec 2020 07:45:47 -0500
In-Reply-To: <20201201115630.GC24837@ircssh-2.c.rugged-nimbus-611.internal>
References: <20201201115630.GC24837@ircssh-2.c.rugged-nimbus-611.internal>
Content-Type: text/plain; charset="UTF-8"
User-Agent: Evolution 3.38.1 (3.38.1-1.fc33) 
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org

On Tue, 2020-12-01 at 11:56 +0000, Sargun Dhillon wrote:
> On Mon, Nov 30, 2020 at 02:33:42PM -0500, Vivek Goyal wrote:
> > On Fri, Nov 27, 2020 at 01:20:58AM -0800, Sargun Dhillon wrote:
> > > Volatile remounts validate the following at the moment:
> > > Â * Has the module been reloaded / the system rebooted
> > > Â * Has the workdir been remounted
> > > 
> > > This adds a new check for errors detected via the superblock's
> > > errseq_t. At mount time, the errseq_t is snapshotted to disk,
> > > and upon remount it's re-verified. This allows for kernel-level
> > > detection of errors without forcing userspace to perform a
> > > sync and allows for the hidden detection of writeback errors.
> > > 
> > > Signed-off-by: Sargun Dhillon <sargun@sargun.me>
> > > Cc: linux-fsdevel@vger.kernel.org
> > > Cc: linux-unionfs@vger.kernel.org
> > > Cc: Miklos Szeredi <miklos@szeredi.hu>
> > > Cc: Amir Goldstein <amir73il@gmail.com>
> > > Cc: Vivek Goyal <vgoyal@redhat.com>
> > > ---
> > > Â fs/overlayfs/overlayfs.h | 1 +
> > > Â fs/overlayfs/readdir.c   | 6 ++++++
> > > Â fs/overlayfs/super.c     | 1 +
> > > Â 3 files changed, 8 insertions(+)
> > > 
> > > diff --git a/fs/overlayfs/overlayfs.h b/fs/overlayfs/overlayfs.h
> > > index de694ee99d7c..e8a711953b64 100644
> > > --- a/fs/overlayfs/overlayfs.h
> > > +++ b/fs/overlayfs/overlayfs.h
> > > @@ -85,6 +85,7 @@ struct ovl_volatile_info {
> > > Â 	 */
> > > Â 	uuid_t		ovl_boot_id;	/* Must stay first member */
> > > Â 	u64		s_instance_id;
> > > +	errseq_t	errseq;	/* Implemented as a u32 */
> > > Â } __packed;
> > > Â 
> > > 
> > > 
> > > 
> > > Â /*
> > > diff --git a/fs/overlayfs/readdir.c b/fs/overlayfs/readdir.c
> > > index 7b66fbb20261..5795b28bb4cf 100644
> > > --- a/fs/overlayfs/readdir.c
> > > +++ b/fs/overlayfs/readdir.c
> > > @@ -1117,6 +1117,12 @@ static int ovl_verify_volatile_info(struct ovl_fs *ofs,
> > > Â 		return -EINVAL;
> > > Â 	}
> > > Â 
> > > 
> > > 
> > > 
> > > +	err = errseq_check(&volatiledir->d_sb->s_wb_err, info.errseq);
> > 
> > Might be a stupid question. Will ask anyway.
> > 
> > But what protects against wrapping of counter. IOW, Say we stored info.errseq
> > value as A. It is possible that bunch of errors occurred and at remount
> > time ->s_wb_err is back to A and we pass the check. (Despite the fact lots
> > of errors have occurred since we sampled).
> > 
> > Thanks
> > Vivek
> > 
> 
> +Jeff Layton <jlayton@redhat.com>
> 
> Nothing. The current errseq API works like this today where if you have 2^20
> (1048576) errors, and syncfs (or other calls that mark the errseq as seen), and
> the error that occured 1048575 times ago was the same error as you just last
> had, and the error on the upperdir has already been marked as seen, the error
> will be swallowed up silently.
 
> This exists throughout all of VFS. I think we're potentially making this more
> likely by checkpointing to disk. The one aspect which is a little different about
> the usecase in the patch is that it relies on this mechanism to determine if
> an error has occured after the entire FS was constructed, so it's somewhat
> more consequential than the current issue in VFS which will just bubble up
> errors in a few files.
> 
> On my system syncfs takes about 2 milliseconds, so you have a chance to
> experience this every ~30 minutes if the syscalls align in the right way. If
> we expanded the errseq_t to u64, we would potentially get a collision
> every 4503599627370496 calls, or assuming the 2 millisecond invariant
> holds, every 285 years. Now, we probably don't want to make errseq_t into
> a u64 because of performance reasons (not all systems have native u64
> cmpxchg), and the extra memory it'd take up.
> 
> If we really want to avoid this case, I can think of one "simple" solution,
> which is something like laying out errseq_t as something like a errseq_t_src
> that's 64-bits, and all readers just look at the lower 32-bits. The longer
> errseq_t would exist on super_blocks, but files would still get the shorter one.
> To potentially avoid the performance penalty of atomic longs, we could also
> do something like this:
> 
> typedef struct {
> Â Â Â Â atomic_t overflow;
> Â Â Â Â u32 errseq;
> } errseq_t_big;
> 
> And in errseq_set, do:
> /* Wraps */
> if (new < old)
> Â Â Â Â Â Â Â Â atomic_inc(&eseq->overflow);
> 
> *shrug*
> I don't think that the above scenario is likely though.
> 


The counter is only bumped if the seen flag is set, so you'd need to
record 2^20 errors _and_ fetch them that many times, and just get
unlucky once and hit the exact counter value that you had previously
sampled.

If that's your situation, then you probably have much bigger problems
than the counter wrapping.

-- 
Jeff Layton <jlayton@redhat.com>


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-17.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,
	URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 36B57C64E7A
	for <linux-unionfs@archiver.kernel.org>; Tue,  1 Dec 2020 13:02:46 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id C7AD4207FF
	for <linux-unionfs@archiver.kernel.org>; Tue,  1 Dec 2020 13:02:45 +0000 (UTC)
Authentication-Results: mail.kernel.org;
	dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="LRZpwzKg"
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1728780AbgLANCp (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Tue, 1 Dec 2020 08:02:45 -0500
Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:55618 "EHLO
        us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK)
        by vger.kernel.org with ESMTP id S1727077AbgLANCp (ORCPT
        <rfc822;linux-unionfs@vger.kernel.org>);
        Tue, 1 Dec 2020 08:02:45 -0500
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;
        s=mimecast20190719; t=1606827676;
        h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
         to:to:cc:cc:mime-version:mime-version:content-type:content-type:
         content-transfer-encoding:content-transfer-encoding:
         in-reply-to:in-reply-to:references:references;
        bh=lxQN4N2AQp3LRLggORzpLN660oZQ5Hx35SxeD4S+HA4=;
        b=LRZpwzKg1IkYn/BdmboSVKFOdDgzNJawCnI4zgOxkQD2g/urX1/PbV5BNek/cp23vKBbCi
        0IwHTle+MBEydonBwPpm3PNnn2AHW8UGxGBUQ7CCIwtOGkcNEuRGzrKUqYWdN6uX/Se8Uz
        g7oXWy/ztre/Fvuz+gqmDMNo5AV/nN8=
Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com
 [209.85.222.198]) (Using TLS) by relay.mimecast.com with ESMTP id
 us-mta-245-sbaLm0juOlWElARfAuJ19A-1; Tue, 01 Dec 2020 08:01:14 -0500
X-MC-Unique: sbaLm0juOlWElARfAuJ19A-1
Received: by mail-qk1-f198.google.com with SMTP id t141so1238415qke.22
        for <linux-unionfs@vger.kernel.org>; Tue, 01 Dec 2020 05:01:14 -0800 (PST)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to
         :references:user-agent:mime-version:content-transfer-encoding;
        bh=lxQN4N2AQp3LRLggORzpLN660oZQ5Hx35SxeD4S+HA4=;
        b=CDLVmoweTbQ9zigXLlqE7VdS7bRYC678RuQnddD8vpoBeHm2xT3Y5dSnzqzqB0uqd9
         LafUOtwU3b9r3suuEGxVSQqUAJLLbRyLU9PPAjK8gnpgScIhgqFoR194JAOq5k+J8FFo
         /HxbVlk7Wa5Qnd18bJG+akBt5L4rVwQB51AY9hhPc2OosbjktpEjCAQYJDMI7AAlgN/u
         BnlB5zDMnnxweuLCn5kqPJf+7DOyG7PGXr7mA5bvH/i28/HymazMe1QZRQsGEkjr1ADO
         yWO0lMmgb9VU5qWkfdF6MjTi9hd6ha7SSFjCHxuJ1FVhTFHsB5+HvmPg4Jgqep2rh3yU
         V97Q==
X-Gm-Message-State: AOAM530V6LievCmcHY96MhnKB4e2VkkNy26QWTBtMPD9iUG6glAa0Xdd
        k7S7X38W1QrdS8tyQg88PMqauLAagkBvlz09FpFN1IWIlb7Lng9XaAscFe+DhBoFtZwyFkx+5GH
        6OrpqLY0at25j+qkX6ERdI5/2CA==
X-Received: by 2002:ad4:4661:: with SMTP id z1mr2625153qvv.19.1606827673592;
        Tue, 01 Dec 2020 05:01:13 -0800 (PST)
X-Google-Smtp-Source: ABdhPJx3PJhxWzDz8G3n9BDF43q/8HsRP003nUuqHywCMosTeoUNJ6+L/0zFi99YimDtKsGBrrVz2Q==
X-Received: by 2002:ad4:4661:: with SMTP id z1mr2625134qvv.19.1606827673215;
        Tue, 01 Dec 2020 05:01:13 -0800 (PST)
Received: from tleilax.poochiereds.net (68-20-15-154.lightspeed.rlghnc.sbcglobal.net. [68.20.15.154])
        by smtp.gmail.com with ESMTPSA id n41sm1933874qtb.18.2020.12.01.05.01.12
        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
        Tue, 01 Dec 2020 05:01:12 -0800 (PST)
Message-ID: <0f831ca2a65b028c2682819228eb5560230a0e4d.camel@redhat.com>
Subject: Re: [PATCH v2 2/4] overlay: Document current outstanding
 shortcoming of volatile
From:   Jeff Layton <jlayton@redhat.com>
To:     Sargun Dhillon <sargun@sargun.me>,
        Amir Goldstein <amir73il@gmail.com>
Cc:     overlayfs <linux-unionfs@vger.kernel.org>,
        Miklos Szeredi <miklos@szeredi.hu>,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Vivek Goyal <vgoyal@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel <linux-fsdevel@vger.kernel.org>,
        David Howells <dhowells@redhat.com>
Date:   Tue, 01 Dec 2020 08:01:11 -0500
In-Reply-To: <20201201110928.GA24837@ircssh-2.c.rugged-nimbus-611.internal>
References: <20201127092058.15117-1-sargun@sargun.me>
         <20201127092058.15117-3-sargun@sargun.me>
         <CAOQ4uxgaLuLb+f6WCMvmKHNTELvcvN8C5_u=t5hhoGT8Op7QuQ@mail.gmail.com>
         <20201127221154.GA23383@ircssh-2.c.rugged-nimbus-611.internal>
         <1338a059d03db0e85cf3f3234fd33434a45606c6.camel@redhat.com>
         <20201128044530.GA28230@ircssh-2.c.rugged-nimbus-611.internal>
         <CAOQ4uxjT6FF03Sq3qXuqDcqJQnzQq2dD_XVbuj_Fb9A2Ag585w@mail.gmail.com>
         <20201128085227.GB28230@ircssh-2.c.rugged-nimbus-611.internal>
         <20201201110928.GA24837@ircssh-2.c.rugged-nimbus-611.internal>
Content-Type: text/plain; charset="UTF-8"
User-Agent: Evolution 3.38.1 (3.38.1-1.fc33) 
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org

On Tue, 2020-12-01 at 11:09 +0000, Sargun Dhillon wrote:
> On Sat, Nov 28, 2020 at 08:52:27AM +0000, Sargun Dhillon wrote:
> > On Sat, Nov 28, 2020 at 09:12:03AM +0200, Amir Goldstein wrote:
> > > On Sat, Nov 28, 2020 at 6:45 AM Sargun Dhillon <sargun@sargun.me> wrote:
> > > > 
> > > > On Fri, Nov 27, 2020 at 09:01:07PM -0500, Jeff Layton wrote:
> > > > > On Fri, 2020-11-27 at 22:11 +0000, Sargun Dhillon wrote:
> > > > > > On Fri, Nov 27, 2020 at 02:52:52PM +0200, Amir Goldstein wrote:
> > > > > > > On Fri, Nov 27, 2020 at 11:21 AM Sargun Dhillon <sargun@sargun.me> wrote:
> > > > > > > > 
> > > > > > > > This documents behaviour that was discussed in a thread about the volatile
> > > > > > > > feature. Specifically, how failures can go hidden from asynchronous writes
> > > > > > > > (such as from mmap, or writes that are not immediately flushed to the
> > > > > > > > filesystem). Although we pass through calls like msync, fallocate, and
> > > > > > > > write, and will still return errors on those, it doesn't guarantee all
> > > > > > > > kinds of errors will happen at those times, and thus may hide errors.
> > > > > > > > 
> > > > > > > > In the future, we can add error checking to all interactions with the
> > > > > > > > upperdir, and pass through errseq_t from the upperdir on mappings,
> > > > > > > > and other interactions with the filesystem[1].
> > > > > > > > 
> > > > > > > > [1]: https://lore.kernel.org/linux-unionfs/20201116045758.21774-1-sargun@sargun.me/T/#m7d501f375e031056efad626e471a1392dd3aad33
> > > > > > > > 
> > > > > > > > Signed-off-by: Sargun Dhillon <sargun@sargun.me>
> > > > > > > > Cc: linux-fsdevel@vger.kernel.org
> > > > > > > > Cc: linux-unionfs@vger.kernel.org
> > > > > > > > Cc: Miklos Szeredi <miklos@szeredi.hu>
> > > > > > > > Cc: Amir Goldstein <amir73il@gmail.com>
> > > > > > > > Cc: Vivek Goyal <vgoyal@redhat.com>
> > > > > > > > ---
> > > > > > > > Â Documentation/filesystems/overlayfs.rst | 6 +++++-
> > > > > > > > Â 1 file changed, 5 insertions(+), 1 deletion(-)
> > > > > > > > 
> > > > > > > > diff --git a/Documentation/filesystems/overlayfs.rst b/Documentation/filesystems/overlayfs.rst
> > > > > > > > index 580ab9a0fe31..c6e30c1bc2f2 100644
> > > > > > > > --- a/Documentation/filesystems/overlayfs.rst
> > > > > > > > +++ b/Documentation/filesystems/overlayfs.rst
> > > > > > > > @@ -570,7 +570,11 @@ Volatile mount
> > > > > > > > Â This is enabled with the "volatile" mount option.  Volatile mounts are not
> > > > > > > > Â guaranteed to survive a crash.  It is strongly recommended that volatile
> > > > > > > > Â mounts are only used if data written to the overlay can be recreated
> > > > > > > > -without significant effort.
> > > > > > > > +without significant effort.  In addition to this, the sync family of syscalls
> > > > > > > > +are not sufficient to determine whether a write failed as sync calls are
> > > > > > > > +omitted.  For this reason, it is important that the filesystem used by the
> > > > > > > > +upperdir handles failure in a fashion that's suitable for the user.  For
> > > > > > > > +example, upon detecting a fault, ext4 can be configured to panic.
> > > > > > > > 
> > > > > > > 
> > > > > > > Reading this now, I think I may have wrongly analysed the issue.
> > > > > > > Specifically, when I wrote that the very minimum is to document the
> > > > > > > issue, it was under the assumption that a proper fix is hard.
> > > > > > > I think I was wrong and that the very minimum is to check for errseq
> > > > > > > since mount on the fsync and syncfs calls.
> > > > > > > 
> > > > > > > Why? first of all because it is very very simple and goes a long way to
> > > > > > > fix the broken contract with applications, not the contract about durability
> > > > > > > obviously, but the contract about write+fsync+read expects to find the written
> > > > > > > data (during the same mount era).
> > > > > > > 
> > > > > > > Second, because the sentence you added above is hard for users to understand
> > > > > > > and out of context. If we properly handle the writeback error in fsync/syncfs,
> > > > > > > then this sentence becomes irrelevant.
> > > > > > > The fact that ext4 can lose data if application did not fsync after
> > > > > > > write is true
> > > > > > > for volatile as well as non-volatile and it is therefore not relevant
> > > > > > > in the context
> > > > > > > of overlayfs documentation at all.
> > > > > > > 
> > > > > > > Am I wrong saying that it is very very simple to fix?
> > > > > > > Would you mind making that fix at the bottom of the patch series, so it can
> > > > > > > be easily applied to stable kernels?
> > > > > > > 
> > > > > > > Thanks,
> > > > > > > Amir.
> > > > > > 
> > > > > > I'm not sure it's so easy. In VFS, there are places where the superblock's
> > > > > > errseq is checked[1]. AFAIK, that's going to check "our" errseq, and not the
> > > > > > errseq of the real corresponding real file's superblock. One solution might be
> > > > > > as part of all these callbacks we set our errseq to the errseq of the filesystem
> > > > > > that the upperdir, and then rely on VFS's checking.
> > > > > > 
> > > > > > I'm having a hard time figuring out how to deal with the per-mapping based
> > > > > > error tracking. It seems like this infrastructure was only partially completed
> > > > > > by Jeff Layton[2]. I don't now how it's actually supposed to work right now,
> > > > > > as not all of his patches landed.
> > > > > > 
> > > > > 
> > > > > The patches in the series were all merged, but we ended up going with a
> > > > > simpler solution [1] than the first series I posted. Instead of plumbing
> > > > > the errseq_t handling down into sync_fs, we did it in the syscall
> > > > > wrapper.
> > > > Jeff,
> > > > 
> > > > Thanks for replying. I'm still a little confused as to what the
> > > > per-address_space wb_err. It seems like we should implement the
> > > > flush method, like so:
> > > > diff --git a/fs/overlayfs/file.c b/fs/overlayfs/file.c
> > > > index efccb7c1f9bc..32e5bc0aacd6 100644
> > > > --- a/fs/overlayfs/file.c
> > > > +++ b/fs/overlayfs/file.c
> > > > @@ -787,6 +787,16 @@ static loff_t ovl_remap_file_range(struct file *file_in, loff_t pos_in,
> > > > Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â remap_flags, op);
> > > > Â }
> > > > 
> > > > +static int ovl_flush(struct file *file, fl_owner_t id)
> > > > +{
> > > > +       struct file *realfile = file->private_data;
> > > > +
> > > > +       if (real_file->f_op->flush)
> > > > +               return real_file->f_op->flush(filp, id);
> > > > +
> > > > +       return 0;
> > > > +}
> > > > +
> > > > Â const struct file_operations ovl_file_operations = {
> > > > Â Â Â Â Â Â Â Â .open           = ovl_open,
> > > > Â Â Â Â Â Â Â Â .release        = ovl_release,
> > > > @@ -798,6 +808,7 @@ const struct file_operations ovl_file_operations = {
> > > > Â Â Â Â Â Â Â Â .fallocate      = ovl_fallocate,
> > > > Â Â Â Â Â Â Â Â .fadvise        = ovl_fadvise,
> > > > Â Â Â Â Â Â Â Â .unlocked_ioctl = ovl_ioctl,
> > > > +       .flush          = ovl_flush,
> > > > Â #ifdef CONFIG_COMPAT
> > > > Â Â Â Â Â Â Â Â .compat_ioctl   = ovl_compat_ioctl,
> > > > Â #endif
> > > > 
> > > > 
> > > > > 
> > > > > I think the tricky part here is that there is no struct file plumbed
> > > > > into ->sync_fs, so you don't have an errseq_t cursor to work with by the
> > > > > time that gets called.
> > > > > 
> > > > > What may be easiest is to just propagate the s_wb_err value from the
> > > > > upper_sb to the overlayfs superblock in ovl_sync_fs(). That should get
> > > > > called before the errseq_check_and_advance in the syncfs syscall wrapper
> > > > > and should ensure that syncfs() calls against the overlayfs sb see any
> > > > > errors that occurred on the upper_sb.
> > > > > 
> > > > > Something like this maybe? Totally untested of course. May also need to
> > > > > think about how to ensure ordering between racing syncfs calls too
> > > > > (don't really want the s_wb_err going "backward"):
> > > > > 
> > > > > ----------------------------8<---------------------------------
> > > > > $ git diff
> > > > > diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
> > > > > index 290983bcfbb3..d725705abdac 100644
> > > > > --- a/fs/overlayfs/super.c
> > > > > +++ b/fs/overlayfs/super.c
> > > > > @@ -283,6 +283,9 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
> > > > > Â Â Â Â Â Â Â Â ret = sync_filesystem(upper_sb);
> > > > > Â Â Â Â Â Â Â Â up_read(&upper_sb->s_umount);
> > > > > 
> > > > > +       /* Propagate s_wb_err from upper_sb to overlayfs sb */
> > > > > +       WRITE_ONCE(sb->s_wb_err, READ_ONCE(upper_sb->s_wb_err));
> > > > > +
> > > > > Â Â Â Â Â Â Â Â return ret;
> > > > > Â }
> > > > > ----------------------------8<---------------------------------
> > > > > 
> > > > > [1]: https://www.spinics.net/lists/linux-api/msg41276.html
> > > > 
> > > > So,
> > > > I think this will have bad behaviour because syncfs() twice in a row will
> > > > return the error twice. The reason is that normally in syncfs it calls
> > > > errseq_check_and_advance marking the error on the super block as seen. If we
> > > > copy-up the error value each time, it will break this semantic, as we do not set
> > > > seen on the upperdir.
> > > > 
> > > > Either we need to set the seen flag on the upperdir's errseq_t, or have a sync
> > > > method, like:
> > > > diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
> > > > index 290983bcfbb3..4931d1797c03 100644
> > > > --- a/fs/overlayfs/super.c
> > > > +++ b/fs/overlayfs/super.c
> > > > @@ -259,6 +259,7 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
> > > > Â {
> > > > Â Â Â Â Â Â Â Â struct ovl_fs *ofs = sb->s_fs_info;
> > > > Â Â Â Â Â Â Â Â struct super_block *upper_sb;
> > > > +       errseq_t src;
> > > > Â Â Â Â Â Â Â Â int ret;
> > > > 
> > > > Â Â Â Â Â Â Â Â if (!ovl_upper_mnt(ofs))
> > > > @@ -283,6 +284,11 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
> > > > Â Â Â Â Â Â Â Â ret = sync_filesystem(upper_sb);
> > > > Â Â Â Â Â Â Â Â up_read(&upper_sb->s_umount);
> > > > 
> > > > +       /* Propagate the error up from the upper_sb once */
> > > > +       src = READ_ONCE(upper_sb->s_wb_err);
> > > > +       if (errseq_counter(src) != errseq_counter(sb->s_wb_err))
> > > > +               WRITE_ONCE(sb->s_wb_err, src & ~ERRSEQ_SEEN);
> > > > +
> > > > Â Â Â Â Â Â Â Â return ret;
> > > > Â }
> > > > 
> > > > @@ -1945,6 +1951,7 @@ static int ovl_fill_super(struct super_block *sb, void *data, int silent)
> > > > 
> > > > Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â sb->s_stack_depth = ovl_upper_mnt(ofs)->mnt_sb->s_stack_depth;
> > > > Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â sb->s_time_gran = ovl_upper_mnt(ofs)->mnt_sb->s_time_gran;
> > > > +               sb->s_wb_err = READ_ONCE(ovl_upper_mnt(ofs)->mnt_sb->s_wb_err);
> > > > 
> > > > diff --git a/lib/errseq.c b/lib/errseq.c
> > > > index 81f9e33aa7e7..53275c168265 100644
> > > > --- a/lib/errseq.c
> > > > +++ b/lib/errseq.c
> > > > @@ -204,3 +204,18 @@ int errseq_check_and_advance(errseq_t *eseq, errseq_t *since)
> > > > Â Â Â Â Â Â Â Â return err;
> > > > Â }
> > > > Â EXPORT_SYMBOL(errseq_check_and_advance);
> > > > +
> > > > +/**
> > > > + * errseq_counter() - Get the value of the errseq counter
> > > > + * @eseq: Value being checked
> > > > + *
> > > > + * This returns the errseq_t counter value. Reminder, it can wrap because
> > > > + * there are only a limited number of counter bits.
> > > > + *
> > > > + * Return: The counter portion of the errseq_t.
> > > > + */
> > > > +int errseq_counter(errseq_t eseq)
> > > > +{
> > > > +       return eseq >> (ERRSEQ_SHIFT + 1);
> > > > +}
> > > > 
> > > > This also needs some locking sprinkled on it because racing can occur with
> > > > sync_fs. This would be much easier if the errseq_t was 64-bits, and didn't go
> > > > backwards, because you could just use a simple cmpxchg64. I know that would make
> > > > a bunch of structures larger, and if it's just overlayfs that has to pay the tax
> > > > we can just sprinkle a mutex on it.
> > > > 
> > > > We can always "copy-up" the errseq_t because if the upperdir's errseq_t is read,
> > > > and we haven't done a copy yet, we'll get it. Since this will be strictly
> > > > serialized a simple equivalence check will work versus actually having to deal
> > > > with happens before behaviour. There is still a correctness flaw here though,
> > > > which is exactly enough errors occur to result in it wrapping back to the same
> > > > value, it will break.
> > > > 
> > > > By the way, how do y'all test this error handling behaviour? I didn't
> > > > find any automated testing for what currently exists.
> > > > > 
> > > > > How about I split this into two patchsets? One, where I add the logic to copy
> > > > > the errseq on callbacks to fsync from the upperdir to the ovl fs superblock,
> > > > > and thus allowing VFS to bubble up errors, and the documentation. We can CC
> > > > > stable on those because I think it has an effect that's universal across
> > > > > all filesystems.
> > > > > 
> > > > > P.S.
> > > > > I notice you maintain overlay tests outside of the kernel. Unfortunately, I
> > > > > think for this kind of test, it requires in kernel code to artificially bump the
> > > > > writeback error count on the upperdir, or it requires the failure injection
> > > > > infrastructure.
> > > > > 
> > > > > Simulating this behaviour is non-trivial without in-kernel support:
> > > > > 
> > > > > P1: Open(f) -> p1.fd
> > > > > P2: Open(f) -> p2.fd
> > > > > P1: syncfs(p1.fd) -> errrno
> > > > > P2: syncfs(p1.fd) -> 0 # This should return an error
> > > > > 
> > > > > 
> > > > > [1]: https://elixir.bootlin.com/linux/latest/source/fs/sync.c#L175
> > > > > [2]: https://lwn.net/Articles/722250/
> > > > > 
> > > > > 
> > > 
> > > Sargun (and Jeff),
> > > 
> > > Thank you for this discussion. It would be very nice to work on testing
> > > and fixing errseq propagation is correct on overlayfs.
> > > Alas, this is not what I suggested.
> > > 
> > > What I suggested was a solution only for the volatile overlay issue
> > > where data can vaporise without applications noticing:
> > > "...the very minimum is to check for errseq since mount on the fsync
> > > Â and syncfs calls."
> > > 
> > Yeah, I was confusing the checking that VFS does on our behalf and the checking 
> > that we can do ourselves in the sync callback. If we return an error prior to 
> > the vfs checking it short-circuits that entirely.
> > 
> > > Do you get it? there is no pre-file state in the game, not for fsync and not 
> > > for syncfs.
> > > 
> > > Any single error, no matter how temporary it is and what damage it may
> > > or may not have caused to upper layer consistency, permanently
> > > invalidates the reliability of the volatile overlay, resulting in:
> > > Effective immediately: every fsync/syncfs returns EIO.
> > > Going forward: maybe implement overlay shutdown, so every access
> > > returns EIO.
> > > 
> > > So now that I hopefully explained myself better, I'll ask again:
> > > Am I wrong saying that it is very very simple to fix?
> > > Would you mind making that fix at the bottom of the patch series, so it can
> > > be easily applied to stable kernels?
> > > 
> > > Thanks,
> > > Amir.
> > 
> > I think that this should be easy enough if the semantic is such that volatile
> > overlayfs mounts will return EIO on syncfs on every syncfs call if the upperdir's
> > super block has experienced errors since the initial mount. I imagine we do not
> > want to make it such that if the upperdir has ever experienced errors, return
> > EIO on syncfs.
> > 
> > The one caveat that I see is that if the errseq wraps, we can silently begin 
> > swallowing errors again. Thus, on the first failed syncfs we should just
> > store a flag indicating that the volatile fs is bad, and to continue to return
> > EIO rather than go through the process of checking errseq_t, but that's easy
> > enough to write.
> 
> It turns out this is much more complicated than I initially thought. I'm not 
> entirely sure why VFS even defines sync_fs (on the superblock) as returning an 
> int. The way that sync_fs works is:
> 
> sys_syncfs ->
> Â Â sync_filesystem(sb) ->
> Â Â Â Â __sync_filesystem(sb, N) ->
> Â Â Â Â Â Â sb->s_op->sync_fs
> Â Â Â Â Â Â /* __sync_blockdev has its own writeback error checking */
> Â Â Â Â Â Â __sync_blockdev
> Â Â errseq_check_and_advance(sb...)
> 
> __sync_filesystem calls the sync_fs callback on the superblock's superblock
> operations:
> 
> static int __sync_filesystem(struct super_block *sb, int wait)
> {
> 	if (wait)
> 		sync_inodes_sb(sb);
> 	else
> 		writeback_inodes_sb(sb, WB_REASON_SYNC);
> 
> 	if (sb->s_op->sync_fs)
> 		sb->s_op->sync_fs(sb, wait);
> 	return __sync_blockdev(sb->s_bdev, wait);
> }
> 
> But it completely discards the result. On the other hand, fsync, and fdatasync
> exhibit different behaviour:
> 
> sys_fdatasync ->
> Â Â do_fsync
> Â Â Â (see below)
> 
> sys_fsync ->
> Â Â do_fsync ->
> Â Â Â Â vfs_fsync ->
> Â Â Â Â Â Â vfs_fsync_range ->
> Â Â Â Â Â Â Â Â return file->f_op->fsync
> ----
> 

This is mainly a result of the evolution of the code. In the beginning
there was only sync() and it doesn't report errors. Eventually syncfs()
was added, but the only error it reliably reported was EBADF.

I added the machinery to record writeback errors of individual inodes at
the superblock level recently, but the sync_fs op was left alone (mainly
because changing this would be very invasive, and the value of doing so
wasn't completely clear).


> 
> syncfs seems to enforce the semantics laid out by VFS[1]. Specifically the 
> statement:
> Â Â When there is an error during writeback, they expect that error to be reported 
> Â Â when a file sync request is made. After an error has been reported on one 
> Â Â request, subsequent requests on the same file descriptor should return 0, unless 
> Â Â further writeback errors have occurred since the previous file syncronization.
> 
> This is enforced by the errseq_check_and_advance logic. We can hack around this
> logic by resetting the errset (setting the error on it) every time we get the
> sync_fs callback, but that to me seems wrong. FWIW, implementing this behaviour
> for fdatasync, and fsync is easier, because the error is bubbled up from the
> filesystem to the VFS. I don't actually think this is a good idea because
> it seems like this sync_fs behaviour is a bit...not neccessarily what all
> filesystems expect. For example, btrfs_sync_fs returns an error if it
> is unable to finish the current transaction. Nowhere in the btrfs code
> does it set the errseq on the superblock if this fails.
> 
> I think we have a couple paths forward:
> 1. Change the semantic of sync_fs so the error is always bubbled up if
> Â Â Â it returns a non-zero value. If we do this, we have to decide whether
> Â Â Â or not we would _also_ call errseq_check_and_advance on the SB,
> Â Â Â or leave that to a subsequent call.

You would want to call errseq_check_and_advance in that situation.
Otherwise a subsequent syncfs call would return an error even when an
error had not occurred since the last syncfs.

> 2. Have overlayfs forcefully set an error on the superblock on every
> Â Â Â callback to sync_fs. This seems ugly, but I wrote a little patch,
> Â Â Â and it seems to solve the problem for all the fsync / fdatasync /
> Â Â Â sync / syncfs variants without having to do plumbing in VFS.

It is ugly, but that's probably the easiest solution.

> 3. Choose a different set of semantics for how we want to handle
> Â Â Â errors in volatile mounts.
> 
> [1]: https://www.kernel.org/doc/html/v5.9/filesystems/vfs.html?highlight=handling%20errors%20during%20writeback#handling-errors-during-writeback
> 

-- 
Jeff Layton <jlayton@redhat.com>


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-17.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,
	URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 78507C83012
	for <linux-unionfs@archiver.kernel.org>; Tue,  1 Dec 2020 15:26:20 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id 097B120770
	for <linux-unionfs@archiver.kernel.org>; Tue,  1 Dec 2020 15:26:19 +0000 (UTC)
Authentication-Results: mail.kernel.org;
	dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="KcKUmPs2"
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S2391844AbgLAP0K (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Tue, 1 Dec 2020 10:26:10 -0500
Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:58469 "EHLO
        us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK)
        by vger.kernel.org with ESMTP id S2391826AbgLAP0I (ORCPT
        <rfc822;linux-unionfs@vger.kernel.org>);
        Tue, 1 Dec 2020 10:26:08 -0500
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;
        s=mimecast20190719; t=1606836280;
        h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
         to:to:cc:cc:mime-version:mime-version:content-type:content-type:
         content-transfer-encoding:content-transfer-encoding:
         in-reply-to:in-reply-to:references:references;
        bh=BrifEUbhc7yaiyQVIbQQamFLVxxNVOPlSKjk5BSZVvc=;
        b=KcKUmPs2g2sBHfYZ/U+7R24MKQB3Iv7nYilwezlEwtHYIepuWfiJ0LKlev+XvBzPZOr+98
        byzs9GpfkCFeSySGw4CalBleSt/322q5rEY7yOCNg/nUK24uxQ1zuSZN+6lLvivbbOmUez
        FusGRNilz6i2ePJgkaaWERTRwpBaZLs=
Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com
 [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id
 us-mta-223-dcdIM0SwOBmpj1uUTT3vig-1; Tue, 01 Dec 2020 10:24:36 -0500
X-MC-Unique: dcdIM0SwOBmpj1uUTT3vig-1
Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23])
        (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
        (No client certificate requested)
        by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 9CEE980476B;
        Tue,  1 Dec 2020 15:24:05 +0000 (UTC)
Received: from horse.redhat.com (ovpn-117-236.rdu2.redhat.com [10.10.117.236])
        by smtp.corp.redhat.com (Postfix) with ESMTP id CD46219C48;
        Tue,  1 Dec 2020 15:24:04 +0000 (UTC)
Received: by horse.redhat.com (Postfix, from userid 10451)
        id 5F01C22054F; Tue,  1 Dec 2020 10:24:04 -0500 (EST)
Date:   Tue, 1 Dec 2020 10:24:04 -0500
From:   Vivek Goyal <vgoyal@redhat.com>
To:     Jeff Layton <jlayton@redhat.com>
Cc:     Sargun Dhillon <sargun@sargun.me>,
        Amir Goldstein <amir73il@gmail.com>,
        overlayfs <linux-unionfs@vger.kernel.org>,
        Miklos Szeredi <miklos@szeredi.hu>,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel <linux-fsdevel@vger.kernel.org>,
        David Howells <dhowells@redhat.com>
Subject: Re: [PATCH v2 2/4] overlay: Document current outstanding shortcoming
 of volatile
Message-ID: <20201201152404.GA86704@redhat.com>
References: <20201127092058.15117-1-sargun@sargun.me>
 <20201127092058.15117-3-sargun@sargun.me>
 <CAOQ4uxgaLuLb+f6WCMvmKHNTELvcvN8C5_u=t5hhoGT8Op7QuQ@mail.gmail.com>
 <20201127221154.GA23383@ircssh-2.c.rugged-nimbus-611.internal>
 <1338a059d03db0e85cf3f3234fd33434a45606c6.camel@redhat.com>
 <20201128044530.GA28230@ircssh-2.c.rugged-nimbus-611.internal>
 <CAOQ4uxjT6FF03Sq3qXuqDcqJQnzQq2dD_XVbuj_Fb9A2Ag585w@mail.gmail.com>
 <20201128085227.GB28230@ircssh-2.c.rugged-nimbus-611.internal>
 <20201201110928.GA24837@ircssh-2.c.rugged-nimbus-611.internal>
 <0f831ca2a65b028c2682819228eb5560230a0e4d.camel@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <0f831ca2a65b028c2682819228eb5560230a0e4d.camel@redhat.com>
X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org

On Tue, Dec 01, 2020 at 08:01:11AM -0500, Jeff Layton wrote:
> On Tue, 2020-12-01 at 11:09 +0000, Sargun Dhillon wrote:
> > On Sat, Nov 28, 2020 at 08:52:27AM +0000, Sargun Dhillon wrote:
> > > On Sat, Nov 28, 2020 at 09:12:03AM +0200, Amir Goldstein wrote:
> > > > On Sat, Nov 28, 2020 at 6:45 AM Sargun Dhillon <sargun@sargun.me> wrote:
> > > > > 
> > > > > On Fri, Nov 27, 2020 at 09:01:07PM -0500, Jeff Layton wrote:
> > > > > > On Fri, 2020-11-27 at 22:11 +0000, Sargun Dhillon wrote:
> > > > > > > On Fri, Nov 27, 2020 at 02:52:52PM +0200, Amir Goldstein wrote:
> > > > > > > > On Fri, Nov 27, 2020 at 11:21 AM Sargun Dhillon <sargun@sargun.me> wrote:
> > > > > > > > > 
> > > > > > > > > This documents behaviour that was discussed in a thread about the volatile
> > > > > > > > > feature. Specifically, how failures can go hidden from asynchronous writes
> > > > > > > > > (such as from mmap, or writes that are not immediately flushed to the
> > > > > > > > > filesystem). Although we pass through calls like msync, fallocate, and
> > > > > > > > > write, and will still return errors on those, it doesn't guarantee all
> > > > > > > > > kinds of errors will happen at those times, and thus may hide errors.
> > > > > > > > > 
> > > > > > > > > In the future, we can add error checking to all interactions with the
> > > > > > > > > upperdir, and pass through errseq_t from the upperdir on mappings,
> > > > > > > > > and other interactions with the filesystem[1].
> > > > > > > > > 
> > > > > > > > > [1]: https://lore.kernel.org/linux-unionfs/20201116045758.21774-1-sargun@sargun.me/T/#m7d501f375e031056efad626e471a1392dd3aad33
> > > > > > > > > 
> > > > > > > > > Signed-off-by: Sargun Dhillon <sargun@sargun.me>
> > > > > > > > > Cc: linux-fsdevel@vger.kernel.org
> > > > > > > > > Cc: linux-unionfs@vger.kernel.org
> > > > > > > > > Cc: Miklos Szeredi <miklos@szeredi.hu>
> > > > > > > > > Cc: Amir Goldstein <amir73il@gmail.com>
> > > > > > > > > Cc: Vivek Goyal <vgoyal@redhat.com>
> > > > > > > > > ---
> > > > > > > > >  Documentation/filesystems/overlayfs.rst | 6 +++++-
> > > > > > > > >  1 file changed, 5 insertions(+), 1 deletion(-)
> > > > > > > > > 
> > > > > > > > > diff --git a/Documentation/filesystems/overlayfs.rst b/Documentation/filesystems/overlayfs.rst
> > > > > > > > > index 580ab9a0fe31..c6e30c1bc2f2 100644
> > > > > > > > > --- a/Documentation/filesystems/overlayfs.rst
> > > > > > > > > +++ b/Documentation/filesystems/overlayfs.rst
> > > > > > > > > @@ -570,7 +570,11 @@ Volatile mount
> > > > > > > > >  This is enabled with the "volatile" mount option.  Volatile mounts are not
> > > > > > > > >  guaranteed to survive a crash.  It is strongly recommended that volatile
> > > > > > > > >  mounts are only used if data written to the overlay can be recreated
> > > > > > > > > -without significant effort.
> > > > > > > > > +without significant effort.  In addition to this, the sync family of syscalls
> > > > > > > > > +are not sufficient to determine whether a write failed as sync calls are
> > > > > > > > > +omitted.  For this reason, it is important that the filesystem used by the
> > > > > > > > > +upperdir handles failure in a fashion that's suitable for the user.  For
> > > > > > > > > +example, upon detecting a fault, ext4 can be configured to panic.
> > > > > > > > > 
> > > > > > > > 
> > > > > > > > Reading this now, I think I may have wrongly analysed the issue.
> > > > > > > > Specifically, when I wrote that the very minimum is to document the
> > > > > > > > issue, it was under the assumption that a proper fix is hard.
> > > > > > > > I think I was wrong and that the very minimum is to check for errseq
> > > > > > > > since mount on the fsync and syncfs calls.
> > > > > > > > 
> > > > > > > > Why? first of all because it is very very simple and goes a long way to
> > > > > > > > fix the broken contract with applications, not the contract about durability
> > > > > > > > obviously, but the contract about write+fsync+read expects to find the written
> > > > > > > > data (during the same mount era).
> > > > > > > > 
> > > > > > > > Second, because the sentence you added above is hard for users to understand
> > > > > > > > and out of context. If we properly handle the writeback error in fsync/syncfs,
> > > > > > > > then this sentence becomes irrelevant.
> > > > > > > > The fact that ext4 can lose data if application did not fsync after
> > > > > > > > write is true
> > > > > > > > for volatile as well as non-volatile and it is therefore not relevant
> > > > > > > > in the context
> > > > > > > > of overlayfs documentation at all.
> > > > > > > > 
> > > > > > > > Am I wrong saying that it is very very simple to fix?
> > > > > > > > Would you mind making that fix at the bottom of the patch series, so it can
> > > > > > > > be easily applied to stable kernels?
> > > > > > > > 
> > > > > > > > Thanks,
> > > > > > > > Amir.
> > > > > > > 
> > > > > > > I'm not sure it's so easy. In VFS, there are places where the superblock's
> > > > > > > errseq is checked[1]. AFAIK, that's going to check "our" errseq, and not the
> > > > > > > errseq of the real corresponding real file's superblock. One solution might be
> > > > > > > as part of all these callbacks we set our errseq to the errseq of the filesystem
> > > > > > > that the upperdir, and then rely on VFS's checking.
> > > > > > > 
> > > > > > > I'm having a hard time figuring out how to deal with the per-mapping based
> > > > > > > error tracking. It seems like this infrastructure was only partially completed
> > > > > > > by Jeff Layton[2]. I don't now how it's actually supposed to work right now,
> > > > > > > as not all of his patches landed.
> > > > > > > 
> > > > > > 
> > > > > > The patches in the series were all merged, but we ended up going with a
> > > > > > simpler solution [1] than the first series I posted. Instead of plumbing
> > > > > > the errseq_t handling down into sync_fs, we did it in the syscall
> > > > > > wrapper.
> > > > > Jeff,
> > > > > 
> > > > > Thanks for replying. I'm still a little confused as to what the
> > > > > per-address_space wb_err. It seems like we should implement the
> > > > > flush method, like so:
> > > > > diff --git a/fs/overlayfs/file.c b/fs/overlayfs/file.c
> > > > > index efccb7c1f9bc..32e5bc0aacd6 100644
> > > > > --- a/fs/overlayfs/file.c
> > > > > +++ b/fs/overlayfs/file.c
> > > > > @@ -787,6 +787,16 @@ static loff_t ovl_remap_file_range(struct file *file_in, loff_t pos_in,
> > > > >                             remap_flags, op);
> > > > >  }
> > > > > 
> > > > > +static int ovl_flush(struct file *file, fl_owner_t id)
> > > > > +{
> > > > > +       struct file *realfile = file->private_data;
> > > > > +
> > > > > +       if (real_file->f_op->flush)
> > > > > +               return real_file->f_op->flush(filp, id);
> > > > > +
> > > > > +       return 0;
> > > > > +}
> > > > > +
> > > > >  const struct file_operations ovl_file_operations = {
> > > > >         .open           = ovl_open,
> > > > >         .release        = ovl_release,
> > > > > @@ -798,6 +808,7 @@ const struct file_operations ovl_file_operations = {
> > > > >         .fallocate      = ovl_fallocate,
> > > > >         .fadvise        = ovl_fadvise,
> > > > >         .unlocked_ioctl = ovl_ioctl,
> > > > > +       .flush          = ovl_flush,
> > > > >  #ifdef CONFIG_COMPAT
> > > > >         .compat_ioctl   = ovl_compat_ioctl,
> > > > >  #endif
> > > > > 
> > > > > 
> > > > > > 
> > > > > > I think the tricky part here is that there is no struct file plumbed
> > > > > > into ->sync_fs, so you don't have an errseq_t cursor to work with by the
> > > > > > time that gets called.
> > > > > > 
> > > > > > What may be easiest is to just propagate the s_wb_err value from the
> > > > > > upper_sb to the overlayfs superblock in ovl_sync_fs(). That should get
> > > > > > called before the errseq_check_and_advance in the syncfs syscall wrapper
> > > > > > and should ensure that syncfs() calls against the overlayfs sb see any
> > > > > > errors that occurred on the upper_sb.
> > > > > > 
> > > > > > Something like this maybe? Totally untested of course. May also need to
> > > > > > think about how to ensure ordering between racing syncfs calls too
> > > > > > (don't really want the s_wb_err going "backward"):
> > > > > > 
> > > > > > ----------------------------8<---------------------------------
> > > > > > $ git diff
> > > > > > diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
> > > > > > index 290983bcfbb3..d725705abdac 100644
> > > > > > --- a/fs/overlayfs/super.c
> > > > > > +++ b/fs/overlayfs/super.c
> > > > > > @@ -283,6 +283,9 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
> > > > > >         ret = sync_filesystem(upper_sb);
> > > > > >         up_read(&upper_sb->s_umount);
> > > > > > 
> > > > > > +       /* Propagate s_wb_err from upper_sb to overlayfs sb */
> > > > > > +       WRITE_ONCE(sb->s_wb_err, READ_ONCE(upper_sb->s_wb_err));
> > > > > > +
> > > > > >         return ret;
> > > > > >  }
> > > > > > ----------------------------8<---------------------------------
> > > > > > 
> > > > > > [1]: https://www.spinics.net/lists/linux-api/msg41276.html
> > > > > 
> > > > > So,
> > > > > I think this will have bad behaviour because syncfs() twice in a row will
> > > > > return the error twice. The reason is that normally in syncfs it calls
> > > > > errseq_check_and_advance marking the error on the super block as seen. If we
> > > > > copy-up the error value each time, it will break this semantic, as we do not set
> > > > > seen on the upperdir.
> > > > > 
> > > > > Either we need to set the seen flag on the upperdir's errseq_t, or have a sync
> > > > > method, like:
> > > > > diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
> > > > > index 290983bcfbb3..4931d1797c03 100644
> > > > > --- a/fs/overlayfs/super.c
> > > > > +++ b/fs/overlayfs/super.c
> > > > > @@ -259,6 +259,7 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
> > > > >  {
> > > > >         struct ovl_fs *ofs = sb->s_fs_info;
> > > > >         struct super_block *upper_sb;
> > > > > +       errseq_t src;
> > > > >         int ret;
> > > > > 
> > > > >         if (!ovl_upper_mnt(ofs))
> > > > > @@ -283,6 +284,11 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
> > > > >         ret = sync_filesystem(upper_sb);
> > > > >         up_read(&upper_sb->s_umount);
> > > > > 
> > > > > +       /* Propagate the error up from the upper_sb once */
> > > > > +       src = READ_ONCE(upper_sb->s_wb_err);
> > > > > +       if (errseq_counter(src) != errseq_counter(sb->s_wb_err))
> > > > > +               WRITE_ONCE(sb->s_wb_err, src & ~ERRSEQ_SEEN);
> > > > > +
> > > > >         return ret;
> > > > >  }
> > > > > 
> > > > > @@ -1945,6 +1951,7 @@ static int ovl_fill_super(struct super_block *sb, void *data, int silent)
> > > > > 
> > > > >                 sb->s_stack_depth = ovl_upper_mnt(ofs)->mnt_sb->s_stack_depth;
> > > > >                 sb->s_time_gran = ovl_upper_mnt(ofs)->mnt_sb->s_time_gran;
> > > > > +               sb->s_wb_err = READ_ONCE(ovl_upper_mnt(ofs)->mnt_sb->s_wb_err);
> > > > > 
> > > > > diff --git a/lib/errseq.c b/lib/errseq.c
> > > > > index 81f9e33aa7e7..53275c168265 100644
> > > > > --- a/lib/errseq.c
> > > > > +++ b/lib/errseq.c
> > > > > @@ -204,3 +204,18 @@ int errseq_check_and_advance(errseq_t *eseq, errseq_t *since)
> > > > >         return err;
> > > > >  }
> > > > >  EXPORT_SYMBOL(errseq_check_and_advance);
> > > > > +
> > > > > +/**
> > > > > + * errseq_counter() - Get the value of the errseq counter
> > > > > + * @eseq: Value being checked
> > > > > + *
> > > > > + * This returns the errseq_t counter value. Reminder, it can wrap because
> > > > > + * there are only a limited number of counter bits.
> > > > > + *
> > > > > + * Return: The counter portion of the errseq_t.
> > > > > + */
> > > > > +int errseq_counter(errseq_t eseq)
> > > > > +{
> > > > > +       return eseq >> (ERRSEQ_SHIFT + 1);
> > > > > +}
> > > > > 
> > > > > This also needs some locking sprinkled on it because racing can occur with
> > > > > sync_fs. This would be much easier if the errseq_t was 64-bits, and didn't go
> > > > > backwards, because you could just use a simple cmpxchg64. I know that would make
> > > > > a bunch of structures larger, and if it's just overlayfs that has to pay the tax
> > > > > we can just sprinkle a mutex on it.
> > > > > 
> > > > > We can always "copy-up" the errseq_t because if the upperdir's errseq_t is read,
> > > > > and we haven't done a copy yet, we'll get it. Since this will be strictly
> > > > > serialized a simple equivalence check will work versus actually having to deal
> > > > > with happens before behaviour. There is still a correctness flaw here though,
> > > > > which is exactly enough errors occur to result in it wrapping back to the same
> > > > > value, it will break.
> > > > > 
> > > > > By the way, how do y'all test this error handling behaviour? I didn't
> > > > > find any automated testing for what currently exists.
> > > > > > 
> > > > > > How about I split this into two patchsets? One, where I add the logic to copy
> > > > > > the errseq on callbacks to fsync from the upperdir to the ovl fs superblock,
> > > > > > and thus allowing VFS to bubble up errors, and the documentation. We can CC
> > > > > > stable on those because I think it has an effect that's universal across
> > > > > > all filesystems.
> > > > > > 
> > > > > > P.S.
> > > > > > I notice you maintain overlay tests outside of the kernel. Unfortunately, I
> > > > > > think for this kind of test, it requires in kernel code to artificially bump the
> > > > > > writeback error count on the upperdir, or it requires the failure injection
> > > > > > infrastructure.
> > > > > > 
> > > > > > Simulating this behaviour is non-trivial without in-kernel support:
> > > > > > 
> > > > > > P1: Open(f) -> p1.fd
> > > > > > P2: Open(f) -> p2.fd
> > > > > > P1: syncfs(p1.fd) -> errrno
> > > > > > P2: syncfs(p1.fd) -> 0 # This should return an error
> > > > > > 
> > > > > > 
> > > > > > [1]: https://elixir.bootlin.com/linux/latest/source/fs/sync.c#L175
> > > > > > [2]: https://lwn.net/Articles/722250/
> > > > > > 
> > > > > > 
> > > > 
> > > > Sargun (and Jeff),
> > > > 
> > > > Thank you for this discussion. It would be very nice to work on testing
> > > > and fixing errseq propagation is correct on overlayfs.
> > > > Alas, this is not what I suggested.
> > > > 
> > > > What I suggested was a solution only for the volatile overlay issue
> > > > where data can vaporise without applications noticing:
> > > > "...the very minimum is to check for errseq since mount on the fsync
> > > >  and syncfs calls."
> > > > 
> > > Yeah, I was confusing the checking that VFS does on our behalf and the checking 
> > > that we can do ourselves in the sync callback. If we return an error prior to 
> > > the vfs checking it short-circuits that entirely.
> > > 
> > > > Do you get it? there is no pre-file state in the game, not for fsync and not 
> > > > for syncfs.
> > > > 
> > > > Any single error, no matter how temporary it is and what damage it may
> > > > or may not have caused to upper layer consistency, permanently
> > > > invalidates the reliability of the volatile overlay, resulting in:
> > > > Effective immediately: every fsync/syncfs returns EIO.
> > > > Going forward: maybe implement overlay shutdown, so every access
> > > > returns EIO.
> > > > 
> > > > So now that I hopefully explained myself better, I'll ask again:
> > > > Am I wrong saying that it is very very simple to fix?
> > > > Would you mind making that fix at the bottom of the patch series, so it can
> > > > be easily applied to stable kernels?
> > > > 
> > > > Thanks,
> > > > Amir.
> > > 
> > > I think that this should be easy enough if the semantic is such that volatile
> > > overlayfs mounts will return EIO on syncfs on every syncfs call if the upperdir's
> > > super block has experienced errors since the initial mount. I imagine we do not
> > > want to make it such that if the upperdir has ever experienced errors, return
> > > EIO on syncfs.
> > > 
> > > The one caveat that I see is that if the errseq wraps, we can silently begin 
> > > swallowing errors again. Thus, on the first failed syncfs we should just
> > > store a flag indicating that the volatile fs is bad, and to continue to return
> > > EIO rather than go through the process of checking errseq_t, but that's easy
> > > enough to write.
> > 
> > It turns out this is much more complicated than I initially thought. I'm not 
> > entirely sure why VFS even defines sync_fs (on the superblock) as returning an 
> > int. The way that sync_fs works is:
> > 
> > sys_syncfs ->
> >   sync_filesystem(sb) ->
> >     __sync_filesystem(sb, N) ->
> >       sb->s_op->sync_fs
> >       /* __sync_blockdev has its own writeback error checking */
> >       __sync_blockdev
> >   errseq_check_and_advance(sb...)
> > 
> > __sync_filesystem calls the sync_fs callback on the superblock's superblock
> > operations:
> > 
> > static int __sync_filesystem(struct super_block *sb, int wait)
> > {
> > 	if (wait)
> > 		sync_inodes_sb(sb);
> > 	else
> > 		writeback_inodes_sb(sb, WB_REASON_SYNC);
> > 
> > 	if (sb->s_op->sync_fs)
> > 		sb->s_op->sync_fs(sb, wait);
> > 	return __sync_blockdev(sb->s_bdev, wait);
> > }
> > 
> > But it completely discards the result. On the other hand, fsync, and fdatasync
> > exhibit different behaviour:
> > 
> > sys_fdatasync ->
> >   do_fsync
> >    (see below)
> > 
> > sys_fsync ->
> >   do_fsync ->
> >     vfs_fsync ->
> >       vfs_fsync_range ->
> >         return file->f_op->fsync
> > ----
> > 
> 
> This is mainly a result of the evolution of the code. In the beginning
> there was only sync() and it doesn't report errors. Eventually syncfs()
> was added, but the only error it reliably reported was EBADF.
> 
> I added the machinery to record writeback errors of individual inodes at
> the superblock level recently, but the sync_fs op was left alone (mainly
> because changing this would be very invasive, and the value of doing so
> wasn't completely clear).

Is following patch enough to capture ->sync_fs return value and return
to user space on syncfs. Or there is more to it which I am missing.


---
 fs/sync.c |    7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

Index: redhat-linux/fs/sync.c
===================================================================
--- redhat-linux.orig/fs/sync.c	2020-08-10 15:40:22.535349278 -0400
+++ redhat-linux/fs/sync.c	2020-12-01 10:19:03.624330578 -0500
@@ -30,14 +30,17 @@
  */
 static int __sync_filesystem(struct super_block *sb, int wait)
 {
+	int ret, ret2;
+
 	if (wait)
 		sync_inodes_sb(sb);
 	else
 		writeback_inodes_sb(sb, WB_REASON_SYNC);
 
 	if (sb->s_op->sync_fs)
-		sb->s_op->sync_fs(sb, wait);
-	return __sync_blockdev(sb->s_bdev, wait);
+		ret = sb->s_op->sync_fs(sb, wait);
+	ret2 = __sync_blockdev(sb->s_bdev, wait);
+	return ret ? ret : ret2;
 }
 
 /*


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-17.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,
	URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id ADE93C64E7B
	for <linux-unionfs@archiver.kernel.org>; Tue,  1 Dec 2020 16:12:16 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id 15C9E2225D
	for <linux-unionfs@archiver.kernel.org>; Tue,  1 Dec 2020 16:12:16 +0000 (UTC)
Authentication-Results: mail.kernel.org;
	dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Ic86Bldi"
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S2403954AbgLAQMA (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Tue, 1 Dec 2020 11:12:00 -0500
Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:56300 "EHLO
        us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK)
        by vger.kernel.org with ESMTP id S2392123AbgLAQL7 (ORCPT
        <rfc822;linux-unionfs@vger.kernel.org>);
        Tue, 1 Dec 2020 11:11:59 -0500
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;
        s=mimecast20190719; t=1606839031;
        h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
         to:to:cc:cc:mime-version:mime-version:content-type:content-type:
         content-transfer-encoding:content-transfer-encoding:
         in-reply-to:in-reply-to:references:references;
        bh=C/8GDlsepmNxbtynUo3UMwYspSr8LBSZN/TvksieEj8=;
        b=Ic86BldiqaD0VVP06q2Y475c5e5Ja0DxnT8VckrDXR/F8zEMdkgFxYCu6C9GzC3f0Aln2+
        zuRES01mp6dFp5x15eqzocQ/Xor7xIzq/iI0LPiM76vPBWPA1y52mblaXedVlHpiMsbOSW
        DuqTK6Zf1FJGCobMjVGjDPD5aoc+VS4=
Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com
 [209.85.219.72]) (Using TLS) by relay.mimecast.com with ESMTP id
 us-mta-475-YavxG80QNLSlnS0-SWwDvA-1; Tue, 01 Dec 2020 11:10:26 -0500
X-MC-Unique: YavxG80QNLSlnS0-SWwDvA-1
Received: by mail-qv1-f72.google.com with SMTP id y8so1543497qvu.22
        for <linux-unionfs@vger.kernel.org>; Tue, 01 Dec 2020 08:10:26 -0800 (PST)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to
         :references:user-agent:mime-version:content-transfer-encoding;
        bh=C/8GDlsepmNxbtynUo3UMwYspSr8LBSZN/TvksieEj8=;
        b=qcF4pb+VV3jOc9pa87JpDhZvg61aNNe271rFqycfPmyw2e8UR7EJAr1SbXB7Ucbygi
         oDBljtkmN5vdbhhh0pbZVw9D0zvMOXpQErQUduRN6mMMkAzwVWMsPW4WU1GRfWravhmJ
         XJ/ZZaDsT3+EK+XLMeYhHHU7PgVueXQ4ZYkt1ttYXhGdTQ40v+eF+0AD2YCfurYvVZyf
         KOQFjB5wUNiNVliDkHUjz2iRYZ5XemOZCyvgFQ1zUd1e6MZbNSQ34L7383kf56z7jIXx
         0xMcA1Paz01iQ+joaJY1g1AZdJUVr/Tgyg5g6lE0JWSeI46t6Rkfqz3Bfl/pxbisAlpj
         7crQ==
X-Gm-Message-State: AOAM531Tu5I5qWlYllZ8bwgCFC02s4ihc7nung8zUXIW/eWgJzdTFuw4
        BphAg4G3V+8/vo2fTwJY5bBhqMXNJnX0McfZZ1ujuW9nPIM5mwa0wjbvvVEdqiIc0ACFB2kB/FX
        9O7B3sZO/t1tvRi3oC6dJVnYbxg==
X-Received: by 2002:a05:620a:22eb:: with SMTP id p11mr3709384qki.224.1606839025715;
        Tue, 01 Dec 2020 08:10:25 -0800 (PST)
X-Google-Smtp-Source: ABdhPJypK/3+cJ5spjC5os9iOVXQtUAlt1i46xak9KjSPp1Y9RwvU5fHz7Ek9YOwbEigLHdOj3FL9A==
X-Received: by 2002:a05:620a:22eb:: with SMTP id p11mr3709347qki.224.1606839025301;
        Tue, 01 Dec 2020 08:10:25 -0800 (PST)
Received: from tleilax.poochiereds.net (68-20-15-154.lightspeed.rlghnc.sbcglobal.net. [68.20.15.154])
        by smtp.gmail.com with ESMTPSA id y24sm2146078qtv.76.2020.12.01.08.10.23
        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
        Tue, 01 Dec 2020 08:10:23 -0800 (PST)
Message-ID: <36d820394c3e7cd1faa1b28a8135136d5001dadd.camel@redhat.com>
Subject: Re: [PATCH v2 2/4] overlay: Document current outstanding
 shortcoming of volatile
From:   Jeff Layton <jlayton@redhat.com>
To:     Vivek Goyal <vgoyal@redhat.com>
Cc:     Sargun Dhillon <sargun@sargun.me>,
        Amir Goldstein <amir73il@gmail.com>,
        overlayfs <linux-unionfs@vger.kernel.org>,
        Miklos Szeredi <miklos@szeredi.hu>,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel <linux-fsdevel@vger.kernel.org>,
        David Howells <dhowells@redhat.com>
Date:   Tue, 01 Dec 2020 11:10:22 -0500
In-Reply-To: <20201201152404.GA86704@redhat.com>
References: <20201127092058.15117-1-sargun@sargun.me>
         <20201127092058.15117-3-sargun@sargun.me>
         <CAOQ4uxgaLuLb+f6WCMvmKHNTELvcvN8C5_u=t5hhoGT8Op7QuQ@mail.gmail.com>
         <20201127221154.GA23383@ircssh-2.c.rugged-nimbus-611.internal>
         <1338a059d03db0e85cf3f3234fd33434a45606c6.camel@redhat.com>
         <20201128044530.GA28230@ircssh-2.c.rugged-nimbus-611.internal>
         <CAOQ4uxjT6FF03Sq3qXuqDcqJQnzQq2dD_XVbuj_Fb9A2Ag585w@mail.gmail.com>
         <20201128085227.GB28230@ircssh-2.c.rugged-nimbus-611.internal>
         <20201201110928.GA24837@ircssh-2.c.rugged-nimbus-611.internal>
         <0f831ca2a65b028c2682819228eb5560230a0e4d.camel@redhat.com>
         <20201201152404.GA86704@redhat.com>
Content-Type: text/plain; charset="UTF-8"
User-Agent: Evolution 3.38.1 (3.38.1-1.fc33) 
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org

On Tue, 2020-12-01 at 10:24 -0500, Vivek Goyal wrote:
> On Tue, Dec 01, 2020 at 08:01:11AM -0500, Jeff Layton wrote:
> > On Tue, 2020-12-01 at 11:09 +0000, Sargun Dhillon wrote:
> > > On Sat, Nov 28, 2020 at 08:52:27AM +0000, Sargun Dhillon wrote:
> > > > On Sat, Nov 28, 2020 at 09:12:03AM +0200, Amir Goldstein wrote:
> > > > > On Sat, Nov 28, 2020 at 6:45 AM Sargun Dhillon <sargun@sargun.me> wrote:
> > > > > > 
> > > > > > On Fri, Nov 27, 2020 at 09:01:07PM -0500, Jeff Layton wrote:
> > > > > > > On Fri, 2020-11-27 at 22:11 +0000, Sargun Dhillon wrote:
> > > > > > > > On Fri, Nov 27, 2020 at 02:52:52PM +0200, Amir Goldstein wrote:
> > > > > > > > > On Fri, Nov 27, 2020 at 11:21 AM Sargun Dhillon <sargun@sargun.me> wrote:
> > > > > > > > > > 
> > > > > > > > > > This documents behaviour that was discussed in a thread about the volatile
> > > > > > > > > > feature. Specifically, how failures can go hidden from asynchronous writes
> > > > > > > > > > (such as from mmap, or writes that are not immediately flushed to the
> > > > > > > > > > filesystem). Although we pass through calls like msync, fallocate, and
> > > > > > > > > > write, and will still return errors on those, it doesn't guarantee all
> > > > > > > > > > kinds of errors will happen at those times, and thus may hide errors.
> > > > > > > > > > 
> > > > > > > > > > In the future, we can add error checking to all interactions with the
> > > > > > > > > > upperdir, and pass through errseq_t from the upperdir on mappings,
> > > > > > > > > > and other interactions with the filesystem[1].
> > > > > > > > > > 
> > > > > > > > > > [1]: https://lore.kernel.org/linux-unionfs/20201116045758.21774-1-sargun@sargun.me/T/#m7d501f375e031056efad626e471a1392dd3aad33
> > > > > > > > > > 
> > > > > > > > > > Signed-off-by: Sargun Dhillon <sargun@sargun.me>
> > > > > > > > > > Cc: linux-fsdevel@vger.kernel.org
> > > > > > > > > > Cc: linux-unionfs@vger.kernel.org
> > > > > > > > > > Cc: Miklos Szeredi <miklos@szeredi.hu>
> > > > > > > > > > Cc: Amir Goldstein <amir73il@gmail.com>
> > > > > > > > > > Cc: Vivek Goyal <vgoyal@redhat.com>
> > > > > > > > > > ---
> > > > > > > > > > Ã‚Â Documentation/filesystems/overlayfs.rst | 6 +++++-
> > > > > > > > > > Ã‚Â 1 file changed, 5 insertions(+), 1 deletion(-)
> > > > > > > > > > 
> > > > > > > > > > diff --git a/Documentation/filesystems/overlayfs.rst b/Documentation/filesystems/overlayfs.rst
> > > > > > > > > > index 580ab9a0fe31..c6e30c1bc2f2 100644
> > > > > > > > > > --- a/Documentation/filesystems/overlayfs.rst
> > > > > > > > > > +++ b/Documentation/filesystems/overlayfs.rst
> > > > > > > > > > @@ -570,7 +570,11 @@ Volatile mount
> > > > > > > > > > Ã‚Â This is enabled with the "volatile" mount option.  Volatile mounts are not
> > > > > > > > > > Ã‚Â guaranteed to survive a crash.  It is strongly recommended that volatile
> > > > > > > > > > Ã‚Â mounts are only used if data written to the overlay can be recreated
> > > > > > > > > > -without significant effort.
> > > > > > > > > > +without significant effort.  In addition to this, the sync family of syscalls
> > > > > > > > > > +are not sufficient to determine whether a write failed as sync calls are
> > > > > > > > > > +omitted.  For this reason, it is important that the filesystem used by the
> > > > > > > > > > +upperdir handles failure in a fashion that's suitable for the user.  For
> > > > > > > > > > +example, upon detecting a fault, ext4 can be configured to panic.
> > > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > Reading this now, I think I may have wrongly analysed the issue.
> > > > > > > > > Specifically, when I wrote that the very minimum is to document the
> > > > > > > > > issue, it was under the assumption that a proper fix is hard.
> > > > > > > > > I think I was wrong and that the very minimum is to check for errseq
> > > > > > > > > since mount on the fsync and syncfs calls.
> > > > > > > > > 
> > > > > > > > > Why? first of all because it is very very simple and goes a long way to
> > > > > > > > > fix the broken contract with applications, not the contract about durability
> > > > > > > > > obviously, but the contract about write+fsync+read expects to find the written
> > > > > > > > > data (during the same mount era).
> > > > > > > > > 
> > > > > > > > > Second, because the sentence you added above is hard for users to understand
> > > > > > > > > and out of context. If we properly handle the writeback error in fsync/syncfs,
> > > > > > > > > then this sentence becomes irrelevant.
> > > > > > > > > The fact that ext4 can lose data if application did not fsync after
> > > > > > > > > write is true
> > > > > > > > > for volatile as well as non-volatile and it is therefore not relevant
> > > > > > > > > in the context
> > > > > > > > > of overlayfs documentation at all.
> > > > > > > > > 
> > > > > > > > > Am I wrong saying that it is very very simple to fix?
> > > > > > > > > Would you mind making that fix at the bottom of the patch series, so it can
> > > > > > > > > be easily applied to stable kernels?
> > > > > > > > > 
> > > > > > > > > Thanks,
> > > > > > > > > Amir.
> > > > > > > > 
> > > > > > > > I'm not sure it's so easy. In VFS, there are places where the superblock's
> > > > > > > > errseq is checked[1]. AFAIK, that's going to check "our" errseq, and not the
> > > > > > > > errseq of the real corresponding real file's superblock. One solution might be
> > > > > > > > as part of all these callbacks we set our errseq to the errseq of the filesystem
> > > > > > > > that the upperdir, and then rely on VFS's checking.
> > > > > > > > 
> > > > > > > > I'm having a hard time figuring out how to deal with the per-mapping based
> > > > > > > > error tracking. It seems like this infrastructure was only partially completed
> > > > > > > > by Jeff Layton[2]. I don't now how it's actually supposed to work right now,
> > > > > > > > as not all of his patches landed.
> > > > > > > > 
> > > > > > > 
> > > > > > > The patches in the series were all merged, but we ended up going with a
> > > > > > > simpler solution [1] than the first series I posted. Instead of plumbing
> > > > > > > the errseq_t handling down into sync_fs, we did it in the syscall
> > > > > > > wrapper.
> > > > > > Jeff,
> > > > > > 
> > > > > > Thanks for replying. I'm still a little confused as to what the
> > > > > > per-address_space wb_err. It seems like we should implement the
> > > > > > flush method, like so:
> > > > > > diff --git a/fs/overlayfs/file.c b/fs/overlayfs/file.c
> > > > > > index efccb7c1f9bc..32e5bc0aacd6 100644
> > > > > > --- a/fs/overlayfs/file.c
> > > > > > +++ b/fs/overlayfs/file.c
> > > > > > @@ -787,6 +787,16 @@ static loff_t ovl_remap_file_range(struct file *file_in, loff_t pos_in,
> > > > > > Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â remap_flags, op);
> > > > > > Ã‚Â }
> > > > > > 
> > > > > > +static int ovl_flush(struct file *file, fl_owner_t id)
> > > > > > +{
> > > > > > +       struct file *realfile = file->private_data;
> > > > > > +
> > > > > > +       if (real_file->f_op->flush)
> > > > > > +               return real_file->f_op->flush(filp, id);
> > > > > > +
> > > > > > +       return 0;
> > > > > > +}
> > > > > > +
> > > > > > Ã‚Â const struct file_operations ovl_file_operations = {
> > > > > > Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â .open           = ovl_open,
> > > > > > Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â .release        = ovl_release,
> > > > > > @@ -798,6 +808,7 @@ const struct file_operations ovl_file_operations = {
> > > > > > Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â .fallocate      = ovl_fallocate,
> > > > > > Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â .fadvise        = ovl_fadvise,
> > > > > > Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â .unlocked_ioctl = ovl_ioctl,
> > > > > > +       .flush          = ovl_flush,
> > > > > > Ã‚Â #ifdef CONFIG_COMPAT
> > > > > > Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â .compat_ioctl   = ovl_compat_ioctl,
> > > > > > Ã‚Â #endif
> > > > > > 
> > > > > > 
> > > > > > > 
> > > > > > > I think the tricky part here is that there is no struct file plumbed
> > > > > > > into ->sync_fs, so you don't have an errseq_t cursor to work with by the
> > > > > > > time that gets called.
> > > > > > > 
> > > > > > > What may be easiest is to just propagate the s_wb_err value from the
> > > > > > > upper_sb to the overlayfs superblock in ovl_sync_fs(). That should get
> > > > > > > called before the errseq_check_and_advance in the syncfs syscall wrapper
> > > > > > > and should ensure that syncfs() calls against the overlayfs sb see any
> > > > > > > errors that occurred on the upper_sb.
> > > > > > > 
> > > > > > > Something like this maybe? Totally untested of course. May also need to
> > > > > > > think about how to ensure ordering between racing syncfs calls too
> > > > > > > (don't really want the s_wb_err going "backward"):
> > > > > > > 
> > > > > > > ----------------------------8<---------------------------------
> > > > > > > $ git diff
> > > > > > > diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
> > > > > > > index 290983bcfbb3..d725705abdac 100644
> > > > > > > --- a/fs/overlayfs/super.c
> > > > > > > +++ b/fs/overlayfs/super.c
> > > > > > > @@ -283,6 +283,9 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
> > > > > > > Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â ret = sync_filesystem(upper_sb);
> > > > > > > Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â up_read(&upper_sb->s_umount);
> > > > > > > 
> > > > > > > +       /* Propagate s_wb_err from upper_sb to overlayfs sb */
> > > > > > > +       WRITE_ONCE(sb->s_wb_err, READ_ONCE(upper_sb->s_wb_err));
> > > > > > > +
> > > > > > > Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â return ret;
> > > > > > > Ã‚Â }
> > > > > > > ----------------------------8<---------------------------------
> > > > > > > 
> > > > > > > [1]: https://www.spinics.net/lists/linux-api/msg41276.html
> > > > > > 
> > > > > > So,
> > > > > > I think this will have bad behaviour because syncfs() twice in a row will
> > > > > > return the error twice. The reason is that normally in syncfs it calls
> > > > > > errseq_check_and_advance marking the error on the super block as seen. If we
> > > > > > copy-up the error value each time, it will break this semantic, as we do not set
> > > > > > seen on the upperdir.
> > > > > > 
> > > > > > Either we need to set the seen flag on the upperdir's errseq_t, or have a sync
> > > > > > method, like:
> > > > > > diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
> > > > > > index 290983bcfbb3..4931d1797c03 100644
> > > > > > --- a/fs/overlayfs/super.c
> > > > > > +++ b/fs/overlayfs/super.c
> > > > > > @@ -259,6 +259,7 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
> > > > > > Ã‚Â {
> > > > > > Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â struct ovl_fs *ofs = sb->s_fs_info;
> > > > > > Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â struct super_block *upper_sb;
> > > > > > +       errseq_t src;
> > > > > > Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â int ret;
> > > > > > 
> > > > > > Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â if (!ovl_upper_mnt(ofs))
> > > > > > @@ -283,6 +284,11 @@ static int ovl_sync_fs(struct super_block *sb, int wait)
> > > > > > Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â ret = sync_filesystem(upper_sb);
> > > > > > Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â up_read(&upper_sb->s_umount);
> > > > > > 
> > > > > > +       /* Propagate the error up from the upper_sb once */
> > > > > > +       src = READ_ONCE(upper_sb->s_wb_err);
> > > > > > +       if (errseq_counter(src) != errseq_counter(sb->s_wb_err))
> > > > > > +               WRITE_ONCE(sb->s_wb_err, src & ~ERRSEQ_SEEN);
> > > > > > +
> > > > > > Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â return ret;
> > > > > > Ã‚Â }
> > > > > > 
> > > > > > @@ -1945,6 +1951,7 @@ static int ovl_fill_super(struct super_block *sb, void *data, int silent)
> > > > > > 
> > > > > > Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â sb->s_stack_depth = ovl_upper_mnt(ofs)->mnt_sb->s_stack_depth;
> > > > > > Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â sb->s_time_gran = ovl_upper_mnt(ofs)->mnt_sb->s_time_gran;
> > > > > > +               sb->s_wb_err = READ_ONCE(ovl_upper_mnt(ofs)->mnt_sb->s_wb_err);
> > > > > > 
> > > > > > diff --git a/lib/errseq.c b/lib/errseq.c
> > > > > > index 81f9e33aa7e7..53275c168265 100644
> > > > > > --- a/lib/errseq.c
> > > > > > +++ b/lib/errseq.c
> > > > > > @@ -204,3 +204,18 @@ int errseq_check_and_advance(errseq_t *eseq, errseq_t *since)
> > > > > > Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â return err;
> > > > > > Ã‚Â }
> > > > > > Ã‚Â EXPORT_SYMBOL(errseq_check_and_advance);
> > > > > > +
> > > > > > +/**
> > > > > > + * errseq_counter() - Get the value of the errseq counter
> > > > > > + * @eseq: Value being checked
> > > > > > + *
> > > > > > + * This returns the errseq_t counter value. Reminder, it can wrap because
> > > > > > + * there are only a limited number of counter bits.
> > > > > > + *
> > > > > > + * Return: The counter portion of the errseq_t.
> > > > > > + */
> > > > > > +int errseq_counter(errseq_t eseq)
> > > > > > +{
> > > > > > +       return eseq >> (ERRSEQ_SHIFT + 1);
> > > > > > +}
> > > > > > 
> > > > > > This also needs some locking sprinkled on it because racing can occur with
> > > > > > sync_fs. This would be much easier if the errseq_t was 64-bits, and didn't go
> > > > > > backwards, because you could just use a simple cmpxchg64. I know that would make
> > > > > > a bunch of structures larger, and if it's just overlayfs that has to pay the tax
> > > > > > we can just sprinkle a mutex on it.
> > > > > > 
> > > > > > We can always "copy-up" the errseq_t because if the upperdir's errseq_t is read,
> > > > > > and we haven't done a copy yet, we'll get it. Since this will be strictly
> > > > > > serialized a simple equivalence check will work versus actually having to deal
> > > > > > with happens before behaviour. There is still a correctness flaw here though,
> > > > > > which is exactly enough errors occur to result in it wrapping back to the same
> > > > > > value, it will break.
> > > > > > 
> > > > > > By the way, how do y'all test this error handling behaviour? I didn't
> > > > > > find any automated testing for what currently exists.
> > > > > > > 
> > > > > > > How about I split this into two patchsets? One, where I add the logic to copy
> > > > > > > the errseq on callbacks to fsync from the upperdir to the ovl fs superblock,
> > > > > > > and thus allowing VFS to bubble up errors, and the documentation. We can CC
> > > > > > > stable on those because I think it has an effect that's universal across
> > > > > > > all filesystems.
> > > > > > > 
> > > > > > > P.S.
> > > > > > > I notice you maintain overlay tests outside of the kernel. Unfortunately, I
> > > > > > > think for this kind of test, it requires in kernel code to artificially bump the
> > > > > > > writeback error count on the upperdir, or it requires the failure injection
> > > > > > > infrastructure.
> > > > > > > 
> > > > > > > Simulating this behaviour is non-trivial without in-kernel support:
> > > > > > > 
> > > > > > > P1: Open(f) -> p1.fd
> > > > > > > P2: Open(f) -> p2.fd
> > > > > > > P1: syncfs(p1.fd) -> errrno
> > > > > > > P2: syncfs(p1.fd) -> 0 # This should return an error
> > > > > > > 
> > > > > > > 
> > > > > > > [1]: https://elixir.bootlin.com/linux/latest/source/fs/sync.c#L175
> > > > > > > [2]: https://lwn.net/Articles/722250/
> > > > > > > 
> > > > > > > 
> > > > > 
> > > > > Sargun (and Jeff),
> > > > > 
> > > > > Thank you for this discussion. It would be very nice to work on testing
> > > > > and fixing errseq propagation is correct on overlayfs.
> > > > > Alas, this is not what I suggested.
> > > > > 
> > > > > What I suggested was a solution only for the volatile overlay issue
> > > > > where data can vaporise without applications noticing:
> > > > > "...the very minimum is to check for errseq since mount on the fsync
> > > > > Ã‚Â and syncfs calls."
> > > > > 
> > > > Yeah, I was confusing the checking that VFS does on our behalf and the checking 
> > > > that we can do ourselves in the sync callback. If we return an error prior to 
> > > > the vfs checking it short-circuits that entirely.
> > > > 
> > > > > Do you get it? there is no pre-file state in the game, not for fsync and not 
> > > > > for syncfs.
> > > > > 
> > > > > Any single error, no matter how temporary it is and what damage it may
> > > > > or may not have caused to upper layer consistency, permanently
> > > > > invalidates the reliability of the volatile overlay, resulting in:
> > > > > Effective immediately: every fsync/syncfs returns EIO.
> > > > > Going forward: maybe implement overlay shutdown, so every access
> > > > > returns EIO.
> > > > > 
> > > > > So now that I hopefully explained myself better, I'll ask again:
> > > > > Am I wrong saying that it is very very simple to fix?
> > > > > Would you mind making that fix at the bottom of the patch series, so it can
> > > > > be easily applied to stable kernels?
> > > > > 
> > > > > Thanks,
> > > > > Amir.
> > > > 
> > > > I think that this should be easy enough if the semantic is such that volatile
> > > > overlayfs mounts will return EIO on syncfs on every syncfs call if the upperdir's
> > > > super block has experienced errors since the initial mount. I imagine we do not
> > > > want to make it such that if the upperdir has ever experienced errors, return
> > > > EIO on syncfs.
> > > > 
> > > > The one caveat that I see is that if the errseq wraps, we can silently begin 
> > > > swallowing errors again. Thus, on the first failed syncfs we should just
> > > > store a flag indicating that the volatile fs is bad, and to continue to return
> > > > EIO rather than go through the process of checking errseq_t, but that's easy
> > > > enough to write.
> > > 
> > > It turns out this is much more complicated than I initially thought. I'm not 
> > > entirely sure why VFS even defines sync_fs (on the superblock) as returning an 
> > > int. The way that sync_fs works is:
> > > 
> > > sys_syncfs ->
> > > Ã‚Â Ã‚Â sync_filesystem(sb) ->
> > > Ã‚Â Ã‚Â Ã‚Â Ã‚Â __sync_filesystem(sb, N) ->
> > > Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â sb->s_op->sync_fs
> > > Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â /* __sync_blockdev has its own writeback error checking */
> > > Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â __sync_blockdev
> > > Ã‚Â Ã‚Â errseq_check_and_advance(sb...)
> > > 
> > > __sync_filesystem calls the sync_fs callback on the superblock's superblock
> > > operations:
> > > 
> > > static int __sync_filesystem(struct super_block *sb, int wait)
> > > {
> > > 	if (wait)
> > > 		sync_inodes_sb(sb);
> > > 	else
> > > 		writeback_inodes_sb(sb, WB_REASON_SYNC);
> > > 
> > > 	if (sb->s_op->sync_fs)
> > > 		sb->s_op->sync_fs(sb, wait);
> > > 	return __sync_blockdev(sb->s_bdev, wait);
> > > }
> > > 
> > > But it completely discards the result. On the other hand, fsync, and fdatasync
> > > exhibit different behaviour:
> > > 
> > > sys_fdatasync ->
> > > Ã‚Â Ã‚Â do_fsync
> > > Ã‚Â Ã‚Â Ã‚Â (see below)
> > > 
> > > sys_fsync ->
> > > Ã‚Â Ã‚Â do_fsync ->
> > > Ã‚Â Ã‚Â Ã‚Â Ã‚Â vfs_fsync ->
> > > Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â vfs_fsync_range ->
> > > Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â Ã‚Â return file->f_op->fsync
> > > ----
> > > 
> > 
> > This is mainly a result of the evolution of the code. In the beginning
> > there was only sync() and it doesn't report errors. Eventually syncfs()
> > was added, but the only error it reliably reported was EBADF.
> > 
> > I added the machinery to record writeback errors of individual inodes at
> > the superblock level recently, but the sync_fs op was left alone (mainly
> > because changing this would be very invasive, and the value of doing so
> > wasn't completely clear).
> 
> Is following patch enough to capture ->sync_fs return value and return
> to user space on syncfs. Or there is more to it which I am missing.
> 
> 
> ---
> Â fs/sync.c |    7 +++++--
> Â 1 file changed, 5 insertions(+), 2 deletions(-)
> 
> Index: redhat-linux/fs/sync.c
> ===================================================================
> --- redhat-linux.orig/fs/sync.c	2020-08-10 15:40:22.535349278 -0400
> +++ redhat-linux/fs/sync.c	2020-12-01 10:19:03.624330578 -0500
> @@ -30,14 +30,17 @@
> Â Â */
> Â static int __sync_filesystem(struct super_block *sb, int wait)
> Â {
> +	int ret, ret2;
> +
> Â 	if (wait)
> Â 		sync_inodes_sb(sb);
> Â 	else
> Â 		writeback_inodes_sb(sb, WB_REASON_SYNC);
> Â 
> 
> Â 	if (sb->s_op->sync_fs)
> -		sb->s_op->sync_fs(sb, wait);
> -	return __sync_blockdev(sb->s_bdev, wait);
> +		ret = sb->s_op->sync_fs(sb, wait);
> +	ret2 = __sync_blockdev(sb->s_bdev, wait);
> +	return ret ? ret : ret2;
> Â }
> Â 
> 
> Â /*
> 

Maybe. The question of course is what will regress once we make that
change. Are there sync_fs routines in place today that regularly return
errors that are just ignored? It's difficult to know without testing all
the affected filesystems.

If we did decide to do that, then there are some other callers of
sync_fs -- dquot_quota_sync, for instance, and we probably need to vet
them carefully.

Also, a sync_fs error could be transient. Suppose you have something
call sync() and it gets an error back from sync_fs (which is ignored),
and then something else calls syncfs() and doesn't see any error because
its sync_fs call succeeded. Arguably, we should return an error in that
situation.

What may work best is to just do an errseq_set on sb->s_wb_err when the
either sync_fs or __sync_blockdev returns an error. Then, the
errseq_check_and_advance in syncfs can eventually report it.
-- 
Jeff Layton <jlayton@redhat.com>


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,
	HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham
	autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id BD3CBC433FE
	for <linux-unionfs@archiver.kernel.org>; Sat,  5 Dec 2020 09:29:04 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id 86B8222D0B
	for <linux-unionfs@archiver.kernel.org>; Sat,  5 Dec 2020 09:29:04 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1726432AbgLEJ1x (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Sat, 5 Dec 2020 04:27:53 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42252 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1727673AbgLEJXn (ORCPT
        <rfc822;linux-unionfs@vger.kernel.org>);
        Sat, 5 Dec 2020 04:23:43 -0500
Received: from mail-io1-xd41.google.com (mail-io1-xd41.google.com [IPv6:2607:f8b0:4864:20::d41])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9C8D5C061A51;
        Sat,  5 Dec 2020 01:13:25 -0800 (PST)
Received: by mail-io1-xd41.google.com with SMTP id n14so8303665iom.10;
        Sat, 05 Dec 2020 01:13:25 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20161025;
        h=mime-version:references:in-reply-to:from:date:message-id:subject:to
         :cc;
        bh=pPN7UPtT6pJjG7XMJVzwK80KCiuqM/7+qfCK3uEZntA=;
        b=t22m7PqMtNTzJhPfZzuOXb2kEljECEe2iqQ2Olmzt/VDu71xwDJob5LZ6LXKxHclsp
         G62/obD8cFlivAI6FiCzMtPdHwf6wRuvjp3kWehGCgNGN8yPmdM5BUQC2Hj5w3k/Jw2Y
         ++GI6e/QIFIosVPazvz8vOrZooRvBH3tudyfeivY6ZUZVTqdhQGTYlqPZgZD0gPE3/6b
         FFAAJrIPj0V0IjfLSbrygZRJ/AbHtubPiDlQQoAJmM3PbxR7zt4XTGEE4XJqhOGjHDG0
         I4e3zvzpViYf7XVNXkH7Q0t2ehb9IrJpwSryo7JAZNOT2OAKdXqkymU/GKGbKDEi1hXG
         /xrg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:mime-version:references:in-reply-to:from:date
         :message-id:subject:to:cc;
        bh=pPN7UPtT6pJjG7XMJVzwK80KCiuqM/7+qfCK3uEZntA=;
        b=YCq4TahD3DV5h49ygJa/GfFDoRCkQkvfdTZqaNzHx1RoFe5JPpaYxt5tSsOon+Peat
         d5rCuzA0XZkHmA0l45OOcjeYBcLhXuB/2oD7vzV7dy93owP7ISvgDa9orDZ1kPc7i+Ev
         Cgx4nPR6t/nZ6vWfnUTqLJ85w6o3az29ES6vhFHcbJ6b9xfNJB7uHGktCfY57nZDr1ki
         /TqPe5se9zV8pnffSlIFyP0nyYnMxqhby4pOIxPtguTo8D6YsnlaSrx874P1tXAdvZ+4
         WRwGOFLKUtyeGEZsDlzpeI3U88qcJPnU5g96F72MQKsddy19sExQr9IS10zj00Lgc0JL
         97EA==
X-Gm-Message-State: AOAM532CXGbdet290QfkP8D2gECJAfK/6ONsptdWzux4PmJJqTizRQhE
        NiitSh0bKM6LMmX3diY21RrAS/k3FP5zOkILcfk=
X-Google-Smtp-Source: ABdhPJzO1kNL+9qzYppuh3BSCu5zYYcgYegA6H8Z2sZldrGEiCj5r4d3+7Gbeh3L//mTw8ZetXHtiXedVIZsASBRfFs=
X-Received: by 2002:a6b:fd03:: with SMTP id c3mr10156175ioi.64.1607159604914;
 Sat, 05 Dec 2020 01:13:24 -0800 (PST)
MIME-Version: 1.0
References: <20201127092058.15117-1-sargun@sargun.me> <20201127092058.15117-5-sargun@sargun.me>
 <20201130191509.GC14328@redhat.com>
In-Reply-To: <20201130191509.GC14328@redhat.com>
From:   Amir Goldstein <amir73il@gmail.com>
Date:   Sat, 5 Dec 2020 11:13:13 +0200
Message-ID: <CAOQ4uxjeG4N7i95D+YFr0zo82nLOjUCdUhD8e1WABFtwtQYzrQ@mail.gmail.com>
Subject: Re: [PATCH v2 4/4] overlay: Add rudimentary checking of writeback
 errseq on volatile remount
To:     Vivek Goyal <vgoyal@redhat.com>, Sargun Dhillon <sargun@sargun.me>
Cc:     overlayfs <linux-unionfs@vger.kernel.org>,
        Miklos Szeredi <miklos@szeredi.hu>,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel <linux-fsdevel@vger.kernel.org>,
        David Howells <dhowells@redhat.com>,
        Jeff Layton <jlayton@redhat.com>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org

On Mon, Nov 30, 2020 at 9:15 PM Vivek Goyal <vgoyal@redhat.com> wrote:
>
> On Fri, Nov 27, 2020 at 01:20:58AM -0800, Sargun Dhillon wrote:
> > Volatile remounts validate the following at the moment:
> >  * Has the module been reloaded / the system rebooted
> >  * Has the workdir been remounted
> >
> > This adds a new check for errors detected via the superblock's
> > errseq_t. At mount time, the errseq_t is snapshotted to disk,
> > and upon remount it's re-verified. This allows for kernel-level
> > detection of errors without forcing userspace to perform a
> > sync and allows for the hidden detection of writeback errors.
> >
> > Signed-off-by: Sargun Dhillon <sargun@sargun.me>
> > Cc: linux-fsdevel@vger.kernel.org
> > Cc: linux-unionfs@vger.kernel.org
> > Cc: Miklos Szeredi <miklos@szeredi.hu>
> > Cc: Amir Goldstein <amir73il@gmail.com>
> > Cc: Vivek Goyal <vgoyal@redhat.com>
> > ---
> >  fs/overlayfs/overlayfs.h | 1 +
> >  fs/overlayfs/readdir.c   | 6 ++++++
> >  fs/overlayfs/super.c     | 1 +
> >  3 files changed, 8 insertions(+)
> >
> > diff --git a/fs/overlayfs/overlayfs.h b/fs/overlayfs/overlayfs.h
> > index de694ee99d7c..e8a711953b64 100644
> > --- a/fs/overlayfs/overlayfs.h
> > +++ b/fs/overlayfs/overlayfs.h
> > @@ -85,6 +85,7 @@ struct ovl_volatile_info {
> >        */
> >       uuid_t          ovl_boot_id;    /* Must stay first member */
> >       u64             s_instance_id;
> > +     errseq_t        errseq; /* Implemented as a u32 */
> >  } __packed;
> >
> >  /*
> > diff --git a/fs/overlayfs/readdir.c b/fs/overlayfs/readdir.c
> > index 7b66fbb20261..5795b28bb4cf 100644
> > --- a/fs/overlayfs/readdir.c
> > +++ b/fs/overlayfs/readdir.c
> > @@ -1117,6 +1117,12 @@ static int ovl_verify_volatile_info(struct ovl_fs *ofs,
> >               return -EINVAL;
> >       }
> >
> > +     err = errseq_check(&volatiledir->d_sb->s_wb_err, info.errseq);
> > +     if (err) {
> > +             pr_debug("Workdir filesystem reports errors: %d\n", err);
> > +             return -EINVAL;
> > +     }
> > +
> >       return 1;
> >  }
> >
> > diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
> > index a8ee3ba4ebbd..2e473f8c75dd 100644
> > --- a/fs/overlayfs/super.c
> > +++ b/fs/overlayfs/super.c
> > @@ -1248,6 +1248,7 @@ static int ovl_set_volatile_info(struct ovl_fs *ofs, struct dentry *volatiledir)
> >       int err;
> >       struct ovl_volatile_info info = {
> >               .s_instance_id = volatiledir->d_sb->s_instance_id,
> > +             .errseq = errseq_sample(&volatiledir->d_sb->s_wb_err),
>
> errse_sample() seems to return 0 if nobody has seen the error yet. That
> means on remount we will fail. It is a false failure from our perspective
> and we are not interested in knowing if somebody else has seen the
> failure or not.
>
> Maybe we need a flag in errseq_sample() to get us current value
> irrespective of the fact whether anybody has seen the error or not?
>
> If we end up making this change, then we probably will have to somehow
> mask ERRSEQ_SEEN bit in errseq_check() comparison. Because if we
> sampled ->s_wb_err when nobody saw it and later by the remount time
> say ERRSEQ_SEEN is set, we don't want remount to fail.
>

Hopping back to this review, looks like for volatile mount we need
something like (in this order):
1. check if re-use and get sampled errseq from volatiledir xattr
2. otherwise errseq_sample() upper_sb and store in volatiledir xattr
3. errseq_check() since stored or sampled errseq (0 for fresh mount
with unseen error)
4. fail volatile mount if errseq_check() failed
5. errseq_check() since stored errseq on fsync()/syncfs()

For fresh volatile mount, syncfs can fix the temporary mount error.
For re-used volatile mount, the mount error is permanent.

Did I miss anything?
Is the mount safe for both seen and unseen error cases? no error case?
Are we safe if a syncfs on upper_sb sneaks in between 2 and 3?

Thanks,
Amir.

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,
	HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable
	autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id E72C9C4361B
	for <linux-unionfs@archiver.kernel.org>; Sat,  5 Dec 2020 15:35:15 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id B2E7923132
	for <linux-unionfs@archiver.kernel.org>; Sat,  5 Dec 2020 15:35:15 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1726472AbgLEO4d (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Sat, 5 Dec 2020 09:56:33 -0500
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36502 "EHLO
        lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1726591AbgLEOxI (ORCPT
        <rfc822;linux-unionfs@vger.kernel.org>);
        Sat, 5 Dec 2020 09:53:08 -0500
Received: from mail-io1-xd43.google.com (mail-io1-xd43.google.com [IPv6:2607:f8b0:4864:20::d43])
        by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8E90DC02B8F4;
        Sat,  5 Dec 2020 06:51:21 -0800 (PST)
Received: by mail-io1-xd43.google.com with SMTP id z136so8896597iof.3;
        Sat, 05 Dec 2020 06:51:21 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20161025;
        h=mime-version:references:in-reply-to:from:date:message-id:subject:to
         :cc;
        bh=j6NQ9ycamfVAHPSF5hrisg0pA0mvJ+mft6vLxWBd6ec=;
        b=Ykzaa6G36yvI6S6WhkxCfe+8kGWMQLZsEiXDoX0LVygALOi/mOhsFeIa3kNvg/svHQ
         LScV163iipg3Z13VnW/1qmOQNjOBR7+924M7sEWFXMW/Kky5g3tvI7HqNLKBOCciCKIQ
         iuXzhd/rp4Suu2YrvwZUhmspcjepbfkTn2Xbk1G9RyuH2wsBHcenw/A49Zi9YOORAgZv
         4etI9j9Iu0HScv7xKLqKQBGCHthq/AAGpH2BnQCs4R3dDj2VPIfZgdafrSkfLoUH91cS
         Nnf37HuVbULxQptpS6njOCdYJuCb52xcXAkbIWDQQs0LIXWEV98xk4C1V3VTlyTCvPUG
         mcOQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:mime-version:references:in-reply-to:from:date
         :message-id:subject:to:cc;
        bh=j6NQ9ycamfVAHPSF5hrisg0pA0mvJ+mft6vLxWBd6ec=;
        b=ULlJfExZofnxBeBrW7AQ4mZA1zQ7W9iBdkn5o69O78QHsghEFFmG6Xq9Yyv91W5wWo
         TgJt8cLdZwmE+urFSCi5/iTcriYsNojs75rQtH5fWwSUFK030SsQoJDK2WI+Tck3weFN
         DtpB9/p76uOD8nJv8odnbez31A5WZpQcPEHLQYjeHT0+ELv7LXhZTHu/fp8nUB8uD+4S
         SijWn7IiS4NzdzpAmji/Soow1P7u86Hn/MbMT9wg24a7xMhqzSEYaC2a4OSZ/VZGm+hD
         qcTU9OkDAeLXqrEbpS6Yt8NPeilkHPLz78WU85dZy7NhBWvbfOTDa95aUDuRmx0Ptyxz
         O9Zg==
X-Gm-Message-State: AOAM530yV63UDRlxlqqCLbLy+FhsFGimeH8djr0XynYEoCmH4DVPIAJi
        JaodE+1BueU9yBYQq2CqjpCc/kUhrNs2d/D/UQ8=
X-Google-Smtp-Source: ABdhPJw3PNkTAx+EM2gJMInq6sThR2pJBiyBwrpBkA6Zw8bmnZrE4m8ErIHgc4hncmUkZr1FVPbO0dwyU5B/SkGDSO0=
X-Received: by 2002:a05:6602:1608:: with SMTP id x8mr11110479iow.72.1607179880992;
 Sat, 05 Dec 2020 06:51:20 -0800 (PST)
MIME-Version: 1.0
References: <20201127092058.15117-1-sargun@sargun.me> <20201127092058.15117-5-sargun@sargun.me>
 <20201130191509.GC14328@redhat.com> <CAOQ4uxjeG4N7i95D+YFr0zo82nLOjUCdUhD8e1WABFtwtQYzrQ@mail.gmail.com>
 <5468a450ac7e9626af1a61a29ef17a6855d6692f.camel@redhat.com>
In-Reply-To: <5468a450ac7e9626af1a61a29ef17a6855d6692f.camel@redhat.com>
From:   Amir Goldstein <amir73il@gmail.com>
Date:   Sat, 5 Dec 2020 16:51:09 +0200
Message-ID: <CAOQ4uxgHKqLjMV2nD5vOfAHbef+geso25DfqG3hu-PWyc25b9Q@mail.gmail.com>
Subject: Re: [PATCH v2 4/4] overlay: Add rudimentary checking of writeback
 errseq on volatile remount
To:     Jeff Layton <jlayton@redhat.com>
Cc:     Vivek Goyal <vgoyal@redhat.com>, Sargun Dhillon <sargun@sargun.me>,
        overlayfs <linux-unionfs@vger.kernel.org>,
        Miklos Szeredi <miklos@szeredi.hu>,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel <linux-fsdevel@vger.kernel.org>,
        David Howells <dhowells@redhat.com>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org

On Sat, Dec 5, 2020 at 3:51 PM Jeff Layton <jlayton@redhat.com> wrote:
>
> On Sat, 2020-12-05 at 11:13 +0200, Amir Goldstein wrote:
> > On Mon, Nov 30, 2020 at 9:15 PM Vivek Goyal <vgoyal@redhat.com> wrote:
> > >
> > > On Fri, Nov 27, 2020 at 01:20:58AM -0800, Sargun Dhillon wrote:
> > > > Volatile remounts validate the following at the moment:
> > > >  * Has the module been reloaded / the system rebooted
> > > >  * Has the workdir been remounted
> > > >
> > > > This adds a new check for errors detected via the superblock's
> > > > errseq_t. At mount time, the errseq_t is snapshotted to disk,
> > > > and upon remount it's re-verified. This allows for kernel-level
> > > > detection of errors without forcing userspace to perform a
> > > > sync and allows for the hidden detection of writeback errors.
> > > >
> > > > Signed-off-by: Sargun Dhillon <sargun@sargun.me>
> > > > Cc: linux-fsdevel@vger.kernel.org
> > > > Cc: linux-unionfs@vger.kernel.org
> > > > Cc: Miklos Szeredi <miklos@szeredi.hu>
> > > > Cc: Amir Goldstein <amir73il@gmail.com>
> > > > Cc: Vivek Goyal <vgoyal@redhat.com>
> > > > ---
> > > >  fs/overlayfs/overlayfs.h | 1 +
> > > >  fs/overlayfs/readdir.c   | 6 ++++++
> > > >  fs/overlayfs/super.c     | 1 +
> > > >  3 files changed, 8 insertions(+)
> > > >
> > > > diff --git a/fs/overlayfs/overlayfs.h b/fs/overlayfs/overlayfs.h
> > > > index de694ee99d7c..e8a711953b64 100644
> > > > --- a/fs/overlayfs/overlayfs.h
> > > > +++ b/fs/overlayfs/overlayfs.h
> > > > @@ -85,6 +85,7 @@ struct ovl_volatile_info {
> > > >        */
> > > >       uuid_t          ovl_boot_id;    /* Must stay first member */
> > > >       u64             s_instance_id;
> > > > +     errseq_t        errseq; /* Implemented as a u32 */
> > > >  } __packed;
> > > >
> > > >  /*
> > > > diff --git a/fs/overlayfs/readdir.c b/fs/overlayfs/readdir.c
> > > > index 7b66fbb20261..5795b28bb4cf 100644
> > > > --- a/fs/overlayfs/readdir.c
> > > > +++ b/fs/overlayfs/readdir.c
> > > > @@ -1117,6 +1117,12 @@ static int ovl_verify_volatile_info(struct ovl_fs *ofs,
> > > >               return -EINVAL;
> > > >       }
> > > >
> > > > +     err = errseq_check(&volatiledir->d_sb->s_wb_err, info.errseq);
> > > > +     if (err) {
> > > > +             pr_debug("Workdir filesystem reports errors: %d\n", err);
> > > > +             return -EINVAL;
> > > > +     }
> > > > +
> > > >       return 1;
> > > >  }
> > > >
> > > > diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
> > > > index a8ee3ba4ebbd..2e473f8c75dd 100644
> > > > --- a/fs/overlayfs/super.c
> > > > +++ b/fs/overlayfs/super.c
> > > > @@ -1248,6 +1248,7 @@ static int ovl_set_volatile_info(struct ovl_fs *ofs, struct dentry *volatiledir)
> > > >       int err;
> > > >       struct ovl_volatile_info info = {
> > > >               .s_instance_id = volatiledir->d_sb->s_instance_id,
> > > > +             .errseq = errseq_sample(&volatiledir->d_sb->s_wb_err),
> > >
> > > errse_sample() seems to return 0 if nobody has seen the error yet. That
> > > means on remount we will fail. It is a false failure from our perspective
> > > and we are not interested in knowing if somebody else has seen the
> > > failure or not.
> > >
> > > Maybe we need a flag in errseq_sample() to get us current value
> > > irrespective of the fact whether anybody has seen the error or not?
> > >
> > > If we end up making this change, then we probably will have to somehow
> > > mask ERRSEQ_SEEN bit in errseq_check() comparison. Because if we
> > > sampled ->s_wb_err when nobody saw it and later by the remount time
> > > say ERRSEQ_SEEN is set, we don't want remount to fail.
> > >
> >
> > Hopping back to this review, looks like for volatile mount we need
> > something like (in this order):
> > 1. check if re-use and get sampled errseq from volatiledir xattr
> > 2. otherwise errseq_sample() upper_sb and store in volatiledir xattr
>
> I'm not sure I follow. Why does this need to go into an xattr?
>
> errseq_t is never persisted on stable storage. It's an entirely
> in-memory thing.
>

We know, but that was the purpose of this patch series [1].
Reusing volatile overlay layers is not allowed in v5.9.
Sargun is trying to allow that by verifying that since the first volatile
mount there was:
* no reboot
* no overlay module reload
* no underlying fs re-mount
* [and with this patch] no writeback error on upper fs

[1] https://lore.kernel.org/linux-unionfs/20201127092058.15117-1-sargun@sargun.me/T/#mb2f1c770a47898d8781e62a46fcc7526535e5dde

>
> > 3. errseq_check() since stored or sampled errseq (0 for fresh mount
> > with unseen error)
> > 4. fail volatile mount if errseq_check() failed
> > 5. errseq_check() since stored errseq on fsync()/syncfs()
> >
>
> I think this is simpler than that. You just need a new errseq_t helper
> that only conditionally samples if the thing is 0 or if the error has
> already been seen. Something like this (hopefully with a better name):
>
> bool errseq_sample_no_unseen(errseq_t *eseq, errseq_t *sample)
> {
>         errseq_t old = READ_ONCE(*eseq);
>
>         if (old && !(old & ERRSEQ_SEEN))
>                 return false;
>         *sample = old;
>         return true;
> }
>
> If that returns false, fail the mount. If it's true, then save off the
> sample and proceed.
>

Yes, but that wasn't the purpose of the original patch set,
it was just something else that needed fixing that we found during
review of this patch for which Sargun posted a separate patch.

Thank,
Amir.

From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,
	URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id 94DDAC433FE
	for <linux-unionfs@archiver.kernel.org>; Sat,  5 Dec 2020 18:22:31 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id 66E522332A
	for <linux-unionfs@archiver.kernel.org>; Sat,  5 Dec 2020 18:22:31 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1725274AbgLESVu (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Sat, 5 Dec 2020 13:21:50 -0500
Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:48170 "EHLO
        us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK)
        by vger.kernel.org with ESMTP id S1725270AbgLESVd (ORCPT
        <rfc822;linux-unionfs@vger.kernel.org>);
        Sat, 5 Dec 2020 13:21:33 -0500
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;
        s=mimecast20190719; t=1607192405;
        h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
         to:to:cc:cc:mime-version:mime-version:content-type:content-type:
         content-transfer-encoding:content-transfer-encoding:
         in-reply-to:in-reply-to:references:references;
        bh=eJ3TlJLyB/D1p6iBoOSpxo7AX6033ynhX3Zy+9CpELA=;
        b=YVOSnSWGW0QRVfWEZg63Rglk6zYGeU23hWFIWvOlKju/EXWa26PZVk67LjZry7Sdj9+V3B
        uDh5Us2Q370jb5G5QwSS0DdlGOkNNfffhp5AOGCE0xZkSfA4BGtmVRP8Q4QqT27ATqtzOA
        BRDK9ByTB4flEJJc2/VVVQVaN2QbhfY=
Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com
 [209.85.219.70]) (Using TLS) by relay.mimecast.com with ESMTP id
 us-mta-432-P9eh51z_PuG04f4cEBFzFA-1; Sat, 05 Dec 2020 08:51:20 -0500
X-MC-Unique: P9eh51z_PuG04f4cEBFzFA-1
Received: by mail-qv1-f70.google.com with SMTP id f2so7463468qvb.7
        for <linux-unionfs@vger.kernel.org>; Sat, 05 Dec 2020 05:51:20 -0800 (PST)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to
         :references:user-agent:mime-version:content-transfer-encoding;
        bh=eJ3TlJLyB/D1p6iBoOSpxo7AX6033ynhX3Zy+9CpELA=;
        b=udUeuyoflyWqOFwNEYrcR3US3rZK+OFBtGIVR4nY84B/rzd0Ysz/SNYC3i5KKGKU1v
         thmVF3GJnyh4Lwuy7UE52jCqZfQkzdoaUCWK0uv/Awm67wODRIPeOInyS4J3RKQFfosk
         tJY0tOXE/YB9yvYxe9duUEujpkQFQ6bFzNDKP6JTLHqe7CsVhhJV2OUzD0X2G2qhAULU
         Twaa4mSCzf2xCDN6ON1Yh7j0ex36Ob7pEHM04U5PWLtaF3immezqJHy98L2oFyCXdAPo
         Jg1UXUtHO3iaBUtr/m88TAYjThx/57hlHiqYbTlkAmSC6aJLqXIaftuy78+umJ/XL3UT
         b+yg==
X-Gm-Message-State: AOAM532tXAHpCHl5KzBhN5GDR8kIh0l94QfU0uLK6FH6Re9OQX1L/kWm
        7D6APsZiBjv+0+XDOqL4LdJAugmMegNecgnCcgCdxakKRs/bwOoxmNzwN/AGfl7MAedWkbw7yiA
        BeN2w5ExkV1dA2ORj2ffmEgfH6g==
X-Received: by 2002:ac8:1486:: with SMTP id l6mr14654702qtj.123.1607176280494;
        Sat, 05 Dec 2020 05:51:20 -0800 (PST)
X-Google-Smtp-Source: ABdhPJzf0TX7HzWr+7ZlUNgaWDbTFRqjLfGWcFqjWR0uZ/edGk7MUPRmkFxWz347muqImSmgQQ8VbA==
X-Received: by 2002:ac8:1486:: with SMTP id l6mr14654671qtj.123.1607176280168;
        Sat, 05 Dec 2020 05:51:20 -0800 (PST)
Received: from tleilax.poochiereds.net (68-20-15-154.lightspeed.rlghnc.sbcglobal.net. [68.20.15.154])
        by smtp.gmail.com with ESMTPSA id d11sm8800449qta.64.2020.12.05.05.51.19
        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
        Sat, 05 Dec 2020 05:51:19 -0800 (PST)
Message-ID: <5468a450ac7e9626af1a61a29ef17a6855d6692f.camel@redhat.com>
Subject: Re: [PATCH v2 4/4] overlay: Add rudimentary checking of writeback
 errseq on volatile remount
From:   Jeff Layton <jlayton@redhat.com>
To:     Amir Goldstein <amir73il@gmail.com>,
        Vivek Goyal <vgoyal@redhat.com>,
        Sargun Dhillon <sargun@sargun.me>
Cc:     overlayfs <linux-unionfs@vger.kernel.org>,
        Miklos Szeredi <miklos@szeredi.hu>,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel <linux-fsdevel@vger.kernel.org>,
        David Howells <dhowells@redhat.com>
Date:   Sat, 05 Dec 2020 08:51:18 -0500
In-Reply-To: <CAOQ4uxjeG4N7i95D+YFr0zo82nLOjUCdUhD8e1WABFtwtQYzrQ@mail.gmail.com>
References: <20201127092058.15117-1-sargun@sargun.me>
         <20201127092058.15117-5-sargun@sargun.me>
         <20201130191509.GC14328@redhat.com>
         <CAOQ4uxjeG4N7i95D+YFr0zo82nLOjUCdUhD8e1WABFtwtQYzrQ@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
User-Agent: Evolution 3.38.1 (3.38.1-1.fc33) 
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org

On Sat, 2020-12-05 at 11:13 +0200, Amir Goldstein wrote:
> On Mon, Nov 30, 2020 at 9:15 PM Vivek Goyal <vgoyal@redhat.com> wrote:
> > 
> > On Fri, Nov 27, 2020 at 01:20:58AM -0800, Sargun Dhillon wrote:
> > > Volatile remounts validate the following at the moment:
> > > Â * Has the module been reloaded / the system rebooted
> > > Â * Has the workdir been remounted
> > > 
> > > This adds a new check for errors detected via the superblock's
> > > errseq_t. At mount time, the errseq_t is snapshotted to disk,
> > > and upon remount it's re-verified. This allows for kernel-level
> > > detection of errors without forcing userspace to perform a
> > > sync and allows for the hidden detection of writeback errors.
> > > 
> > > Signed-off-by: Sargun Dhillon <sargun@sargun.me>
> > > Cc: linux-fsdevel@vger.kernel.org
> > > Cc: linux-unionfs@vger.kernel.org
> > > Cc: Miklos Szeredi <miklos@szeredi.hu>
> > > Cc: Amir Goldstein <amir73il@gmail.com>
> > > Cc: Vivek Goyal <vgoyal@redhat.com>
> > > ---
> > > Â fs/overlayfs/overlayfs.h | 1 +
> > > Â fs/overlayfs/readdir.c   | 6 ++++++
> > > Â fs/overlayfs/super.c     | 1 +
> > > Â 3 files changed, 8 insertions(+)
> > > 
> > > diff --git a/fs/overlayfs/overlayfs.h b/fs/overlayfs/overlayfs.h
> > > index de694ee99d7c..e8a711953b64 100644
> > > --- a/fs/overlayfs/overlayfs.h
> > > +++ b/fs/overlayfs/overlayfs.h
> > > @@ -85,6 +85,7 @@ struct ovl_volatile_info {
> > > Â Â Â Â Â Â Â */
> > > Â Â Â Â Â Â uuid_t          ovl_boot_id;    /* Must stay first member */
> > > Â Â Â Â Â Â u64             s_instance_id;
> > > +     errseq_t        errseq; /* Implemented as a u32 */
> > > Â } __packed;
> > > 
> > > Â /*
> > > diff --git a/fs/overlayfs/readdir.c b/fs/overlayfs/readdir.c
> > > index 7b66fbb20261..5795b28bb4cf 100644
> > > --- a/fs/overlayfs/readdir.c
> > > +++ b/fs/overlayfs/readdir.c
> > > @@ -1117,6 +1117,12 @@ static int ovl_verify_volatile_info(struct ovl_fs *ofs,
> > > Â Â Â Â Â Â Â Â Â Â Â Â Â Â return -EINVAL;
> > > Â Â Â Â Â Â }
> > > 
> > > +     err = errseq_check(&volatiledir->d_sb->s_wb_err, info.errseq);
> > > +     if (err) {
> > > +             pr_debug("Workdir filesystem reports errors: %d\n", err);
> > > +             return -EINVAL;
> > > +     }
> > > +
> > > Â Â Â Â Â Â return 1;
> > > Â }
> > > 
> > > diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
> > > index a8ee3ba4ebbd..2e473f8c75dd 100644
> > > --- a/fs/overlayfs/super.c
> > > +++ b/fs/overlayfs/super.c
> > > @@ -1248,6 +1248,7 @@ static int ovl_set_volatile_info(struct ovl_fs *ofs, struct dentry *volatiledir)
> > > Â Â Â Â Â Â int err;
> > > Â Â Â Â Â Â struct ovl_volatile_info info = {
> > > Â Â Â Â Â Â Â Â Â Â Â Â Â Â .s_instance_id = volatiledir->d_sb->s_instance_id,
> > > +             .errseq = errseq_sample(&volatiledir->d_sb->s_wb_err),
> > 
> > errse_sample() seems to return 0 if nobody has seen the error yet. That
> > means on remount we will fail. It is a false failure from our perspective
> > and we are not interested in knowing if somebody else has seen the
> > failure or not.
> > 
> > Maybe we need a flag in errseq_sample() to get us current value
> > irrespective of the fact whether anybody has seen the error or not?
> > 
> > If we end up making this change, then we probably will have to somehow
> > mask ERRSEQ_SEEN bit in errseq_check() comparison. Because if we
> > sampled ->s_wb_err when nobody saw it and later by the remount time
> > say ERRSEQ_SEEN is set, we don't want remount to fail.
> > 
> 
> Hopping back to this review, looks like for volatile mount we need
> something like (in this order):
> 1. check if re-use and get sampled errseq from volatiledir xattr
> 2. otherwise errseq_sample() upper_sb and store in volatiledir xattr

I'm not sure I follow. Why does this need to go into an xattr?

errseq_t is never persisted on stable storage. It's an entirely
in-memory thing.


> 3. errseq_check() since stored or sampled errseq (0 for fresh mount
> with unseen error)
> 4. fail volatile mount if errseq_check() failed
> 5. errseq_check() since stored errseq on fsync()/syncfs()
> 

I think this is simpler than that. You just need a new errseq_t helper
that only conditionally samples if the thing is 0 or if the error has
already been seen. Something like this (hopefully with a better name):

bool errseq_sample_no_unseen(errseq_t *eseq, errseq_t *sample)         
{                                                                      
        errseq_t old = READ_ONCE(*eseq);                               
                                                                       
        if (old && !(old & ERRSEQ_SEEN))                               
                return false;                                          
        *sample = old;                                                 
        return true;                                                   
}                                                                      
       
If that returns false, fail the mount. If it's true, then save off the
sample and proceed.

> For fresh volatile mount, syncfs can fix the temporary mount error.
> For re-used volatile mount, the mount error is permanent.
> 
> Did I miss anything?
> Is the mount safe for both seen and unseen error cases? no error case?
> Are we safe if a syncfs on upper_sb sneaks in between 2 and 3?
> 
> Thanks,
> Amir.
> 

-- 
Jeff Layton <jlayton@redhat.com>


From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-unionfs-owner@kernel.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
	aws-us-west-2-korg-lkml-1.web.codeaurora.org
X-Spam-Level: 
X-Spam-Status: No, score=-10.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FAKE_REPLY_C,
	HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,
	SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY,URIBL_BLOCKED,USER_AGENT_SANE_1
	autolearn=ham autolearn_force=no version=3.4.0
Received: from mail.kernel.org (mail.kernel.org [198.145.29.99])
	by smtp.lore.kernel.org (Postfix) with ESMTP id CA2C5C4361B
	for <linux-unionfs@archiver.kernel.org>; Mon,  7 Dec 2020 11:41:08 +0000 (UTC)
Received: from vger.kernel.org (vger.kernel.org [23.128.96.18])
	by mail.kernel.org (Postfix) with ESMTP id 8239323356
	for <linux-unionfs@archiver.kernel.org>; Mon,  7 Dec 2020 11:41:08 +0000 (UTC)
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1726891AbgLGLlH (ORCPT
        <rfc822;linux-unionfs@archiver.kernel.org>);
        Mon, 7 Dec 2020 06:41:07 -0500
Received: from userp2130.oracle.com ([156.151.31.86]:42142 "EHLO
        userp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1726855AbgLGLlG (ORCPT
        <rfc822;linux-unionfs@vger.kernel.org>);
        Mon, 7 Dec 2020 06:41:06 -0500
Received: from pps.filterd (userp2130.oracle.com [127.0.0.1])
        by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 0B7BXv2W113193;
        Mon, 7 Dec 2020 11:40:12 GMT
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=date : from : to : cc
 : subject : message-id : mime-version : content-type : in-reply-to;
 s=corp-2020-01-29; bh=TzrMLNTmBiDKT8ADel4wOnUX0sW9EJgdgSdoP6KVlUU=;
 b=CXxP0npI+sCXb6HyMf9jNZzAacEZ8ycyGUmP68cZ9GM836LDZKOHzExX/qg+4jLj981/
 Av6r3HTrZe8MUQr7q0zuR7baOufuikprVFqKinqg4Ee+Me6jO2Uiip1hx2ZrSoCNyJP6
 hRxnBwoUsaZ2fh4FNdZgUuWUbU+bdfNpWtOEplonVX+fyO1o/sKbdk5Nng5DEDuy/WFK
 Yz1u3OVwJwwMCFcX40fjmL0zSGB4nhH6+m0iIV5fiVBmCFZEYO3bUN6zGPgzpGzYFpT2
 /P+/4msrf787sdQJ/ZGue+712dYGhfePHlXRon886nTvCIZPDx9wfgUbvfqNyLRov5YW Ag== 
Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71])
        by userp2130.oracle.com with ESMTP id 3581mqmv24-1
        (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL);
        Mon, 07 Dec 2020 11:40:12 +0000
Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1])
        by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 0B7BYiH0130203;
        Mon, 7 Dec 2020 11:40:11 GMT
Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75])
        by aserp3030.oracle.com with ESMTP id 358ksm2t9p-1
        (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);
        Mon, 07 Dec 2020 11:40:11 +0000
Received: from abhmp0007.oracle.com (abhmp0007.oracle.com [141.146.116.13])
        by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id 0B7Be7Vs027094;
        Mon, 7 Dec 2020 11:40:08 GMT
Received: from kadam (/102.36.221.92)
        by default (Oracle Beehive Gateway v4.0)
        with ESMTP ; Mon, 07 Dec 2020 03:40:04 -0800
Date:   Mon, 7 Dec 2020 14:39:54 +0300
From:   Dan Carpenter <dan.carpenter@oracle.com>
To:     kbuild@lists.01.org, Sargun Dhillon <sargun@sargun.me>,
        linux-unionfs@vger.kernel.org, miklos@szeredi.hu,
        Alexander Viro <viro@zeniv.linux.org.uk>,
        Amir Goldstein <amir73il@gmail.com>
Cc:     lkp@intel.com, kbuild-all@lists.01.org,
        Sargun Dhillon <sargun@sargun.me>,
        Giuseppe Scrivano <gscrivan@redhat.com>,
        Vivek Goyal <vgoyal@redhat.com>,
        Daniel J Walsh <dwalsh@redhat.com>,
        linux-fsdevel@vger.kernel.org, David Howells <dhowells@redhat.com>
Subject: Re: [PATCH v2 3/4] overlay: Add the ability to remount volatile
 directories when safe
Message-ID: <20201207113954.GW2767@kadam>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="UlsYxwg8UDQn+EKZ"
Content-Disposition: inline
In-Reply-To: <20201127092058.15117-4-sargun@sargun.me>
User-Agent: Mutt/1.9.4 (2018-02-28)
X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9827 signatures=668682
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 suspectscore=0
 bulkscore=0 malwarescore=0 phishscore=0 mlxscore=0 spamscore=0
 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2009150000 definitions=main-2012070075
X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9827 signatures=668682
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999
 clxscore=1011 malwarescore=0 priorityscore=1501 adultscore=0
 lowpriorityscore=0 phishscore=0 spamscore=0 impostorscore=0 mlxscore=0
 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2009150000 definitions=main-2012070075
Precedence: bulk
List-ID: <linux-unionfs.vger.kernel.org>
X-Mailing-List: linux-unionfs@vger.kernel.org


--UlsYxwg8UDQn+EKZ
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

Hi Sargun,

url:    https://github.com/0day-ci/linux/commits/Sargun-Dhillon/Make-overlayfs-volatile-mounts-reusable/20201127-172416
base:    be4df0cea08a8b59eb38d73de988b7ba8022df41
config: x86_64-randconfig-m001-20201127 (attached as .config)
compiler: gcc-9 (Debian 9.3.0-15) 9.3.0

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
Reported-by: Dan Carpenter <dan.carpenter@oracle.com>

smatch warnings:
fs/overlayfs/readdir.c:1149 ovl_check_incompat() error: uninitialized symbol 'd'.

vim +/d +1149 fs/overlayfs/readdir.c

a9871ab1de6c660 Sargun Dhillon 2020-11-27  1129  static int ovl_check_incompat(struct ovl_fs *ofs, struct ovl_cache_entry *p,
a9871ab1de6c660 Sargun Dhillon 2020-11-27  1130  			      struct path *path)
a9871ab1de6c660 Sargun Dhillon 2020-11-27  1131  {
a9871ab1de6c660 Sargun Dhillon 2020-11-27  1132  	int err = -EINVAL;
a9871ab1de6c660 Sargun Dhillon 2020-11-27  1133  	struct dentry *d;
a9871ab1de6c660 Sargun Dhillon 2020-11-27  1134  
a9871ab1de6c660 Sargun Dhillon 2020-11-27  1135  	if (!strcmp(p->name, OVL_VOLATILEDIR_NAME)) {
a9871ab1de6c660 Sargun Dhillon 2020-11-27  1136  		d = lookup_one_len(p->name, path->dentry, p->len);
a9871ab1de6c660 Sargun Dhillon 2020-11-27  1137  		if (IS_ERR(d))
a9871ab1de6c660 Sargun Dhillon 2020-11-27  1138  			return PTR_ERR(d);
a9871ab1de6c660 Sargun Dhillon 2020-11-27  1139  
a9871ab1de6c660 Sargun Dhillon 2020-11-27  1140  		err = ovl_verify_volatile_info(ofs, d);
a9871ab1de6c660 Sargun Dhillon 2020-11-27  1141  		dput(d);
                                                                ^^^^^^^
dput() d here.

a9871ab1de6c660 Sargun Dhillon 2020-11-27  1142  	}
a9871ab1de6c660 Sargun Dhillon 2020-11-27  1143  
a9871ab1de6c660 Sargun Dhillon 2020-11-27  1144  	if (err == -EINVAL)
a9871ab1de6c660 Sargun Dhillon 2020-11-27  1145  		pr_err("incompat feature '%s' cannot be mounted\n", p->name);
a9871ab1de6c660 Sargun Dhillon 2020-11-27  1146  	else
a9871ab1de6c660 Sargun Dhillon 2020-11-27  1147  		pr_debug("incompat '%s' handled: %d\n", p->name, err);
a9871ab1de6c660 Sargun Dhillon 2020-11-27  1148  
a9871ab1de6c660 Sargun Dhillon 2020-11-27 @1149  	dput(d);
                                                        ^^^^^^
"d" is uninitialized and this is probably a double free.

a9871ab1de6c660 Sargun Dhillon 2020-11-27  1150  	return err;
a9871ab1de6c660 Sargun Dhillon 2020-11-27  1151  }

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org

--UlsYxwg8UDQn+EKZ
Content-Type: application/gzip
Content-Disposition: attachment; filename=".config.gz"
Content-Transfer-Encoding: base64

H4sICPsSwV8AAy5jb25maWcAjFzLd9w2r9/3r5iTbtpFUttJfNNzjxcciZKY0SskNePxhsd1
Jvl86ti9fnxN/vsLkHqQFDRtF6kFQHyCwA8gND//9POKvTw/fLt+vr25vrv7sfp6uD88Xj8f
Pq++3N4d/neVNqu60SueCv0GhMvb+5fvv33/cG7O363ev/n9zcnrx5u3q83h8f5wt0oe7r/c
fn2B928f7n/6+aekqTORmyQxWy6VaGqj+aW+ePX15ub176tf0sMft9f3q9/fvIVmTt//6v56
5b0mlMmT5OLHQMqnpi5+P3l7cjIwynSkn719f2L/G9spWZ2P7BOv+YTVphT1ZurAIxqlmRZJ
wCuYMkxVJm90QzJEDa9yj9XUSssu0Y1UE1XIT2bXSK/fdSfKVIuKG83WJTeqkXri6kJylkLj
WQP/gIjCV2GBf17ldr/uVk+H55e/piVfy2bDawMrrqrW67gW2vB6a5iENROV0Bdvz6CVcbRV
K6B3zZVe3T6t7h+eseFxkZuElcM6vnpFkQ3r/JWx0zKKldqTL9iWmw2XNS9NfiW84fmcNXDO
aFZ5VTGac3m19EazxHhHM66UToEzLo03Xn9lYr4dNbF04cjjty6vjrUJgz/OfneMjRMhBpTy
jHWlthrh7c1ALhqla1bxi1e/3D/cH359NbWr9mor2oRos22UuDTVp4533hHwqfhyosuJuWM6
KUz0RiIbpUzFq0buDdOaJYW/aJ3ipViTc2Yd2CliZHZnmYSurASOgpXlcIbgOK6eXv54+vH0
fPg2naGc11yKxJ7WVjZrb4Q+SxXNjuaI+iNPNB4WT8VkCixl1M5Irnid0q8mhX8ukJI2FRN1
SFOiooRMIbjE2e7njVdKoOQiY9aPP6qKaQl7CUsHRx5MGi2F85JbhhM3VZPycIhZIxOe9iZN
1PnEVS2TitOjsyPj6y7PlFWFw/3n1cOXaOcmg98kG9V00JFTsLTxurFq4IvYI/CDennLSpEy
zU3JlDbJPikJHbBWezupVMS27fEtr7U6ykSTzdIEOjouVsH+svRjR8pVjTJdi0OOrJo7hEnb
2eFKZX3I4IPsIdC33w6PT9Q5ACe4AU/CQdG9PuvGFFfoMSqr3+MRBGILg2lSQZkI95ZIy9BH
IiwwWrJkE2hEzHHK4/dm26MOvMgLVMR+ur7OzCY6GirJedVqaLMO+hjo26bsas3knjQ9vRRl
Fvv3kwZeH5YbtuI3ff305+oZhrO6hqE9PV8/P62ub24eXu6fb++/ThuwFVLbvWOJbcOt0diz
3Z+QTYyCaAT1xm8ID5nVZrqhUW6tUjSICQcrDaKaFEIFQwSl6NVSIqT3m/MvlsUun0y6laJU
td4b4E0qBA+GX4JGeqqrAgn7TkTCsdtX+5NFsGakLuUUHXX3OMNYcFetfSUN5xfCqbWoz7wR
iY37Y06xm+STC+gIT8O3Cbphoxn4MJHpi7OTSWVFrQEBs4xHMqdvA7vSAbx1gDUpwKpbQzWo
uLr5z+Hzy93hcfXlcP388nh4suR+hgQ3sNCqa1sAwcrUXcXMmgGITwLjYKV2rNbA1Lb3rq5Y
a3S5NlnZqWIG0GFOp2cfohbGfmJuksuma5V/QACSJNTZcqJuDaYGMiakITlJBuae1elOpNob
JhxQWtxRW5GqGVGmIaLsyRnYnCsul8ea8q1I+Kw5OH94oud9c5nNiNYhe8eqQTvUs5gOhoVY
Evw7mAxqSAVPNm0DG4AmG3CFNy6nVxhV2Ib9NsHlwjqmHOwroBGekoZG8pLtiT7X5QbXwDp/
6a21fWYVNOwwgIeNZRqFK0CIohSghMEJEPyYxPKb6Pld8BwHHuumQdeBf1NLl5imBZstrjg6
R7tPjazgrAQuLBZT8AfRWozO3TPYz4S3Fs5ZmxVDi0S1G+i5ZBq79oK/1lOZ2AZXEGUIQPIy
2NGc6wqBSY+m6CHi3sRoKyvgOJWh27aQx/l/0iujifPCFGfy6kr44aun3vN5TnvEALdmHTng
rAMI4w0UH+EoeyvTNv5MlMhrVmaeStopZIFSWCyYUXGdKsBIeSZONIGHb0wnl9w6S7dC8WFt
qTWbwijcORspZqnZBacSul8zKQVpfDbY8L7yzNhAMcF2jlS7sniGtdgGuwvKdURLJt8wBLQo
/9HH+T1hFPCjNG92kbdANzLNEEZQA+p2BmsyOYp/IsYEb/E09e26Oz7QlRnjg0l7k9OTIK63
vrPPs7WHxy8Pj9+u728OK/7fwz0AJAZeNUGIBNh2wkMLjVvD7ZgwVbOtbERHArJ/2eOIMivX
nQO7A/Ie4XzVMtgKuaEUt2TrwBaUHR3jq7JZYrA1bI7M+bCpC91Y71gKCOkkWI2mCrv1+Riu
AwKkD1qXZQB8Wgb9ERExKKfmlXWEmHsUmUhYmAsAdJaJMsA01r5aTxiELGGSbxA+f7f2FfrS
JmWDZ9+tuTQkGvGUJxCVe0NtOt122ljHoS9eHe6+nL97/f3D+evzd36SbwOudoBL3jw1xGYO
y854VdVF56dChCZrBLEulr04+3BMgF1igpIUGLRpaGihnUAMmjs9j6PmwMp7xNHuGLsjXFIB
PCvFWmKKIEWgQVgLDLmwoUuKxwDdYLKZW49MSICCQMemzUFZdGQ5FNcOmbmwDqIILzrngJoG
lrU80JTEJEbR+fnuQM6qMinmxiPWXNYurwPOVYl1GQ9ZdarlsOgLbGuR7dKx0hQdePtyPYlc
QdxtAM6+9QCVTdfZl5dge2/LYOj2EC6JdTaD5+1gBuCAM1nuE0xT+T4z3QNahb1ti72CQ1ua
yuXbh2Obu4inBENXqov3UZChGO4mHgrcMp44o2Ctd/v4cHN4enp4XD3/+MtFt0FkFK0DbeSq
ljBGePYzznQnuYPavklD5uUZa0VCtojsqrW5N6LlvCnTTPjBlOQaEIsIcyTYiNN0QIyScsko
wS81aAdqHAGiUGDobXGgeA5LU7aKTiugCKum9vswh5QVjcog6hYLYx01p89DQzRXdqGjd7FJ
U4GGZhAzjHaCQk57OGQArwB75x33c3ew8AxzM4Hz72lOuRcGWGzR+pRr0DRwML2eTSvBawp/
gX+O+nfp0LbDnBwocKl7ODoNZktvxzjIIymjWHTID4yNfIRVLRoEIXZYNC5NZH2EXW0+0PRW
0QpfIWA7o1ngwyvqFAzmvvUc2qCPsgZ32ttylyQ590XK02WeVknYXlK1l0mRR34cE7vbkAIe
T1RdZQ9dBuaq3F+cv/MFrOpA9FYpz9MLMK7WUJggzkP5bXU5MyETUMFcH8aNvORBdgB6h4Pi
juWcDEdxTiz2eZgxHhgJ4EbWkTmLXuKqYM2lfzlRtNzpn4xoHOJL9MpSewuc2thu7DdnoJGi
AZBCJaqtS1SIEMEprnkO3Z7STLzEmbF6BDpjTASYjx1ieP1gNQYvVA0a7EjZmoEYGDzJJSA4
F9j39742aYD3TAu2o/Kj+J6Aeb6S5yzZz1jx/g7kYH8HIt7sqAIMOdXMx0B97BEoOMDOcjJh
zlV68ca3h/vb54fHICXuBTa9T+jqOA6by0jWlrQtm4kmmLumHYcvbJ1Nswtt/ojbF2YRHLs+
iO01VoRHw+16W+I/sEzEbooPgT0F3AKHFezR0sb71qB32iLaqfcW2YS0VEjYH5OvEWjN0EXS
MlctobRIKJ3DZQI/Ckclkfs28HURC6y9hdzr/ZEgzkE7C1ncq4yAoiN7OIoR35qywbvjPaU3
ZVHiSSgHh44XgR2/OPn++XD9+cT7L1hJzGJCnNEozAjIrqU2Ew8mesNq6HgSdQ0s7Ju7U8Vk
/M6z85WWgdPHZwSgQourWCW9xiAeoh0trosLiRffVRBXLYyxq/zk6ITW+pn2cBhnuuF7RUlq
dWl3wjRZNjsGkUT9DxhxlMTcMSnLM0HSFU8waCR5xZU5PTlZYp29P6Gw35V5e3Liz8e1Qste
eFVNDmAWEm/4/Pc3/JJTt6uWjjEjFUo6ZtvJHBMV+1l7mHkkJ5ZIpgqTdmTkMQZJcP4BxJ58
Pw1PBoS+mBkJD6jTMUxGY34v1AQbkdq3fAQy9ALhdl5DL2dBJ0PE1msaBOLgEIOU3NihEyHn
6ZxfbJEpSxpLXjZ1GSxoLIBXyvTaVqkN/8Eu0J4JVFhkMKVUH0l32nRACWazxZswP3N0LOSc
aQhLUzPYd5/n7OuwugVYsLKLL+J6GdWWEA216BW1fxfYPvx9eFyBJ7z+evh2uH+2I2FJK1YP
f2GxoJez7HMIXoapTyr0d1pzhtqI1qZiPYWpjCo5bwMKWoI5dcc23NZ80NS+TO500raAmwed
BvCjWgzfgJWUXkph98mBCDBJmUgEn/LWS+kMXDqPN3saNNAePJhD02y6NmqsEnmh+5w+vtKm
SdRIn8N0Y7OASHlJPi9AA1k715wMfl1bbSJNZAfcSFsf/zrZcJMsTfKtabZcSpFyP78UjgLM
WF8mtDQOFk9yzTS4+H1M7bQO3bclb6H3ZqnpjM1f0Iy+l3Rr1pAO3/JsYCc5aIZS0dimeMyh
1EV2WGcTMmcjFW1F+8OoUZbnklsfsjT0HtJHPSedguDapApMmXVR083qZIHckqEl6dpcsjQe
fswjdPDIHBJUrWYpy4BjbCAEBXu8OLXe+AEejyMxp7hrOi/l3l24ovZXp+K6aI6ISZ52WD6H
FxM7JhEHldT19nSqWcs92xDS+9vOsAtkkANIW51Rwc5o1ATeMoNuiAVwNiwx/E2eT4tNqzie
V5m4mIqmVtnj4f9eDvc3P1ZPN9d3LigMUgl4aJZqi4i3x4bF57uDV7MOLYnoRnmgmbzZAshI
U7rAwpeqeN0tNqE5DUwDoSElR26yYw3pO9/xjzPyEp4WF6MgHaz+o4u2S7V+eRoIq1/gRK0O
zzdvfvUiczhkLlr0HCPQqso9+LdD+Admr05PilA4qddnJ7AAnzohA+QrFAPTTB8y5KUVwxzJ
Qghae7cNNpjZqywou1qYnJv47f31448V//Zydz1AlqlvTKyNIfxiaHL59oxc+3nbtvHs9vHb
39ePh1X6ePvf4GKXp+nFNy+UAeQGMRNVfiBkZQ0FoDcI3YJkmB/4w6OrrYhI+DVExZICQSqg
WAyaYMdc2tnfmGxnkqwvzyCnnzdNXvJxPNQFOTQ93MgMJ14fvj5er74M6/DZroNfRrYgMLBn
KxgYws02QGyYou4gwriabeKgQ+ChtpfvT/0LKkBEBTs1tYhpZ+/PYyqE/p2Nd4JPN64fb/5z
+3y4QVT++vPhLxg6Hr8ZMHZxWBKUR7nILaQNyWqAkT6sadx9M/f1ZqD1l/a2KKct+eWSRxnb
mLWKrmS03FNg6W7dSIX4CAElWNE1X4xoJjTc1Tbww0qwBGFIhGTx/gBrI7WozVrtWPxJiYD1
wftg4hJ1E98LOipeclGMpqXpfTP4wU1GFUllXe1SHwBgEZhRXwZseVh9NFXb2BYLwPARE60d
QhaRd01H3E4rWGHrSFy1PZEUyADKY3zZF7vNBRQfcmQLzD4zWM0W3Y3cfbnkig/MrhDaFlZE
beG9sBqjeFua7t6Im1QVBsT9t0bxHgDygPOJ8SDevvaagt4gllP809L24HdRiy8WO7OG6bhS
xYhXiUvQzomt7HAiIawrwvvTTtZgTGHhRXBxEZUDEdqAsA/jTlt86S6Xo9LMqRGi/6HiR/ZL
hDkdatemo3mc61dmjU69MxAcQATQY3WM2kk2Vj5TIr12udPgSpD7W7FoMD3VXZcs8NKmCyLO
aRZ9lq+vsCAlcI1K2NCIObvcH4BqTJ8yigEHdbsh70mnvndCg8vtt8leLsd7mSx+NWHZy+X+
gWGcV/zHet2g3lRx1dpglmpMs6OFxmoOzB39WznTdmSbyMeCtDhZYUtHLBOTS+BeJb2rTWZN
kt7P5pEO9wI8wWIsTyebtMMkCXoRXmZWqQljZ1lDHpHqOyhdigT4pdC0FQ7fmqqhJlUbPkOa
uwsYqXBpt7EIa5LooXJox/pqqLdna+EuTqmJ4PK7JgN0NFKXbmSsS9DgePTw8aHcXfrnY5EV
v+62hHydYk1Dh/CzBCTeZ51DJzFCBfBnFB5Aw+pXL8av9rWf3iWWg3FJs339x/XT4fPqT1cs
+dfjw5fbu+C+EoX6mROtWu4ApMKvyOacqSrwSMfByuBX05i1EDVZVfgP+HNoSiI+BJPjWyJb
nquwfHS61+6Pk686/Z7Zr7dgmRdSYr1UVx+TGHz/sRaUTMbPkcvFtLqVXLjy6Nl4WiRfqDXq
ZbDqbAfuXyk0seN3DkZUNlNLnJWuBiUEG7av1k2p5nZIgzucMrZTxXO5kC1U9enUSFe7T9DB
zoH/wLVM4jq+KYmsG8RbEJARh8R+k5vaZqLseCwid5QA6jUGjJizLVnb4uqwNMXlNHaFqNM/
lFObNc/wf4hVwk9IPVl3tbKT0LjvwKdrAns8+ffDzcvz9R93B/uLBit7If/shVVrUWeVRvM+
tQEPYUzVC6lEilbPyLD7QS4Q343vy8ZjtzQgO9rq8O3h8ceqmrIw8/uRYxfX0613xeqOURxK
GIAEWEROsbYucTC7ZJ9JxJAaP5nN/UuHfsRCNfNCh/BuiSq3dhdL9lLJFdK8i9pd40n0XWRP
cD4virYomoUlkuPZCHBQJXLJ4tcx6DJRxSneUlodNzouznaFcg065CA+VlRNx/BJgl13971v
Ki/enfx+Tp/jWYViuDAzerGDEEXByoz1ONNXmwRUO/Z9BVirAlBBEHIHNcUbT98SANO1rYXz
aNHnaxDRLt2ajTw/SYVELH1WF//jTZvEgVdt05TTt49Xax+BXr3NsGRp4ir3UYSfLRlo9irs
SJWgTWINSQd/djYWt2o2oPZjUKq1VeUEFgarqNwXzsA0Wclyyvi2YZUFbIgto8NPcj18AZHk
GgBNUTE5q/fux2ARMwugx7KZmpRgBEj14fnvh8c/AZbMjRkcwg0PqmHxGXaO5dNegE/zwB4+
gSEOMneWhi/RnyKXC9XCmaysiyG5MAMsVKFqrtzkpm1tXeYMP96nM9MtflyFH+WBt8RaPere
AITa2v/RB/ts0iJpo86QbAuJljpDAckkzcd5iXbhZ0wcM5eoeVVH5QGdhNFd7bDsFOTu0dQ2
G7GQ73MvbjV9uYTcrOmO8aZuF/L/KMfoYmXLA2S2zBRtXPTjc8fp+kSrpSFJJ+1ADpvv0nZZ
Qa2EZLt/kEAu7AumD+gKFuwd/sxHbSOmM8ok3dqPggenM/AvXt28/HF78ypsvUrfK/JDVNjZ
81BNt+e9rmNQli2oKgi5D1OxctCkC7gfZ39+bGvPj+7tObG54Rgq0Z4vcyOd9VlK6NmsgWbO
JbX2ll2ngBYN1ozrfctnbztNOzLUPkffl3YcEbSrv8xXPD835e6f+rNi4B7oWnq3zW15vKGq
Bd1Z5phNh79Lhb86RcVLsIP4QyWYseu9lG8ZLAuQl81mgHer2qXvW0HYJftI7ro9wgTLkyYL
U8Dao2TBFsuU3iBN/4AS04FXg0fATIIySsgqWfgNDtIg6KS/G0LmWp6df6B/LKo801Q3SnsO
KQeP4qFrKVI/3eeejcgrWI+6adoAQ/fcLYy5z7hGP2LSC1SSGobLBqMFVCzafiQRb9iOPpyc
nXqZ/olm8q0MnKrHqrbkEFKeRF7fUZb9eVl61hUeziZAwzQrN9Mj/g4BBLElD8miTdPWx5+W
gOknRmvb5dl7ahys9a6926LBeXitnpfNrmXUdafgnOOivH8XWKmRauqy/8N+3Q/nr4YZHG3I
/khE2D+YFsdbABvDj2FYLPnp5fByACT5W//zIVH9Ry9vkvWnJRNg+YVeL/dmiiwM5wc6nKGj
rbZSULVhA9ta7k/+1AeOJD31/3N2Jc2N40r6r/g00e/Q0SIlWdKhDxAJSihxM0FJdF0Y7rJ7
2jF2ucJ2Tc/795MJcAHAhNQzh1qUmViINZHI/NBztY/ChEjmVPM7egkeBLb0Vjy2nF97Qz6s
oZfzZ1daAXRL45jVU2NJLO2KA/9y6pA8pKyqaetkd1iLaTHysO0Y0w/fFwfqqNvz71SDu4ls
60tPTu46zjQBO3BKfkrb75MpsRScaiMoDzgXKq8cdomP5h54pqFxpyEYerq9PHx8PP/5/M1B
+sR0USrdOgIJTcWCRATr+HUk8pg3bi2RpZZYet/qRZLzRfZxToEyDvnLU0lVGem3F9IlqYn8
11M1Ao7d9aoByoQqBDMh949eIEOfbmbHZKrTjGJc/GwWUWaKYdSIxAIHiSNqUYxzvMqWBSJ8
jh+1hbWQoWHqRNH6/xoxiybTvF4z6DGrSbrp122Qsw4rb1QhjKymPsBeMeKbDZHeJ4pKjkYc
n7JZlDw/ybNwuqjXN/Q2aGXc0/yHwEEiBfVqS1/EoaFRFGMBrx5Gj9xnTzeFe+uWP6jq5mUF
DiKktDtpLaeKhhP2ghbe5pJqlL2sJoubakDQj7xZpXMMz8dTpiPVydxVtbE/4K9WZhaWjqLB
4YUsQjGzvd9mkUeSOhx2MFgooTacfxOMKGVSmj54Sp1s0EZ839qgQNs788cAZWOa124+nz4+
CXWoPNQ7Tn+cUp+rAk71BZy/CjqGcZK9wzDNemPWe5ZVLCYVgcj2kkcHw4rRyzfythG1+yNn
d3bz+RJs5pvpZgXKbfz038/fCD9KTHXCGr3aOZ2aiNGNhlyZRqTCjDwYhm61IpZGeL2Pdg7S
5ItCScoboml21aWKfGH511bA/+ZekcOJocNPGQme0NYVVcX2UjFRtFpR0WrIE4nAf02kJOUz
SvWzIg7gDJ4MtVANfy2aZTMuYCopZ4fuU2yG/MIw8MztRp5JFPd+WLIObmeBpyJjy9GVcAsb
KkepOfrDm2luXc3RXYNmDFgZNrdIuluqYYxL9HtGOKY/H749OWN8L+ZB0Lg1zqIyXAYNOe+J
HO3EGnhFA2xIbxbOxBvWsNrQGxASisfGMg2UKsENyqzwQGzrmkT0g2xyXjpJkATf2U6NO44M
+oIUozeYmcdexPSBG3mUxQro5lWp+mliNwIhk4nCwjdpk8stoEmeJjV3FJ2R3PIopvZSU0Qj
wJupe5yHif6jHd1ffj59vr19/nXzqDvvcVg1rS+PxJFVlIKpmae9OdTxk6tTahPqA+ageqx3
hPeVbViyEtgkq5K2RwLzQG4YZ1Hx1PIcjpId2h0CSwVSBo1AmSIzJ4p+HPxdQhz7PMXA/PbM
qhxmIwlu2UtHHF1tOxSwtsiPRF1A08aQB+WYkqs42F28JcTQ+6L3o0IRJ6J6kMN7fTaKxKLC
qKvpl4AE1C9NjymrYLjnnmOhJa9ArRCAWZBRMGMj6XuFkqrf9Dp3aK0qZlSs4SBwhm+jTcra
nkSt6j0Lb0HxenOPiGAaems2DhRECXu1fnZrncZFGDz0quQgUmNE69/OVO+IIi+diGRN35Ve
Y8mmtDXDTdnp8K6SvCn9V+ZMmLCt8MsFbVU0yAXVlleLeJRba7/g5b51cPf7SiSRcdxNIjhB
7ETNUpuYm8tBR0D3Imv+dWTPwoLsfTRJIfdxGk2Wsfzp4f0meX56QZjC19ef3ztrxc0vkOZf
3fpibJOYU10lq81qxtwSMi7wjoHWorEGgjRRASexTbgdqRUhaQsBbpkv53O7oRQJk0zJYTus
oGZ1atWKTiGUiCRjpVQ3NGXXZXbvaPLlvOfJucqXFz5S1pvlPjEX/n/YW4MZWzI4aXLbyiIS
K7iFupLqWDFC3NmOKHAuU0ugCRHHRIrua+aE4/W+Loq0Pyz7Liu4c0jznT+0sLCvNfC3L2PU
H0djvvOje7zBARIVyqXJiZozuEw6gesdjbL+TYVUQKiEqv0DMVzQp8ITUQvD2OC2ZZ3Z35tJ
MSGQr1ggTwUTum1zIWYYuZVGFezhDVwsEUNS1setXR4i1CLx1SRa3jdIQIc2VEy6wFibKUzY
L5VnJdwPKJkU1PWByryL4RjP5V38snMo0hgNQPv29v3z/e0FYdkf3bGKGSY1/B3YaCZIx/di
ev8n70BoG4Q7pZxKTlk8zpSP5//8fsaYPaxQ9Ab/kT9//Hh7/7SqAtrr2Rke8VnVY0rFIxRN
pRLoOFnbuqDGFpduaGe3eF2qs3brfPsDGvP5BdlP7jeNXlV+Ka17Pzw+IZSUYo89hW9ZjHmZ
XxKxmMMsGr/UO86/rMKAEyL9Se5qyUMUMT2IhgHGvz/+eIMzpVNXBFFTcVNk8VbCIauPv58/
v/1FD1krb3nuDIk1p0GOL+dmZhYx0rGjYqWIzQunjtAqXw/0TEBAmvnM0P86gW5hqZq2btqJ
x/hEHCP+eL4THjiZQcyjD46lHjN9IzOtMvoC5lOycmdvI60n6jc4Hn48P4L6KnXLEY3fp62l
WK6aizWOStk21NJg5nG7JqoLCWHhCc2dv+dVjeLNyU73VH8MzX3+1u3XN4XrtnjUYSJ7nlqu
5xYZYbb21tNapzor7cuEntZmGHBCNhAcVfKYpQ6AUD8gK13iEOWtXgjpl9Ih+PnlDebv+1j9
5KyiMCyv+Z6k/FNjfPFjZOpjZF+I8U1jKhVV6LYHyfaFj3eSVMzFKNQrbdMA7+4bh3ObBn8/
Db74lqeMitkwuWTTdxauSpw8bkGDCazyOGJpAWWF0dmAQoERdaSwEmMqBqITVkHFREMYcKRK
JfG8Tobs0zFFuOUt7Ly1MEN34AhveRbr3/Ypo6NJM4JsoGViIphl5lVHn6P5nldPmxul4Jqm
AvrUqEtsfRuZidrEVHgbOZM9s3WAq5gc9bKiqblhgYTjG3rUge65Na0y2V4MDvsGQIR7FoF/
ch3yaSxBu5wcwpn9uAf8nLq96Q3u4f3zWR2Afjy8f1h6GCZi1QpRdc0LPiT3cGI9yyqoSDSd
rhY6oSjYUCLbnqVjvVWwhYor+jWwS7CyUEH7KgzPg3szTYHhkS6gzbhJT1pENdQR/guaEz4l
pN8kqN8fvn9oGI2b9OHfk6YrCudRnzpWxQu0y8EY1NeJkw6pWPZbVWS/JS8PH6Am/PX8Y6oh
qy5IjMmChC885pEzQ5EOs3R4VtCqDOSgrpcLBQvp6y6cRluWH1r1iFAb2Jk73PAid2FzsXwR
ELSQqqnSlh1bnPsxWawftXHosLUxZ/wC9ViL1JaFpncIRWanY1sJ26BpT7jQXVojf/jxA68u
OyJGYWmph28ImOf0aYHrQ4Pthm6N0+Gzv3dxMS0+NMDqtqlIaGnki2iPXDdbLrdh5cHiVNU6
rGeLC9nKaBtieIiJ3o70nNefTy92C6aLxWzX2HLOHZeqqTobnao2J+G/VCo4bOgeGw82Vxpb
Pxf29PLnr6h+Pzx/f3q8gaym9w9mMVm0XAb2R2gavt2QCPdbNKu3flofhW90qGbytnQW7ctw
fgiXlDuQampZh8vUzVim0BC+ZtpjI1nVhz/uUEek2LqoEV8TrdRmDFjHhc1dds9HBOG6O0M/
f/zXr8X3XyNsY5/pSX15Ee3mxn2TcmnLQQ3Jfg8WU2r9+2Ls1Ov9pc2woL7ahSLFgaxR63LO
keO2YUfWb6jct+dKkJCcpujk0T6TWdSlr4iwwQV55+8zJcWjCE+Ue5bhbb47QQiRVmaUHVSv
bGeVwh4HZh5bhefZnbf+/g22vgc4nL6oZr35Uy9u43mcaOiYI3YOUYBm2BfSLjOuCV7EbDvr
wMga4ftQ3S+l7Xo5MC54BBilKlsGUR8GM0CdWPXC/vzxzW4GmY3OTm5a/Atf0Z1yYAAVe6pl
hDwUuXqfl/qUka0390uhOJcSqUDp8VaKEt1uazUb7HEOyno7zKPeuI5zULVOWkLON/+h/w1v
yii7edWxfOQiq8Ts/O/UK+Sj4tIVcT3jSSWLym7ejqguNxcq3KN7NX08C5edvqH+53PscKSI
sWV125F8AgQ56tEO60QQm88JFJY/J2jXGMXieToduLDD1LWFdANEHSpKsg7F9otF6PCRLFo/
UEyadeAqki4Uc/zdYYnH3as35hfoYH3Ku8JFjC0jVJrd29mORFkqzNBGFdeojsYZVL5DHO6f
yPl8+/b2Ypqd87LDt9X7ySnjlPHUog8rwfTsB4qihGHVpkLO09MstHD0WLwMl00blx6Q0PiY
ZffYwMQXim2GAFWGuX/P8row9vNaJJne+IwiFXHVNNR9tYjkZh7KxcxyUoBlMC0k+m9gP7re
N53QHg7ZqbXasjKWm/UsZL7wSZmGm9mM9iTTzJBGPu+btAahJQmB3kts98FqZXlp9RxVu82M
NhLus+h2vqRcuGMZ3K6NyBqcSNAisHuW8/5GZTzmW9qVaXVua+tNHH1P0co44aYWgbZcOKYa
3gHlqWS5aUSNQjUlzHtzRYGRA4Wzqg0Du4E0RgUvUYsnzPia07I6pMJjRu7SMNZpYveUx6tD
zlhzu14tJ/TNPGpuCWrTLG4necOpsV1v9iWXlktZx+U8mM0cd/0e+8L+UKOVtqtgpubGpHHq
p/95+LgR3z8+33++qpcHP/56eAeN8xMP+pjPzQtooDePMN2ff+B/zQas8ZhJ1uX/kS+1hnQW
M1Ume/l8en+4ScodMyAp3/7+jsbJm1dlpLj5BSFon9+foOwwMoBTGXqeqXc0SsNzon9CQRAk
+GOFhw30uqEtjCdtJD5lxO2f+A5nwhvYX2ATf396efiEliAGZFeIeiGPXkdkJBIv81SUU17X
IZdqYJg2z3eGEqh/j+99aXjFike4k93/bly48GhPo+2qac3SqKhcjcKd9+6BeM+2LGcto58y
t7afYVlWqGvxgAEq0X28OzZ9uLecyESAF1PPohIYZvSjdJBLdd9yzm+C+WZx80sCI+8Mf/5F
dW4iKo6OcpTtvWOBhibvzQpdzNtoZQyVqAt8SELZtj2Bvp07qe355D7ssy3y2BdeoHZokoO1
3x2d+7txeNwp5NcLWBA1Zx4lkkUYP0rvm6WXdWp8HDy/eu4ItjB/jx5v1J0nCBnqJ92Lz/G7
8GRfuJDQHbtCiyitCfkCJoDenlSnVYWEFdKzEPGaDIvRMSpOgGqeZj5A8Spy6tcz6qwfZ9a9
D5K9AwS5vjj1Lt7aneoGl+d+Hk4eCQq+ZwChyFdfcCkyQcXA51W8fNiPV6twSb9phwIs2zJQ
tGM3xsMQ2ReV+OprZyzDH1eOUF3hbEZ3tcrbz4KxV3icTZW34nSx6C1cn+/Pf/zEPaK7wGUG
jJ11Gd07afzDJIOugyil1tkJh/UJVEbYKeaRba49gY7HaeW1vi/3BQknbeTHYlbW3H5QQJPU
wzoJvSibGey4vUjyOpgHPgCXPlHKIrQgdG/G9NtOKqKCvL2yktbcfR8DpoDHK1urNjX5wqiZ
aca+2ply2F/7jriW1jIdws91gB7Qtef1Slw5XDT2MW3b7MibV7NA2DHyWliee+zOgxxupqsi
ckgpVNzCWa9S35xOAy/DN9nSwNc714bJEZQq+zsVpc236zX5jJWReFsVLHZmy3ZBR/Nuowz3
PnrH2eYN3RiRb9jVYld4IqMwM3q66jdvXCuGmfDKQIQPjpzHSbY55c9ppOmNm6ZhmpHhuFai
kzha7Vrvjzn6UOT4ejEdp2+KnK6LbHeeRc2QqTwyqbg7CicGcMJ0KkF85Z6n0nZ670htTc+B
ge0JiuvZ9Bgc2VdrBueCwl6shC/CvU+isAWtqbTj+JQruciNdWrg+OJ5tC++ujLGfIJmUR9p
WBkzFbo3W54YaXigZwwMBTRxX84PH8fgjTUreHi17vyrsrSbjawobV5iZHwO2x6ijrTuqjHN
Sb8/YbU86YZsJNkf2Zlb/rZ7cbWLxTpcNg25wvevqI7fEpCrJ1ehjI6cR70Su62P7pneovEl
cfc8m+PLbuGrGTB8aTxPcidZMKPHmNjRS/yX7EofZqw6cRs8ITtlvlVJHnZ0zeTh3ocg0RcE
pbC8sEZ4ljaL1hN8DbzlxOBlcuX5IttGvCDqI6LKHm0HuV4v6SVTsyBbGjzmIL+u14vGYx9x
Ci0mMzaPwvWXW9p2DMwmXACXZkOTrhbzK/qJKlVy87UEk3tv+8zj72Dm6eeEs5T0UDcyzFnd
FTauqZpEn7Hler4Or2hJiHJTOZi4MvSM0lNDgrLa2VVFXmQ2WkxyZcnP7W8SoATz/9siu55v
ZvZeEx6uj5r8BJqAtSkqpPCYNhIYCYuDVWN8O+3K6tzhi2oXakvX3jP1zBHZ4PccHUYTcUWz
L3ku8YEAM1vo02s7xl1a7Oy35O5SNm8aWqu6S736LuTZ8Lz1se+8KFN9RY5oBs0slfIuYiuM
QncCxSZ8N2LPEMArAR9+YJVdHVNVbLVNdTtbXJlMGLRTc0t5WQfzjce0g6y6oGdatQ5uN9cK
g4HEJLn0VIi1UpEsyTLQmyxXJIm7rHtgJVJyfkdnWaSsSuCPNetlQre8xEhE7O8rg1qK1AZy
kNEmnM2p60orlX1xL+TGs8IDK9hc6VCZ2YBsvBSR7xlmlN0ELqiByVxcW4xlEaHhs6FNP7JW
+431eXUGA/wfdN0xt5ecsrzPOKM3XRwenLYJRggc47E35uJ4pRL3eVHKezsO4Ry1TbpzZuk0
bc33x9paczXlSio7Bb5tCsoNYnZKD2BonZL4KkaeJ3vDgJ9ttfeFxSAXQ/UjQSJHGNmexdfc
hnnUlPa89A24QWB+zRSir5XNzLuLZtYI/xLZyaQptLVPJoljejSAKuZZuBW80tZ9hnzUknQY
0cmnqkPvOcHgY1KleaJOudksM9psXJb0Yiydc6ky4O7fPj5//Xh+fLo5ym1/t6Sknp4enx6V
Oylyeqgi9vjw4/PpfXqhdnaWsh4Uoj3HlLkTxUcDbaa3FIpXW/ZT+HnpJdp6r+AT+tfOujfB
dhdgFyDJcqJGmdzbAz2NziK9DQNqVNrVz+yDgiJcSUSa9aooS+g1xEza200I1uSELMpz6Jt4
yAt9vHN6Fgm1LrnFVVI4EbJ4A09PJl5lnkCfshIyIyFMzfKIwyhMIl7VzPPEDH4D+RqjlSsH
7dk7OMkjGQyMKCDUOiJ9xTpDEMUbFC2KacZmmwzT08ak1x75r/exqV+ZLGWC57ltUburcxyF
KgTWsx1UsF47w0OtNOfnjDU3eIf98vTxcbN9f3t4/APfwR3dyLTDznf1fou5HH2+QTZPXQ7I
IK6grmZv9NAVRF7qevOUNXipQe8Qxy+ilsfWM36hwovWZ37U9/IejAvYRgacBNMZRcaEH8L3
Hz8/ve4OPUaK+VOjqbzatCRB10WF5uNwEDVPR6iOFVEM/VrNwQnSsEQyVleiQZHenQdDi16w
eyhUrS5RgW85mdgpNh2BKo6N+1EDV8JOCSe25vdgFi4uy9z/vrpd2yJfinuiaH4iiVuFAGt0
gy8uQCc48PttwSojaqenwFZoOLwZ1HK5XK/Npnd41EFqFKkPW6qwuzqYLWdkrsgiYekMiTC4
pRPHHdRldbumQLMHufSA9Zp+bufLTpHVKOQxWWwdsdtFQEWRmCLrRUA3pB6jF+ubrefhnEyM
rPn8UmJYnVbz5Yb4riySFLWsgjAgGDk/16a//cBAWFS0alK59UdiolWLNE6E3HcPo1Jp6+LM
zuyeYh1zugvh1FZysqUKWASovXzsoyxs6+IY7YFC5tDgcKZtqb1IxEo4pl7szG2U0cOgBnUw
I+1KxuJheK7iT1iKDEfZgdSy1ITmGunb+5gio7EK/i3tKJqBDadLVtYiIsMWp1JwIneexBqF
ovtLj+8N9REJ3xYFfW0wiqkHlwiEiYkgT1GlIFF7jfpzVFRto51RlhoZggyJHIQSfLXWdb0Y
2adM/f9yLei2gwOF8NgUtIB+awAreUEIxt5ys6LvSLVEdM9K+nSn+diSXpwoLXKSTdMw6nJc
8+1Vtvu8Yexoz1v36wc2KmG+CQJbJ744Y6iuPaVlOYMxPpY6MuYxRY0FQY2KbWWhiA2cXRJS
qFEjvxIlkSGSYb2lOEcB201W1ARPHXBYRLGkiPkZIdcrspp1FlPLy5izMtGTSTWrDUnM9UHq
zKpKmCFAAydjO3WXRlUaH44sqi1ZrmJufS9+jmKIm0pCgY4ffxYx/CAq8HXP8/2R7lkmlzMS
dnCQQIXtmJVk6qb0PH40SJQSZbyW+FGuqS52XSIFu7WaUM8J9aAQtWh1bFwwtDJqhLSMRIwJ
KxHj0nTfNfkslqv1wggysJmr9Wp1gbe5xLNhKwi+gzNoS1CtZUlUoJ4H7nJjSaDFpM1I1yBL
7gjaoWgiUfly2h7DYBZQWtpEKtz4MsHjfpHzVkT5eh6saROrR345o/RhS/p+Hf0vY9fS5bat
pP9KljOLzOWb1CILCqQkpgmKJqAW2xsdX8cz8RnncRzfO8m/HxQAkngUqCzc7q6viPejUChU
cVrHWYQ3ucLPcRwFy/fGORvD19s+b+YxI6xNfYjyJJQpKNzEEH2SxqWmI7t0U4vXrW15F0DO
dV/PocwVimzNGO9MUuXLGgH1OT6Uz/l6bdADglVHsfS3I55+13diaAXrwQr2Vha4EYFVjtvw
/ll3tS/8lMRJYNK3ve2V3caedeO9hmvHexXZT+d8lpB8YnKKY1EcV6ifcIuNiA0g1G+UsjjO
AljbnyC2eDeGGOQfgQ6jc3HrH5wFFsBuaGdTiLLSfSnjBIfEOU36pAp0TsMfJ57PUWA1l79P
8Fo01Pry9zt6o26xgfeQNM3ncAX319N7w6tyngO+SC1OcTyOgwNfCJPSDciVdYGgevZQiNOy
wjVxXit0PIn/Bisjcvl4NvIFXxJF8/LONMiRBXtGws82AcUVmLsafHS23abVawRVxZksE32Y
Z31rGer6tm6Ci1TH/kZ/Mx4LETWYBqcn1FWSw2RrECzwNp2EQJoGBTaLea4K9PrAataRFXlU
znibvG95kSRpAFzEdWyHvfbdceoer6c8sHhN1wvVMkcaqm73juWoNkNrJJwQ6YpaVSOtxGC9
DqFIu8AlZLg4M2ptUm3Jz0KUzGcjUk4TQ29ZUJ3yHIVYgz5g1lrUdI5EO3BLt6U1z4SNL5NL
BYVaKfpMVdBD1SrxGO8TniildZWZnaJrMdZD27tUqXo8io3dUcRvYNNCVLJAhMuN7VWMBuxI
rlgIzFyryE4aNe/FZnbkQ+CZuWbqpCs83uI2/asSWhzqBs25xzjzHw87uPQ+TOvdNN7a2vXo
7HAQGkd7ucD7wr7mYDksVRDBNpxafttrQjnRk7jaeMJzYx4TMYHG9sUdDzd1n+IlPhIxz4tU
DDuKWYqsTFVeZv7X00sV5VCuvQkrh9F05fX0Bgae16b1ZoaS1fF5AViR4pjapR/YBKxHfyGY
+xRbOSQZXzoU5JwXFSiWuKQ47EwNWtuCu0V2z486zaat5am+F78dA2/89B3WlejFR5xIJ9RL
h26/6TUpxKBQo9C7JJNwke/DpQ9PtMs89xWSiG+2EnIaUtEoppiT0Cky3FEtFCX3OAVJGv0M
3+WPY4+SuJTUslLVNFzXqcDckoeUVcqHrz/JB/7dP67fwU2m5WTECvqFOE9xOOSfj66KssQl
ip/azYpFJrxKSBkbo03RR9JZin5FFTs8UE2bNUkPRd5SqH63J75E+ktnxxKIa2r2sf52Irsf
1uMRKae6PzPpN6fzzzVt7fZYKI+B5XllvfBYkB4TrVa0pbc4eonRL0/UOfitl/lY/2+uDJCL
bvXS/ucPXz98BOskzzMN52/WbT6mTLoN3XwQ+wI3w70oVx9BopizcJBL8tV1XC/dUoPnWHC6
u/oi+PT184cvviMopbRQrorIdbAHogCqJI9QohA4xqmVDkAXZ5Y4n3IOZA2gBYqLPI/qx2st
SCH7KJP/BCpvTMFuMhH15j1QGDOoggm0cz2FiknloQ9b1kyuYZJ20uyHDEMn0UsdbVcWNKN2
5u3QBDyqmow1G1vR7q+BUCpWN93F8hCqWRNeINaC86SqMNHfZOpHFuh92jVI5uCtFnHdpXxA
/fbr9/CpoMghKw1wEH8WOikhhadBc2GTJWA0rFigIXvc+aDmsH0aGkRjwLmp/shwE18Ns+7U
BVxBaA64ruzwaNdLGoQMc8DccuGIi46VgRcHmklvBj/y+hw09rdZn7F1p7mYA++CNIs2fh3Z
08TqKWDnruBpxA8aGj4x0ZLjszwkVzdAqMZnrAQMyKUP8u7ciRN2wPPCMvzgiB6n+W4nja7r
ktURp7VuO+OPEj6pOFPI6BvArSf4lw94RVkNL5yQe5tp2OMcGMDD9f019CLqBsbMgRSlm28x
7tHYOrrYYBZl+eUz6LK6InHXNZ0ggZXmwPETnnY9QnxvKItwPNJOSG9D01uHGaA28E+erQ1D
LwBk3A3wLmubtAICbsOUOQsu8ct0pfm1usQ91egDW8nH7LOKJImFI8R+h7jRjXnFrYoEZ+Tr
6WSmdbkLCXFoUA/DYEXQkatloULvoUBAoi408JpBQC8Uj/H+qnzkmqzBl/mXMWC+KDrtTC4t
3OuK7RW7+OVE/BtNT31A6Jizomuqz2Zpmgzig0y5ddJYMHFgClpZmzxiremG1lYRmPhwe73y
gNMW4BsYqgUl2sbbTXbJLpgemXCTf8BeOcRumq4zdjBdG4Wn6fsxybBGWbCQ7tZls8IS8LYn
0ie96cSvf4NVwqjkQgvbBq8c1xO62Pry+3qylBNFrF83xh/g1H4NsaGMLkWlfJNXSwMhOlTa
d4GHVWMlEWS4eTBDGEnaRbBalp6CSG+g7VDuL//15dvn3798+lOUFTKXzqCxEoit86gOXjLO
fDucbbdyKtlQqMINVnk75J6TLDWviRZgJPUhz+IQ8CcCdAMs7j4wtWeb2LS7/LSfydir64PF
WdteY5nf6xgmcHayu0MZY1l51f35euy4TxRVXHoJMltPkhBdYushHejoO5GyoP/82x/fdsNz
qcS7OE9zt/skucCMClZ0Tr2PaFOi/s41CO51kG8eNCBpyfUMv0WVEDNvOBWFOm03dt2c2aRB
XmwkKPHBskOVO5B8ayxG682ms47l+SH3iEUaebRD4Qx0eIzmEsRKuPQxuK/FwiPJ5AhFfDHC
WvHXH98+/fLdPyHgiHab/x+/iDHw5a/vPv3yz08/wcOnf2iu78V5CPzp/6c9GgisY7YJnZod
EF1WeoJ0tXkOzPrQdu4w7jjwdjltF4qAtrR9DQ+ZnWXnpaXLNDaoV2lfHPhEzDwkoDYg00s6
uz1NwSWXRdNP+HTHtn+KjeBXIXsL6B9qmn7Qb88C3c3rKxNin3+kvX77WS09Oh2j3+1ORRav
4Bpi1UaFIrQH35P+BQ/NQSOdjQWWtCcsQZ+fxq64ltYMDkSagQFFh2XZgOaOkpVQsJ1zRiTC
o4Hpz/+yaNIttlKJiYlLP/wBHUq2pdd7MgFfqTOvnRK864T/lbsBq5APsTEcazt+gJQYlSeo
QHG3ieRV8g6hkkKf3R2tlqLZ4Zqk+/V5fMDp1tHaAxSYhwCpY/HRTgqIloQGRKWsEKcT4iZ/
hTB5A34klG7c5zoJ6ScEDC/twYotyMBIXInVPEIV0oBLRYvdRRDPwKbMblB6SQxFtQTw/dvw
jo6P8zvrjCAHxxbzUg4zQwLxPcFCaTbRDvgXT+l6fDqjUfwD2dBu/Ot1hLhrytm1VRbet0Uy
R+6g8tYHc/y43ujtiGAXZv9hSbbq/oR1TuiKjfzlMzhkNsLkigRA3t2SHEdLsBd/BqLZC2RJ
z29X+Iz0HTg3eZGHQzdNDUqFOdIQBoveZ9c8/weCkn349ttXX6TjoyjRbx//1wX0q0L9tBoe
ng0tv1+nF/lWHsrHeE0hAtHy2lBsGmLH+UmGxRLbkEz1j/8ynxz6ma1l1+KxFzZOAxAR+WY+
DRF0ar5lM/hBqj7dxGe2jh9SEr/hWViA2iW8Ii1FkbfclrnrilA09q1G5U2yIRwudErGJGVR
ZR/APNRav1zUR5joGfNWYaVzepr9bORVOlanK2n7QACCheVYv/Gp7nBT94WJXNppenvtWlyF
v7D1b2LZD76MWXMUJ/uQqmHNsB6G69DXLwGfAQtb29QQ4hnXxK191w6v7fQsy1ZsYpwdb1Mg
hvMylqWXvacl60TLP+P5EW5WpqdsfXvvnpeL3YapY+3z5ufd2c/UHZmgb6j9gUZYVvZx7g9M
CaQhwHbltDb4u1sn7cVumLQPq6AlCmjC4yQELQiIKgQF0WE/5HGycFxPi3ba+OShQ344qXTT
O/2U3Vo3kO/ZGzsxh7bFhVIKEhWf5pcPv/8uzlJSvvEkbvldmc2zIyypMi4i36Yvl2TajNiV
m1KxuGGMlGXWvR6PXkJw4Yir5QE9cfgvivH7E7PCe8czxTe5oZkk+dLfA9cCgHboGzkJSYdW
r8SrED1WBQsEAlYM7fA+TspQuqymdd4kYixejzcv9Z27Mo1fMSltGS7E9r6mTOHmKscsYSXo
u3NZuv9xcp/WLWqm8JBTooHYoL/XKJgV7AzKUxlX1eyMo45XpUNi5OJODXJJY9vSWdLv3QCe
9UPVvbO4IFllHjx3i7tqMiT105+/C4HGORGr9lIPx8P9VjcDZiqsRi6EcW+8qsiHzKg7ng1O
3MaTisfUbxdNd8P+IEzlzlxUFnU7Y5+PHUkqdzob52SnHdUadmr89rUaT3p19xepYyOKG9P7
a7hAykQvjP9YD+8fnOMyiORQCprgKjFWJdLcQM4L/CpU9x5sc3t9Cya37jSQptpuKyyGlqHE
JH4w30go8js6V4UzrYz3A8ts9ztnDV6/32lau2pncOTV7I5ZKmSWqzvBIbKR9K0cF8hg7loF
Jri9m7LdbEiaoN7jVWNem/oVXp3+YAQCQWq1HlZ3ayu217jI3IqBQcYh9pdXNbUxLbKCSZpW
VeTXu2NXht39qaV8gudsqVuGJUz0dtXu18XuuPN5as81N9+16qTIy82wT70bnXuPH2rvkc0V
f/9/n7UabzvZr3URvEplJV1BoPvZxtKwJDODY5lIfLf0GBsU0PNsDOzcmU2ClNesB/vy4d+m
kZtIR2oVHuDv2i2CQhh+GbziUK0ot6plAJXVsCYgw0u7gectnsAzHDsd7ErE4rDfSJhQFeHL
mvV5GnAzZ/HgL/BsHuy+x+aoQi2RR4GRtXKUVYQ3dFnFOFC1UYZ3WtXGJTKk9NAxjiBgofCo
XzHlvsIgtKFxTWoQnTOCi8CvvJ4C3/acJIfcMp81YcqLNMGHjsmms3jKp2TR/Tpqps1oYzNw
bmVAdWpZo2huG9vsccA0wwSDebPbOPZvfjso+o7XO4vtcqd4/ZpaMRqLpz7B1A15HGvOIajX
NobUWwCY0jfjTKjJS0qbnYFof0VF8gbt4hlGl5BGo8K62NT5iiMprw5ZjkkeCwvRrxC8b8k9
iWLsJLEwwMwprAO3iVT4mmCxYFuixZBgBevbszhovmJrxcLCjsa99tJOFlH5iHaIy+fHd0k5
m4KLA9g6OBe8NO/CYMMfNzFoRMfCGN5GwFpt9SDcdG2on8sERgHAVfU43dr+ca5v59bPGx4D
l5H59t1B0GaWWBLwSru06fIyZ2d4inSqgymoLABIzeZT6oVuX/9uycgOM1tmTYinRY6NJaMI
cZaXSF5Ny2XMXsVS5Jb0aXwu5fO9HMakSA5++qLbszifA8AhwrIDKMkxjYLJUdo2EwaUiwx3
Ow14KtSB8TpV6DHNSn+8yAGm9pYMXXAW68udtCcuVqTcb5EbYXEUJUhTNYfDITc2YrkYb1NH
/vl47RqXpG+DlfZNWWF/+CZO+th7AR3S9djx2/k2WZoaD8T3zZWtKdMYe7xhMGTmu3qLbjlR
2xAK7jyeZAs8IXNck6f4Gzz4m0GLB/WnbXLEZRmozCFB/aFvHLycYzzmroBS1D2syZGFP87i
Z80oeIqQ4bXBE1Cc2DzY9rlysDQQV5iRsnjW23P3ONXwEHoQZ6qAN27N+1JBdKidkrzEEXCY
tskKONU0zi+ugLOWgTYQX2E6v/lfSn9djBLkK+nBGa83PP/YazE+j7GfIhE/6m4Sksx09dGG
FUmEkeMiQRJr2r4Xyx/1q6Sff1quJBesy18gYCG2aoDSM8oxy2aTo0pOZz/ZU5mnZc58YHmG
XTcE6TVGLrTxPzr3eVwxijW9gJKIYUfYlUOIejWSZlkkCFVqe+vBL9uluxRxinZ/d6Q1eoo2
GMZ2Rhof9P5yS8BSzXNUqWoMnlYPfv9bXmG78AL/SDKk7mKyTHGCDbm+G9ralM9WQO6neQgo
g4Ath7qgbb5hggesdJwISQWZEQAkMV66LEkSrNUllO1vRpKn2OsbxRFjGUj/NqjnMpOjiAqk
3BKJDwGgqHDgUGJDROr8ygR/M2qypEiTQ2BzdBGSQHoIVLwosif5FUWOTjEJHfbGtCosNj4o
GdMIKywnRZ6hubXDKYmPlARPz9tmScz4Y2v/0yJFhh0tI4y3TNGRSMsnw5CW5TMG/KZnY6h2
R7E422O1qHK8vLtrTk/RuSskKpSKZnzIkxQRPyWQYQuABNDSjqQq0905DBxZgs6dgROlUu0Y
v2JH25WRcDEtkboAUJa5PxoEUFYR0ibDSGg5z3hdTlV+wJaUkTpvMtZPaDCivCEHJ7ti4FGc
28dTiyUvNrwHOZ3G/Sy6gY236dGNbMS0jCvblOYJNn8FUEUFMiC6aWR5FmGfsL6ohASCD+Ak
j4r9M4bchErsCsvgSCts09HLOlJctXpHoe0iicqAmtpmQhUJ9vKIz1zAsmz3YAOqkKJCj3jj
3IpNae9jPrIsyhJkUAskT4sS2dFupDlEuLQNUCgUxcIzN2MbJ/uHofd9EX6orFjYhaOKRAPH
BqYgp3+iZIJ2sn4PsZNRQ1uxE6PDthUidRbtn+wFTxJHmK7L4ChAbYqUmjKSlXQHwZZwhR3T
A1pmxjkrA+ERtxRoUeweQxsSJ1VTxYjcI92fJgGVhIDKXRWAaIsKXXCGWllGIvQZk++HOlUr
l1cKTso9TQu/UJIj+yWnYxyhgqtE9npYMiBNJejoUgl0VGqiYx6jEgtEfiLj7cmZXXAVVVH7
W98rjxPzHn6jV0kaY/vwvUrLMkWfGxgcVYwcKQE4BIEkBKD1lsjeSBUMvVh7OXIeVlDhvE3Y
wCIpL3tHcMXSXk5o+8jLGfPr3SdS6/gnY+dr7X02/hLF6MIvZaPaCr2jSRDCJOj4a+FhvOYd
C/i1X5ha2k7ndgCvK/paDjQg9duDsh8il9nRuy7k68mn3adO+mZ+8KkzDbIXvGlP9a3nj/P1
VRS0HR/3jrVYTU3GE2h52KUOueBFPpGhqqSX7d1PwqkjjLvlBQZ4JCN/PEloK5x1qzHeFq7d
MkN06hqs13e5wAYUZVB25lhOOuDKt09fwFj/6y+Wg531exn3So0a0tfoWiWEncf4AreYdFxH
8y9uEuAjrOEsWBI5zwRrmkXzkwIBC952+pp+Ny2vbuSymxjeRIYRj3ERvNehi3MDbH1iR9HE
jHVHxxMLGifgSGiNsgPgtat8vfPf//r1I7zEWBw/eTcj9NQ4LgWAstwrG/epQJWuKuExlvKx
4EGXntiRUQGSURsi1A+nhBerPydBeSuK0WylGNBdI/ONFuK1dWeyDVyD9JUojdGtGkkyGqNn
RW1D9Y2MKXYgbMp2L2x8ojXS1pOPhW6qZVeate9qquM81ILPNW/hMQ97nNG3RLLBSAyBf+0S
aKJftAXw2925QAXapSuE7CTrb1z+cfIYa9aR1G1BNWvf3erpZe8Zbj8SMATfUgQCs4OabCuT
bHxy4TCHcdcwW9bgL0pu+3+HLxQbUbK9Y0USmhDShpXQa2MaBwGwvi02aMpbbeT2uyKHRqh/
Xa6pi5WqlZaiV/iJf2M4YFK1hqtDVHrJ8gLXLC3gwS3fom7cyO37WTnZtGet7XcTSODd1KYs
hg1msVY3pE78BZ/BHXpWzRBLVRvneYRaVEjwpYoqt62mIecFahIMKOuysnC9a0uA5lFst40k
ea6XJPLyVolRgS1Q9XHOowjZJhYLZmVUy+nnj19/+/Tl08dvX3/79fPHP76TuNxHZag645Xy
ttMBi++PejFu/ftpWuVank4YNMttu7rks+rfj+khC/WK+LinxggCI+w4ym3X7NIVNi7pb16y
rTwVfWdqaWNwTGO8FGuxWPfJuXk5YqRWIVRlUO7nLegB3zcWi/M4G7B7HydlGoqRIVucpnma
OoVZzNittEJvXqRQ4L8mMMg7hV84cFHA9gckK0TzOMJ1ZQscePik4OpwwC8iVjg0wbUZ/18+
zfUtsCaFaU30jD1Zw3CS5s1Y4BPTDU1IjFySXt1Mm4XZfE+H3B1sHKduBs+Q157XtqufjQXc
od2UGzt2w51mbcxw9JInr5UdL5nYKM9VgW3FFo/eYz0I5OTKnGY2ZIvQBtbk6aHCq6mEXHSg
bEzoUxmPy9kIjf5YotKgSBFGUrzMAksCY99hwpWZxjiohzzNc/xKz2Gr0Gu5jcnd5Qxf6FLw
3P1YsbzmKdrvHesPaYT2O9ydJGWM9rtY8Qr7eZGBif0H1bc6LGjnSFPLYMKwQzxJWOwWaG28
fcSAOEnz6hDIVIBFiW9sG9ciiO4WDpjyqsCK4D+ssrCqyA5ByA7CaoNCXH1e8uoQeInmcJWY
UOHUoUqKQGl2orbYXBV6tjR5xlg0FTp86JhnMd7CY/X/pF1Lc+M4kv4rij5sdEdMb/Mt6tAH
iKQklkmRJilZrovCbatcirItr62Kac+vXyTABxJIyjMxl3IpvwSINxJAPkJ/pI8B+2TpzMvr
6cwZa2ku5JMqHpjFGVl0APOpLROzqMeIAQHbSI8eOLq+vIqYytAU22LzNbFJrSSFacuXr4Au
AEDhODQjoWsIHYUdnmggxAvaIo9wA0N37DCAOlv6tvbEqKD8zGAFlPkB4gkdNeKBBk3XFARv
nnbgksNVkb5JzHHphpXi9th46oT0i5URTPZ4sbB4rWPeyBLdydEXP73FfkgGoLfypLs9Y/N0
PlfgSDsqVuCySpETsrRCl3hV1IVpoRUVBA5uYamHiCjRP5cn4O0O6JWi3DhQwSIHOdMXWaym
LlZFE6y8WGSRYKiXm6xOQuAbZalYuq5XLC5uRtlkudoyGXesy7e71+9wKjXc+LAlWkT4T9Cn
DWgDV0DF1dEoWqdU4wKCHP1tlwxcZw5t1xJglwEHg/WfdjBkDGB9kzbglKUYeUUgXJ0zThsi
BQx37wpZ0Bdvd8+HyV8/v30DB2VKgjbvhea4tc2HTCbSze/ufzwdH7+fJ/8zyaJYD2agZM3R
fZSxum5HJnWZzqKrTMS/UBmHphxw/Zw6IGZ0xwETGtJkmw48Yn7eZCOu8we+mq0YaV4+sJih
4pSixLBTU/uRxoP11QfwghUQaiap02gg2OBHSbH1HWualRQ2j/kSP6UQVkW7aL0mM0yQE8LP
xkvHZ8xi5Xqq2GD3E9IhWBqbM54TlVvsNB6M1ZoqWS8bxSSeoxW7GX5vjLSdX5jOu9rr4R4c
qsOHjScc4GeeiNuN8mBRpbq/7Ul71TZTUGGoaoybKmGZVp8ku0qREjZQpUcnYmRIMOW/bo00
xWY54kcG4JxFLMtG8xRLMi5aG5cdE3kbLwvhy0j9/kDl7TDyiSSv98LruErLEs2zuKB+pYNO
yT7M52mld+xC9fIpKFlRpQVWQAQ6z9gIiK7Ct4me4oZlTUE91AIIPrfqYo2eWuDrt5V4ZMbU
FHwCaaRGI3xhWjxxIDY36XrFKF1gWaU1OCRr9M9lkWaCIohJrOeeJetiS735CLBYpu0kwIla
OvwoqdbpGRZIPQTI1SafZ0nJYoceLMCznHkWkfRmlSRZPT7GcrZMIxEfXi9wzruxIvWpJXq7
4BuWUc0qkSN7fFalUVXUxYLyAyXwApz6q+HNBHWTNakRmhCQNX4vUxAuKamx34BUsjU8kfOh
rswHhWisSWXSMPACp3+1hMgVEWVFJFDw5lnBKDemU1mlXAgaSVez1ChyzfJ6s15qRDBhAtUc
Pfu6SRgdS6BF+WDgKzopIwuOzbrMNtoKViH3lTBbIa4lq1M073qiNthwCXJWNV+KW/jIKFOT
js4uvrDUSaItZs2Kz+Zcp4G3ed0Hrko1+noDu+G+rF1jQUtTPcqigu7SdV7gj3/lgmzbjH1G
He1S63y9jfnWODrnpC7XfrWZa6NB0iNeNTh8i196HVim63d3b1vEnj74X6ckDOHsPTX9LCu8
im4PWOnibLTnbwiiAgzjui9GFn2gMfWTyoGqWEUpPz82TZbskzXfppVHOMCHsx06h/GlHhTJ
aD8NwLDJwD3xyMiVcZDX6zGFAMBFuLsVq/erKNa+PpJCeicWrQZMIp7TIHr19PL7x/vxnndj
dvdB+9heF6XIcBclKe3aClDp222sig1bbQu9sH1vXCiH9hEWL0cikjS35aWDcsE7VB4V6VNy
Tr6ycTmqSdUAGB1Fvghhr4f1+Xj/g1BQ6pJs1jVbJOBrZJMnVNIVhCO45BO7z6pJFzloQKpP
lh32ReyT671Lhg/r2Sp/5hD1AqmZ7yStT/luBCQ3MMRRJLy4ludOtQwDdS+2eErgGFjEzizC
OBl5zEWc8zWXiCFcTgTRZhLzFAPnTqO5RXq2di3HnylCtiSrEQokBZThXY04j/LAdUKK6utU
7YlK0irLsj0Uml7Qk8zm50UczlQA4oBOEh2KqJcXDq4ewRnMVH98gtrfR6tE6YzPMXqhpY8r
iwiuMS0mUQjQSdCbAYi+UdzS93egA5LnqnDdY6qK+kA0WoITA7MiWRn6pHuzDg3xY0o7SJMt
uEZLqYBCQ/v4ehO3VM05Ug8Frp6gewlvWKPKTwIzL2hacmQ7Xm2ROnyCY3jQ1kZx7CC7M0Fs
tclqzzGHZuP6M72Z20cdjdpEDO6fdWoW+TMbW7TJTMadlvRD3f9b/7KiQ4WzS2vXXmSuPaOf
nlUezem9tpZMvp3eJn89HV9+/Gr/JramajmftHdcP8FBHSX5TH4d5MnftNVoDvJ2rtdEiTmh
0XnfjTUKqAUYSfhpYRrOL9RbxEa4JeVQ2UdCO2hk8sE6oncqEB0cylpm1D5skM3bvB0fH821
GuSmZVLV6PypAKMRxhBTwTeLVdGYJWrxvKGOXIhllfATxjxhjT6CW7y/phvBo3KjzfcOYRE/
mKTN7UhCYqXooM5UQHSLaMnj6xk82r9PzrI5h1G5Ppy/HZ8grsb96eXb8XHyK7T6+e7t8XDW
h2TfthVb1+Brf+T7Ecs1d8wILiHk3mftuk4aFFNLywHuGNdjDbeJsQoEi6IE1MdTLqPTgS1E
vLV0zkinuwlfOPd8VQSd2TqqNspDg4AI8R7o1LNWE+H4HEAAE70gtEMTkaISIq2ipqhvaWJ3
mf/L2/ne+kVl4GDDDyk4VUvUUg3vW000rsHURNJXXSeQcsLk2CknKnMVGPk+sdC9gPf0sirw
q1sH8FKNfDiutjJwhhLPDb5vCHUds9RkUp8KW4DN5/7XBB/ABywpvtL+ggaWXUi6Z+wY4tp2
sQowRvYRn0Mb8hpZZcRLpoIEU/LttGVY3eahH5CVA48JM/qpfuBo1Zc1wNDU7YHaj9yLBUrr
zHboxBIiHU5oLIFZpB2n+1SuwvZ9xB8k4rHIkGyIxVWdNiBkFAgJIPfsBik5IPr+Jm702QDo
/Np1aLO4jqPmEv7MIlUSWo5F7tou8eWKD2KbpvuhTfY0T+FQMmTHkOT8IEWNnS2n04MHNDwu
9UEd8/kS/qmE3cFT3hzivEFJ5XzE4JmFFFPToXpBICM6egoLqU6NGIimAfqMGhcwU21i0Fez
qarlPvSNN9prAW2QiSawR3aPXC9G9L6GIe/Yn0y2PCppY1ix7EPYwnXcaib3/QzRd8wlnmh5
fkS9tH7I8o2NylnkjCG9gah0Gf50d+ay/vPlLSfKi9rsHN7JThiMLOb+mJaowuJfbl7YEkIf
/Jul5DuiwjfF3jEHxPEsWlujZzGc0RMM1JpYN1f2tGEhtfiFDVZ/VxH30lIDDP7MbOq8zgPH
I/p0fu2FlkNMnNKPLHLiwCC4NG8MRaRhwHW2NhoiQ5F1I+r08js/BHwyntpgJUaxFw3/H6zf
BqJpQfczsLNrM9taqA1eWiDA3TuVUlxnkeoC9eHlnR+ML9atc66JnqvBYNFQrhLZcmi+WUxO
r6CZrzq7vF1H+0WqWZTeCDp12S3zUZklZZ8X22S/Lpp0QZ8VWrY6yRYghlJPWy0LPxuWNfEF
QRfytx6Iu73a1urYZcs2uzity4wprglXsechF+BpvgRfi2kK6kDo1THJ2kvRfc6PQ2Bx8IFR
6Xa6xX5RzgLgsxSUi+ZgFk+/J6ks1IOSgsuH3A8NUZ5MVL2ZDYTqSVEsdCCVYuQkay0UiMIR
gyttyYFzY0mk51YnVVTU1L69aaM4DJopKCE/pdI3KCJdtamp0QFYvuDLk5ob2Hnw6ZBuNX1D
BVbP+/I33GxtDKJ8w0E5t7XIkiWLqJ2h5dnGpapWJ4lzlmWFalfW0tN1uWn0KkCZcvJqV+St
Fov/hsiu9Hl8EW0pFYLtqqibrtZDVoIK7/x1+/5G1FQ+nIBd3fvp23my+ng9vP2+nTz+PLyf
qdfC1W2ZVFtyen6WS1faZZWIuOeD15KGLdP1Uhn7Baj1ID8JgjJ68u5hecMjFqL0a7K/mv/p
WF54gY3LcCqnZXwyT+vowghsudKadUx6NfgQy6a2cuWukIVhGUEOSG71nDKQQ9uhMgntgCaH
RCa5O3U8g87yMuN1TwvHsqCGRn6SoYwcNxD4s9F6PUfgAge9RkpWPktC8vCt4o5RyJhF+FzS
07nYl5OOwnsGCLBYMyLL2g7oLC+WENIh27CBHniW2Utx44RYdFAA0hhCxT2iuQVACYYqPjUL
yMnq81ZHznPXUa9wW/oi87GFaNfZsI2khe3saR+KCluaVhD+58KEgmGZOtZVZHw+Cvh+v8Tx
4LrJWkbBSNyg7uPxte1Qz/wtvuYszZ45tm8Z/dViBQ3kyJE9BuwgJkq7hoDDc/BQcHlm8PnJ
qBvYAY4ZuQjkVJk4eZOadRAvyNeuwV775GqUji53oeP7+Mm7b3r+T+f/hEYZZGxbrjnNFdi3
zK5RYdssrwoHxEI3wMHOnAcD7FguOfAHhjEvewana5PHJ5PPt8y9Q4F3+DmwZwCXNmngWJRR
FGaa7lyqzgILZeSrkexn9sjp3GC7vCDEcC5L7Sl5EaMzOeYCO2AuWdgOpQyhdaaAGlxbOeKJ
2YT2SHLUK3ukJoISe6TmKIRmTB2H7JQeHvF92a2foNsZdTX6bINEgdn7/QIrXnTk2zUTjWjh
o3QLL7kgtSrjCxXk4v/OI1KmUSnXp0vVitm1iOHNO+Ji9b9URjPrLFdgKb4BDdjxskZCK0xs
7Ub79BhRlxaLL674kokv6xdEzo4nZsRX8sSzLokqeQLtRW1kgY8d+KrISIB7hSUgX34Uhqll
rjX9PkiNtbXYbWJyu5cYfbDqpcbYdyyiQnXgXBA/cqThPuxsZl/DdkdkL3bBmnp56MaY/Ite
N4k15ZLMbc5LhuJ+a81LAQ0ttFTFpkmxg8WqySDqu350THnjv5/vHo8vj7pGJLu/Pzwd3k7P
hzO64WJxCvEYLHcoUEvypOZQZ0OG08s8X+6eTo8QXv7h+Hg8Q0z50wv/qB6HkMXTkBQvOeCE
cjx0n7mUpfrRDv7r+PvD8e0gfWCMfR4il2gW7/h7n+Ums7t7vbvnbC/3h9E6DxWzVRtm/nvq
BWp7fp6ZvEkUpeF/JFx/vJy/H96PWv1mIfmsIABPbd7R7GSMnsP5n6e3H6JRPv51ePvHJH1+
PTyIMkZkLf1Z66epzf/fzKEdkSIi0OHl8Pb4MRHjCsZtGuG6JdPQ9+i+G81Avvcf3k9PoPb0
aV85te3YaLR/lrZXxiYmnHaRIu1Uu4t09vLwdjo+IOvMlqSnE/vnMCu7K+hedaSj1/tFuWRw
L4rUp9dpfVvXJSPDmMpLH3GZWqkOAztgkVb5DauQAUyHaYrpGqrpEfXkYqmWbiAXJWgfXcjQ
iMzXARW7uZBsm86rVkvQSDmv0niZxPtyRd+el6kWD1NaFN+9/zicFftarb+WrL5Kmv2iYrlw
3Tc0QsfBymTXntTUwaZlPJRjl2YQbgi6Y0HrqS7SJIuhSrQqyhXfmSx122oJmt13R9V8RXVk
zVGTgcc5LT5dZ0vazFZ4J22jzF8wUC9zqdJEzIEyLdHYFM6hqqLeJ9Q7SrTigzzpP4kFQYHx
lBkrNRM9k6eEcAl0nXqeZp7TzUWUD2NXc2FaN6jjkax5kmVsXez66hAVXoHrqChT1Pv5D97Q
MN9Q9MuOcV9WSclUJ3JSZVLLpKcRRs1AXdUxNZWVdDnbzTzsxV9BheLO5Qzq1Hc9fAZXId8e
yZqDZGg4zOJ5YzlPLRKJ4iiZWsEoNnN8GqvlPCxJdBvRqaQvLnGRpPbsPlvm+2ipvLKsbvhc
XmeFsPCQ++nT6f7HpD79fKN8zQpNVOxOWlDKqpgn6FvJttmnXJ5W5EXxc99+bOCcZ7HOyak1
+LRAFRBWHRAqkM/qJvDmqrxClrpPyNJsXqDDbb+q5KsNvbRH1AIBliEV2+eQ27OWvdTjU30n
8/7YjPqdqA7Pp/Ph9e10T7wmJ2Az16oTGjQ+XpKtWnkiK/mJ1+f3RyL3Mq+VZxvxUzxcKi/s
giZ8GyxbU8kRBAg62r/vDSVEJem3O7DMv0mrwf/k6efLww1EaR58TUigiCa/1h/v58PzpHiZ
RN+Pr79N3kEH/dvxXrESkqLTM5fUObk+YSWbTowiYJmOZ3h4GE1motKjxdvp7uH+9DyWjsSl
FL0r/1i8HQ7v93dPh8n16S29HsvkM1apF/2/+W4sAwMT4PXPuydetNGyk/jQe+Bvt+u63fHp
+PK3ltEgnqTrHV+sNuqYoFL0hpX/Vn8rs1WIB4sqoR7Qk10TDTpPyd9nLqC3A8w0MJPMwiH4
FwjqrsznDhLRfEe/s1/UjO9cFpFy1IqoxVubTvB9PqP9rrWMlNM1gsd1SZebLUPZrH109mzp
VQMOzxhR/jr3fYs6P7Z4ZwSpLrQ5X7ZIpeBUXdohPuB8s1gIH48GbR/NSTKXJ8foyZoL0gmJ
gjnd4FBSwa9Aet5LPRqF3Grig7zVlXBY4kGhQ/x3QQmmSnKcZ1cALikKEwTJ4uCM6xvC9Y3O
0aY1tpfPrnGQ+NMRZ9SWF+8yd+pgdkHSXf5qKA7AnTNbVRXmvx1HOW/w356FcU+eKAwacvbK
ZWg+hoVRREZT9TwUBOUUM0ctYMxcG7URH1RVbFGXUxKZGcwj7jQVq3NZDJf2HSQ6uOl44HhH
H652dUz129Uu+nJlWziATR652jNDB+Rs6vmq73xJ0LzAt0TdZS0nB7Qj8JyFnmrpyAkz37e1
c2VL1QmqY+FdxPveR4TA8XGUr4i5Y68IdXPFzyFkTEaOzJn/395h9oOfnwGWOYMQLQ264mfx
1JrZFbUgwyWgqsgBv1XTZLj9DAL8e2Yjfkfnn4UI96YB+h1YgVY4TtmnC3C3y893LMsSyugT
8WmjAK4ug5HL22kQ7nGBp6F26zmzUQUgAhHOPCSjLnJgphrBwm9vhler6WxGvW+weOYFU3Sd
LYI2MxwAow1ZoTmUV8AwxBGHowi8Ldo48q9QK8V8yXqbZEWZ9BHu1a+u0tBzaf381W468oAr
w3ON+r6HmH4e6aBWIKEyvQRhpow5SVD9+XMZxHKU5pPxXtVFXFJCtFRwkuNRRQDE1cJfsN0s
GKkqxDp1yGcrQDzsaBBIM1I1J0/W+6+23oMy0sVeczG/ZhuIWEnkIo5DW5AZI82AtHfpuU+1
3AZkSw+tgYHjapjFWEineRH3/rr7pQwc8aOaNCK1Fdro2x2VfAzoQK9GsWQl2XZsNzSIVljb
qpJXxxvWKCBKSw5seMUzysOzsOkBL+HpjLSUlmDoep6ZI0RyvJCjNCAfZZChBujO4XiTRZ7v
KavWdhHYFm7+9uSz6/r+P328WrydXs6T5OVB2WJALqgSvtm12uE4TyVFexh+feKHJm2TCt0A
dcAqjzxHa/z+uNxnIO+Hvh+ehZcSqQ6vZttkjAu0K8KLjoSSr0WLES06z5MAS4jwW5feBA3J
bVFUh6qeZsqusXBR5vXUsnA8uyh25d02LSyAj7MqhRPPsiTFpbqsVbXO7ddwtkOPdnojSSOC
40NnRADvQRE/WZ9esBPAVjCUZwS8lGhwd2hQvkrnr46bvB5CNjqDa8K67NL1ZcJiaF226VYb
2n+NmUX7KimHNx/pd3J8jr28+lZA3bqCm33smQIoIS3kcYj2CQ+AhwQg/nuGfvszB+zb60SV
BwRVI7gVTmd5iCFwvArHVAJiiD/Of+tvKECdBSOxLjiInLuL3yH6xDSwtd8elp/86dSi5h0g
muzl4kf+MFSN8+KyAK+pimgT156niq9cVIAov1h2AD/Qg6gQOC5WyeNbvU/HSOEAClHK93Jv
6iDBH0gz0t62kZrGoQOOQtBGxMm+P7W1PYNTp7RD8xYMbKS1LJd8zT5eeUS/MPp7PYyHn8/P
H+0VmHr5ZmCtR97D//08vNx/9G/y/wLfG3Fc/1FmWXcFKq/Dl/DifXc+vf0RH9/Pb8e/foK6
groPQJxx8xp9JJ00Gfx+9374PeNsh4dJdjq9Tn7l3/1t8q0v17tSLjzHF1yipVZTgbR90Rbk
P/3M4Hn4YvOgRenx4+30fn96PfCy6FuZuBCxVFMkSUJWxx0p0ElOYOHJt6tqj6z6PF/aAboH
gd/6ridoaFFZ7FjtcGHbiSgaTq/QUR7KhrK8rYq9qwYmKzeupV4OtoQ2Z/2SQqbX7yg6nmbp
OhY6YI93gdwoD3dP5++KhNFR386T6u58mOSnl+MZ99gi8TwLnxkFidYrh9tRi3bw30IO2ln/
n7IjW24jx73vV7jmabcqM6XTlh/mgeqmpI77Sh+S7Jcux9YkqomP8lGb7NcvQPYBkqCSeUjF
AtA8QRAkcXBVEyRtrW7r+8Px/vj2g2GtZDIdG+Ir3FSsyNmgZj1ywmf2kQyTKPSFANlU5WTC
H5w2Vc3ukWV0YVyw4O+JMWlOn9pIhSDQMBTQw+H29f3l8HAA/fMdxshZTsYFXwuyF4oCXvCX
JAq3MBZLND43VMaIWTxRt3jIgsjKxQU1d+gg9sJpocayuUr251TfTLdNFCQzWPMjHmoWamCM
ZiEGltm5WmbUws9AWOuPoPjr2HaFxmVyHpYkA5UJt9po4ozu299NjZPNCW6gBeBkmiFiKHS4
aNeRk1T872EhDeyC6ZtFzIbVDz/CSpmOjUunGm8sTGUynvLp3gABMos4GYk8LC+nBgcj5NLg
v834wky+iBA2xVGQTCfjBTVPSNDl37BJgPPnhHPjBMS5GSMEIeeeLPPrfCLykSfegUZCR0ej
U7m3ozKeXI649PMaQyMFKsiYWjPQC++4ZOF5QR/UP5ZiPBlTf/e8GM25FO1OMMCqMBwv4i1M
7ywglYJ0n82sZCgtjA+Qk2bCGykjy9GknpOmOfRABTqkjY7GY9pY/D0zb7Crq+mU5UdYcfU2
Kumw9iBz7Q5gQ7hUQTmd0XiMCkBTUfVp72ECrVA7CuRJYYS4iwuevQA3m0+58anL+XhBk89v
gzSeGe4JGjI1Bmgrk/h8xB7JNYpa32zj87H5DnsDEwbTMma1dVPMaA/b2y+Phzf9DMDs5FeL
ywt6bY+/jeaKq9HlpecCs32JSsQ69RrOURpeugMKhJzRSbKs8ENZZYnEkM1TzhozSYLpvLPg
NiW8qtWn03XMskmC+WI2dbmoRVi8aSENDu2QRTI1bpFNuL0BWlhfFlJ2Jv/V59h+/nb4bucq
x+uSes+XRr9ptaC7b8dHH6fQq5s0iKO0nxJWGupX36bIqi6uP9lfmXpUC7rYgme/o6Hx4z2c
Nx8PdofQwqAo6rzi3o3p9GOEM/J03dfP19Ju04+gJasQN7ePX96/wd/PT69HZSFPN+9+uf2c
3DiuPT+9gTJxZF605xPzfTpEh142D6zYz2fUTVEBFoYzrwZ50nwG+WzEJ/EFzHhKM/UCAESf
fU0x9mWXje0jh6fb7JDAVLwZzBsn+eXYeQv1lKy/1qf7l8Mrqm2MsFvmo/NRYjiXLJN8wmo3
YbwBWWw4z4Z5ye9shhZgJgDJzVvbKMjHvjNbHo+NzO/qt/V4rWH223UegwBlM9OW83OqP+rf
VpkaZuafBdj0ghGoqnvc5M9nZkc3+WR0zu8IN7kANZF3UXEmb9CeH9HvwJ3Tcno5ndPl7RK3
bPH0/fiARz9crvfHV+2O4hSodD9TAYtCUWA4edls6bpbji1tN49SPnZ8sUKXGI/+WhYrzwG/
3F9OPQYYgJqzXISlGSHDUAXxBEzaxvNpPNr321E/BydH6h/7m1waN1DofzL6J/4nemc4PDzj
fR67rvFu9pIG9wP5FyUNJvVIsiCr81iyN0aVTIzsaEm8vxyds5bTGmVOd5XAmYQzFlAI8sxc
wVZEGUr9nhiiBS9sxos5vyi4zhPlvuKCCmwT2ehQJ2r84OfZ8uV4/4UxF0TSQFyOg/2M3G8j
tAL1frYwYStxJbtHF1Xq0+3LPZdSfZtESA9nR0My9R861ovkS6lCJPNGzTs3FRwGHbr7enxm
clUUn9DamjBg3KwimvBPhGgNrYMSDSqKXWBfXi6Cq2ZppvZQzkuw/2HUCM9ylUUENUd5FlSC
T68HglVWykO6yOKYsYZDB57y/fOrsigdetgGQkL/HuI7MgCbJMoj2LoUemhykDRXmHa8LpcT
v28QfN5GOmuqrChkymXtoVSqnge+hFLEW952FKlWJYxPsl8kn9wEE4QsifYyHvrkaU6+F81k
kSbNpqSTbaCw54Y5PTZVWbicrF/k+SZLZZOEyfk5K4KRLAtknFU476Es7QHRNrJBliz9w6Fp
ZGL713Qy2mCGvmZ0H8Mwc9S8M4wllPZRBtzcJQHJJZNox24TEOf9G2t+eMFIi2o3eNBXyEag
pq5tJ8j62xDTQx1+2qkv+0P2ctbVT/0Ju+WchkXmSR7T+xoOChybeakLnEx/9vGRBzGrwWjL
UobClUGb3dnby+2d0jtsKVRWRsY2+Im3RRWG8gL+ZK9KOgoMI01cDBER1klybYLKrC7aJPMZ
zaFHcDQuOrl6RS6rNuz4MT3qb5XztaAyVXmU5HAyzDurSHI9aSGV3wrTZyyzSdZF/0VpPRdZ
+GCbM8jWWoX/Ek6kM/sRqsMlItjss4l1y45Y7UHJ9GlVSHkjWzzTo7YteaES16IaUlhFF3Jt
JL3LVha8r1KBwxW/d6xK7l6lkr1TCvzJ+UZQcL/mMdoFNHWvGmtfL3ARX5MaLYzWF5cTLuhA
iy3HM1MvRbgn6Qii2gBC3HUFl1VIOQA32wjUOit5UbcSoow8HOAv3MUdU/8yjhK+AHXtAH+n
IEgNn0o3XMfQx6ys2KVl6T76Vfz4DdQ7JdCJchYGwJey2WFyOh0233i7F3g4gYPJqkST15K9
DEFcVkYwDQFJ3Cz36NJlyrgO1izRwa3Jcm4gMKSncoAzAuclIInRku/axhMmbUANKK5zvA/i
m7kFpaUiakwP6hM10dJa1LKOgGNTtFlORVUXrHvtqtQBRInFhw2INEAn5xiaIGy6T3VmWkUr
AEaeVL5kik/QtpjTDArAtvQ7UaTWCGmEL9KgxlYgc+gy+rRKqmbLXU9rDDmwqgKCivCAqKts
Vc5gdMgNsII1NDnAqsYspTTeVk1tjNowk5Qgg7mJxbUHhkkiowLWUQP/nSYQ8U7Awl6BSpzt
WNIoDaXhFklwe5hT1SFmeAhZImFcsvy6k5bB7d1XK0d0qZYhu5Zbaq0hvR7e75/O/oKl7Kxk
5fZorjcFurJt+CgSpJoxYwqYCwwZm6WRYTmr3So3URyCnm5/gRkAMfdcm55oeAiWRUonyUoL
ASdZs8UKMIgT/sVE0exFVXHSSGNhYkJpmndt6jUsoSXrhASql3J6B/WFGrd12fTW0VqkVaRH
hrCp+q9j5kE/dSeprwdDbqIA07GAKfcWGKhXlzVYvytxpkGDEO2AbeRe61Kopfq4WpUTo7AO
0sq6kQPfgbiTtofZgMWYoigSV8ZBT+NLUBlFwR/y+hKcGbNI4MCirv/QqDRTUpybKk17YzyM
a1iBbtZkdgqRmCOnIZ4kP0WWWKOvIRhAAP3arvG7oXSNRB9CCtUxDezfGJY1xl2066FDEN9k
FDks4A4969HcSu6pNsFQzIOFXswmp+q4KavwFyo5UYLdyy4aLX8GdTvO0Z/qYUfPNIX29efF
OgX+9u1/s693vznFtuefUx1C525/TcCAxplPVhjchEoE7vwYE4aBH0Mzj69Pi8X88vcxaSoS
BCD8lBCfTTl7UYPkYkpuEk3Mxdyst8csTAsOC8cZmVokRlwxC/fTFi9MgygLx78kW0Q/b+L5
1Nd3GhXPwnjH6/zcM8aL80tvXy6n3OWvSULND62PJ94pupxxzpBmu8wcRIiDsw8yW8M97Rnf
jic0wqyNGtv9VbHrPWV2dY7N8jrwxC6rQ3APmxQ/833IPXJRvDWPHfjCV55voPuOTX1fss8E
BoHFbVdZtGgKs30KVtszmYgA9y82f32HDySmXTVL03A4f9RFZtatMEUmKkyFzNQWXBdRHLNX
UR3JWsjYjLffY+BIwkXC6fARtBVOh26LorSOKm/no5P9h4PeVUSzFiCirlbGHUMYcwGD6jRC
hh8+bQFNmhUJHKhvlMVCn0+Cqo7GGV37nRzu3l/wrczJf3Elrw1F+xqPNZ9qiZHx8TBhbIuy
KCPYX+B0CIRwSlx7EjBj+nLQdrA0pmft+boloCMBv5twA2qbLISjuRlU6rgbBSeoShnU+lAO
Cq56uKiKKODViI72JJJV+ZU8qsQSb69hU9dWJGYsp40oQplCb2uVLyK/bjAzQiBsN1SbjL9D
ywp1/NfXpXx70ZYlUMUkwC8bGefshUsX5X8YKOpqEpcJKDBPd3/fP/338cOP24fbD9+ebu+f
j48fXm//OkA5x/sPmLvvC/LVh8/Pf/2mWe3q8PJ4+Hb29fbl/qDetweW+9eQhPrs+HhE69Xj
/25NT4kIzovYheAK5tiIK4EIDAqCg0dzVdJXhJYGrzwJCW9ZxLejQ/u70Tti2WtqODIAd2f9
Mf3lx/Pb09nd08vh7Onl7Ovh27PyPjGIoVdrQS98DfDEhUsRskCXtLwKonxDD2MWwv1kYyZa
GYAuaWGkpehhLCHRtK2Ge1sifI2/ynOXGoBuCaiWu6QgucWaKbeFux+0yXBZ6iaMSiUCVGIf
59P1ajxZJHXsfJ7WcexQI9CtXv3HTHldbUCcOvA2QJY14VHilrCOa3x1QRmBQQAdfB/eRd8c
vX/+drz7/e/Dj7M7xddfXm6fv/5w2LkwkmFoWOjylAwCp5UyCDfmcbsFF6EvC0A7RHWxlZP5
fMxpSw6N6mz3Yvf+9hXts+5u3w73Z/JRdQ2t4/57fPt6Jl5fn+6OChXevt06fQ2CxB3VIHE6
FmxgPxWTUZ7F12jnzKzfdYSp3ox7exMFf5Rp1JSl5K10upmWnyIu+GQ/lhsBonLbzepS+aI9
PN3TGJpdq5cB055gxaaIaJFVwX3CprzqW7R0hiMuds4YZqslwxs5NNJf9p5ZkqBC7Aqa6Kxb
fJt+dtxqBuRPJ4CQiu2eOyd2E4tph6ra5Ra8weonaHP7+tU3P6BmOh9vEuHKhL2eShO41ZSd
rePh9c2toQimE/dLDdZvszySYwKEY5odEIcnZmyvdiB3BpaxuJIT3uzBIOFVQpMEJcAJJi6C
ajwKoxXfC437aU/WbUcsfvVJgJ5pMByrkYej3W9CDjZ3YRGscGWA4vJGkYRaxrhg80ZkQEzm
JwYK8FMj5UQrgjZi7FSCQFg7pZxyKKimRzoSbSPm44lG+9uiC+HKno8ZzWgjpm67E6Zt+Jy1
zNYMR1brYnx5UhTs8vn4JIHikUYxEoaWVwvKsdsIjs9fDXO4fmNwhRvAMFyg3TMEd+UzPRFp
vfQEvuooioAN19qtqmy3ihjFsUN0YRbcye0pNNufWJQCQ+1Gwt1bW0S3cBx51OH1/gky+dcp
J35SPCR3nXJx7rpUUFo7R+Dyr4KearS23nJg00aGcvjGHvaV+v/UjF9txA2fWaldGCIuBQ0u
Z6k6nOxsUcxcu1qMPFW3LHKZMlquhqstehhmt2xNNQzq6Zb01JOfMmmZcBVWks110SJ3WZvK
lIX7mKxDe7jCRDfTnbhmJqSj4kdCC5+nh2e0eDcO6j2bqecaV3u7yRzYYsbpBPHNicFUbzXO
uOArU6e8F7eP908PZ+n7w+fDSxdQgWupSMuoCXLuxBoWy7WVFJNiWnXKbrjGgfA+xTqKKGDf
mAmFU+/HqKokWrIW+sXfPYw2wgxrYqF+2rCesLsH8LewJy3StTMXPVLdSTirsRKM+q12vChd
2Zck346fX25ffpy9PL2/HR8ZbRddlrktT7kyt2pdl3P1BA2L05Lp5OeaxN1Z9Qv/Vmqi/ozp
EJpoLjssS3hieQBd6BmPXpksMIHnn+PxyVaTc8+Jok4NzskSnMMvR+RR3zY7ZsPbNrkIVYBz
d/QIFrnlxNIjhOVGeIoSVYJGzoEvT4BFiL0YzU5IeyQNgpyTKBrThCf2PaT5JNxLtBbehJvF
5fw7c7HSEQRtSjwP9nyy94wDLZ1Ns8tVtF15W4pVbVeeytIIRN++CdJ0bqXVcmntLMcEVYqV
3BsRa42h1mZqzEwmcbaOgma9578k+N7or6US5XWSSHxrUA8V1XUuWWReL+OWpqyXJtl+Prps
Alm0bxyytZokBh9XQblAQ70tYrEMjuKiy8rtweItIX5MORHtEzEPi9QWkspUp31ncfUCjEDx
l7o2ez37C+3nj18etQPQ3dfD3d/Hxy/EulxZJtC3ocIwzXTxpZFMvMXLfYVW1cPYcFa0Ev4I
RXH909pgP8DEHmX1CxRq08K/dLM6u7pfGIOuyGWUYqOUfeXqzz7mhm/Pi6NUiqJRJl2mpYxQ
ZqtMz5ewaiRmHSXM1HnZwCk2DfLrZlVkiWU+SklimXqwqayauoqoHUmHWkVpiBmKYJigCYZk
y4qQfbGFUUhkk9bJEpOkDuZSitVE7NaB+RijLBG5i7LAamdDa9AgyffBZq3sfAu5sijQJm+F
Z7jWmDyine7LgBULqmPaeogbW20AEgT0NANkJFcFiv4SicCiqm7Mr6YT66f5rmtiQGzI5bUn
Yygl4fVqRSCKndbarS9h9nzles48gXGKDUiYWtjS3bvDgFw9tfd8gzWcSMMsMTvfoiyzMwJF
pwUbjnZ9qF+aJ5MbrS1ZUMtojkC5kn3WcWhbx7aE2sMN5ArM0e9vEEznRUPsy0obrXyqcm6d
tQSRMO8AWrBgkxUNyGoDC5T5DrOYnahtGXy0+9SYEzp0vlnfUNdHglgCYsJi4huakIAg9jce
+swDn7FwdcB0xAx9t+9YVsLuUmZxZpzPKRTNHRYeFNR4AkUT1C8Dsk5EWWZBBPIIdHdRFIIc
EFGmgTSknmIahC4TjSElEW7kdUhV/SqkfgO7wLraWDhEQBHqUGcbOCNOhGHRVM35TO8B3bCq
GO1BLJTp40adajlJnKH/FxLXaW8hQrbjXZRV8dIstisO2D+LKZcqpCcHmmopOmZ6PBjKdaxn
moitjcQTQue6QRrxiW5UcWYsFfzdizLWSAhNmUkt8Q2aktAiouITnqu4F4YkN3Ojwo9VSAY2
i0LloAV7usEfwDMdP2/DMnO5fC0rNIHOVqFgfHXxG5WJqqF74SrDGzedf8mCLr5TPlYg9HWA
cdEuSjYf5OjAaNhC9KhauxY1q7guN511ko8oCVD3twiUYclO0AxqChTKPKsoDHVB1rbKUddM
o5dO6VXQ55fj49vfOj7Bw+H1i2t9pVTBq6Y1OSceFwocCNvFuleqlAUxpnCMQd2Le8uKCy/F
pzqS1Z+znnvag4FTQk+BiSi7hoQyNu8OMZN1EgXeNWTgtbMlPWNcJ8sMDz+yKICOU+H1h/AP
lNllVhrxt73D2l9WHr8dfn87PrQq+KsivdPwF3cSdF3tdZQDQy+fOlCiaOj+gO22DMmnFiGU
JWiYvHJFiMKdKFb8XfQ6BGkSFFHOPqTLVFmgJDW+SaC4ImsO018qT64/J6PZglqKQWmwm6Af
ccI//RRShKpgoGIJNhJDE6CrDCwkVlDp3sGZTNkrJlGZiIruZjZGtbTJ0vjang69RazqVH8g
4gjjZk2WdlfzTLm3WWu8c0M0LPRowTsprlSWoCA30mf9Mkcp/lM30Me7TiSEh8/vX1Qy2ujx
9e3lHcM0Et5LBF4jwFmzIMnGCbC3YtPz++fo+3gYe0qnYzd4h9/0j+pgap/bNafmDT19olLT
JehEeqIcNNhjClJ7jhK/V8DD9Hv8zXwwyPJlKVI4rKRRFd1IrIV+rbCsbd8vzYHZT3Qxk7Hb
OfS1ci4+WtvBvlwi0FGoyn2Fce3Nt05dHOKVcuGzJc12qZn4SkGBo8ss9QXuGYqG9bo6QVJk
wP+i8egj/bBr4t3eXiYU0p/Kq7BOiFapfzsivwWrclhDWl1DtsQIEKVdcQtmD8UmBVp+nhiA
jkyl0uQlnkmIjjU/a21TBLWSg/5mgVQBocJ5YbPkrTDv9uSxXWwZC27dqIXWsjNoUTHIM7dJ
HeZE57W4rFFB4PRj2F7Clkamob3bWFy0TZp8rQyk7UndJi5E2SOZil2PKpZuZ1TpcKBf+1mK
a4DdxqioasEs/hbhLVun3FNmx5aKrcWdAAHlRWBfrWNGoBqlse5Ti8YiR6JymmaDXIRzl+VM
p8o4Zfc8yC5n9jcYV8gWeYr+LHt6fv1whsHi35/1Jri5ffxiOCHn0KoAjbAz3jHfwOOeXMvB
i1Qj1fGjrgYw3t/VOc3e1PFitqpcZN8W1GExx1ZCCVUd3N2pl9huJZrqW7XiCl/Rqe4p9NkR
uwRLJslZmtNtJ4Q/b7tN3LedzDJW1mxqYKJKlLwg2H0CNQyUsTDj1Hv1cKBrMQIpnuQR7YgC
+tP9OypNzOapJYv1nqKBpnauYN3T8mCQz5RtrlechSspc33Rr6/f0fR1UBD+/fp8fERzWOjC
w/vb4fsB/ji83f3xxx//ITfz+GCoilRJ5J2Db15kWza6hEYUYqeLSGEcfZu6fpSs2CA07W6O
N9eV3EtnvySZpk15xZPvdhoD+0q2ywW992lr2pWGm7mG6vdVU4QhDA7TrihtEd7OYFZ5VGZj
6fsaR1rZK7RKACdcVJNgIeEtjZMleugmcyczHPD/AUP066HALH0gNtVWNIyGCW/ShNxqqA1U
EQwwdfKBEW3qFI2fgO/1lTizi2tVwSOm/9Z67/3t2+0ZKrx3+B5lSOl2WCNWEfx/Y9fS2zYM
g/9ShxVFd9jBr9Ra7NiV7cQ9GTsEOw0DtmHozx8fsi1RordTEJFSZEXi4yNFO3uSYl1yZ73E
c+G7WJo9RXbNZSHDE6xCLIlrlCs5h5MP51GAV15dRpM1W/FAsMKSFjkfOfdGZNmoLYHYQ6s3
DB1Q0CeatU2HNDDKF3qLWGR5B2y0GRJzQVr1OmyCcS9JGTyx/FtAhLNfa8mjTZ2WDPyU4m3s
PK1E6Tz73oxl26Xreab2c2jZbH75MfXFZn2d5llRo5M4FgnicjNjjfjm8B9srkoLwmmS3bG1
ZJnDeBiuFCxY3ATPJHEStBANgnlaEmQt3Gg8tJQJRSieCW6URTOqK0LbyB8EpuEDhNiIiDii
JXIlvaGcWz7cfNi/Bw+phTNoX9NPFP2ea/B2wn4bMdqywQkzJfizdWE+fPz0SLA7GsopjwLs
oiYMcXPTkk1zaYa+yVIlFB0PLygtlhdhC4gMYypEjqBImhOwnrzm9vq25BbcJ1q2oFye64ev
tk7fy2QGi7Ud4P836TKVjou/hYVHHOnALvM8jCJwPTyHhwoEGgfZEKxJ0vP9+SklPYUGi6RB
rOFiniqzzdsKJU+DB9dhvqkDeMmEnfp0L2WsMn9ROtAL0ucyvLvkbMAmp/iBtnxY0k0RfThd
DBKWKCKjELXpGCdfHubnIPPZIygo8cYx0ccxj4KyOcScUHv0BILjVPR6xS7uuIowqdtbo5hL
wZoQDBjqmF0STHgrF+27eAo7lne5YYksu3TJsPJGlgDvpgvD/esHZcb7r99oyaEnUvz4c//5
9dvdu/+Os9tPP31dYar9v+Vmqdu5tZqdNNGWiNlIhUizdeNZbSgMidC7C74wKp5YC8aDN45A
HmemkdhQQGSQMAIgQ54TGsn//mEfWZZi5lx01wi5AIkEzatI9pyGkBu/rQgYRlcyi9jnIBgw
3mCnlrLy/Vu0TATlloGUXijz9eEdX3eyee8W1DHGL0d2vta8793xOpdjOtbBbi/mgg0gF3SW
1lwQD+x1DrV/vhtecOoO7MUc7yQd0P1cApWLTtE1a5bjwRx6qdLZd3t6PJYU9OB1NSMQfLAy
HHrlG8nJGv6Oayj6oIYzJzMCYexmfXjOvdOGzc3YZn00KjTD2WzSsps4pklW9/WpM5kYOh2L
/J1AG+scFhOlCMQ8WFot4Z+opkxnQ/CePqeygNZn7/ohWhKHXWq9yIugYoXfw9H6k2zB/Mq6
I7T7GpSSxZRC+PUlBx+lbjObBqpokJOxLfjOKbXI22WtT+hnk0KLJ9dT7hHlhfqSfyN4iZmC
BjPemsQeiXR7uKOpaoesiMLypGqLDPbu0amhlFAlvrwOopjfQJFZDoeqMyqRwUkPfwGAjR6J
jPQBAA==

--UlsYxwg8UDQn+EKZ--

