BCACHE stability patches

BCACHE stability patches

[Denis Bychkov]

Hi,

There is a set of bcache stability patches elevating bcache stability
to production level. As far as I know, there is no single reported and
peer confirmed bug that is not solved by this set. Unfortunately, for
some reason, Kent does not have enough time and/or energy to review
them and send them upstream. Let's come up with a solution that would
allow to review all these patches (some of them written by Ken
himself, some of them produced by the community), review them and hand
them to the maintainer who is willing to apply them upstream. Without
that, bcache is just another half-assed unstable and buggy cache
layer.
These patches will allow people to start use bcache in production systems.
Please find the patch set attached. (The patches apply cleanly to 4.3
and 4.4 kernel series).

---
From: Zheng Liu <wenqing.lz@taobao.com>

In bcache_init() function it forgot to unregister reboot notifier if
bcache fails to unregister a block device.  This commit fixes this.

Signed-off-by: Zheng Liu <wenqing.lz@taobao.com>
Tested-by: Joshua Schmid <jschmid@suse.com>
---
 drivers/md/bcache/super.c | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)

--- a/drivers/md/bcache/super.c
+++ b/drivers/md/bcache/super.c
@@ -2066,8 +2066,10 @@ static int __init bcache_init(void)
  closure_debug_init();

  bcache_major = register_blkdev(0, "bcache");
- if (bcache_major < 0)
+ if (bcache_major < 0) {
+ unregister_reboot_notifier(&reboot);
  return bcache_major;
+ }

  if (!(bcache_wq = create_workqueue("bcache")) ||
     !(bcache_kobj = kobject_create_and_add("bcache", fs_kobj)) ||
From: Zheng Liu <gnehzuil.liu@gmail.com>
To: linux-bcache@vger.kernel.org
Cc: Zheng Liu <wenqing.lz@taobao.com>, Joshua Schmid
<jschmid@suse.com>, Zhu Yanhai <zhu.yanhai@gmail.com>, Kent Overstreet
<kmo@daterainc.com>
Subject: [PATCH v2] bcache: fix a livelock in btree lock
Date: Wed, 25 Feb 2015 20:32:09 +0800 (02/25/2015 04:32:09 AM)
From: Zheng Liu <wenqing.lz@taobao.com>

This commit tries to fix a livelock in bcache.  This livelock might
happen when we causes a huge number of cache misses simultaneously.

When we get a cache miss, bcache will execute the following path.

->cached_dev_make_request()
  ->cached_dev_read()
    ->cached_lookup()
      ->bch->btree_map_keys()
        ->btree_root()  <------------------------
          ->bch_btree_map_keys_recurse()        |
            ->cache_lookup_fn()                 |
              ->cached_dev_cache_miss()         |
                ->bch_btree_insert_check_key() -|
                  [If btree->seq is not equal to seq + 1, we should return
                   EINTR and traverse btree again.]

In bch_btree_insert_check_key() function we first need to check upgrade
flag (op->lock == -1), and when this flag is true we need to release
read btree->lock and try to take write btree->lock.  During taking and
releasing this write lock, btree->seq will be monotone increased in
order to prevent other threads modify this in cache miss (see btree.h:74).
But if there are some cache misses caused by some requested, we could
meet a livelock because btree->seq is always changed by others.  Thus no
one can make progress.

This commit will try to take write btree->lock if it encounters a race
when we traverse btree.  Although it sacrifice the scalability but we
can ensure that only one can modify the btree.

Signed-off-by: Zheng Liu <wenqing.lz@taobao.com>
Tested-by: Joshua Schmid <jschmid@suse.com>
Cc: Joshua Schmid <jschmid@suse.com>
Cc: Zhu Yanhai <zhu.yanhai@gmail.com>
Cc: Kent Overstreet <kmo@daterainc.com>
---
changelog:
v2: fix a bug that stops all concurrency writes unconditionally.

 drivers/md/bcache/btree.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

--- a/drivers/md/bcache/btree.c
+++ b/drivers/md/bcache/btree.c
@@ -2162,8 +2162,10 @@ int bch_btree_insert_check_key(struct bt
  rw_lock(true, b, b->level);

  if (b->key.ptr[0] != btree_ptr ||
-    b->seq != seq + 1)
+                   b->seq != seq + 1) {
+                       op->lock = b->level;
  goto out;
+               }
  }

  SET_KEY_PTRS(check_key, 1);

From: Joshua Schmid <jschmid@suse.com>
Subject: [PATCH] fix a leak in bch_cached_dev_run()
Newsgroups: gmane.linux.kernel.bcache.devel
Date: 2015-02-03 11:24:06 GMT (3 weeks, 2 days, 11 hours and 43 minutes ago)

From: Al Viro <viro@ZenIV.linux.org.uk>

Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
Tested-by: Joshua Schmid <jschmid@suse.com>
---
 drivers/md/bcache/super.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

--- a/drivers/md/bcache/super.c
+++ b/drivers/md/bcache/super.c
@@ -847,8 +847,11 @@ void bch_cached_dev_run(struct cached_de
  buf[SB_LABEL_SIZE] = '\0';
  env[2] = kasprintf(GFP_KERNEL, "CACHED_LABEL=%s", buf);

- if (atomic_xchg(&dc->running, 1))
+ if (atomic_xchg(&dc->running, 1)) {
+ kfree(env[1]);
+ kfree(env[2]);
  return;
+ }

  if (!d->c &&
     BDEV_STATE(&dc->sb) != BDEV_STATE_NONE) {
From: Denis Bychkov <manover@gmail.com>

Allows to use register, not register_quiet in udev to avoid "device_busy" error.
The initial patch proposed at https://lkml.org/lkml/2013/8/26/549 by
Gabriel de Perthuis
<g2p.code@gmail.com> does not unlock the mutex and hangs the kernel.

See http://thread.gmane.org/gmane.linux.kernel.bcache.devel/2594 for
the discussion.
--- a/drivers/md/bcache/super.c
+++ b/drivers/md/bcache/super.c
@@ -1936,6 +1936,8 @@ static ssize_t register_bcache(struct ko
  else
  err = "device busy";
  mutex_unlock(&bch_register_lock);
+ if (attr == &ksysfs_register_quiet)
+ goto out;
  }
  goto err;
  }
@@ -1974,8 +1976,7 @@ out:
 err_close:
  blkdev_put(bdev, FMODE_READ|FMODE_WRITE|FMODE_EXCL);
 err:
- if (attr != &ksysfs_register_quiet)
- pr_info("error opening %s: %s", path, err);
+ pr_info("error opening %s: %s", path, err);
  ret = -EINVAL;
  goto out;
 }
From f0e6320a7874af434575f37a11ec6e4992cef790 Mon Sep 17 00:00:00 2001
From: Kent Overstreet <kmo@daterainc.com>
Date: Sat, 1 Nov 2014 13:44:47 -0700
Subject: [PATCH 1/5] bcache: Add a cond_resched() call to gc
Git-commit: f0e6320a7874af434575f37a11ec6e4992cef790
Patch-mainline: Submitted
References: bnc#910440

Change-id: Id4f18c533b80ddb40df94ed0bb5e2a236a4bc325
Signed-off-by: Takashi Iwai <tiwai@suse.de>

---
 drivers/md/bcache/btree.c | 1 +
  1 file changed, 1 insertion(+)

--- a/drivers/md/bcache/btree.c
+++ b/drivers/md/bcache/btree.c
@@ -1741,6 +1741,7 @@ static void bch_btree_gc(struct cache_se
  do {
  ret = btree_root(gc_root, c, &op, &writes, &stats);
  closure_sync(&writes);
+ cond_resched();

  if (ret && ret != -EAGAIN)
  pr_warn("gc failed!");
From: Joshua Schmid <jschmid@suse.com>
Subject: [PATCH] bcache: [BUG] clear BCACHE_DEV_UNLINK_DONE flag when
attaching a backing device
Newsgroups: gmane.linux.kernel.bcache.devel
Date: 2015-02-03 11:18:01 GMT (3 weeks, 2 days, 11 hours and 45 minutes ago)

From: Zheng Liu <wenqing.lz@taobao.com>

This bug can be reproduced by the following script:

  #!/bin/bash

  bcache_sysfs="/sys/fs/bcache"

  function clear_cache()
  {
  if [ ! -e $bcache_sysfs ]; then
  echo "no bcache sysfs"
  exit
  fi

  cset_uuid=$(ls -l $bcache_sysfs|head -n 2|tail -n 1|awk '{print $9}')
  sudo sh -c "echo $cset_uuid > /sys/block/sdb/sdb1/bcache/detach"
  sleep 5
  sudo sh -c "echo $cset_uuid > /sys/block/sdb/sdb1/bcache/attach"
  }

  for ((i=0;i<10;i++)); do
  clear_cache
  done

The warning messages look like below:
[  275.948611] ------------[ cut here ]------------
[  275.963840] WARNING: at fs/sysfs/dir.c:512
sysfs_add_one+0xb8/0xd0() (Tainted: P        W
---------------   )
[  275.979253] Hardware name: Tecal RH2285
[  275.994106] sysfs: cannot create duplicate filename
'/devices/pci0000:00/0000:00:09.0/0000:08:00.0/host4/target4:2:1/4:2:1:0/block/sdb/sdb1/bcache/cache'
[  276.024105] Modules linked in: bcache tcp_diag inet_diag
ipmi_devintf ipmi_si ipmi_msghandler
bonding 8021q garp stp llc ipv6 ext3 jbd loop sg iomemory_vsl(P) bnx2
microcode serio_raw i2c_i801
i2c_core iTCO_wdt iTCO_vendor_support i7core_edac edac_core shpchp
ext4 jbd2 mbcache megaraid_sas
pata_acpi ata_generic ata_piix dm_mod [last unloaded: scsi_wait_scan]
[  276.072643] Pid: 2765, comm: sh Tainted: P        W
---------------    2.6.32 #1
[  276.089315] Call Trace:
[  276.105801]  [<ffffffff81070fe7>] ? warn_slowpath_common+0x87/0xc0
[  276.122650]  [<ffffffff810710d6>] ? warn_slowpath_fmt+0x46/0x50
[  276.139361]  [<ffffffff81205c08>] ? sysfs_add_one+0xb8/0xd0
[  276.156012]  [<ffffffff8120609b>] ? sysfs_do_create_link+0x12b/0x170
[  276.172682]  [<ffffffff81206113>] ? sysfs_create_link+0x13/0x20
[  276.189282]  [<ffffffffa03bda21>] ? bcache_device_link+0xc1/0x110 [bcache]
[  276.205993]  [<ffffffffa03bfa08>] ?
bch_cached_dev_attach+0x478/0x4f0 [bcache]
[  276.222794]  [<ffffffffa03c4a17>] ? bch_cached_dev_store+0x627/0x780 [bcache]
[  276.239680]  [<ffffffff8116783a>] ? alloc_pages_current+0xaa/0x110
[  276.256594]  [<ffffffff81203b15>] ? sysfs_write_file+0xe5/0x170
[  276.273364]  [<ffffffff811887b8>] ? vfs_write+0xb8/0x1a0
[  276.290133]  [<ffffffff811890b1>] ? sys_write+0x51/0x90
[  276.306368]  [<ffffffff8100c072>] ? system_call_fastpath+0x16/0x1b
[  276.322301] ---[ end trace 9f5d4fcdd0c3edfb ]---
[  276.338241] ------------[ cut here ]------------
[  276.354109] WARNING: at /home/wenqing.lz/bcache/bcache/super.c:720
bcache_device_link+0xdf/0x110 [bcache]() (Tainted: P        W
---------------   )
[  276.386017] Hardware name: Tecal RH2285
[  276.401430] Couldn't create device <-> cache set symlinks
[  276.401759] Modules linked in: bcache tcp_diag inet_diag
ipmi_devintf ipmi_si ipmi_msghandler
bonding 8021q garp stp llc ipv6 ext3 jbd loop sg iomemory_vsl(P) bnx2
microcode serio_raw i2c_i801
i2c_core iTCO_wdt iTCO_vendor_support i7core_edac edac_core shpchp
ext4 jbd2 mbcache megaraid_sas
pata_acpi ata_generic ata_piix dm_mod [last unloaded: scsi_wait_scan]
[  276.465477] Pid: 2765, comm: sh Tainted: P        W
---------------    2.6.32 #1
[  276.482169] Call Trace:
[  276.498610]  [<ffffffff81070fe7>] ? warn_slowpath_common+0x87/0xc0
[  276.515405]  [<ffffffff810710d6>] ? warn_slowpath_fmt+0x46/0x50
[  276.532059]  [<ffffffffa03bda3f>] ? bcache_device_link+0xdf/0x110 [bcache]
[  276.548808]  [<ffffffffa03bfa08>] ?
bch_cached_dev_attach+0x478/0x4f0 [bcache]
[  276.565569]  [<ffffffffa03c4a17>] ? bch_cached_dev_store+0x627/0x780 [bcache]
[  276.582418]  [<ffffffff8116783a>] ? alloc_pages_current+0xaa/0x110
[  276.599341]  [<ffffffff81203b15>] ? sysfs_write_file+0xe5/0x170
[  276.616142]  [<ffffffff811887b8>] ? vfs_write+0xb8/0x1a0
[  276.632607]  [<ffffffff811890b1>] ? sys_write+0x51/0x90
[  276.648671]  [<ffffffff8100c072>] ? system_call_fastpath+0x16/0x1b
[  276.664756] ---[ end trace 9f5d4fcdd0c3edfc ]---

We forget to clear BCACHE_DEV_UNLINK_DONE flag in bcache_device_attach()
function when we attach a backing device first time.  After detaching this
backing device, this flag will be true and sysfs_remove_link() isn't called in
bcache_device_unlink().  Then when we attach this backing device again,
sysfs_create_link() will return EEXIST error in bcache_device_link().

So the fix is trival and we clear this flag in bcache_device_link().

Signed-off-by: Zheng Liu <wenqing.lz@taobao.com>
Tested-by: Joshua Schmid <jschmid@suse.com>
---
 drivers/md/bcache/super.c | 2 ++
 1 file changed, 2 insertions(+)

--- a/drivers/md/bcache/super.c
+++ b/drivers/md/bcache/super.c
@@ -685,6 +685,8 @@ static void bcache_device_link(struct bc
  WARN(sysfs_create_link(&d->kobj, &c->kobj, "cache") ||
      sysfs_create_link(&c->kobj, &d->kobj, d->name),
      "Couldn't create device <-> cache set symlinks");
+
+ clear_bit(BCACHE_DEV_UNLINK_DONE, &d->flags);
 }

 static void bcache_device_detach(struct bcache_device *d)
---
From e949c64fa6acbdaab999410250855a6a4fc6bad1 Mon Sep 17 00:00:00 2001
From: Stefan Bader <stefan.bader@canonical.com>
Date: Mon, 18 Aug 2014 20:00:13 +0200
Subject: [PATCH] bcache: prevent crash on changing writeback_running

commit a664d0f05a2e ("bcache: fix crash on shutdown in passthrough mode")

added a safeguard in the shutdown case. At least while not being
attached it is also possible to trigger a kernel bug by writing into
writeback_running. This change  adds the same check before trying to
wake up the thread for that case.

Signed-off-by: Stefan Bader <stefan.bader@canonical.com>
---
 drivers/md/bcache/writeback.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
--- a/drivers/md/bcache/writeback.h
+++ b/drivers/md/bcache/writeback.h
@@ -63,7 +63,8 @@ static inline bool should_writeback(stru

 static inline void bch_writeback_queue(struct cached_dev *dc)
 {
- wake_up_process(dc->writeback_thread);
+ if (!IS_ERR_OR_NULL(dc->writeback_thread))
+ wake_up_process(dc->writeback_thread);
 }

 static inline void bch_writeback_add(struct cached_dev *dc)
Subject: [PATCH] bcache: Change refill_dirty() to always scan entire
disk if necessary

Previously, it would only scan the entire disk if it was starting from the very
start of the disk - i.e. if the previous scan got to the end.

This was broken by refill_full_stripes(), which updates last_scanned so that
refill_dirty was never triggering the searched_from_start path.

But if we change refill_dirty() to always scan the entire disk if necessary,
regardless of what last_scanned was, the code gets cleaner and we fix that bug
too.

Signed-off-by: Kent Overstreet <kent.overstreet@gmail.com>
---
 drivers/md/bcache/writeback.c | 37 ++++++++++++++++++++++++++++++-------
 1 file changed, 30 insertions(+), 7 deletions(-)

--- a/drivers/md/bcache/writeback.c
+++ b/drivers/md/bcache/writeback.c
@@ -323,6 +323,10 @@ void bcache_dev_sectors_dirty_add(struct

 static bool dirty_pred(struct keybuf *buf, struct bkey *k)
 {
+ struct cached_dev *dc = container_of(buf, struct cached_dev, writeback_keys);
+
+ BUG_ON(KEY_INODE(k) != dc->disk.id);
+
  return KEY_DIRTY(k);
 }

@@ -372,11 +376,24 @@ next:
  }
 }

+/*
+ * Returns true if we scanned the entire disk
+ */
 static bool refill_dirty(struct cached_dev *dc)
 {
  struct keybuf *buf = &dc->writeback_keys;
+ struct bkey start = KEY(dc->disk.id, 0, 0);
  struct bkey end = KEY(dc->disk.id, MAX_KEY_OFFSET, 0);
- bool searched_from_start = false;
+ struct bkey start_pos;
+
+ /*
+ * make sure keybuf pos is inside the range for this disk - at bringup
+ * we might not be attached yet so this disk's inode nr isn't
+ * initialized then
+ */
+ if (bkey_cmp(&buf->last_scanned, &start) < 0 ||
+    bkey_cmp(&buf->last_scanned, &end) > 0)
+ buf->last_scanned = start;

  if (dc->partial_stripes_expensive) {
  refill_full_stripes(dc);
@@ -384,14 +401,20 @@ static bool refill_dirty(struct cached_d
  return false;
  }

- if (bkey_cmp(&buf->last_scanned, &end) >= 0) {
- buf->last_scanned = KEY(dc->disk.id, 0, 0);
- searched_from_start = true;
- }
-
+ start_pos = buf->last_scanned;
  bch_refill_keybuf(dc->disk.c, buf, &end, dirty_pred);

- return bkey_cmp(&buf->last_scanned, &end) >= 0 && searched_from_start;
+ if (bkey_cmp(&buf->last_scanned, &end) < 0)
+ return false;
+
+ /*
+ * If we get to the end start scanning again from the beginning, and
+ * only scan up to where we initially started scanning from:
+ */
+ buf->last_scanned = start;
+ bch_refill_keybuf(dc->disk.c, buf, &start_pos, dirty_pred);
+
+ return bkey_cmp(&buf->last_scanned, &start_pos) >= 0;
 }

 static int bch_writeback_thread(void *arg)
---

-- 

Denis

[PULL] Re: bcache stability patches

[Eric Wheeler]

Hi Jens and Kent,

This affects many users, so please take a look when you have a moment:

There is a growing bcache user community with a well-tested patchset that 
is necessary for production bcache use.  The diffstat is small and we all 
want someone to pull it in and get it into mainline.  This would serve 
many people if this can get pulled in upstream.

More below:

On Tue, 22 Dec 2015, Denis Bychkov wrote:
> There is a set of bcache stability patches elevating bcache stability to 
> production level. As far as I know, there is no single reported and peer 
> confirmed bug that is not solved by this set. Unfortunately, for some 
> reason, Kent does not have enough time and/or energy to review them and 
> send them upstream. Let's come up with a solution that would allow to 
> review all these patches (some of them written by Ken himself, some of 
> them produced by the community), review them and hand them to the 
> maintainer who is willing to apply them upstream. Without that, bcache 
> is just another half-assed unstable and buggy cache layer. These patches 
> will allow people to start use bcache in production systems. Please find 
> the patch set attached. (The patches apply cleanly to 4.3 and 4.4 kernel 
> series).

Hi Dennis,

I'm maintaining a branch here that is ready to merge.  We have been 
testing this for about a year in production and works great.  All Cc's and 
authors are correct and it (should) have every stability patch below, 
possibly others too.  Please tell me if there are any patches missing:

git pull https://github.com/ewheelerinc/linux.git bcache-patches-for-3.17
(Yes, github for hosting only, I don't edit with their web interfaces.) 

Note that this branch still merges cleanly through v4.4-rc7 and as far 
back as 3.17-rc1 (maybe earlier).  Each patch provides Cc: stable@vger.kernel.org.

It is ready to merge!  We just need Jens or Kent or someone to pull it in.  
Here is the diffstat and shortlog against v4.4-rc7:

 drivers/md/bcache/btree.c     |  5 ++++-
 drivers/md/bcache/super.c     | 16 ++++++++++++----
 drivers/md/bcache/writeback.c | 37 ++++++++++++++++++++++++++++++-------
 drivers/md/bcache/writeback.h |  3 ++-
 4 files changed, 48 insertions(+), 13 deletions(-)

Al Viro (1):
      bcache: fix a leak in bch_cached_dev_run()

Gabriel de Perthuis (1):
      bcache: allows use of register in udev to avoid "device_busy" error.

Kent Overstreet (2):
      bcache: Add a cond_resched() call to gc
      bcache: Change refill_dirty() to always scan entire disk if 
necessary

Stefan Bader (1):
      bcache: prevent crash on changing writeback_running

Zheng Liu (3):
      bcache: fix a livelock when we cause a huge number of cache misses
      bcache: clear BCACHE_DEV_UNLINK_DONE flag when attaching a backing device
      bcache: unregister reboot notifier if bcache fails to unregister device

See also these threads:
  https://lkml.org/lkml/2015/12/5/38
  https://www.mail-archive.com/linux-bcache@vger.kernel.org/msg03159.html

Quickly view commits here, too:
  https://github.com/ewheelerinc/linux/commits/bcache-patches-for-3.17


Cheers,

-Eric


-- 
Eric Wheeler, President           eWheeler, Inc. dba Global Linux Security
888-LINUX26 (888-546-8926)        Fax: 503-716-3878           PO Box 25107
www.GlobalLinuxSecurity.pro       Linux since 1996!     Portland, OR 97298

Re: [PULL] Re: bcache stability patches

[Jens Axboe]

On 12/29/2015 08:00 PM, Eric Wheeler wrote:
> Hi Jens and Kent,
>
> This affects many users, so please take a look when you have a moment:
>
> There is a growing bcache user community with a well-tested patchset that
> is necessary for production bcache use.  The diffstat is small and we all
> want someone to pull it in and get it into mainline.  This would serve
> many people if this can get pulled in upstream.
>
> More below:
>
> On Tue, 22 Dec 2015, Denis Bychkov wrote:
>> There is a set of bcache stability patches elevating bcache stability to
>> production level. As far as I know, there is no single reported and peer
>> confirmed bug that is not solved by this set. Unfortunately, for some
>> reason, Kent does not have enough time and/or energy to review them and
>> send them upstream. Let's come up with a solution that would allow to
>> review all these patches (some of them written by Ken himself, some of
>> them produced by the community), review them and hand them to the
>> maintainer who is willing to apply them upstream. Without that, bcache
>> is just another half-assed unstable and buggy cache layer. These patches
>> will allow people to start use bcache in production systems. Please find
>> the patch set attached. (The patches apply cleanly to 4.3 and 4.4 kernel
>> series).
>
> Hi Dennis,
>
> I'm maintaining a branch here that is ready to merge.  We have been
> testing this for about a year in production and works great.  All Cc's and
> authors are correct and it (should) have every stability patch below,
> possibly others too.  Please tell me if there are any patches missing:
>
> git pull https://github.com/ewheelerinc/linux.git bcache-patches-for-3.17
> (Yes, github for hosting only, I don't edit with their web interfaces.)
>
> Note that this branch still merges cleanly through v4.4-rc7 and as far
> back as 3.17-rc1 (maybe earlier).  Each patch provides Cc: stable@vger.kernel.org.
>
> It is ready to merge!  We just need Jens or Kent or someone to pull it in.
> Here is the diffstat and shortlog against v4.4-rc7:
>
>   drivers/md/bcache/btree.c     |  5 ++++-
>   drivers/md/bcache/super.c     | 16 ++++++++++++----
>   drivers/md/bcache/writeback.c | 37 ++++++++++++++++++++++++++++++-------
>   drivers/md/bcache/writeback.h |  3 ++-
>   4 files changed, 48 insertions(+), 13 deletions(-)
>
> Al Viro (1):
>        bcache: fix a leak in bch_cached_dev_run()
>
> Gabriel de Perthuis (1):
>        bcache: allows use of register in udev to avoid "device_busy" error.
>
> Kent Overstreet (2):
>        bcache: Add a cond_resched() call to gc
>        bcache: Change refill_dirty() to always scan entire disk if
> necessary
>
> Stefan Bader (1):
>        bcache: prevent crash on changing writeback_running
>
> Zheng Liu (3):
>        bcache: fix a livelock when we cause a huge number of cache misses
>        bcache: clear BCACHE_DEV_UNLINK_DONE flag when attaching a backing device
>        bcache: unregister reboot notifier if bcache fails to unregister device

Looking over these, most are really simple one-liners, and nothing 
sticks out as being overly complicated. Kent, do you have any plans to 
maintain the in-kernel bcache?

If I don't hear otherwise, I'll pull these in for 4.5.

-- 
Jens Axboe

Re: [PULL] Re: bcache stability patches

[Kent Overstreet]

On Wed, Dec 30, 2015 at 10:59:39AM -0700, Jens Axboe wrote:
> Looking over these, most are really simple one-liners, and nothing sticks
> out as being overly complicated. Kent, do you have any plans to maintain the
> in-kernel bcache?

Yeah - these patches are all fine, go ahead and pull.

I may start doing maintainence again at some point (but if there's someone
willing to step up and take over and do a good job of it, I'd gladly hand things
off)

Re: [PULL] Re: bcache stability patches

[Jens Axboe]

On 12/30/2015 08:15 PM, Kent Overstreet wrote:
> On Wed, Dec 30, 2015 at 10:59:39AM -0700, Jens Axboe wrote:
>> Looking over these, most are really simple one-liners, and nothing sticks
>> out as being overly complicated. Kent, do you have any plans to maintain the
>> in-kernel bcache?
>
> Yeah - these patches are all fine, go ahead and pull.

Great, thanks.

> I may start doing maintainence again at some point (but if there's someone
> willing to step up and take over and do a good job of it, I'd gladly hand things
> off)

As long as we have a path into mainline for stability fixes, at least 
that's better than before.

Thanks Eric for collecting these. I've reformatted some of them a bit, 
not sure if that's github crappery, or if they came like that. It's 
pushed out now.

-- 
Jens Axboe

Re: [PULL] Re: bcache stability patches

[Kent Overstreet]

On Wed, Dec 30, 2015 at 08:25:36PM -0700, Jens Axboe wrote:
> On 12/30/2015 08:15 PM, Kent Overstreet wrote:
> >On Wed, Dec 30, 2015 at 10:59:39AM -0700, Jens Axboe wrote:
> >>Looking over these, most are really simple one-liners, and nothing sticks
> >>out as being overly complicated. Kent, do you have any plans to maintain the
> >>in-kernel bcache?
> >
> >Yeah - these patches are all fine, go ahead and pull.
> 
> Great, thanks.
> 
> >I may start doing maintainence again at some point (but if there's someone
> >willing to step up and take over and do a good job of it, I'd gladly hand things
> >off)
> 
> As long as we have a path into mainline for stability fixes, at least that's
> better than before.

I'd really like to get the improvements from the bcache-dev branch upstream -
there's a lot of _huge_ improvements (performance and otherwise), but
backporting the non on disk format changes has turned out to be... not really
practical.

So one of the major obstacles has been that there's a ton of very worthwhile
code I'd really like to get upstream, but at this point it's pretty much going
to have to be as drivers/md/bcache2 - effectively a fork that wouldn't support
the original on disk format. And that's a high hurdle.

If the user community is willing to step up and help out (and realistically,
that'd have to include financial support) - that's probably what I'd like to see
happen. But it's a non trivial amount of work to get sufficient testing and to
get the new on disk format stabilized enough to go upstream.

> Thanks Eric for collecting these. I've reformatted some of them a bit, not
> sure if that's github crappery, or if they came like that. It's pushed out
> now.

Thanks Eric - and let me know if you'd be willing to take on more maintainence
work, I may have had some more patches in my backlog and we could talk about
testing.

Re: [PULL] Re: bcache stability patches

[Denis Bychkov]

On Tue, Dec 29, 2015 at 10:00 PM, Eric Wheeler
<bcache@lists.ewheeler.net> wrote:
> Hi Jens and Kent,
>
> This affects many users, so please take a look when you have a moment:
>
> There is a growing bcache user community with a well-tested patchset that
> is necessary for production bcache use.  The diffstat is small and we all
> want someone to pull it in and get it into mainline.  This would serve
> many people if this can get pulled in upstream.
>
> More below:
>
> On Tue, 22 Dec 2015, Denis Bychkov wrote:
>> There is a set of bcache stability patches elevating bcache stability to
>> production level. As far as I know, there is no single reported and peer
>> confirmed bug that is not solved by this set. Unfortunately, for some
>> reason, Kent does not have enough time and/or energy to review them and
>> send them upstream. Let's come up with a solution that would allow to
>> review all these patches (some of them written by Ken himself, some of
>> them produced by the community), review them and hand them to the
>> maintainer who is willing to apply them upstream. Without that, bcache
>> is just another half-assed unstable and buggy cache layer. These patches
>> will allow people to start use bcache in production systems. Please find
>> the patch set attached. (The patches apply cleanly to 4.3 and 4.4 kernel
>> series).
>
> Hi Dennis,
>
> I'm maintaining a branch here that is ready to merge.  We have been
> testing this for about a year in production and works great.  All Cc's and
> authors are correct and it (should) have every stability patch below,
> possibly others too.  Please tell me if there are any patches missing:

Just went through all the patches I have, looks like you did not miss any. The
only small correction - the author of commit
53803810fce7826feff7d5632a7ab3cc991e6243
is me, not Gabriel de Perthuis. Gabriel wrote the original patch which
I mention in
the comment, but it was broken (did not release the mutex), so I
created a new one,
that actually fixed the problem with device_busy error.

>
> git pull https://github.com/ewheelerinc/linux.git bcache-patches-for-3.17
> (Yes, github for hosting only, I don't edit with their web interfaces.)
>
> Note that this branch still merges cleanly through v4.4-rc7 and as far
> back as 3.17-rc1 (maybe earlier).  Each patch provides Cc: stable@vger.kernel.org.
>
> It is ready to merge!  We just need Jens or Kent or someone to pull it in.
> Here is the diffstat and shortlog against v4.4-rc7:
>
>  drivers/md/bcache/btree.c     |  5 ++++-
>  drivers/md/bcache/super.c     | 16 ++++++++++++----
>  drivers/md/bcache/writeback.c | 37 ++++++++++++++++++++++++++++++-------
>  drivers/md/bcache/writeback.h |  3 ++-
>  4 files changed, 48 insertions(+), 13 deletions(-)
>
> Al Viro (1):
>       bcache: fix a leak in bch_cached_dev_run()
>
> Gabriel de Perthuis (1):
>       bcache: allows use of register in udev to avoid "device_busy" error.
>
> Kent Overstreet (2):
>       bcache: Add a cond_resched() call to gc
>       bcache: Change refill_dirty() to always scan entire disk if
> necessary
>
> Stefan Bader (1):
>       bcache: prevent crash on changing writeback_running
>
> Zheng Liu (3):
>       bcache: fix a livelock when we cause a huge number of cache misses
>       bcache: clear BCACHE_DEV_UNLINK_DONE flag when attaching a backing device
>       bcache: unregister reboot notifier if bcache fails to unregister device
>
> See also these threads:
>   https://lkml.org/lkml/2015/12/5/38
>   https://www.mail-archive.com/linux-bcache@vger.kernel.org/msg03159.html
>
> Quickly view commits here, too:
>   https://github.com/ewheelerinc/linux/commits/bcache-patches-for-3.17
>
>
> Cheers,
>
> -Eric
>
>
> --
> Eric Wheeler, President           eWheeler, Inc. dba Global Linux Security
> 888-LINUX26 (888-546-8926)        Fax: 503-716-3878           PO Box 25107
> www.GlobalLinuxSecurity.pro       Linux since 1996!     Portland, OR 97298
>



-- 

Denis

Re: [PULL] Re: bcache stability patches

[Denis Bychkov]

On Thu, Dec 31, 2015 at 12:18 AM, Kent Overstreet
<kent.overstreet@gmail.com> wrote:
> On Wed, Dec 30, 2015 at 08:25:36PM -0700, Jens Axboe wrote:
>> On 12/30/2015 08:15 PM, Kent Overstreet wrote:
>> >On Wed, Dec 30, 2015 at 10:59:39AM -0700, Jens Axboe wrote:
>> >>Looking over these, most are really simple one-liners, and nothing sticks
>> >>out as being overly complicated. Kent, do you have any plans to maintain the
>> >>in-kernel bcache?
>> >
>> >Yeah - these patches are all fine, go ahead and pull.
>>
>> Great, thanks.
>>
>> >I may start doing maintainence again at some point (but if there's someone
>> >willing to step up and take over and do a good job of it, I'd gladly hand things
>> >off)
>>
>> As long as we have a path into mainline for stability fixes, at least that's
>> better than before.
>
> I'd really like to get the improvements from the bcache-dev branch upstream -
> there's a lot of _huge_ improvements (performance and otherwise), but
> backporting the non on disk format changes has turned out to be... not really
> practical.
>
> So one of the major obstacles has been that there's a ton of very worthwhile
> code I'd really like to get upstream, but at this point it's pretty much going
> to have to be as drivers/md/bcache2 - effectively a fork that wouldn't support
> the original on disk format. And that's a high hurdle.

Hey Kent,

Why is it so important to keep the same on-disk format? We are are
talking about the
caching device, not the backing device (which does not have its own
on-disk layout, it's
just the layout of the FS it backs, correct?)
So what's so big of a deal if the caching device format changes? You
just disconnect the cache set
before the upgrade, flushing all the cached data that is not on the
backing device, disable caching for
this device (bcache can work without the caching device in
write-through mode), then upgrade the
kernel and re-create the caching device with the new format. Yes, all
you cache is invalidated, but it
will take few days or, in case of very intensive use/lots of new data,
even few hours. And those who
can't tolerate this warm-up period can stick with the old code. But,
if you say there is A LOT of performance
improvements, it definitely should be worth it.
It's not like you are going to lose your backing device data, only
invalidate the cache.
So, can you please tell me where I am wrong here and why can't we do this?

> If the user community is willing to step up and help out (and realistically,
> that'd have to include financial support) - that's probably what I'd like to see
> happen. But it's a non trivial amount of work to get sufficient testing and to
> get the new on disk format stabilized enough to go upstream.

Speaking for myself I can help with maintenance/coding/unit test
writing/code reviewing.
I realize, you have no idea about my skills, but I do have some
experience with low level/
systems programming. I don't have a lot of DEEP knowledge about linux
kernel, but I did a lot of
driver-related programming back in the day, when memory was a scarce
resource (OS/2 in 90s :).
It was long ago, I admit, but I can learn pretty quick and, besides,
can help with some trivial stuff
like regression tests/debugging, etc


>
>> Thanks Eric for collecting these. I've reformatted some of them a bit, not
>> sure if that's github crappery, or if they came like that. It's pushed out
>> now.
>
> Thanks Eric - and let me know if you'd be willing to take on more maintainence
> work, I may have had some more patches in my backlog and we could talk about
> testing.

-- 

Denis

Re: [PULL] Re: bcache stability patches

[Kent Overstreet]

On Thu, Dec 31, 2015 at 04:19:03PM -0500, Denis Bychkov wrote:
> On Thu, Dec 31, 2015 at 12:18 AM, Kent Overstreet
> <kent.overstreet@gmail.com> wrote:
> > On Wed, Dec 30, 2015 at 08:25:36PM -0700, Jens Axboe wrote:
> >> On 12/30/2015 08:15 PM, Kent Overstreet wrote:
> >> >On Wed, Dec 30, 2015 at 10:59:39AM -0700, Jens Axboe wrote:
> >> >>Looking over these, most are really simple one-liners, and nothing sticks
> >> >>out as being overly complicated. Kent, do you have any plans to maintain the
> >> >>in-kernel bcache?
> >> >
> >> >Yeah - these patches are all fine, go ahead and pull.
> >>
> >> Great, thanks.
> >>
> >> >I may start doing maintainence again at some point (but if there's someone
> >> >willing to step up and take over and do a good job of it, I'd gladly hand things
> >> >off)
> >>
> >> As long as we have a path into mainline for stability fixes, at least that's
> >> better than before.
> >
> > I'd really like to get the improvements from the bcache-dev branch upstream -
> > there's a lot of _huge_ improvements (performance and otherwise), but
> > backporting the non on disk format changes has turned out to be... not really
> > practical.
> >
> > So one of the major obstacles has been that there's a ton of very worthwhile
> > code I'd really like to get upstream, but at this point it's pretty much going
> > to have to be as drivers/md/bcache2 - effectively a fork that wouldn't support
> > the original on disk format. And that's a high hurdle.
> 
> Hey Kent,
> 
> Why is it so important to keep the same on-disk format? We are are
> talking about the
> caching device, not the backing device (which does not have its own
> on-disk layout, it's
> just the layout of the FS it backs, correct?)
> So what's so big of a deal if the caching device format changes? You
> just disconnect the cache set
> before the upgrade, flushing all the cached data that is not on the
> backing device, disable caching for
> this device (bcache can work without the caching device in
> write-through mode), then upgrade the
> kernel and re-create the caching device with the new format. Yes, all
> you cache is invalidated, but it
> will take few days or, in case of very intensive use/lots of new data,
> even few hours. And those who
> can't tolerate this warm-up period can stick with the old code. But,
> if you say there is A LOT of performance
> improvements, it definitely should be worth it.
> It's not like you are going to lose your backing device data, only
> invalidate the cache.
> So, can you please tell me where I am wrong here and why can't we do this?

We certainly can do all that, but: since new bcache can't read the old bcache
format (I can go into why that's impractical, if people are curious) - that
means there's a pretty high cost to switching to the new format:
 - people have to manually upgrade
 - the kernel would have to carry around both the old and the new
   implementations of bcache for as long as people are using the old format -
   we can't force people to upgrade

So this isn't something we want to do more than once, which means we need to
make sure the new on disk format is 100% done. And it's not quite done - the
main thing that's left for it to really be considered done is big endian support
and endian compatibility (writing the code so a little endian machine can read a
big endian layout and vice versa; due to the way bkeys work it's not possible to
just have an endian agnostic layout, we'll have to do swabbing).

And this isn't a trivial amount of work - and besides finishing the on disk
format, there's a fair amount of work on tooling and related stuff to make sure
everything is ready for the switch.

And, I can't work for free, so somehow funding has to be secured. Given the
number of companies that are using bcache, and the fact that Canonical and SuSe
are both apparantly putting in at least a little bit of engineering time into
supporting bcache, you'd think it should be possible but offers have not been
forthcoming.

> Speaking for myself I can help with maintenance/coding/unit test
> writing/code reviewing.
> I realize, you have no idea about my skills, but I do have some
> experience with low level/
> systems programming. I don't have a lot of DEEP knowledge about linux
> kernel, but I did a lot of
> driver-related programming back in the day, when memory was a scarce
> resource (OS/2 in 90s :).
> It was long ago, I admit, but I can learn pretty quick and, besides,
> can help with some trivial stuff
> like regression tests/debugging, etc

That would be useful, but I've had a fair number of offers for help before but
no one has actually committed the time so far.

Re: [PULL] Re: bcache stability patches

[Denis Bychkov]

On Fri, Jan 1, 2016 at 5:36 PM, Kent Overstreet
<kent.overstreet@gmail.com> wrote:
> On Thu, Dec 31, 2015 at 04:19:03PM -0500, Denis Bychkov wrote:
>> On Thu, Dec 31, 2015 at 12:18 AM, Kent Overstreet
>> <kent.overstreet@gmail.com> wrote:
>> > On Wed, Dec 30, 2015 at 08:25:36PM -0700, Jens Axboe wrote:
>> >> On 12/30/2015 08:15 PM, Kent Overstreet wrote:
>> >> >On Wed, Dec 30, 2015 at 10:59:39AM -0700, Jens Axboe wrote:
>> >> >>Looking over these, most are really simple one-liners, and nothing sticks
>> >> >>out as being overly complicated. Kent, do you have any plans to maintain the
>> >> >>in-kernel bcache?
>> >> >
>> >> >Yeah - these patches are all fine, go ahead and pull.
>> >>
>> >> Great, thanks.
>> >>
>> >> >I may start doing maintainence again at some point (but if there's someone
>> >> >willing to step up and take over and do a good job of it, I'd gladly hand things
>> >> >off)
>> >>
>> >> As long as we have a path into mainline for stability fixes, at least that's
>> >> better than before.
>> >
>> > I'd really like to get the improvements from the bcache-dev branch upstream -
>> > there's a lot of _huge_ improvements (performance and otherwise), but
>> > backporting the non on disk format changes has turned out to be... not really
>> > practical.
>> >
>> > So one of the major obstacles has been that there's a ton of very worthwhile
>> > code I'd really like to get upstream, but at this point it's pretty much going
>> > to have to be as drivers/md/bcache2 - effectively a fork that wouldn't support
>> > the original on disk format. And that's a high hurdle.
>>
>> Hey Kent,
>>
>> Why is it so important to keep the same on-disk format? We are are
>> talking about the
>> caching device, not the backing device (which does not have its own
>> on-disk layout, it's
>> just the layout of the FS it backs, correct?)
>> So what's so big of a deal if the caching device format changes? You
>> just disconnect the cache set
>> before the upgrade, flushing all the cached data that is not on the
>> backing device, disable caching for
>> this device (bcache can work without the caching device in
>> write-through mode), then upgrade the
>> kernel and re-create the caching device with the new format. Yes, all
>> you cache is invalidated, but it
>> will take few days or, in case of very intensive use/lots of new data,
>> even few hours. And those who
>> can't tolerate this warm-up period can stick with the old code. But,
>> if you say there is A LOT of performance
>> improvements, it definitely should be worth it.
>> It's not like you are going to lose your backing device data, only
>> invalidate the cache.
>> So, can you please tell me where I am wrong here and why can't we do this?
>
> We certainly can do all that, but: since new bcache can't read the old bcache
> format (I can go into why that's impractical, if people are curious) - that
> means there's a pretty high cost to switching to the new format:
>  - people have to manually upgrade
>  - the kernel would have to carry around both the old and the new
>    implementations of bcache for as long as people are using the old format -
>    we can't force people to upgrade

Not that hard technically, you could just leave the existing bcache
module as-is to avoid regressions when adding the new format support,
since the existing code does not require a lot of maintenance and add
a new module that would only recognize a new superblock. Another
question is how easy it is to convince Linus/top maintainers to keep 2
modules with a lot of duplication with intention to retire the old
code eventually, but something tells me that you know ways around this
problem.

>
> So this isn't something we want to do more than once, which means we need to
> make sure the new on disk format is 100% done. And it's not quite done - the
> main thing that's left for it to really be considered done is big endian support
> and endian compatibility (writing the code so a little endian machine can read a
> big endian layout and vice versa; due to the way bkeys work it's not possible to
> just have an endian agnostic layout, we'll have to do swabbing).

But this problem is not unique to bcache at all, AFAIK, almost any FS
linux supports would not be able to work on a different endianness
than it was created for.

>
> And this isn't a trivial amount of work - and besides finishing the on disk
> format, there's a fair amount of work on tooling and related stuff to make sure
> everything is ready for the switch.
>
> And, I can't work for free, so somehow funding has to be secured. Given the
> number of companies that are using bcache, and the fact that Canonical and SuSe
> are both apparantly putting in at least a little bit of engineering time into
> supporting bcache, you'd think it should be possible but offers have not been
> forthcoming.

I don't know, IMHO bcache was hurt a lot because of a host of small
problems that nobody was able to address for quite some time. It
gained a bad reputation as a production system, unfortunately, which
means not much interest from the enterprise world, which means
Canonical & co. did not want to invest into it. Don't get me wrong, I
am not blaming you. Of all people, I might understand pretty well what
was going on, just explaining why RH or Canonical or Suse did not
fight for the privilege to financially support this project.

>
>> Speaking for myself I can help with maintenance/coding/unit test
>> writing/code reviewing.
>> I realize, you have no idea about my skills, but I do have some
>> experience with low level/
>> systems programming. I don't have a lot of DEEP knowledge about linux
>> kernel, but I did a lot of
>> driver-related programming back in the day, when memory was a scarce
>> resource (OS/2 in 90s :).
>> It was long ago, I admit, but I can learn pretty quick and, besides,
>> can help with some trivial stuff
>> like regression tests/debugging, etc
>
> That would be useful, but I've had a fair number of offers for help before but
> no one has actually committed the time so far.

Well, I get your bitterness, but there is only one way to find out, right?

-- 

Denis

Re: [PULL] Re: bcache stability patches

[Vojtech Pavlik]

On Fri, Jan 01, 2016 at 08:28:39PM -0500, Denis Bychkov wrote:

> > We certainly can do all that, but: since new bcache can't read the old bcache
> > format (I can go into why that's impractical, if people are curious) - that
> > means there's a pretty high cost to switching to the new format:
> >  - people have to manually upgrade
> >  - the kernel would have to carry around both the old and the new
> >    implementations of bcache for as long as people are using the old format -
> >    we can't force people to upgrade
> 
> Not that hard technically, you could just leave the existing bcache
> module as-is to avoid regressions when adding the new format support,
> since the existing code does not require a lot of maintenance and add
> a new module that would only recognize a new superblock. Another
> question is how easy it is to convince Linus/top maintainers to keep 2
> modules with a lot of duplication with intention to retire the old
> code eventually, but something tells me that you know ways around this
> problem.

We had that with the UHCI drivers, it is there with ext3/ext4, I don't
think that is a real problem.

> > So this isn't something we want to do more than once, which means we need to
> > make sure the new on disk format is 100% done. And it's not quite done - the
> > main thing that's left for it to really be considered done is big endian support
> > and endian compatibility (writing the code so a little endian machine can read a
> > big endian layout and vice versa; due to the way bkeys work it's not possible to
> > just have an endian agnostic layout, we'll have to do swabbing).
> 
> But this problem is not unique to bcache at all, AFAIK, almost any FS
> linux supports would not be able to work on a different endianness
> than it was created for.

On the contrary, all modern filesystems cope with endianness
portability. The only major filesystem in use where endianness is not
handled is, as far I know, UFS.

At the same time, I don't see endianness portability, the ability to
create a cache on a machine with one endian and then mounting it on a
machine with the opposite endian a real use case.

Unlike fileystems, which can be used to transfer valuable data between
machines, the cache only contains ephemeral data, which can easily be
recreated from the backing device.

Hence I believe that it is reasonable to require the user to nuke the
contents of the cache when moving the cache set between machines of
different endianity.

Ideally this would happen automatically and error out if the cache isn't
clean.

Actually, the same would be fine for format version changes.

> > And this isn't a trivial amount of work - and besides finishing the on disk
> > format, there's a fair amount of work on tooling and related stuff to make sure
> > everything is ready for the switch.
> >
> > And, I can't work for free, so somehow funding has to be secured. Given the
> > number of companies that are using bcache, and the fact that Canonical and SuSe
> > are both apparantly putting in at least a little bit of engineering time into
> > supporting bcache, you'd think it should be possible but offers have not been
> > forthcoming.
> 
> I don't know, IMHO bcache was hurt a lot because of a host of small
> problems that nobody was able to address for quite some time. It
> gained a bad reputation as a production system, unfortunately, which
> means not much interest from the enterprise world, which means
> Canonical & co. did not want to invest into it. Don't get me wrong, I
> am not blaming you. Of all people, I might understand pretty well what
> was going on, just explaining why RH or Canonical or Suse did not
> fight for the privilege to financially support this project.

SUSE had plans for bcache, however, since upstram stable branch
maintenance has been more than unreliable, we postponed most of them and
are building knowledge in-house to be able to fully support it before we
deploy.

The structure of the code doesn't really help, either.

-- 
Vojtech Pavlik
Director SUSE Labs

Re: [PULL] Re: bcache stability patches

[Denis Bychkov]

On Sat, Jan 2, 2016 at 6:50 AM, Vojtech Pavlik <vojtech@suse.com> wrote:
> On Fri, Jan 01, 2016 at 08:28:39PM -0500, Denis Bychkov wrote:
>
>> > We certainly can do all that, but: since new bcache can't read the old bcache
>> > format (I can go into why that's impractical, if people are curious) - that
>> > means there's a pretty high cost to switching to the new format:
>> >  - people have to manually upgrade
>> >  - the kernel would have to carry around both the old and the new
>> >    implementations of bcache for as long as people are using the old format -
>> >    we can't force people to upgrade
>>
>> Not that hard technically, you could just leave the existing bcache
>> module as-is to avoid regressions when adding the new format support,
>> since the existing code does not require a lot of maintenance and add
>> a new module that would only recognize a new superblock. Another
>> question is how easy it is to convince Linus/top maintainers to keep 2
>> modules with a lot of duplication with intention to retire the old
>> code eventually, but something tells me that you know ways around this
>> problem.
>
> We had that with the UHCI drivers, it is there with ext3/ext4, I don't
> think that is a real problem.
>
>> > So this isn't something we want to do more than once, which means we need to
>> > make sure the new on disk format is 100% done. And it's not quite done - the
>> > main thing that's left for it to really be considered done is big endian support
>> > and endian compatibility (writing the code so a little endian machine can read a
>> > big endian layout and vice versa; due to the way bkeys work it's not possible to
>> > just have an endian agnostic layout, we'll have to do swabbing).
>>
>> But this problem is not unique to bcache at all, AFAIK, almost any FS
>> linux supports would not be able to work on a different endianness
>> than it was created for.
>
> On the contrary, all modern filesystems cope with endianness
> portability. The only major filesystem in use where endianness is not
> handled is, as far I know, UFS.
>
> At the same time, I don't see endianness portability, the ability to
> create a cache on a machine with one endian and then mounting it on a
> machine with the opposite endian a real use case.
>
> Unlike fileystems, which can be used to transfer valuable data between
> machines, the cache only contains ephemeral data, which can easily be
> recreated from the backing device.
>
> Hence I believe that it is reasonable to require the user to nuke the
> contents of the cache when moving the cache set between machines of
> different endianity.
>
> Ideally this would happen automatically and error out if the cache isn't
> clean.
>
> Actually, the same would be fine for format version changes.

Yeah, I totally agree with you here. I just think that dirty cache
situation might be much more common and less avoidable, which means it
requires a lot of dancing around in terms of tooling, documentation,
testing, etc. But it can easily be solved, it's not a hard problem,
it's just time-consuming and this is something that Kent might use
some help with.

>
>> > And this isn't a trivial amount of work - and besides finishing the on disk
>> > format, there's a fair amount of work on tooling and related stuff to make sure
>> > everything is ready for the switch.
>> >
>> > And, I can't work for free, so somehow funding has to be secured. Given the
>> > number of companies that are using bcache, and the fact that Canonical and SuSe
>> > are both apparantly putting in at least a little bit of engineering time into
>> > supporting bcache, you'd think it should be possible but offers have not been
>> > forthcoming.
>>
>> I don't know, IMHO bcache was hurt a lot because of a host of small
>> problems that nobody was able to address for quite some time. It
>> gained a bad reputation as a production system, unfortunately, which
>> means not much interest from the enterprise world, which means
>> Canonical & co. did not want to invest into it. Don't get me wrong, I
>> am not blaming you. Of all people, I might understand pretty well what
>> was going on, just explaining why RH or Canonical or Suse did not
>> fight for the privilege to financially support this project.
>
> SUSE had plans for bcache, however, since upstram stable branch
> maintenance has been more than unreliable, we postponed most of them and
> are building knowledge in-house to be able to fully support it before we
> deploy.

Yeah, I was able to deduce that much. Not good.

 >
> The structure of the code doesn't really help, either.
>
> --
> Vojtech Pavlik
> Director SUSE Labs



-- 

Denis

Re: [PULL] Re: bcache stability patches

[Kent Overstreet]

On Sat, Jan 02, 2016 at 10:48:04AM -0500, Denis Bychkov wrote:
> On Sat, Jan 2, 2016 at 6:50 AM, Vojtech Pavlik <vojtech@suse.com> wrote:
> > On the contrary, all modern filesystems cope with endianness
> > portability. The only major filesystem in use where endianness is not
> > handled is, as far I know, UFS.
> >
> > At the same time, I don't see endianness portability, the ability to
> > create a cache on a machine with one endian and then mounting it on a
> > machine with the opposite endian a real use case.
> >
> > Unlike fileystems, which can be used to transfer valuable data between
> > machines, the cache only contains ephemeral data, which can easily be
> > recreated from the backing device.
> >
> > Hence I believe that it is reasonable to require the user to nuke the
> > contents of the cache when moving the cache set between machines of
> > different endianity.
> >
> > Ideally this would happen automatically and error out if the cache isn't
> > clean.
> >
> > Actually, the same would be fine for format version changes.
> 
> Yeah, I totally agree with you here. I just think that dirty cache
> situation might be much more common and less avoidable, which means it
> requires a lot of dancing around in terms of tooling, documentation,
> testing, etc. But it can easily be solved, it's not a hard problem,
> it's just time-consuming and this is something that Kent might use
> some help with.

The bcache2 on disk format is the same on disk format as bcachefs - so I do want
to get endian portability done right.

It shouldn't be an outrageous amount of work though, the biggest hassle is just
going to be getting a test environment set up.

> >> > And this isn't a trivial amount of work - and besides finishing the on disk
> >> > format, there's a fair amount of work on tooling and related stuff to make sure
> >> > everything is ready for the switch.
> >> >
> >> > And, I can't work for free, so somehow funding has to be secured. Given the
> >> > number of companies that are using bcache, and the fact that Canonical and SuSe
> >> > are both apparantly putting in at least a little bit of engineering time into
> >> > supporting bcache, you'd think it should be possible but offers have not been
> >> > forthcoming.
> >>
> >> I don't know, IMHO bcache was hurt a lot because of a host of small
> >> problems that nobody was able to address for quite some time. It
> >> gained a bad reputation as a production system, unfortunately, which
> >> means not much interest from the enterprise world, which means
> >> Canonical & co. did not want to invest into it. Don't get me wrong, I
> >> am not blaming you. Of all people, I might understand pretty well what
> >> was going on, just explaining why RH or Canonical or Suse did not
> >> fight for the privilege to financially support this project.
> >
> > SUSE had plans for bcache, however, since upstram stable branch
> > maintenance has been more than unreliable, we postponed most of them and
> > are building knowledge in-house to be able to fully support it before we
> > deploy.

The biggest reason for maintainence dropping off was me going off to a certain
startup that shall not be named, which ended up being fairly all-consuming, and
left me pretty burned out in the end. I'm not going to revisit that topic right
now, except to say that upstream maintainence is not the only reason I have
mixed feelings about that decision...

I do want to say though that I never knew Suse or Canonical engineers were ever
looking at the code or that either companies were ever considering supporting it
- if I had, I would've certainly made an effort to work with your engineers on
getting them up to speed.

Anyways, what's done is done but if the demand is there I'd really like to see
the codebase live a long happy life, and figure out if we can make that happen
now.

Re: [PULL] Re: bcache stability patches

[Eric Wheeler]

On Wed, 30 Dec 2015, Jens Axboe wrote:

> On 12/30/2015 08:15 PM, Kent Overstreet wrote:
> > On Wed, Dec 30, 2015 at 10:59:39AM -0700, Jens Axboe wrote:
> > > Looking over these, most are really simple one-liners, and nothing sticks
> > > out as being overly complicated. Kent, do you have any plans to maintain
> > > the
> > > in-kernel bcache?
> >
> > Yeah - these patches are all fine, go ahead and pull.
> 
> Great, thanks.
> 
> > I may start doing maintainence again at some point (but if there's someone
> > willing to step up and take over and do a good job of it, I'd gladly hand
> > things
> > off)

We use this in production internally, so I'll keep collecting patches and 
may ask for pulls from time to time.  

> As long as we have a path into mainline for stability fixes, at least that's
> better than before.
> 
> Thanks Eric for collecting these. I've reformatted some of them a bit, not
> sure if that's github crappery, or if they came like that. It's pushed out
> now.

You're welcome.  It could be github crappery, however, these were 
copy-pasted and cleaned up in various ways such that they merge into 
3.17-rc1 and then merge forward into 4.1.15 which we build against---so 
who knows about formatting, whitspace, and other minor issues.  I'm just 
glad they are on their way to 4.5 with stable@ Cc's!

BTW, can someone suggest an alternate git host with less crappery?

-Eric

> 
> -- 
> Jens Axboe
> 
> 
> 

Re: bcache stability patches: Now mainlined!

[Eric Wheeler]

On Wed, 30 Dec 2015, Jens Axboe wrote:
> On 12/30/2015 08:15 PM, Kent Overstreet wrote:
> > Yeah - these patches are all fine, go ahead and pull.
> >
> > I may start doing maintainence again at some point (but if there's 
> > someone willing to step up and take over and do a good job of it, I'd 
> > gladly hand things off)
> 
> As long as we have a path into mainline for stability fixes, at least that's
> better than before.

I would like to thank everyone involved in getting these patches 
mainlined.  Bcache should now be stable on the following releases:

	mainline: 4.5-rc1 (and later)
	longterm: 4.1.18
	longterm: 3.18.27

Please test on your favorite branch and let us know how it goes.

For those working from another branch, please see:
	git log 2ef9ccbf~1..627ccd20

and cherry-pick with
	git cherry-pick 2ef9ccbf~1..627ccd20


Cheers,	


--
Eric Wheeler
Linux since 1996!  

Re: bcache stability patches: Now mainlined!

[Stefan Priebe - Profihost AG]

Am 24.02.2016 um 07:45 schrieb Eric Wheeler:
> On Wed, 30 Dec 2015, Jens Axboe wrote:
>> On 12/30/2015 08:15 PM, Kent Overstreet wrote:
>>> Yeah - these patches are all fine, go ahead and pull.
>>>
>>> I may start doing maintainence again at some point (but if there's 
>>> someone willing to step up and take over and do a good job of it, I'd 
>>> gladly hand things off)
>>
>> As long as we have a path into mainline for stability fixes, at least that's
>> better than before.
> 
> I would like to thank everyone involved in getting these patches 
> mainlined.  Bcache should now be stable on the following releases:
> 
> 	mainline: 4.5-rc1 (and later)
> 	longterm: 4.1.18
> 	longterm: 3.18.27

the new longterm 4.4.2 should also contain them all.

> 
> Please test on your favorite branch and let us know how it goes.
> 
> For those working from another branch, please see:
> 	git log 2ef9ccbf~1..627ccd20
> 
> and cherry-pick with
> 	git cherry-pick 2ef9ccbf~1..627ccd20
> 
> 
> Cheers,	
> 
> 
> --
> Eric Wheeler
> Linux since 1996!  
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: bcache stability patches: Now mainlined!

[Eric Wheeler]

On Wed, 24 Feb 2016, Stefan Priebe - Profihost AG wrote:
> Am 24.02.2016 um 07:45 schrieb Eric Wheeler:
> > I would like to thank everyone involved in getting these patches 
> > mainlined.  Bcache should now be stable on the following releases:
> > 
> > 	mainline: 4.5-rc1 (and later)
> > 	longterm: 4.1.18
> > 	longterm: 3.18.27
> 
> the new longterm 4.4.2 should also contain them all.

I agree, it should, but it doesn't appear to:

https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/log/?id=refs/tags/v4.4.2

# git log --grep="bcache:" --oneline | head
532eb52 bcache: remove driver private bio splitting code
9ea18f8 Merge branch 'for-3.19/drivers' of git://git.kernel.dk/linux-block
aae4933 md/bcache: use generic io stats accounting functions to simplify 
  io stat accounting d429a36 Merge branch 'for-3.17/drivers' of 
  git://git.kernel.dk/linux-block
[...]

Maybe in 4.4.3 after GKH merges the Cc: stable@ tagged commits.
  
It does, however, have the bio splitting code removal in 532eb52, which 
earlier releases do not.

-Eric

> > 
> > Please test on your favorite branch and let us know how it goes.
> > 
> > For those working from another branch, please see:
> > 	git log 2ef9ccbf~1..627ccd20
> > 
> > and cherry-pick with
> > 	git cherry-pick 2ef9ccbf~1..627ccd20
> > 
> > 
> > Cheers,	
> > 
> > 
> > --
> > Eric Wheeler
> > Linux since 1996!  
> > 
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 





--
Eric Wheeler, President           eWheeler, Inc. dba Global Linux Security
888-LINUX26 (888-546-8926)        Fax: 503-716-3878           PO Box 25107
www.GlobalLinuxSecurity.pro       Linux since 1996!     Portland, OR 97298

Re: bcache stability patches: Now mainlined!

[Tim Small]

On 24/02/16 06:45, Eric Wheeler wrote:
> I would like to thank everyone involved in getting these patches 
> mainlined.  Bcache should now be stable on the following releases:
> 
> 	mainline: 4.5-rc1 (and later)
> 	longterm: 4.1.18
> 	longterm: 3.18.27
> 
> Please test on your favorite branch and let us know how it goes.

According to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=779628#25

... these patches are now in the Debian testing ("Stretch") kernel
(currently 4.3.x based), and it suggests that they may also be applied
to Debian stable ("Jessie") - which is 3.16.x based in a future Debian
"point release".

Cheers,

Tim.