bcache: register_cache() error opening sdh2: cannot allocate memory => BUG NULL pointer

bcache: register_cache() error opening sdh2: cannot allocate memory => BUG NULL pointer

[Marc MERLIN]

I understand I have an underlying memory problem, but could bcache handle
memory issues and not crash on null pointer?

gargamel:/sys/block/md5/bcache# echo /dev/sdh2 > /sys/fs/bcache/register
gave:
bcache: register_cache() error opening sdh2: cannot allocate memory
BUG: unable to handle kernel NULL pointer dereference at 00000000000009b8
IP: [<ffffffffc05a7e8d>] cache_set_flush+0x102/0x15c [bcache]
PGD 120dff067 PUD 1119a3067 PMD 0 
Oops: 0000 [#1] SMP 
Modules linked in: veth ip6table_filter ip6_tables
(...)
CPU: 4 PID: 3371 Comm: kworker/4:3 Not tainted 4.4.2-amd64-i915-volpreempt-20160213bc1 #3
Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
Workqueue: events cache_set_flush [bcache]
task: ffff88020d5dc280 ti: ffff88020b6f8000 task.ti: ffff88020b6f8000
RIP: 0010:[<ffffffffc05a7e8d>]  [<ffffffffc05a7e8d>] cache_set_flush+0x102/0x15c [bcache]

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: bcache: register_cache() error opening sdh2: cannot allocate memory => BUG NULL pointer

[Eric Wheeler]

On Wed, 24 Feb 2016, Marc MERLIN wrote:

> I understand I have an underlying memory problem, but could bcache handle
> memory issues and not crash on null pointer?

Try this patch.  It adds some error handling:

=======================================================================================
diff --git a/drivers/md/bcache/super.c b/drivers/md/bcache/super.c
index a542b58..5d95816 100644
--- a/drivers/md/bcache/super.c
+++ b/drivers/md/bcache/super.c
@@ -1829,11 +1829,12 @@ static int cache_alloc(struct cache_sb *sb, struct cache *ca)
 	return 0;
 }
 
-static void register_cache(struct cache_sb *sb, struct page *sb_page,
+static int register_cache(struct cache_sb *sb, struct page *sb_page,
 				struct block_device *bdev, struct cache *ca)
 {
 	char name[BDEVNAME_SIZE];
-	const char *err = "cannot allocate memory";
+	const char *err = NULL; 
+	int ret = 0;
 
 	memcpy(&ca->sb, sb, sizeof(struct cache_sb));
 	ca->bdev = bdev;
@@ -1848,27 +1849,34 @@ static void register_cache(struct cache_sb *sb, struct page *sb_page,
 	if (blk_queue_discard(bdev_get_queue(ca->bdev)))
 		ca->discard = CACHE_DISCARD(&ca->sb);
 
-	if (cache_alloc(sb, ca) != 0)
+	if ((ret = cache_alloc(sb, ca)) != 0) 
 		goto err;
 
-	err = "error creating kobject";
-	if (kobject_add(&ca->kobj, &part_to_dev(bdev->bd_part)->kobj, "bcache"))
-		goto err;
+	if (kobject_add(&ca->kobj, &part_to_dev(bdev->bd_part)->kobj, "bcache")) {
+		err = "error calling kobject_add";
+		ret = -ENOMEM;
+		goto out;
+	}
 
 	mutex_lock(&bch_register_lock);
 	err = register_cache_set(ca);
 	mutex_unlock(&bch_register_lock);
 
-	if (err)
-		goto err;
-
+	if (err) {
+		ret = -ENODEV;
+		goto out;
+	}
+	
 	pr_info("registered cache device %s", bdevname(bdev, name));
+
 out:
 	kobject_put(&ca->kobj);
-	return;
+
 err:
-	pr_notice("error opening %s: %s", bdevname(bdev, name), err);
-	goto out;
+	if (err)
+		pr_notice("error opening %s: %s", bdevname(bdev, name), err);
+
+	return ret;
 }
 
 /* Global interfaces/init */
@@ -1964,7 +1972,8 @@ static ssize_t register_bcache(struct kobject *k, struct kobj_attribute *attr,
 		if (!ca)
 			goto err_close;
 
-		register_cache(sb, sb_page, bdev, ca);
+		if (register_cache(sb, sb_page, bdev, ca) != 0)
+			goto err_close;
 	}
 out:
 	if (sb_page)
===================================================================


-Eric


> 
> gargamel:/sys/block/md5/bcache# echo /dev/sdh2 > /sys/fs/bcache/register
> gave:
> bcache: register_cache() error opening sdh2: cannot allocate memory
> BUG: unable to handle kernel NULL pointer dereference at 00000000000009b8
> IP: [<ffffffffc05a7e8d>] cache_set_flush+0x102/0x15c [bcache]
> PGD 120dff067 PUD 1119a3067 PMD 0 
> Oops: 0000 [#1] SMP 
> Modules linked in: veth ip6table_filter ip6_tables
> (...)
> CPU: 4 PID: 3371 Comm: kworker/4:3 Not tainted 4.4.2-amd64-i915-volpreempt-20160213bc1 #3
> Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
> Workqueue: events cache_set_flush [bcache]
> task: ffff88020d5dc280 ti: ffff88020b6f8000 task.ti: ffff88020b6f8000
> RIP: 0010:[<ffffffffc05a7e8d>]  [<ffffffffc05a7e8d>] cache_set_flush+0x102/0x15c [bcache]
> 
> Thanks,
> Marc
> -- 
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/  
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: bcache: register_cache() error opening sdh2: cannot allocate memory => BUG NULL pointer

[Marc MERLIN]

On Thu, Feb 25, 2016 at 07:29:36AM +0000, Eric Wheeler wrote:
> Try this patch.  It adds some error handling:

Thanks, applied. I'll report back if I get output from it.

I think one reason I'm having problem on that server is that I run
zoneminder which requires a lot of shared memory:
kernel.shmall = 234217728
kernel.shmmax = 234217728

Is there a chance that's putting memory pressure in the wrong memory
pool on the system?

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: bcache: register_cache() error opening sdh2: cannot allocate memory => BUG NULL pointer

[Eric Wheeler]

On Thu, 25 Feb 2016, Marc MERLIN wrote:

> On Thu, Feb 25, 2016 at 07:29:36AM +0000, Eric Wheeler wrote:
> > Try this patch.  It adds some error handling:
> 
> Thanks, applied. I'll report back if I get output from it.
> 
> I think one reason I'm having problem on that server is that I run
> zoneminder which requires a lot of shared memory:
> kernel.shmall = 234217728
> kernel.shmmax = 234217728
> 
> Is there a chance that's putting memory pressure in the wrong memory
> pool on the system?

Perhaps, though I'm thinking your shm values are far too large.  Those are 
in units of pages, so with 4k pages you have set almost 900GB of shared 
memory limit.  Does this machine have 1TB of ram?

I like to set vm.min_free_kbytes = 262144 on my machines.  Its <1% on our 
systems and we never have memory issues, even with high memory pressure 
and zram for swap.

-Eric



> 
> Marc
> -- 
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: bcache: register_cache() error opening sdh2: cannot allocate memory => BUG NULL pointer

[Marc MERLIN]

On Thu, Feb 25, 2016 at 11:30:05PM +0000, Eric Wheeler wrote:
> On Thu, 25 Feb 2016, Marc MERLIN wrote:
> 
> > On Thu, Feb 25, 2016 at 07:29:36AM +0000, Eric Wheeler wrote:
> > > Try this patch.  It adds some error handling:
> > 
> > Thanks, applied. I'll report back if I get output from it.
> > 
> > I think one reason I'm having problem on that server is that I run
> > zoneminder which requires a lot of shared memory:
> > kernel.shmall = 234217728
> > kernel.shmmax = 234217728
> > 
> > Is there a chance that's putting memory pressure in the wrong memory
> > pool on the system?
> 
> Perhaps, though I'm thinking your shm values are far too large.  Those are 
> in units of pages, so with 4k pages you have set almost 900GB of shared 
> memory limit.  Does this machine have 1TB of ram?

Indeed, no.

> I like to set vm.min_free_kbytes = 262144 on my machines.  Its <1% on our 
> systems and we never have memory issues, even with high memory pressure 
> and zram for swap.

Very good catch. I'll fix that.
I use zoneminder which requires stupid amounts of shared memory, but not 1TB
:)

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/