[Regression, bisected 9e30cc] "sysfs, kernfs: no need to kern_mount() sysfs from sysfs_init()" prevents system from booting correctly

[Regression, bisected 9e30cc] "sysfs, kernfs: no need to kern_mount() sysfs from sysfs_init()" prevents system from booting correctly

[Alexandre Demers]

I was told to send this bug
(https://bugzilla.kernel.org/show_bug.cgi?id=72061) directly by email.

So, I've been struggling with something that looks like a bug, but I
may be wrong. Since 3.14-rc1 up until yesterday's latest commit, I
can't boot my system correctly. The kernel will boot, but when it
comes to mounting the disk's partitions, the computer seem to hit
something while mounting the root partition: the light associated to
the disk's activity stays on; if I hit ctrl+alt+del in the first 10 to
15 seconds, the computer will restart; beyond that point, the system
just continues to work on the disk for a long period (I was painting a
door the other day and it continued reading or writing the whole time)
until it begins outputting errors. I'll have to add it in another
comment later. It is about not being able to copy because disk is full
(my partitions have plenty of free space). So I've been bisecting
until I found out when the problem had begun.

------
Bisecting points to the following first bad commit:
9e30cc9595303b27b48be49b7bcd4d0679e34253 is the first bad commit
commit 9e30cc9595303b27b48be49b7bcd4d0679e34253
Author: Tejun Heo <tj@kernel.org>
Date:   Thu Nov 28 14:54:38 2013 -0500

    sysfs, kernfs: no need to kern_mount() sysfs from sysfs_init()

    It has been very long since sysfs depended on vfs to keep track of
    internal states and whether sysfs is mounted or not doesn't make any
    difference to sysfs's internal operation.

    In addition to init and filesystem type registration, sysfs_init()
    invokes kern_mount() to create in-kernel mount of sysfs.  This
    internal mounting doesn't server any purpose anymore.  Remove it.

    Signed-off-by: Tejun Heo <tj@kernel.org>
    Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

:040000 040000 d85991e8a4e333ad651dc593cacff5e6b8d4f916
d3bd9c50967807c9145a7bd9d3ec7978101c93f3 M fs
------

Using any kernel compiled before that commit boots fine. Am I missing something?

Re: [Regression, bisected 9e30cc] "sysfs, kernfs: no need to kern_mount() sysfs from sysfs_init()" prevents system from booting correctly

[Tejun Heo]

Hello, Alexandre.

On Fri, Mar 14, 2014 at 09:27:35AM -0400, Alexandre Demers wrote:
> I was told to send this bug
> (https://bugzilla.kernel.org/show_bug.cgi?id=72061) directly by email.
> 
> So, I've been struggling with something that looks like a bug, but I
> may be wrong. Since 3.14-rc1 up until yesterday's latest commit, I
> can't boot my system correctly. The kernel will boot, but when it
> comes to mounting the disk's partitions, the computer seem to hit
> something while mounting the root partition: the light associated to
> the disk's activity stays on; if I hit ctrl+alt+del in the first 10 to
> 15 seconds, the computer will restart; beyond that point, the system
> just continues to work on the disk for a long period (I was painting a
> door the other day and it continued reading or writing the whole time)
> until it begins outputting errors. I'll have to add it in another
> comment later. It is about not being able to copy because disk is full
> (my partitions have plenty of free space). So I've been bisecting
> until I found out when the problem had begun.
> 
> ------
> Bisecting points to the following first bad commit:
> 9e30cc9595303b27b48be49b7bcd4d0679e34253 is the first bad commit
> commit 9e30cc9595303b27b48be49b7bcd4d0679e34253
> Author: Tejun Heo <tj@kernel.org>
> Date:   Thu Nov 28 14:54:38 2013 -0500
> 
>     sysfs, kernfs: no need to kern_mount() sysfs from sysfs_init()
> 
>     It has been very long since sysfs depended on vfs to keep track of
>     internal states and whether sysfs is mounted or not doesn't make any
>     difference to sysfs's internal operation.
> 
>     In addition to init and filesystem type registration, sysfs_init()
>     invokes kern_mount() to create in-kernel mount of sysfs.  This
>     internal mounting doesn't server any purpose anymore.  Remove it.
> 
>     Signed-off-by: Tejun Heo <tj@kernel.org>
>     Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> 
> :040000 040000 d85991e8a4e333ad651dc593cacff5e6b8d4f916
> d3bd9c50967807c9145a7bd9d3ec7978101c93f3 M fs
> ------
> 
> Using any kernel compiled before that commit boots fine. Am I missing something?

Hah, that's interesting and completely unexpected.  What userland are
you using?  Also, after such failure fills up the filesystem, can you
determine which files are filling it up?  We'll need to restore the
original behavior but need to first find out how userland depends on
it at all.

Thanks.

-- 
tejun

Re: [Regression, bisected 9e30cc] "sysfs, kernfs: no need to kern_mount() sysfs from sysfs_init()" prevents system from booting correctly

[Alexandre Demers]

Hello Tejun,

Since it takes a while before the system begins to display errors,
I'll have to test it and report it later today when I'll get back from
work.

Now, about the userland part, this seems a broad question... I'm using
Arch Linux 64bit (updated mostly on a daily basis). But I suspect you
are expecting a narrower information. Is there anything in particular
you want to know?

I'll be waiting for your answer to give you more information about the
userland part.
Alexandre

On Fri, Mar 14, 2014 at 10:08 AM, Tejun Heo <tj@kernel.org> wrote:
> Hello, Alexandre.
>
> On Fri, Mar 14, 2014 at 09:27:35AM -0400, Alexandre Demers wrote:
>> I was told to send this bug
>> (https://bugzilla.kernel.org/show_bug.cgi?id=72061) directly by email.
>>
>> So, I've been struggling with something that looks like a bug, but I
>> may be wrong. Since 3.14-rc1 up until yesterday's latest commit, I
>> can't boot my system correctly. The kernel will boot, but when it
>> comes to mounting the disk's partitions, the computer seem to hit
>> something while mounting the root partition: the light associated to
>> the disk's activity stays on; if I hit ctrl+alt+del in the first 10 to
>> 15 seconds, the computer will restart; beyond that point, the system
>> just continues to work on the disk for a long period (I was painting a
>> door the other day and it continued reading or writing the whole time)
>> until it begins outputting errors. I'll have to add it in another
>> comment later. It is about not being able to copy because disk is full
>> (my partitions have plenty of free space). So I've been bisecting
>> until I found out when the problem had begun.
>>
>> ------
>> Bisecting points to the following first bad commit:
>> 9e30cc9595303b27b48be49b7bcd4d0679e34253 is the first bad commit
>> commit 9e30cc9595303b27b48be49b7bcd4d0679e34253
>> Author: Tejun Heo <tj@kernel.org>
>> Date:   Thu Nov 28 14:54:38 2013 -0500
>>
>>     sysfs, kernfs: no need to kern_mount() sysfs from sysfs_init()
>>
>>     It has been very long since sysfs depended on vfs to keep track of
>>     internal states and whether sysfs is mounted or not doesn't make any
>>     difference to sysfs's internal operation.
>>
>>     In addition to init and filesystem type registration, sysfs_init()
>>     invokes kern_mount() to create in-kernel mount of sysfs.  This
>>     internal mounting doesn't server any purpose anymore.  Remove it.
>>
>>     Signed-off-by: Tejun Heo <tj@kernel.org>
>>     Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
>>
>> :040000 040000 d85991e8a4e333ad651dc593cacff5e6b8d4f916
>> d3bd9c50967807c9145a7bd9d3ec7978101c93f3 M fs
>> ------
>>
>> Using any kernel compiled before that commit boots fine. Am I missing something?
>
> Hah, that's interesting and completely unexpected.  What userland are
> you using?  Also, after such failure fills up the filesystem, can you
> determine which files are filling it up?  We'll need to restore the
> original behavior but need to first find out how userland depends on
> it at all.
>
> Thanks.
>
> --
> tejun

Re: [Regression, bisected 9e30cc] "sysfs, kernfs: no need to kern_mount() sysfs from sysfs_init()" prevents system from booting correctly

[Tejun Heo]

Hello,

On Fri, Mar 14, 2014 at 11:48:10AM -0400, Alexandre Demers wrote:
> Since it takes a while before the system begins to display errors,
> I'll have to test it and report it later today when I'll get back from
> work.

Any update?

> Now, about the userland part, this seems a broad question... I'm using
> Arch Linux 64bit (updated mostly on a daily basis). But I suspect you
> are expecting a narrower information. Is there anything in particular
> you want to know?

I was just wondering whether you were running something really exotic.
Doesn't seem that way.  Weird that other people aren't hitting this
issue.  Hmm...

Thanks.

-- 
tejun

Re: [Regression, bisected 9e30cc] "sysfs, kernfs: no need to kern_mount() sysfs from sysfs_init()" prevents system from booting correctly

[Alexandre Demers]

By the way, I tried to get the error message, but my display had gone 
in standby mode before it completed and it wouldn't wake up. I'll have 
to test with an earlier 3.14 kernel (rc1 or rc2), when it was answering 
to my wake up signal. Even though my display doesn't wake up with rc6, 
it is still running since I can see some activities on my disk from 
time to time.

I'll be back with more info ASAP.

Alexandre Demers

On Mon 17 Mar 2014 04:13:46 PM EDT, Tejun Heo wrote:
> Hello,
>
> On Fri, Mar 14, 2014 at 11:48:10AM -0400, Alexandre Demers wrote:
>> Since it takes a while before the system begins to display errors,
>> I'll have to test it and report it later today when I'll get back from
>> work.
>
> Any update?
>
>> Now, about the userland part, this seems a broad question... I'm using
>> Arch Linux 64bit (updated mostly on a daily basis). But I suspect you
>> are expecting a narrower information. Is there anything in particular
>> you want to know?
>
> I was just wondering whether you were running something really exotic.
> Doesn't seem that way.  Weird that other people aren't hitting this
> issue.  Hmm...
>
> Thanks.
>

Re: [Regression, bisected 9e30cc] "sysfs, kernfs: no need to kern_mount() sysfs from sysfs_init()" prevents system from booting correctly

[Alexandre Demers]

So, I tried RC8 and I'm still getting the error. It start too fast to 
be able to read whatever it is saying, but here is what I can read:
...
cp: error writing /run/initramfs/new_root/...
cp: failed to extend ...
... (and so on)

It goes on like that for a long time.

Alexandre Demers

On Mon 17 Mar 2014 10:08:40 PM EDT, Alexandre Demers wrote:
> By the way, I tried to get the error message, but my display had gone
> in standby mode before it completed and it wouldn't wake up. I'll have
> to test with an earlier 3.14 kernel (rc1 or rc2), when it was
> answering to my wake up signal. Even though my display doesn't wake up
> with rc6, it is still running since I can see some activities on my
> disk from time to time.
>
> I'll be back with more info ASAP.
>
> Alexandre Demers
>
> On Mon 17 Mar 2014 04:13:46 PM EDT, Tejun Heo wrote:
>> Hello,
>>
>> On Fri, Mar 14, 2014 at 11:48:10AM -0400, Alexandre Demers wrote:
>>> Since it takes a while before the system begins to display errors,
>>> I'll have to test it and report it later today when I'll get back from
>>> work.
>>
>> Any update?
>>
>>> Now, about the userland part, this seems a broad question... I'm using
>>> Arch Linux 64bit (updated mostly on a daily basis). But I suspect you
>>> are expecting a narrower information. Is there anything in particular
>>> you want to know?
>>
>> I was just wondering whether you were running something really exotic.
>> Doesn't seem that way.  Weird that other people aren't hitting this
>> issue.  Hmm...
>>
>> Thanks.
>>

Re: [Regression, bisected 9e30cc] "sysfs, kernfs: no need to kern_mount() sysfs from sysfs_init()" prevents system from booting correctly

[Tejun Heo]

On Tue, Mar 25, 2014 at 09:03:51PM -0400, Alexandre Demers wrote:
> So, I tried RC8 and I'm still getting the error. It start too fast
> to be able to read whatever it is saying, but here is what I can
> read:
> ...
> cp: error writing /run/initramfs/new_root/...
> cp: failed to extend ...
> ... (and so on)
> 
> It goes on like that for a long time.

Ooh, I thought you were gonna find out what filled up the filesystem
after rebooting into the older kernel.  Can you please do that?  Also,
pressing "ScrLk" should stop the screen from scrolling.

Thanks.

-- 
tejun

Re: [Regression, bisected 9e30cc] "sysfs, kernfs: no need to kern_mount() sysfs from sysfs_init()" prevents system from booting correctly

[Alexandre Demers]

I'll do my best, but I just don't have enough time right now for
everything I have to do at home and dig this bug. I may be able to
look at it in the next couple of days though, or it may go somewhere
next week... That being said, I tried yesterday to have a better idea
and I thank you for the "ScrLck" trick, I didn't know that it was
working. Nevertheless I didn't find anything.

It seems as if the transition from the temporary kernel's fs to the
hard disk's fs is not done correctly. If I boot with a working kernel
after that, I can't find any trace of where it was trying to copy the
files or of any partition being full. Could it be filling the RAM
until it reaches its maximum capacity (16GB to fill before hitting the
limit and throwing the errors)? That would explain at least why it
takes so much time before the flood of errors begins...

On Wed, Mar 26, 2014 at 9:18 AM, Tejun Heo <tj@kernel.org> wrote:
> On Tue, Mar 25, 2014 at 09:03:51PM -0400, Alexandre Demers wrote:
>> So, I tried RC8 and I'm still getting the error. It start too fast
>> to be able to read whatever it is saying, but here is what I can
>> read:
>> ...
>> cp: error writing /run/initramfs/new_root/...
>> cp: failed to extend ...
>> ... (and so on)
>>
>> It goes on like that for a long time.
>
> Ooh, I thought you were gonna find out what filled up the filesystem
> after rebooting into the older kernel.  Can you please do that?  Also,
> pressing "ScrLk" should stop the screen from scrolling.
>
> Thanks.
>
> --
> tejun

Re: [Regression, bisected 9e30cc] "sysfs, kernfs: no need to kern_mount() sysfs from sysfs_init()" prevents system from booting correctly

[Tejun Heo]

Hello, Alexander.

On Thu, Mar 27, 2014 at 01:47:53PM -0400, Alexandre Demers wrote:
> I'll do my best, but I just don't have enough time right now for
> everything I have to do at home and dig this bug. I may be able to
> look at it in the next couple of days though, or it may go somewhere
> next week... That being said, I tried yesterday to have a better idea
> and I thank you for the "ScrLck" trick, I didn't know that it was
> working. Nevertheless I didn't find anything.
>
> It seems as if the transition from the temporary kernel's fs to the
> hard disk's fs is not done correctly. If I boot with a working kernel
> after that, I can't find any trace of where it was trying to copy the
> files or of any partition being full. Could it be filling the RAM
> until it reaches its maximum capacity (16GB to fill before hitting the
> limit and throwing the errors)? That would explain at least why it
> takes so much time before the flood of errors begins...

That could definitely be the case.  I'd really like to get to the
bottom of it.  It's really curious that there aren't more people
reporting the same problem.  Can you please try whether the following
patch makes the regression go away?

Thanks!

diff --git a/fs/sysfs/mount.c b/fs/sysfs/mount.c
index a66ad61..a3223e2 100644
--- a/fs/sysfs/mount.c
+++ b/fs/sysfs/mount.c
@@ -75,5 +75,7 @@ int __init sysfs_init(void)
 		return err;
 	}
 
+	WARN_ON(IS_ERR(kern_mount(&sysfs_fs_type)));
+
 	return 0;
 }

Re: [Regression, bisected 9e30cc] "sysfs, kernfs: no need to kern_mount() sysfs from sysfs_init()" prevents system from booting correctly

[Alexandre Demers]

That fixes the problem, I'm now running 3.14-rc8 with your patch 
applied.

Alexandre Demers

On Sat 29 Mar 2014 02:12:58 PM EDT, Tejun Heo wrote:
> Hello, Alexander.
>
> On Thu, Mar 27, 2014 at 01:47:53PM -0400, Alexandre Demers wrote:
>> I'll do my best, but I just don't have enough time right now for
>> everything I have to do at home and dig this bug. I may be able to
>> look at it in the next couple of days though, or it may go somewhere
>> next week... That being said, I tried yesterday to have a better idea
>> and I thank you for the "ScrLck" trick, I didn't know that it was
>> working. Nevertheless I didn't find anything.
>>
>> It seems as if the transition from the temporary kernel's fs to the
>> hard disk's fs is not done correctly. If I boot with a working kernel
>> after that, I can't find any trace of where it was trying to copy the
>> files or of any partition being full. Could it be filling the RAM
>> until it reaches its maximum capacity (16GB to fill before hitting the
>> limit and throwing the errors)? That would explain at least why it
>> takes so much time before the flood of errors begins...
>
> That could definitely be the case.  I'd really like to get to the
> bottom of it.  It's really curious that there aren't more people
> reporting the same problem.  Can you please try whether the following
> patch makes the regression go away?
>
> Thanks!
>
> diff --git a/fs/sysfs/mount.c b/fs/sysfs/mount.c
> index a66ad61..a3223e2 100644
> --- a/fs/sysfs/mount.c
> +++ b/fs/sysfs/mount.c
> @@ -75,5 +75,7 @@ int __init sysfs_init(void)
>   		return err;
>   	}
>
> +	WARN_ON(IS_ERR(kern_mount(&sysfs_fs_type)));
> +
>   	return 0;
>   }

Re: [Regression, bisected 9e30cc] "sysfs, kernfs: no need to kern_mount() sysfs from sysfs_init()" prevents system from booting correctly

[Tejun Heo]

On Sat, Mar 29, 2014 at 06:08:30PM -0400, Alexandre Demers wrote:
> That fixes the problem, I'm now running 3.14-rc8 with your patch
> applied.

Just sent out the patch but I think we still need to find out what's
going on.  Is there any chance you can create a qemu image which shows
the same behavior?

Thanks.

-- 
tejun

Re: [Regression, bisected 9e30cc] "sysfs, kernfs: no need to kern_mount() sysfs from sysfs_init()" prevents system from booting correctly

[Alexandre Demers]

Oh boy... Maybe. I'll look at this option this week.

Alexandre Demers

On Sun 30 Mar 2014 08:31:28 AM EDT, Tejun Heo wrote:
> On Sat, Mar 29, 2014 at 06:08:30PM -0400, Alexandre Demers wrote:
>> That fixes the problem, I'm now running 3.14-rc8 with your patch
>> applied.
>
> Just sent out the patch but I think we still need to find out what's
> going on.  Is there any chance you can create a qemu image which shows
> the same behavior?
>
> Thanks.
>

Re: [Regression, bisected 9e30cc] "sysfs, kernfs: no need to kern_mount() sysfs from sysfs_init()" prevents system from booting correctly

[Alexandre Demers]

I should have a qemu image running before the end of the week based on 
the latest Archlinux 64 packages. Then, I'll install the 3.14 kernel 
and see what I'll get.

By the way, since kernel 3.14 is now available, it has been pushed to 
Arch's repository. It will be deployed soon on many systems when 
they'll be updated. As you may have seen from the latest comments in 
bug 72061, another Arch user has reported today encountering the same 
problem as myself.

Alexandre Demers

On Sun 30 Mar 2014 12:53:56 PM EDT, Alexandre Demers wrote:
> Oh boy... Maybe. I'll look at this option this week.
>
> Alexandre Demers
>
> On Sun 30 Mar 2014 08:31:28 AM EDT, Tejun Heo wrote:
>> On Sat, Mar 29, 2014 at 06:08:30PM -0400, Alexandre Demers wrote:
>>> That fixes the problem, I'm now running 3.14-rc8 with your patch
>>> applied.
>>
>> Just sent out the patch but I think we still need to find out what's
>> going on.  Is there any chance you can create a qemu image which shows
>> the same behavior?
>>
>> Thanks.
>>