ramdisk problems in 4.0-rc1? (was Re: [PATCH 1/4] xfs/104: log size too small for 4k sector drives)

ramdisk problems in 4.0-rc1? (was Re: [PATCH 1/4] xfs/104: log size too small for 4k sector drives)

[Dave Chinner]

[cc linux-fsdevel, Boaz and others]

On Wed, Feb 25, 2015 at 11:11:51AM -0500, Brian Foster wrote:
> On Wed, Feb 25, 2015 at 09:54:36AM +1100, Dave Chinner wrote:
> > From: Dave Chinner <dchinner@redhat.com>
> > 
> > xfs/104, xfs/119, xfs/291 and xfs/297 have small fixed log sizes. A
> > recent change to the kernel ramdisk changed it's physical sector
> > size from 512B to 4kB, and this results in mkfs calculating a log
> > size larger than the fixed test size and hence the tests fail.
> > 
> > Change the log size to a larger size that works with 4k sectors, and
> > also increase the size of the filesystem being created so that the
> > amount of data space in the filesystem does not change and hence
> > does not perturb the rest of the test.
> > 
> > Signed-off-by: Dave Chinner <dchinner@redhat.com>
> > ---
> 
> Well for some reason I can't mount a ramdisk on the current tot to test
> this. In fact, I can't mount _anything_ after the ramdisk mount attempt.
> The mount actually reports success too, but there's nothing there... :/
> 
> # modprobe brd
> # mkfs.xfs -f /dev/ram0 
> meta-data=/dev/ram0              isize=256    agcount=1, agsize=4096
> blks
>          =                       sectsz=4096  attr=2, projid32bit=1
>          =                       crc=0        finobt=0
> data     =                       bsize=4096   blocks=4096, imaxpct=25
>          =                       sunit=0      swidth=0 blks
> naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
> log      =internal log           bsize=4096   blocks=1605, version=2
>          =                       sectsz=4096  sunit=1 blks, lazy-count=1
> realtime =none                   extsz=4096   blocks=0, rtextents=0
> # mount /dev/ram0 /mnt/
> # mount | grep mnt
> # umount  /mnt/
> umount: /mnt/: not mounted
> 
> ... and then I can't even mount my normal scratch device until after a
> reboot:
> 
> # mount /dev/test/scratch /mnt/
> # mount | grep mnt
> # umount  /mnt/
> umount: /mnt/: not mounted

Ok, so that's just plain broken. What's in dmesg?

As it is, I'm seeing plenty of weirdness in 4.0-rc1 on ramdisks as
well. Apart from the change to 4k physical sector size causing all
sorts of chaos with xfstests results due to it changing mkfs.xfs
behaviour, I'm also seeing this happen randomly:

....
Feb 25 11:48:35 test4 dave: run xfstest generic/083
Feb 25 11:48:37 test4 kernel: [ 8732.316223] XFS (ram1): Mounting V5 Filesystem
Feb 25 11:48:37 test4 kernel: [ 8732.318904] XFS (ram1): Ending clean mount
Feb 25 11:48:40 test4 kernel: [ 8735.871968] XFS (ram1): Unmounting Filesystem
Feb 25 11:48:40 test4 kernel: [ 8735.930160]  ram1: [POWERTEC] p1 p2 p3 p4 p5 p6 p7
Feb 25 11:48:40 test4 kernel: [ 8735.932081] ram1: p2 start 3158599292 is beyond EOD, truncated
Feb 25 11:48:40 test4 kernel: [ 8735.933983] ram1: p3 size 1627389952 extends beyond EOD, truncated
Feb 25 11:48:40 test4 kernel: [ 8735.936177] ram1: p4 size 1158021120 extends beyond EOD, truncated
Feb 25 11:48:40 test4 kernel: [ 8735.938269] ram1: p5 start 50924556 is beyond EOD, truncated
Feb 25 11:48:40 test4 kernel: [ 8735.940103] ram1: p6 size 67108864 extends beyond EOD, truncated
Feb 25 11:48:40 test4 kernel: [ 8735.942101] ram1: p7 start 4294967295 is beyond EOD, truncated
Feb 25 11:48:40 test4 dave: run xfstest generic/088
....

Something is causing partition rescans on ram devices that don't
have partitions, and this is new behaviour. Boaz, your commit
937af5ecd05 ("brd: Fix all partitions BUGs") seems the likely cause
of this problem I'm seeing - looks likea behaviour regression to
me as no other block device I have on any machine running the
same kernel throws these strange warnings from partition probing...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: ramdisk problems in 4.0-rc1? (was Re: [PATCH 1/4] xfs/104: log size too small for 4k sector drives)

[Brian Foster]

On Thu, Feb 26, 2015 at 09:32:48AM +1100, Dave Chinner wrote:
> [cc linux-fsdevel, Boaz and others]
> 
> On Wed, Feb 25, 2015 at 11:11:51AM -0500, Brian Foster wrote:
> > On Wed, Feb 25, 2015 at 09:54:36AM +1100, Dave Chinner wrote:
> > > From: Dave Chinner <dchinner@redhat.com>
> > > 
> > > xfs/104, xfs/119, xfs/291 and xfs/297 have small fixed log sizes. A
> > > recent change to the kernel ramdisk changed it's physical sector
> > > size from 512B to 4kB, and this results in mkfs calculating a log
> > > size larger than the fixed test size and hence the tests fail.
> > > 
> > > Change the log size to a larger size that works with 4k sectors, and
> > > also increase the size of the filesystem being created so that the
> > > amount of data space in the filesystem does not change and hence
> > > does not perturb the rest of the test.
> > > 
> > > Signed-off-by: Dave Chinner <dchinner@redhat.com>
> > > ---
> > 
> > Well for some reason I can't mount a ramdisk on the current tot to test
> > this. In fact, I can't mount _anything_ after the ramdisk mount attempt.
> > The mount actually reports success too, but there's nothing there... :/
> > 
> > # modprobe brd
> > # mkfs.xfs -f /dev/ram0 
> > meta-data=/dev/ram0              isize=256    agcount=1, agsize=4096
> > blks
> >          =                       sectsz=4096  attr=2, projid32bit=1
> >          =                       crc=0        finobt=0
> > data     =                       bsize=4096   blocks=4096, imaxpct=25
> >          =                       sunit=0      swidth=0 blks
> > naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
> > log      =internal log           bsize=4096   blocks=1605, version=2
> >          =                       sectsz=4096  sunit=1 blks, lazy-count=1
> > realtime =none                   extsz=4096   blocks=0, rtextents=0
> > # mount /dev/ram0 /mnt/
> > # mount | grep mnt
> > # umount  /mnt/
> > umount: /mnt/: not mounted
> > 
> > ... and then I can't even mount my normal scratch device until after a
> > reboot:
> > 
> > # mount /dev/test/scratch /mnt/
> > # mount | grep mnt
> > # umount  /mnt/
> > umount: /mnt/: not mounted
> 
> Ok, so that's just plain broken. What's in dmesg?
> 

Once I got back to this I found that for some reason systemd is
immediately invoking a umount on the mount. :/ No idea why or how to
stop it, but if I do something like this:

mount /dev/ram0 /mnt; cd /mnt

... I can occasionally win the race and get systemd to spin in a
umount() cycle trying to undo the mount. I haven't gone back to confirm
it's the same behavior with the normal devices at that point, but I
suspect it is, perhaps due to getting into some kind of bad state.

So fyi that this particular problem doesn't appear to be directly kernel
related...

Brian

> As it is, I'm seeing plenty of weirdness in 4.0-rc1 on ramdisks as
> well. Apart from the change to 4k physical sector size causing all
> sorts of chaos with xfstests results due to it changing mkfs.xfs
> behaviour, I'm also seeing this happen randomly:
> 
> ....
> Feb 25 11:48:35 test4 dave: run xfstest generic/083
> Feb 25 11:48:37 test4 kernel: [ 8732.316223] XFS (ram1): Mounting V5 Filesystem
> Feb 25 11:48:37 test4 kernel: [ 8732.318904] XFS (ram1): Ending clean mount
> Feb 25 11:48:40 test4 kernel: [ 8735.871968] XFS (ram1): Unmounting Filesystem
> Feb 25 11:48:40 test4 kernel: [ 8735.930160]  ram1: [POWERTEC] p1 p2 p3 p4 p5 p6 p7
> Feb 25 11:48:40 test4 kernel: [ 8735.932081] ram1: p2 start 3158599292 is beyond EOD, truncated
> Feb 25 11:48:40 test4 kernel: [ 8735.933983] ram1: p3 size 1627389952 extends beyond EOD, truncated
> Feb 25 11:48:40 test4 kernel: [ 8735.936177] ram1: p4 size 1158021120 extends beyond EOD, truncated
> Feb 25 11:48:40 test4 kernel: [ 8735.938269] ram1: p5 start 50924556 is beyond EOD, truncated
> Feb 25 11:48:40 test4 kernel: [ 8735.940103] ram1: p6 size 67108864 extends beyond EOD, truncated
> Feb 25 11:48:40 test4 kernel: [ 8735.942101] ram1: p7 start 4294967295 is beyond EOD, truncated
> Feb 25 11:48:40 test4 dave: run xfstest generic/088
> ....
> 
> Something is causing partition rescans on ram devices that don't
> have partitions, and this is new behaviour. Boaz, your commit
> 937af5ecd05 ("brd: Fix all partitions BUGs") seems the likely cause
> of this problem I'm seeing - looks likea behaviour regression to
> me as no other block device I have on any machine running the
> same kernel throws these strange warnings from partition probing...
> 
> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> david@fromorbit.com

Re: ramdisk problems in 4.0-rc1? (was Re: [PATCH 1/4] xfs/104: log size too small for 4k sector drives)

[Dave Chinner]

On Wed, Feb 25, 2015 at 06:31:15PM -0500, Brian Foster wrote:
> On Thu, Feb 26, 2015 at 09:32:48AM +1100, Dave Chinner wrote:
> > [cc linux-fsdevel, Boaz and others]
> > 
> > On Wed, Feb 25, 2015 at 11:11:51AM -0500, Brian Foster wrote:
> > > On Wed, Feb 25, 2015 at 09:54:36AM +1100, Dave Chinner wrote:
> > > > From: Dave Chinner <dchinner@redhat.com>
> > > > 
> > > > xfs/104, xfs/119, xfs/291 and xfs/297 have small fixed log sizes. A
> > > > recent change to the kernel ramdisk changed it's physical sector
> > > > size from 512B to 4kB, and this results in mkfs calculating a log
> > > > size larger than the fixed test size and hence the tests fail.
> > > > 
> > > > Change the log size to a larger size that works with 4k sectors, and
> > > > also increase the size of the filesystem being created so that the
> > > > amount of data space in the filesystem does not change and hence
> > > > does not perturb the rest of the test.
> > > > 
> > > > Signed-off-by: Dave Chinner <dchinner@redhat.com>
> > > > ---
> > > 
> > > Well for some reason I can't mount a ramdisk on the current tot to test
> > > this. In fact, I can't mount _anything_ after the ramdisk mount attempt.
> > > The mount actually reports success too, but there's nothing there... :/
> > > 
> > > # modprobe brd
> > > # mkfs.xfs -f /dev/ram0 
> > > meta-data=/dev/ram0              isize=256    agcount=1, agsize=4096
> > > blks
> > >          =                       sectsz=4096  attr=2, projid32bit=1
> > >          =                       crc=0        finobt=0
> > > data     =                       bsize=4096   blocks=4096, imaxpct=25
> > >          =                       sunit=0      swidth=0 blks
> > > naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
> > > log      =internal log           bsize=4096   blocks=1605, version=2
> > >          =                       sectsz=4096  sunit=1 blks, lazy-count=1
> > > realtime =none                   extsz=4096   blocks=0, rtextents=0
> > > # mount /dev/ram0 /mnt/
> > > # mount | grep mnt
> > > # umount  /mnt/
> > > umount: /mnt/: not mounted
> > > 
> > > ... and then I can't even mount my normal scratch device until after a
> > > reboot:
> > > 
> > > # mount /dev/test/scratch /mnt/
> > > # mount | grep mnt
> > > # umount  /mnt/
> > > umount: /mnt/: not mounted
> > 
> > Ok, so that's just plain broken. What's in dmesg?
> > 
> 
> Once I got back to this I found that for some reason systemd is
> immediately invoking a umount on the mount. :/ No idea why or how to
> stop it, but if I do something like this:
> 
> mount /dev/ram0 /mnt; cd /mnt
> 
> ... I can occasionally win the race and get systemd to spin in a
> umount() cycle trying to undo the mount. I haven't gone back to confirm
> it's the same behavior with the normal devices at that point, but I
> suspect it is, perhaps due to getting into some kind of bad state.
> 
> So fyi that this particular problem doesn't appear to be directly kernel
> related...

It may still be related to the kernel changes  e.g. by triggering
udev events when they didn't previously. The only machine I have
that is triggering the partition probing is also the only test
machine that I have that runs systemd and it didn't have this
problem on 3.19.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: ramdisk problems in 4.0-rc1? (was Re: [PATCH 1/4] xfs/104: log size too small for 4k sector drives)

[Boaz Harrosh]

On 02/26/2015 01:43 AM, Dave Chinner wrote:
> On Wed, Feb 25, 2015 at 06:31:15PM -0500, Brian Foster wrote:
>> On Thu, Feb 26, 2015 at 09:32:48AM +1100, Dave Chinner wrote:
>>> [cc linux-fsdevel, Boaz and others]
>>>
>>> On Wed, Feb 25, 2015 at 11:11:51AM -0500, Brian Foster wrote:
>>>> On Wed, Feb 25, 2015 at 09:54:36AM +1100, Dave Chinner wrote:
>>>>> From: Dave Chinner <dchinner@redhat.com>
>>>>>
>>>>> xfs/104, xfs/119, xfs/291 and xfs/297 have small fixed log sizes. A
>>>>> recent change to the kernel ramdisk changed it's physical sector
>>>>> size from 512B to 4kB, and this results in mkfs calculating a log
>>>>> size larger than the fixed test size and hence the tests fail.
>>>>>
>>>>> Change the log size to a larger size that works with 4k sectors, and
>>>>> also increase the size of the filesystem being created so that the
>>>>> amount of data space in the filesystem does not change and hence
>>>>> does not perturb the rest of the test.
>>>>>
>>>>> Signed-off-by: Dave Chinner <dchinner@redhat.com>
>>>>> ---
>>>>
>>>> Well for some reason I can't mount a ramdisk on the current tot to test
>>>> this. In fact, I can't mount _anything_ after the ramdisk mount attempt.
>>>> The mount actually reports success too, but there's nothing there... :/
>>>>
>>>> # modprobe brd
>>>> # mkfs.xfs -f /dev/ram0 
>>>> meta-data=/dev/ram0              isize=256    agcount=1, agsize=4096
>>>> blks
>>>>          =                       sectsz=4096  attr=2, projid32bit=1
>>>>          =                       crc=0        finobt=0
>>>> data     =                       bsize=4096   blocks=4096, imaxpct=25
>>>>          =                       sunit=0      swidth=0 blks
>>>> naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
>>>> log      =internal log           bsize=4096   blocks=1605, version=2
>>>>          =                       sectsz=4096  sunit=1 blks, lazy-count=1
>>>> realtime =none                   extsz=4096   blocks=0, rtextents=0
>>>> # mount /dev/ram0 /mnt/
>>>> # mount | grep mnt
>>>> # umount  /mnt/
>>>> umount: /mnt/: not mounted
>>>>
>>>> ... and then I can't even mount my normal scratch device until after a
>>>> reboot:
>>>>
>>>> # mount /dev/test/scratch /mnt/
>>>> # mount | grep mnt
>>>> # umount  /mnt/
>>>> umount: /mnt/: not mounted
>>>
>>> Ok, so that's just plain broken. What's in dmesg?
>>>
>>
>> Once I got back to this I found that for some reason systemd is
>> immediately invoking a umount on the mount. :/ No idea why or how to
>> stop it, but if I do something like this:
>>
>> mount /dev/ram0 /mnt; cd /mnt
>>
>> ... I can occasionally win the race and get systemd to spin in a
>> umount() cycle trying to undo the mount. I haven't gone back to confirm
>> it's the same behavior with the normal devices at that point, but I
>> suspect it is, perhaps due to getting into some kind of bad state.
>>
>> So fyi that this particular problem doesn't appear to be directly kernel
>> related...
> 
> It may still be related to the kernel changes  e.g. by triggering
> udev events when they didn't previously. The only machine I have
> that is triggering the partition probing is also the only test
> machine that I have that runs systemd and it didn't have this
> problem on 3.19.
> 

Sigh, thanks Dave. Yes you are correct my patch enabled the
udev events, as part of fixing ramdisk with partitions.
This is because if you do not enable them then mount by UUID
and all sort of lsblk and friends do not work.

I did try to test this in all kind of ways, xfstest+ext4
as well, and ran with it on Fedora 20 for a while, sorry
about that.

It looks like the system anticipates that ramdisk "should
not have these events"

I will send a patch ASAP that re-instates the module_parameter
for enabling notification, and leaving the default off. It should
be easy to set the param if one intends to use these utilities.

That said, please do agree with me that there is brokenness in
systemd?

BTW: You also said something about the 4k sectors thing, It looks
like we are pulled in two different directions here. If you will
want to use DAX on ramdisk then you want it on, if you are not
using DAX, and wants to use smaller-then-page_size FS blocks than
you do not want it.

Please advise what we should do? Maybe only do 4k if BLK_DEV_RAM_DAX
is set in Kconfig ?


Sorry for the mess, I'll send a fix ASAP

> Cheers,
> Dave.
> 

Thanks
Boaz

Re: ramdisk problems in 4.0-rc1? (was Re: [PATCH 1/4] xfs/104: log size too small for 4k sector drives)

[Brian Foster]

On Thu, Feb 26, 2015 at 09:46:27AM +0200, Boaz Harrosh wrote:
> On 02/26/2015 01:43 AM, Dave Chinner wrote:
> > On Wed, Feb 25, 2015 at 06:31:15PM -0500, Brian Foster wrote:
> >> On Thu, Feb 26, 2015 at 09:32:48AM +1100, Dave Chinner wrote:
> >>> [cc linux-fsdevel, Boaz and others]
> >>>
> >>> On Wed, Feb 25, 2015 at 11:11:51AM -0500, Brian Foster wrote:
> >>>> On Wed, Feb 25, 2015 at 09:54:36AM +1100, Dave Chinner wrote:
> >>>>> From: Dave Chinner <dchinner@redhat.com>
> >>>>>
> >>>>> xfs/104, xfs/119, xfs/291 and xfs/297 have small fixed log sizes. A
> >>>>> recent change to the kernel ramdisk changed it's physical sector
> >>>>> size from 512B to 4kB, and this results in mkfs calculating a log
> >>>>> size larger than the fixed test size and hence the tests fail.
> >>>>>
> >>>>> Change the log size to a larger size that works with 4k sectors, and
> >>>>> also increase the size of the filesystem being created so that the
> >>>>> amount of data space in the filesystem does not change and hence
> >>>>> does not perturb the rest of the test.
> >>>>>
> >>>>> Signed-off-by: Dave Chinner <dchinner@redhat.com>
> >>>>> ---
> >>>>
> >>>> Well for some reason I can't mount a ramdisk on the current tot to test
> >>>> this. In fact, I can't mount _anything_ after the ramdisk mount attempt.
> >>>> The mount actually reports success too, but there's nothing there... :/
> >>>>
> >>>> # modprobe brd
> >>>> # mkfs.xfs -f /dev/ram0 
> >>>> meta-data=/dev/ram0              isize=256    agcount=1, agsize=4096
> >>>> blks
> >>>>          =                       sectsz=4096  attr=2, projid32bit=1
> >>>>          =                       crc=0        finobt=0
> >>>> data     =                       bsize=4096   blocks=4096, imaxpct=25
> >>>>          =                       sunit=0      swidth=0 blks
> >>>> naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
> >>>> log      =internal log           bsize=4096   blocks=1605, version=2
> >>>>          =                       sectsz=4096  sunit=1 blks, lazy-count=1
> >>>> realtime =none                   extsz=4096   blocks=0, rtextents=0
> >>>> # mount /dev/ram0 /mnt/
> >>>> # mount | grep mnt
> >>>> # umount  /mnt/
> >>>> umount: /mnt/: not mounted
> >>>>
> >>>> ... and then I can't even mount my normal scratch device until after a
> >>>> reboot:
> >>>>
> >>>> # mount /dev/test/scratch /mnt/
> >>>> # mount | grep mnt
> >>>> # umount  /mnt/
> >>>> umount: /mnt/: not mounted
> >>>
> >>> Ok, so that's just plain broken. What's in dmesg?
> >>>
> >>
> >> Once I got back to this I found that for some reason systemd is
> >> immediately invoking a umount on the mount. :/ No idea why or how to
> >> stop it, but if I do something like this:
> >>
> >> mount /dev/ram0 /mnt; cd /mnt
> >>
> >> ... I can occasionally win the race and get systemd to spin in a
> >> umount() cycle trying to undo the mount. I haven't gone back to confirm
> >> it's the same behavior with the normal devices at that point, but I
> >> suspect it is, perhaps due to getting into some kind of bad state.
> >>
> >> So fyi that this particular problem doesn't appear to be directly kernel
> >> related...
> > 
> > It may still be related to the kernel changes  e.g. by triggering
> > udev events when they didn't previously. The only machine I have
> > that is triggering the partition probing is also the only test
> > machine that I have that runs systemd and it didn't have this
> > problem on 3.19.
> > 
> 
> Sigh, thanks Dave. Yes you are correct my patch enabled the
> udev events, as part of fixing ramdisk with partitions.
> This is because if you do not enable them then mount by UUID
> and all sort of lsblk and friends do not work.
> 
> I did try to test this in all kind of ways, xfstest+ext4
> as well, and ran with it on Fedora 20 for a while, sorry
> about that.
> 
> It looks like the system anticipates that ramdisk "should
> not have these events"
> 
> I will send a patch ASAP that re-instates the module_parameter
> for enabling notification, and leaving the default off. It should
> be easy to set the param if one intends to use these utilities.
> 

Thanks Boaz, but I still see the same behavior with the part_show patch.
It seems to be something that broke in systemd on Fedora between
versions systemd-218 and systemd-219. The latter is broken on a 3.19
kernel as well.

I've filed a systemd bug so we'll see what comes of it from that end:

https://bugzilla.redhat.com/show_bug.cgi?id=1196452

Brian

> That said, please do agree with me that there is brokenness in
> systemd?
> 
> BTW: You also said something about the 4k sectors thing, It looks
> like we are pulled in two different directions here. If you will
> want to use DAX on ramdisk then you want it on, if you are not
> using DAX, and wants to use smaller-then-page_size FS blocks than
> you do not want it.
> 
> Please advise what we should do? Maybe only do 4k if BLK_DEV_RAM_DAX
> is set in Kconfig ?
> 
> 
> Sorry for the mess, I'll send a fix ASAP
> 
> > Cheers,
> > Dave.
> > 
> 
> Thanks
> Boaz
> 

Re: ramdisk problems in 4.0-rc1? (was Re: [PATCH 1/4] xfs/104: log size too small for 4k sector drives)

[Boaz Harrosh]

On 02/26/2015 07:23 PM, Brian Foster wrote:
<>
> 
> Thanks Boaz, but I still see the same behavior with the part_show patch.
> It seems to be something that broke in systemd on Fedora between
> versions systemd-218 and systemd-219. The latter is broken on a 3.19
> kernel as well.
> 
> I've filed a systemd bug so we'll see what comes of it from that end:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1196452
> 
> Brian
> 

Hi Brian

It says in bugzilla (link above) that this issue is "fixed in git" so
I guess we should be fine ?

Jens does *not* need to take
	[PATCH] brd: Re-instate ram disk visibility option (part_show)

Please confirm.

Please tell me if there is anything I can help with?

Thanks
Boaz

Re: ramdisk problems in 4.0-rc1? (was Re: [PATCH 1/4] xfs/104: log size too small for 4k sector drives)

[Brian Foster]

On Sun, Mar 01, 2015 at 10:49:16AM +0200, Boaz Harrosh wrote:
> On 02/26/2015 07:23 PM, Brian Foster wrote:
> <>
> > 
> > Thanks Boaz, but I still see the same behavior with the part_show patch.
> > It seems to be something that broke in systemd on Fedora between
> > versions systemd-218 and systemd-219. The latter is broken on a 3.19
> > kernel as well.
> > 
> > I've filed a systemd bug so we'll see what comes of it from that end:
> > 
> > https://bugzilla.redhat.com/show_bug.cgi?id=1196452
> > 
> > Brian
> > 
> 
> Hi Brian
> 
> It says in bugzilla (link above) that this issue is "fixed in git" so
> I guess we should be fine ?
> 

Yes, I picked up a more recent systemd version and it seems to work fine
now without the patch referenced below.

Brian

> Jens does *not* need to take
> 	[PATCH] brd: Re-instate ram disk visibility option (part_show)
> 
> Please confirm.
> 
> Please tell me if there is anything I can help with?
> 
> Thanks
> Boaz
> 

Re: ramdisk problems in 4.0-rc1? (was Re: [PATCH 1/4] xfs/104: log size too small for 4k sector drives)

[Dave Chinner]

On Thu, Feb 26, 2015 at 09:46:27AM +0200, Boaz Harrosh wrote:
> On 02/26/2015 01:43 AM, Dave Chinner wrote:
> > On Wed, Feb 25, 2015 at 06:31:15PM -0500, Brian Foster wrote:
> >> On Thu, Feb 26, 2015 at 09:32:48AM +1100, Dave Chinner wrote:
> >>> [cc linux-fsdevel, Boaz and others]
> >>>
> >>> On Wed, Feb 25, 2015 at 11:11:51AM -0500, Brian Foster wrote:
> >>>> On Wed, Feb 25, 2015 at 09:54:36AM +1100, Dave Chinner wrote:
> >>>>> From: Dave Chinner <dchinner@redhat.com>
> >>>>>
> >>>>> xfs/104, xfs/119, xfs/291 and xfs/297 have small fixed log sizes. A
> >>>>> recent change to the kernel ramdisk changed it's physical sector
> >>>>> size from 512B to 4kB, and this results in mkfs calculating a log
> >>>>> size larger than the fixed test size and hence the tests fail.
> >>>>>
> >>>>> Change the log size to a larger size that works with 4k sectors, and
> >>>>> also increase the size of the filesystem being created so that the
> >>>>> amount of data space in the filesystem does not change and hence
> >>>>> does not perturb the rest of the test.
> >>>>>
> >>>>> Signed-off-by: Dave Chinner <dchinner@redhat.com>
> >>>>> ---
> >>>>
> >>>> Well for some reason I can't mount a ramdisk on the current tot to test
> >>>> this. In fact, I can't mount _anything_ after the ramdisk mount attempt.
> >>>> The mount actually reports success too, but there's nothing there... :/
> >>>>
> >>>> # modprobe brd
> >>>> # mkfs.xfs -f /dev/ram0 
> >>>> meta-data=/dev/ram0              isize=256    agcount=1, agsize=4096
> >>>> blks
> >>>>          =                       sectsz=4096  attr=2, projid32bit=1
> >>>>          =                       crc=0        finobt=0
> >>>> data     =                       bsize=4096   blocks=4096, imaxpct=25
> >>>>          =                       sunit=0      swidth=0 blks
> >>>> naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
> >>>> log      =internal log           bsize=4096   blocks=1605, version=2
> >>>>          =                       sectsz=4096  sunit=1 blks, lazy-count=1
> >>>> realtime =none                   extsz=4096   blocks=0, rtextents=0
> >>>> # mount /dev/ram0 /mnt/
> >>>> # mount | grep mnt
> >>>> # umount  /mnt/
> >>>> umount: /mnt/: not mounted
> >>>>
> >>>> ... and then I can't even mount my normal scratch device until after a
> >>>> reboot:
> >>>>
> >>>> # mount /dev/test/scratch /mnt/
> >>>> # mount | grep mnt
> >>>> # umount  /mnt/
> >>>> umount: /mnt/: not mounted
> >>>
> >>> Ok, so that's just plain broken. What's in dmesg?
> >>>
> >>
> >> Once I got back to this I found that for some reason systemd is
> >> immediately invoking a umount on the mount. :/ No idea why or how to
> >> stop it, but if I do something like this:
> >>
> >> mount /dev/ram0 /mnt; cd /mnt
> >>
> >> ... I can occasionally win the race and get systemd to spin in a
> >> umount() cycle trying to undo the mount. I haven't gone back to confirm
> >> it's the same behavior with the normal devices at that point, but I
> >> suspect it is, perhaps due to getting into some kind of bad state.
> >>
> >> So fyi that this particular problem doesn't appear to be directly kernel
> >> related...
> > 
> > It may still be related to the kernel changes  e.g. by triggering
> > udev events when they didn't previously. The only machine I have
> > that is triggering the partition probing is also the only test
> > machine that I have that runs systemd and it didn't have this
> > problem on 3.19.
> > 
> 
> Sigh, thanks Dave. Yes you are correct my patch enabled the
> udev events, as part of fixing ramdisk with partitions.
> This is because if you do not enable them then mount by UUID
> and all sort of lsblk and friends do not work.

Sure, that's what the gendisk abstraction just you. But why am I
seeing random partition probes on a ramdisk that *isn't using
partitions*?

> I did try to test this in all kind of ways, xfstest+ext4
> as well, and ran with it on Fedora 20 for a while, sorry
> about that.
> 
> It looks like the system anticipates that ramdisk "should
> not have these events"

Right, but not because it's a ramdisk. Those events should not be
occurring because I'm not creating or destroying devices, I'm not
changing partition tables, I'm not resizing ramdisks or partitions,
and so on. I'm simply mkfs'ing, mounting and unmounting filesystems
on the ramdisks - nothing should be generating device based udev
events...

Finding the trigger that is causing these events will tell us what
the bug is - restricting the config won't help, especially as DAX
will *always* be enabled on my test machines as it's something
needed in my test matrix. I'm not sure how to go about finding that
trigger right now and as such I won't really have time to look at it
until after lsfmm/vault...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: ramdisk problems in 4.0-rc1? (was Re: [PATCH 1/4] xfs/104: log size too small for 4k sector drives)

[Boaz Harrosh]

On 02/27/2015 02:58 AM, Dave Chinner wrote:
<>
>>
>> Sigh, thanks Dave. Yes you are correct my patch enabled the
>> udev events, as part of fixing ramdisk with partitions.
>> This is because if you do not enable them then mount by UUID
>> and all sort of lsblk and friends do not work.
> 
> Sure, that's what the gendisk abstraction just you. But why am I
> seeing random partition probes on a ramdisk that *isn't using
> partitions*?
> 

Yes, There should be the one new event on create (modprobe or
mknod) which was not there before. Perhaps it triggers a systemd
process that never used to run before. (And is now sitting there
and making a mess)

<>
>> It looks like the system anticipates that ramdisk "should
>> not have these events"
> 
> Right, but not because it's a ramdisk. Those events should not be
> occurring because I'm not creating or destroying devices, I'm not
> changing partition tables, I'm not resizing ramdisks or partitions,
> and so on. I'm simply mkfs'ing, mounting and unmounting filesystems
> on the ramdisks - nothing should be generating device based udev
> events...
> 
> Finding the trigger that is causing these events will tell us what
> the bug is - 

> restricting the config won't help, especially as DAX
> will *always* be enabled on my test machines as it's something
> needed in my test matrix.

No the "if DAX" is for the 4k thing. The enablement of the uevents is
with a new "part_show" module parameter (See patch-1).

> I'm not sure how to go about finding that
> trigger right now and as such I won't really have time to look at it
> until after lsfmm/vault...
> 

I'll try to reproduce this here. What Fedora version do I need?

> Cheers,
> Dave.
> 

Thanks
Boaz

Re: ramdisk problems in 4.0-rc1? (was Re: [PATCH 1/4] xfs/104: log size too small for 4k sector drives)

[Dave Chinner]

On Sun, Mar 01, 2015 at 10:27:49AM +0200, Boaz Harrosh wrote:
> On 02/27/2015 02:58 AM, Dave Chinner wrote:
> >> It looks like the system anticipates that ramdisk "should
> >> not have these events"
> > 
> > Right, but not because it's a ramdisk. Those events should not be
> > occurring because I'm not creating or destroying devices, I'm not
> > changing partition tables, I'm not resizing ramdisks or partitions,
> > and so on. I'm simply mkfs'ing, mounting and unmounting filesystems
> > on the ramdisks - nothing should be generating device based udev
> > events...
> > 
> > Finding the trigger that is causing these events will tell us what
> > the bug is - 
> 
> > restricting the config won't help, especially as DAX
> > will *always* be enabled on my test machines as it's something
> > needed in my test matrix.
> 
> No the "if DAX" is for the 4k thing. The enablement of the uevents is
> with a new "part_show" module parameter (See patch-1).

Sure, but that doesn't answer my question: what is generating device
level uevents when all I'm doing is mkfs/mount/umount on the device?

> > I'm not sure how to go about finding that
> > trigger right now and as such I won't really have time to look at it
> > until after lsfmm/vault...
> > 
> 
> I'll try to reproduce this here. What Fedora version do I need?

I'm running debian unstable w/ systemd-215 on the particular test
machine that is hitting this problem.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: ramdisk problems in 4.0-rc1? (was Re: [PATCH 1/4] xfs/104: log size too small for 4k sector drives)

[Boaz Harrosh]

On 03/02/2015 03:09 AM, Dave Chinner wrote:
> On Sun, Mar 01, 2015 at 10:27:49AM +0200, Boaz Harrosh wrote:
>> On 02/27/2015 02:58 AM, Dave Chinner wrote:
<>
>> No the "if DAX" is for the 4k thing. The enablement of the uevents is
>> with a new "part_show" module parameter (See patch-1).
> 
> Sure, but that doesn't answer my question: what is generating device
> level uevents when all I'm doing is mkfs/mount/umount on the device?
> 

I was suspecting it is this systemd bug which keeps trying to tier-down
the devices.

>>> I'm not sure how to go about finding that
>>> trigger right now and as such I won't really have time to look at it
>>> until after lsfmm/vault...
>>>
>>
>> I'll try to reproduce this here. What Fedora version do I need?
> 
> I'm running debian unstable w/ systemd-215 on the particular test
> machine that is hitting this problem.
> 

Oooff, On my fedora 20 I'm at systemd 208. I'll see if I have time to
install an fc21 vm or maybe upgrade from source. (Any easy way?)

I have setup my xfs rig and ran "./check -g auto" by now. I tried both
part_show=1/0 and both look working as expected.
(Do I need any special $MKFS_OPTIONS or anything else)

I'll probably be giving up soon, and will just wait for more reports.

With the patch-1 I sent I am reverting to old behavior so I need a
reproducer to try and run with patch-1 and part_show=1 should show
the problem and part_show=0 should not. Else this is something else

> Cheers,
> Dave.
> 

Thanks Dave, sorry for trapping you in this boring mess, life
at Kernel the one reproducing the problem needs to help fix it ;-)

Have an enjoyable and productive LSF
Thanks
Boaz

[PATCH] brd: Re-instate ram disk visibility option (part_show)

[Boaz Harrosh]

This re-enables part_show option, so we can keep it false by
default.

Here is what Dmitry said in the original patch:
  [aeac318] brd: add ram disk visibility option
	Currently ram disk is not visible inside /proc/partitions.
	This was done for compatibility reasons here: 53978d0a7a27.
	But some utilities expect disk presents in /proc/partitions.
	Let's add module's option and let's administrator chose
	visibility behavior. By default, old behavior preserved.

Dave Chinner and other have reported problems with current system
when udev events start firing for ramdisk:
	http://marc.info/?l=linux-fsdevel&m=142490357918672&w=2

It was me who enabled these notifications through this patch:
   [937af5e] brd: Fix all partitions BUGs
Sorry for the mess.

CC: Dave Chinner <david@fromorbit.com>
CC: Dmitry Monakhov <dmonakhov@openvz.org>
Signed-off-by: Boaz Harrosh <boaz@plexistor.com>
---
 drivers/block/brd.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/block/brd.c b/drivers/block/brd.c
index 64ab495..6e0775b 100644
--- a/drivers/block/brd.c
+++ b/drivers/block/brd.c
@@ -450,6 +450,10 @@ static int max_part = 1;
 module_param(max_part, int, S_IRUGO);
 MODULE_PARM_DESC(max_part, "Num Minors to reserve between devices");
 
+static int part_show = 0;
+module_param(part_show, int, S_IRUGO);
+MODULE_PARM_DESC(part_show, "Control RAM disk visibility in /proc/partitions");
+
 MODULE_LICENSE("GPL");
 MODULE_ALIAS_BLOCKDEV_MAJOR(RAMDISK_MAJOR);
 MODULE_ALIAS("rd");
@@ -513,6 +517,8 @@ static struct brd_device *brd_alloc(int i)
 	disk->private_data	= brd;
 	disk->queue		= brd->brd_queue;
 	disk->flags		= GENHD_FL_EXT_DEVT;
+	if (!part_show)
+		disk->flags |= GENHD_FL_SUPPRESS_PARTITION_INFO;
 	sprintf(disk->disk_name, "ram%d", i);
 	set_capacity(disk, rd_size * 2);
 
-- 
1.9.3

[PATCH] brd: Only request 4K sectors if DAX is enabled

[Boaz Harrosh]

People systems have been using ramdisk with
smaller-than-page-size blocks in their mkfs.

The 4K sectors is only important if we will be
using brd with a DAX filesystem.

So only enable 4K sectors if DAX is configured

Signed-off-by: Boaz Harrosh <boaz@plexistor.com>
---
 drivers/block/brd.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/block/brd.c b/drivers/block/brd.c
index 6e0775b..e875f12 100644
--- a/drivers/block/brd.c
+++ b/drivers/block/brd.c
@@ -495,6 +495,7 @@ static struct brd_device *brd_alloc(int i)
 	blk_queue_max_hw_sectors(brd->brd_queue, 1024);
 	blk_queue_bounce_limit(brd->brd_queue, BLK_BOUNCE_ANY);
 
+#ifdef CONFIG_BLK_DEV_RAM_DAX
 	/* This is so fdisk will align partitions on 4k, because of
 	 * direct_access API needing 4k alignment, returning a PFN
 	 * (This is only a problem on very small devices <= 4M,
@@ -502,6 +503,7 @@ static struct brd_device *brd_alloc(int i)
 	 *  is harmless)
 	 */
 	blk_queue_physical_block_size(brd->brd_queue, PAGE_SIZE);
+#endif
 
 	brd->brd_queue->limits.discard_granularity = PAGE_SIZE;
 	brd->brd_queue->limits.max_discard_sectors = UINT_MAX;
-- 
1.9.3

Re: [PATCH] brd: Only request 4K sectors if DAX is enabled

[Boaz Harrosh]

On 02/26/2015 10:41 AM, Boaz Harrosh wrote:
> 
> People systems have been using ramdisk with
> smaller-than-page-size blocks in their mkfs.
> 
> The 4K sectors is only important if we will be
> using brd with a DAX filesystem.
> 
> So only enable 4K sectors if DAX is configured
> 
> Signed-off-by: Boaz Harrosh <boaz@plexistor.com>
> ---
>  drivers/block/brd.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/drivers/block/brd.c b/drivers/block/brd.c
> index 6e0775b..e875f12 100644
> --- a/drivers/block/brd.c
> +++ b/drivers/block/brd.c
> @@ -495,6 +495,7 @@ static struct brd_device *brd_alloc(int i)
>  	blk_queue_max_hw_sectors(brd->brd_queue, 1024);
>  	blk_queue_bounce_limit(brd->brd_queue, BLK_BOUNCE_ANY);
>  
> +#ifdef CONFIG_BLK_DEV_RAM_DAX
>  	/* This is so fdisk will align partitions on 4k, because of
>  	 * direct_access API needing 4k alignment, returning a PFN
>  	 * (This is only a problem on very small devices <= 4M,
> @@ -502,6 +503,7 @@ static struct brd_device *brd_alloc(int i)
>  	 *  is harmless)
>  	 */

Hi Dave

I was just thinking should we also do

+	if (part_show)
>  		blk_queue_physical_block_size(brd->brd_queue, PAGE_SIZE);

The rational is that if part_show is off then we are not using partitions
at all and then there can be no problems?

I'll try to run your tests here. I never test xfs always ext4, maybe I should
start to ;-). Actually I do have xfsprogs so why not. But if you could run
a quick test as well it could be great.

Thanks
Boaz

<>

Re: [PATCH] brd: Only request 4K sectors if DAX is enabled

[Dave Chinner]

On Thu, Feb 26, 2015 at 10:41:46AM +0200, Boaz Harrosh wrote:
> 
> People systems have been using ramdisk with
> smaller-than-page-size blocks in their mkfs.
> 
> The 4K sectors is only important if we will be
> using brd with a DAX filesystem.
> 
> So only enable 4K sectors if DAX is configured

4k sectors are not a problem - we should be handling them fine and
because it makes the ramdisk look like a 512e drive, no applications
should fail, either.

The main "unexpected" part about it was how much of xfstests didn't
handle 4k sectors in mkfs output properly. This isn't a problem with
the kernel change and so doesn't need fixing.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [PATCH] brd: Only request 4K sectors if DAX is enabled

[Boaz Harrosh]

On 02/27/2015 02:23 AM, Dave Chinner wrote:
> On Thu, Feb 26, 2015 at 10:41:46AM +0200, Boaz Harrosh wrote:
>>
>> People systems have been using ramdisk with
>> smaller-than-page-size blocks in their mkfs.
>>
>> The 4K sectors is only important if we will be
>> using brd with a DAX filesystem.
>>
>> So only enable 4K sectors if DAX is configured
> 
> 4k sectors are not a problem - we should be handling them fine and
> because it makes the ramdisk look like a 512e drive, no applications
> should fail, either.
> 
> The main "unexpected" part about it was how much of xfstests didn't
> handle 4k sectors in mkfs output properly. This isn't a problem with
> the kernel change and so doesn't need fixing.
> 

Glad to be of service ;-)

Please tell me if there is anything I can help with

> Cheers,
> Dave.
> 

Thanks
Boaz