* Re: [PATCH 0/3] dm-crypt: Adds support for wiping key when doing suspend/hibernation
From: Mike Snitzer @ 2015-04-09 14:08 UTC (permalink / raw)
To: Pali Rohár
Cc: Alasdair Kergon, Neil Brown, Rafael J. Wysocki, Len Brown,
Pavel Machek, dm-devel, linux-raid, linux-kernel, linux-pm
In-Reply-To: <20150409132836.GD12339@atrey.karlin.mff.cuni.cz>
On Thu, Apr 09 2015 at 9:28am -0400,
Pali Rohár <pali.rohar@gmail.com> wrote:
> On Thursday 09 April 2015 09:12:08 Mike Snitzer wrote:
> > On Mon, Apr 06 2015 at 9:29am -0400,
> > Pali Rohár <pali.rohar@gmail.com> wrote:
> >
> > > On Monday 06 April 2015 15:00:46 Mike Snitzer wrote:
> > > > On Sun, Apr 05 2015 at 1:20pm -0400,
> > > >
> > > > Pali Rohár <pali.rohar@gmail.com> wrote:
> > > > > This patch series increase security of suspend and hibernate
> > > > > actions. It allows user to safely wipe crypto keys before
> > > > > suspend and hibernate actions starts without race
> > > > > conditions on userspace process with heavy I/O.
> > > > >
> > > > > To automatically wipe cryto key for <device> before
> > > > > hibernate action call: $ dmsetup message <device> 0 key
> > > > > wipe_on_hibernation 1
> > > > >
> > > > > To automatically wipe cryto key for <device> before suspend
> > > > > action call: $ dmsetup message <device> 0 key
> > > > > wipe_on_suspend 1
> > > > >
> > > > > (Value 0 after wipe_* string reverts original behaviour - to
> > > > > not wipe key)
> > > >
> > > > Can you elaborate on the attack vector your changes are meant
> > > > to protect against? The user already authorized access, why
> > > > is it inherently dangerous to _not_ wipe the associated key
> > > > across these events?
> > >
> > > Hi,
> > >
> > > yes, I will try to explain current problems with cryptsetup
> > > luksSuspend command and hibernation.
> > >
> > > First, sometimes it is needed to put machine into other hands.
> > > You can still watch other person what is doing with machine, but
> > > once if you let machine unlocked (e.g opened luks disk), she/he
> > > can access encrypted data.
> > >
> > > If you turn off machine, it could be safe, because luks disk
> > > devices are locked. But if you enter machine into suspend or
> > > hibernate state luks devices are still open. And my patches try
> > > to achieve similar security as when machine is off (= no crypto
> > > keys in RAM or on swap).
> > >
> > > When doing hibernate on unencrypted swap it is to prevent leaking
> > > crypto keys to hibernate image (which is stored in swap).
> > >
> > > When doing suspend action it is again to prevent leaking crypto
> > > keys. E.g when you suspend laptop and put it off (somebody can
> > > remove RAMs and do some cold boot attack).
> > >
> > > The most common situation is:
> > > You have mounted partition from dm-crypt device (e.g. /home/),
> > > some userspace processes access it (e.g opened firefox which
> > > still reads/writes to cache ~/.firefox/) and you want to drop
> > > crypto keys from kernel for some time.
> > >
> > > For that operation there is command cryptsetup luksSuspend, which
> > > suspend dm device and then tell kernel to wipe crypto keys. All
> > > I/O operations are then stopped and userspace processes which
> > > want to do some those I/O operations are stopped too (until you
> > > call cryptsetup luksResume and enter correct key).
> > >
> > > Now if you want to suspend/hiberate your machine (when some of dm
> > > devices are suspeneded and some processes are stopped due to
> > > pending I/O) it is not possible. Kernel freeze_processes function
> > > will fail because userspace processes are still stopped inside
> > > some I/O syscall (read/write, etc,...).
> > >
> > > My patches fixes this problem and do those operations (suspend dm
> > > device, wipe crypto keys, enter suspend/hiberate) in correct
> > > order and without race condition.
> > >
> > > dm device is suspended *after* userspace processes are freezed
> > > and after that are crypto keys wiped. And then computer/laptop
> > > enters into suspend/hibernate state.
> >
> > Wouldn't it be better to fix freeze_processes() to be tolerant of
> > processes that are hung as a side-effect of their backing storage being
> > suspended? A hibernate shouldn't fail simply because a user chose to
> > suspend a DM device.
> >
> > Then this entire problem goes away and the key can be wiped from
> > userspace (like you said above).
>
> Still there will be race condition. Before hibernation (and device
> poweroff) we should have synced disks and filesystems to prevent data
> lose (or other damage) as more as we can. And if there will be some
> application which using lot of I/O (e.g normal firefox) then there
> always will be race condtion.
The DM suspend will take care of flushing any pending I/O. So I don't
see where the supposed race is...
Anything else that is trapped in userspace memory will be there when the
machine resumes.
> So proper way is to wipe luks crypto keys *after* userspace processes
> are freezed.
I know you believe that I'm just not accepting that at face value.
^ permalink raw reply
* Re: [PATCH 0/3] dm-crypt: Adds support for wiping key when doing suspend/hibernation
From: Pali Rohár @ 2015-04-09 14:16 UTC (permalink / raw)
To: Mike Snitzer
Cc: Alasdair Kergon, Neil Brown, Rafael J. Wysocki, Len Brown,
Pavel Machek, dm-devel, linux-raid, linux-kernel, linux-pm
In-Reply-To: <20150409140842.GA9779@redhat.com>
On Thursday 09 April 2015 10:08:43 Mike Snitzer wrote:
> On Thu, Apr 09 2015 at 9:28am -0400,
> Pali Rohár <pali.rohar@gmail.com> wrote:
>
> > On Thursday 09 April 2015 09:12:08 Mike Snitzer wrote:
> > > On Mon, Apr 06 2015 at 9:29am -0400,
> > > Pali Rohár <pali.rohar@gmail.com> wrote:
> > >
> > > > On Monday 06 April 2015 15:00:46 Mike Snitzer wrote:
> > > > > On Sun, Apr 05 2015 at 1:20pm -0400,
> > > > >
> > > > > Pali Rohár <pali.rohar@gmail.com> wrote:
> > > > > > This patch series increase security of suspend and hibernate
> > > > > > actions. It allows user to safely wipe crypto keys before
> > > > > > suspend and hibernate actions starts without race
> > > > > > conditions on userspace process with heavy I/O.
> > > > > >
> > > > > > To automatically wipe cryto key for <device> before
> > > > > > hibernate action call: $ dmsetup message <device> 0 key
> > > > > > wipe_on_hibernation 1
> > > > > >
> > > > > > To automatically wipe cryto key for <device> before suspend
> > > > > > action call: $ dmsetup message <device> 0 key
> > > > > > wipe_on_suspend 1
> > > > > >
> > > > > > (Value 0 after wipe_* string reverts original behaviour - to
> > > > > > not wipe key)
> > > > >
> > > > > Can you elaborate on the attack vector your changes are meant
> > > > > to protect against? The user already authorized access, why
> > > > > is it inherently dangerous to _not_ wipe the associated key
> > > > > across these events?
> > > >
> > > > Hi,
> > > >
> > > > yes, I will try to explain current problems with cryptsetup
> > > > luksSuspend command and hibernation.
> > > >
> > > > First, sometimes it is needed to put machine into other hands.
> > > > You can still watch other person what is doing with machine, but
> > > > once if you let machine unlocked (e.g opened luks disk), she/he
> > > > can access encrypted data.
> > > >
> > > > If you turn off machine, it could be safe, because luks disk
> > > > devices are locked. But if you enter machine into suspend or
> > > > hibernate state luks devices are still open. And my patches try
> > > > to achieve similar security as when machine is off (= no crypto
> > > > keys in RAM or on swap).
> > > >
> > > > When doing hibernate on unencrypted swap it is to prevent leaking
> > > > crypto keys to hibernate image (which is stored in swap).
> > > >
> > > > When doing suspend action it is again to prevent leaking crypto
> > > > keys. E.g when you suspend laptop and put it off (somebody can
> > > > remove RAMs and do some cold boot attack).
> > > >
> > > > The most common situation is:
> > > > You have mounted partition from dm-crypt device (e.g. /home/),
> > > > some userspace processes access it (e.g opened firefox which
> > > > still reads/writes to cache ~/.firefox/) and you want to drop
> > > > crypto keys from kernel for some time.
> > > >
> > > > For that operation there is command cryptsetup luksSuspend, which
> > > > suspend dm device and then tell kernel to wipe crypto keys. All
> > > > I/O operations are then stopped and userspace processes which
> > > > want to do some those I/O operations are stopped too (until you
> > > > call cryptsetup luksResume and enter correct key).
> > > >
> > > > Now if you want to suspend/hiberate your machine (when some of dm
> > > > devices are suspeneded and some processes are stopped due to
> > > > pending I/O) it is not possible. Kernel freeze_processes function
> > > > will fail because userspace processes are still stopped inside
> > > > some I/O syscall (read/write, etc,...).
> > > >
> > > > My patches fixes this problem and do those operations (suspend dm
> > > > device, wipe crypto keys, enter suspend/hiberate) in correct
> > > > order and without race condition.
> > > >
> > > > dm device is suspended *after* userspace processes are freezed
> > > > and after that are crypto keys wiped. And then computer/laptop
> > > > enters into suspend/hibernate state.
> > >
> > > Wouldn't it be better to fix freeze_processes() to be tolerant of
> > > processes that are hung as a side-effect of their backing storage being
> > > suspended? A hibernate shouldn't fail simply because a user chose to
> > > suspend a DM device.
> > >
> > > Then this entire problem goes away and the key can be wiped from
> > > userspace (like you said above).
> >
> > Still there will be race condition. Before hibernation (and device
> > poweroff) we should have synced disks and filesystems to prevent data
> > lose (or other damage) as more as we can. And if there will be some
> > application which using lot of I/O (e.g normal firefox) then there
> > always will be race condtion.
>
> The DM suspend will take care of flushing any pending I/O. So I don't
> see where the supposed race is...
>
Any I/O operation after DM suspend is race condition and could cause
data lost.
> Anything else that is trapped in userspace memory will be there when the
> machine resumes.
>
You are expecting that machine resumes always at 100% and correctly. But
this is not truth in real world. There are planty of users who reported
lot of random problems with suspend or hibernate...
> > So proper way is to wipe luks crypto keys *after* userspace processes
> > are freezed.
>
> I know you believe that I'm just not accepting that at face value.
If disks are synced before any DM suspend operation then we have higher
chance of preventing data corruption.
I still think that correct order is only:
* freeze processes (which doing continous I/O)
* fs & disk sync
* DM suspend
* wipe crypto keys
* enter hibernate
--
Pali Rohár
pali.rohar@gmail.com
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 0/3] dm-crypt: Adds support for wiping key when doing suspend/hibernation
From: Mike Snitzer @ 2015-04-09 14:26 UTC (permalink / raw)
To: Pali Rohár
Cc: Alasdair Kergon, Neil Brown, Rafael J. Wysocki, Len Brown,
Pavel Machek, dm-devel, linux-raid, linux-kernel, linux-pm
In-Reply-To: <20150409141633.GJ12339@atrey.karlin.mff.cuni.cz>
On Thu, Apr 09 2015 at 10:16am -0400,
Pali Rohár <pali.rohar@gmail.com> wrote:
> On Thursday 09 April 2015 10:08:43 Mike Snitzer wrote:
> > On Thu, Apr 09 2015 at 9:28am -0400,
> > Pali Rohár <pali.rohar@gmail.com> wrote:
> >
> > > On Thursday 09 April 2015 09:12:08 Mike Snitzer wrote:
> > > > On Mon, Apr 06 2015 at 9:29am -0400,
> > > > Pali Rohár <pali.rohar@gmail.com> wrote:
> > > >
> > > > > On Monday 06 April 2015 15:00:46 Mike Snitzer wrote:
> > > > > > On Sun, Apr 05 2015 at 1:20pm -0400,
> > > > > >
> > > > > > Pali Rohár <pali.rohar@gmail.com> wrote:
> > > > > > > This patch series increase security of suspend and hibernate
> > > > > > > actions. It allows user to safely wipe crypto keys before
> > > > > > > suspend and hibernate actions starts without race
> > > > > > > conditions on userspace process with heavy I/O.
> > > > > > >
> > > > > > > To automatically wipe cryto key for <device> before
> > > > > > > hibernate action call: $ dmsetup message <device> 0 key
> > > > > > > wipe_on_hibernation 1
> > > > > > >
> > > > > > > To automatically wipe cryto key for <device> before suspend
> > > > > > > action call: $ dmsetup message <device> 0 key
> > > > > > > wipe_on_suspend 1
> > > > > > >
> > > > > > > (Value 0 after wipe_* string reverts original behaviour - to
> > > > > > > not wipe key)
> > > > > >
> > > > > > Can you elaborate on the attack vector your changes are meant
> > > > > > to protect against? The user already authorized access, why
> > > > > > is it inherently dangerous to _not_ wipe the associated key
> > > > > > across these events?
> > > > >
> > > > > Hi,
> > > > >
> > > > > yes, I will try to explain current problems with cryptsetup
> > > > > luksSuspend command and hibernation.
> > > > >
> > > > > First, sometimes it is needed to put machine into other hands.
> > > > > You can still watch other person what is doing with machine, but
> > > > > once if you let machine unlocked (e.g opened luks disk), she/he
> > > > > can access encrypted data.
> > > > >
> > > > > If you turn off machine, it could be safe, because luks disk
> > > > > devices are locked. But if you enter machine into suspend or
> > > > > hibernate state luks devices are still open. And my patches try
> > > > > to achieve similar security as when machine is off (= no crypto
> > > > > keys in RAM or on swap).
> > > > >
> > > > > When doing hibernate on unencrypted swap it is to prevent leaking
> > > > > crypto keys to hibernate image (which is stored in swap).
> > > > >
> > > > > When doing suspend action it is again to prevent leaking crypto
> > > > > keys. E.g when you suspend laptop and put it off (somebody can
> > > > > remove RAMs and do some cold boot attack).
> > > > >
> > > > > The most common situation is:
> > > > > You have mounted partition from dm-crypt device (e.g. /home/),
> > > > > some userspace processes access it (e.g opened firefox which
> > > > > still reads/writes to cache ~/.firefox/) and you want to drop
> > > > > crypto keys from kernel for some time.
> > > > >
> > > > > For that operation there is command cryptsetup luksSuspend, which
> > > > > suspend dm device and then tell kernel to wipe crypto keys. All
> > > > > I/O operations are then stopped and userspace processes which
> > > > > want to do some those I/O operations are stopped too (until you
> > > > > call cryptsetup luksResume and enter correct key).
> > > > >
> > > > > Now if you want to suspend/hiberate your machine (when some of dm
> > > > > devices are suspeneded and some processes are stopped due to
> > > > > pending I/O) it is not possible. Kernel freeze_processes function
> > > > > will fail because userspace processes are still stopped inside
> > > > > some I/O syscall (read/write, etc,...).
> > > > >
> > > > > My patches fixes this problem and do those operations (suspend dm
> > > > > device, wipe crypto keys, enter suspend/hiberate) in correct
> > > > > order and without race condition.
> > > > >
> > > > > dm device is suspended *after* userspace processes are freezed
> > > > > and after that are crypto keys wiped. And then computer/laptop
> > > > > enters into suspend/hibernate state.
> > > >
> > > > Wouldn't it be better to fix freeze_processes() to be tolerant of
> > > > processes that are hung as a side-effect of their backing storage being
> > > > suspended? A hibernate shouldn't fail simply because a user chose to
> > > > suspend a DM device.
> > > >
> > > > Then this entire problem goes away and the key can be wiped from
> > > > userspace (like you said above).
> > >
> > > Still there will be race condition. Before hibernation (and device
> > > poweroff) we should have synced disks and filesystems to prevent data
> > > lose (or other damage) as more as we can. And if there will be some
> > > application which using lot of I/O (e.g normal firefox) then there
> > > always will be race condtion.
> >
> > The DM suspend will take care of flushing any pending I/O. So I don't
> > see where the supposed race is...
> >
>
> Any I/O operation after DM suspend is race condition and could cause
> data lost.
>
> > Anything else that is trapped in userspace memory will be there when the
> > machine resumes.
> >
>
> You are expecting that machine resumes always at 100% and correctly. But
> this is not truth in real world. There are planty of users who reported
> lot of random problems with suspend or hibernate...
But the system was left in a crash consistent state. Properly written
apps will wait for I/O to ensure data loss (in the event of a failed
resume) isn't a problem.
> > > So proper way is to wipe luks crypto keys *after* userspace processes
> > > are freezed.
> >
> > I know you believe that I'm just not accepting that at face value.
>
> If disks are synced before any DM suspend operation then we have higher
> chance of preventing data corruption.
disks are already synced as part of the DM suspend operation!
But you're saying that all user processes are frozen (and associated
I/O flushed) before the DM suspend, that is different:
> I still think that correct order is only:
>
> * freeze processes (which doing continous I/O)
> * fs & disk sync
> * DM suspend
> * wipe crypto keys
> * enter hibernate
I just don't think that extreme is _required_ to have a hibernate/resume
that incorporates dm-crypt key wiping.
^ permalink raw reply
* Re: [PATCH 0/3] dm-crypt: Adds support for wiping key when doing suspend/hibernation
From: Pali Rohár @ 2015-04-09 14:38 UTC (permalink / raw)
To: Mike Snitzer
Cc: Alasdair Kergon, Neil Brown, Rafael J. Wysocki, Len Brown,
Pavel Machek, dm-devel, linux-raid, linux-kernel, linux-pm
In-Reply-To: <20150409142658.GA9839@redhat.com>
On Thursday 09 April 2015 10:26:58 Mike Snitzer wrote:
> On Thu, Apr 09 2015 at 10:16am -0400,
> Pali Rohár <pali.rohar@gmail.com> wrote:
>
> > On Thursday 09 April 2015 10:08:43 Mike Snitzer wrote:
> > > On Thu, Apr 09 2015 at 9:28am -0400,
> > > Pali Rohár <pali.rohar@gmail.com> wrote:
> > >
> > > > On Thursday 09 April 2015 09:12:08 Mike Snitzer wrote:
> > > > > On Mon, Apr 06 2015 at 9:29am -0400,
> > > > > Pali Rohár <pali.rohar@gmail.com> wrote:
> > > > >
> > > > > > On Monday 06 April 2015 15:00:46 Mike Snitzer wrote:
> > > > > > > On Sun, Apr 05 2015 at 1:20pm -0400,
> > > > > > >
> > > > > > > Pali Rohár <pali.rohar@gmail.com> wrote:
> > > > > > > > This patch series increase security of suspend and hibernate
> > > > > > > > actions. It allows user to safely wipe crypto keys before
> > > > > > > > suspend and hibernate actions starts without race
> > > > > > > > conditions on userspace process with heavy I/O.
> > > > > > > >
> > > > > > > > To automatically wipe cryto key for <device> before
> > > > > > > > hibernate action call: $ dmsetup message <device> 0 key
> > > > > > > > wipe_on_hibernation 1
> > > > > > > >
> > > > > > > > To automatically wipe cryto key for <device> before suspend
> > > > > > > > action call: $ dmsetup message <device> 0 key
> > > > > > > > wipe_on_suspend 1
> > > > > > > >
> > > > > > > > (Value 0 after wipe_* string reverts original behaviour - to
> > > > > > > > not wipe key)
> > > > > > >
> > > > > > > Can you elaborate on the attack vector your changes are meant
> > > > > > > to protect against? The user already authorized access, why
> > > > > > > is it inherently dangerous to _not_ wipe the associated key
> > > > > > > across these events?
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > yes, I will try to explain current problems with cryptsetup
> > > > > > luksSuspend command and hibernation.
> > > > > >
> > > > > > First, sometimes it is needed to put machine into other hands.
> > > > > > You can still watch other person what is doing with machine, but
> > > > > > once if you let machine unlocked (e.g opened luks disk), she/he
> > > > > > can access encrypted data.
> > > > > >
> > > > > > If you turn off machine, it could be safe, because luks disk
> > > > > > devices are locked. But if you enter machine into suspend or
> > > > > > hibernate state luks devices are still open. And my patches try
> > > > > > to achieve similar security as when machine is off (= no crypto
> > > > > > keys in RAM or on swap).
> > > > > >
> > > > > > When doing hibernate on unencrypted swap it is to prevent leaking
> > > > > > crypto keys to hibernate image (which is stored in swap).
> > > > > >
> > > > > > When doing suspend action it is again to prevent leaking crypto
> > > > > > keys. E.g when you suspend laptop and put it off (somebody can
> > > > > > remove RAMs and do some cold boot attack).
> > > > > >
> > > > > > The most common situation is:
> > > > > > You have mounted partition from dm-crypt device (e.g. /home/),
> > > > > > some userspace processes access it (e.g opened firefox which
> > > > > > still reads/writes to cache ~/.firefox/) and you want to drop
> > > > > > crypto keys from kernel for some time.
> > > > > >
> > > > > > For that operation there is command cryptsetup luksSuspend, which
> > > > > > suspend dm device and then tell kernel to wipe crypto keys. All
> > > > > > I/O operations are then stopped and userspace processes which
> > > > > > want to do some those I/O operations are stopped too (until you
> > > > > > call cryptsetup luksResume and enter correct key).
> > > > > >
> > > > > > Now if you want to suspend/hiberate your machine (when some of dm
> > > > > > devices are suspeneded and some processes are stopped due to
> > > > > > pending I/O) it is not possible. Kernel freeze_processes function
> > > > > > will fail because userspace processes are still stopped inside
> > > > > > some I/O syscall (read/write, etc,...).
> > > > > >
> > > > > > My patches fixes this problem and do those operations (suspend dm
> > > > > > device, wipe crypto keys, enter suspend/hiberate) in correct
> > > > > > order and without race condition.
> > > > > >
> > > > > > dm device is suspended *after* userspace processes are freezed
> > > > > > and after that are crypto keys wiped. And then computer/laptop
> > > > > > enters into suspend/hibernate state.
> > > > >
> > > > > Wouldn't it be better to fix freeze_processes() to be tolerant of
> > > > > processes that are hung as a side-effect of their backing storage being
> > > > > suspended? A hibernate shouldn't fail simply because a user chose to
> > > > > suspend a DM device.
> > > > >
> > > > > Then this entire problem goes away and the key can be wiped from
> > > > > userspace (like you said above).
> > > >
> > > > Still there will be race condition. Before hibernation (and device
> > > > poweroff) we should have synced disks and filesystems to prevent data
> > > > lose (or other damage) as more as we can. And if there will be some
> > > > application which using lot of I/O (e.g normal firefox) then there
> > > > always will be race condtion.
> > >
> > > The DM suspend will take care of flushing any pending I/O. So I don't
> > > see where the supposed race is...
> > >
> >
> > Any I/O operation after DM suspend is race condition and could cause
> > data lost.
> >
> > > Anything else that is trapped in userspace memory will be there when the
> > > machine resumes.
> > >
> >
> > You are expecting that machine resumes always at 100% and correctly. But
> > this is not truth in real world. There are planty of users who reported
> > lot of random problems with suspend or hibernate...
>
> But the system was left in a crash consistent state. Properly written
> apps will wait for I/O to ensure data loss (in the event of a failed
> resume) isn't a problem.
>
I think you are too optimistic about ideal world...
"Properly written apps" "ensure data loss"
> > > > So proper way is to wipe luks crypto keys *after* userspace processes
> > > > are freezed.
> > >
> > > I know you believe that I'm just not accepting that at face value.
> >
> > If disks are synced before any DM suspend operation then we have higher
> > chance of preventing data corruption.
>
> disks are already synced as part of the DM suspend operation!
>
Yes, but part of hibernate operation is also sync call.
> But you're saying that all user processes are frozen (and associated
> I/O flushed) before the DM suspend, that is different:
>
Yes, I want to ensure that. So processes wont be able to do any other
I/O.
> > I still think that correct order is only:
> >
> > * freeze processes (which doing continous I/O)
> > * fs & disk sync
> > * DM suspend
> > * wipe crypto keys
> > * enter hibernate
>
> I just don't think that extreme is _required_ to have a hibernate/resume
> that incorporates dm-crypt key wiping.
Ok, and what other developers think?
I'm saying that if I want to wipe luks keys before suspend/hibernate and
have system in consistant state as much as possible, keys must be wiped
*after* userspace processes are freezed. Or do you have relevant or
functional argument why not? Or is there any problem in my thinking?
--
Pali Rohár
pali.rohar@gmail.com
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: Dan Williams @ 2015-04-09 15:37 UTC (permalink / raw)
To: Shaohua Li; +Cc: NeilBrown, linux-raid, Song Liu, Kernel-team
In-Reply-To: <20150409061545.GA864165@devbig257.prn2.facebook.com>
On Wed, Apr 8, 2015 at 11:15 PM, Shaohua Li <shli@fb.com> wrote:
> On Thu, Apr 09, 2015 at 03:04:59PM +1000, NeilBrown wrote:
>> On Wed, 8 Apr 2015 17:43:11 -0700 Shaohua Li <shli@fb.com> wrote:
>>
>> > Hi,
>> > This is what I'm working on now, and hopefully had the basic code
>> > running next week. The new design will do cache and fix the write hole
>> > issue too. Before I post the code out, I'd like to check if the design
>> > has obvious issues.
>>
>> I can't say I'm excited about it....
>>
>> You still haven't explained why you would ever want to read data from the
>> "cache"? Why not just keep everything in the stripe-cache until it is safe
>> in the RAID. I asked before and you said:
>>
>> >> I'm not enthusiastic to use stripe cache though, we can't keep all data
>> >> in stripe cache. What we really need is an index.
>>
>> which is hardly an answer. Why cannot you keep all the data in the stripe
>> cache? How much data is there? How much memory can you afford to dedicate?
>>
>> You must have some very long sustained bursts of writes which are much faster
>> than the RAID can accept in order to not be able to keep everything in memory.
>>
>>
>> Your cache layout seems very rigid. I would much rather a layout that was
>> very general and flexible. If you want to always allocate a chunk at a time
>> then fine, but don't force that on the cache layout.
>>
>> The log really should be very simple. A block describing what comes next,
>> then lots of data/parity. Then another block and more data etc etc.
>> Each metadata block points to the next one.
>> If you need an index of the cache, you keep that in memory. On restart, you
>> read all of the metadata blocks and built up the index.
>>
>> I think that space in the log should be reclaimed in exactly the order that
>> it is written, so the active part of the log is contiguous. Obviously
>> individual blocks become inactive in arbitrary order as they are written to
>> the RAID, but each extent of the log becomes free in order.
>> If you want that to happen out of order, you would need to present a very
>> good reason.
>
> I came to the same idea when I'm thinking about a caching layer, but the
> memory size is the main blocking issue. If the solution requires a large
> amount of extra memory, it's not cost effective, so a hard sell to
> replace hardware raid with software raid. The design completely depends
> on if we can store all data in memory. I don't have an anwser yet how
> much memory we should use to make the aggregation efficient. Guess only
> number can talk. I'll try to collect some data and get back to you.
>
Another consideration to keep in mind is persistent memory. I'm
working on an in-kernel mechanism to claim and map pmem and a
raid-write-cache is an obvious first application. I'll include you on
the initial submission of that capability.
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: Shaohua Li @ 2015-04-09 16:03 UTC (permalink / raw)
To: Dan Williams; +Cc: NeilBrown, linux-raid, Song Liu, Kernel-team
In-Reply-To: <CAPcyv4j2RjqY=Ns8rXMUypGovU7_beNfpuL-iW0Fx9522ogPkA@mail.gmail.com>
On Thu, Apr 09, 2015 at 08:37:03AM -0700, Dan Williams wrote:
> On Wed, Apr 8, 2015 at 11:15 PM, Shaohua Li <shli@fb.com> wrote:
> > On Thu, Apr 09, 2015 at 03:04:59PM +1000, NeilBrown wrote:
> >> On Wed, 8 Apr 2015 17:43:11 -0700 Shaohua Li <shli@fb.com> wrote:
> >>
> >> > Hi,
> >> > This is what I'm working on now, and hopefully had the basic code
> >> > running next week. The new design will do cache and fix the write hole
> >> > issue too. Before I post the code out, I'd like to check if the design
> >> > has obvious issues.
> >>
> >> I can't say I'm excited about it....
> >>
> >> You still haven't explained why you would ever want to read data from the
> >> "cache"? Why not just keep everything in the stripe-cache until it is safe
> >> in the RAID. I asked before and you said:
> >>
> >> >> I'm not enthusiastic to use stripe cache though, we can't keep all data
> >> >> in stripe cache. What we really need is an index.
> >>
> >> which is hardly an answer. Why cannot you keep all the data in the stripe
> >> cache? How much data is there? How much memory can you afford to dedicate?
> >>
> >> You must have some very long sustained bursts of writes which are much faster
> >> than the RAID can accept in order to not be able to keep everything in memory.
> >>
> >>
> >> Your cache layout seems very rigid. I would much rather a layout that was
> >> very general and flexible. If you want to always allocate a chunk at a time
> >> then fine, but don't force that on the cache layout.
> >>
> >> The log really should be very simple. A block describing what comes next,
> >> then lots of data/parity. Then another block and more data etc etc.
> >> Each metadata block points to the next one.
> >> If you need an index of the cache, you keep that in memory. On restart, you
> >> read all of the metadata blocks and built up the index.
> >>
> >> I think that space in the log should be reclaimed in exactly the order that
> >> it is written, so the active part of the log is contiguous. Obviously
> >> individual blocks become inactive in arbitrary order as they are written to
> >> the RAID, but each extent of the log becomes free in order.
> >> If you want that to happen out of order, you would need to present a very
> >> good reason.
> >
> > I came to the same idea when I'm thinking about a caching layer, but the
> > memory size is the main blocking issue. If the solution requires a large
> > amount of extra memory, it's not cost effective, so a hard sell to
> > replace hardware raid with software raid. The design completely depends
> > on if we can store all data in memory. I don't have an anwser yet how
> > much memory we should use to make the aggregation efficient. Guess only
> > number can talk. I'll try to collect some data and get back to you.
> >
>
> Another consideration to keep in mind is persistent memory. I'm
> working on an in-kernel mechanism to claim and map pmem and a
> raid-write-cache is an obvious first application. I'll include you on
> the initial submission of that capability.
Exactly, we are planing to use pmem in the future when it's mature and
popular. SSD is still the best option before pmem is popular and widely
used.
Thanks,
Shaohua
^ permalink raw reply
* Re: [PATCH 1/3] PM suspend/hibernate: Call notifier after freezing processes
From: Pali Rohár @ 2015-04-09 16:55 UTC (permalink / raw)
To: Rafael J. Wysocki
Cc: Alasdair Kergon, Mike Snitzer, Neil Brown, Len Brown,
Pavel Machek, dm-devel, linux-raid, linux-kernel, linux-pm
In-Reply-To: <1856560.gNHsvxOS0D@vostro.rjw.lan>
[-- Attachment #1: Type: Text/Plain, Size: 1776 bytes --]
On Thursday 09 April 2015 19:13:55 Rafael J. Wysocki wrote:
> On Thursday, April 09, 2015 08:36:57 AM Pali Rohár wrote:
> > --nextPart2566388.gOmNIJrIqI
> > Content-Type: Text/Plain;
> >
> > charset="utf-8"
> >
> > Content-Transfer-Encoding: quoted-printable
> >
> > On Thursday 09 April 2015 02:28:41 Rafael J. Wysocki wrote:
> > > On Sunday, April 05, 2015 07:20:17 PM Pali Roh=C3=A1r
> > > wrote:
> > > > To prevent race conditions on userspace processes with
> > > > I/O some taks must be called after processes are
> > > > freezed. This patch adds new events which are delivered
> > > > by
> > > > pm_notifier_call_chain() after freezing processes when
> > > > doing suspend or hibernate action.
> > > >
> > > >=20
> > > >
> > > > Signed-off-by: Pali Roh=C3=A1r <pali.rohar@gmail.com>
> > >
> > >=20
> > >
> > > Please don't add more notifiers. Just call whatever you
> > > need directly from where you need to call that.
> > >
> > >=20
> > >
> > > If that is device-related, try to use device PM
> > > suspend/hibernate callbacks instead.
> > >
> > >=20
> >
> > Hi! It is not possible to use any exiting pm notifiers! This
> > is=20 reason why I added new ones. As I wrote wiping dm
> > crypt keys must=20 be done *after* userspace processes are
> > freezed to prevent race=20 conditions...
>
> I'm not talking about using the existing notifiers. I'm
> talking about calling the function you need to call directly
> from a suitable place in the system suspend code.
I need to wipe crypto keys from dm-crypt module. That module can
be compiled as external .ko file and so kernel cannot call
directly needed function. This is reason why I'm adding new
notifier event.
--
Pali Rohár
pali.rohar@gmail.com
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply
* Re: [PATCH 1/3] PM suspend/hibernate: Call notifier after freezing processes
From: Rafael J. Wysocki @ 2015-04-09 17:13 UTC (permalink / raw)
To: Pali Rohár
Cc: Alasdair Kergon, Mike Snitzer, Neil Brown, Len Brown,
Pavel Machek, dm-devel, linux-raid, linux-kernel, linux-pm
In-Reply-To: <201504090836.57365@pali>
On Thursday, April 09, 2015 08:36:57 AM Pali Rohár wrote:
>
> --nextPart2566388.gOmNIJrIqI
> Content-Type: Text/Plain;
> charset="utf-8"
> Content-Transfer-Encoding: quoted-printable
>
> On Thursday 09 April 2015 02:28:41 Rafael J. Wysocki wrote:
> > On Sunday, April 05, 2015 07:20:17 PM Pali Roh=C3=A1r wrote:
> > > To prevent race conditions on userspace processes with I/O
> > > some taks must be called after processes are freezed. This
> > > patch adds new events which are delivered by
> > > pm_notifier_call_chain() after freezing processes when
> > > doing suspend or hibernate action.
> > >=20
> > > Signed-off-by: Pali Roh=C3=A1r <pali.rohar@gmail.com>
> >=20
> > Please don't add more notifiers. Just call whatever you need
> > directly from where you need to call that.
> >=20
> > If that is device-related, try to use device PM
> > suspend/hibernate callbacks instead.
> >=20
>
> Hi! It is not possible to use any exiting pm notifiers! This is=20
> reason why I added new ones. As I wrote wiping dm crypt keys must=20
> be done *after* userspace processes are freezed to prevent race=20
> conditions...
I'm not talking about using the existing notifiers. I'm talking about
calling the function you need to call directly from a suitable place
in the system suspend code.
--
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.
^ permalink raw reply
* md: NULL ptr deref on xfstests generic/040
From: Sasha Levin @ 2015-04-09 19:37 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, LKML
Hi all,
I'm seeing the following warnings and NULL ptr deref when running xfstest generic/040
on the latest -next kernel.
[ 7023.673973] run fstests generic/040 at 2015-04-09 10:31:57
[ 7025.777329] kobject: 'sdd' (ffff8837b7c5e0a8): kobject_uevent_env
[ 7025.777344] kobject: 'sdd' (ffff8837b7c5e0a8): fill_kobj_path: path = '/devices/pci0000:00/0000:00:03.2/0000:50:00.0/host0/target0:2:3/0:2:3:0/block/sdd'
[ 7025.969112] kobject: '251:0' (ffff881ff2554810): kobject_add_internal: parent: 'bdi', set: 'devices'
[ 7025.969161] ------------[ cut here ]------------
[ 7025.969181] WARNING: CPU: 7 PID: 30467 at fs/sysfs/dir.c:31 sysfs_warn_dup+0x86/0xa0()
[ 7025.969187] sysfs: cannot create duplicate filename '/devices/virtual/bdi/251:0'
[ 7025.969192] Modules linked in: dm_flakey intel_rapl ast iosf_mbi x86_pkg_temp_thermal ttm intel_powerclamp coretemp drm_kms_helper kvm_intel kvm drm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw glue_helper ablk_helper cryptd joydev i2c_algo_bit syscopyarea sysfillrect sysimgblt ipmi_si sb_edac ipmi_msghandler edac_core ioatdma shpchp lpc_ich mac_hid btrfs xor mlx4_en vxlan raid6_pq hid_generic usbhid hid ixgbe mlx4_core ahci dca ptp libahci megaraid_sas pps_core mdio
[ 7025.969328] CPU: 7 PID: 30467 Comm: dmsetup Not tainted 4.0.0-rc7-next-20150408+ #6
[ 7025.969335] Hardware name: Oracle Corporation OVCA X3-2 /ASSY,MOTHERBOARD,1U , BIOS 17021300 06/19/2012
[ 7025.969342] ffffffff82b37a40 ffff881fda8073f8 ffffffff82947148 0000000000000000
[ 7025.969354] ffff881fda807478 ffff881fda807448 ffffffff8115a04a 0000000000000001
[ 7025.969365] ffffffff81770d56 ffff881fda807498 ffffed03fb500e8b ffff881ff2533a30
[ 7025.969376] Call Trace:
[ 7025.969389] dump_stack (lib/dump_stack.c:52)
[ 7025.969400] warn_slowpath_common (kernel/panic.c:447)
[ 7025.969410] ? sysfs_warn_dup (fs/sysfs/dir.c:33)
[ 7025.969418] warn_slowpath_fmt (kernel/panic.c:453)
[ 7025.969427] ? warn_slowpath_common (kernel/panic.c:453)
[ 7025.969439] ? trace_hardirqs_on (kernel/locking/lockdep.c:2630)
[ 7025.969448] sysfs_warn_dup (fs/sysfs/dir.c:33)
[ 7025.969458] sysfs_create_dir_ns (fs/sysfs/dir.c:59)
[ 7025.969471] kobject_add_internal (lib/kobject.c:72 lib/kobject.c:229)
[ 7025.969481] ? debug_lockdep_rcu_enabled (kernel/rcu/update.c:195)
[ 7025.969493] ? __mutex_unlock_slowpath (./arch/x86/include/asm/paravirt.h:809 kernel/locking/mutex.c:755 kernel/locking/mutex.c:766)
[ 7025.969502] kobject_add (lib/kobject.c:384)
[ 7025.969509] ? kobject_add_internal (lib/kobject.c:384)
[ 7025.969518] ? mutex_unlock (kernel/locking/mutex.c:444)
[ 7025.969532] device_add (drivers/base/core.c:1025)
[ 7025.969541] ? device_private_init (drivers/base/core.c:977)
[ 7025.969554] ? kfree (include/trace/events/kmem.h:136 mm/slub.c:3422)
[ 7025.969564] device_create_groups_vargs (drivers/base/core.c:1618)
[ 7025.969572] ? debug_check_no_locks_freed (kernel/locking/lockdep.c:3091)
[ 7025.969581] device_create_vargs (drivers/base/core.c:1660)
[ 7025.969592] bdi_register (mm/backing-dev.c:347)
[ 7025.969600] ? wait_iff_congested (mm/backing-dev.c:337)
[ 7025.969609] ? vsnprintf (lib/vsprintf.c:2008)
[ 7025.969617] bdi_register_dev (mm/backing-dev.c:367)
[ 7025.969629] add_disk (block/genhd.c:616)
[ 7025.969636] ? pointer.isra.23 (lib/vsprintf.c:1878)
[ 7025.969644] ? lockdep_init_map_type (kernel/locking/lockdep.c:3009)
[ 7025.969651] ? trace_hardirqs_on (kernel/locking/lockdep.c:2630)
[ 7025.969660] ? blk_alloc_devt (block/genhd.c:583)
[ 7025.969667] ? sprintf (lib/vsprintf.c:2138)
[ 7025.969673] ? scnprintf (lib/vsprintf.c:2138)
[ 7025.969682] ? lockdep_init_map (kernel/locking/lockdep.c:3041)
[ 7025.969692] dm_create (drivers/md/dm.c:2318 drivers/md/dm.c:2598)
[ 7025.969701] dev_create (drivers/md/dm-ioctl.c:747)
[ 7025.969709] ? list_version_get_info (drivers/md/dm-ioctl.c:735)
[ 7025.969716] ctl_ioctl (drivers/md/dm-ioctl.c:1848)
[ 7025.969726] ? semctl_main (ipc/sem.c:1330)
[ 7025.969734] ? debug_lockdep_rcu_enabled (kernel/rcu/update.c:195)
[ 7025.969741] ? list_version_get_info (drivers/md/dm-ioctl.c:735)
[ 7025.969751] ? free_params (drivers/md/dm-ioctl.c:1793)
[ 7025.969760] ? SYSC_semtimedop (ipc/sem.c:2010)
[ 7025.969771] dm_ctl_ioctl (drivers/md/dm-ioctl.c:1866)
[ 7025.969783] do_vfs_ioctl (fs/ioctl.c:44 fs/ioctl.c:607)
[ 7025.969792] ? ioctl_preallocate (fs/ioctl.c:557)
[ 7025.969803] ? mntput (fs/namespace.c:1106)
[ 7025.969812] ? SyS_semctl (ipc/sem.c:1601 ipc/sem.c:1577)
[ 7025.969820] ? debug_lockdep_rcu_enabled (kernel/rcu/update.c:195)
[ 7025.969828] ? __fget_light (fs/file.c:684)
[ 7025.969836] SyS_ioctl (fs/ioctl.c:622 fs/ioctl.c:613)
[ 7025.969845] system_call_fastpath (arch/x86/kernel/entry_64.S:261)
[ 7025.969853] ---[ end trace 734c93b316c19e43 ]---
[ 7025.969862] ------------[ cut here ]------------
[ 7025.969873] WARNING: CPU: 7 PID: 30467 at lib/kobject.c:240 kobject_add_internal+0x6ff/0x920()
[ 7025.969879] kobject_add_internal failed for 251:0 with -EEXIST, don't try to register things with the same name in the same directory.
[ 7025.969883] Modules linked in: dm_flakey intel_rapl ast iosf_mbi x86_pkg_temp_thermal ttm intel_powerclamp coretemp drm_kms_helper kvm_intel kvm drm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw glue_helper ablk_helper cryptd joydev i2c_algo_bit syscopyarea sysfillrect sysimgblt ipmi_si sb_edac ipmi_msghandler edac_core ioatdma shpchp lpc_ich mac_hid btrfs xor mlx4_en vxlan raid6_pq hid_generic usbhid hid ixgbe mlx4_core ahci dca ptp libahci megaraid_sas pps_core mdio
[ 7025.969997] CPU: 7 PID: 30467 Comm: dmsetup Tainted: G W 4.0.0-rc7-next-20150408+ #6
[ 7025.970003] Hardware name: Oracle Corporation OVCA X3-2 /ASSY,MOTHERBOARD,1U , BIOS 17021300 06/19/2012
[ 7025.970008] ffffffff82bc7b60 ffff881fda807458 ffffffff82947148 0000000000000000
[ 7025.970019] ffff881fda8074d8 ffff881fda8074a8 ffffffff8115a04a 0000000000000001
[ 7025.970029] ffffffff81bdf32f ffff881fda8074f8 ffffed03fb500e97 00000000ffffffef
[ 7025.970040] Call Trace:
[ 7025.970048] dump_stack (lib/dump_stack.c:52)
[ 7025.970056] warn_slowpath_common (kernel/panic.c:447)
[ 7025.970064] ? kobject_add_internal (lib/kobject.c:237 (discriminator 1))
[ 7025.970072] warn_slowpath_fmt (kernel/panic.c:453)
[ 7025.970080] ? warn_slowpath_common (kernel/panic.c:453)
[ 7025.970089] ? _raw_spin_unlock (./arch/x86/include/asm/preempt.h:77 include/linux/spinlock_api_smp.h:154 kernel/locking/spinlock.c:183)
[ 7025.970097] kobject_add_internal (lib/kobject.c:237 (discriminator 1))
[ 7025.970105] ? debug_lockdep_rcu_enabled (kernel/rcu/update.c:195)
[ 7025.970113] ? __mutex_unlock_slowpath (./arch/x86/include/asm/paravirt.h:809 kernel/locking/mutex.c:755 kernel/locking/mutex.c:766)
[ 7025.970121] kobject_add (lib/kobject.c:384)
[ 7025.970129] ? kobject_add_internal (lib/kobject.c:384)
[ 7025.970137] ? mutex_unlock (kernel/locking/mutex.c:444)
[ 7025.970146] device_add (drivers/base/core.c:1025)
[ 7025.970155] ? device_private_init (drivers/base/core.c:977)
[ 7025.970164] ? kfree (include/trace/events/kmem.h:136 mm/slub.c:3422)
[ 7025.970173] device_create_groups_vargs (drivers/base/core.c:1618)
[ 7025.970181] ? debug_check_no_locks_freed (kernel/locking/lockdep.c:3091)
[ 7025.970190] device_create_vargs (drivers/base/core.c:1660)
[ 7025.970197] bdi_register (mm/backing-dev.c:347)
[ 7025.970205] ? wait_iff_congested (mm/backing-dev.c:337)
[ 7025.970213] ? vsnprintf (lib/vsprintf.c:2008)
[ 7025.970221] bdi_register_dev (mm/backing-dev.c:367)
[ 7025.970229] add_disk (block/genhd.c:616)
[ 7025.970235] ? pointer.isra.23 (lib/vsprintf.c:1878)
[ 7025.970243] ? lockdep_init_map_type (kernel/locking/lockdep.c:3009)
[ 7025.970250] ? trace_hardirqs_on (kernel/locking/lockdep.c:2630)
[ 7025.970259] ? blk_alloc_devt (block/genhd.c:583)
[ 7025.970265] ? sprintf (lib/vsprintf.c:2138)
[ 7025.970271] ? scnprintf (lib/vsprintf.c:2138)
[ 7025.970280] ? lockdep_init_map (kernel/locking/lockdep.c:3041)
[ 7025.970287] dm_create (drivers/md/dm.c:2318 drivers/md/dm.c:2598)
[ 7025.970295] dev_create (drivers/md/dm-ioctl.c:747)
[ 7025.970303] ? list_version_get_info (drivers/md/dm-ioctl.c:735)
[ 7025.970310] ctl_ioctl (drivers/md/dm-ioctl.c:1848)
[ 7025.970318] ? semctl_main (ipc/sem.c:1330)
[ 7025.970326] ? debug_lockdep_rcu_enabled (kernel/rcu/update.c:195)
[ 7025.970333] ? list_version_get_info (drivers/md/dm-ioctl.c:735)
[ 7025.970341] ? free_params (drivers/md/dm-ioctl.c:1793)
[ 7025.970350] ? SYSC_semtimedop (ipc/sem.c:2010)
[ 7025.970361] dm_ctl_ioctl (drivers/md/dm-ioctl.c:1866)
[ 7025.970369] do_vfs_ioctl (fs/ioctl.c:44 fs/ioctl.c:607)
[ 7025.970377] ? ioctl_preallocate (fs/ioctl.c:557)
[ 7025.970385] ? mntput (fs/namespace.c:1106)
[ 7025.970393] ? SyS_semctl (ipc/sem.c:1601 ipc/sem.c:1577)
[ 7025.970402] ? debug_lockdep_rcu_enabled (kernel/rcu/update.c:195)
[ 7025.970409] ? __fget_light (fs/file.c:684)
[ 7025.970417] SyS_ioctl (fs/ioctl.c:622 fs/ioctl.c:613)
[ 7025.970426] system_call_fastpath (arch/x86/kernel/entry_64.S:261)
[ 7025.970433] ---[ end trace 734c93b316c19e44 ]---
[ 7025.970445] kobject: '251:0' (ffff881ff2554810): kobject_release, parent (null) (delayed 100)
[ 7025.970484] kobject: 'dm-0' (ffff881ff25578a8): kobject_add_internal: parent: 'block', set: 'devices'
[ 7025.970968] kobject: 'dm-0' (ffff881ff25578a8): kobject_uevent_env
[ 7025.970976] kobject: 'dm-0' (ffff881ff25578a8): kobject_uevent_env: uevent_suppress caused the event to drop!
[ 7025.971006] kobject: 'holders' (ffff881fea16ae00): kobject_add_internal: parent: 'dm-0', set: '<NULL>'
[ 7025.971022] kobject: 'slaves' (ffff881fea16ac00): kobject_add_internal: parent: 'dm-0', set: '<NULL>'
[ 7025.971034] kobject: 'dm-0' (ffff881ff25578a8): kobject_uevent_env
[ 7025.971044] kobject: 'dm-0' (ffff881ff25578a8): fill_kobj_path: path = '/devices/virtual/block/dm-0'
[ 7025.971141] kobject: 'queue' (ffff881fe5f007a8): kobject_add_internal: parent: 'dm-0', set: '<NULL>'
[ 7025.971286] kobject: 'queue' (ffff881fe5f007a8): kobject_uevent_env
[ 7025.971291] kobject: 'queue' (ffff881fe5f007a8): kobject_uevent_env: filter function caused the event to drop!
[ 7025.971307] CONFIG_KASAN_INLINE enabled
[ 7025.971382] GPF could be caused by NULL-ptr deref or user memory accessgeneral protection fault: 0000 [#1] SMP DEBUG_PAGEALLOC KASAN
[ 7025.971571] Modules linked in: dm_flakey intel_rapl ast iosf_mbi x86_pkg_temp_thermal ttm intel_powerclamp coretemp drm_kms_helper kvm_intel kvm drm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw glue_helper ablk_helper cryptd joydev i2c_algo_bit syscopyarea sysfillrect sysimgblt ipmi_si sb_edac ipmi_msghandler edac_core ioatdma shpchp lpc_ich mac_hid btrfs xor mlx4_en vxlan raid6_pq hid_generic usbhid hid ixgbe mlx4_core ahci dca ptp libahci megaraid_sas pps_core mdio
[ 7025.972551] CPU: 23 PID: 30467 Comm: dmsetup Tainted: G W 4.0.0-rc7-next-20150408+ #6
[ 7025.972770] Hardware name: Oracle Corporation OVCA X3-2 /ASSY,MOTHERBOARD,1U , BIOS 17021300 06/19/2012
[ 7025.973033] task: ffff881ff0878000 ti: ffff881fda800000 task.ti: ffff881fda800000
[ 7025.973213] RIP: sysfs_do_create_link_sd.isra.2 (fs/sysfs/symlink.c:35)
[ 7025.973475] RSP: 0018:ffff881fda807878 EFLAGS: 00010202
[ 7025.973615] RAX: dffffc0000000000 RBX: 0000000000000040 RCX: 000000001a561a54
[ 7025.973802] RDX: 0000000000000008 RSI: 00000000000000db RDI: ffffffff833dd704
[ 7025.973982] RBP: ffff881fda8078a8 R08: 0000000000000000 R09: 0000000000000000
[ 7025.974163] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000001
[ 7025.974348] R13: ffffffff82bbc720 R14: ffff881fe5ffd400 R15: ffff881fe5f00000
[ 7025.974532] FS: 00007fc654bcc840(0000) GS:ffff881fffdc0000(0000) knlGS:0000000000000000
[ 7025.974745] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 7025.974874] CR2: 00007fed40c85798 CR3: 0000001fe0ab5000 CR4: 00000000000407e0
[ 7025.974972] Stack:
[ 7025.975003] ffff881fda807888 ffff881ff25578a8 0000000000000001 ffff881ff2557800
[ 7025.975119] ffff881fda8079e8 ffff881fe5f00000 ffff881fda8078d8 ffffffff81771658
[ 7025.975234] ffff881fda8078d8 ffffffff81b25cb6 ffff881fda8079e8 ffff881fda8079a8
[ 7025.975349] Call Trace:
[ 7025.975393] sysfs_create_link (fs/sysfs/symlink.c:93)
[ 7025.975474] ? blk_get_queue (block/blk-core.c:794)
[ 7025.975552] add_disk (block/genhd.c:629 (discriminator 8))
[ 7025.975625] ? lockdep_init_map_type (kernel/locking/lockdep.c:3009)
[ 7025.975715] ? trace_hardirqs_on (kernel/locking/lockdep.c:2630)
[ 7025.975798] ? blk_alloc_devt (block/genhd.c:583)
[ 7025.975879] ? sprintf (lib/vsprintf.c:2138)
[ 7025.975949] ? scnprintf (lib/vsprintf.c:2138)
[ 7025.976023] ? lockdep_init_map (kernel/locking/lockdep.c:3041)
[ 7025.976103] dm_create (drivers/md/dm.c:2318 drivers/md/dm.c:2598)
[ 7025.976177] dev_create (drivers/md/dm-ioctl.c:747)
[ 7025.976252] ? list_version_get_info (drivers/md/dm-ioctl.c:735)
[ 7025.976341] ctl_ioctl (drivers/md/dm-ioctl.c:1848)
[ 7025.976414] ? semctl_main (ipc/sem.c:1330)
[ 7025.976492] ? debug_lockdep_rcu_enabled (kernel/rcu/update.c:195)
[ 7025.976583] ? list_version_get_info (drivers/md/dm-ioctl.c:735)
[ 7025.976672] ? free_params (drivers/md/dm-ioctl.c:1793)
[ 7025.976749] ? SYSC_semtimedop (ipc/sem.c:2010)
[ 7025.980019] dm_ctl_ioctl (drivers/md/dm-ioctl.c:1866)
[ 7025.983301] do_vfs_ioctl (fs/ioctl.c:44 fs/ioctl.c:607)
[ 7025.986587] ? ioctl_preallocate (fs/ioctl.c:557)
[ 7025.989788] ? mntput (fs/namespace.c:1106)
[ 7025.992915] ? SyS_semctl (ipc/sem.c:1601 ipc/sem.c:1577)
[ 7025.995989] ? debug_lockdep_rcu_enabled (kernel/rcu/update.c:195)
[ 7025.998981] ? __fget_light (fs/file.c:684)
[ 7026.001895] SyS_ioctl (fs/ioctl.c:622 fs/ioctl.c:613)
[ 7026.004724] system_call_fastpath (arch/x86/kernel/entry_64.S:261)
[ 7026.007455] Code: d7 3d 83 41 55 49 89 d5 41 54 41 89 cc 53 48 89 f3 48 83 ec 08 e8 10 b6 1e 01 48 89 da 48 b8 00 00 00 00 00 fc ff df 48 c1 ea 03 <80> 3c 02 00 0f 85 9a 00 00 00 48 8b 1b 48 85 db 74 4d 48 89 df
All code
========
0: d7 xlat %ds:(%rbx)
1: 3d 83 41 55 49 cmp $0x49554183,%eax
6: 89 d5 mov %edx,%ebp
8: 41 54 push %r12
a: 41 89 cc mov %ecx,%r12d
d: 53 push %rbx
e: 48 89 f3 mov %rsi,%rbx
11: 48 83 ec 08 sub $0x8,%rsp
15: e8 10 b6 1e 01 callq 0x11eb62a
1a: 48 89 da mov %rbx,%rdx
1d: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
24: fc ff df
27: 48 c1 ea 03 shr $0x3,%rdx
2b:* 80 3c 02 00 cmpb $0x0,(%rdx,%rax,1) <-- trapping instruction
2f: 0f 85 9a 00 00 00 jne 0xcf
35: 48 8b 1b mov (%rbx),%rbx
38: 48 85 db test %rbx,%rbx
3b: 74 4d je 0x8a
3d: 48 89 df mov %rbx,%rdi
Code starting with the faulting instruction
===========================================
0: 80 3c 02 00 cmpb $0x0,(%rdx,%rax,1)
4: 0f 85 9a 00 00 00 jne 0xa4
a: 48 8b 1b mov (%rbx),%rbx
d: 48 85 db test %rbx,%rbx
10: 74 4d je 0x5f
12: 48 89 df mov %rbx,%rdi
[ 7026.013551] RIP sysfs_do_create_link_sd.isra.2 (fs/sysfs/symlink.c:35)
[ 7026.016243] RSP <ffff881fda807878>
[ 7026.142471] kobject: 'holders' (ffff881ff0e9fe00): kobject_cleanup, parent ffff8837b82ee0a8
[ 7026.144127] kobject: 'holders' (ffff881ff0e9fe00): auto cleanup kobject_del
[ 7026.498540] ---[ end trace 734c93b316c19e45 ]---
Thanks,
Sasha
^ permalink raw reply
* interesting MD-xfs bug
From: Joe Landman @ 2015-04-09 21:02 UTC (permalink / raw)
To: xfs, linux-raid
If I build an MD raid0 with a non power of 2 chunk size, it appears that
I can mkfs.xfs a file system, but it doesn't show up in blkid and is not
mountable. Yet, using a power of 2 chunk size, this does work
correctly. This is kernel 3.18.9.
For example, non-power of 2 chunk:
root@unison:~# wipefs -a /dev/sdb
4 bytes were erased at offset 0x1000 (linux_raid_member)
they were: fc 4e 2b a9
root@unison:~# wipefs -a /dev/sda
4 bytes were erased at offset 0x1000 (linux_raid_member)
they were: fc 4e 2b a9
root@unison:~# mdadm --create /dev/md20 --level=0 --metadata=1.2
--chunk=1152 --auto=yes --raid-disks=2 /dev/sd[ab]
mdadm: array /dev/md20 started.
root@unison:~# mkfs.xfs /dev/md20
log stripe unit (1179648 bytes) is too large (maximum is 256KiB)
log stripe unit adjusted to 32KiB
meta-data=/dev/md20 isize=256 agcount=50,
agsize=268435296 blks
= sectsz=512 attr=2, projid32bit=0
data = bsize=4096 blocks=13164865984, imaxpct=5
= sunit=288 swidth=576 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal log bsize=4096 blocks=521728, version=2
= sectsz=512 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
root@unison:~# blkid | grep xfs
root@unison:~#
Same system, with power of 2 chunk size:
root@unison:~# mdadm -S /dev/md20
mdadm: stopped /dev/md20
root@unison:~# wipefs -a /dev/sda
4 bytes were erased at offset 0x1000 (linux_raid_member)
they were: fc 4e 2b a9
root@unison:~# wipefs -a /dev/sdb
4 bytes were erased at offset 0x1000 (linux_raid_member)
they were: fc 4e 2b a9
root@unison:~# mdadm --create /dev/md20 --level=0 --metadata=1.2
--chunk=1024 --auto=yes --raid-disks=2 /dev/sd[ab]
mdadm: array /dev/md20 started.
root@unison:~# mkfs.xfs /dev/md20
log stripe unit (1048576 bytes) is too large (maximum is 256KiB)
log stripe unit adjusted to 32KiB
meta-data=/dev/md20 isize=256 agcount=50,
agsize=268435200 blks
= sectsz=512 attr=2, projid32bit=0
data = bsize=4096 blocks=13164866048, imaxpct=5
= sunit=256 swidth=512 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal log bsize=4096 blocks=521728, version=2
= sectsz=512 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
root@unison:~# blkid | grep xfs
/dev/md20: UUID="5e965ae7-198e-4e58-8920-a65c4b6bbe60" TYPE="xfs"
I am not sure which code base might be at "fault" or even if there is a
"fault" (beyond simply saying "don't do non-power-of-two chunks"). If
its the latter, happy to work on a warning message patch for mdadm if
needed. If it should work, then happy to poke around if someone can
give me a pointer where something might be relevant.
^ permalink raw reply
* Re: interesting MD-xfs bug
From: Dave Chinner @ 2015-04-09 22:18 UTC (permalink / raw)
To: Joe Landman; +Cc: xfs, linux-raid
In-Reply-To: <5526E8E9.3030805@gmail.com>
On Thu, Apr 09, 2015 at 05:02:33PM -0400, Joe Landman wrote:
> If I build an MD raid0 with a non power of 2 chunk size, it appears
> that I can mkfs.xfs a file system, but it doesn't show up in blkid
> and is not mountable. Yet, using a power of 2 chunk size, this does
> work correctly. This is kernel 3.18.9.
>
>
> For example, non-power of 2 chunk:
>
> root@unison:~# wipefs -a /dev/sdb
> 4 bytes were erased at offset 0x1000 (linux_raid_member)
> they were: fc 4e 2b a9
> root@unison:~# wipefs -a /dev/sda
> 4 bytes were erased at offset 0x1000 (linux_raid_member)
> they were: fc 4e 2b a9
> root@unison:~# mdadm --create /dev/md20 --level=0 --metadata=1.2
> --chunk=1152 --auto=yes --raid-disks=2 /dev/sd[ab]
> mdadm: array /dev/md20 started.
>
> root@unison:~# mkfs.xfs /dev/md20
> log stripe unit (1179648 bytes) is too large (maximum is 256KiB)
> log stripe unit adjusted to 32KiB
> meta-data=/dev/md20 isize=256 agcount=50,
> agsize=268435296 blks
> = sectsz=512 attr=2, projid32bit=0
> data = bsize=4096 blocks=13164865984, imaxpct=5
> = sunit=288 swidth=576 blks
> naming =version 2 bsize=4096 ascii-ci=0
> log =internal log bsize=4096 blocks=521728, version=2
> = sectsz=512 sunit=8 blks, lazy-count=1
> realtime =none extsz=4096 blocks=0, rtextents=0
>
> root@unison:~# blkid | grep xfs
That looks more like a blkid or udev problem. try using blkid -p so
that it doesn't look up the cache but directly probes devices for
the signatures. strace might tell you a bit more, too. And if the
filesystem mounts, then it definitely isn't an XFS problem ;)
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply
* Re: interesting MD-xfs bug
From: Joe Landman @ 2015-04-09 22:20 UTC (permalink / raw)
To: Dave Chinner; +Cc: xfs, linux-raid
In-Reply-To: <20150409221846.GG13731@dastard>
On 04/09/2015 06:18 PM, Dave Chinner wrote:
> On Thu, Apr 09, 2015 at 05:02:33PM -0400, Joe Landman wrote:
>> If I build an MD raid0 with a non power of 2 chunk size, it appears
>> that I can mkfs.xfs a file system, but it doesn't show up in blkid
>> and is not mountable. Yet, using a power of 2 chunk size, this does
>> work correctly. This is kernel 3.18.9.
>>
[...]
> That looks more like a blkid or udev problem. try using blkid -p so
> that it doesn't look up the cache but directly probes devices for
> the signatures. strace might tell you a bit more, too. And if the
> filesystem mounts, then it definitely isn't an XFS problem ;)
Thats the thing, it didn't mount, even when I used the device name
directly.
Good point on stracing though. I'll do that tomorrow and report back.
Thanks!
Joe
>
> Cheers,
>
> Dave.
--
Joe Landman
e: joe.landman@gmail.com
t: @sijoe
^ permalink raw reply
* Re: interesting MD-xfs bug
From: Dave Chinner @ 2015-04-09 22:53 UTC (permalink / raw)
To: Joe Landman; +Cc: xfs, linux-raid
In-Reply-To: <5526FB2A.8060704@gmail.com>
On Thu, Apr 09, 2015 at 06:20:26PM -0400, Joe Landman wrote:
>
>
> On 04/09/2015 06:18 PM, Dave Chinner wrote:
> >On Thu, Apr 09, 2015 at 05:02:33PM -0400, Joe Landman wrote:
> >>If I build an MD raid0 with a non power of 2 chunk size, it appears
> >>that I can mkfs.xfs a file system, but it doesn't show up in blkid
> >>and is not mountable. Yet, using a power of 2 chunk size, this does
> >>work correctly. This is kernel 3.18.9.
> >>
>
> [...]
>
> >That looks more like a blkid or udev problem. try using blkid -p so
> >that it doesn't look up the cache but directly probes devices for
> >the signatures. strace might tell you a bit more, too. And if the
> >filesystem mounts, then it definitely isn't an XFS problem ;)
>
> Thats the thing, it didn't mount, even when I used the device name
> directly.
Ok, that's interesting. Let me see if I can reproduce it locally. If
you don't hear otherwise, tracing would still be useful. Thanks for
the bug report, Joe.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply
* Re: interesting MD-xfs bug
From: Dave Chinner @ 2015-04-09 23:10 UTC (permalink / raw)
To: Joe Landman; +Cc: linux-raid, xfs
In-Reply-To: <20150409225322.GH13731@dastard>
On Fri, Apr 10, 2015 at 08:53:22AM +1000, Dave Chinner wrote:
> On Thu, Apr 09, 2015 at 06:20:26PM -0400, Joe Landman wrote:
> >
> >
> > On 04/09/2015 06:18 PM, Dave Chinner wrote:
> > >On Thu, Apr 09, 2015 at 05:02:33PM -0400, Joe Landman wrote:
> > >>If I build an MD raid0 with a non power of 2 chunk size, it appears
> > >>that I can mkfs.xfs a file system, but it doesn't show up in blkid
> > >>and is not mountable. Yet, using a power of 2 chunk size, this does
> > >>work correctly. This is kernel 3.18.9.
> > >>
> >
> > [...]
> >
> > >That looks more like a blkid or udev problem. try using blkid -p so
> > >that it doesn't look up the cache but directly probes devices for
> > >the signatures. strace might tell you a bit more, too. And if the
> > >filesystem mounts, then it definitely isn't an XFS problem ;)
> >
> > Thats the thing, it didn't mount, even when I used the device name
> > directly.
>
> Ok, that's interesting. Let me see if I can reproduce it locally. If
> you don't hear otherwise, tracing would still be useful. Thanks for
> the bug report, Joe.
No luck - md doesn't allow the device to be activated on 4.0-rc7:
$ sudo mdadm --version
mdadm - v3.3.2 - 21st August 2014
$ uname -a
Linux test4 4.0.0-rc7-dgc+ #882 SMP Fri Apr 10 08:50:52 AEST 2015 x86_64 GNU/Linux
$ sudo wipefs -a /dev/vd[ab]
/dev/vda: 4 bytes were erased at offset 0x00001000 (linux_raid_member): fc 4e 2b a9
/dev/vdb: 4 bytes were erased at offset 0x00001000 (linux_raid_member): fc 4e 2b a9
$ sudo mdadm --create /dev/md20 --level=0 --metadata=1.2 --chunk=1152 --auto=yes --raid-disks=2 /dev/vd[ab]
mdadm: RUN_ARRAY failed: Invalid argument
Problem may be that chunk size is not a power of 2
$ cat /proc/mdstat
Personalities : [raid1] [raid10] [raid6] [raid5] [raid4]
unused devices: <none>
$
So I can't actually reproduce what you are seeing because MD doesn't
allow the device to be activated and so mdadm tears it back down.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply
* Re: interesting MD-xfs bug
From: NeilBrown @ 2015-04-09 23:36 UTC (permalink / raw)
To: Dave Chinner; +Cc: Joe Landman, linux-raid, xfs
In-Reply-To: <20150409231035.GI13731@dastard>
[-- Attachment #1: Type: text/plain, Size: 2288 bytes --]
On Fri, 10 Apr 2015 09:10:35 +1000 Dave Chinner <david@fromorbit.com> wrote:
> On Fri, Apr 10, 2015 at 08:53:22AM +1000, Dave Chinner wrote:
> > On Thu, Apr 09, 2015 at 06:20:26PM -0400, Joe Landman wrote:
> > >
> > >
> > > On 04/09/2015 06:18 PM, Dave Chinner wrote:
> > > >On Thu, Apr 09, 2015 at 05:02:33PM -0400, Joe Landman wrote:
> > > >>If I build an MD raid0 with a non power of 2 chunk size, it appears
> > > >>that I can mkfs.xfs a file system, but it doesn't show up in blkid
> > > >>and is not mountable. Yet, using a power of 2 chunk size, this does
> > > >>work correctly. This is kernel 3.18.9.
> > > >>
> > >
> > > [...]
> > >
> > > >That looks more like a blkid or udev problem. try using blkid -p so
> > > >that it doesn't look up the cache but directly probes devices for
> > > >the signatures. strace might tell you a bit more, too. And if the
> > > >filesystem mounts, then it definitely isn't an XFS problem ;)
> > >
> > > Thats the thing, it didn't mount, even when I used the device name
> > > directly.
> >
> > Ok, that's interesting. Let me see if I can reproduce it locally. If
> > you don't hear otherwise, tracing would still be useful. Thanks for
> > the bug report, Joe.
>
> No luck - md doesn't allow the device to be activated on 4.0-rc7:
>
> $ sudo mdadm --version
> mdadm - v3.3.2 - 21st August 2014
> $ uname -a
> Linux test4 4.0.0-rc7-dgc+ #882 SMP Fri Apr 10 08:50:52 AEST 2015 x86_64 GNU/Linux
> $ sudo wipefs -a /dev/vd[ab]
> /dev/vda: 4 bytes were erased at offset 0x00001000 (linux_raid_member): fc 4e 2b a9
> /dev/vdb: 4 bytes were erased at offset 0x00001000 (linux_raid_member): fc 4e 2b a9
> $ sudo mdadm --create /dev/md20 --level=0 --metadata=1.2 --chunk=1152 --auto=yes --raid-disks=2 /dev/vd[ab]
Weird. Works for me.
Any messages in 'dmesg' ??
How big are /dev/vd[ab]??
NeilBrown
> mdadm: RUN_ARRAY failed: Invalid argument
> Problem may be that chunk size is not a power of 2
> $ cat /proc/mdstat
> Personalities : [raid1] [raid10] [raid6] [raid5] [raid4]
> unused devices: <none>
> $
>
> So I can't actually reproduce what you are seeing because MD doesn't
> allow the device to be activated and so mdadm tears it back down.
>
> Cheers,
>
> Dave.
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: interesting MD-xfs bug
From: Dave Chinner @ 2015-04-10 1:31 UTC (permalink / raw)
To: NeilBrown; +Cc: Joe Landman, linux-raid, xfs
In-Reply-To: <20150410093652.73204748@notabene.brown>
On Fri, Apr 10, 2015 at 09:36:52AM +1000, NeilBrown wrote:
> On Fri, 10 Apr 2015 09:10:35 +1000 Dave Chinner <david@fromorbit.com> wrote:
>
> > On Fri, Apr 10, 2015 at 08:53:22AM +1000, Dave Chinner wrote:
> > > On Thu, Apr 09, 2015 at 06:20:26PM -0400, Joe Landman wrote:
> > > >
> > > >
> > > > On 04/09/2015 06:18 PM, Dave Chinner wrote:
> > > > >On Thu, Apr 09, 2015 at 05:02:33PM -0400, Joe Landman wrote:
> > > > >>If I build an MD raid0 with a non power of 2 chunk size, it appears
> > > > >>that I can mkfs.xfs a file system, but it doesn't show up in blkid
> > > > >>and is not mountable. Yet, using a power of 2 chunk size, this does
> > > > >>work correctly. This is kernel 3.18.9.
> > > > >>
> > > >
> > > > [...]
> > > >
> > > > >That looks more like a blkid or udev problem. try using blkid -p so
> > > > >that it doesn't look up the cache but directly probes devices for
> > > > >the signatures. strace might tell you a bit more, too. And if the
> > > > >filesystem mounts, then it definitely isn't an XFS problem ;)
> > > >
> > > > Thats the thing, it didn't mount, even when I used the device name
> > > > directly.
> > >
> > > Ok, that's interesting. Let me see if I can reproduce it locally. If
> > > you don't hear otherwise, tracing would still be useful. Thanks for
> > > the bug report, Joe.
> >
> > No luck - md doesn't allow the device to be activated on 4.0-rc7:
> >
> > $ sudo mdadm --version
> > mdadm - v3.3.2 - 21st August 2014
> > $ uname -a
> > Linux test4 4.0.0-rc7-dgc+ #882 SMP Fri Apr 10 08:50:52 AEST 2015 x86_64 GNU/Linux
> > $ sudo wipefs -a /dev/vd[ab]
> > /dev/vda: 4 bytes were erased at offset 0x00001000 (linux_raid_member): fc 4e 2b a9
> > /dev/vdb: 4 bytes were erased at offset 0x00001000 (linux_raid_member): fc 4e 2b a9
> > $ sudo mdadm --create /dev/md20 --level=0 --metadata=1.2 --chunk=1152 --auto=yes --raid-disks=2 /dev/vd[ab]
>
> Weird. Works for me.
> Any messages in 'dmesg' ??
> How big are /dev/vd[ab]??
vda is 5GB, vdb is 20GB
dmesg:
[ 125.131340] md: bind<vda>
[ 125.134547] md: bind<vdb>
[ 125.139669] md: personality for level 0 is not loaded!
[ 125.141302] md: md20 stopped.
[ 125.141986] md: unbind<vdb>
[ 125.160100] md: export_rdev(vdb)
[ 125.161751] md: unbind<vda>
[ 125.180126] md: export_rdev(vda)
Oh, curious. Going from 4.0-rc4 to 4.0-rc7, and make oldconfig
has resulted in:
# CONFIG_MD_RAID0 is not set
Ok, so with that fixed, it's still horribly broken.
RAID 0 on different sized devices should result in a device that is
twice the size of the smallest devices:
$ sudo mdadm --create /dev/md20 --level=raid0 --metadata=1.2 --chunk=1024 --auto=yes --raid-disks=2 /dev/vd[ab]
mdadm: array /dev/md20 started.
$ cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md20 : active raid0 vdb[1] vda[0]
26206208 blocks super 1.2 1024k chunks
unused devices: <none>
$ grep "md\|vd[ab]" /proc/partitions
253 0 5242880 vda
253 16 20971520 vdb
9 20 26206208 md20
$
Oh, "RAID0" is not actually RAID 0 - that's the size I'd expect from
a linear mapping. Half way through writing that block device, the IO
stats change in an obvious way:
Device: r/s w/s rMB/s wMB/s
vda 0.00 144.00 0.00 48.00
vdb 0.00 145.20 0.00 48.40
md20 0.00 290.40 0.00 96.80
Device: r/s w/s rMB/s wMB/s
vda 0.00 56.40 0.00 18.80
vdb 0.00 229.20 0.00 76.40
md20 0.00 285.20 0.00 95.10
Device: r/s w/s rMB/s wMB/s
vda 0.00 0.00 0.00 0.00
vdb 0.00 290.40 0.00 96.80
md20 0.00 290.80 0.00 96.90
So it's actually a stripe for the first 10GB, then some kind of
concatenated mapping of the remainder of the single device. That's
not what I expected, but it's also clearly not the problem.
Anyway, change the stripe size to 1152:
sudo mdadm --stop /dev/md20
mdadm: stopped /dev/md20
$ sudo wipefs -a /dev/vd[ab]
/dev/vda: 4 bytes were erased at offset 0x00001000 (linux_raid_member): fc 4e 2b a9
/dev/vdb: 4 bytes were erased at offset 0x00001000 (linux_raid_member): fc 4e 2b a9
$ sudo mdadm --create /dev/md20 --level=raid0 --metadata=1.2 --chunk=1152 --auto=yes --raid-disks=2 /dev/vd[ab]
mdadm: array /dev/md20 started.
$ sudo xfs_io -fd -c "pwrite -b 4m 0 25g" /dev/md20
wrote 26831355904/26843545600 bytes at offset 0
24.989 GiB, 6398 ops; 0:00:16.00 (1.530 GiB/sec and 391.8556 ops/sec)
$
Wait, what? Neil, did you put a flux capacitor in MD? :P
The underlying drive is only capable of 100MB/s - 25GB of sequential
direct IO does not complete in 16 seconds on such a drive. But
there's also a 1GB BBWC in front of the physical drives (HW RAID1),
but even so, this write rate could only occur if every write is
hitting the BBWC. And so it is:
$ sudo xfs_io -fd -c "pwrite -b 4m 0 25g" /dev/md20 & iostat -d -m 1
...
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
vda 4214.00 0.00 1516.99 0 1516
vdb 0.00 0.00 0.00 0 0
md20 4223.00 0.00 1520.00 0 1520
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
vda 2986.00 0.00 1075.01 0 1075
vdb 1174.00 0.00 422.88 0 422
md20 4154.00 0.00 1496.00 0 1496
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
vda 0.00 0.00 0.00 0 0
vdb 4376.00 0.00 1575.12 0 1575
md20 4378.00 0.00 1576.00 0 1576
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
vda 2682.00 0.00 965.74 0 965
vdb 1650.00 0.00 594.00 0 594
md20 4334.00 0.00 1560.00 0 1560
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
vda 4518.00 0.00 1626.26 0 1626
vdb 138.00 0.00 49.50 0 49
md20 4656.00 0.00 1676.00 0 1676
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
vda 0.00 0.00 0.00 0 0
vdb 4214.00 0.00 1517.48 0 1517
md20 4210.00 0.00 1516.00 0 1516
.....
Note how it is cycling from one drive to the other with about a 2s
period?
Yup, blocktrace on /dev/vda shows it is, indeed, hitting the BBWC
because the block mapping is clearly broken:
253,0 4 1 0.000000000 6972 Q WS 8192 + 1008 [xfs_io]
253,0 4 5 0.000068012 6972 Q WS 8192 + 1008 [xfs_io]
253,0 4 9 0.000093266 6972 Q WS 8192 + 288 [xfs_io]
253,0 4 13 0.000129722 6972 Q WS 8193 + 1008 [xfs_io]
253,0 4 17 0.000176872 6972 Q WS 8193 + 1008 [xfs_io]
253,0 4 21 0.000205566 6972 Q WS 8193 + 288 [xfs_io]
253,0 4 25 0.000240846 6972 Q WS 8194 + 1008 [xfs_io]
253,0 4 29 0.000284990 6972 Q WS 8194 + 1008 [xfs_io]
253,0 4 33 0.000313276 6972 Q WS 8194 + 288 [xfs_io]
253,0 4 37 0.000352330 6972 Q WS 8195 + 1008 [xfs_io]
253,0 4 41 0.000374272 6972 Q WS 8195 + 272 [xfs_io]
253,0 4 56 0.001215857 6972 Q WS 8195 + 1008 [xfs_io]
253,0 4 60 0.001252697 6972 Q WS 8195 + 16 [xfs_io]
253,0 4 64 0.001284517 6972 Q WS 8196 + 1008 [xfs_io]
253,0 4 68 0.001326130 6972 Q WS 8196 + 1008 [xfs_io]
253,0 4 72 0.001355050 6972 Q WS 8196 + 288 [xfs_io]
253,0 4 76 0.001393777 6972 Q WS 8197 + 1008 [xfs_io]
253,0 4 80 0.001439547 6972 Q WS 8197 + 1008 [xfs_io]
253,0 4 84 0.001466097 6972 Q WS 8197 + 288 [xfs_io]
253,0 4 88 0.001501267 6972 Q WS 8198 + 1008 [xfs_io]
253,0 4 92 0.001545863 6972 Q WS 8198 + 1008 [xfs_io]
253,0 4 96 0.001571500 6972 Q WS 8198 + 288 [xfs_io]
253,0 4 100 0.001584620 6972 Q WS 8199 + 256 [xfs_io]
253,0 4 116 0.002730034 6972 Q WS 8199 + 1008 [xfs_io]
253,0 4 120 0.002792351 6972 Q WS 8199 + 1008 [xfs_io]
253,0 4 124 0.002810937 6972 Q WS 8199 + 32 [xfs_io]
253,0 4 128 0.002842047 6972 Q WS 8200 + 1008 [xfs_io]
253,0 4 132 0.002889087 6972 Q WS 8200 + 1008 [xfs_io]
253,0 4 136 0.002916894 6972 Q WS 8200 + 288 [xfs_io]
253,0 4 140 0.002952334 6972 Q WS 8201 + 1008 [xfs_io]
253,0 4 144 0.002996101 6972 Q WS 8201 + 1008 [xfs_io]
253,0 4 148 0.003022401 6972 Q WS 8201 + 288 [xfs_io]
Multiple IOs to teh same sector, then the sector increments by 1 and
we get more IOs to the same sector offset. After about a second the
mapping shifts IO to the other block device as it slowly increments
the sector, and that's why we see that cycling behaviour.
IOWs, something is going wrong with the MD block mapping when the
RAID chunk size is not a power of 2....
Over to you, Neil....
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply
* RAID IO stuck under heavy load (writes)
From: Tejas Rao @ 2015-04-10 1:54 UTC (permalink / raw)
To: linux-raid
We are running GPFS over md raid devices.
The GPFS storage servers each have 60 4TB jbod drives. This is setup in
a 6 RAID6 md devices(8+2), with default chunk size 512K. We are running
RHEL 6.5 , kernel 2.6.32-431.23.3.el6.x86_64.
stripe_cache_size for each md device is set to maximum 32768. If I set
the stripe_cache_size to 16384 or lower, I see stuck IO even at lower
work loads.
Under heavy write load we see IO getting stuck for several minutes (GPFS
waiters), sometimes as long as 30 minutes, eventually they all complete.
I see stripe_cache_active on the stuck md device close to maximum and
not changing(stuck?).
This happens randomly on different md devices on different servers, so I
am sure this is not a hardware problem tied to a failing disk/SAS port etc.
How can troubleshoot this further to isolate the cause? I am reproduce
this problem 100%.
This is what I see in the /var/log/messages file. mmfslinux/mmfs26 is
the GPFS application.
Feb 5 12:24:10 host12 kernel: Not tainted 2.6.32-431.23.3.el6.x86_64 #1
Feb 5 12:24:10 host12 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 5 12:24:10 host12 kernel: mmfsd D 0000000000000012 0 28987 28418
0x00000080
Feb 5 12:24:10 host12 kernel: ffff880c9f1ffbe8 0000000000000082
0000000000000000 ffffffffa02833e8
Feb 5 12:24:10 host12 kernel: ffff880c9f1ffc48 ffffffffa088c133
0000000000016840 ffff880872b07740
Feb 5 12:24:10 host12 kernel: ffff88007dc3faf8 ffff880c9f1fffd8
000000000000fbc8 ffff88007dc3faf8
Feb 5 12:24:10 host12 kernel: Call Trace:
Feb 5 12:24:10 host12 kernel: [<ffffffffa02833e8>] ?
raid5_unplug_queue+0x18/0x20 [raid456]
Feb 5 12:24:10 host12 kernel: [<ffffffffa088c133>] ?
cxiStartIO+0x2a3/0x6b0 [mmfslinux]
Feb 5 12:24:10 host12 kernel: [<ffffffffa0888b6c>] cxiWaitIO+0x13c/0x1a0
[mmfslinux]
Feb 5 12:24:10 host12 kernel: [<ffffffff8109afa0>] ?
autoremove_wake_function+0x0/0x40
Feb 5 12:24:10 host12 kernel: [<ffffffffa0913c8d>]
_ZN9DiskSched7localIOEPP15MBDoDiskIOParmsiiP15KernelOperation+0x49d/0x6d0 [mmfs26]
Feb 5 12:24:10 host12 kernel: [<ffffffffa09132d0>] ?
_Z22LinuxIODoneIntCallbackPvj+0x0/0x2a0 [mmfs26]
Feb 5 12:24:10 host12 kernel: [<ffffffffa0913f8d>] ?
kxLocalIO+0xcd/0x110 [mmfs26]
Feb 5 12:24:10 host12 kernel: [<ffffffff810129de>] ?
copy_user_generic+0xe/0x20
Feb 5 12:24:10 host12 kernel: [<ffffffffa09e0755>] ?
_Z8ss_ioctljm+0x345/0x1650 [mmfs26]
Feb 5 12:24:10 host12 kernel: [<ffffffff8100b9ce>] ?
common_interrupt+0xe/0x13
Feb 5 12:24:10 host12 kernel: [<ffffffff8100b9ce>] ?
common_interrupt+0xe/0x13
Feb 5 12:24:10 host12 kernel: [<ffffffffa089a199>] ?
ss_fs_unlocked_ioctl+0x89/0x3e0 [mmfslinux]
Feb 5 12:24:10 host12 kernel: [<ffffffff8100b9ce>] ?
common_interrupt+0xe/0x13
Feb 5 12:24:10 host12 kernel: [<ffffffff8119e532>] ? vfs_ioctl+0x22/0xa0
Feb 5 12:24:10 host12 kernel: [<ffffffff8119e6e7>] ? do_vfs_ioctl+0x97/0x580
Feb 5 12:24:10 host12 kernel: [<ffffffff8119e6d4>] ? do_vfs_ioctl+0x84/0x580
Feb 5 12:24:10 host12 kernel: [<ffffffff8119ec51>] ? sys_ioctl+0x81/0xa0
Feb 5 12:24:10 host12 kernel: [<ffffffff810e1cde>] ?
__audit_syscall_exit+0x25e/0x290
Feb 5 12:24:10 host12 kernel: [<ffffffff8100b072>] ?
system_call_fastpath+0x16/0x1b
^ permalink raw reply
* Personal insight - Re: Can't mount partitions after "mdadm --zero-superblock"
From: Eduard Rozenberg @ 2015-04-10 2:53 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <22789EEC-4F93-4B65-A93F-987DEE075C43@pobox.com>
I finally figured out why I panicked and why things eventually worked
thanks to this very helpful post:
http://unix.stackexchange.com/questions/64889/how-to-mount-recover-data-on-a-disk-that-was-part-of-a-mdadm-raid-1-on-another-m
In particular this paragraph:
"Linux mdraid has several metadata formats. Formats 0.9 and 1.0 put the metadata
at the end of the containing device, and the payload (the filesystem) starts at
the beginning of the device and can be accessed directly without going through
the raid layer. Formats 1.1 and 1.2 put the metadata at the middle and
beginning of the containing device respectively, so the payload is at an offset."
I knew about the various metadata versions but I’d been working under the false
assumption (based on my 0.9/1.0 metadata format mdadm experience) that
I would be able to directly mount an ext4 partition that had been previously part of
a RAID1 mdadm device.
According to the paragraph above this is clearly no longer true with metadata
v1.1/1.2, and mounting a previously raid1 partition requires either the decimal
offset to mount it directly or creating a new md device with the partition in question
(without of course formatting the new md device). Certainly not as convenient
as it used to be but I'm sure there were good reasons to change the location of
the metadata to the front of each partition.
Perhaps the mdadm wiki could include information to this effect, that mounting
formerly raid1 mdadm partitions with metadata 1.1/1.2 requires these extra
steps and not to panic :).
Regards,
—Ed
> On Apr 6, 2015, at 20:31, Eduard Rozenberg <eduardr@pobox.com> wrote:
>
> Hello Neil,
>
> Success! This is stronger voodoo magic than I’ve ever
> had to perform so really didn’t have the faith to continue
> without the extra encouragement :). My initial mistake
> had been to use the octal value with losetup instead of
> getting the decimal value.
>
> Documenting my steps below for anybody else who
> might come here later.
>
> In the examples below, we are using partition "/dev/sdac2"
> Replace this with the appropriate partition you’re recovering.
>
> Step 1: find the decimal value for the start of the partition
> ----------------------------------------------------------------------------
>
> Note:
> ext4 partitions have a “magic" octal value of “ef53"
> to indicate the start of the partition. Note that "ef53"
> may show more than once as you read further into
> the partition. We are interested in the location the
> very first occurrence of "ef53". Other types of
> partitions (ext2, ext3, etc) probably have other magic
> values to look for so this “ef53" may not apply there.
>
> The "od" hex viewer command to search for “ef53":
>
> od -x /dev/sdac2 | awk '$6 == "ef53"'
>
> The results will look something like:
>
> 4002060 f3fd 5521 0004 0025 ef53 0001 0001 0000
> 1004000060 64be 4ec9 0000 0025 ef53 0000 0001 0000
> 1042630400 17f8 a7dd bb6e ee40 ef53 000d 3cfb 9e22
>
> We are only interested in the first line. So we now have
> the octal address of the ef53 magic value: it’s the first
> long number on the line: "4002060" (octal value!)
>
> According to Neil’s instructions then we have to subtract
> the octal value “0002060" from this number we found.
> We then have to convert the octal result into decimal.
>
> Luckily an online calculator makes this easy:
>
> http://www.csgnetwork.com/octaddsubcalc.html
>
> "Enter a octal value" - Enter “4002060" here
> "Enter Second Octal Value" - Enter “0002060" here
>
> Then take the value from the line:
> "Calculated Decimal Subtraction" - 1048576
>
> This is the decimal value for the start of our partition.
>
>
> Step 2: use the decimal start value to mount partition
> ———————————————————————————————————
>
> First create a loop device loop0 as a handle to the
> partition. We tell losetup where the start of the partition is:
>
> losetup -o 1048576 /dev/loop0 /dev/sdac2
>
> Next, try to mount loop0 read-only (hopefully it will work!)
>
> mount -o ro /dev/loop0 /mnt
>
> If the partition is unclean and needs to be fscked:
>
> fsck.ext4 /dev/loop0
>
>
> Thanks again Neil! Maybe a few years from now I’ll
> understand why this worked when nothing else did
> ( linux tools still have some ways to go to being
> intelligent enough to do this kind of recovery).
>
> Regards,
> —Ed
>
>
>> On Apr 6, 2015, at 17:52, NeilBrown <neilb@suse.de> wrote:
>>
>> On Mon, 6 Apr 2015 16:45:58 -0700 Eduard Rozenberg <eduardr@pobox.com> wrote:
>>
>>> Hello folks,
>>>
>>> I previously had the following setup:
>>>
>>> sda & sdb partitioned w/ GPT, 7 partitions each (usr, opt, var etc...)
>>> 7 raid1’s with 2 devices for each pair of partitions (/dev/sda1 & /dev/sdab1, etc)
>>> They’d been created under Slackware 13.37.
>>>
>>> I was trying to clean out mdadm from those partitions but keep the data so I ran
>>> "mdadm --zero-superblock” on each of those previously RAID1 mdadm 1.2 ext4
>>> partitions.
>>
>> The "1.2" metadata is stored 4k from the start of the device. The actual
>> data is some megabytes further in. I don't suppose you still have the output
>> of "mdadm --examine" from before you destroyed the superblocks??
>>
>>>
>>> As a result I am now currently unable to mount any partition after the first one on either
>>> disk. The first partition does mount. The partition table is visible and looks fine in gdisk.
>>>
>>> mount -t ext4 /dev/sdac2 /mnt
>>> mount: wrong fs type, bad option, bad superblock on /dev/sdac2,
>>> missing codepage or helper program, or other error
>>> In some cases useful info is found in syslog - try
>>> dmesg | tail or so
>>>
>>> I did try superblock recovery with each backup superblock that ext4 normally creates,
>>> but none of the superblock locations worked.
>>>
>>> For example:
>>>
>>> fsck.ext4 -b 4096000 /dev/sdac2
>>> e2fsck 1.42.8 (20-Jun-2013)
>>> /sbin/e2fsck: Invalid argument while trying to open /dev/sdac2
>>>
>>> The superblock could not be read or does not describe a correct ext2
>>> filesystem. If the device is valid and it really contains an ext2
>>> filesystem (and not swap or ufs or something else), then the superblock
>>> is corrupt, and you might try running e2fsck with an alternate superblock:
>>> e2fsck -b 8193 <device>
>>>
>>>
>>> Would be grateful for any advice on anything else I can try.
>>
>> You need to find where the filesystem actually starts, then you need to
>> create some way to access it as a block device, then it should "just work".
>>
>> An ext4 filesystem superblock has 0xef53 at an offset of 0x38, and the
>> superblock is typically 1K from the start of the partition.
>>
>> So you could:
>> od -x /dev/sdac2 | awk '$6 == "ef53"'
>>
>> Then subtrace 0002060 (octal) from the leading number, and that might be the
>> start of the partition.
>>
>> Then
>> losetup -o "start in decimal" /dev/loop0 /dev/sdac2
>>
>> and try 'fsck' on /dev/loop0
>>
>> Good luck.
>>
>> NeilBrown
>>
>>
>>
>>>
>>> Regards,
>>> —Ed--
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: interesting MD-xfs bug
From: NeilBrown @ 2015-04-10 3:22 UTC (permalink / raw)
To: Dave Chinner; +Cc: Joe Landman, linux-raid, xfs
In-Reply-To: <20150410013156.GH15810@dastard>
[-- Attachment #1: Type: text/plain, Size: 11947 bytes --]
On Fri, 10 Apr 2015 11:31:57 +1000 Dave Chinner <david@fromorbit.com> wrote:
> On Fri, Apr 10, 2015 at 09:36:52AM +1000, NeilBrown wrote:
> > On Fri, 10 Apr 2015 09:10:35 +1000 Dave Chinner <david@fromorbit.com> wrote:
> >
> > > On Fri, Apr 10, 2015 at 08:53:22AM +1000, Dave Chinner wrote:
> > > > On Thu, Apr 09, 2015 at 06:20:26PM -0400, Joe Landman wrote:
> > > > >
> > > > >
> > > > > On 04/09/2015 06:18 PM, Dave Chinner wrote:
> > > > > >On Thu, Apr 09, 2015 at 05:02:33PM -0400, Joe Landman wrote:
> > > > > >>If I build an MD raid0 with a non power of 2 chunk size, it appears
> > > > > >>that I can mkfs.xfs a file system, but it doesn't show up in blkid
> > > > > >>and is not mountable. Yet, using a power of 2 chunk size, this does
> > > > > >>work correctly. This is kernel 3.18.9.
> > > > > >>
> > > > >
> > > > > [...]
> > > > >
> > > > > >That looks more like a blkid or udev problem. try using blkid -p so
> > > > > >that it doesn't look up the cache but directly probes devices for
> > > > > >the signatures. strace might tell you a bit more, too. And if the
> > > > > >filesystem mounts, then it definitely isn't an XFS problem ;)
> > > > >
> > > > > Thats the thing, it didn't mount, even when I used the device name
> > > > > directly.
> > > >
> > > > Ok, that's interesting. Let me see if I can reproduce it locally. If
> > > > you don't hear otherwise, tracing would still be useful. Thanks for
> > > > the bug report, Joe.
> > >
> > > No luck - md doesn't allow the device to be activated on 4.0-rc7:
> > >
> > > $ sudo mdadm --version
> > > mdadm - v3.3.2 - 21st August 2014
> > > $ uname -a
> > > Linux test4 4.0.0-rc7-dgc+ #882 SMP Fri Apr 10 08:50:52 AEST 2015 x86_64 GNU/Linux
> > > $ sudo wipefs -a /dev/vd[ab]
> > > /dev/vda: 4 bytes were erased at offset 0x00001000 (linux_raid_member): fc 4e 2b a9
> > > /dev/vdb: 4 bytes were erased at offset 0x00001000 (linux_raid_member): fc 4e 2b a9
> > > $ sudo mdadm --create /dev/md20 --level=0 --metadata=1.2 --chunk=1152 --auto=yes --raid-disks=2 /dev/vd[ab]
> >
> > Weird. Works for me.
> > Any messages in 'dmesg' ??
> > How big are /dev/vd[ab]??
>
> vda is 5GB, vdb is 20GB
>
> dmesg:
>
> [ 125.131340] md: bind<vda>
> [ 125.134547] md: bind<vdb>
> [ 125.139669] md: personality for level 0 is not loaded!
> [ 125.141302] md: md20 stopped.
> [ 125.141986] md: unbind<vdb>
> [ 125.160100] md: export_rdev(vdb)
> [ 125.161751] md: unbind<vda>
> [ 125.180126] md: export_rdev(vda)
>
> Oh, curious. Going from 4.0-rc4 to 4.0-rc7, and make oldconfig
> has resulted in:
>
> # CONFIG_MD_RAID0 is not set
>
> Ok, so with that fixed, it's still horribly broken.
>
> RAID 0 on different sized devices should result in a device that is
> twice the size of the smallest devices:
>
> $ sudo mdadm --create /dev/md20 --level=raid0 --metadata=1.2 --chunk=1024 --auto=yes --raid-disks=2 /dev/vd[ab]
> mdadm: array /dev/md20 started.
> $ cat /proc/mdstat
> Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
> md20 : active raid0 vdb[1] vda[0]
> 26206208 blocks super 1.2 1024k chunks
>
> unused devices: <none>
> $ grep "md\|vd[ab]" /proc/partitions
> 253 0 5242880 vda
> 253 16 20971520 vdb
> 9 20 26206208 md20
> $
>
> Oh, "RAID0" is not actually RAID 0 - that's the size I'd expect from
> a linear mapping. Half way through writing that block device, the IO
> stats change in an obvious way:
>
> Device: r/s w/s rMB/s wMB/s
> vda 0.00 144.00 0.00 48.00
> vdb 0.00 145.20 0.00 48.40
> md20 0.00 290.40 0.00 96.80
>
> Device: r/s w/s rMB/s wMB/s
> vda 0.00 56.40 0.00 18.80
> vdb 0.00 229.20 0.00 76.40
> md20 0.00 285.20 0.00 95.10
>
> Device: r/s w/s rMB/s wMB/s
> vda 0.00 0.00 0.00 0.00
> vdb 0.00 290.40 0.00 96.80
> md20 0.00 290.80 0.00 96.90
>
> So it's actually a stripe for the first 10GB, then some kind of
> concatenated mapping of the remainder of the single device. That's
> not what I expected, but it's also clearly not the problem.
>
> Anyway, change the stripe size to 1152:
>
> sudo mdadm --stop /dev/md20
> mdadm: stopped /dev/md20
> $ sudo wipefs -a /dev/vd[ab]
> /dev/vda: 4 bytes were erased at offset 0x00001000 (linux_raid_member): fc 4e 2b a9
> /dev/vdb: 4 bytes were erased at offset 0x00001000 (linux_raid_member): fc 4e 2b a9
> $ sudo mdadm --create /dev/md20 --level=raid0 --metadata=1.2 --chunk=1152 --auto=yes --raid-disks=2 /dev/vd[ab]
> mdadm: array /dev/md20 started.
> $ sudo xfs_io -fd -c "pwrite -b 4m 0 25g" /dev/md20
> wrote 26831355904/26843545600 bytes at offset 0
> 24.989 GiB, 6398 ops; 0:00:16.00 (1.530 GiB/sec and 391.8556 ops/sec)
> $
>
> Wait, what? Neil, did you put a flux capacitor in MD? :P
>
> The underlying drive is only capable of 100MB/s - 25GB of sequential
> direct IO does not complete in 16 seconds on such a drive. But
> there's also a 1GB BBWC in front of the physical drives (HW RAID1),
> but even so, this write rate could only occur if every write is
> hitting the BBWC. And so it is:
>
> $ sudo xfs_io -fd -c "pwrite -b 4m 0 25g" /dev/md20 & iostat -d -m 1
> ...
> Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
> vda 4214.00 0.00 1516.99 0 1516
> vdb 0.00 0.00 0.00 0 0
> md20 4223.00 0.00 1520.00 0 1520
>
> Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
> vda 2986.00 0.00 1075.01 0 1075
> vdb 1174.00 0.00 422.88 0 422
> md20 4154.00 0.00 1496.00 0 1496
>
> Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
> vda 0.00 0.00 0.00 0 0
> vdb 4376.00 0.00 1575.12 0 1575
> md20 4378.00 0.00 1576.00 0 1576
>
> Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
> vda 2682.00 0.00 965.74 0 965
> vdb 1650.00 0.00 594.00 0 594
> md20 4334.00 0.00 1560.00 0 1560
>
> Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
> vda 4518.00 0.00 1626.26 0 1626
> vdb 138.00 0.00 49.50 0 49
> md20 4656.00 0.00 1676.00 0 1676
>
> Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
> vda 0.00 0.00 0.00 0 0
> vdb 4214.00 0.00 1517.48 0 1517
> md20 4210.00 0.00 1516.00 0 1516
> .....
>
> Note how it is cycling from one drive to the other with about a 2s
> period?
>
> Yup, blocktrace on /dev/vda shows it is, indeed, hitting the BBWC
> because the block mapping is clearly broken:
>
> 253,0 4 1 0.000000000 6972 Q WS 8192 + 1008 [xfs_io]
> 253,0 4 5 0.000068012 6972 Q WS 8192 + 1008 [xfs_io]
> 253,0 4 9 0.000093266 6972 Q WS 8192 + 288 [xfs_io]
> 253,0 4 13 0.000129722 6972 Q WS 8193 + 1008 [xfs_io]
> 253,0 4 17 0.000176872 6972 Q WS 8193 + 1008 [xfs_io]
> 253,0 4 21 0.000205566 6972 Q WS 8193 + 288 [xfs_io]
> 253,0 4 25 0.000240846 6972 Q WS 8194 + 1008 [xfs_io]
> 253,0 4 29 0.000284990 6972 Q WS 8194 + 1008 [xfs_io]
> 253,0 4 33 0.000313276 6972 Q WS 8194 + 288 [xfs_io]
> 253,0 4 37 0.000352330 6972 Q WS 8195 + 1008 [xfs_io]
> 253,0 4 41 0.000374272 6972 Q WS 8195 + 272 [xfs_io]
> 253,0 4 56 0.001215857 6972 Q WS 8195 + 1008 [xfs_io]
> 253,0 4 60 0.001252697 6972 Q WS 8195 + 16 [xfs_io]
> 253,0 4 64 0.001284517 6972 Q WS 8196 + 1008 [xfs_io]
> 253,0 4 68 0.001326130 6972 Q WS 8196 + 1008 [xfs_io]
> 253,0 4 72 0.001355050 6972 Q WS 8196 + 288 [xfs_io]
> 253,0 4 76 0.001393777 6972 Q WS 8197 + 1008 [xfs_io]
> 253,0 4 80 0.001439547 6972 Q WS 8197 + 1008 [xfs_io]
> 253,0 4 84 0.001466097 6972 Q WS 8197 + 288 [xfs_io]
> 253,0 4 88 0.001501267 6972 Q WS 8198 + 1008 [xfs_io]
> 253,0 4 92 0.001545863 6972 Q WS 8198 + 1008 [xfs_io]
> 253,0 4 96 0.001571500 6972 Q WS 8198 + 288 [xfs_io]
> 253,0 4 100 0.001584620 6972 Q WS 8199 + 256 [xfs_io]
> 253,0 4 116 0.002730034 6972 Q WS 8199 + 1008 [xfs_io]
> 253,0 4 120 0.002792351 6972 Q WS 8199 + 1008 [xfs_io]
> 253,0 4 124 0.002810937 6972 Q WS 8199 + 32 [xfs_io]
> 253,0 4 128 0.002842047 6972 Q WS 8200 + 1008 [xfs_io]
> 253,0 4 132 0.002889087 6972 Q WS 8200 + 1008 [xfs_io]
> 253,0 4 136 0.002916894 6972 Q WS 8200 + 288 [xfs_io]
> 253,0 4 140 0.002952334 6972 Q WS 8201 + 1008 [xfs_io]
> 253,0 4 144 0.002996101 6972 Q WS 8201 + 1008 [xfs_io]
> 253,0 4 148 0.003022401 6972 Q WS 8201 + 288 [xfs_io]
>
>
> Multiple IOs to teh same sector, then the sector increments by 1 and
> we get more IOs to the same sector offset. After about a second the
> mapping shifts IO to the other block device as it slowly increments
> the sector, and that's why we see that cycling behaviour.
>
> IOWs, something is going wrong with the MD block mapping when the
> RAID chunk size is not a power of 2....
>
> Over to you, Neil....
That's .... not good. Not good at all.
This should help. It seems that non-power-of-2 chunksizes aren't widely used.
Thanks,
NeilBrown
From: NeilBrown <neilb@suse.de>
Date: Fri, 10 Apr 2015 13:19:04 +1000
Subject: [PATCH] md/raid0: fix bug with chunksize not a power of 2.
Since commit 20d0189b1012a37d2533a87fb451f7852f2418d1
in v3.14-rc1 RAID0 has performed incorrect calculations
when the chunksize is not a power of 2.
This happens because "sector_div()" modifies its first argument, but
this wasn't taken into account in the patch.
So restore that first arg before re-using the variable.
Reported-by: Joe Landman <joe.landman@gmail.com>
Reported-by: Dave Chinner <david@fromorbit.com>
Fixes: 20d0189b1012a37d2533a87fb451f7852f2418d1
Cc: stable@vger.kernel.org (3.14 and later).
Signed-off-by: NeilBrown <neilb@suse.de>
diff --git a/drivers/md/raid0.c b/drivers/md/raid0.c
index e074813da6c0..2cb59a641cd2 100644
--- a/drivers/md/raid0.c
+++ b/drivers/md/raid0.c
@@ -315,7 +315,7 @@ static struct strip_zone *find_zone(struct r0conf *conf,
/*
* remaps the bio to the target device. we separate two flows.
- * power 2 flow and a general flow for the sake of perfromance
+ * power 2 flow and a general flow for the sake of performance
*/
static struct md_rdev *map_sector(struct mddev *mddev, struct strip_zone *zone,
sector_t sector, sector_t *sector_offset)
@@ -530,6 +530,7 @@ static void raid0_make_request(struct mddev *mddev, struct bio *bio)
split = bio;
}
+ sector = bio->bi_iter.bi_sector;
zone = find_zone(mddev->private, §or);
tmp_dev = map_sector(mddev, zone, sector, §or);
split->bi_bdev = tmp_dev->bdev;
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply related
* Re: [PATCH 4/4] md-cluster: re-add
From: Goldwyn Rodrigues @ 2015-04-10 3:49 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid, GQJiang
In-Reply-To: <20150409095501.536f6216@notabene.brown>
On 04/08/2015 06:55 PM, NeilBrown wrote:
> On Wed, 8 Apr 2015 14:24:14 -0500 Goldwyn Rodrigues <rgoldwyn@suse.de> wrote:
>
>> This extends the capabilites of re-adding a failed device
>> to the clustering environment.
>>
>> A new function gather_bitmaps gathers set bits from bitmaps of
>> all nodes, sends a message to all nodes to readd the disk
>> and then initiates the recovery process.
>>
>> Question: Do you see a race in sending a READD and then performing
>> the bitmap resync/recovery? Should the initiating node perform the
>> recovery before sending the READD message? The recovery will send a
>> METADATA_UPDATE anyways.
>
> The RE-ADD has to happen *before* the bitmaps are gathered.
> After the RE-ADD, all writes will go to the new device.
> Any write before that RE-ADD will be recorded in the bitmap.
> To ensure that the recovery handles all regions affected by writes, it needs
> to know about all writes that didn't go to the new device. So it needs to
> collect bitmaps only once new writes have started going to the new device.
>
> Is that clear? If not, I'll try again.
>
Yes, I understood your point. Performing the re-add later would miss on
the ones between the recovery and the re-add.
--
Goldwyn
^ permalink raw reply
* Re: interesting MD-xfs bug
From: Roman Mamedov @ 2015-04-10 4:43 UTC (permalink / raw)
To: Dave Chinner; +Cc: NeilBrown, Joe Landman, linux-raid, xfs
In-Reply-To: <20150410013156.GH15810@dastard>
[-- Attachment #1: Type: text/plain, Size: 937 bytes --]
On Fri, 10 Apr 2015 11:31:57 +1000
Dave Chinner <david@fromorbit.com> wrote:
> RAID 0 on different sized devices should result in a device that is
> twice the size of the smallest devices
> Oh, "RAID0" is not actually RAID 0 - that's the size I'd expect from
> a linear mapping.
> it's actually a stripe for the first 10GB, then some kind of
> concatenated mapping of the remainder of the single device.
It might be not what you expected, but it's also not a bug of any kind, just
the regular behavior of mdadm RAID0 with different sized devices (man md):
If devices in the array are not all the same size, then once the small‐
est device has been exhausted, the RAID0 driver starts collecting
chunks into smaller stripes that only span the drives which still have
remaining space.
Once or twice this came VERY handy for me in real life usage.
--
With respect,
Roman
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply
* Re: interesting MD-xfs bug
From: Dave Chinner @ 2015-04-10 6:05 UTC (permalink / raw)
To: NeilBrown; +Cc: Joe Landman, linux-raid, xfs
In-Reply-To: <20150410132253.644e3660@notabene.brown>
On Fri, Apr 10, 2015 at 01:22:53PM +1000, NeilBrown wrote:
> On Fri, 10 Apr 2015 11:31:57 +1000 Dave Chinner <david@fromorbit.com> wrote:
>
> > On Fri, Apr 10, 2015 at 09:36:52AM +1000, NeilBrown wrote:
> > > On Fri, 10 Apr 2015 09:10:35 +1000 Dave Chinner <david@fromorbit.com> wrote:
> > >
> > > > On Fri, Apr 10, 2015 at 08:53:22AM +1000, Dave Chinner wrote:
> > > > > On Thu, Apr 09, 2015 at 06:20:26PM -0400, Joe Landman wrote:
> > > > > >
> > > > > >
> > > > > > On 04/09/2015 06:18 PM, Dave Chinner wrote:
> > > > > > >On Thu, Apr 09, 2015 at 05:02:33PM -0400, Joe Landman wrote:
> > > > > > >>If I build an MD raid0 with a non power of 2 chunk size, it appears
> > > > > > >>that I can mkfs.xfs a file system, but it doesn't show up in blkid
> > > > > > >>and is not mountable. Yet, using a power of 2 chunk size, this does
> > > > > > >>work correctly. This is kernel 3.18.9.
> > > > > > >>
> > > > > >
> > > > > > [...]
> > > > > >
> > > > > > >That looks more like a blkid or udev problem. try using blkid -p so
> > > > > > >that it doesn't look up the cache but directly probes devices for
> > > > > > >the signatures. strace might tell you a bit more, too. And if the
> > > > > > >filesystem mounts, then it definitely isn't an XFS problem ;)
> > > > > >
> > > > > > Thats the thing, it didn't mount, even when I used the device name
> > > > > > directly.
> > > > >
> > > > > Ok, that's interesting. Let me see if I can reproduce it locally. If
> > > > > you don't hear otherwise, tracing would still be useful. Thanks for
> > > > > the bug report, Joe.
> > > >
> > > > No luck - md doesn't allow the device to be activated on 4.0-rc7:
> > > >
> > > > $ sudo mdadm --version
> > > > mdadm - v3.3.2 - 21st August 2014
> > > > $ uname -a
> > > > Linux test4 4.0.0-rc7-dgc+ #882 SMP Fri Apr 10 08:50:52 AEST 2015 x86_64 GNU/Linux
> > > > $ sudo wipefs -a /dev/vd[ab]
> > > > /dev/vda: 4 bytes were erased at offset 0x00001000 (linux_raid_member): fc 4e 2b a9
> > > > /dev/vdb: 4 bytes were erased at offset 0x00001000 (linux_raid_member): fc 4e 2b a9
> > > > $ sudo mdadm --create /dev/md20 --level=0 --metadata=1.2 --chunk=1152 --auto=yes --raid-disks=2 /dev/vd[ab]
> > >
> > > Weird. Works for me.
> > > Any messages in 'dmesg' ??
> > > How big are /dev/vd[ab]??
> >
> > vda is 5GB, vdb is 20GB
> >
> > dmesg:
> >
> > [ 125.131340] md: bind<vda>
> > [ 125.134547] md: bind<vdb>
> > [ 125.139669] md: personality for level 0 is not loaded!
> > [ 125.141302] md: md20 stopped.
> > [ 125.141986] md: unbind<vdb>
> > [ 125.160100] md: export_rdev(vdb)
> > [ 125.161751] md: unbind<vda>
> > [ 125.180126] md: export_rdev(vda)
> >
> > Oh, curious. Going from 4.0-rc4 to 4.0-rc7, and make oldconfig
> > has resulted in:
> >
> > # CONFIG_MD_RAID0 is not set
> >
> > Ok, so with that fixed, it's still horribly broken.
> >
> > RAID 0 on different sized devices should result in a device that is
> > twice the size of the smallest devices:
> >
> > $ sudo mdadm --create /dev/md20 --level=raid0 --metadata=1.2 --chunk=1024 --auto=yes --raid-disks=2 /dev/vd[ab]
> > mdadm: array /dev/md20 started.
> > $ cat /proc/mdstat
> > Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
> > md20 : active raid0 vdb[1] vda[0]
> > 26206208 blocks super 1.2 1024k chunks
> >
> > unused devices: <none>
> > $ grep "md\|vd[ab]" /proc/partitions
> > 253 0 5242880 vda
> > 253 16 20971520 vdb
> > 9 20 26206208 md20
> > $
> >
> > Oh, "RAID0" is not actually RAID 0 - that's the size I'd expect from
> > a linear mapping. Half way through writing that block device, the IO
> > stats change in an obvious way:
> >
> > Device: r/s w/s rMB/s wMB/s
> > vda 0.00 144.00 0.00 48.00
> > vdb 0.00 145.20 0.00 48.40
> > md20 0.00 290.40 0.00 96.80
> >
> > Device: r/s w/s rMB/s wMB/s
> > vda 0.00 56.40 0.00 18.80
> > vdb 0.00 229.20 0.00 76.40
> > md20 0.00 285.20 0.00 95.10
> >
> > Device: r/s w/s rMB/s wMB/s
> > vda 0.00 0.00 0.00 0.00
> > vdb 0.00 290.40 0.00 96.80
> > md20 0.00 290.80 0.00 96.90
> >
> > So it's actually a stripe for the first 10GB, then some kind of
> > concatenated mapping of the remainder of the single device. That's
> > not what I expected, but it's also clearly not the problem.
> >
> > Anyway, change the stripe size to 1152:
> >
> > sudo mdadm --stop /dev/md20
> > mdadm: stopped /dev/md20
> > $ sudo wipefs -a /dev/vd[ab]
> > /dev/vda: 4 bytes were erased at offset 0x00001000 (linux_raid_member): fc 4e 2b a9
> > /dev/vdb: 4 bytes were erased at offset 0x00001000 (linux_raid_member): fc 4e 2b a9
> > $ sudo mdadm --create /dev/md20 --level=raid0 --metadata=1.2 --chunk=1152 --auto=yes --raid-disks=2 /dev/vd[ab]
> > mdadm: array /dev/md20 started.
> > $ sudo xfs_io -fd -c "pwrite -b 4m 0 25g" /dev/md20
> > wrote 26831355904/26843545600 bytes at offset 0
> > 24.989 GiB, 6398 ops; 0:00:16.00 (1.530 GiB/sec and 391.8556 ops/sec)
> > $
> >
> > Wait, what? Neil, did you put a flux capacitor in MD? :P
> >
> > The underlying drive is only capable of 100MB/s - 25GB of sequential
> > direct IO does not complete in 16 seconds on such a drive. But
> > there's also a 1GB BBWC in front of the physical drives (HW RAID1),
> > but even so, this write rate could only occur if every write is
> > hitting the BBWC. And so it is:
> >
> > $ sudo xfs_io -fd -c "pwrite -b 4m 0 25g" /dev/md20 & iostat -d -m 1
> > ...
> > Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
> > vda 4214.00 0.00 1516.99 0 1516
> > vdb 0.00 0.00 0.00 0 0
> > md20 4223.00 0.00 1520.00 0 1520
> >
> > Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
> > vda 2986.00 0.00 1075.01 0 1075
> > vdb 1174.00 0.00 422.88 0 422
> > md20 4154.00 0.00 1496.00 0 1496
> >
> > Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
> > vda 0.00 0.00 0.00 0 0
> > vdb 4376.00 0.00 1575.12 0 1575
> > md20 4378.00 0.00 1576.00 0 1576
> >
> > Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
> > vda 2682.00 0.00 965.74 0 965
> > vdb 1650.00 0.00 594.00 0 594
> > md20 4334.00 0.00 1560.00 0 1560
> >
> > Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
> > vda 4518.00 0.00 1626.26 0 1626
> > vdb 138.00 0.00 49.50 0 49
> > md20 4656.00 0.00 1676.00 0 1676
> >
> > Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
> > vda 0.00 0.00 0.00 0 0
> > vdb 4214.00 0.00 1517.48 0 1517
> > md20 4210.00 0.00 1516.00 0 1516
> > .....
> >
> > Note how it is cycling from one drive to the other with about a 2s
> > period?
> >
> > Yup, blocktrace on /dev/vda shows it is, indeed, hitting the BBWC
> > because the block mapping is clearly broken:
> >
> > 253,0 4 1 0.000000000 6972 Q WS 8192 + 1008 [xfs_io]
> > 253,0 4 5 0.000068012 6972 Q WS 8192 + 1008 [xfs_io]
> > 253,0 4 9 0.000093266 6972 Q WS 8192 + 288 [xfs_io]
> > 253,0 4 13 0.000129722 6972 Q WS 8193 + 1008 [xfs_io]
> > 253,0 4 17 0.000176872 6972 Q WS 8193 + 1008 [xfs_io]
> > 253,0 4 21 0.000205566 6972 Q WS 8193 + 288 [xfs_io]
> > 253,0 4 25 0.000240846 6972 Q WS 8194 + 1008 [xfs_io]
> > 253,0 4 29 0.000284990 6972 Q WS 8194 + 1008 [xfs_io]
> > 253,0 4 33 0.000313276 6972 Q WS 8194 + 288 [xfs_io]
> > 253,0 4 37 0.000352330 6972 Q WS 8195 + 1008 [xfs_io]
> > 253,0 4 41 0.000374272 6972 Q WS 8195 + 272 [xfs_io]
> > 253,0 4 56 0.001215857 6972 Q WS 8195 + 1008 [xfs_io]
> > 253,0 4 60 0.001252697 6972 Q WS 8195 + 16 [xfs_io]
> > 253,0 4 64 0.001284517 6972 Q WS 8196 + 1008 [xfs_io]
> > 253,0 4 68 0.001326130 6972 Q WS 8196 + 1008 [xfs_io]
> > 253,0 4 72 0.001355050 6972 Q WS 8196 + 288 [xfs_io]
> > 253,0 4 76 0.001393777 6972 Q WS 8197 + 1008 [xfs_io]
> > 253,0 4 80 0.001439547 6972 Q WS 8197 + 1008 [xfs_io]
> > 253,0 4 84 0.001466097 6972 Q WS 8197 + 288 [xfs_io]
> > 253,0 4 88 0.001501267 6972 Q WS 8198 + 1008 [xfs_io]
> > 253,0 4 92 0.001545863 6972 Q WS 8198 + 1008 [xfs_io]
> > 253,0 4 96 0.001571500 6972 Q WS 8198 + 288 [xfs_io]
> > 253,0 4 100 0.001584620 6972 Q WS 8199 + 256 [xfs_io]
> > 253,0 4 116 0.002730034 6972 Q WS 8199 + 1008 [xfs_io]
> > 253,0 4 120 0.002792351 6972 Q WS 8199 + 1008 [xfs_io]
> > 253,0 4 124 0.002810937 6972 Q WS 8199 + 32 [xfs_io]
> > 253,0 4 128 0.002842047 6972 Q WS 8200 + 1008 [xfs_io]
> > 253,0 4 132 0.002889087 6972 Q WS 8200 + 1008 [xfs_io]
> > 253,0 4 136 0.002916894 6972 Q WS 8200 + 288 [xfs_io]
> > 253,0 4 140 0.002952334 6972 Q WS 8201 + 1008 [xfs_io]
> > 253,0 4 144 0.002996101 6972 Q WS 8201 + 1008 [xfs_io]
> > 253,0 4 148 0.003022401 6972 Q WS 8201 + 288 [xfs_io]
> >
> >
> > Multiple IOs to teh same sector, then the sector increments by 1 and
> > we get more IOs to the same sector offset. After about a second the
> > mapping shifts IO to the other block device as it slowly increments
> > the sector, and that's why we see that cycling behaviour.
> >
> > IOWs, something is going wrong with the MD block mapping when the
> > RAID chunk size is not a power of 2....
> >
> > Over to you, Neil....
>
> That's .... not good. Not good at all.
>
> This should help. It seems that non-power-of-2 chunksizes aren't widely used.
I haven't tested the patch, but if you want to make sure that you
get regular smoke testing on this sort of config, write a simple
test for xfstests and throw it in the generic group. e.g. create
multiple loop devices, then iterate over various MD configurations
running a basic data integrity tests on them. e.g. mkfs, mount,
write a 20MB pattened file, fsync, unmount, mount, md5sum it,
unlink, unmount, check filesystem.
Something like that will get run all the time by FS developers and
QE departments, so it's a good way of smoke testing configurations
that don't usually get tested without even having to think about
it...
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply
* Re.
From: Mrs. Zhang Xiao @ 2015-04-10 7:24 UTC (permalink / raw)
To: Recipients
Dear,
I seek for your sincerity and trust in a deal which involved a total sum of 60,000,000.00 United states dollars. I need to know you and know your location. This really matter to the success of this deal.If interested respond to me for more details.
Yours Sincerely.
Mrs. Zhang Xiao (Accounts book Keeper)
Angang Steel Company Limited
396 Nan Zhong Hua Lu, Tie Dong District Anshan, Liaoning 114021, China.
---
This email is free from viruses and malware because avast! Antivirus protection is active.
http://www.avast.com
^ permalink raw reply
* [PATCH] md-cluster: correct the num for comparison
From: gqjiang @ 2015-04-10 8:06 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, rgoldwyn, Guoqing Jiang
From: Guoqing Jiang <gqjiang@suse.com>
Since the node num of md-cluster is from zero, and
cinfo->slot_number represents the slot num of dlm,
here also need to deduct one to keep consistency
with other codes.
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
drivers/md/md-cluster.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index ae8bb54..531cc2b 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -612,7 +612,7 @@ static int join(struct mddev *mddev, int nodes)
if (ret)
goto err;
wait_for_completion(&cinfo->completion);
- if (nodes <= cinfo->slot_number) {
+ if (nodes <= cinfo->slot_number - 1) {
pr_err("md-cluster: Slot allotted(%d) greater than available slots(%d)", cinfo->slot_number - 1,
nodes);
ret = -ERANGE;
--
1.7.12.4
^ permalink raw reply related
* Re: [PATCH 4/4] md-cluster: re-add
From: Guoqing Jiang @ 2015-04-10 9:43 UTC (permalink / raw)
To: Goldwyn Rodrigues; +Cc: neilb, linux-raid
In-Reply-To: <20150408192414.GA9693@shrek.lan>
Hi Goldwyn,
> diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
> index d036c83..afffbee 100644
> --- a/drivers/md/md-cluster.c
> +++ b/drivers/md/md-cluster.c
> @@ -50,6 +50,7 @@ struct md_cluster_info {
> /* dlm lock space and resources for clustered raid. */
> dlm_lockspace_t *lockspace;
> int slot_number;
> + int total_slots;
>
There are two "nodes" in bitmap_info and bitmap_super_t, is it possible
to use
the existed nodes? Instead of introduce a new member in md_cluster_info.
drivers/md/bitmap.c:440: sb->nodes =
cpu_to_le32(bitmap->mddev->bitmap_info.nodes);
drivers/md/bitmap.c:552: int nodes = 0;
drivers/md/bitmap.c:610: nodes = le32_to_cpu(sb->nodes);
drivers/md/bitmap.c:688: bitmap->mddev->bitmap_info.nodes = nodes;
> @@ -653,6 +671,7 @@ static int join(struct mddev *mddev, int nodes)
> ret = -ERANGE;
> goto err;
> }
> + cinfo->total_slots = nodes;
>
The nodes is got from "nodes = le32_to_cpu(sb->nodes);", right?
> cinfo->sb_lock = lockres_init(mddev, "cmd-super",
> NULL, 0);
> if (!cinfo->sb_lock) {
> @@ -900,6 +919,34 @@ static int remove_disk(struct mddev *mddev, struct md_rdev *rdev)
> return __sendmsg(cinfo, &cmsg);
> }
>
> +static int gather_bitmaps(struct md_rdev *rdev)
> +{
> + int sn, err;
> + sector_t lo, hi;
> + struct cluster_msg cmsg;
> + struct mddev *mddev = rdev->mddev;
> + struct md_cluster_info *cinfo = mddev->cluster_info;
> + struct mdp_superblock_1 *sb = page_address(rdev->sb_page);
> + char *uuid = sb->device_uuid;
> +
> + for (sn = 0; sn < cinfo->total_slots; sn++) {
>
The last place which used total_slots.
Thanks,
Guoqing
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox