* error injection @ 2011-09-28 19:48 Jojy Varghese 2011-09-28 23:11 ` NeilBrown 0 siblings, 1 reply; 6+ messages in thread From: Jojy Varghese @ 2011-09-28 19:48 UTC (permalink / raw) To: linux-raid Hi I am trying to dynamically add error injection to my virtual disk(LVM) for testing+ debugging purpose. I saw "faulty" personality module in the kernel and was wondering if there was any documentation on its usage. I am not looking to set up a RAID but a simple mapped device. So the basic use case is that I need to be able to dynamically add/remove error sectors and also be able to have granular error configuration like read error, read+write error etc. thanks in advance Jojy ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: error injection 2011-09-28 19:48 error injection Jojy Varghese @ 2011-09-28 23:11 ` NeilBrown 2011-09-29 0:59 ` Jojy Varghese 0 siblings, 1 reply; 6+ messages in thread From: NeilBrown @ 2011-09-28 23:11 UTC (permalink / raw) To: Jojy Varghese; +Cc: linux-raid [-- Attachment #1: Type: text/plain, Size: 4287 bytes --] On Wed, 28 Sep 2011 12:48:37 -0700 Jojy Varghese <jojy.varghese@gmail.com> wrote: > Hi > I am trying to dynamically add error injection to my virtual > disk(LVM) for testing+ debugging purpose. I saw "faulty" personality > module in the kernel and was wondering if there was any documentation > on its usage. I am not looking to set up a RAID but a simple mapped > device. So the basic use case is that I need to be able to dynamically > add/remove error sectors and also be able to have granular error > configuration like read error, read+write error etc. > > thanks in advance > Jojy > -- > To unsubscribe from this list: send the line "unsubscribe linux-raid" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html The 'faulty' md personality is described briefly in the 'md.4' man page which is included in the mdadm distribution. I've included the relevant part below. Configuring the type of faults is described in mdadm.8 under the '-p --layout=' section. So can adjust the settings using mdadm --grow. so: mdadm -B /dev/md0 -l faulty -n1 /dev/sda will build a 'faulty' device which provides access to /dev/sda, but introduces faults. Initially no faults will be introduces. mdadm -G /dev/md0 --layout=rt400 will tell md0 to generate a read error every 400 requests, but not to remember the error - rt == readtransient --layout=rp400 will create a persistent error every 400 reads subsequent reads of the same block will produce the same error. at most 50 persistent errors can be recorded. mdadm -G /dev/md0 --layout=clear will stop producing new errors mdadm -G /dev/md0 --layout=flush will forget all persistent errors. from md.4: FAULTY The FAULTY md module is provided for testing purposes. A faulty array has exactly one component device and is normally assembled without a superblock, so the md array created provides direct access to all of the data in the component device. The FAULTY module may be requested to simulate faults to allow testing of other md levels or of filesystems. Faults can be chosen to trigger on read requests or write requests, and can be transient (a subsequent read/write at the address will probably succeed) or persistent (subse- quent read/write of the same address will fail). Further, read faults can be "fixable" meaning that they persist until a write request at the same address. Fault types can be requested with a period. In this case, the fault will recur repeatedly after the given number of requests of the rele- vant type. For example if persistent read faults have a period of 100, then every 100th read request would generate a fault, and the faulty sector would be recorded so that subsequent reads on that sector would also fail. There is a limit to the number of faulty sectors that are remembered. Faults generated after this limit is exhausted are treated as tran- sient. The list of faulty sectors can be flushed, and the active list of fail- ure modes can be cleared. from mdadm.8: When setting the failure mode for level faulty, the options are: write-transient, wt, read-transient, rt, write-persistent, wp, read-persistent, rp, write-all, read-fixable, rf, clear, flush, none. Each failure mode can be followed by a number, which is used as a period between fault generation. Without a number, the fault is generated once on the first relevant request. With a number, the fault will be generated after that many requests, and will continue to be generated every time the period elapses. Multiple failure modes can be current simultaneously by using the --grow option to set subsequent failure modes. "clear" or "none" will remove any pending or periodic failure modes, and "flush" will clear any persistent faults. NeilBrown [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 190 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: error injection 2011-09-28 23:11 ` NeilBrown @ 2011-09-29 0:59 ` Jojy Varghese 2011-09-29 1:08 ` NeilBrown 0 siblings, 1 reply; 6+ messages in thread From: Jojy Varghese @ 2011-09-29 0:59 UTC (permalink / raw) To: NeilBrown; +Cc: linux-raid Thanks Neil. I tried setting my sda7 partition to generate write errors every 40 bytes(writing 1 byte at a time). I did : 1. Create a array with: mdadm -C /dev/md/me0 -l faulty -n1 /dev/sda7 After this step I can see /dev/md127 and when i do a mdadm -D /dev/md127, i get: /dev/md127: Version : 1.2 Creation Time : Wed Sep 28 17:35:50 2011 Raid Level : faulty Array Size : 969410424 (924.50 GiB 992.68 GB) Raid Devices : 1 Total Devices : 1 Persistence : Superblock is persistent Update Time : Wed Sep 28 17:35:50 2011 State : clean Active Devices : 1 Working Devices : 1 Failed Devices : 0 Spare Devices : 0 Name : eng-dev16.lab.local:me0 (local to host eng-dev16.lab.local) UUID : 96f4be10:312f9574:f40107aa:d9f278ba Events : 0 Number Major Minor RaidDevice State 0 8 7 0 active sync /dev/sda7 2. Set write fault level with: mdadm -G /dev/md/me0 --layout=wp40 After this when i write > 40 bytes into /dev/md127, i dont get any I/O errors. I am sure i am doing something wrong here. Any help is much appreciated. Thanks Jojy On Wed, Sep 28, 2011 at 4:11 PM, NeilBrown <neilb@suse.de> wrote: > On Wed, 28 Sep 2011 12:48:37 -0700 Jojy Varghese <jojy.varghese@gmail.com> > wrote: > >> Hi >> I am trying to dynamically add error injection to my virtual >> disk(LVM) for testing+ debugging purpose. I saw "faulty" personality >> module in the kernel and was wondering if there was any documentation >> on its usage. I am not looking to set up a RAID but a simple mapped >> device. So the basic use case is that I need to be able to dynamically >> add/remove error sectors and also be able to have granular error >> configuration like read error, read+write error etc. >> >> thanks in advance >> Jojy >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > The 'faulty' md personality is described briefly in the 'md.4' man page which > is included in the mdadm distribution. > I've included the relevant part below. > > Configuring the type of faults is described in mdadm.8 under the '-p > --layout=' section. So can adjust the settings using mdadm --grow. > so: > mdadm -B /dev/md0 -l faulty -n1 /dev/sda > > will build a 'faulty' device which provides access to /dev/sda, but > introduces faults. Initially no faults will be introduces. > > mdadm -G /dev/md0 --layout=rt400 > > will tell md0 to generate a read error every 400 requests, but not to > remember the error - rt == readtransient > --layout=rp400 > will create a persistent error every 400 reads subsequent reads of the same > block will produce the same error. at most 50 persistent errors can be > recorded. > mdadm -G /dev/md0 --layout=clear > will stop producing new errors > mdadm -G /dev/md0 --layout=flush > will forget all persistent errors. > > > from md.4: > > FAULTY > The FAULTY md module is provided for testing purposes. A faulty array > has exactly one component device and is normally assembled without a > superblock, so the md array created provides direct access to all of > the data in the component device. > > The FAULTY module may be requested to simulate faults to allow testing > of other md levels or of filesystems. Faults can be chosen to trigger > on read requests or write requests, and can be transient (a subsequent > read/write at the address will probably succeed) or persistent (subse- > quent read/write of the same address will fail). Further, read faults > can be "fixable" meaning that they persist until a write request at the > same address. > > Fault types can be requested with a period. In this case, the fault > will recur repeatedly after the given number of requests of the rele- > vant type. For example if persistent read faults have a period of 100, > then every 100th read request would generate a fault, and the faulty > sector would be recorded so that subsequent reads on that sector would > also fail. > > There is a limit to the number of faulty sectors that are remembered. > Faults generated after this limit is exhausted are treated as tran- > sient. > > The list of faulty sectors can be flushed, and the active list of fail- > ure modes can be cleared. > > > from mdadm.8: > > When setting the failure mode for level faulty, the options are: > write-transient, wt, read-transient, rt, write-persistent, wp, > read-persistent, rp, write-all, read-fixable, rf, clear, flush, > none. > > Each failure mode can be followed by a number, which is used as > a period between fault generation. Without a number, the fault > is generated once on the first relevant request. With a number, > the fault will be generated after that many requests, and will > continue to be generated every time the period elapses. > > Multiple failure modes can be current simultaneously by using > the --grow option to set subsequent failure modes. > > "clear" or "none" will remove any pending or periodic failure > modes, and "flush" will clear any persistent faults. > > > > NeilBrown > > -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: error injection 2011-09-29 0:59 ` Jojy Varghese @ 2011-09-29 1:08 ` NeilBrown 2011-09-29 2:06 ` Jojy Varghese 0 siblings, 1 reply; 6+ messages in thread From: NeilBrown @ 2011-09-29 1:08 UTC (permalink / raw) To: Jojy Varghese; +Cc: linux-raid [-- Attachment #1: Type: text/plain, Size: 1893 bytes --] On Wed, 28 Sep 2011 17:59:49 -0700 Jojy Varghese <jojy.varghese@gmail.com> wrote: > Thanks Neil. I tried setting my sda7 partition to generate write > errors every 40 bytes(writing 1 byte at a time). I did : md doesn't see byte writes. It sees sectors or more - usually whole pages or groups of pages. > > 1. Create a array with: > mdadm -C /dev/md/me0 -l faulty -n1 /dev/sda7 -C will write a superblock to /dev/sda7 which you don't really want. It doesn't hurt, but I always used -B (--build) to avoid any metadata. > > After this step I can see /dev/md127 and when i do a mdadm -D /dev/md127, i get: > > /dev/md127: > Version : 1.2 > Creation Time : Wed Sep 28 17:35:50 2011 > Raid Level : faulty > Array Size : 969410424 (924.50 GiB 992.68 GB) > Raid Devices : 1 > Total Devices : 1 > Persistence : Superblock is persistent > > Update Time : Wed Sep 28 17:35:50 2011 > State : clean > Active Devices : 1 > Working Devices : 1 > Failed Devices : 0 > Spare Devices : 0 > > Name : eng-dev16.lab.local:me0 (local to host eng-dev16.lab.local) > UUID : 96f4be10:312f9574:f40107aa:d9f278ba > Events : 0 > > Number Major Minor RaidDevice State > 0 8 7 0 active sync /dev/sda7 > > > 2. Set write fault level with: > > mdadm -G /dev/md/me0 --layout=wp40 > > > > After this when i write > 40 bytes into /dev/md127, i dont get any > I/O errors. I am sure i am doing something wrong here. When you write to /dev/md127 it will just go into the page cache and eventually be flushed to the device in one write. Use O_DIRECT or O_SYNC and it will be flushed out more quickly, but always write at least 512 bytes at a time. NeilBrown > > > Any help is much appreciated. > > Thanks > Jojy [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 190 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: error injection 2011-09-29 1:08 ` NeilBrown @ 2011-09-29 2:06 ` Jojy Varghese 2011-09-29 2:12 ` NeilBrown 0 siblings, 1 reply; 6+ messages in thread From: Jojy Varghese @ 2011-09-29 2:06 UTC (permalink / raw) To: NeilBrown; +Cc: linux-raid Thanks Neil. Also, is there any way to find the current fault blocks being set? On Wed, Sep 28, 2011 at 6:08 PM, NeilBrown <neilb@suse.de> wrote: > On Wed, 28 Sep 2011 17:59:49 -0700 Jojy Varghese <jojy.varghese@gmail.com> > wrote: > >> Thanks Neil. I tried setting my sda7 partition to generate write >> errors every 40 bytes(writing 1 byte at a time). I did : > > md doesn't see byte writes. It sees sectors or more - usually whole pages or > groups of pages. > >> >> 1. Create a array with: >> mdadm -C /dev/md/me0 -l faulty -n1 /dev/sda7 > > -C will write a superblock to /dev/sda7 which you don't really want. It > doesn't hurt, but I always used -B (--build) to avoid any metadata. > >> >> After this step I can see /dev/md127 and when i do a mdadm -D /dev/md127, i get: >> >> /dev/md127: >> Version : 1.2 >> Creation Time : Wed Sep 28 17:35:50 2011 >> Raid Level : faulty >> Array Size : 969410424 (924.50 GiB 992.68 GB) >> Raid Devices : 1 >> Total Devices : 1 >> Persistence : Superblock is persistent >> >> Update Time : Wed Sep 28 17:35:50 2011 >> State : clean >> Active Devices : 1 >> Working Devices : 1 >> Failed Devices : 0 >> Spare Devices : 0 >> >> Name : eng-dev16.lab.local:me0 (local to host eng-dev16.lab.local) >> UUID : 96f4be10:312f9574:f40107aa:d9f278ba >> Events : 0 >> >> Number Major Minor RaidDevice State >> 0 8 7 0 active sync /dev/sda7 >> >> >> 2. Set write fault level with: >> >> mdadm -G /dev/md/me0 --layout=wp40 >> >> >> >> After this when i write > 40 bytes into /dev/md127, i dont get any >> I/O errors. I am sure i am doing something wrong here. > > When you write to /dev/md127 it will just go into the page cache and > eventually be flushed to the device in one write. > Use O_DIRECT or O_SYNC and it will be flushed out more quickly, but always > write at least 512 bytes at a time. > > NeilBrown > > > >> >> >> Any help is much appreciated. >> >> Thanks >> Jojy > -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: error injection 2011-09-29 2:06 ` Jojy Varghese @ 2011-09-29 2:12 ` NeilBrown 0 siblings, 0 replies; 6+ messages in thread From: NeilBrown @ 2011-09-29 2:12 UTC (permalink / raw) To: Jojy Varghese; +Cc: linux-raid [-- Attachment #1: Type: text/plain, Size: 348 bytes --] On Wed, 28 Sep 2011 19:06:17 -0700 Jojy Varghese <jojy.varghese@gmail.com> wrote: > Thanks Neil. Also, is there any way to find the current fault blocks being set? > No. All you can get is what is shown in "/proc/mdstat". It wouldn't be too hard to add something to /proc/mdstat or /sys/.... to show that information. NeilBrown [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 190 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2011-09-29 2:12 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-09-28 19:48 error injection Jojy Varghese 2011-09-28 23:11 ` NeilBrown 2011-09-29 0:59 ` Jojy Varghese 2011-09-29 1:08 ` NeilBrown 2011-09-29 2:06 ` Jojy Varghese 2011-09-29 2:12 ` NeilBrown
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).