Re: Thought about delayed sync

[Thomas Fjellstrom <thomas@fjellstrom.ca>]

On October 9, 2011, Wakko Warner wrote:
> Thomas Fjellstrom wrote:
> > On October 8, 2011, Wakko Warner wrote:
> > > A few days ago, I thought about creating raid arrays w/o syncing.  I
> > > understand why sync is needed.  Please correct me if I'm wrong in any
> > > of my statements.
> > > 
> > > Currently, if someone uses large disks (1tb or larger), the initial
> > > sync can take a long time and until it has completed, the array isn't
> > > fully protected.  I noted on a raid1 of a pair of 1tb disks took hours
> > > to complete when there was no activity.
> > > 
> > > Here is my thought.  There is already a bitmap to indicate which blocks
> > > are dirty.  Thus by using that, a drop of a disk (accidental or
> > > intentional), a resync only syncs those blocks that the bitmap knows
> > > were dirtied.
> > > 
> > > What if another bitmap could be utilized.  This would be an "in use"
> > > bitmap. The purpose of this could be that there would never be an
> > > initial sync. When data is written to an area that has not been
> > > synced, a sync will happen of that region.  Once the sync is complete,
> > > that region will be marked as synced in the bitmap.  Only the parts
> > > that have been written to will be synced.  The other data is of no
> > > consequence.  As with the current bitmap, this would have to be asked
> > > for.
> > > 
> > > Lets say someone has been using this array for some time and a disk
> > > dropped out and had to be replaced.  Lets also say that the actual
> > > usage was about 25-30% of the array (of course, that would be wasted
> > > space).  With the "in use" bitmap, they would replace the disk and
> > > only the areas that had been written to would be resynced over to the
> > > new disk.  The rest, since it had not been used, would not need to be.
> > > 
> > > A side effect of this would be that a check or a resync could use this
> > > to check the real data (IE on a weekly basis) and take less time.
> > > 
> > > Over all, depending on the usage, this can keep the wear and tear on a
> > > disk down.  I'm speaking of personal experience with my systems.  I
> > > have arrays that are not 100% or even 80% used.  I have some
> > > production servers that have extra space for expansion and not fully
> > > used.
> > > 
> > > I'm sure this would take some time to implement if someone does this. 
> > > As I mentioned at the beginning, this was just a thought, but I think
> > > it could benefit people if it were implemented.
> > > 
> > > I am on the list, but feel free to keep me in the CC.
> > 
> > I think theres at least one, probably fatal problem with that idea. There
> > is currently no reliable way for md to tell which areas are actually in
> > use. That is, once a section is written to the first time, it will stay
> > in use, even if it isn't. "Now what about TRIM?" you ask? Not all file
> > systems support it, and I /think/ (based on a quick search of the list)
> > mdraid doesn't fully support TRIM either. LVM may not either. (a quick
> > search also suggested lvm2 doesn't pass on trim properly/at-all).
> 
> Actually, I was completely aware of this before I wrote my thought to the
> list.  I don't know exactly how it could be told.  I thought about a
> program that could read lvm data and tell MD what blocks are not in use. 
> It could go further and attempt to read the filesystem.  TRIM is a nice
> idea, but as you alread mentioned, not all filesystems support it and not
> all layers support passing it.
> 
> > I've been using the current bitmap support on my raid5 array for some
> > time, and it has made the few resync's that were needed, very fast
> > compared to a full resync. Instead of 15+ hours, they finished in 20
> > minutes or less. I call that a win.
> 
> Try this instead.  Create a raid5 (or 6) on 4 2tb drives.  Add about 100gb
> of data to it and replace one of the disks with a fresh disk.  You'll
> notice you have to resync the entire array.  The current bitmap only tells
> which blocks have changed and a resync of an existing member is quick. 
> But a new member has no known in sync blocks and has to resync the whole
> thing.  I know, I already had this happen to me last month.

Yeah, after reading the link to Neil's blog, it hit me how useful it could be.

> On another note, I used this feature to clean the dust out of my disk array
> in another system.  Fail a drive, read the array to verify which drive I
> physically failed, remove it, clean the dust off, add it back, wait for
> resync to complete and then do another disk.  Resync on that was quick for
> the 750gb member.  Without a bitmap, resync time on that system is 3 hours.

Try it on a 7 1TB drive raid5. fun times. I imagine its much worse with 2, 3 
or 4 TB drives. (though not many people have a bunch of internal 4TB drives I 
imagine).

> Thanks for your input though.


-- 
Thomas Fjellstrom
thomas@fjellstrom.ca