using git on flash media

using git on flash media

[David Tweed]

I was wondering if anyone here could answer two silly
questions arising from a lack of knowledge about the
precise architecture of git (unfortunately
websearching "using git flash" turns up pages about
miserable people and Macromedia):

Is it reasonable to use git reasonably intensively
keeping it's database on a flash media drive? (Ie, the
flash drive is plugged into a standard desktop machine
and the "actual versions" of the file that're being
managed are on the machine's hard disc, but all of the
git repository data are on a flash media drive. What
I'm basically checking is that it doesn't, I dunno,
rewrite files so frequently that on a modern flash
drive it would wear out the entire drive unreasonably
quickly.

Likewise, supposing that there are several machines
with the same filesystem tree devoted to the
git-chronicled project. Supposing you've got a careful
user who checks out the latest tree from the flash
drive into the machine's hard disk upon sitting down
at one of these machines, and commits to the flash
drive & properly umounts the flash before pulling it
out. The user does this switching between several
machines randomly. (So the git repository on flash
acts to ensure that whenever I sit down at a machine
I'm presented with my latest versions of everything.)
Is there any obvious problem that could come up which
could lead to git getting confused and somehow
corrupting the archive contents on the flash drive?
(Not the kind of catastrophic loss thing that'd be
caught by taking regular backups but some corruption
that'd silently make parts of the repository
un-checkoutable in the main flash repository.) I'm
trying to imagine some program somehow getting
confused and somehow writing some vital piece of
repository data to the "current" hard disc without
realising that means it's not "always readable all the
time", unlike the flash drive proper.

(I know the answer probably ought to be "just make all
the machines networked and communicate via the network
rather than a flash drive", but assume I'm not
amenable to changing.)

Many thanks for any insight,

cheers, david tweed


Send instant messages to your online friends http://uk.messenger.yahoo.com 

Re: using git on flash media

[Linus Torvalds]

On Thu, 20 Apr 2006, David Tweed wrote:
>
> What I'm basically checking is that it doesn't, I dunno, rewrite files 
> so frequently that on a modern flash drive it would wear out the entire 
> drive unreasonably quickly.

The largely write-once nature of git should mean that the only files that 
get rewritten a lot are
 - the directories get rewritten to, since git creates new objects at a 
   reasonable pace
 - the branch references get rewritten.

In general, I'd say that git probably does less writing than most other 
SCM's are likely to do.

That said, when you say "modern flash drive", I really suspect you 
shouldn't care deeply any more. Modern flash devices can be rewritten a 
lot more than old ones could (by an order of magnitude or more), and they 
almost always have wear levelling in hw, making it even less of an issue 
(but if they don't, your biggest issue will be that you should use a 
filesystem that does it for you).

That said, if you want to be safe, I think flash memory card vendors 
guarantee only up to 10,000 write cycles (and it used to be much less). 

That's _complete_ rewrites, though, which is more than just a single 
sector write. They tend to guarantee 100,000 single-sector re-writes (ie 
more like the "directory update" things when you create a new object).

And assuming you'd count one commit as one "total rewrite" (which sounds 
unlikely - but it's certainly more than one sector - I don't know what 
they consider a total rewrite when they make up their numbers), that 
implies that to be really safe, you shouldn't do more than 10,000 commits 
before you replace your flash. Quite frankly, I suspect that's _way_ more 
conservative than you should be, but hey, since you asked..

10,000 commits is actually a fair number. The kernel has gotten 25,000 in 
a year, but the kernel is a pretty active and large project. I suspect 
that 10,000 commits is quite a lot of years for most projects.

One rule: NEVER mount your flash with the "sync" option, and use "noatime" 
to avoid unnecessary inode access time updates (that's especially true for 
git, where archive atimes aren't interesting, but it's usually a good idea 
for flash in general). Otherwise you'll get normal accesses ending up 
doign "writes" too and writes will do a lot more of them, and the above 
"one commit = one rewrite" rule-of-thumb is suddenly not at all 
conservative.

Btw, backups are still good. Flash or no flash, and whether you're very 
conservative in your flash usage or not.

		Linus

Re: using git on flash media

[Josh Boyer]

On 4/19/06, Linus Torvalds <torvalds@osdl.org> wrote:
>
>
> On Thu, 20 Apr 2006, David Tweed wrote:
> >
> > What I'm basically checking is that it doesn't, I dunno, rewrite files
> > so frequently that on a modern flash drive it would wear out the entire
> > drive unreasonably quickly.
>
> The largely write-once nature of git should mean that the only files that
> get rewritten a lot are
>  - the directories get rewritten to, since git creates new objects at a
>    reasonable pace
>  - the branch references get rewritten.
>
> In general, I'd say that git probably does less writing than most other
> SCM's are likely to do.
>
> That said, when you say "modern flash drive", I really suspect you
> shouldn't care deeply any more. Modern flash devices can be rewritten a
> lot more than old ones could (by an order of magnitude or more), and they
> almost always have wear levelling in hw, making it even less of an issue
> (but if they don't, your biggest issue will be that you should use a
> filesystem that does it for you).

Finding one that claims to have HW wear leveling is still hard, so you
can never really tell.

> That said, if you want to be safe, I think flash memory card vendors
> guarantee only up to 10,000 write cycles (and it used to be much less).
>
> That's _complete_ rewrites, though, which is more than just a single
> sector write. They tend to guarantee 100,000 single-sector re-writes (ie
> more like the "directory update" things when you create a new object).

When talking about flash, it's the erase cycles that are counted. 
Most NOR chips guarantee 100,000 erase cycles per eraseblock.  NAND
chips, which are what most flash drives use, can vary between 10,000 -
100,000 erases depending on the technology used.

> And assuming you'd count one commit as one "total rewrite" (which sounds
> unlikely - but it's certainly more than one sector - I don't know what
> they consider a total rewrite when they make up their numbers), that

A "total rewrite" is simply something that causes an eraseblock to be
erased.  That can vary alot, but it essentially boils down to needing
to write new data to a location in an eraseblock that has already been
written to.

> implies that to be really safe, you shouldn't do more than 10,000 commits
> before you replace your flash. Quite frankly, I suspect that's _way_ more
> conservative than you should be, but hey, since you asked..

To be really conservative, just don't do it ;).  Most flash drives
don't allow access to the raw flash chips, so while you can use
something like JFFS2 on them still, the benefits of that are somewhat
lost since you never really know what is going on underneath.

> One rule: NEVER mount your flash with the "sync" option, and use "noatime"
> to avoid unnecessary inode access time updates (that's especially true for

Right.

josh