* Re: Linus kernel tree corrupt?
From: Petr Baudis @ 2005-07-10 15:23 UTC (permalink / raw)
To: H. Peter Anvin; +Cc: Russell King, Tony Luck, Jon Smirl, git
In-Reply-To: <42D01174.4050501@zytor.com>
Dear diary, on Sat, Jul 09, 2005 at 08:03:32PM CEST, I got a letter
where "H. Peter Anvin" <hpa@zytor.com> told me that...
> Petr Baudis wrote:
> >
> >Yes, please do. I deprecated rsync a day before Linus "broke" http-pull.
> >It's un-deprecated again for now in the latest Cogito.
> >
>
> Presumably for packed repos you want to drop the --ignore-existing
> --whole-file options I assume?
It _is_ unsafe for individual objects, and your packfile will be corrupt
if you break it in the middle and not have --whole-file turned on, I
assume. It would be ideal if we could make rsync allow resuming download
of the file if interrupted, but not under the final name but in that
hidden file it uses.
> sent 339 bytes received 1802 bytes 4282.00 bytes/sec
> total size is 410 speedup is 0.19
> Missing object of tag v2.6.11... different source (obsolete tag?)
> Missing object of tag v2.6.11-tree... different source (obsolete tag?)
> Missing object of tag v2.6.12... different source (obsolete tag?)
> Missing object of tag v2.6.12-rc2... different source (obsolete tag?)
> Missing object of tag v2.6.12-rc3... different source (obsolete tag?)
> Missing object of tag v2.6.12-rc4... different source (obsolete tag?)
> Missing object of tag v2.6.12-rc5... different source (obsolete tag?)
> Missing object of tag v2.6.12-rc6... different source (obsolete tag?)
> Missing object of tag v2.6.13-rc1... different source (obsolete tag?)
> Missing object of tag v2.6.13-rc2... different source (obsolete tag?)
> New branch: 0109fd37046de64e8459f8c4f4706df9ac7cc82c
> Cloned (origin
> rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
> available as branch "origin")
> Cloned to ./ (origin
> rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
> available as branch "origin")
>
> Is the "missing objects" thing spurious?
Yes, already fixed. Will release 0.12.1 soon.
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
<Espy> be careful, some twit might quote you out of context..
^ permalink raw reply
* Re: New script: cg-clean
From: Petr Baudis @ 2005-07-10 15:46 UTC (permalink / raw)
To: Pavel Roskin; +Cc: git
In-Reply-To: <1120862084.17812.6.camel@dv>
Dear diary, on Sat, Jul 09, 2005 at 12:34:44AM CEST, I got a letter
where Pavel Roskin <proski@gnu.org> told me that...
> Hello, Petr!
Hello,
> Please consider this script for Cogito.
>
> Signed-off-by: Pavel Roskin <proski@gnu.org>
the script is definitively interesting, but I have couple of notes
about it first:
(i) -i sounds wrong for anything but being interactive here ;-) What
about -A?
(ii) I'm confused - if -a is all of the above, how do I clean _only_
regular files, and only those not ignored by cg-status?
(iii) Makes it any sense to remove only special files?
(iv) -r implies being recursive, but it has nothing to do with that
here.
(v) Semantically, I think it's quite close to cg-reset. What about
making it part of cg-reset instead of a separate command? I tend to be
careful about command inflation. (That's part of being careful about the
usability in general.) That's just an idea and possibly a bad one, what
do you think?
Thanks,
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
<Espy> be careful, some twit might quote you out of context..
^ permalink raw reply
* Re: [ANNOUNCE] Cogito-0.12
From: Linus Torvalds @ 2005-07-10 16:51 UTC (permalink / raw)
To: Russell King; +Cc: Junio C Hamano, Tony Luck, Petr Baudis, git
In-Reply-To: <20050710134624.B3279@flint.arm.linux.org.uk>
On Sun, 10 Jul 2005, Russell King wrote:
>
> It means that rsync --delete-after can (in theory) be used when
> making changes available to the upstream maintainer.
I'd suggest against that from a safety standpoint (no backups), but what
you _can_ do is to upload only the objects I don't have.
This actually works - I already synced several weeks ago with Paul
Mackerras, who had made his ppc64 git thing contain only the objects that
I didn't have.
In other words, if you have my tree pointed to by
GIT_ALTERNATE_OBJECT_DIRECTORIES, and you populate your tree only with new
files, you can actually upload that small "sparsely populated" tree as-is
(without any of the objects that came from my tree), and I should be able
to pull it as-is.
Well, at least with rsync. I think my git "pack" send/receive thing might
be unhappy about a partial tree, but that's something I can fix, so if
this makes it easier for people (you can create a totally new tre _really_
cheaply and also upload it and move it around very cheaply), then I'm ok
with pulling from partial repositories, and I have indeed already done so
in the past.
Btw, if people start doing this, then I really think we want a
".git/config" file, so that you can have different alternate object
directories for different git directories without having to remember to
set the environment variables all the time.
Linus
^ permalink raw reply
* cg-clone failure
From: YOSHIFUJI Hideaki / 吉藤英明 @ 2005-07-10 17:02 UTC (permalink / raw)
To: git
Am I doing something wrong?
% cg-version
cogito-0.12 (a2503fd85e6bb7f25d134a5634a1d8efc93fee5f)
% cg-clone -s ../linux-2.6
defaulting to local storage area
`/home/yoshfuji/GIT/linux-2.6+test/../linux-2.6/.git/refs/heads/master' -> `.git/refs/heads/origin'
progress: 1011 objects, 2976247 bytes
does not exist /home/yoshfuji/GIT/linux-2.6+test/../linux-2.6/.git/objects/3b/791b97522eb19f885a5c906d70c296a628ed25
Cannot obtain needed tree 3b791b97522eb19f885a5c906d70c296a628ed25
while processing commit 0000000000000000000000000000000000000000.
cg-pull: objects pull failed
cg-init: pull failed
rm: cannot remove `.' or `..'
--yoshfuji
^ permalink raw reply
* Re: What broke snapshots now?
From: Linus Torvalds @ 2005-07-10 17:08 UTC (permalink / raw)
To: David Woodhouse; +Cc: git
In-Reply-To: <1121006378.23706.65.camel@baythorne.infradead.org>
On Sun, 10 Jul 2005, David Woodhouse wrote:
>
> Now the mailing list feed isn't happy though -- it stopped being able to
> pull from your tree at around 0600 UTC (which I think is then the last
> DRM fix was added). I got this when trying to update...
Which script is this? I'm looking at your scripts, but
"cg-feedmaillist.sh" is unreadable for me, so I can't see all of it.
Anyway, it's possible that this is a temporary problem: one of the issues
is that since you seem to be using the "rsync:" protocol for updating
things, what happens is that if the mirroring is off a bit, you may have
gotten a new head, but not all the objects. Then you'd get exactly this.
I don't know how to solve that, really. It's fundamental to any mirroring
that doesn't guarantee some kind of ordering, ie the dumb kind.
All the "native git" pack-file synchronization protocols should be
strictly ordered, ie they will unpack and write the new objects _before_
they write the new refs that point to them..
But you can just imagine what happens if the mirror script is busy
mirroring at the same time, and has already mirrored the object
directories before we wrote all of them, but then gets to the refs/
directory by the time we have written the new refs (and all the new object
- but it didn't notice that because it had already mirrored the object
dirs).
See what I'm saying? The thing fixes itself after a while when you try
again and the next round of mirroring has gotten the objects it missed the
last time, but there fundamentally isn't anything to be done about the
fact that the mirroring wasn't "atomic". Only a git-aware mirror could do
an atomic snapshot.
I'm actually surprised that people haven't complained more about this than
they have. I know it's happened a couple of times: it bit Dave Jones'
scripts once, and it's caused webgit to sometimes report "permission
denied" problems (or whatever) and then showing empty logs.
The fact that my upload scripts _do_ upload in the right order, together
with the fact that rsync normally syncs alphabetically (ie "objects"
before "refs") means that we _normally_ don't see this, but it's
occasionally unavoidable when I happen to do a new push while a previous
mirror sweep is going on.
Of course, in this case it may also be just some cogito thing that doesn't
like pack-files. The fact that it complains about an object that is not
actually either of the "heads" of the diff makes me suspect it's really a
missing object, though, which in turn makes me suspect the above mirror
mis-feature.
Linus
^ permalink raw reply
* Re: What broke snapshots now?
From: David Woodhouse @ 2005-07-10 17:22 UTC (permalink / raw)
To: Linus Torvalds; +Cc: git
In-Reply-To: <Pine.LNX.4.58.0507100954080.17536@g5.osdl.org>
On Sun, 2005-07-10 at 10:08 -0700, Linus Torvalds wrote:
> Which script is this? I'm looking at your scripts, but
> "cg-feedmaillist.sh" is unreadable for me, so I can't see all of it.
Hm. Dunno why that happened -- it's readable now, and also at
http://david.woodhou.se/cg-feedmaillist.sh
> Anyway, it's possible that this is a temporary problem: one of the issues
> is that since you seem to be using the "rsync:" protocol for updating
> things, what happens is that if the mirroring is off a bit, you may have
> gotten a new head, but not all the objects. Then you'd get exactly this.
It's done locally on hera though -- so the mirroring shouldn't be a
problem. IIRC the reason it uses rsync is because I wasn't getting tags
when it was using whatever other method was the default for a local
'parent repository'.
That was actually more relevant for the snapshots than the mailing list
feed, though -- so even if it isn't fixed now, I could live without
tags.
More usefully though, if ordering really isn't a problem on your
repository then I should probably rewrite the script to work directly
from that instead of from a copy.
--
dwmw2
^ permalink raw reply
* Re: What broke snapshots now?
From: Linus Torvalds @ 2005-07-10 17:31 UTC (permalink / raw)
To: David Woodhouse; +Cc: git
In-Reply-To: <1121016147.23706.91.camel@baythorne.infradead.org>
On Sun, 10 Jul 2005, David Woodhouse wrote:
>
> > Anyway, it's possible that this is a temporary problem: one of the issues
> > is that since you seem to be using the "rsync:" protocol for updating
> > things, what happens is that if the mirroring is off a bit, you may have
> > gotten a new head, but not all the objects. Then you'd get exactly this.
>
> It's done locally on hera though -- so the mirroring shouldn't be a
> problem.
No it's not, as far as I can tell:
torvalds@hera:/home/dwmw2/git/mail-2.6(0)$ cat .git/branches/origin
rsync://www.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
so your scripts will go out to rsync with www.kernel.org to get the data,
when you use "cg-update origin".
> More usefully though, if ordering really isn't a problem on your
> repository then I should probably rewrite the script to work directly
> from that instead of from a copy.
It should not be a problem, but you obviously have to be careful about
ordering, ie you should fetch the references first, and act on them later.
Linus
^ permalink raw reply
* [RFC] Design for http-pull on repo with packs
From: Daniel Barkalow @ 2005-07-10 18:42 UTC (permalink / raw)
To: git
I have a design for using http-pull on a packed repository, and it only
requires one extra file in the repository: an append-only list of the pack
files (because getting the directory listing is very painful and
failure-prone).
The first thing to note is that fetch() is allowed to get more than just
the requested object. This means that we can get the pack file with the
requested object, and this will fulfill the contract of fetch(), and,
hopefully, be extra-helpful (since we expect the repository owner to have
packed stuff together usefully). So I do this:
Try to get individual files. So long as this works, everything is as
before.
If an individual file is not available, figure out what packs are
available:
Get the list of pack files the repository has
(currently, I just use "e3117bbaf6a59cb53c3f6f0d9b17b9433f0e4135")
For any packs we don't have, get the index files.
Keep a list of the struct packed_gits for the packs the server has
(these are not used as places to look for objects)
Each time we need an object, check the list for it. If it is in there,
download the corresponding pack and report success.
I've nearly got an implementation ready, except for not having a way of
getting a list of available packs. It seems to work for getting
e3117bbaf6a59cb53c3f6f0d9b17b9433f0e4135 when necessary, although I'm
still debugging the last few things.
-Daniel
*This .sig left intentionally blank*
^ permalink raw reply
* Re: [ANNOUNCE] Cogito-0.12
From: Russell King @ 2005-07-10 19:15 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Junio C Hamano, Petr Baudis, git
In-Reply-To: <Pine.LNX.4.58.0507100942020.17536@g5.osdl.org>
On Sun, Jul 10, 2005 at 09:51:16AM -0700, Linus Torvalds wrote:
> On Sun, 10 Jul 2005, Russell King wrote:
> > It means that rsync --delete-after can (in theory) be used when
> > making changes available to the upstream maintainer.
>
> I'd suggest against that from a safety standpoint (no backups), but what
> you _can_ do is to upload only the objects I don't have.
>
> This actually works - I already synced several weeks ago with Paul
> Mackerras, who had made his ppc64 git thing contain only the objects that
> I didn't have.
Ok, let's give this a go then. However, I'm not confident in this
working, especially after seeing the output of git-fsck-cache --full...
and I've no idea _why_ it's complaining.
I've pushed this (partial) tree out to
master.kernel.org:~rmk/linux-2.6-arm.git
Below is the usual mail.
$ export | grep GIT_
declare -x GIT_ALTERNATE_OBJECT_DIRECTORIES="/home/rmk/git/linux-2.6/.git/objects"
$ git-fsck-cache --full
error: cannot read sha1_file for 0084227438c28d26bc2d089b1facc4675310f741
bad sha1 entry '0084227438c28d26bc2d089b1facc4675310f741'
error: cannot read sha1_file for 008c1ddc1fc2854b64fcb49a40f1c933d116fb5c
bad sha1 entry '008c1ddc1fc2854b64fcb49a40f1c933d116fb5c'
...
error: cannot read sha1_file for 83c28d2c90fe720b5a315b89301cf3a519ffed88
bad sha1 entry '83c28d2c90fe720b5a315b89301cf3a519ffed88'
dangling commit 043d051615aa5da09a7e44f1edbb69798458e067
dangling commit a92b7b80579fe68fe229892815c750f6652eb6a9
$ grep . .git/refs/heads/*
.git/refs/heads/master:ec6bced6c7b92904f5ead39c9c1b8dc734e6eff0
.git/refs/heads/origin:f179bc77d09b9087bfc559d0368bba350342ac76
.git/refs/heads/smp:053a7b5b7617a72d7c61b6f84196d1c0f79b9849
$ cd $GIT_ALTERNATE_OBJECT_DIRECTORIES/../..
$ git-fsck-cache --full
$
Could this be because cogito doesn't know how to handle this setup
properly yet? Have I just destroyed my git tree by trying to apply
stuff to it?
---
Linus, Andrew,
Please incorporate the latest ARM changes, which can be found at:
master.kernel.org:/home/rmk/linux-2.6-arm.git
This will update the following files:
arch/arm/mach-omap/Kconfig | 221 -----
arch/arm/mach-omap/Makefile | 40
arch/arm/mach-omap/Makefile.boot | 4
arch/arm/mach-omap/board-generic.c | 100 --
arch/arm/mach-omap/board-h2.c | 189 ----
arch/arm/mach-omap/board-h3.c | 207 -----
arch/arm/mach-omap/board-innovator.c | 282 ------
arch/arm/mach-omap/board-netstar.c | 153 ---
arch/arm/mach-omap/board-osk.c | 171 ----
arch/arm/mach-omap/board-perseus2.c | 191 ----
arch/arm/mach-omap/board-voiceblue.c | 258 ------
arch/arm/mach-omap/clock.c | 1076 --------------------------
arch/arm/mach-omap/clock.h | 112 --
arch/arm/mach-omap/common.c | 549 -------------
arch/arm/mach-omap/common.h | 36
arch/arm/mach-omap/dma.c | 1086 --------------------------
arch/arm/mach-omap/fpga.c | 188 ----
arch/arm/mach-omap/gpio.c | 762 ------------------
arch/arm/mach-omap/irq.c | 219 -----
arch/arm/mach-omap/leds-h2p2-debug.c | 144 ---
arch/arm/mach-omap/leds-innovator.c | 103 --
arch/arm/mach-omap/leds-osk.c | 198 ----
arch/arm/mach-omap/leds.c | 61 -
arch/arm/mach-omap/leds.h | 3
arch/arm/mach-omap/mcbsp.c | 685 ----------------
arch/arm/mach-omap/mux.c | 163 ---
arch/arm/mach-omap/ocpi.c | 114 --
arch/arm/mach-omap/pm.c | 632 ---------------
arch/arm/mach-omap/sleep.S | 314 -------
arch/arm/mach-omap/time.c | 424 ----------
arch/arm/mach-omap/usb.c | 593 --------------
arch/arm/Kconfig | 6
arch/arm/Makefile | 6
arch/arm/configs/enp2611_defconfig | 20
arch/arm/configs/ixdp2400_defconfig | 20
arch/arm/configs/ixdp2401_defconfig | 20
arch/arm/configs/ixdp2800_defconfig | 20
arch/arm/configs/ixdp2801_defconfig | 20
arch/arm/configs/omap_h2_1610_defconfig | 117 +-
arch/arm/mach-ixp2000/core.c | 55 -
arch/arm/mach-ixp2000/enp2611.c | 1
arch/arm/mach-ixp2000/ixdp2x00.c | 1
arch/arm/mach-ixp2000/ixdp2x01.c | 1
arch/arm/mach-omap1/Kconfig | 144 +++
arch/arm/mach-omap1/Makefile | 30
arch/arm/mach-omap1/Makefile.boot | 3
arch/arm/mach-omap1/board-generic.c | 99 ++
arch/arm/mach-omap1/board-h2.c | 188 ++++
arch/arm/mach-omap1/board-h3.c | 206 ++++
arch/arm/mach-omap1/board-innovator.c | 281 ++++++
arch/arm/mach-omap1/board-netstar.c | 152 +++
arch/arm/mach-omap1/board-osk.c | 170 ++++
arch/arm/mach-omap1/board-perseus2.c | 190 ++++
arch/arm/mach-omap1/board-voiceblue.c | 257 ++++++
arch/arm/mach-omap1/fpga.c | 188 ++++
arch/arm/mach-omap1/id.c | 188 ++++
arch/arm/mach-omap1/io.c | 115 ++
arch/arm/mach-omap1/irq.c | 234 +++++
arch/arm/mach-omap1/leds-h2p2-debug.c | 144 +++
arch/arm/mach-omap1/leds-innovator.c | 103 ++
arch/arm/mach-omap1/leds-osk.c | 194 ++++
arch/arm/mach-omap1/leds.c | 61 +
arch/arm/mach-omap1/leds.h | 3
arch/arm/mach-omap1/serial.c | 200 ++++
arch/arm/mach-omap1/time.c | 436 ++++++++++
arch/arm/mm/Kconfig | 2
arch/arm/mm/mm-armv.c | 4
arch/arm/plat-omap/Kconfig | 112 ++
arch/arm/plat-omap/Makefile | 17
arch/arm/plat-omap/clock.c | 1323 ++++++++++++++++++++++++++++++++
arch/arm/plat-omap/clock.h | 120 ++
arch/arm/plat-omap/common.c | 135 +++
arch/arm/plat-omap/cpu-omap.c | 128 +++
arch/arm/plat-omap/dma.c | 1116 ++++++++++++++++++++++++++
arch/arm/plat-omap/gpio.c | 762 ++++++++++++++++++
arch/arm/plat-omap/mcbsp.c | 758 ++++++++++++++++++
arch/arm/plat-omap/mux.c | 160 +++
arch/arm/plat-omap/ocpi.c | 114 ++
arch/arm/plat-omap/pm.c | 632 +++++++++++++++
arch/arm/plat-omap/sleep.S | 314 +++++++
arch/arm/plat-omap/usb.c | 593 ++++++++++++++
include/asm-arm/arch-ixp2000/platform.h | 1
include/asm-arm/arch-omap/board-h2.h | 5
include/asm-arm/arch-omap/board-h3.h | 5
include/asm-arm/arch-omap/board-osk.h | 5
include/asm-arm/arch-omap/board.h | 12
include/asm-arm/arch-omap/common.h | 36
include/asm-arm/arch-omap/dma.h | 1
include/asm-arm/arch-omap/hardware.h | 24
include/asm-arm/arch-omap/irqs.h | 3
include/asm-arm/arch-omap/mux.h | 28
include/asm-arm/arch-omap/omap16xx.h | 32
include/asm-arm/arch-omap/system.h | 21
93 files changed, 10164 insertions(+), 9450 deletions(-)
through these changes:
From: Tony Lindgren: Sun Jul 10 19:58:20 BST 2005
[PATCH] ARM: 2803/1: OMAP update 11/11: Add cpufreq support
Patch from Tony Lindgren
This patch adds minimal cpufreq support for OMAP
taking advantage of the clock framework.
Signed-off-by: Tony Lindgren <tony@atomide.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
From: Tony Lindgren: Sun Jul 10 19:58:19 BST 2005
[PATCH] ARM: 2805/1: OMAP update 10/11: Update H2 defconfig
Patch from Tony Lindgren
This patch updates H2 defconfig.
Signed-off-by: Tony Lindgren <tony@atomide.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
From: Tony Lindgren: Sun Jul 10 19:58:18 BST 2005
[PATCH] ARM: 2804/1: OMAP update 9/11: Update OMAP arch files
Patch from Tony Lindgren
This patch by various OMAP developers syncs the OMAP
specific arch files with the linux-omap tree.
Signed-off-by: Tony Lindgren <tony@atomide.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
From: Tony Lindgren: Sun Jul 10 19:58:17 BST 2005
[PATCH] ARM: 2802/1: OMAP update 8/11: Update OMAP arch files
Patch from Tony Lindgren
This patch by various OMAP developers syncs the OMAP
specific arch files with the linux-omap tree.
Signed-off-by: Tony Lindgren <tony@atomide.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
From: Tony Lindgren: Sun Jul 10 19:58:15 BST 2005
[PATCH] ARM: 2812/1: OMAP update 7c/11: Move arch-omap to plat-omap
Patch from Tony Lindgren
This patch move common OMAP code from arch-omap to plat-omap
directory.
Signed-off-by: Tony Lindgren <tony@atomide.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
From: Tony Lindgren: Sun Jul 10 19:58:14 BST 2005
[PATCH] ARM: 2809/1: OMAP update 7b/11: Move arch-omap to plat-omap
Patch from Tony Lindgren
This patch move common OMAP code from arch-omap to plat-omap
directory.
Signed-off-by: Tony Lindgren <tony@atomide.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
From: Tony Lindgren: Sun Jul 10 19:58:13 BST 2005
[PATCH] ARM: 2807/1: OMAP update 7a/11: Move arch-omap to plat-omap
Patch from Tony Lindgren
This patch move common OMAP code from arch-omap to plat-omap
directory.
Signed-off-by: Tony Lindgren <tony@atomide.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
From: Tony Lindgren: Sun Jul 10 19:58:12 BST 2005
[PATCH] ARM: 2801/1: OMAP update 6/11: Split OMAP1 common code into id, io and serial
Patch from Tony Lindgren
This patch by Juha Yrjölä and other OMAP developers splits
OMAP1 specific common code into OMAP1 id, io, and serial
code in mach-omap1 directory.
Signed-off-by: Tony Lindgren <tony@atomide.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
From: Tony Lindgren: Sun Jul 10 19:58:11 BST 2005
[PATCH] ARM: 2806/1: OMAP update 5/11: Move board files into mach-omap1 directory
Patch from Tony Lindgren
This patch by Paul Mundt and other OMAP developers
moves OMAP1 board files into mach-omap1 directory.
Signed-off-by: Tony Lindgren <tony@atomide.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
From: Tony Lindgren: Sun Jul 10 19:58:10 BST 2005
[PATCH] ARM: 2799/1: OMAP update 4/11: Move OMAP1 LED code into mach-omap1 directory
Patch from Tony Lindgren
This patch by Paul Mundt and other OMAP developers
moves OMAP1 specific LED code into mach-omap1 directory.
Signed-off-by: Tony Lindgren <tony@atomide.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
From: Tony Lindgren: Sun Jul 10 19:58:09 BST 2005
[PATCH] ARM: 2800/1: OMAP update 3/11: Move OMAP1 core code into mach-omap1 directory
Patch from Tony Lindgren
This patch by Paul Mundt and other OMAP developers
moves OMAP1 specific IRQ, time, and FPGA code into
mach-omap1 directory.
Signed-off-by: Tony Lindgren <tony@atomide.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
From: Tony Lindgren: Sun Jul 10 19:58:08 BST 2005
[PATCH] ARM: 2798/1: OMAP update 2/11: Change ARM Kconfig to support omap1 and omap2
Patch from Tony Lindgren
This patch by Paul Mundt and other OMAP developers modifies
ARM specific Kconfig to allow sharing code between OMAP1 and
OMAP2 architectures.
In order to share code between OMAP1 and OMAP2, all OMAP1
specific code is moved into mach-omap1 directory in the
following patch. A new mach-omap2 directory will be added
later on.
Signed-off-by: Tony Lindgren <tony@atomide.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
From: Tony Lindgren: Sun Jul 10 19:58:06 BST 2005
[PATCH] ARM: 2797/1: OMAP update 1/11: Update include files
Patch from Tony Lindgren
This patch by various OMAP developers syncs the OMAP
specific include files with the linux-omap tree.
Signed-off-by: Tony Lindgren <tony@atomide.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
From: Deepak Saxena: Sun Jul 10 19:44:55 BST 2005
[PATCH] ARM: 2796/1: Fix ARMv5[TEJ] check in MMU initalization
Patch from Deepak Saxena
The code in mm-armv.c checks for the condition (cpu_architecture()<= ARMv5)
in a few places but should be checking for ARMv5TEJ as the MMU is shared
across all v5 variations.
Signed-off-by: Deepak Saxena <dsaxena@plexity.net>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
From: Lennert Buytenhek: Sun Jul 10 19:44:54 BST 2005
[PATCH] ARM: 2795/1: update ixp2000 defconfigs
Patch from Lennert Buytenhek
Update the ixp2000 defconfigs from 2.6.12-git6 to 2.6.13-rc2.
Signed-off-by: Lennert Buytenhek <buytenh@wantstofly.org>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
From: Lennert Buytenhek: Sun Jul 10 19:44:53 BST 2005
[PATCH] ARM: 2793/1: platform serial support for ixp2000
Patch from Lennert Buytenhek
This patch converts the ixp2000 serial port over to a platform
serial device.
Signed-off-by: Lennert Buytenhek <buytenh@wantstofly.org>
Signed-off-by: Deepak Saxena <dsaxena@plexity.net>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
--
Russell King
^ permalink raw reply
* [PATCH 1/2] Management of packs not yet installed
From: Daniel Barkalow @ 2005-07-10 19:55 UTC (permalink / raw)
To: git; +Cc: Linus Torvalds, Petr Baudis
In-Reply-To: <Pine.LNX.4.21.0507101539220.30848-100000@iabervon.org>
Support for parsing index files without pack files, installing pack
files while running, and checking what pack files are available.
Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
---
commit b686d7a0377c24e05dbed0dafe909dda6c3dfb48
tree ce285b1a0adb4f8d415f72668a77bc1f1f92e1e1
parent 167a4a3308f4a1606e268c2204c98d6999046ae0
author Daniel Barkalow <barkalow@iabervon.org> 1121024924 -0400
committer Daniel Barkalow <barkalow@silva-tulga.(none)> 1121024924 -0400
Index: cache.h
===================================================================
--- dbae854c7c91182c8a124d0b85d802945d1c6223/cache.h (mode:100644 sha1:84d43d366c6145a30865aa65d92ada88ab95bb9f)
+++ ce285b1a0adb4f8d415f72668a77bc1f1f92e1e1/cache.h (mode:100644 sha1:719a77dfabb24e58abd21b7f3a4b846a114e000a)
@@ -161,6 +161,8 @@
extern char *mkpath(const char *fmt, ...);
extern char *git_path(const char *fmt, ...);
extern char *sha1_file_name(const unsigned char *sha1);
+extern char *sha1_pack_name(const unsigned char *sha1);
+extern char *sha1_pack_index_name(const unsigned char *sha1);
int safe_create_leading_directories(char *path);
@@ -189,6 +191,9 @@
extern int has_sha1_pack(const unsigned char *sha1);
extern int has_sha1_file(const unsigned char *sha1);
+extern int has_pack_file(const unsigned char *sha1);
+extern int has_pack_index(const unsigned char *sha1);
+
/* Convert to/from hex/sha1 representation */
extern int get_sha1(const char *str, unsigned char *sha1);
extern int get_sha1_hex(const char *hex, unsigned char *sha1);
@@ -260,6 +265,7 @@
void *pack_base;
unsigned int pack_last_used;
unsigned int pack_use_cnt;
+ unsigned char sha1[20];
char pack_name[0]; /* something like ".git/objects/pack/xxxxx.pack" */
} *packed_git;
@@ -274,7 +280,14 @@
extern int path_match(const char *path, int nr, char **match);
extern int get_ack(int fd, unsigned char *result_sha1);
+extern struct packed_git *parse_pack_index(unsigned char *sha1);
+
extern void prepare_packed_git(void);
+extern void install_packed_git(struct packed_git *pack);
+
+struct packed_git *find_sha1_pack(const unsigned char *sha1,
+ struct packed_git *packs);
+
extern int use_packed_git(struct packed_git *);
extern void unuse_packed_git(struct packed_git *);
extern struct packed_git *add_packed_git(char *, int);
Index: sha1_file.c
===================================================================
--- dbae854c7c91182c8a124d0b85d802945d1c6223/sha1_file.c (mode:100644 sha1:b2914dd2ea629ae974fd4b4c272e77cb04e5c0e0)
+++ ce285b1a0adb4f8d415f72668a77bc1f1f92e1e1/sha1_file.c (mode:100644 sha1:27136fdba0fbf2dd943f2634cb49660cdbf95ec4)
@@ -200,6 +200,56 @@
return base;
}
+char *sha1_pack_name(const unsigned char *sha1)
+{
+ static const char hex[] = "0123456789abcdef";
+ static char *name, *base, *buf;
+ int i;
+
+ if (!base) {
+ const char *sha1_file_directory = get_object_directory();
+ int len = strlen(sha1_file_directory);
+ base = xmalloc(len + 60);
+ sprintf(base, "%s/pack/pack-1234567890123456789012345678901234567890.pack", sha1_file_directory);
+ name = base + len + 11;
+ }
+
+ buf = name;
+
+ for (i = 0; i < 20; i++) {
+ unsigned int val = *sha1++;
+ *buf++ = hex[val >> 4];
+ *buf++ = hex[val & 0xf];
+ }
+
+ return base;
+}
+
+char *sha1_pack_index_name(const unsigned char *sha1)
+{
+ static const char hex[] = "0123456789abcdef";
+ static char *name, *base, *buf;
+ int i;
+
+ if (!base) {
+ const char *sha1_file_directory = get_object_directory();
+ int len = strlen(sha1_file_directory);
+ base = xmalloc(len + 60);
+ sprintf(base, "%s/pack/pack-1234567890123456789012345678901234567890.idx", sha1_file_directory);
+ name = base + len + 11;
+ }
+
+ buf = name;
+
+ for (i = 0; i < 20; i++) {
+ unsigned int val = *sha1++;
+ *buf++ = hex[val >> 4];
+ *buf++ = hex[val & 0xf];
+ }
+
+ return base;
+}
+
struct alternate_object_database *alt_odb;
/*
@@ -360,6 +410,14 @@
int use_packed_git(struct packed_git *p)
{
+ if (!p->pack_size) {
+ struct stat st;
+ // We created the struct before we had the pack
+ stat(p->pack_name, &st);
+ if (!S_ISREG(st.st_mode))
+ die("packfile %s not a regular file", p->pack_name);
+ p->pack_size = st.st_size;
+ }
if (!p->pack_base) {
int fd;
struct stat st;
@@ -387,8 +445,10 @@
* this is cheap.
*/
if (memcmp((char*)(p->index_base) + p->index_size - 40,
- p->pack_base + p->pack_size - 20, 20))
+ p->pack_base + p->pack_size - 20, 20)) {
+
die("packfile %s does not match index.", p->pack_name);
+ }
}
p->pack_last_used = pack_used_ctr++;
p->pack_use_cnt++;
@@ -426,6 +486,37 @@
return p;
}
+struct packed_git *parse_pack_index(unsigned char *sha1)
+{
+ struct packed_git *p;
+ unsigned long idx_size;
+ void *idx_map;
+ char *path = sha1_pack_index_name(sha1);
+
+ if (check_packed_git_idx(path, &idx_size, &idx_map))
+ return NULL;
+
+ path = sha1_pack_name(sha1);
+
+ p = xmalloc(sizeof(*p) + strlen(path) + 2);
+ strcpy(p->pack_name, path);
+ p->index_size = idx_size;
+ p->pack_size = 0;
+ p->index_base = idx_map;
+ p->next = NULL;
+ p->pack_base = NULL;
+ p->pack_last_used = 0;
+ p->pack_use_cnt = 0;
+ memcpy(p->sha1, sha1, 20);
+ return p;
+}
+
+void install_packed_git(struct packed_git *pack)
+{
+ pack->next = packed_git;
+ packed_git = pack;
+}
+
static void prepare_packed_git_one(char *objdir)
{
char path[PATH_MAX];
@@ -999,6 +1090,20 @@
return 0;
}
+struct packed_git *find_sha1_pack(const unsigned char *sha1,
+ struct packed_git *packs)
+{
+ struct packed_git *p;
+ struct pack_entry e;
+
+ for (p = packs; p; p = p->next) {
+ if (find_pack_entry_one(sha1, &e, p))
+ return p;
+ }
+ return NULL;
+
+}
+
int sha1_object_info(const unsigned char *sha1, char *type, unsigned long *sizep)
{
int status;
@@ -1283,6 +1388,22 @@
return 0;
}
+int has_pack_index(const unsigned char *sha1)
+{
+ struct stat st;
+ if (stat(sha1_pack_index_name(sha1), &st))
+ return 0;
+ return 1;
+}
+
+int has_pack_file(const unsigned char *sha1)
+{
+ struct stat st;
+ if (stat(sha1_pack_name(sha1), &st))
+ return 0;
+ return 1;
+}
+
int has_sha1_pack(const unsigned char *sha1)
{
struct pack_entry e;
^ permalink raw reply
* Re: [RFC] Design for http-pull on repo with packs
From: Dan Holmsand @ 2005-07-10 19:56 UTC (permalink / raw)
To: Daniel Barkalow; +Cc: git
In-Reply-To: <Pine.LNX.4.21.0507101226011.30848-100000@iabervon.org>
[-- Attachment #1: Type: text/plain, Size: 3689 bytes --]
Daniel Barkalow wrote:
> I have a design for using http-pull on a packed repository, and it only
> requires one extra file in the repository: an append-only list of the pack
> files (because getting the directory listing is very painful and
> failure-prone).
A few comments (as I've been tinkering with a way to solve the problem
myself).
As long as the pack files are named sensibly (i.e. if they are created
by git-repack-script), it's not very error-prone to just get the
directory listing, and look for matches for pack-<sha1>.idx. It seems to
work quite well (see below). It isn't beautiful in any way, but it works...
[snip]
> If an individual file is not available, figure out what packs are
> available:
>
> Get the list of pack files the repository has
> (currently, I just use "e3117bbaf6a59cb53c3f6f0d9b17b9433f0e4135")
> For any packs we don't have, get the index files.
This part might be slightly expensive, for large repositories. If one
assumes that packs are named as by git-repack-script, however, one might
cache indexes we've already seen (again, see below). Or, if you go for
the mandatory "pack-index-file", require that it has a reliable order,
so that you can get the last added index first.
> Keep a list of the struct packed_gits for the packs the server has
> (these are not used as places to look for objects)
>
> Each time we need an object, check the list for it. If it is in there,
> download the corresponding pack and report success.
Here you will need some strategy to deal with packs that overlap with
what we've already got. Basically, small and overlapping packs should be
unpacked, big and non-overlapping ones saved as is (since
git-unpack-objects is painfully slow and memory-hungry...).
One could also optimize the pack-download bit, by figuring out the last
object in the pack that we need (easy enough to do from the index file),
and just get the part of the pack file leading up to that object. That
could be a huge win for independently packed repositories (I don't do
that in my code below, though).
Anyway, here's my attempt at the same thing. It introduces
"git-dumb-fetch", with usage like git-fetch-pack (except that it works
with http and rsync). And it adds some uglyness to git-cat-file, for
figuring out which objects we already have.
I'm sort of using the same basic strategy as you, except that I check
the pack files first (I didn't want to mess with http-pull.c, and I
wanted something that would work with rsync as well).
The strategy is this:
o Check if the repository has some pack files we haven't seen
already
o If there are new pack files, download indexes, and see if
they contain anything new. If so, download pack file and
store or unpack. In either case, note that we have seen the
pack file in question (I've used $GIT_DIR/checked_packs).
o Then
o if http: do the git-http-pull stuff, and we're done
o if rsync: get a list of all object files in the
repository, and download the ones we're still missing.
Feel free to take a look, and use anything that might be useful (if
anything...)
I'm not claiming that this method is better than your way; the only main
differences are the caching of seen index files, and that I download
packs first.
My way is faster if the repository contains overlapping object files and
packs. And doesn't require any new infrastructure.
On the other hand, my method risks fetching too many objects, if a pack
file solely contains stuff from a branch we don't want. And it requires
the git-repack-script naming convention to be used on the remote side.
/dan
[-- Attachment #2: git-dumb-fetch.patch.txt --]
[-- Type: text/plain, Size: 5718 bytes --]
diff --git a/cat-file.c b/cat-file.c
--- a/cat-file.c
+++ b/cat-file.c
@@ -11,6 +11,42 @@ int main(int argc, char **argv)
char type[20];
void *buf;
unsigned long size;
+ int obj_count = 0;
+ int missing_count = 0;
+ char line[1000];
+
+ if (argc == 2 && !strcmp("--count", argv[1])) {
+ while (fgets(line, sizeof(line), stdin)) {
+ if (get_sha1(line, sha1))
+ die("invalid id %s", line);
+ if (has_sha1_file(sha1))
+ ++obj_count;
+ else
+ ++missing_count;
+ }
+ printf("%i %i\n", obj_count, missing_count);
+ return 0;
+ }
+
+ if (argc == 2 && !strcmp("--existing", argv[1])) {
+ while (fgets(line, sizeof(line), stdin)) {
+ if (get_sha1(line, sha1))
+ die("invalid id %s", line);
+ if (has_sha1_file(sha1))
+ printf ("%s", line);
+ }
+ return 0;
+ }
+
+ if (argc == 2 && !strcmp("--missing", argv[1])) {
+ while (fgets(line, sizeof(line), stdin)) {
+ if (get_sha1(line, sha1))
+ die("invalid id %s", line);
+ if (!has_sha1_file(sha1))
+ printf ("%s", line);
+ }
+ return 0;
+ }
if (argc != 3 || get_sha1(argv[2], sha1))
usage("git-cat-file [-t | -s | tagname] <sha1>");
diff --git a/git-dumb-fetch b/git-dumb-fetch
new file mode 100755
--- /dev/null
+++ b/git-dumb-fetch
@@ -0,0 +1,182 @@
+#! /bin/sh
+
+# git-dumb-fetch pulls objects from (optionally) packed remote
+# git repositories.
+
+. git-sh-setup-script || die "Not a git archive"
+
+checked_packs=$GIT_DIR/checked_packs
+
+usage() {
+ die "usage: git-dumb-fetch [-w ref] commit-id url"
+}
+
+http_download() {
+ tmpf=$(basename "$1")
+ wget -O "$tmpd/$tmpf" "$1"
+}
+
+http_cat() {
+ wget -q -O - "$1"
+}
+
+http_list_packs() {
+ # XXX: It would be nice to be able to differentiate between failed
+ # connections and missing pack dir. For now, assume the latter.
+ pindex=$(http_cat "$1/objects/pack/") || return 0
+ # die "error getting $1"
+ echo "$pindex" |
+ sed -n 's,.*pack-\([0-9a-f]\{40\}\)\.idx.*,\1\n,gp' |
+ sed '/^$/d' | sort | uniq
+}
+
+http_pull() {
+ git-http-pull -v -a "$1" "$2/"
+}
+
+rsync_download() {
+ rsync "$1" "$tmpd/" > /dev/null
+}
+
+rsync_cat() {
+ tmpf=$(basename "$1")
+ rsync_download "$1" && cat "$tmpd/$tmpf"
+}
+
+rsync_list_packs() {
+ # list every file on the remote side. we'll use that later
+ echo "Listing remote objects" >&2
+ rsync -zr "$1/objects/" > "$tmpd/files" &&
+ LANG=C sed -n 's,.*pack/pack-\([0-9a-f]\{40\}\)\.idx.*,\1,p' < \
+ "$tmpd/files"
+}
+
+rsync_pull() {
+ LANG=C sed -n 's,.*\([0-9a-f][0-9a-f]\)/\([0-9a-f]\{38\}\).*,\1\2,p' \
+ < "$tmpd/files" |
+ git-cat-file --missing > "$tmpd/missing" &&
+ LANG=C sed 's,^..,\0/,' < "$tmpd/missing" > "$tmpd/tofetch" || exit 1
+
+ [ -s "$tmpd/tofetch" ] || { echo "Nothing new to fetch" >&2; return; }
+
+ if rsync --help 2>&1 | grep -q files-from; then
+ rsync -avz --ignore-existing --whole-file \
+ --files-from="$tmpd/tofetch" \
+ "$2/objects/" "$GIT_OBJECT_DIRECTORY/" >&2
+ else
+ LANG=C sed -n \
+ 's,.*\([0-9a-f][0-9a-f]\)/\([0-9a-f]\{38\}\).*,\1\2,p' \
+ < "$tmpd/files" |
+ git-cat-file --existing > "$tmpd/got" &&
+ LANG=C sed 's,^..,\0/,' < "$tmpd/got" > "$tmpd/excl" || exit 1
+ if [ -f "$checked_packs" ]; then
+ sed 's,^.*,pack/pack-&.idx,' < "$checked_packs"
+ sed 's,^.*,pack/pack-&.pack,' < "$checked_packs"
+ fi >> "$tmpd/excl"
+ rsync -avz --ignore-existing --whole-file \
+ --exclude-from="$tmpd/excl" \
+ "$2/objects/" "$GIT_OBJECT_DIRECTORY/" >&2
+ fi
+}
+
+idx_policy() {
+ # existing=$1 missing=$2
+ if [ $1 -eq 0 -a $2 -eq 0 ]; then
+ echo empty
+ elif [ $2 -eq 0 ]; then
+ echo all
+ elif [ $1 -eq 0 ]; then
+ echo none
+ else
+ if [ $2 -gt 5000 -a $1 -lt $2 ]; then
+ # It's a really big pack. Don't unpack
+ echo all
+ else
+ echo partial
+ fi
+ fi
+}
+
+check_idx() {
+ counts=$(git-show-index | cut -d' ' -f2 | git-cat-file --count) ||
+ exit 1
+ idx_policy $counts
+}
+
+has_pack() {
+ [ -f "$GIT_OBJECT_DIRECTORY/pack/pack-$1.idx" -a \
+ "$GIT_OBJECT_DIRECTORY/pack/pack-$1.pack" ] && return 0
+ [ -f "$checked_packs" ] && grep -q $1 < "$checked_packs"
+}
+
+fetch_packs() {
+ idx=$($list_packs "$1") || exit 1
+ [ "$idx" ] || return 0
+ echo "Examining remote packs: $idx" >&2
+ for i in $idx; do
+ has_pack $i && continue
+ echo "Downloading pack $i" >&2
+ $download "$1/objects/pack/pack-$i.idx" &&
+ gotit=$(check_idx < "$tmpd/pack-$i.idx") || exit 1
+
+ case $gotit in
+ partial | none)
+ $download "$1/objects/pack/pack-$i.pack" &&
+ git-verify-pack "$tmpd/pack-$i" || die "invalid pack" ;;
+ *)
+ echo "Already got all objects in pack $i" >&2 ;;
+ esac
+
+ case $gotit in
+ partial)
+ git-unpack-objects < "$tmpd/pack-$i.pack" || exit 1 ;;
+ none)
+ mv "$tmpd/pack-$i.idx" "$tmpd/pack-$i.pack" \
+ "$GIT_OBJECT_DIRECTORY/pack/" 2>/dev/null ||
+ cp "$tmpd/pack-$i.idx" "$tmpd/pack-$i.pack" \
+ "$GIT_OBJECT_DIRECTORY/pack/" || exit 1 ;;
+ esac
+ echo $i >> "$checked_packs"
+ done
+}
+
+
+while true; do
+ case $1 in
+ --) shift; break ;;
+ -*) die "unknown option: $1" ;;
+ *) break ;;
+ esac
+ shift
+done
+
+url=$1 srchead=$2
+[ -n "$srchead" -a -n "$url" ] || usage
+
+case $url in
+ http://*) proto=http ;;
+ rsync://*) proto=rsync ;;
+ *) die "don't know how to fetch from $url" ;;
+esac
+
+download=${proto}_download
+cat=${proto}_cat
+list_packs=${proto}_list_packs
+pull=${proto}_pull
+
+tmpd=$(mktemp -d "${TMPDIR:-/tmp}/dumbfetch.XXXXXX") || exit 1
+trap "rm -rf '$tmpd'" 0 1 2 3 15
+
+echo "Fetching from $url" >&2
+remoteid=$($cat "$url/refs/$srchead") || die "error reading $srchead"
+
+if [ "$previd" = "$remoteid" ]; then
+ echo "Up to date" >&2
+ exit 0
+fi
+
+fetch_packs "$url" &&
+$pull "$remoteid" "$url" || die "fetch failed"
+
+echo $remoteid
+
^ permalink raw reply
* [PATCH 0/2] Support for packs in HTTP
From: Daniel Barkalow @ 2005-07-10 19:53 UTC (permalink / raw)
To: git; +Cc: Linus Torvalds, Petr Baudis
This series has one patch which is ready to go in and one that's not
(although it's a reasonable phony for the current state of the git world).
1: Several additional functions are needed in the library to support
progressively getting pack data from some remote location and using it
to determine what else to get.
2: git-http-pull can get packs as appropriate by getting all the index
files first, and then using them to figure out whether the object it's
looking for is in some pack it could get.
Currently, there's no sane way to figure out what pack/index files are
available from an HTTP server. But there only seems to be one pack file
available on an HTTP server at the moment, so this tries to get that
one.
-Daniel
*This .sig left intentionally blank*
^ permalink raw reply
* Re: [ANNOUNCE] Cogito-0.12
From: Linus Torvalds @ 2005-07-10 20:03 UTC (permalink / raw)
To: Russell King; +Cc: Junio C Hamano, Petr Baudis, git
In-Reply-To: <20050710201504.A22477@flint.arm.linux.org.uk>
On Sun, 10 Jul 2005, Russell King wrote:
>
> Ok, let's give this a go then. However, I'm not confident in this
> working, especially after seeing the output of git-fsck-cache --full...
> and I've no idea _why_ it's complaining.
Ok, I've downloaded your objects, and it all looks fine. Nothing is
missing.
So something is wrong with the git-fsck-cache handling of
GIT_ALTERNATE_OBJECT_DIRECTORIES, but I don't see what. Other programs
happily see the objects, git-fsck-cache for some reason does not, and thus
complains. I'll try to figure it out.
However, the more I try to make "git-pack-objects" work with a partial
repository, the less happy I am about it. It works wonderfully well with
rsync:, since rsync just doesn't know that something is missing, but
generating the object list when there are objects missing is quite hard.
I can be trivial and say "missing objects aren't interesting", and it
would _work_, but that just doesn't make me happy. So I'm almost getting
ready to say "let's not do this thing after all".
> Could this be because cogito doesn't know how to handle this setup
> properly yet? Have I just destroyed my git tree by trying to apply
> stuff to it?
This is definitely not a cogito problem, that fsck thing is in git itself.
And no, you didn't destroy your tree - I just merged it, and the merged
results look fine and fsck correctly (and I get the same diffstat you do).
It's just a bug in fsck somewhere that makes it look bad.
That said, my inability to check the pack for completeness for a partial
archive makes me think this partial rsync wasn't such a good idea after
all. It _is_ convenient, though, so I'll have to think about the send-pack
issues some more and see if I can resolve the difficulty without too much
problems. And clearly I need to fix git-fsck-cache.
Anyway, I pushed out the merge, so don't worry about your tree. But let's
hold off on this partial thing for a while, ok?
Linus
^ permalink raw reply
* [PATCH 2/2] Demo support for packs via HTTP
From: Daniel Barkalow @ 2005-07-10 19:56 UTC (permalink / raw)
To: git; +Cc: Linus Torvalds, Petr Baudis
In-Reply-To: <Pine.LNX.4.21.0507101539220.30848-100000@iabervon.org>
Support for downloading the pack file
"e3117bbaf6a59cb53c3f6f0d9b17b9433f0e4135" when appropriate. (Will
support other pack files when the repository has a list of them.)
Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
---
commit 74132562a2f6cfce9690a5091de7e85bd51d88af
tree c0ae9cb936abac4412aa4a89928f4609d111fd2c
parent b686d7a0377c24e05dbed0dafe909dda6c3dfb48
author Daniel Barkalow <barkalow@iabervon.org> 1121024943 -0400
committer Daniel Barkalow <barkalow@silva-tulga.(none)> 1121024943 -0400
Index: http-pull.c
===================================================================
--- ce285b1a0adb4f8d415f72668a77bc1f1f92e1e1/http-pull.c (mode:100644 sha1:1f9d60b9b1d5eed85b24d96c240666bbfc5a22ed)
+++ c0ae9cb936abac4412aa4a89928f4609d111fd2c/http-pull.c (mode:100644 sha1:2a8d7e71d9447483668cb4a1eb01a096e736f8e3)
@@ -56,6 +56,126 @@
return size;
}
+static int got_indices = 0;
+
+static struct packed_git *packs = NULL;
+
+static int fetch_index(unsigned char *sha1)
+{
+ char *filename;
+ char *url;
+
+ FILE *indexfile;
+
+ if (has_pack_index(sha1))
+ return 0;
+
+ if (get_verbosely)
+ fprintf(stderr, "Getting index for pack %s\n",
+ sha1_to_hex(sha1));
+
+ url = xmalloc(strlen(base) + 64);
+ sprintf(url, "%s/objects/pack/pack-%s.idx",
+ base, sha1_to_hex(sha1));
+
+ filename = sha1_pack_index_name(sha1);
+ indexfile = fopen(filename, "w");
+ if (!indexfile)
+ return error("Unable to open local file %s for pack index",
+ filename);
+
+ curl_easy_setopt(curl, CURLOPT_FILE, indexfile);
+ curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite);
+ curl_easy_setopt(curl, CURLOPT_URL, url);
+
+ if (curl_easy_perform(curl)) {
+ fclose(indexfile);
+ return error("Unable to get pack index %s", url);
+ }
+
+ fclose(indexfile);
+ return 0;
+}
+
+static int setup_index(unsigned char *sha1)
+{
+ struct packed_git *new_pack;
+ if (has_pack_file(sha1))
+ return 0; // don't list this as something we can get
+
+ if (fetch_index(sha1))
+ return -1;
+
+ new_pack = parse_pack_index(sha1);
+ new_pack->next = packs;
+ packs = new_pack;
+ return 0;
+}
+
+static int fetch_indices(void)
+{
+ unsigned char sha1[20];
+ if (got_indices)
+ return 0;
+ get_sha1_hex("e3117bbaf6a59cb53c3f6f0d9b17b9433f0e4135", sha1);
+ setup_index(sha1);
+ got_indices = 1;
+ return 0;
+}
+
+static int fetch_pack(unsigned char *sha1)
+{
+ char *url;
+ struct packed_git *target;
+ struct packed_git **lst;
+ FILE *packfile;
+ char *filename;
+
+ if (fetch_indices())
+ return -1;
+ target = find_sha1_pack(sha1, packs);
+ if (!target)
+ return error("Couldn't get %s: not separate or in any pack",
+ sha1_to_hex(sha1));
+
+ if (get_verbosely) {
+ fprintf(stderr, "Getting pack %s\n",
+ sha1_to_hex(target->sha1));
+ fprintf(stderr, " which contains %s\n",
+ sha1_to_hex(sha1));
+ }
+
+ url = xmalloc(strlen(base) + 65);
+ sprintf(url, "%s/objects/pack/pack-%s.pack",
+ base, sha1_to_hex(target->sha1));
+
+ filename = sha1_pack_name(target->sha1);
+ packfile = fopen(filename, "w");
+ if (!packfile)
+ return error("Unable to open local file %s for pack",
+ filename);
+
+ curl_easy_setopt(curl, CURLOPT_FILE, packfile);
+ curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite);
+ curl_easy_setopt(curl, CURLOPT_URL, url);
+
+ if (curl_easy_perform(curl)) {
+ fclose(packfile);
+ return error("Unable to get pack file %s", url);
+ }
+
+ fclose(packfile);
+
+ install_packed_git(target);
+
+ lst = &packs;
+ while (*lst != target)
+ lst = &((*lst)->next);
+ *lst = (*lst)->next;
+
+ return 0;
+}
+
int fetch(unsigned char *sha1)
{
char *hex = sha1_to_hex(sha1);
@@ -67,7 +187,7 @@
local = open(filename, O_WRONLY | O_CREAT | O_EXCL, 0666);
if (local < 0)
- return error("Couldn't open %s\n", filename);
+ return error("Couldn't open local object %s\n", filename);
memset(&stream, 0, sizeof(stream));
@@ -75,6 +195,7 @@
SHA1_Init(&c);
+ curl_easy_setopt(curl, CURLOPT_FAILONERROR, 1);
curl_easy_setopt(curl, CURLOPT_FILE, NULL);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);
@@ -90,8 +211,12 @@
curl_easy_setopt(curl, CURLOPT_URL, url);
- if (curl_easy_perform(curl))
- return error("Couldn't get %s for %s\n", url, hex);
+ if (curl_easy_perform(curl)) {
+ unlink(filename);
+ if (fetch_pack(sha1))
+ return error("Tried %s", url);
+ return 0;
+ }
close(local);
inflateEnd(&stream);
^ permalink raw reply
* Re: [RFC] Design for http-pull on repo with packs
From: Daniel Barkalow @ 2005-07-10 20:29 UTC (permalink / raw)
To: Dan Holmsand; +Cc: git
In-Reply-To: <42D17D89.9080808@innehallsbolaget.se>
On Sun, 10 Jul 2005, Dan Holmsand wrote:
> Daniel Barkalow wrote:
> > I have a design for using http-pull on a packed repository, and it only
> > requires one extra file in the repository: an append-only list of the pack
> > files (because getting the directory listing is very painful and
> > failure-prone).
>
> A few comments (as I've been tinkering with a way to solve the problem
> myself).
>
> As long as the pack files are named sensibly (i.e. if they are created
> by git-repack-script), it's not very error-prone to just get the
> directory listing, and look for matches for pack-<sha1>.idx. It seems to
> work quite well (see below). It isn't beautiful in any way, but it works...
I may grab your code for that; the version I just sent seems to be working
except for that.
> > If an individual file is not available, figure out what packs are
> > available:
> >
> > Get the list of pack files the repository has
> > (currently, I just use "e3117bbaf6a59cb53c3f6f0d9b17b9433f0e4135")
> > For any packs we don't have, get the index files.
>
> This part might be slightly expensive, for large repositories. If one
> assumes that packs are named as by git-repack-script, however, one might
> cache indexes we've already seen (again, see below). Or, if you go for
> the mandatory "pack-index-file", require that it has a reliable order,
> so that you can get the last added index first.
Nothing bad happens if you have index files for pack files you don't have,
as it turns out; the library ignores them. So we can keep the index files
around so we can quickly check if they have the objects we want. That way,
we don't have to worry about skipping something now (because it's not
needed) and then ignoring it when the branch gets merged in.
So what I actually do is make a list of the pack files that aren't already
downloaded that are available from the server, and download the index
files for any where the index file isn't downloaded, either.
> > Keep a list of the struct packed_gits for the packs the server has
> > (these are not used as places to look for objects)
> >
> > Each time we need an object, check the list for it. If it is in there,
> > download the corresponding pack and report success.
>
> Here you will need some strategy to deal with packs that overlap with
> what we've already got. Basically, small and overlapping packs should be
> unpacked, big and non-overlapping ones saved as is (since
> git-unpack-objects is painfully slow and memory-hungry...).
I don't think there's an issue to having overlapping packs, either with
each other or with separate objects. If the user wants, stuff can be
repacked outside of the pull operation (note, though, that the index files
should be truncated rather than removed, so that the program doesn't fetch
them again next time some object can't be found easily).
> One could also optimize the pack-download bit, by figuring out the last
> object in the pack that we need (easy enough to do from the index file),
> and just get the part of the pack file leading up to that object. That
> could be a huge win for independently packed repositories (I don't do
> that in my code below, though).
That's only possible if you can figure out what you want to have before
you get it. My code is walking the reachability graph on the client; it
can only figure out what other objects it needs after it's mapped the pack
file.
> Anyway, here's my attempt at the same thing. It introduces
> "git-dumb-fetch", with usage like git-fetch-pack (except that it works
> with http and rsync). And it adds some uglyness to git-cat-file, for
> figuring out which objects we already have.
I might use that method for listing the available packs, although I'd sort
of like to encourage a clean solution first.
-Daniel
*This .sig left intentionally blank*
^ permalink raw reply
* Re: [ANNOUNCE] Cogito-0.12
From: Russell King @ 2005-07-10 20:32 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Junio C Hamano, Petr Baudis, git
In-Reply-To: <Pine.LNX.4.58.0507101228300.17536@g5.osdl.org>
On Sun, Jul 10, 2005 at 01:03:30PM -0700, Linus Torvalds wrote:
> Anyway, I pushed out the merge, so don't worry about your tree. But let's
> hold off on this partial thing for a while, ok?
Thanks, that's good news. I was fearing having to reconstruct stuff.
Do you want me to re-populate linux-2.6-arm.git to be fully populated
or are you happy for it to just grow the new objects as they become
available?
--
Russell King
^ permalink raw reply
* Re: Linus kernel tree corrupt?
From: H. Peter Anvin @ 2005-07-10 20:35 UTC (permalink / raw)
To: Petr Baudis; +Cc: Russell King, Tony Luck, Jon Smirl, git
In-Reply-To: <20050710152309.GE24249@pasky.ji.cz>
Petr Baudis wrote:
>
> It _is_ unsafe for individual objects, and your packfile will be corrupt
> if you break it in the middle and not have --whole-file turned on, I
> assume. It would be ideal if we could make rsync allow resuming download
> of the file if interrupted, but not under the final name but in that
> hidden file it uses.
>
I think if you just don't give it --partial you're fine (rsync always
creates a second copy, --partial decides if a partial file should be
thrown away or not.)
-hpa
^ permalink raw reply
* [PATCH] Make --recover cause pull to trace everything
From: Daniel Barkalow @ 2005-07-10 20:43 UTC (permalink / raw)
To: git; +Cc: Linus Torvalds
Make the --recover flag check the parents of commits which are already
available. This is needed currently to deal with cases where a parent is
pulled along with a commit (in a pack, e.g.) and references above that
parent aren't also pulled together.
Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
---
commit 75e8c1be7a778e0a0fa119fe1bc408341932e7e5
tree ffbe708117543c356eb2981f1e0540b89b7a95e2
parent a7336ae514738f159dad314d6674961427f043a6
author Daniel Barkalow <barkalow@iabervon.org> 1121024019 -0400
committer Daniel Barkalow <barkalow@silva-tulga.(none)> 1121024019 -0400
Index: http-pull.c
===================================================================
--- 248f72f3e4dcb40693488b0c06f93d0b38122b8e/http-pull.c (mode:100644 sha1:1f9d60b9b1d5eed85b24d96c240666bbfc5a22ed)
+++ ffbe708117543c356eb2981f1e0540b89b7a95e2/http-pull.c (mode:100644 sha1:3fa56f08b0b8e7316afcaab3a7bfa3f2d26b550f)
@@ -146,7 +146,10 @@
int arg = 1;
while (arg < argc && argv[arg][0] == '-') {
- if (argv[arg][1] == 't') {
+ if (argv[arg][1] == '-') {
+ if (!strcmp(argv[arg] + 2, "recover"))
+ careful = 1;
+ } else if (argv[arg][1] == 't') {
get_tree = 1;
} else if (argv[arg][1] == 'c') {
get_history = 1;
Index: local-pull.c
===================================================================
--- 248f72f3e4dcb40693488b0c06f93d0b38122b8e/local-pull.c (mode:100644 sha1:2f06fbee8b840a7ae642f5a22e2cb993687f3470)
+++ ffbe708117543c356eb2981f1e0540b89b7a95e2/local-pull.c (mode:100644 sha1:0d10c07844030bc7cb615cf916dce89592151be7)
@@ -116,7 +116,10 @@
int arg = 1;
while (arg < argc && argv[arg][0] == '-') {
- if (argv[arg][1] == 't')
+ if (argv[arg][1] == '-') {
+ if (!strcmp(argv[arg] + 2, "recover"))
+ careful = 1;
+ } else if (argv[arg][1] == 't')
get_tree = 1;
else if (argv[arg][1] == 'c')
get_history = 1;
Index: pull.c
===================================================================
--- 248f72f3e4dcb40693488b0c06f93d0b38122b8e/pull.c (mode:100644 sha1:ed3078e3b27c62c07558fd94f339801cbd685593)
+++ ffbe708117543c356eb2981f1e0540b89b7a95e2/pull.c (mode:100644 sha1:d9763840c7ebcb1e5838c3b960695cafcca3ac73)
@@ -11,6 +11,7 @@
const unsigned char *current_ref = NULL;
+int careful = 0;
int get_tree = 0;
int get_history = 0;
int get_all = 0;
@@ -91,7 +92,8 @@
if (get_history) {
struct commit_list *parents = obj->parents;
for (; parents; parents = parents->next) {
- if (has_sha1_file(parents->item->object.sha1))
+ if (!careful &&
+ has_sha1_file(parents->item->object.sha1))
continue;
if (make_sure_we_have_it(NULL,
parents->item->object.sha1)) {
Index: pull.h
===================================================================
--- 248f72f3e4dcb40693488b0c06f93d0b38122b8e/pull.h (mode:100644 sha1:e173ae3337c4465da87d849f4e5c9da203fdf01d)
+++ ffbe708117543c356eb2981f1e0540b89b7a95e2/pull.h (mode:100644 sha1:d1076468b71b31dd5e59ec55d98de830cf9df60e)
@@ -21,6 +21,12 @@
/* If set, the hash that the current value of write_ref must be. */
extern const unsigned char *current_ref;
+/*
+ * Set to check on everything, instead of stopping at points where we think
+ * we must have everything.
+ */
+extern int careful;
+
/* Set to fetch the target tree. */
extern int get_tree;
Index: ssh-pull.c
===================================================================
--- 248f72f3e4dcb40693488b0c06f93d0b38122b8e/ssh-pull.c (mode:100644 sha1:26356dd7d84ea1bc9f7320b18562ed4117d4fac0)
+++ ffbe708117543c356eb2981f1e0540b89b7a95e2/ssh-pull.c (mode:100644 sha1:7ca4243f3bd84590e7bb94467fd5acccd7d4d6f9)
@@ -61,7 +61,10 @@
const char *prog = getenv("GIT_SSH_PUSH") ? : "git-ssh-push";
while (arg < argc && argv[arg][0] == '-') {
- if (argv[arg][1] == 't') {
+ if (argv[arg][1] == '-') {
+ if (!strcmp(argv[arg] + 2, "recover"))
+ careful = 1;
+ } else if (argv[arg][1] == 't') {
get_tree = 1;
} else if (argv[arg][1] == 'c') {
get_history = 1;
^ permalink raw reply
* Re: [RFC] Design for http-pull on repo with packs
From: Dan Holmsand @ 2005-07-10 21:39 UTC (permalink / raw)
To: Daniel Barkalow; +Cc: git
In-Reply-To: <Pine.LNX.4.21.0507101557510.30848-100000@iabervon.org>
Daniel Barkalow wrote:
> On Sun, 10 Jul 2005, Dan Holmsand wrote:
>>Daniel Barkalow wrote:
>>> If an individual file is not available, figure out what packs are
>>> available:
>>>
>>> Get the list of pack files the repository has
>>> (currently, I just use "e3117bbaf6a59cb53c3f6f0d9b17b9433f0e4135")
>>> For any packs we don't have, get the index files.
>>
>>This part might be slightly expensive, for large repositories. If one
>>assumes that packs are named as by git-repack-script, however, one might
>>cache indexes we've already seen (again, see below). Or, if you go for
>>the mandatory "pack-index-file", require that it has a reliable order,
>>so that you can get the last added index first.
>
>
> Nothing bad happens if you have index files for pack files you don't have,
> as it turns out; the library ignores them. So we can keep the index files
> around so we can quickly check if they have the objects we want. That way,
> we don't have to worry about skipping something now (because it's not
> needed) and then ignoring it when the branch gets merged in.
>
> So what I actually do is make a list of the pack files that aren't already
> downloaded that are available from the server, and download the index
> files for any where the index file isn't downloaded, either.
Aah. In other words, you do the caching thing as well. It seems a little
ugly, though, to store the index-only index files with the rest of the
pack. It might be preferable to introduce something like
$GIT_DIR/index-cache or something, so than it can be easily cleaned (and
don't follow us around forever when
cloning-by-hardlinking-the-entire-object-directory).
You might end up with quite a large number of index files, after a while
though, if you pull from several repositories that are regularly repacked.
>>> Keep a list of the struct packed_gits for the packs the server has
>>> (these are not used as places to look for objects)
>>>
>>> Each time we need an object, check the list for it. If it is in there,
>>> download the corresponding pack and report success.
>>
>>Here you will need some strategy to deal with packs that overlap with
>>what we've already got. Basically, small and overlapping packs should be
>>unpacked, big and non-overlapping ones saved as is (since
>>git-unpack-objects is painfully slow and memory-hungry...).
>
>
> I don't think there's an issue to having overlapping packs, either with
> each other or with separate objects. If the user wants, stuff can be
> repacked outside of the pull operation (note, though, that the index files
> should be truncated rather than removed, so that the program doesn't fetch
> them again next time some object can't be found easily).
Well, the only issue is obviously waste of space. If you fetch a lot of
branches from independently packed repos, it might mean a lot of waste,
though.
About truncating index files: this seems a bit ugly. You get a file that
doesn't contain what it says it contains, which may cause trouble if for
example the git prune thing is used.
You might be better off with a simple list of index files we know we
have all the objects of (and make sure that git-prune-script deletes
this file, since it possibly breaks the contract).
>>One could also optimize the pack-download bit, by figuring out the last
>>object in the pack that we need (easy enough to do from the index file),
>> and just get the part of the pack file leading up to that object. That
>>could be a huge win for independently packed repositories (I don't do
>>that in my code below, though).
>
>
> That's only possible if you can figure out what you want to have before
> you get it. My code is walking the reachability graph on the client; it
> can only figure out what other objects it needs after it's mapped the pack
> file.
No, but we can find out which objects we *don't* want (i.e. the ones we
have). And that may be a lot, e.g. if a repository is fully repacked, or
if we track branches on several similar but independently packed
repositories. And as far as I understand git-pack-objects, it tries to
put recent objects in the front.
I don't have any numbers to back this up with, though. Some testing may
be needed, but since the population of packed public repositories is 1,
this is tricky...
> I might use that method for listing the available packs, although I'd sort
> of like to encourage a clean solution first.
Encouraging cleanliness is obviously a good thing :-)
/dan
^ permalink raw reply
* Re: [ANNOUNCE] Cogito-0.12
From: Linus Torvalds @ 2005-07-10 21:40 UTC (permalink / raw)
To: Russell King; +Cc: Junio C Hamano, Petr Baudis, git
In-Reply-To: <20050710213209.B22477@flint.arm.linux.org.uk>
On Sun, 10 Jul 2005, Russell King wrote:
>
> On Sun, Jul 10, 2005 at 01:03:30PM -0700, Linus Torvalds wrote:
> > Anyway, I pushed out the merge, so don't worry about your tree. But let's
> > hold off on this partial thing for a while, ok?
>
> Thanks, that's good news. I was fearing having to reconstruct stuff.
>
> Do you want me to re-populate linux-2.6-arm.git to be fully populated
> or are you happy for it to just grow the new objects as they become
> available?
We can try just letting it grow. That way I'll have more reason to try to
make the partial-repo thing just work.
Linus
^ permalink raw reply
* Re: [PATCH] rev-list: add "--full-objects" flag.
From: Sven Verdoolaege @ 2005-07-10 21:48 UTC (permalink / raw)
To: Eric W. Biederman; +Cc: Linus Torvalds, Junio C Hamano, git
In-Reply-To: <m1pstrr8k1.fsf@ebiederm.dsl.xmission.com>
On Sat, Jul 09, 2005 at 03:09:02PM -0600, Eric W. Biederman wrote:
> The current intelligent fetch currently has a problem that it cannot
> be used to bootstrap a repository. If you don't have an ancestor
> of what you are fetching you can't fetch it.
>
Not sure if this is what you want, but you could use the
following gitweb patch (to be applied on top of my previous
patches) to get a git tree snapshot for bootstrapping.
http://www.liacs.nl/~sverdool/gitweb.cgi?p=gitweb.git;a=summary
http://www.liacs.nl/~sverdool/gitweb.git/
skimo
--
Support pack snapshots.
---
commit f76a442a0e2166b3f17db0e496545a600a33f94c
tree f8f089ab738864e69e0155b10262dbec832b4a11
parent 8392280de17a89a451c1f7db4e268f2047d4aa83
author Sven Verdoolaege <skimo@liacs.nl> Sun, 10 Jul 2005 23:56:42 +0200
committer Sven Verdoolaege <skimo@liacs.nl> Sun, 10 Jul 2005 23:56:42 +0200
gitweb.cgi | 11 ++++++++---
1 files changed, 8 insertions(+), 3 deletions(-)
diff --git a/gitweb.cgi b/gitweb.cgi
--- a/gitweb.cgi
+++ b/gitweb.cgi
@@ -2058,8 +2058,9 @@ sub git_snapshot {
"<th></th>\n" .
"</tr>\n";
my %types = (
- 'Bzipped tar archive' => 'tar.bz2',
- 'Gzipped tar archive' => 'tar.gz',
+ 'Source tree (bzipped tar archive)' => 'tar.bz2',
+ 'Source tree (gzipped tar archive)' => 'tar.gz',
+ 'Git tree (pack file)' => 'pack',
);
my $alternate = 0;
for my $type (sort keys %types) {
@@ -2094,6 +2095,7 @@ sub git_serve_snapshot {
my %info = (
'tar.bz2' => [ 'application/x-bzip2', 'bzip2' ],
'tar.gz' => [ 'application/x-gzip', 'gzip' ],
+ 'pack' => [ 'application/x-git-pack' ],
);
if (!exists $info{$st}) {
die_error(undef, "Unknown snapshot type.");
@@ -2101,7 +2103,10 @@ sub git_serve_snapshot {
my ($type, $zip) = @{$info{$st}};
print $cgi->header(-type => $type,
-attachment => "$project-$hash.$st");
- open my $fd, "-|", "$gitbin/git-tar-tree $hash '$project-$hash' | $zip"
+ open my $fd, "-|", ($st eq 'pack' ?
+ "$gitbin/git-rev-list --max-count=1 --objects $hash | ".
+ "$gitbin/git-pack-objects --stdout" :
+ "$gitbin/git-tar-tree $hash '$project-$hash' | $zip")
or return;
undef $/;
print <$fd>;
^ permalink raw reply
* [PATCH 0/2] Handing sending objects from packs
From: Daniel Barkalow @ 2005-07-10 22:22 UTC (permalink / raw)
To: git; +Cc: Linus Torvalds
This series adds support for sending individual objects from packs in in
git-ssh-push and removes map_sha1_file.
-Daniel
*This .sig left intentionally blank*
^ permalink raw reply
* [PATCH 2/2] Remove map_sha1_file
From: Daniel Barkalow @ 2005-07-10 22:27 UTC (permalink / raw)
To: git; +Cc: Linus Torvalds
In-Reply-To: <Pine.LNX.4.21.0507101811390.30848-100000@iabervon.org>
Remove map_sha1_file(), now unused.
Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
---
commit c21a02262f770a25b005378e06354e582aa1bfd8
tree 7ac9fabe666f00f37572e7b349fdb859bf8a6491
parent 264ff9f3dcde5553728b34fa08e04643b2b55946
author Daniel Barkalow <barkalow@iabervon.org> 1121033599 -0400
committer Daniel Barkalow <barkalow@silva-tulga.(none)> 1121033599 -0400
Index: cache.h
===================================================================
--- 353fe33ae9c7265d7b685bca864d657e3efe2849/cache.h (mode:100644 sha1:38dac6d6a413f1c788e5331ef4741fc15d72d9bd)
+++ 7ac9fabe666f00f37572e7b349fdb859bf8a6491/cache.h (mode:100644 sha1:11ba95c8aa9202fa3b1a3cbc07bc976641cd1908)
@@ -167,7 +167,6 @@
int safe_create_leading_directories(char *path);
/* Read and unpack a sha1 file into memory, write memory to a sha1 file */
-extern void * map_sha1_file(const unsigned char *sha1, unsigned long *size);
extern int unpack_sha1_header(z_stream *stream, void *map, unsigned long mapsize, void *buffer, unsigned long size);
extern int parse_sha1_header(char *hdr, char *type, unsigned long *sizep);
extern int sha1_object_info(const unsigned char *, char *, unsigned long *);
Index: sha1_file.c
===================================================================
--- 353fe33ae9c7265d7b685bca864d657e3efe2849/sha1_file.c (mode:100644 sha1:08560b2c7a6dff400a46160501c247081f9bb4c7)
+++ 7ac9fabe666f00f37572e7b349fdb859bf8a6491/sha1_file.c (mode:100644 sha1:e082f2e6cb985caca11979311c291aa51d6c37fd)
@@ -578,8 +578,7 @@
}
static void *map_sha1_file_internal(const unsigned char *sha1,
- unsigned long *size,
- int say_error)
+ unsigned long *size)
{
struct stat st;
void *map;
@@ -587,8 +586,6 @@
char *filename = find_sha1_file(sha1, &st);
if (!filename) {
- if (say_error)
- error("cannot map sha1 file %s", sha1_to_hex(sha1));
return NULL;
}
@@ -602,8 +599,6 @@
break;
/* Fallthrough */
case 0:
- if (say_error)
- perror(filename);
return NULL;
}
@@ -620,11 +615,6 @@
return map;
}
-void *map_sha1_file(const unsigned char *sha1, unsigned long *size)
-{
- return map_sha1_file_internal(sha1, size, 1);
-}
-
int unpack_sha1_header(z_stream *stream, void *map, unsigned long mapsize, void *buffer, unsigned long size)
{
/* Get the data stream */
@@ -1112,7 +1102,7 @@
z_stream stream;
char hdr[128];
- map = map_sha1_file_internal(sha1, &mapsize, 0);
+ map = map_sha1_file_internal(sha1, &mapsize);
if (!map) {
struct pack_entry e;
@@ -1151,7 +1141,7 @@
unsigned long mapsize;
void *map, *buf;
- map = map_sha1_file_internal(sha1, &mapsize, 0);
+ map = map_sha1_file_internal(sha1, &mapsize);
if (map) {
buf = unpack_sha1_file(map, mapsize, type, size);
munmap(map, mapsize);
@@ -1331,7 +1321,7 @@
ssize_t size;
unsigned long objsize;
int posn = 0;
- char *buf = map_sha1_file_internal(sha1, &objsize, 0);
+ char *buf = map_sha1_file_internal(sha1, &objsize);
z_stream stream;
if (!buf) {
unsigned char *unpacked;
^ permalink raw reply
* [PATCH 1/2] write_sha1_to_fd()
From: Daniel Barkalow @ 2005-07-10 22:25 UTC (permalink / raw)
To: git; +Cc: Linus Torvalds
In-Reply-To: <Pine.LNX.4.21.0507101811390.30848-100000@iabervon.org>
Add write_sha1_to_fd(), which writes an object to a file descriptor. This
includes support for unpacking it and recompressing it.
Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
---
commit 264ff9f3dcde5553728b34fa08e04643b2b55946
tree 353fe33ae9c7265d7b685bca864d657e3efe2849
parent c3eb461762b1d65e424fc4ede6a1d4f3e0a679f7
author Daniel Barkalow <barkalow@iabervon.org> 1121033477 -0400
committer Daniel Barkalow <barkalow@silva-tulga.(none)> 1121033477 -0400
Index: cache.h
===================================================================
--- 545ef8191b517b7f9e4ea558edaf526038ed1895/cache.h (mode:100644 sha1:719a77dfabb24e58abd21b7f3a4b846a114e000a)
+++ 353fe33ae9c7265d7b685bca864d657e3efe2849/cache.h (mode:100644 sha1:38dac6d6a413f1c788e5331ef4741fc15d72d9bd)
@@ -187,6 +187,7 @@
extern int read_tree(void *buffer, unsigned long size, int stage);
extern int write_sha1_from_fd(const unsigned char *sha1, int fd);
+extern int write_sha1_to_fd(int fd, const unsigned char *sha1);
extern int has_sha1_pack(const unsigned char *sha1);
extern int has_sha1_file(const unsigned char *sha1);
Index: sha1_file.c
===================================================================
--- 545ef8191b517b7f9e4ea558edaf526038ed1895/sha1_file.c (mode:100644 sha1:27136fdba0fbf2dd943f2634cb49660cdbf95ec4)
+++ 353fe33ae9c7265d7b685bca864d657e3efe2849/sha1_file.c (mode:100644 sha1:08560b2c7a6dff400a46160501c247081f9bb4c7)
@@ -1326,6 +1326,65 @@
return 0;
}
+int write_sha1_to_fd(int fd, const unsigned char *sha1)
+{
+ ssize_t size;
+ unsigned long objsize;
+ int posn = 0;
+ char *buf = map_sha1_file_internal(sha1, &objsize, 0);
+ z_stream stream;
+ if (!buf) {
+ unsigned char *unpacked;
+ unsigned long len;
+ char type[20];
+ char hdr[50];
+ int hdrlen;
+ // need to unpack and recompress it by itself
+ unpacked = read_packed_sha1(sha1, type, &len);
+
+ hdrlen = sprintf(hdr, "%s %lu", type, len) + 1;
+
+ /* Set it up */
+ memset(&stream, 0, sizeof(stream));
+ deflateInit(&stream, Z_BEST_COMPRESSION);
+ size = deflateBound(&stream, len + hdrlen);
+ buf = xmalloc(size);
+
+ /* Compress it */
+ stream.next_out = buf;
+ stream.avail_out = size;
+
+ /* First header.. */
+ stream.next_in = hdr;
+ stream.avail_in = hdrlen;
+ while (deflate(&stream, 0) == Z_OK)
+ /* nothing */;
+
+ /* Then the data itself.. */
+ stream.next_in = unpacked;
+ stream.avail_in = len;
+ while (deflate(&stream, Z_FINISH) == Z_OK)
+ /* nothing */;
+ deflateEnd(&stream);
+
+ objsize = stream.total_out;
+ }
+
+ do {
+ size = write(fd, buf + posn, objsize - posn);
+ if (size <= 0) {
+ if (!size) {
+ fprintf(stderr, "write closed");
+ } else {
+ perror("write ");
+ }
+ return -1;
+ }
+ posn += size;
+ } while (posn < objsize);
+ return 0;
+}
+
int write_sha1_from_fd(const unsigned char *sha1, int fd)
{
char *filename = sha1_file_name(sha1);
Index: ssh-push.c
===================================================================
--- 545ef8191b517b7f9e4ea558edaf526038ed1895/ssh-push.c (mode:100644 sha1:090d6f9f8fbde2d736ac5bf563415b0fa402b5aa)
+++ 353fe33ae9c7265d7b685bca864d657e3efe2849/ssh-push.c (mode:100644 sha1:aac70af514e0dc5507fa4997ebad54352c973215)
@@ -7,13 +7,13 @@
static unsigned char local_version = 1;
static unsigned char remote_version = 0;
+static int verbose = 0;
+
static int serve_object(int fd_in, int fd_out) {
ssize_t size;
- int posn = 0;
unsigned char sha1[20];
- unsigned long objsize;
- void *buf;
signed char remote;
+ int posn = 0;
do {
size = read(fd_in, sha1 + posn, 20 - posn);
if (size < 0) {
@@ -25,12 +25,12 @@
posn += size;
} while (posn < 20);
- /* fprintf(stderr, "Serving %s\n", sha1_to_hex(sha1)); */
+ if (verbose)
+ fprintf(stderr, "Serving %s\n", sha1_to_hex(sha1));
+
remote = 0;
- buf = map_sha1_file(sha1, &objsize);
-
- if (!buf) {
+ if (!has_sha1_file(sha1)) {
fprintf(stderr, "git-ssh-push: could not find %s\n",
sha1_to_hex(sha1));
remote = -1;
@@ -41,20 +41,7 @@
if (remote < 0)
return 0;
- posn = 0;
- do {
- size = write(fd_out, buf + posn, objsize - posn);
- if (size <= 0) {
- if (!size) {
- fprintf(stderr, "git-ssh-push: write closed");
- } else {
- perror("git-ssh-push: write ");
- }
- return -1;
- }
- posn += size;
- } while (posn < objsize);
- return 0;
+ return write_sha1_to_fd(fd_out, sha1);
}
static int serve_version(int fd_in, int fd_out)
@@ -76,6 +63,10 @@
return -1;
posn++;
} while (ref[posn - 1]);
+
+ if (verbose)
+ fprintf(stderr, "Serving %s\n", ref);
+
if (get_ref_sha1(ref, sha1))
remote = -1;
write(fd_out, &remote, 1);
^ permalink raw reply
* Re: [PATCH] rev-list: add "--full-objects" flag.
From: Linus Torvalds @ 2005-07-10 22:36 UTC (permalink / raw)
To: Eric W. Biederman; +Cc: Junio C Hamano, git
In-Reply-To: <m1pstrr8k1.fsf@ebiederm.dsl.xmission.com>
On Sat, 9 Jul 2005, Eric W. Biederman wrote:
>
> The current intelligent fetch currently has a problem that it cannot
> be used to bootstrap a repository. If you don't have an ancestor
> of what you are fetching you can't fetch it.
Sure you can.
See the current "git clone". It's actually quite good, it's a pleasure to
use now that it gives updates on how much it has done.
Just do
git clone src dest
to try it out. It starts out silent (for big repositories) because it
takes a while to get the whole rev list, but once it gets going it's quite
nice and gives a nice progress report..
It uses the exact same server side code that "git-fetch-pack" does (ie it
just starts "git-upload-pack" on the server).
Now, one thing you cannot do is to start a totally new _project_ on the
server side. In order to do a "git-send-pack", you need to first create a
directory and do a "git-init-db" on the remote side.
So to create a new project, what you need to do is
src$ ssh target
target$ mkdir new-project
target$ cd new-project
target$ git-init-db
target$ exit
src$ git-send-pack target:new-project master
and you've now sent your "master" branch to the new project at
"target:new-project".
You can even populate multiple branches at a time: just list them all (you
do have to list them, because by default "git-send-pack" will update the
_common_ branches, and since the other end is empty, there obviously are
no common branches to start with).
Ahh, you should even be able to automate the sending of all branches by
doing
git-send-pack target:new-project $(cd .git ; find refs -type f)
I think - that will end up being equivalent to a "reverse clone".
The smart clients are doing pretty damn well, I think.
Linus
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox