Mirrors

Mirrors

[William Roberts]

Recently, I used a SE Android mirror.

After the repository was initialized with --mirror I had a cron job
the updated us every night through script. The script would repo init
(on the appropriate branch) pull the local manifest over .repo and
sync again. This all worked until the projects moved to bitbucket, now
for some reason it dies, it literally hangs checking out the objects.

I then merged the local manifest project paths and branches into my
manifest to other people in my team can just do a checkout with repo
init -b seandroid (and others)

Did anyone else have something similar setup and did they have similar
issues? Does anyone possibly know the cause?

-- 
Respectfully,

William C Roberts

--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.

Re: Mirrors

[Joshua Brindle]

William Roberts wrote:
> Recently, I used a SE Android mirror.
>
> After the repository was initialized with --mirror I had a cron job
> the updated us every night through script. The script would repo init
> (on the appropriate branch) pull the local manifest over .repo and
> sync again. This all worked until the projects moved to bitbucket, now
> for some reason it dies, it literally hangs checking out the objects.
>
> I then merged the local manifest project paths and branches into my
> manifest to other people in my team can just do a checkout with repo
> init -b seandroid (and others)
>
> Did anyone else have something similar setup and did they have similar
> issues? Does anyone possibly know the cause?

I'm doing the same thing. The issue I had was that the bitbucket repos 
are formatted like "device-moto-wingray" instead of device/moto/wingray 
so my mirror started checking out new repos and generally changing the 
directory layout of the mirror. I guess I could have copied everything 
over to the new locations and updated all the manifests but I found it 
lazier to "fix" repo, I'm not sure if they intended to do this or not:

diff --git a/manifest_xml.py b/manifest_xml.py
index a46cf24..a4c56e6 100644
--- a/manifest_xml.py
+++ b/manifest_xml.py
@@ -570,9 +570,9 @@ class XmlManifest(object):
        groups.append('default')

      if self.IsMirror:
-      relpath = None
+      #relpath = None
        worktree = None
-      gitdir = os.path.join(self.topdir, '%s.git' % name)
+      gitdir = os.path.join(self.topdir, '%s.git' % path)
      else:
        worktree = os.path.join(self.topdir, path).replace('\\', '/')
        gitdir = os.path.join(self.repodir, 'projects/%s.git' % path)

--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.

Re: Mirrors

[William Roberts]

That's strange because it worked before with stuff that was outside of
the normal path, like selinux/libselinux. I definitely thing they
intended to use name as that matches up with he remote
and not the local checkout path. I started syncing with the new local
manifest 1 project at a time and it seems to be working ok. Somewhere
it just hangs and I haven't thrown repo into a debugger to see why.

Bill

On Sat, Jul 28, 2012 at 10:00 AM, Joshua Brindle <method@manicmethod.com> wrote:
> William Roberts wrote:
>>
>> Recently, I used a SE Android mirror.
>>
>> After the repository was initialized with --mirror I had a cron job
>> the updated us every night through script. The script would repo init
>> (on the appropriate branch) pull the local manifest over .repo and
>> sync again. This all worked until the projects moved to bitbucket, now
>> for some reason it dies, it literally hangs checking out the objects.
>>
>> I then merged the local manifest project paths and branches into my
>> manifest to other people in my team can just do a checkout with repo
>> init -b seandroid (and others)
>>
>> Did anyone else have something similar setup and did they have similar
>> issues? Does anyone possibly know the cause?
>
>
> I'm doing the same thing. The issue I had was that the bitbucket repos are
> formatted like "device-moto-wingray" instead of device/moto/wingray so my
> mirror started checking out new repos and generally changing the directory
> layout of the mirror. I guess I could have copied everything over to the new
> locations and updated all the manifests but I found it lazier to "fix" repo,
> I'm not sure if they intended to do this or not:
>
> diff --git a/manifest_xml.py b/manifest_xml.py
> index a46cf24..a4c56e6 100644
> --- a/manifest_xml.py
> +++ b/manifest_xml.py
> @@ -570,9 +570,9 @@ class XmlManifest(object):
>        groups.append('default')
>
>      if self.IsMirror:
> -      relpath = None
> +      #relpath = None
>        worktree = None
> -      gitdir = os.path.join(self.topdir, '%s.git' % name)
> +      gitdir = os.path.join(self.topdir, '%s.git' % path)
>      else:
>        worktree = os.path.join(self.topdir, path).replace('\\', '/')
>        gitdir = os.path.join(self.repodir, 'projects/%s.git' % path)



-- 
Respectfully,

William C Roberts

--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.

Re: Mirrors

[William Roberts]

I even tried with -j1 to make sure it wasn't a deadlock in git or the
repo tool and it still would hang. It seems syncing one by one from
the local manifest (even with a high -j option) works fine. Once the
initial mirror sync for master works, ill try it with a full local
manifest and post my results. If it works, then their is some weird
flaw with repo....

Bill

On Sat, Jul 28, 2012 at 1:28 PM, William Roberts
<bill.c.roberts@gmail.com> wrote:
> That's strange because it worked before with stuff that was outside of
> the normal path, like selinux/libselinux. I definitely thing they
> intended to use name as that matches up with he remote
> and not the local checkout path. I started syncing with the new local
> manifest 1 project at a time and it seems to be working ok. Somewhere
> it just hangs and I haven't thrown repo into a debugger to see why.
>
> Bill
>
> On Sat, Jul 28, 2012 at 10:00 AM, Joshua Brindle <method@manicmethod.com> wrote:
>> William Roberts wrote:
>>>
>>> Recently, I used a SE Android mirror.
>>>
>>> After the repository was initialized with --mirror I had a cron job
>>> the updated us every night through script. The script would repo init
>>> (on the appropriate branch) pull the local manifest over .repo and
>>> sync again. This all worked until the projects moved to bitbucket, now
>>> for some reason it dies, it literally hangs checking out the objects.
>>>
>>> I then merged the local manifest project paths and branches into my
>>> manifest to other people in my team can just do a checkout with repo
>>> init -b seandroid (and others)
>>>
>>> Did anyone else have something similar setup and did they have similar
>>> issues? Does anyone possibly know the cause?
>>
>>
>> I'm doing the same thing. The issue I had was that the bitbucket repos are
>> formatted like "device-moto-wingray" instead of device/moto/wingray so my
>> mirror started checking out new repos and generally changing the directory
>> layout of the mirror. I guess I could have copied everything over to the new
>> locations and updated all the manifests but I found it lazier to "fix" repo,
>> I'm not sure if they intended to do this or not:
>>
>> diff --git a/manifest_xml.py b/manifest_xml.py
>> index a46cf24..a4c56e6 100644
>> --- a/manifest_xml.py
>> +++ b/manifest_xml.py
>> @@ -570,9 +570,9 @@ class XmlManifest(object):
>>        groups.append('default')
>>
>>      if self.IsMirror:
>> -      relpath = None
>> +      #relpath = None
>>        worktree = None
>> -      gitdir = os.path.join(self.topdir, '%s.git' % name)
>> +      gitdir = os.path.join(self.topdir, '%s.git' % path)
>>      else:
>>        worktree = os.path.join(self.topdir, path).replace('\\', '/')
>>        gitdir = os.path.join(self.repodir, 'projects/%s.git' % path)
>
>
>
> --
> Respectfully,
>
> William C Roberts



-- 
Respectfully,

William C Roberts

--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.

Re: Mirrors

[William Roberts]

It looks like repo and tuna are not getting along, with the only thing
in the local_manifest.xml file is tuna:

git@sta105992:/repo/seandroid$ repo sync -j8
remote: Counting objects: 5870, done.
remote: Compressing objects: 100% (3180/3180), done.
Receiving objects:  95% (5610/5870), 1023.46 MiB | 311 KiB/s
1/5870), 17.44 MiB | 715 KiB/s

It is not over a gig....

On Sat, Jul 28, 2012 at 3:08 PM, William Roberts
<bill.c.roberts@gmail.com> wrote:
> I even tried with -j1 to make sure it wasn't a deadlock in git or the
> repo tool and it still would hang. It seems syncing one by one from
> the local manifest (even with a high -j option) works fine. Once the
> initial mirror sync for master works, ill try it with a full local
> manifest and post my results. If it works, then their is some weird
> flaw with repo....
>
> Bill
>
> On Sat, Jul 28, 2012 at 1:28 PM, William Roberts
> <bill.c.roberts@gmail.com> wrote:
>> That's strange because it worked before with stuff that was outside of
>> the normal path, like selinux/libselinux. I definitely thing they
>> intended to use name as that matches up with he remote
>> and not the local checkout path. I started syncing with the new local
>> manifest 1 project at a time and it seems to be working ok. Somewhere
>> it just hangs and I haven't thrown repo into a debugger to see why.
>>
>> Bill
>>
>> On Sat, Jul 28, 2012 at 10:00 AM, Joshua Brindle <method@manicmethod.com> wrote:
>>> William Roberts wrote:
>>>>
>>>> Recently, I used a SE Android mirror.
>>>>
>>>> After the repository was initialized with --mirror I had a cron job
>>>> the updated us every night through script. The script would repo init
>>>> (on the appropriate branch) pull the local manifest over .repo and
>>>> sync again. This all worked until the projects moved to bitbucket, now
>>>> for some reason it dies, it literally hangs checking out the objects.
>>>>
>>>> I then merged the local manifest project paths and branches into my
>>>> manifest to other people in my team can just do a checkout with repo
>>>> init -b seandroid (and others)
>>>>
>>>> Did anyone else have something similar setup and did they have similar
>>>> issues? Does anyone possibly know the cause?
>>>
>>>
>>> I'm doing the same thing. The issue I had was that the bitbucket repos are
>>> formatted like "device-moto-wingray" instead of device/moto/wingray so my
>>> mirror started checking out new repos and generally changing the directory
>>> layout of the mirror. I guess I could have copied everything over to the new
>>> locations and updated all the manifests but I found it lazier to "fix" repo,
>>> I'm not sure if they intended to do this or not:
>>>
>>> diff --git a/manifest_xml.py b/manifest_xml.py
>>> index a46cf24..a4c56e6 100644
>>> --- a/manifest_xml.py
>>> +++ b/manifest_xml.py
>>> @@ -570,9 +570,9 @@ class XmlManifest(object):
>>>        groups.append('default')
>>>
>>>      if self.IsMirror:
>>> -      relpath = None
>>> +      #relpath = None
>>>        worktree = None
>>> -      gitdir = os.path.join(self.topdir, '%s.git' % name)
>>> +      gitdir = os.path.join(self.topdir, '%s.git' % path)
>>>      else:
>>>        worktree = os.path.join(self.topdir, path).replace('\\', '/')
>>>        gitdir = os.path.join(self.repodir, 'projects/%s.git' % path)
>>
>>
>>
>> --
>> Respectfully,
>>
>> William C Roberts
>
>
>
> --
> Respectfully,
>
> William C Roberts



-- 
Respectfully,

William C Roberts

--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.

Re: Mirrors

[Joshua Brindle]

It is working here:

$ repo sync -j8
Fetching projects: 100% (386/386), done.

William Roberts wrote:
> It looks like repo and tuna are not getting along, with the only thing
> in the local_manifest.xml file is tuna:
>
> git@sta105992:/repo/seandroid$ repo sync -j8
> remote: Counting objects: 5870, done.
> remote: Compressing objects: 100% (3180/3180), done.
> Receiving objects:  95% (5610/5870), 1023.46 MiB | 311 KiB/s
> 1/5870), 17.44 MiB | 715 KiB/s
>
> It is not over a gig....
>
> On Sat, Jul 28, 2012 at 3:08 PM, William Roberts
> <bill.c.roberts@gmail.com>  wrote:
>> I even tried with -j1 to make sure it wasn't a deadlock in git or the
>> repo tool and it still would hang. It seems syncing one by one from
>> the local manifest (even with a high -j option) works fine. Once the
>> initial mirror sync for master works, ill try it with a full local
>> manifest and post my results. If it works, then their is some weird
>> flaw with repo....
>>
>> Bill
>>
>> On Sat, Jul 28, 2012 at 1:28 PM, William Roberts
>> <bill.c.roberts@gmail.com>  wrote:
>>> That's strange because it worked before with stuff that was outside of
>>> the normal path, like selinux/libselinux. I definitely thing they
>>> intended to use name as that matches up with he remote
>>> and not the local checkout path. I started syncing with the new local
>>> manifest 1 project at a time and it seems to be working ok. Somewhere
>>> it just hangs and I haven't thrown repo into a debugger to see why.
>>>
>>> Bill
>>>
>>> On Sat, Jul 28, 2012 at 10:00 AM, Joshua Brindle<method@manicmethod.com>  wrote:
>>>> William Roberts wrote:
>>>>> Recently, I used a SE Android mirror.
>>>>>
>>>>> After the repository was initialized with --mirror I had a cron job
>>>>> the updated us every night through script. The script would repo init
>>>>> (on the appropriate branch) pull the local manifest over .repo and
>>>>> sync again. This all worked until the projects moved to bitbucket, now
>>>>> for some reason it dies, it literally hangs checking out the objects.
>>>>>
>>>>> I then merged the local manifest project paths and branches into my
>>>>> manifest to other people in my team can just do a checkout with repo
>>>>> init -b seandroid (and others)
>>>>>
>>>>> Did anyone else have something similar setup and did they have similar
>>>>> issues? Does anyone possibly know the cause?
>>>>
>>>> I'm doing the same thing. The issue I had was that the bitbucket repos are
>>>> formatted like "device-moto-wingray" instead of device/moto/wingray so my
>>>> mirror started checking out new repos and generally changing the directory
>>>> layout of the mirror. I guess I could have copied everything over to the new
>>>> locations and updated all the manifests but I found it lazier to "fix" repo,
>>>> I'm not sure if they intended to do this or not:
>>>>
>>>> diff --git a/manifest_xml.py b/manifest_xml.py
>>>> index a46cf24..a4c56e6 100644
>>>> --- a/manifest_xml.py
>>>> +++ b/manifest_xml.py
>>>> @@ -570,9 +570,9 @@ class XmlManifest(object):
>>>>         groups.append('default')
>>>>
>>>>       if self.IsMirror:
>>>> -      relpath = None
>>>> +      #relpath = None
>>>>         worktree = None
>>>> -      gitdir = os.path.join(self.topdir, '%s.git' % name)
>>>> +      gitdir = os.path.join(self.topdir, '%s.git' % path)
>>>>       else:
>>>>         worktree = os.path.join(self.topdir, path).replace('\\', '/')
>>>>         gitdir = os.path.join(self.repodir, 'projects/%s.git' % path)
>>>
>>>
>>> --
>>> Respectfully,
>>>
>>> William C Roberts
>>
>>
>> --
>> Respectfully,
>>
>> William C Roberts
>
>
>

--
This message was distributed to subscribers of the selinux mailing list.
If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.gov with
the words "unsubscribe selinux" without quotes as the message.

Re: Mirrors

[Robert Craig]

[-- Attachment #1: Type: text/plain, Size: 4713 bytes --]

Seems that with a repo sync everything works as expected:

rpcraig@zombie:~/bitbucket-seandroid$ repo sync device-samsung-tuna
Fetching projects: 100% (1/1), done.

However with just using Bill's command, seems to be an issue for me.

rpcraig@zombie:/tmp$ time git clone --mirror
https://bitbucket.org/seandroid/device-samsung-tuna
Cloning into bare repository device-samsung-tuna.git...
remote: Counting objects: 5870, done.
remote: Compressing objects: 100% (3180/3180), done.
fatal: The remote end hung up unexpectedly29 MiB | 1.02 MiB/s
fatal: early EOF
fatal: index-pack failed

real 16m20.657s
user 0m0.000s
sys 0m0.148s



On Sun, Jul 29, 2012 at 7:15 PM, Joshua Brindle <method@manicmethod.com>wrote:

> It is working here:
>
> $ repo sync -j8
> Fetching projects: 100% (386/386), done.
>
>
> William Roberts wrote:
>
>> It looks like repo and tuna are not getting along, with the only thing
>> in the local_manifest.xml file is tuna:
>>
>> git@sta105992:/repo/seandroid$ repo sync -j8
>> remote: Counting objects: 5870, done.
>> remote: Compressing objects: 100% (3180/3180), done.
>> Receiving objects:  95% (5610/5870), 1023.46 MiB | 311 KiB/s
>> 1/5870), 17.44 MiB | 715 KiB/s
>>
>> It is not over a gig....
>>
>> On Sat, Jul 28, 2012 at 3:08 PM, William Roberts
>> <bill.c.roberts@gmail.com>  wrote:
>>
>>> I even tried with -j1 to make sure it wasn't a deadlock in git or the
>>> repo tool and it still would hang. It seems syncing one by one from
>>> the local manifest (even with a high -j option) works fine. Once the
>>> initial mirror sync for master works, ill try it with a full local
>>> manifest and post my results. If it works, then their is some weird
>>> flaw with repo....
>>>
>>> Bill
>>>
>>> On Sat, Jul 28, 2012 at 1:28 PM, William Roberts
>>> <bill.c.roberts@gmail.com>  wrote:
>>>
>>>> That's strange because it worked before with stuff that was outside of
>>>> the normal path, like selinux/libselinux. I definitely thing they
>>>> intended to use name as that matches up with he remote
>>>> and not the local checkout path. I started syncing with the new local
>>>> manifest 1 project at a time and it seems to be working ok. Somewhere
>>>> it just hangs and I haven't thrown repo into a debugger to see why.
>>>>
>>>> Bill
>>>>
>>>> On Sat, Jul 28, 2012 at 10:00 AM, Joshua Brindle<method@manicmethod.com
>>>> **>  wrote:
>>>>
>>>>> William Roberts wrote:
>>>>>
>>>>>> Recently, I used a SE Android mirror.
>>>>>>
>>>>>> After the repository was initialized with --mirror I had a cron job
>>>>>> the updated us every night through script. The script would repo init
>>>>>> (on the appropriate branch) pull the local manifest over .repo and
>>>>>> sync again. This all worked until the projects moved to bitbucket, now
>>>>>> for some reason it dies, it literally hangs checking out the objects.
>>>>>>
>>>>>> I then merged the local manifest project paths and branches into my
>>>>>> manifest to other people in my team can just do a checkout with repo
>>>>>> init -b seandroid (and others)
>>>>>>
>>>>>> Did anyone else have something similar setup and did they have similar
>>>>>> issues? Does anyone possibly know the cause?
>>>>>>
>>>>>
>>>>> I'm doing the same thing. The issue I had was that the bitbucket repos
>>>>> are
>>>>> formatted like "device-moto-wingray" instead of device/moto/wingray so
>>>>> my
>>>>> mirror started checking out new repos and generally changing the
>>>>> directory
>>>>> layout of the mirror. I guess I could have copied everything over to
>>>>> the new
>>>>> locations and updated all the manifests but I found it lazier to "fix"
>>>>> repo,
>>>>> I'm not sure if they intended to do this or not:
>>>>>
>>>>> diff --git a/manifest_xml.py b/manifest_xml.py
>>>>> index a46cf24..a4c56e6 100644
>>>>> --- a/manifest_xml.py
>>>>> +++ b/manifest_xml.py
>>>>> @@ -570,9 +570,9 @@ class XmlManifest(object):
>>>>>         groups.append('default')
>>>>>
>>>>>       if self.IsMirror:
>>>>> -      relpath = None
>>>>> +      #relpath = None
>>>>>         worktree = None
>>>>> -      gitdir = os.path.join(self.topdir, '%s.git' % name)
>>>>> +      gitdir = os.path.join(self.topdir, '%s.git' % path)
>>>>>       else:
>>>>>         worktree = os.path.join(self.topdir, path).replace('\\', '/')
>>>>>         gitdir = os.path.join(self.repodir, 'projects/%s.git' % path)
>>>>>
>>>>
>>>>
>>>> --
>>>> Respectfully,
>>>>
>>>> William C Roberts
>>>>
>>>
>>>
>>> --
>>> Respectfully,
>>>
>>> William C Roberts
>>>
>>
>>
>>
>>
> --
> This message was distributed to subscribers of the selinux mailing list.
> If you no longer wish to subscribe, send mail to majordomo@tycho.nsa.govwith
> the words "unsubscribe selinux" without quotes as the message.
>

[-- Attachment #2: Type: text/html, Size: 6485 bytes --]