* [PATCH 1/2] dim: Move all rerere updating into helpers @ 2017-07-12 12:12 Daniel Vetter 2017-07-12 12:12 ` [PATCH 2/2] dim: Try to gc the rr-cache Daniel Vetter 2017-07-13 20:14 ` [PATCH 1/2] dim: Move all rerere updating into helpers Sean Paul 0 siblings, 2 replies; 6+ messages in thread From: Daniel Vetter @ 2017-07-12 12:12 UTC (permalink / raw) To: DRI Development; +Cc: Daniel Vetter, Intel Graphics Development, Daniel Vetter Just prep work. Signed-off-by: Daniel Vetter <daniel.vetter@intel.com> --- dim | 61 ++++++++++++++++++++++++++++++++++--------------------------- 1 file changed, 34 insertions(+), 27 deletions(-) diff --git a/dim b/dim index ca11287535cc..b788edd29653 100755 --- a/dim +++ b/dim @@ -493,11 +493,41 @@ function rr_cache_dir function update_rerere_cache { - cd $DIM_PREFIX/drm-rerere/ + echo -n "Updating rerere cache... " + git pull mkdir $(rr_cache_dir) &> /dev/null || true - cp rr-cache/* $(rr_cache_dir) -r + cp rr-cache/* $(rr_cache_dir) -r --preserve=timestamps cd - > /dev/null + + echo "Done." +} + +function commit_rerere_cache +{ + echo -n "Updating rerere cache... " + + cd $DIM_PREFIX/drm-rerere/ + if git_is_current_branch rerere-cache ; then + remote=$(branch_to_remote rerere-cache) + + git pull >& /dev/null + rm $(rr_cache_dir)/rr-cache -Rf &> /dev/null || true + cp $(rr_cache_dir)/* rr-cache -r + git add ./*.patch >& /dev/null || true + git add rr-cache/* > /dev/null + git rm rr-cache/rr-cache &> /dev/null || true + if git commit -m "$time: $integration_branch rerere cache update" >& /dev/null; then + echo -n "New commit. " + else + echo -n "Nothing changed. " + fi + echo -n "Pushing rerere cache... " + git push $DRY_RUN $remote HEAD >& /dev/null && echo "Done." + else + echo "Fail: Branch setup for the rerere-cache is borked." + exit 1 + fi } function dim_revert_rerere @@ -528,9 +558,7 @@ function dim_rebuild_tip warn_or_fail "integration configuration file $integration_config not commited" fi - echo -n "Updating rerere cache... " - update_rerere_cache >& /dev/null - echo "Done." + update_rerere_cache echo -n "Reloading $integration_config... " read_integration_config @@ -618,28 +646,7 @@ function dim_rebuild_tip echo -n "Pushing $integration_branch... " git push $DRY_RUN $remote +HEAD >& /dev/null && echo "Done." - echo -n "Updating rerere cache... " - cd $rerere - if git_is_current_branch rerere-cache ; then - remote=$(branch_to_remote rerere-cache) - - git pull >& /dev/null - rm $(rr_cache_dir)/rr-cache -Rf &> /dev/null || true - cp $(rr_cache_dir)/* rr-cache -r - git add ./*.patch >& /dev/null || true - git add rr-cache/* > /dev/null - git rm rr-cache/rr-cache &> /dev/null || true - if git commit -m "$time: $integration_branch rerere cache update" >& /dev/null; then - echo -n "New commit. " - else - echo -n "Nothing changed. " - fi - echo -n "Pushing rerere cache... " - git push $DRY_RUN $remote HEAD >& /dev/null && echo "Done." - else - echo "Fail: Branch setup for the rerere-cache is borked." - exit 1 - fi + commit_rerere_cache } # additional patch checks before pushing, e.g. for r-b tags -- 2.13.2 _______________________________________________ Intel-gfx mailing list Intel-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/intel-gfx ^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 2/2] dim: Try to gc the rr-cache 2017-07-12 12:12 [PATCH 1/2] dim: Move all rerere updating into helpers Daniel Vetter @ 2017-07-12 12:12 ` Daniel Vetter 2017-07-13 20:16 ` Sean Paul 2017-07-13 20:14 ` [PATCH 1/2] dim: Move all rerere updating into helpers Sean Paul 1 sibling, 1 reply; 6+ messages in thread From: Daniel Vetter @ 2017-07-12 12:12 UTC (permalink / raw) To: DRI Development; +Cc: Daniel Vetter, Intel Graphics Development, Daniel Vetter The problem is that we have a distributed cache - every committer has a copy. Which means even just a slight clock skew will make sure that a naive gc algorithm results in lots of thrashing around. To fix this add a huge hysteresis: Only add files newer than 1 day, and only remove them when older than 60 days. As long as people have reasonable accurate clocks on their machines this should work. A different problem is that we can't use filesystem timestamps (and hence can't use git rerere gc): When someone comes back from vacations and updates git rerere, all the files will have current timestamps, even when they've been pushed out weeks ago. To fix that, use the git log to judge old files to remove. Also, remove old files before adding new ones, to avoid confusion. Also, we need to teach the cp -r to preserve timestamps, otherwise this won't work. v2: Use git log to remove old files. v3: Remove the debug uncommenting (Sean). v4: Split out code movement and explain better what's going on (Jani). Signed-off-by: Daniel Vetter <daniel.vetter@intel.com> --- dim | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/dim b/dim index b788edd29653..79d616cbf354 100755 --- a/dim +++ b/dim @@ -513,9 +513,15 @@ function commit_rerere_cache git pull >& /dev/null rm $(rr_cache_dir)/rr-cache -Rf &> /dev/null || true - cp $(rr_cache_dir)/* rr-cache -r + cp $(rr_cache_dir)/* rr-cache -r --preserve=timestamps git add ./*.patch >& /dev/null || true - git add rr-cache/* > /dev/null + for file in $(git ls-files); do + if ! git log --since="60 days ago" --name-only -- $file | grep $file &> /dev/null; then + git rm $file &> /dev/null + echo deleting $file + fi + done + find rr-cache/ -ctime -1 -type f -print0 | xargs -0 git add > /dev/null git rm rr-cache/rr-cache &> /dev/null || true if git commit -m "$time: $integration_branch rerere cache update" >& /dev/null; then echo -n "New commit. " -- 2.13.2 _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH 2/2] dim: Try to gc the rr-cache 2017-07-12 12:12 ` [PATCH 2/2] dim: Try to gc the rr-cache Daniel Vetter @ 2017-07-13 20:16 ` Sean Paul 2017-07-14 9:57 ` Jani Nikula 0 siblings, 1 reply; 6+ messages in thread From: Sean Paul @ 2017-07-13 20:16 UTC (permalink / raw) To: Daniel Vetter; +Cc: Daniel Vetter, Intel Graphics Development, DRI Development On Wed, Jul 12, 2017 at 02:12:24PM +0200, Daniel Vetter wrote: > The problem is that we have a distributed cache - every committer has > a copy. Which means even just a slight clock skew will make sure that > a naive gc algorithm results in lots of thrashing around. > > To fix this add a huge hysteresis: Only add files newer than 1 day, > and only remove them when older than 60 days. As long as people have > reasonable accurate clocks on their machines this should work. > > A different problem is that we can't use filesystem timestamps (and > hence can't use git rerere gc): When someone comes back from vacations > and updates git rerere, all the files will have current timestamps, > even when they've been pushed out weeks ago. To fix that, use the git > log to judge old files to remove. Also, remove old files before adding > new ones, to avoid confusion. > > Also, we need to teach the cp -r to preserve timestamps, otherwise > this won't work. > > v2: Use git log to remove old files. > > v3: Remove the debug uncommenting (Sean). > > v4: Split out code movement and explain better what's going on (Jani). Yeah, much easier to digest with the split. Reviewed-by: Sean Paul <seanpaul@chromium.org> > > Signed-off-by: Daniel Vetter <daniel.vetter@intel.com> > --- > dim | 10 ++++++++-- > 1 file changed, 8 insertions(+), 2 deletions(-) > > diff --git a/dim b/dim > index b788edd29653..79d616cbf354 100755 > --- a/dim > +++ b/dim > @@ -513,9 +513,15 @@ function commit_rerere_cache > > git pull >& /dev/null > rm $(rr_cache_dir)/rr-cache -Rf &> /dev/null || true > - cp $(rr_cache_dir)/* rr-cache -r > + cp $(rr_cache_dir)/* rr-cache -r --preserve=timestamps > git add ./*.patch >& /dev/null || true > - git add rr-cache/* > /dev/null > + for file in $(git ls-files); do > + if ! git log --since="60 days ago" --name-only -- $file | grep $file &> /dev/null; then > + git rm $file &> /dev/null > + echo deleting $file > + fi > + done > + find rr-cache/ -ctime -1 -type f -print0 | xargs -0 git add > /dev/null > git rm rr-cache/rr-cache &> /dev/null || true > if git commit -m "$time: $integration_branch rerere cache update" >& /dev/null; then > echo -n "New commit. " > -- > 2.13.2 > > _______________________________________________ > dri-devel mailing list > dri-devel@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/dri-devel -- Sean Paul, Software Engineer, Google / Chromium OS _______________________________________________ Intel-gfx mailing list Intel-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/intel-gfx ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 2/2] dim: Try to gc the rr-cache 2017-07-13 20:16 ` Sean Paul @ 2017-07-14 9:57 ` Jani Nikula 2017-07-14 13:46 ` Daniel Vetter 0 siblings, 1 reply; 6+ messages in thread From: Jani Nikula @ 2017-07-14 9:57 UTC (permalink / raw) To: Sean Paul, Daniel Vetter Cc: Daniel Vetter, Intel Graphics Development, DRI Development On Thu, 13 Jul 2017, Sean Paul <seanpaul@chromium.org> wrote: > On Wed, Jul 12, 2017 at 02:12:24PM +0200, Daniel Vetter wrote: >> The problem is that we have a distributed cache - every committer has >> a copy. Which means even just a slight clock skew will make sure that >> a naive gc algorithm results in lots of thrashing around. >> >> To fix this add a huge hysteresis: Only add files newer than 1 day, >> and only remove them when older than 60 days. As long as people have >> reasonable accurate clocks on their machines this should work. >> >> A different problem is that we can't use filesystem timestamps (and >> hence can't use git rerere gc): When someone comes back from vacations >> and updates git rerere, all the files will have current timestamps, >> even when they've been pushed out weeks ago. To fix that, use the git >> log to judge old files to remove. Also, remove old files before adding >> new ones, to avoid confusion. >> >> Also, we need to teach the cp -r to preserve timestamps, otherwise >> this won't work. >> >> v2: Use git log to remove old files. >> >> v3: Remove the debug uncommenting (Sean). >> >> v4: Split out code movement and explain better what's going on (Jani). > > Yeah, much easier to digest with the split. > > Reviewed-by: Sean Paul <seanpaul@chromium.org> I'll trust that and hope for the best. ;) BR, Jani. > >> >> Signed-off-by: Daniel Vetter <daniel.vetter@intel.com> >> --- >> dim | 10 ++++++++-- >> 1 file changed, 8 insertions(+), 2 deletions(-) >> >> diff --git a/dim b/dim >> index b788edd29653..79d616cbf354 100755 >> --- a/dim >> +++ b/dim >> @@ -513,9 +513,15 @@ function commit_rerere_cache >> >> git pull >& /dev/null >> rm $(rr_cache_dir)/rr-cache -Rf &> /dev/null || true >> - cp $(rr_cache_dir)/* rr-cache -r >> + cp $(rr_cache_dir)/* rr-cache -r --preserve=timestamps >> git add ./*.patch >& /dev/null || true >> - git add rr-cache/* > /dev/null >> + for file in $(git ls-files); do >> + if ! git log --since="60 days ago" --name-only -- $file | grep $file &> /dev/null; then >> + git rm $file &> /dev/null >> + echo deleting $file >> + fi >> + done >> + find rr-cache/ -ctime -1 -type f -print0 | xargs -0 git add > /dev/null >> git rm rr-cache/rr-cache &> /dev/null || true >> if git commit -m "$time: $integration_branch rerere cache update" >& /dev/null; then >> echo -n "New commit. " >> -- >> 2.13.2 >> >> _______________________________________________ >> dri-devel mailing list >> dri-devel@lists.freedesktop.org >> https://lists.freedesktop.org/mailman/listinfo/dri-devel -- Jani Nikula, Intel Open Source Technology Center _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 2/2] dim: Try to gc the rr-cache 2017-07-14 9:57 ` Jani Nikula @ 2017-07-14 13:46 ` Daniel Vetter 0 siblings, 0 replies; 6+ messages in thread From: Daniel Vetter @ 2017-07-14 13:46 UTC (permalink / raw) To: Jani Nikula Cc: Daniel Vetter, Intel Graphics Development, DRI Development, Daniel Vetter On Fri, Jul 14, 2017 at 12:57:23PM +0300, Jani Nikula wrote: > On Thu, 13 Jul 2017, Sean Paul <seanpaul@chromium.org> wrote: > > On Wed, Jul 12, 2017 at 02:12:24PM +0200, Daniel Vetter wrote: > >> The problem is that we have a distributed cache - every committer has > >> a copy. Which means even just a slight clock skew will make sure that > >> a naive gc algorithm results in lots of thrashing around. > >> > >> To fix this add a huge hysteresis: Only add files newer than 1 day, > >> and only remove them when older than 60 days. As long as people have > >> reasonable accurate clocks on their machines this should work. > >> > >> A different problem is that we can't use filesystem timestamps (and > >> hence can't use git rerere gc): When someone comes back from vacations > >> and updates git rerere, all the files will have current timestamps, > >> even when they've been pushed out weeks ago. To fix that, use the git > >> log to judge old files to remove. Also, remove old files before adding > >> new ones, to avoid confusion. > >> > >> Also, we need to teach the cp -r to preserve timestamps, otherwise > >> this won't work. > >> > >> v2: Use git log to remove old files. > >> > >> v3: Remove the debug uncommenting (Sean). > >> > >> v4: Split out code movement and explain better what's going on (Jani). > > > > Yeah, much easier to digest with the split. > > > > Reviewed-by: Sean Paul <seanpaul@chromium.org> > > I'll trust that and hope for the best. ;) Pushed and asked everyone I managed to ping over irc to upgrade. Let's see how often we have to push out a revert until the old stuff is dead for good (I already screwed up twice locally ...). -Daniel > > BR, > Jani. > > > > > >> > >> Signed-off-by: Daniel Vetter <daniel.vetter@intel.com> > >> --- > >> dim | 10 ++++++++-- > >> 1 file changed, 8 insertions(+), 2 deletions(-) > >> > >> diff --git a/dim b/dim > >> index b788edd29653..79d616cbf354 100755 > >> --- a/dim > >> +++ b/dim > >> @@ -513,9 +513,15 @@ function commit_rerere_cache > >> > >> git pull >& /dev/null > >> rm $(rr_cache_dir)/rr-cache -Rf &> /dev/null || true > >> - cp $(rr_cache_dir)/* rr-cache -r > >> + cp $(rr_cache_dir)/* rr-cache -r --preserve=timestamps > >> git add ./*.patch >& /dev/null || true > >> - git add rr-cache/* > /dev/null > >> + for file in $(git ls-files); do > >> + if ! git log --since="60 days ago" --name-only -- $file | grep $file &> /dev/null; then > >> + git rm $file &> /dev/null > >> + echo deleting $file > >> + fi > >> + done > >> + find rr-cache/ -ctime -1 -type f -print0 | xargs -0 git add > /dev/null > >> git rm rr-cache/rr-cache &> /dev/null || true > >> if git commit -m "$time: $integration_branch rerere cache update" >& /dev/null; then > >> echo -n "New commit. " > >> -- > >> 2.13.2 > >> > >> _______________________________________________ > >> dri-devel mailing list > >> dri-devel@lists.freedesktop.org > >> https://lists.freedesktop.org/mailman/listinfo/dri-devel > > -- > Jani Nikula, Intel Open Source Technology Center -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 1/2] dim: Move all rerere updating into helpers 2017-07-12 12:12 [PATCH 1/2] dim: Move all rerere updating into helpers Daniel Vetter 2017-07-12 12:12 ` [PATCH 2/2] dim: Try to gc the rr-cache Daniel Vetter @ 2017-07-13 20:14 ` Sean Paul 1 sibling, 0 replies; 6+ messages in thread From: Sean Paul @ 2017-07-13 20:14 UTC (permalink / raw) To: Daniel Vetter; +Cc: Daniel Vetter, Intel Graphics Development, DRI Development On Wed, Jul 12, 2017 at 02:12:23PM +0200, Daniel Vetter wrote: > Just prep work. > > Signed-off-by: Daniel Vetter <daniel.vetter@intel.com> Reviewed-by: Sean Paul <seanpaul@chromium.org> > --- > dim | 61 ++++++++++++++++++++++++++++++++++--------------------------- > 1 file changed, 34 insertions(+), 27 deletions(-) > > diff --git a/dim b/dim > index ca11287535cc..b788edd29653 100755 > --- a/dim > +++ b/dim > @@ -493,11 +493,41 @@ function rr_cache_dir > > function update_rerere_cache > { > - cd $DIM_PREFIX/drm-rerere/ > + echo -n "Updating rerere cache... " > + > git pull > mkdir $(rr_cache_dir) &> /dev/null || true > - cp rr-cache/* $(rr_cache_dir) -r > + cp rr-cache/* $(rr_cache_dir) -r --preserve=timestamps > cd - > /dev/null > + > + echo "Done." > +} > + > +function commit_rerere_cache > +{ > + echo -n "Updating rerere cache... " > + > + cd $DIM_PREFIX/drm-rerere/ > + if git_is_current_branch rerere-cache ; then > + remote=$(branch_to_remote rerere-cache) > + > + git pull >& /dev/null > + rm $(rr_cache_dir)/rr-cache -Rf &> /dev/null || true > + cp $(rr_cache_dir)/* rr-cache -r > + git add ./*.patch >& /dev/null || true > + git add rr-cache/* > /dev/null > + git rm rr-cache/rr-cache &> /dev/null || true > + if git commit -m "$time: $integration_branch rerere cache update" >& /dev/null; then > + echo -n "New commit. " > + else > + echo -n "Nothing changed. " > + fi > + echo -n "Pushing rerere cache... " > + git push $DRY_RUN $remote HEAD >& /dev/null && echo "Done." > + else > + echo "Fail: Branch setup for the rerere-cache is borked." > + exit 1 > + fi > } > > function dim_revert_rerere > @@ -528,9 +558,7 @@ function dim_rebuild_tip > warn_or_fail "integration configuration file $integration_config not commited" > fi > > - echo -n "Updating rerere cache... " > - update_rerere_cache >& /dev/null > - echo "Done." > + update_rerere_cache > > echo -n "Reloading $integration_config... " > read_integration_config > @@ -618,28 +646,7 @@ function dim_rebuild_tip > echo -n "Pushing $integration_branch... " > git push $DRY_RUN $remote +HEAD >& /dev/null && echo "Done." > > - echo -n "Updating rerere cache... " > - cd $rerere > - if git_is_current_branch rerere-cache ; then > - remote=$(branch_to_remote rerere-cache) > - > - git pull >& /dev/null > - rm $(rr_cache_dir)/rr-cache -Rf &> /dev/null || true > - cp $(rr_cache_dir)/* rr-cache -r > - git add ./*.patch >& /dev/null || true > - git add rr-cache/* > /dev/null > - git rm rr-cache/rr-cache &> /dev/null || true > - if git commit -m "$time: $integration_branch rerere cache update" >& /dev/null; then > - echo -n "New commit. " > - else > - echo -n "Nothing changed. " > - fi > - echo -n "Pushing rerere cache... " > - git push $DRY_RUN $remote HEAD >& /dev/null && echo "Done." > - else > - echo "Fail: Branch setup for the rerere-cache is borked." > - exit 1 > - fi > + commit_rerere_cache > } > > # additional patch checks before pushing, e.g. for r-b tags > -- > 2.13.2 > > _______________________________________________ > dri-devel mailing list > dri-devel@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/dri-devel -- Sean Paul, Software Engineer, Google / Chromium OS _______________________________________________ Intel-gfx mailing list Intel-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/intel-gfx ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2017-07-14 13:46 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2017-07-12 12:12 [PATCH 1/2] dim: Move all rerere updating into helpers Daniel Vetter 2017-07-12 12:12 ` [PATCH 2/2] dim: Try to gc the rr-cache Daniel Vetter 2017-07-13 20:16 ` Sean Paul 2017-07-14 9:57 ` Jani Nikula 2017-07-14 13:46 ` Daniel Vetter 2017-07-13 20:14 ` [PATCH 1/2] dim: Move all rerere updating into helpers Sean Paul
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.