* [BUG] 2.46.0 t7701.09 fails on NonStop ia64 @ 2024-08-01 18:03 rsbecker 2024-08-01 19:23 ` Taylor Blau 0 siblings, 1 reply; 12+ messages in thread From: rsbecker @ 2024-08-01 18:03 UTC (permalink / raw) To: git Hi Team, I think this is low priority but would like to understand the situation. It only happens on NonStop ia64 (consistently), not x86. The t7701.09 subtest fails missing files: Total 3 (delta 0), reused 3 (delta 0), pack-reused 0 (from 0) ls: cannot access '.git/objects/pack/pack-*.mtimes': No such file or directory test_line_count: line count for cruft.after != 2 not ok 9 - --max-cruft-size with pruning The test directory contains: ./.git/objects/pack ./.git/objects/pack/pack-68c6c8c8538900694c32380ac1484201c8b60d8d.idx ./.git/objects/pack/pack-68c6c8c8538900694c32380ac1484201c8b60d8d.pack ./.git/objects/pack/pack-68c6c8c8538900694c32380ac1484201c8b60d8d.rev but no mtimes files. I'm not sure how to proceed with this - currently ignoring the failure but I think it might be significant. This only fails on NonStop ia64 (which is much older) and passes on NonStop x86. There is no real file system reason why this would be a difference between the two platforms. There are some limit differences, but that's pretty much it. Any assistance tracking this down would be appreciated. I can set up breakpoints to catch whatever is needed. Regards, Randall ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [BUG] 2.46.0 t7701.09 fails on NonStop ia64 2024-08-01 18:03 [BUG] 2.46.0 t7701.09 fails on NonStop ia64 rsbecker @ 2024-08-01 19:23 ` Taylor Blau 2024-08-01 22:25 ` rsbecker 0 siblings, 1 reply; 12+ messages in thread From: Taylor Blau @ 2024-08-01 19:23 UTC (permalink / raw) To: rsbecker; +Cc: git On Thu, Aug 01, 2024 at 02:03:31PM -0400, rsbecker@nexbridge.com wrote: > Hi Team, > > I think this is low priority but would like to understand the situation. It > only happens on NonStop ia64 (consistently), not x86. > > The t7701.09 subtest fails missing files: Hmm. Script t7701 only has 8 sub-tests, but I think you're referring to t7704.9 > > Total 3 (delta 0), reused 3 (delta 0), pack-reused 0 (from 0) > ls: cannot access '.git/objects/pack/pack-*.mtimes': No such file or > directory > test_line_count: line count for cruft.after != 2 > not ok 9 - --max-cruft-size with pruning > > The test directory contains: > ./.git/objects/pack > ./.git/objects/pack/pack-68c6c8c8538900694c32380ac1484201c8b60d8d.idx > ./.git/objects/pack/pack-68c6c8c8538900694c32380ac1484201c8b60d8d.pack > ./.git/objects/pack/pack-68c6c8c8538900694c32380ac1484201c8b60d8d.rev Interesting. Can you attach the full -vx output of this test, as well as a ls -la from $GIT_DIR/objects/pack? I suspect that this is a mtime resolution issue in whatever filesystem is in use in your ia64 environment, but the full logs will help us confirm that, or at least point is in a better direction. Thanks in advance. Thanks, Taylor ^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: [BUG] 2.46.0 t7701.09 fails on NonStop ia64 2024-08-01 19:23 ` Taylor Blau @ 2024-08-01 22:25 ` rsbecker 2024-08-02 0:58 ` Taylor Blau 0 siblings, 1 reply; 12+ messages in thread From: rsbecker @ 2024-08-01 22:25 UTC (permalink / raw) To: 'Taylor Blau'; +Cc: git On Thursday, August 1, 2024 3:23 PM, Taylor Blau wrote: >On Thu, Aug 01, 2024 at 02:03:31PM -0400, rsbecker@nexbridge.com wrote: >> Hi Team, >> >> I think this is low priority but would like to understand the >> situation. It only happens on NonStop ia64 (consistently), not x86. >> >> The t7701.09 subtest fails missing files: > >Hmm. Script t7701 only has 8 sub-tests, but I think you're referring to >t7704.9 Yes, t7704.9. Sorry. >> Total 3 (delta 0), reused 3 (delta 0), pack-reused 0 (from 0) >> ls: cannot access '.git/objects/pack/pack-*.mtimes': No such file or >> directory >> test_line_count: line count for cruft.after != 2 not ok 9 - >> --max-cruft-size with pruning >> >> The test directory contains: >> ./.git/objects/pack >> ./.git/objects/pack/pack-68c6c8c8538900694c32380ac1484201c8b60d8d.idx >> ./.git/objects/pack/pack-68c6c8c8538900694c32380ac1484201c8b60d8d.pack >> ./.git/objects/pack/pack-68c6c8c8538900694c32380ac1484201c8b60d8d.rev > >Interesting. Can you attach the full -vx output of this test, as well as a ls -la from >$GIT_DIR/objects/pack? > >I suspect that this is a mtime resolution issue in whatever filesystem is in use in your >ia64 environment, but the full logs will help us confirm that, or at least point is in a >better direction. Here is the output as requested, with ls following. This is using OpenSSL 3.0 - OpenSSL 1.0.2 is further down with slightly different results. ++ git init max-cruft-size-prune Initialized empty Git repository in /home/ituglib/randall/git/t/trash directory.t7704-repack-cruft/max-cruft-size-prune/.git/ ++ cd max-cruft-size-prune ++ test_commit base ++ local notick= ++ local echo=echo ++ local append= ++ local author= ++ local signoff= ++ local indir= ++ local tag=light ++ test 1 '!=' 0 ++ case "$1" in ++ break ++ indir= ++ local file=base.t ++ test -n '' ++ echo base ++ git add -- base.t ++ test -z '' ++ test_tick ++ test -z '' ++ test_tick=1112911993 ++ GIT_COMMITTER_DATE='1112911993 -0700' ++ GIT_AUTHOR_DATE='1112911993 -0700' ++ export GIT_COMMITTER_DATE GIT_AUTHOR_DATE ++ git commit -m base [master (root-commit) d1ff1c9] base Author: A U Thor <author@example.com> 1 file changed, 1 insertion(+) create mode 100644 base.t ++ case "$tag" in ++ git tag base +++ generate_random_blob foo 1048576 +++ test-tool genrandom foo 1048576 +++ git hash-object -w -t blob blob +++ rm blob ++ foo=4f336f3df31054eabc05ac05f98bc024c8e05423 +++ generate_random_blob bar 1048576 +++ test-tool genrandom bar 1048576 +++ git hash-object -w -t blob blob +++ rm blob ++ bar=bd44d8aa2d22eb47ff70cef4b0bb45d1549ee49c +++ generate_random_blob baz 1048576 +++ test-tool genrandom baz 1048576 +++ git hash-object -w -t blob blob +++ rm blob ++ baz=ad16bed54cc61cf036075879deeff95ae977514b +++ test_oid_to_path 4f336f3df31054eabc05ac05f98bc024c8e05423 +++ local basename=336f3df31054eabc05ac05f98bc024c8e05423 +++ echo 4f/336f3df31054eabc05ac05f98bc024c8e05423 ++ test-tool chmtime -10000 .git/objects/4f/336f3df31054eabc05ac05f98bc024c8e05423 ++ git repack -d --cruft --max-cruft-size=1M Enumerating objects: 3, done. Counting objects: 100% (3/3), done. Writing objects: 100% (3/3), done. Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0) Enumerating cruft objects: 6, done. Counting objects: 100% (3/3), done. Compressing objects: 100% (3/3), done. Writing objects: 100% (3/3), done. Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0) ++ ls .git/objects/pack/pack-8f53f837597ebbdfc6f8ed173349e5cecbcfb97e.mtimes .git/objects/pack/pack-b8dc9aadaadc12c82b0053fdee0039ae1025a8f8.mtimes .git/objects/pack/pack-c2357b2b204fda52bc1f5515de94227e1db012af.mtimes ++ sort +++ cat cruft.before ++ for cruft in $(cat cruft.before) +++ test-tool chmtime --get -10000 .git/objects/pack/pack-8f53f837597ebbdfc6f8ed173349e5cecbcfb97e.mtimes ++ mtime=1722540725 ++ echo .git/objects/pack/pack-8f53f837597ebbdfc6f8ed173349e5cecbcfb97e.mtimes 1722540725 ++ for cruft in $(cat cruft.before) +++ test-tool chmtime --get -10000 .git/objects/pack/pack-b8dc9aadaadc12c82b0053fdee0039ae1025a8f8.mtimes ++ mtime=1722540723 ++ echo .git/objects/pack/pack-b8dc9aadaadc12c82b0053fdee0039ae1025a8f8.mtimes 1722540723 ++ for cruft in $(cat cruft.before) +++ test-tool chmtime --get -10000 .git/objects/pack/pack-c2357b2b204fda52bc1f5515de94227e1db012af.mtimes ++ mtime=1722540726 ++ echo .git/objects/pack/pack-c2357b2b204fda52bc1f5515de94227e1db012af.mtimes 1722540726 ++ git repack -d --cruft --max-cruft-size=1M --cruft-expiration=10.seconds.ago Enumerating objects: 3, done. Counting objects: 100% (3/3), done. Writing objects: 100% (3/3), done. Total 3 (delta 0), reused 3 (delta 0), pack-reused 0 (from 0) ++ ls '.git/objects/pack/pack-*.mtimes' ++ sort ls: cannot access '.git/objects/pack/pack-*.mtimes': No such file or directory +++ cat cruft.after ++ test_line_count = 3 cruft.before ++ test 3 '!=' 3 +++ wc -l ++ test 3 = 3 ++ test_line_count = 2 cruft.after ++ test 3 '!=' 3 +++ wc -l ++ test 0 = 2 ++ echo 'test_line_count: line count for cruft.after != 2' test_line_count: line count for cruft.after != 2 ++ cat cruft.after ++ return 1 error: last command exited with $?=1 not ok 9 - --max-cruft-size with pruning ls output with second resolution is here - the file system does not have nanosecond resolution despite showing zeros: /home/ituglib/randall/git/t/trash directory.t7704-repack-cruft/max-cruft-size-prune/.git/objects/pack: ls -la --full-time total 11 drwxrwxrwx 1 ITUGLIB.RANDALL ITUGLIB 4096 2024-08-01 16:18:55.000000000 -0600 . drwxrwxrwx 1 ITUGLIB.RANDALL ITUGLIB 4096 2024-08-01 16:18:48.000000000 -0600 .. -r--r--r-- 1 ITUGLIB.RANDALL ITUGLIB 1156 2024-08-01 16:18:52.000000000 -0600 pack-68c6c8c8538900694c32380ac1484201c8b60d8d.idx -r--r--r-- 1 ITUGLIB.RANDALL ITUGLIB 217 2024-08-01 16:18:52.000000000 -0600 pack-68c6c8c8538900694c32380ac1484201c8b60d8d.pack -r--r--r-- 1 ITUGLIB.RANDALL ITUGLIB 64 2024-08-01 16:18:52.000000000 -0600 pack-68c6c8c8538900694c32380ac1484201c8b60d8d.rev When running with OpenSSL 1.0.2, the failure is different: ++ git init max-cruft-size-prune Initialized empty Git repository in /home/ituglib/randall/jenkins/.jenkins/workspace/Git_Pipeline/t/trash directory.t7704-repack-cruft/max-cruft-size-prune/.git/ ++ cd max-cruft-size-prune ++ test_commit base ++ local notick= ++ local echo=echo ++ local append= ++ local author= ++ local signoff= ++ local indir= ++ local tag=light ++ test 1 '!=' 0 ++ case "$1" in ++ break ++ indir= ++ local file=base.t ++ test -n '' ++ echo base ++ git add -- base.t ++ test -z '' ++ test_tick ++ test -z '' ++ test_tick=1112911993 ++ GIT_COMMITTER_DATE='1112911993 -0700' ++ GIT_AUTHOR_DATE='1112911993 -0700' ++ export GIT_COMMITTER_DATE GIT_AUTHOR_DATE ++ git commit -m base [master (root-commit) d1ff1c9] base Author: A U Thor <author@example.com> 1 file changed, 1 insertion(+) create mode 100644 base.t ++ case "$tag" in ++ git tag base +++ generate_random_blob foo 1048576 +++ test-tool genrandom foo 1048576 +++ git hash-object -w -t blob blob +++ rm blob ++ foo=4f336f3df31054eabc05ac05f98bc024c8e05423 +++ generate_random_blob bar 1048576 +++ test-tool genrandom bar 1048576 +++ git hash-object -w -t blob blob +++ rm blob ++ bar=bd44d8aa2d22eb47ff70cef4b0bb45d1549ee49c +++ generate_random_blob baz 1048576 +++ test-tool genrandom baz 1048576 +++ git hash-object -w -t blob blob +++ rm blob ++ baz=ad16bed54cc61cf036075879deeff95ae977514b +++ test_oid_to_path 4f336f3df31054eabc05ac05f98bc024c8e05423 +++ local basename=336f3df31054eabc05ac05f98bc024c8e05423 +++ echo 4f/336f3df31054eabc05ac05f98bc024c8e05423 ++ test-tool chmtime -10000 .git/objects/4f/336f3df31054eabc05ac05f98bc024c8e05423 ++ git repack -d --cruft --max-cruft-size=1M Enumerating objects: 3, done. Counting objects: 100% (3/3), done. Writing objects: 100% (3/3), done. Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0) Enumerating cruft objects: 6, done. Counting objects: 100% (3/3), done. Compressing objects: 100% (3/3), done. Writing objects: 100% (3/3), done. Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0) ++ ls .git/objects/pack/pack-8f53f837597ebbdfc6f8ed173349e5cecbcfb97e.mtimes .git/objects/pack/pack-b8dc9aadaadc12c82b0053fdee0039ae1025a8f8.mtimes .git/objects/pack/pack-c2357b2b204fda52bc1f5515de94227e1db012af.mtimes ++ sort +++ cat cruft.before ++ for cruft in $(cat cruft.before) +++ test-tool chmtime --get -10000 .git/objects/pack/pack-8f53f837597ebbdfc6f8ed173349e5cecbcfb97e.mtimes ++ mtime=1722531262 ++ echo .git/objects/pack/pack-8f53f837597ebbdfc6f8ed173349e5cecbcfb97e.mtimes 1722531262 ++ for cruft in $(cat cruft.before) +++ test-tool chmtime --get -10000 .git/objects/pack/pack-b8dc9aadaadc12c82b0053fdee0039ae1025a8f8.mtimes ++ mtime=1722531261 ++ echo .git/objects/pack/pack-b8dc9aadaadc12c82b0053fdee0039ae1025a8f8.mtimes 1722531261 ++ for cruft in $(cat cruft.before) +++ test-tool chmtime --get -10000 .git/objects/pack/pack-c2357b2b204fda52bc1f5515de94227e1db012af.mtimes ++ mtime=1722531263 ++ echo .git/objects/pack/pack-c2357b2b204fda52bc1f5515de94227e1db012af.mtimes 1722531263 ++ git repack -d --cruft --max-cruft-size=1M --cruft-expiration=10.seconds.ago Enumerating objects: 3, done. Counting objects: 100% (3/3), done. Writing objects: 100% (3/3), done. Total 3 (delta 0), reused 3 (delta 0), pack-reused 0 (from 0) Enumerating cruft objects: 1, done. Traversing cruft objects: 1, done. Counting objects: 100% (1/1), done. Writing objects: 100% (1/1), done. Total 1 (delta 0), reused 1 (delta 0), pack-reused 0 (from 0) ++ ls .git/objects/pack/pack-8f53f837597ebbdfc6f8ed173349e5cecbcfb97e.mtimes ++ sort +++ cat cruft.after ++ for cruft in $(cat cruft.after) +++ grep .git/objects/pack/pack-8f53f837597ebbdfc6f8ed173349e5cecbcfb97e.mtimes mtimes +++ cut '-d ' -f2 ++ old_mtime=1722531262 +++ test-tool chmtime --get .git/objects/pack/pack-8f53f837597ebbdfc6f8ed173349e5cecbcfb97e.mtimes ++ new_mtime=1722541267 ++ test 1722531262 -lt 1722541267 ++ test_line_count = 3 cruft.before ++ test 3 '!=' 3 +++ wc -l ++ test 3 = 3 ++ test_line_count = 2 cruft.after ++ test 3 '!=' 3 +++ wc -l ++ test 1 = 2 ++ echo 'test_line_count: line count for cruft.after != 2' test_line_count: line count for cruft.after != 2 ++ cat cruft.after .git/objects/pack/pack-8f53f837597ebbdfc6f8ed173349e5cecbcfb97e.mtimes ++ return 1 error: last command exited with $?=1 not ok 9 - --max-cruft-size with pruning ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [BUG] 2.46.0 t7701.09 fails on NonStop ia64 2024-08-01 22:25 ` rsbecker @ 2024-08-02 0:58 ` Taylor Blau 2024-08-02 1:51 ` rsbecker 2024-08-02 22:07 ` rsbecker 0 siblings, 2 replies; 12+ messages in thread From: Taylor Blau @ 2024-08-02 0:58 UTC (permalink / raw) To: rsbecker; +Cc: git On Thu, Aug 01, 2024 at 06:25:51PM -0400, rsbecker@nexbridge.com wrote: > ls output with second resolution is here - the file system does not have nanosecond resolution despite showing zeros: > > /home/ituglib/randall/git/t/trash directory.t7704-repack-cruft/max-cruft-size-prune/.git/objects/pack: ls -la --full-time > total 11 > drwxrwxrwx 1 ITUGLIB.RANDALL ITUGLIB 4096 2024-08-01 16:18:55.000000000 -0600 . > drwxrwxrwx 1 ITUGLIB.RANDALL ITUGLIB 4096 2024-08-01 16:18:48.000000000 -0600 .. > -r--r--r-- 1 ITUGLIB.RANDALL ITUGLIB 1156 2024-08-01 16:18:52.000000000 -0600 pack-68c6c8c8538900694c32380ac1484201c8b60d8d.idx > -r--r--r-- 1 ITUGLIB.RANDALL ITUGLIB 217 2024-08-01 16:18:52.000000000 -0600 pack-68c6c8c8538900694c32380ac1484201c8b60d8d.pack > -r--r--r-- 1 ITUGLIB.RANDALL ITUGLIB 64 2024-08-01 16:18:52.000000000 -0600 pack-68c6c8c8538900694c32380ac1484201c8b60d8d.rev Ah, I suspect that this is even less interesting than imprecise mtime resolution. The test expects that the packs are larger than 1M so that we can exercise writing multiple cruft packs as part of the setup. But that pack is only 217 bytes, which wouldn't trigger the split. I'm suspicious that it's even packing the cruft objects at all, so I'm curious if you can find $foo, $bar, and $baz in the cruft pack's .idx file(s, if multiple) after the first 'git repack -d --cruft'. Assuming they are there, it's possible that setting repack.cruftWindow in the test repository would do the trick. But I'm suspicious that that's even what's going on since generate_random_blob()'s first argument is a seed, and all three objects have different seeds, so I don't think they would even be considered good delta candidates for one another. But it's also possible that your generate_random_blob() behaves differently from my own. Thanks, Taylor ^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: [BUG] 2.46.0 t7701.09 fails on NonStop ia64 2024-08-02 0:58 ` Taylor Blau @ 2024-08-02 1:51 ` rsbecker 2024-08-02 22:07 ` rsbecker 1 sibling, 0 replies; 12+ messages in thread From: rsbecker @ 2024-08-02 1:51 UTC (permalink / raw) To: 'Taylor Blau'; +Cc: git On Thursday, August 1, 2024 8:59 PM, Taylor Blau wrote: >On Thu, Aug 01, 2024 at 06:25:51PM -0400, rsbecker@nexbridge.com wrote: >> ls output with second resolution is here - the file system does not have >nanosecond resolution despite showing zeros: >> >> /home/ituglib/randall/git/t/trash >> directory.t7704-repack-cruft/max-cruft-size-prune/.git/objects/pack: >> ls -la --full-time total 11 drwxrwxrwx 1 ITUGLIB.RANDALL ITUGLIB 4096 2024-08- >01 16:18:55.000000000 -0600 . >> drwxrwxrwx 1 ITUGLIB.RANDALL ITUGLIB 4096 2024-08-01 >16:18:48.000000000 -0600 .. >> -r--r--r-- 1 ITUGLIB.RANDALL ITUGLIB 1156 2024-08-01 >> 16:18:52.000000000 -0600 >> pack-68c6c8c8538900694c32380ac1484201c8b60d8d.idx >> -r--r--r-- 1 ITUGLIB.RANDALL ITUGLIB 217 2024-08-01 16:18:52.000000000 - >0600 pack-68c6c8c8538900694c32380ac1484201c8b60d8d.pack >> -r--r--r-- 1 ITUGLIB.RANDALL ITUGLIB 64 2024-08-01 16:18:52.000000000 - >0600 pack-68c6c8c8538900694c32380ac1484201c8b60d8d.rev > >Ah, I suspect that this is even less interesting than imprecise mtime resolution. The >test expects that the packs are larger than 1M so that we can exercise writing >multiple cruft packs as part of the setup. > >But that pack is only 217 bytes, which wouldn't trigger the split. I'm suspicious that >it's even packing the cruft objects at all, so I'm curious if you can find $foo, $bar, >and $baz in the cruft pack's .idx file(s, if multiple) after the first 'git repack -d --cruft'. > >Assuming they are there, it's possible that setting repack.cruftWindow in the test >repository would do the trick. > >But I'm suspicious that that's even what's going on since generate_random_blob()'s >first argument is a seed, and all three objects have different seeds, so I don't think >they would even be considered good delta candidates for one another. But it's also >possible that your >generate_random_blob() behaves differently from my own. I will try to look tomorrow for $foo, $bar, and $baz. On generate_random_blob(), from OpenSSL's standpoint, on ia64, we use PRNGD. It should be of the same order of magnitude as anywhere else, but depends on the randomness measure - we are able to start it, so randomness is proper. On my x86 machine, I use the hardware randomizer, which is better (and FIPS compatible) and might be why we don't see it there. ^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: [BUG] 2.46.0 t7701.09 fails on NonStop ia64 2024-08-02 0:58 ` Taylor Blau 2024-08-02 1:51 ` rsbecker @ 2024-08-02 22:07 ` rsbecker 2024-08-04 16:02 ` Taylor Blau 1 sibling, 1 reply; 12+ messages in thread From: rsbecker @ 2024-08-02 22:07 UTC (permalink / raw) To: 'Taylor Blau'; +Cc: git On Thursday, August 1, 2024 8:59 PM, Taylor Blau wrote: >On Thu, Aug 01, 2024 at 06:25:51PM -0400, rsbecker@nexbridge.com wrote: >> ls output with second resolution is here - the file system does not have >nanosecond resolution despite showing zeros: >> >> /home/ituglib/randall/git/t/trash >> directory.t7704-repack-cruft/max-cruft-size-prune/.git/objects/pack: >> ls -la --full-time total 11 drwxrwxrwx 1 ITUGLIB.RANDALL ITUGLIB 4096 2024-08- >01 16:18:55.000000000 -0600 . >> drwxrwxrwx 1 ITUGLIB.RANDALL ITUGLIB 4096 2024-08-01 >16:18:48.000000000 -0600 .. >> -r--r--r-- 1 ITUGLIB.RANDALL ITUGLIB 1156 2024-08-01 >> 16:18:52.000000000 -0600 >> pack-68c6c8c8538900694c32380ac1484201c8b60d8d.idx >> -r--r--r-- 1 ITUGLIB.RANDALL ITUGLIB 217 2024-08-01 16:18:52.000000000 - >0600 pack-68c6c8c8538900694c32380ac1484201c8b60d8d.pack >> -r--r--r-- 1 ITUGLIB.RANDALL ITUGLIB 64 2024-08-01 16:18:52.000000000 - >0600 pack-68c6c8c8538900694c32380ac1484201c8b60d8d.rev > >Ah, I suspect that this is even less interesting than imprecise mtime resolution. The >test expects that the packs are larger than 1M so that we can exercise writing >multiple cruft packs as part of the setup. > >But that pack is only 217 bytes, which wouldn't trigger the split. I'm suspicious that >it's even packing the cruft objects at all, so I'm curious if you can find $foo, $bar, >and $baz in the cruft pack's .idx file(s, if multiple) after the first 'git repack -d --cruft'. > >Assuming they are there, it's possible that setting repack.cruftWindow in the test >repository would do the trick. > >But I'm suspicious that that's even what's going on since generate_random_blob()'s >first argument is a seed, and all three objects have different seeds, so I don't think >they would even be considered good delta candidates for one another. But it's also >possible that your >generate_random_blob() behaves differently from my own. The entire final .idx file is: 0000000 377 t O c \0 \0 \0 002 \0 \0 \0 \0 \0 \0 \0 \0 0000020 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0001500 \0 \0 \0 001 \0 \0 \0 001 \0 \0 \0 001 \0 \0 \0 002 0001520 \0 \0 \0 002 \0 \0 \0 002 \0 \0 \0 002 \0 \0 \0 002 * 0001600 \0 \0 \0 002 \0 \0 \0 003 \0 \0 \0 003 \0 \0 \0 003 0001620 \0 \0 \0 003 \0 \0 \0 003 \0 \0 \0 003 \0 \0 \0 003 * 0002000 \0 \0 \0 003 \0 \0 \0 003 316 a 1 002 , : H 241 0002020 _ K z h 257 266 354 363 263 274 260 315 321 377 034 222 0002040 $ 256 ^ X 247 e o 271 354 311 X e 324 . 327 036 0002060 337 226 { 226 245 y 344 Z 030 270 % 027 2 321 h 004 0002100 262 345 j U 003 302 375 225 d 335 = 272 207 227 ) 265 0002120 \0 \0 \0 212 \0 \0 \0 \f \0 \0 \0 267 h 306 310 310 0002140 S 211 \0 i L 2 8 \n 301 H B 001 310 266 \r 215 0002160 273 y 312 253 265 V 225 350 246 212 h Z d ? 313 f 0002200 372 Z 336 3 0002204 After the first repack, I have the following idx files. No foo/bar/baz inside. The generate_random_blob() does generate the proper amount of bytes. I tried changing 0xff to 0x00ff at the putchar just in case we had bad sign extension - that wasn't it. pack-68c6c8c8538900694c32380ac1484201c8b60d8d.idx: 0000000 377 t O c \0 \0 \0 002 \0 \0 \0 \0 \0 \0 \0 \0 0000020 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0001500 \0 \0 \0 001 \0 \0 \0 001 \0 \0 \0 001 \0 \0 \0 002 0001520 \0 \0 \0 002 \0 \0 \0 002 \0 \0 \0 002 \0 \0 \0 002 * 0001600 \0 \0 \0 002 \0 \0 \0 003 \0 \0 \0 003 \0 \0 \0 003 0001620 \0 \0 \0 003 \0 \0 \0 003 \0 \0 \0 003 \0 \0 \0 003 * 0002000 \0 \0 \0 003 \0 \0 \0 003 316 a 1 002 , : H 241 0002020 _ K z h 257 266 354 363 263 274 260 315 321 377 034 222 0002040 $ 256 ^ X 247 e o 271 354 311 X e 324 . 327 036 0002060 337 226 { 226 245 y 344 Z 030 270 % 027 2 321 h 004 0002100 262 345 j U 003 302 375 225 d 335 = 272 207 227 ) 265 0002120 \0 \0 \0 212 \0 \0 \0 \f \0 \0 \0 267 h 306 310 310 0002140 S 211 \0 i L 2 8 \n 301 H B 001 310 266 \r 215 0002160 273 y 312 253 265 V 225 350 246 212 h Z d ? 313 f 0002200 372 Z 336 3 0002204 pack-8f53f837597ebbdfc6f8ed173349e5cecbcfb97e.idx: 0000000 377 t O c \0 \0 \0 002 \0 \0 \0 \0 \0 \0 \0 \0 0000020 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0001260 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 001 0001300 \0 \0 \0 001 \0 \0 \0 001 \0 \0 \0 001 \0 \0 \0 001 * 0002000 \0 \0 \0 001 \0 \0 \0 001 255 026 276 325 L 306 034 360 0002020 6 \a X y 336 357 371 Z 351 w Q K # 236 346 377 0002040 \0 \0 \0 \f 217 S 370 7 Y ~ 273 337 306 370 355 027 0002060 3 I 345 316 313 317 271 ~ 9 033 9 316 363 306 G - 0002100 207 f 224 323 1 372 021 W 037 335 206 372 0002114 pack-b8dc9aadaadc12c82b0053fdee0039ae1025a8f8.idx: 0000000 377 t O c \0 \0 \0 002 \0 \0 \0 \0 \0 \0 \0 \0 0000020 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0000500 \0 \0 \0 \0 \0 \0 \0 001 \0 \0 \0 001 \0 \0 \0 001 0000520 \0 \0 \0 001 \0 \0 \0 001 \0 \0 \0 001 \0 \0 \0 001 * 0002000 \0 \0 \0 001 \0 \0 \0 001 O 3 o = 363 020 T 352 0002020 274 005 254 005 371 213 300 $ 310 340 T # $ 274 l 342 0002040 \0 \0 \0 \f 270 334 232 255 252 334 022 310 + \0 S 375 0002060 356 \0 9 256 020 % 250 370 204 242 7 033 = 0 375 207 0002100 343 352 257 374 265 021 j 214 5 026 355 243 0002114 pack-c2357b2b204fda52bc1f5515de94227e1db012af.idx: 0000000 377 t O c \0 \0 \0 002 \0 \0 \0 \0 \0 \0 \0 \0 0000020 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0001360 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 001 0001400 \0 \0 \0 001 \0 \0 \0 001 \0 \0 \0 001 \0 \0 \0 001 * 0002000 \0 \0 \0 001 \0 \0 \0 001 275 D 330 252 - " 353 G 0002020 377 p 316 364 260 273 E 321 T 236 344 234 302 311 8 h 0002040 \0 \0 \0 \f 302 5 { + O 332 R 274 037 U 025 0002060 336 224 " ~ 035 260 022 257 373 U ) - 204 " 315 v 0002100 / E 272 e 7 342 261 037 6 255 < 306 0002114 --Randall ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [BUG] 2.46.0 t7701.09 fails on NonStop ia64 2024-08-02 22:07 ` rsbecker @ 2024-08-04 16:02 ` Taylor Blau 2024-08-04 16:15 ` rsbecker 0 siblings, 1 reply; 12+ messages in thread From: Taylor Blau @ 2024-08-04 16:02 UTC (permalink / raw) To: rsbecker; +Cc: git On Fri, Aug 02, 2024 at 06:07:55PM -0400, rsbecker@nexbridge.com wrote: > After the first repack, I have the following idx files. No foo/bar/baz inside. > The generate_random_blob() does generate the proper amount of bytes. > I tried changing 0xff to 0x00ff at the putchar just in case we had bad > sign extension - that wasn't it. Hmm. What is in these index files if not the three randomly generated objects? Can you run show-index instead and share the results of that on the pack(s) in your repository after the first repack, and share the results with the list? I think those would be more readable than a hexdump, especially if your machine random source is different than mine (in which case I can't guess the values of $foo, $bar, and $baz). Thanks, Taylor ^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: [BUG] 2.46.0 t7701.09 fails on NonStop ia64 2024-08-04 16:02 ` Taylor Blau @ 2024-08-04 16:15 ` rsbecker 2024-08-04 16:19 ` Taylor Blau 0 siblings, 1 reply; 12+ messages in thread From: rsbecker @ 2024-08-04 16:15 UTC (permalink / raw) To: 'Taylor Blau'; +Cc: git >-----Original Message----- >From: Taylor Blau <me@ttaylorr.com> >Sent: Sunday, August 4, 2024 12:03 PM >To: rsbecker@nexbridge.com >Cc: git@vger.kernel.org >Subject: Re: [BUG] 2.46.0 t7701.09 fails on NonStop ia64 > >On Fri, Aug 02, 2024 at 06:07:55PM -0400, rsbecker@nexbridge.com wrote: >> After the first repack, I have the following idx files. No foo/bar/baz inside. >> The generate_random_blob() does generate the proper amount of bytes. >> I tried changing 0xff to 0x00ff at the putchar just in case we had bad >> sign extension - that wasn't it. > >Hmm. What is in these index files if not the three randomly generated objects? Can >you run show-index instead and share the results of that on the pack(s) in your >repository after the first repack, and share the results with the list? > >I think those would be more readable than a hexdump, especially if your machine >random source is different than mine (in which case I can't guess the values of $foo, >$bar, and $baz). After git repack -d --cruft --max-cruft-size=1M &&... ./trash directory.t7704-repack-cruft/max-cruft-size-prune: git show-index < .git/objects/pack/pack-68c6c8c8538900694c32380ac1484201c8b60d8d.pack fatal: unable to read index ./trash directory.t7704-repack-cruft/max-cruft-size-prune: git show-index < .git/objects/pack/pack-8f53f837597ebbdfc6f8ed173349e5cecbcfb97e.pack fatal: corrupt index file ./trash directory.t7704-repack-cruft/max-cruft-size-prune: git show-index < .git/objects/pack/pack-b8dc9aadaadc12c82b0053fdee0039ae1025a8f8.pack fatal: corrupt index file ./trash directory.t7704-repack-cruft/max-cruft-size-prune: git show-index < .git/objects/pack/pack-c2357b2b204fda52bc1f5515de94227e1db012af.pack fatal: corrupt index file Apparently, something is wrong. This is an ia64 Big Endian. Not sure that matters. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [BUG] 2.46.0 t7701.09 fails on NonStop ia64 2024-08-04 16:15 ` rsbecker @ 2024-08-04 16:19 ` Taylor Blau 2024-08-04 16:38 ` rsbecker 0 siblings, 1 reply; 12+ messages in thread From: Taylor Blau @ 2024-08-04 16:19 UTC (permalink / raw) To: rsbecker; +Cc: git On Sun, Aug 04, 2024 at 12:15:41PM -0400, rsbecker@nexbridge.com wrote: > After git repack -d --cruft --max-cruft-size=1M &&... > > ./trash directory.t7704-repack-cruft/max-cruft-size-prune: git show-index < .git/objects/pack/pack-68c6c8c8538900694c32380ac1484201c8b60d8d.pack > fatal: unable to read index > ./trash directory.t7704-repack-cruft/max-cruft-size-prune: git show-index < .git/objects/pack/pack-8f53f837597ebbdfc6f8ed173349e5cecbcfb97e.pack > fatal: corrupt index file > ./trash directory.t7704-repack-cruft/max-cruft-size-prune: git show-index < .git/objects/pack/pack-b8dc9aadaadc12c82b0053fdee0039ae1025a8f8.pack > fatal: corrupt index file > ./trash directory.t7704-repack-cruft/max-cruft-size-prune: git show-index < .git/objects/pack/pack-c2357b2b204fda52bc1f5515de94227e1db012af.pack > fatal: corrupt index file > > Apparently, something is wrong. This is an ia64 Big Endian. Not sure that matters. show-index expects the contents of the *.idx file on stdin, not the *.pack. Thanks, Taylor ^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: [BUG] 2.46.0 t7701.09 fails on NonStop ia64 2024-08-04 16:19 ` Taylor Blau @ 2024-08-04 16:38 ` rsbecker 2024-08-04 18:12 ` Taylor Blau 0 siblings, 1 reply; 12+ messages in thread From: rsbecker @ 2024-08-04 16:38 UTC (permalink / raw) To: 'Taylor Blau'; +Cc: git On Sunday, August 4, 2024 12:20 PM, Taylor Blau wrote: >On Sun, Aug 04, 2024 at 12:15:41PM -0400, rsbecker@nexbridge.com wrote: >> After git repack -d --cruft --max-cruft-size=1M &&... >> >> ./trash directory.t7704-repack-cruft/max-cruft-size-prune: git show-index < >.git/objects/pack/pack-68c6c8c8538900694c32380ac1484201c8b60d8d.pack >> fatal: unable to read index >> ./trash directory.t7704-repack-cruft/max-cruft-size-prune: git show-index < >.git/objects/pack/pack-8f53f837597ebbdfc6f8ed173349e5cecbcfb97e.pack >> fatal: corrupt index file >> ./trash directory.t7704-repack-cruft/max-cruft-size-prune: git show-index < >.git/objects/pack/pack-b8dc9aadaadc12c82b0053fdee0039ae1025a8f8.pack >> fatal: corrupt index file >> ./trash directory.t7704-repack-cruft/max-cruft-size-prune: git show-index < >.git/objects/pack/pack-c2357b2b204fda52bc1f5515de94227e1db012af.pack >> fatal: corrupt index file >> >> Apparently, something is wrong. This is an ia64 Big Endian. Not sure that matters. > >show-index expects the contents of the *.idx file on stdin, not the *.pack. This is what I get for being 1/2 asleep on Sunday. ./trash directory.t7704-repack-cruft/max-cruft-size-prune: git show-index < .git/objects/pack/pack-68c6c8c8538900694c32380ac1484201c8b60d8d.idx 138 ce6131022c3a48a15f4b7a68afb6ecf3b3bcb0cd (03c2fd95) 12 d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e (64dd3dba) 183 df967b96a579e45a18b8251732d16804b2e56a55 (879729b5) ./trash directory.t7704-repack-cruft/max-cruft-size-prune: git show-index <.git/objects/pack/pack-8f53f837597ebbdfc6f8ed173349e5cecbcfb97e.idx 12 ad16bed54cc61cf036075879deeff95ae977514b (239ee6ff) ./trash directory.t7704-repack-cruft/max-cruft-size-prune: git show-index <.git/objects/pack/pack-b8dc9aadaadc12c82b0053fdee0039ae1025a8f8.idx 12 4f336f3df31054eabc05ac05f98bc024c8e05423 (24bc6ce2) ./trash directory.t7704-repack-cruft/max-cruft-size-prune: git show-index <.git/objects/pack/pack-c2357b2b204fda52bc1f5515de94227e1db012af.idx 12 bd44d8aa2d22eb47ff70cef4b0bb45d1549ee49c (c2c93868) ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [BUG] 2.46.0 t7701.09 fails on NonStop ia64 2024-08-04 16:38 ` rsbecker @ 2024-08-04 18:12 ` Taylor Blau 2024-08-04 19:15 ` rsbecker 0 siblings, 1 reply; 12+ messages in thread From: Taylor Blau @ 2024-08-04 18:12 UTC (permalink / raw) To: rsbecker; +Cc: git On Sun, Aug 04, 2024 at 12:38:38PM -0400, rsbecker@nexbridge.com wrote: > ./trash directory.t7704-repack-cruft/max-cruft-size-prune: git show-index < .git/objects/pack/pack-68c6c8c8538900694c32380ac1484201c8b60d8d.idx > 138 ce6131022c3a48a15f4b7a68afb6ecf3b3bcb0cd (03c2fd95) > 12 d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e (64dd3dba) > 183 df967b96a579e45a18b8251732d16804b2e56a55 (879729b5) > ./trash directory.t7704-repack-cruft/max-cruft-size-prune: git show-index <.git/objects/pack/pack-8f53f837597ebbdfc6f8ed173349e5cecbcfb97e.idx > 12 ad16bed54cc61cf036075879deeff95ae977514b (239ee6ff) > ./trash directory.t7704-repack-cruft/max-cruft-size-prune: git show-index <.git/objects/pack/pack-b8dc9aadaadc12c82b0053fdee0039ae1025a8f8.idx > 12 4f336f3df31054eabc05ac05f98bc024c8e05423 (24bc6ce2) > ./trash directory.t7704-repack-cruft/max-cruft-size-prune: git show-index <.git/objects/pack/pack-c2357b2b204fda52bc1f5515de94227e1db012af.idx > 12 bd44d8aa2d22eb47ff70cef4b0bb45d1549ee49c (c2c93868) OK, so it looks like the first repack is doing what we expect: presumably the first pack containing three objects holds the result of "test_commit base" (and contains the blob, tree, and commit objects for commit "base"). Then I'm assuming that the remaining three packs hold $foo, $bar, and $baz in some unspecified order. So I think that everything up through the first repack is doing the right thing. (As an aside, it took me a while to remember what was going on with this test. Since it was confusing to me as its author, I figured it may be helpful to spell out some of the details. We're just making sure that pruning a cruft object works with multiple cruft packs, and we're adjusting the mtimes of the *.mtimes files themselves to ensure that they are rewritten during the pruning repack. Those adjustments have no bearing on what actually gets pruned, unlike when we update the mtime of the loose copy of $foo, which does affect pruning). Anyway... in your setup it looks like all three objects are gone, and we expected only $foo to be pruned. So I suspect what's going on is that this test takes longer than 10 seconds to run on your machine, in which case it would fail as all objects would have passed the 10-second grace period. If you apply: --- 8< --- diff --git a/t/t7704-repack-cruft.sh b/t/t7704-repack-cruft.sh index 71e1ef3a10..959e6e2648 100755 --- a/t/t7704-repack-cruft.sh +++ b/t/t7704-repack-cruft.sh @@ -330,7 +330,7 @@ test_expect_success '--max-cruft-size with pruning' ' # repack (and prune) with a --max-cruft-size to ensure # that we appropriately split the resulting set of packs git repack -d --cruft --max-cruft-size=1M \ - --cruft-expiration=10.seconds.ago && + --cruft-expiration=1000.seconds.ago && ls $packdir/pack-*.mtimes | sort >cruft.after && for cruft in $(cat cruft.after) --- >8 --- , does it still fail? (There's really no reason to have --cruft-expiration as short as 10 seconds here, since we backdate the cruft mtimes by 10,000 seconds.) Thanks, Taylor ^ permalink raw reply related [flat|nested] 12+ messages in thread
* RE: [BUG] 2.46.0 t7701.09 fails on NonStop ia64 2024-08-04 18:12 ` Taylor Blau @ 2024-08-04 19:15 ` rsbecker 0 siblings, 0 replies; 12+ messages in thread From: rsbecker @ 2024-08-04 19:15 UTC (permalink / raw) To: 'Taylor Blau'; +Cc: git On Sunday, August 4, 2024 2:13 PM, Taylor Blau wrote: >On Sun, Aug 04, 2024 at 12:38:38PM -0400, rsbecker@nexbridge.com wrote: >> ./trash directory.t7704-repack-cruft/max-cruft-size-prune: git >> show-index < >> .git/objects/pack/pack-68c6c8c8538900694c32380ac1484201c8b60d8d.idx >> 138 ce6131022c3a48a15f4b7a68afb6ecf3b3bcb0cd (03c2fd95) >> 12 d1ff1c9224ae5e58a7656fb9ecc95865d42ed71e (64dd3dba) >> 183 df967b96a579e45a18b8251732d16804b2e56a55 (879729b5) ./trash >> directory.t7704-repack-cruft/max-cruft-size-prune: git show-index >> <.git/objects/pack/pack-8f53f837597ebbdfc6f8ed173349e5cecbcfb97e.idx >> 12 ad16bed54cc61cf036075879deeff95ae977514b (239ee6ff) ./trash >> directory.t7704-repack-cruft/max-cruft-size-prune: git show-index >> <.git/objects/pack/pack-b8dc9aadaadc12c82b0053fdee0039ae1025a8f8.idx >> 12 4f336f3df31054eabc05ac05f98bc024c8e05423 (24bc6ce2) ./trash >> directory.t7704-repack-cruft/max-cruft-size-prune: git show-index >> <.git/objects/pack/pack-c2357b2b204fda52bc1f5515de94227e1db012af.idx >> 12 bd44d8aa2d22eb47ff70cef4b0bb45d1549ee49c (c2c93868) > >OK, so it looks like the first repack is doing what we expect: >presumably the first pack containing three objects holds the result of "test_commit >base" (and contains the blob, tree, and commit objects for commit "base"). Then >I'm assuming that the remaining three packs hold $foo, $bar, and $baz in some >unspecified order. > >So I think that everything up through the first repack is doing the right thing. > >(As an aside, it took me a while to remember what was going on with this test. Since >it was confusing to me as its author, I figured it may be helpful to spell out some of >the details. We're just making sure that pruning a cruft object works with multiple >cruft packs, and we're adjusting the mtimes of the *.mtimes files themselves to >ensure that they are rewritten during the pruning repack. Those adjustments have >no bearing on what actually gets pruned, unlike when we update the mtime of the >loose copy of $foo, which does affect pruning). > >Anyway... in your setup it looks like all three objects are gone, and we expected only >$foo to be pruned. So I suspect what's going on is that this test takes longer than 10 >seconds to run on your machine, in which case it would fail as all objects would >have passed the 10-second grace period. > >If you apply: > >--- 8< --- >diff --git a/t/t7704-repack-cruft.sh b/t/t7704-repack-cruft.sh index >71e1ef3a10..959e6e2648 100755 >--- a/t/t7704-repack-cruft.sh >+++ b/t/t7704-repack-cruft.sh >@@ -330,7 +330,7 @@ test_expect_success '--max-cruft-size with pruning' ' > # repack (and prune) with a --max-cruft-size to ensure > # that we appropriately split the resulting set of packs > git repack -d --cruft --max-cruft-size=1M \ >- --cruft-expiration=10.seconds.ago && >+ --cruft-expiration=1000.seconds.ago && > ls $packdir/pack-*.mtimes | sort >cruft.after && > > for cruft in $(cat cruft.after) >--- >8 --- > >, does it still fail? > >(There's really no reason to have --cruft-expiration as short as 10 seconds here, >since we backdate the cruft mtimes by 10,000 seconds.) This worked. Even taking it down to 100 seconds also works. I suspect that the reason this previously passed was that we have an unrelated heavy use looping process on the box right now so things are taking longer. Running verbose also takes longer. ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2024-08-04 19:15 UTC | newest] Thread overview: 12+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2024-08-01 18:03 [BUG] 2.46.0 t7701.09 fails on NonStop ia64 rsbecker 2024-08-01 19:23 ` Taylor Blau 2024-08-01 22:25 ` rsbecker 2024-08-02 0:58 ` Taylor Blau 2024-08-02 1:51 ` rsbecker 2024-08-02 22:07 ` rsbecker 2024-08-04 16:02 ` Taylor Blau 2024-08-04 16:15 ` rsbecker 2024-08-04 16:19 ` Taylor Blau 2024-08-04 16:38 ` rsbecker 2024-08-04 18:12 ` Taylor Blau 2024-08-04 19:15 ` rsbecker
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).