Re: pack operation is thrashing my server

["Björn Steinbrink" <B.Steinbrink@gmx.de>]

[-- Attachment #1: Type: text/plain, Size: 996 bytes --]

On 2008.08.14 17:06:13 -0700, Linus Torvalds wrote:
> The "hash" we use for looking things up is also pretty much a joke, and it 
> has no overflow capability, it just replaces the old entry with a new one.

So I added some stupid tracing to cache_or_unpack entry to see how often
we reread the same stuff. The whole thing just logs the base_offset in
case of a cache miss. I've gc'ed my linux-2.6.git before the run, so
that there's only a single packed_git around (at least I hope so), and I
can ignore that for the tracing.

The whole log for a "git rev-list --objects HEAD" has about 1.2M
entries, while the output of the rev-list command has about 870k lines.
Some postprocessing of the trace shows that the majority of objects is
read only once or twice. A few percent are read three to ten times, and
some are read more than two hundred times.

I'll attach the post-processed thing. The format is:
 x y

Meaning that there were x base_offset values for which we had y cache
misses.

Björn

[-- Attachment #2: cache-misses --]
[-- Type: text/plain, Size: 2437 bytes --]

 391805 1
 110622 2
  27830 3
  13995 4
   8583 5
   5834 6
   4275 7
   3242 8
   2514 9
   2168 10
   1632 11
   1336 12
   1197 13
    947 14
    788 15
    704 16
    565 17
    514 18
    422 19
    348 20
    304 21
    276 22
    227 23
    233 24
    180 25
    160 26
    145 27
    106 28
    123 29
    109 30
     86 31
     91 32
     72 33
     63 34
     55 35
     73 36
     61 37
     56 38
     48 39
     44 40
     47 41
     36 42
     44 43
     47 44
     32 45
     36 46
     27 47
     19 48
     34 49
     28 50
     22 51
     21 52
     26 53
     18 54
     19 55
     16 56
     22 57
     16 58
     16 59
     11 60
     13 61
     19 62
     17 63
      8 64
     21 65
      8 66
      8 67
     16 68
      9 69
     12 70
     11 71
      8 72
      5 73
      6 74
      9 75
      6 76
      9 77
      7 78
      8 79
      7 80
      8 81
      6 82
      5 83
     13 84
      9 85
      8 86
      4 87
      5 89
      6 90
      3 91
      7 92
      4 93
      5 94
      5 95
      5 96
      4 97
      3 98
      7 99
      2 100
      4 101
      4 102
      7 103
      4 104
      4 105
      5 106
      3 107
      1 108
      4 109
      1 110
      1 111
      1 112
      6 113
      5 114
      2 115
      5 116
      2 117
      2 118
      2 119
      7 120
      1 121
      4 122
      3 123
      3 124
      3 125
      4 126
      1 127
      2 128
      2 129
      2 130
      1 131
      4 132
      1 133
      4 134
      1 135
      2 136
      4 137
      1 139
      3 140
      3 141
      5 142
      5 143
      4 144
      1 148
      2 149
      3 150
      1 151
      2 152
      6 153
      1 154
      2 155
      2 156
      3 157
      2 158
      1 159
      3 160
      2 161
      4 162
      2 163
      5 164
      2 165
      2 166
      2 169
      2 170
      1 171
      1 172
      1 173
      1 176
      2 177
      2 178
      2 179
      1 180
      3 181
      3 182
      1 183
      1 184
      1 186
      1 187
      1 190
      1 192
      1 194
      2 195
      3 196
      1 197
      1 200
      1 201
      1 202
      1 208
      2 214
      2 216
      2 217
      3 224
      1 225
      1 228
      1 230
      2 232
      2 233
      1 234
      2 236
      1 239
      1 241
      1 245
      1 246
      2 249
      2 250
      1 252
      1 259
      1 261
      2 263
      1 266
      2 268
      1 272
      1 282