RE: scheduling oddity on 2.6.20.3 stock

["David Schwartz" <davids@webmaster.com>]

> On Thu, 3 May 2007, David Schwartz wrote:
>
> >> I needed to recompress some files from .bz2 to .gz so I setup
> a script to
> >> do
> >>
> >> bunzip2 -c $file.bz2 |gzip -9 >$file.gz
> >>
> >> I expected that the two CPU heavy processes would end up on different
> >> cpu's and spend a little time shuffling data between the two cpu's on a
> >> system (dual core opteron)
> >>
> >> however, instead what I find is that each process is getting 50% of one
> >> cpu while the other cpu is 97% idle.
> >
> > That would only be possible if the compression/decompression
> block size is
> > small compared to the maximum pipe buffer size. I suspect the
> reverse is the
> > case.
>
> I'm still running into this problem in various forms
>
> is there an easy way to change the maximum pipe buffer size? (including a
> simple change to the kernel source, I do compile my own kernels)

No. Changing the size will not do what you want it to do since that only
tells the kernel what the size is, it does not determine what it is.

> > It would be interesting to write an intermediate process that basically
> > enlarged the pipe buffers and see if that changed anything.
> > Basically, the
> > intermediate process would allocate a large buffer (16MB or so)
> > and fill it
> > from 'bunzip2' while draining it to 'gzip' in a non-blocking
> > way (unless the
> > buffer was full/empty, of course).

It is not particularly hard to write such a process. I have a proxy that I
can easily tweak to do this. I'm going to give it a shot and see if it
helps.

DS