Re: [dm-crypt] Poor performane (idle cpu) [SOLVED; problem with "pv"]

[Arno Wagner <arno@wagner.name>]

[-- Attachment #1: Type: text/plain, Size: 2979 bytes --]

On Tue, Feb 09, 2010 at 03:48:29PM +0100, Jakob Sandgren wrote:
> On Tue, Feb 09, 2010 at 01:28:06AM +0100, Arno Wagner wrote:
> > On Tue, Feb 09, 2010 at 12:54:16AM +0100, Jakob Sandgren wrote:
> > > Hi,
> > > 
> > > (please keep me on CC since I'm not subscribed yet)
> > > 
> > > To me it seems like there is some serious flaw within kcryptd that
> > > ends up to wait for "something" instead of sending enough requests to
> > > the disks to make sure it has data to decrypt. What do you think?
> > 
> > The same thing. 
> > 
> > Here is a reference test (I have notebook disks in this server):
> > 
> >   Raw read: 54MB/s 14% CPU
> >   Read with decrypt: 53MB/s 65% CPU
> 
> For reference this is the exact output of my benchmark, maybe there
> are some difference in setup or benchmark? 
> 
> OOOOOooops! While putting toghether this information I actually found
> the cause of the problem, it was my benchark that was wrong!
> 
> This was the benchmark I used to get the performance was:
> dd if=/dev/mapper/bench1  bs=4M iflag=direct |pv | dd of=/dev/null
> and the number reported by "pv" during the run was ~75MB/s and dd
> reported the same number when finished.
> 
> Changing this to:
> dd if=/dev/mapper/bench1  bs=4M iflag=direct of=/dev/null count=1000
> gave a more correct number; 125MB/s
> 
> 
> I was not aware of that piping the data through pv would cause such a
> big degradation in performance.
> 

Ah, yes. I think it is historic and uses some things
that are slow.

I have my own tool (wcs for "wc-stream") that is a bit 
faster and provides real-time speeds on a vt100. It is 
really small. Sources are attached. 
 
> > That would mean the crypto is pretty slow on your new CPU.
> > As a reference, my 53MB/s at 65% CPU is on an 2800MHz Athlon 
> > 64 X2 5600+ with aes-cbc-plain. 
> > 
> > Here is an OpenSSL crypto speed test:
> >  openssl speed -evp aes-256-cbc
> >  [...]
> >  The 'numbers' are in 1000s of bytes per second processed.
> >  type           16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
> >  aes-256-cbc    71848.00k    98649.49k   110187.78k   113646.25k  114666.15k
> > 
> > You might want to compare this with the numbers on your CPU.
> 
> The numbers from my system (Core I7) are below
> 
> root@mvh:~#  openssl speed -evp aes-256-cbc
> ...
> The 'numbers' are in 1000s of bytes per second processed.
> type             16 bytes     64 bytes    256 bytes   1024 bytes  8192 bytes
> aes-256-cbc     110769.28k   118629.67k   120600.15k   121138.86k  121206.10k
> 
> I has now been able to get a 175MB/sec from my main raid partition.

Good.

Arno

-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

[-- Attachment #2: wcs.c --]
[-- Type: text/x-csrc, Size: 2396 bytes --]

/* "wc-follow": wc with incremental output every second or so. 
 * Line positioning is done like in dd_rescue with 
 * "optimistic positioning" i.e. the hope that we have a terminal that 
 * understands vt100 positioning.
 *
 * (C) Arno Wagner <arno@wagner.name> 2008. Distributed under
 * The GNU public license v2
 *
 * Version 1.0
 *
 * Compile with "gcc -O6 -o wcs wcs.c"
 */

#include <stdlib.h>
#include <stdio.h>
#include <errno.h>

const char* up = "\x1b[A"; //] 
const char* down = "\n";
const char* right = "\x1b[C"; //]

char * usage() {
  return("\n"
"  Prints out running count and rate statistics about the data\n"
"  read on stdin to stderr. Does use cursor positioning, which\n" 
"  should work on most terminals (vt100 or later).\n"
"\n"
"  stdin is copied through to stdout, much like in tee.\n"
" \n" 
"  This programm does not support any commandline arguments.\n"  
"\n"
"\n");
}




void printlong(long long l) {
      if (l < 1000000L) 
        fprintf(stderr, "%7.3f kB", l/1000.0);
      else if (l < 1000000000) 
        fprintf(stderr, "%7.3f MB", l/1000000.0);
      else if (l < 1000000000000LL) 
        fprintf(stderr, "%7.3f GB", l/1000000000.0);
      else 
        fprintf(stderr, "%7.3f TB", l/1000000000000.0);
}  

int main(int argc, char ** argv) {
  char buf[4096];
  long long cnt = 0;
  time_t to, tn, ts, dt;
  int read_count;
  double rate;

  if (argc > 1) {
    fprintf(stderr, "%s", usage());
    exit(1);
  }

  to = time(NULL);
  ts = to;
  fprintf(stderr, down);

  while (1) {
    read_count = read (0, buf, sizeof(buf));
    if (read_count < 0 && errno == EINTR)  continue; // not an error
    if (read_count < 0) {
      perror("Abnormal condition in read from stdin:");
      exit(1);
    }

    if (read_count > 0) {
      cnt += read_count;
      if (fwrite (buf, 1, read_count, stdout) != read_count) {
        perror("Error in wcs writing to stdout:");  
        exit(1);
      }
    }

    tn = time(NULL);
    if (tn != to || read_count == 0) {
      to = tn;
      fprintf(stderr, "%s read: ", up);
      printlong(cnt);
      fprintf(stderr, " [ %12Ld B]    avg: ", cnt);
        
      // Calculate the rate
      dt = tn - ts;
      rate = cnt / (double)dt;
      printlong((long long) rate);
      fprintf(stderr, "/sec [ %5d sec]%s", dt, down);
    }
    if (read_count == 0)  break;                     // we are done 
  }
}