Re: [dm-crypt] [BUG] bad performance and system stalls when using dm-crypt

[Heinz Diehl <htd@fancy-poultry.org>]

On 11.09.2010, dave b wrote: 

> A simple test is just to dd if=/dev/zero of=DELETEME for a short time
> and the system will stall rather a lot - it is not a complete lock up
> - just the entire system is unresponsive for large periods at a time
> (from around 30seconds - to 2 minutes).

This is most likely due to the massive disk i/o which is generated by the
dd command.

> This may be related to
> http://thread.gmane.org/gmane.linux.kernel.mm/51444 .

I think this is completely unrelated, it's more a kind of a disk scheduler
issue. I can't compare directly, because I'm running XFS on all of my
machines, but you could try to fine-tune your disk scheduler. In any case,
writing a big file and work on the same disk at the same time will always
give you, hmmm.. "some delay", even without using encryption.

Ok, let's give it a short run on one of my machines,
a 10 GB file zeroed out using the dd command you mentioned, on /home,
encrypted with dmcrypt/LUKS, and running Theodore Tso's fsync-tester
on the same partition:

htd@liesel:~/fs> ./fsync-tester
fsync time: 0.5354
fsync time: 0.1496
fsync time: 0.2655
fsync time: 5.1740
fsync time: 6.0126
fsync time: 3.7945
fsync time: 19.0478
fsync time: 11.2006
fsync time: 4.5590
fsync time: 0.1663
fsync time: 0.3466
fsync time: 2.2969
fsync time: 0.7246
fsync time: 0.3036
fsync time: 6.2057
fsync time: 0.1042
fsync time: 0.7644
fsync time: 0.9193
fsync time: 0.1738
fsync time: 0.7413
fsync time: 18.2807
fsync time: 1.1971
fsync time: 0.2808
fsync time: 0.2364
fsync time: 0.3220
fsync time: 0.2108
fsync time: 0.2831
fsync time: 2.6023
fsync time: 14.0767
fsync time: 23.4450
fsync time: 3.2213
fsync time: 4.3488
fsync time: 12.5578
fsync time: 0.2359
^C
htd@liesel:~/fs> 

You can see stalls up to 23 secs here, too. I'm using the latest kernel
2.6.36-rc3 from Linus' git repository, with the "global workqueue per cpu"
patch from Andi Kleen on top of it (which should not give any performance
boost here in this case). Scheduler is cfq, tuned this way:

echo cfq > /sys/block/sda/queue/scheduler
echo "1" > /sys/block/sda/queue/iosched/low_latency
echo "8" > /sys/block/sda/queue/iosched/slice_idle
echo "8" > /sys/block/sda/queue/iosched/quantum

liesel:/home/htd/fs # xfs_repair -V
xfs_repair version 2.10.1

mount: /dev/mapper/home on /home type xfs (rw,noatime,logbsize=256k,logbufs=2,nobarrier,inode64)


/*
 * fsync-tester.c
 *
 * Written by Theodore Ts'o, 3/21/09.
 *
 * This file may be redistributed under the terms of the GNU Public
 * License, version 2.
 */

#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <time.h>
#include <fcntl.h>
#include <string.h>

#define SIZE (32768*32)

static float timeval_subtract(struct timeval *tv1, struct timeval *tv2)
{
	return ((tv1->tv_sec - tv2->tv_sec) +
		((float) (tv1->tv_usec - tv2->tv_usec)) / 1000000);
}

int main(int argc, char **argv)
{
	int	fd;
	struct timeval tv, tv2;
	char buf[SIZE];

	fd = open("fsync-tester.tst-file", O_WRONLY|O_CREAT);
	if (fd < 0) {
		perror("open");
		exit(1);
	}
	memset(buf, 'a', SIZE);
	while (1) {
		pwrite(fd, buf, SIZE, 0);
		gettimeofday(&tv, NULL);
		fsync(fd);
		gettimeofday(&tv2, NULL);
		printf("fsync time: %5.4f\n", timeval_subtract(&tv2, &tv));
		sleep(1);
	}
}