Re: Linux 2.6.29

Linus Torvalds
2009-04-02T17:01:48Z

On Thu, 2 Apr 2009, Andrew Morton wrote:
>
> A suitable design for the streaming might be, every 4MB:
>
> - run sync_file_range(SYNC_FILE_RANGE_WRITE) to get the 4MB underway
> to the disk
>
> - run fadvise(POSIX_FADV_DONTNEED) against the previous 4MB to
> discard it from pagecache.

Here's an example. I call it "overwrite.c" for obvious reasons.

Except I used 8MB ranges, and I "stream" random data. Very useful for
"secure delete" of harddisks. It gives pretty optimal speed, while not
destroying your system experience.

Of course, I do think the kernel could/should do this kind of thing
automatically. We really could do something like this with a "dirty LRU"
queue. Make the logic be:

- if you have more than "2*limit" pages in your dirty LRU queue, start
writeout on "limit" pages (default value: 8MB, tunable in /proc).
Remove from LRU queues.

- On writeback IO completion, if it's not on any LRU list, insert page
into "done_write" LRU list.

- if you have more than "2*limit" pages on the done_write LRU queue,
try to just get rid of the first "limit" pages.

It would probably work fine in general. Temp-files (smaller than 8MB
total) would go into the dirty LRU queue, but wouldn't be written out to
disk if they get deleted before you've generated 8MB of dirty data.

But this does the queue-handling by hand, and gives you a throughput
indicator. It should get fairly close to disk speeds.

Linus

---
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <errno.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <sys/time.h>
#include <linux/fs.h>

#define BUFSIZE (8*1024*1024ul)

int main(int argc, char **argv)
{
static char buffer[BUFSIZE];
struct timeval start, now;
unsigned int index;
int fd;

mlockall(MCL_CURRENT | MCL_FUTURE);
fd = open("/dev/urandom", O_RDONLY);
if (read(fd, buffer, BUFSIZE) != BUFSIZE) {
perror("/dev/urandom");
exit(1);
}
close(fd);
fd = open(argv[1], O_RDWR | O_CREAT, 0666);
if (fd < 0) {
perror(argv[1]);
exit(1);
}
gettimeofday(&start, NULL);
for (index = 0; ;index++) {
double s;
unsigned long MBps;
unsigned long MB;

if (write(fd, buffer, BUFSIZE) != BUFSIZE)
break;
sync_file_range(fd, index*BUFSIZE, BUFSIZE, SYNC_FILE_RANGE_WRITE);
if (index)
sync_file_range(fd, (index-1)*BUFSIZE, BUFSIZE, SYNC_FILE_RANGE_WAIT_BEFORE|SYNC_FILE_RANGE_WRITE|SYNC_FILE_RANGE_WAIT_AFTER);
gettimeofday(&now, NULL);
s = (now.tv_sec - start.tv_sec) + ((double) now.tv_usec - start.tv_usec)/ 1000000;

MB = index * (BUFSIZE >> 20);
MBps = MB;
if (s > 1)
MBps = MBps / s;
printf("%8lu.%03lu GB written in %5.2f (%lu MB/s) \r",
MB >> 10, (MB & 1023) * 1000 >> 10, s, MBps);
fflush(stdout);
}
close(fd);
printf("\n");
return 0;
}