pin files in memory after read

pin files in memory after read

[Olaf Hering]

Is there a way to always keep a file (once read from disk) in memory, no
matter how much memory pressure exists?
There are always complains that updatedb and similar tools wipe out all
caches. So I guess there is no such thing yet.

I simply want to avoid the spinup of my ibook harddisk when something
has been 'forgotten' and must be loaded again (like opening a new screen
window after a while).

The best I could do so far was a cramfs image. I copied it to tmpfs
during early boot, then mount -o bind every cramfs file over the real
binary on disk. Of course that will fail as soon as I want to update an
affected package because the binary is busy (readonly). So there must be
a better way to achieve this.

How can one tell the kernel to pin a file in memory once it was read?
Maybe with an xattr or something?
Unfortunately I dont know about the block layer and other things
involved, so I cant attach a patch that does what I want.

Re: pin files in memory after read

[Arjan van de Ven]

On Mon, 2005-01-03 at 19:07 +0100, Olaf Hering wrote:
> Is there a way to always keep a file (once read from disk) in memory, no
> matter how much memory pressure exists?
> There are always complains that updatedb and similar tools wipe out all
> caches. So I guess there is no such thing yet.
> 
> I simply want to avoid the spinup of my ibook harddisk when something
> has been 'forgotten' and must be loaded again (like opening a new screen
> window after a while).
> 
> The best I could do so far was a cramfs image. I copied it to tmpfs
> during early boot, then mount -o bind every cramfs file over the real
> binary on disk. Of course that will fail as soon as I want to update an
> affected package because the binary is busy (readonly). So there must be
> a better way to achieve this.
> 
> How can one tell the kernel to pin a file in memory once it was read?
> Maybe with an xattr or something?
> Unfortunately I dont know about the block layer and other things
> involved, so I cant attach a patch that does what I want.

you could write a small userspace daemon that mmaps the file and mlock's
it....

Re: pin files in memory after read

[Olaf Hering]

 On Mon, Jan 03, Arjan van de Ven wrote:

> you could write a small userspace daemon that mmaps the file and mlock's
> it....

Thanks.
It seems to work ok with this thing. I used this patch to find the files
with an absolute path. Any idea how to get to the relative path like
"./x" and print an absolute path for these files?

--- ../linux-2.6.10.orig/fs/open.c      2004-12-31 09:29:25.000000000 +0100
+++ ./fs/open.c 2005-01-04 00:48:30.000000000 +0100
@@ -961,6 +961,17 @@ asmlinkage long sys_open(const char __us
 out:
                putname(tmp);
        }
+       if (0 && fd >= 0) {
+               if (filename[0] == '/' && filename[1] != '\0' && !(
+                                       !memcmp(filename,"/home/olaf/Mail",15) ||
+                                       !memcmp(filename,"/events",7) ||
+                                       !memcmp(filename,"/proc",5) ||
+                                       !memcmp(filename,"/sys",4) ||
+                                       !memcmp(filename,"/dev",4) ||
+                                       !memcmp(filename,"/var",4)
+                                       ))
+                       printk("OP%s %s\n",current->comm, filename);
+       }
        return fd;
 
 out_error:



#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>

#define file_list "/home/olaf/x"

size_t total;
void map(unsigned char *file, struct stat *sb)
{
	int fd;
	register unsigned char *p, c;
	if (stat(file, sb) < 0)
		return;
	fd = open(file, O_RDONLY);
	if (fd < 0)
		return;
	p = mmap(NULL, sb->st_size, PROT_READ, MAP_SHARED | MAP_LOCKED, fd, 0);
	if (p != MAP_FAILED && (total += sb->st_size))
		while (sb->st_size)
			c = p[sb->st_size--];
	close(fd);
	return;
}

int main(int argc, char *argv[])
{
	struct stat sb;
	int fd, ret, line_count;
	size_t len;
	off_t fs;
	void *flp;
	unsigned char *p1, *p2;
	fd = open(file_list, O_RDONLY);
	if (fd < 0) {
		perror(file_list);
		ret = fd;
		goto out;
	}
	ret = stat(file_list, &sb);
	if (ret < 0)
		goto out;
	flp = mmap(NULL, sb.st_size, PROT_READ, MAP_SHARED | MAP_LOCKED, fd, 0);
	if (flp == MAP_FAILED) {
		perror("mmap");
		goto out;
	}
	total += sb.st_size;
	p1 = flp;
	line_count = 0;
	fs = sb.st_size;
	while (fs > 0) {
		line_count++;
		printf("line %d ", line_count);
		len = 0;
		while (1) {
			//              printf("len %d\n", len);
			fs--;
			if (p1[len] != '\n') {
				len++;
				continue;
			}
			p2 = malloc(len);
			if (p2) {
				memcpy(p2, p1, len);
				p2[len] = '\0';
				printf("%u %s\n", len, p2);
				map(p2, &sb);
				free(p2);
				p1 = p1 + len + 1;
			}
			break;
		}
	}
	printf("sleeping ... %u\n", total);
	while (1)sleep(123456789);
      out:
	return ret;
}

Re: pin files in memory after read

[Chris Wright]

* Olaf Hering (olh@suse.de) wrote:
> with an absolute path. Any idea how to get to the relative path like
> "./x" and print an absolute path for these files?

d_path() will give that to you, once you've resolved the path to dentry
and vfsmount pair.

thanks,
-chris
-- 
Linux Security Modules     http://lsm.immunix.org     http://lsm.bkbits.net