All of lore.kernel.org
 help / color / mirror / Atom feed
* large files
@ 2004-05-17 19:48 Bernd Schubert
  2004-05-17 20:12 ` Chris Mason
  0 siblings, 1 reply; 19+ messages in thread
From: Bernd Schubert @ 2004-05-17 19:48 UTC (permalink / raw)
  To: Reiserfs mail-list

[-- Attachment #1: signed data --]
[-- Type: text/plain, Size: 983 bytes --]

Hello,

I'm currently testing our new server and though it will primarily not serve 
really large files (about 40-60 users will have a quota of 25GB each on a 2TB 
array), I'm still testing the performance for large files.

So I created an about 300GB fil and the problem is to remove it now. 
Removing it took much more than 15 minutes. Here's the the relevant top line:

 5012 root      18   0   368  368   312 D    21.9  0.0   5:48 rm

Since I didn't expect it to take so much time, I didn't measure the time to 
delete this file.

system specifications:
	- dual opteron 242 (1600 MHz)
	- linux-2.4.26 with all patches from Chris, no further patches
	- reiserfs-3.6 format

The partition with the 300GB file has a size of 1.7TB.


Any ideas whats going on? 


Thanks,
	Bernd


-- 
Bernd Schubert
Physikalisch Chemisches Institut / Theoretische Chemie
Universität Heidelberg
INF 229
69120 Heidelberg
e-mail: bernd.schubert@pci.uni-heidelberg.de

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

^ permalink raw reply	[flat|nested] 19+ messages in thread
* Re: Large files
@ 2003-06-10 22:38 Ray Lee
  0 siblings, 0 replies; 19+ messages in thread
From: Ray Lee @ 2003-06-10 22:38 UTC (permalink / raw)
  To: root, Linux Kernel

[-- Attachment #1: Type: text/plain, Size: 1286 bytes --]

> With 32 bit return values, ix86 Linux has a file-size limitation 
> which is currently about 0x7fffffff. Unfortunately, instead of 
> returning from a write() with a -1 and errno being set, so that 
> a program can do something about it, write() executes a signal(25) 
> which kills the task even if trapped

Works For Me(tm)

        ray:~/work/test/signals$ ls
        sig.c
        ray:~/work/test/signals$ tcc -run sig.c
        write errored
        seem to have gone overboard, switching to next log file...
        write errored
        seem to have gone overboard, switching to next log file...
        write errored
        seem to have gone overboard, switching to next log file...
        ray:~/work/test/signals$ ls -l
        total 4
        -rw-------    1 ray      ray      2147483647 Jun 10 15:35 log.0
        -rw-------    1 ray      ray      2147483647 Jun 10 15:35 log.1
        -rw-------    1 ray      ray      2147483647 Jun 10 15:35 log.2
        -rw-------    1 ray      ray           259 Jun 10 15:35 log.3
        -rw-r--r--    1 ray      ray          2119 Jun 10 15:33 sig.c
        ray:~/work/test/signals$
        
Test code attached. Please excuse the somewhat haphazard structure, it
was tossed together from code I'd written for other projects.

Ray


[-- Attachment #2: sig.c --]
[-- Type: text/x-c, Size: 2119 bytes --]

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>
#include <termios.h>

#include <errno.h>
#include <signal.h>

typedef void sigfunc_t(int);

void signal_handler(int param) {
	static int fd;
	unsigned char sig;

	if (fd && param==0) {
		close(fd);
		return;
	}
	
	if (param < 1) {
		fd = -param;
		return;
	}

	if (fd) {
		sig=param;
		while (-1 == write(fd, &sig, 1) && EINTR == errno)
			;
	}
}

sigfunc_t *connect_signal(int signo, sigfunc_t *func) {
	struct sigaction act, oact;

	act.sa_handler = func;
	sigemptyset(&act.sa_mask);
	act.sa_flags = 0;
	if (sigaction(signo, &act, &oact) < 0)
		return SIG_ERR;
	return oact.sa_handler;
}

int open_signal_file(void) {
	int fd[2];
	
	pipe(fd);
	signal_handler(-fd[1]);    // hand off the write side of the pipe

	connect_signal(SIGHUP, signal_handler);
	connect_signal(SIGINT, signal_handler);
	connect_signal(SIGXFSZ, signal_handler);

	return fd[0];  //return the read side of the pipe
}

void close_signal_file(int fd) {
	signal_handler(0);
	close(fd);
}

int main(void) {
	fd_set read_fds;
	char buf[256], fname[]="log.0";
	int sigfd, fd, i;
	unsigned long long bytes_left=3ull * (1ull<<31) + 256ull;

	sigfd = open_signal_file();

	if (!sigfd)
		return 1;
	
	while (bytes_left) {
		int len;
		fd = creat(fname, S_IRUSR | S_IWUSR);
		if (fd == -1)
			return 2;
		
		len = 0x7fffffff;
		if (len > bytes_left)
			len = bytes_left;
		ftruncate(fd, len);
		bytes_left -= len;
		
		if (!bytes_left)
			break;
		
		lseek(fd, 0, SEEK_END);
		i = write(fd, buf, 256);

		if (i>0)
			bytes_left -= i;
		if (i<0)
			puts("write errored");
		
		FD_SET(sigfd, &read_fds);
		select(sigfd + 1, &read_fds, NULL, NULL, NULL);
		
		if (FD_ISSET(sigfd, &read_fds)) {
			int sigs = read(sigfd, buf, 256);
			if (sigs > 0)
				for (i=0; i<sigs; i++)
					switch(buf[i]) {
						case SIGINT:
							return 0;
						case SIGXFSZ:
							puts("seem to have gone overboard, switching to next log file...");
							break;
						default:
							break;
					}
		}
		
		close(fd);
		fname[4]++;
	}

	close_signal_file(sigfd);
	return 0;
}

^ permalink raw reply	[flat|nested] 19+ messages in thread
* Large files
@ 2003-06-10 13:57 Richard B. Johnson
  2003-06-10 14:16 ` ZCane, Ed (Test Purposes)
                   ` (2 more replies)
  0 siblings, 3 replies; 19+ messages in thread
From: Richard B. Johnson @ 2003-06-10 13:57 UTC (permalink / raw)
  To: Linux kernel


With 32 bit return values, ix86 Linux has a file-size limitation
which is currently about 0x7fffffff. Unfortunately, instead of
returning from a write() with a -1 and errno being set, so that
a program can do something about it, write() executes a signal(25)
which kills the task even if trapped. Is this one of those <expletive
deleted> POSIX requirements or is somebody going to fix it?

Cheers,
Dick Johnson
Penguin : Linux version 2.4.20 on an i686 machine (797.90 BogoMips).
Why is the government concerned about the lunatic fringe? Think about it.


^ permalink raw reply	[flat|nested] 19+ messages in thread
end of thread, other threads:[~2004-05-18 15:40 UTC | newest]

Thread overview: 19+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2004-05-17 19:48 large files Bernd Schubert
2004-05-17 20:12 ` Chris Mason
2004-05-17 20:25   ` Bernd Schubert
2004-05-18 13:42   ` Bernd Schubert
2004-05-18 13:57     ` Chris Mason
2004-05-18 14:49       ` Bernd Schubert
2004-05-18 15:07         ` Chris Mason
2004-05-18 15:19           ` Hans Reiser
2004-05-18 15:40             ` Chris Mason
  -- strict thread matches above, loose matches on Subject: below --
2003-06-10 22:38 Large files Ray Lee
2003-06-10 13:57 Richard B. Johnson
2003-06-10 14:16 ` ZCane, Ed (Test Purposes)
2003-06-10 14:17 ` Matti Aarnio
2003-06-10 15:12   ` Richard B. Johnson
2003-06-10 17:25     ` Martin Mares
2003-06-10 18:14     ` Andreas Dilger
2003-06-10 22:12     ` Rob Landley
2003-06-10 20:14 ` David Schwartz
2003-06-10 20:31   ` Richard B. Johnson

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.