[uml-devel] 2.4.23(24?)-2 TT mode hangs on request_module

[uml-devel] 2.4.23(24?)-2 TT mode hangs on request_module

[Henrik Nordstrom]

On all our boxes not having the SKAS patch recent UML 2.4 kernels has
started hanging on the first request_module call from within the kernel.

It hangs quite hard, not responding to either gdb or mconsole.

Host: Various RedHat 7.x, 8.0, 9 and FC1 kernels.
Kernel: 2.4.26
UML: Current CVS (2.4.23-2)
Root: hostfs with the hostfs and aio patches sent earlier

Last known working before the last CVS update (2.4.23-1 CVS version 
works).

SKAS mode works, problem only seen in TT mode.

As there has been quite large changes from the earlier version and the
fact that the deugger also hangs I am not entirely sure where to look or
how to approach the problem. Trying to manually trace the kernel thread 
creation and execution to more exacly pinpoint where during the 
request_module or resulting modprobe thread the UML hangs has proven to be 
a little difficult and confusing..

I can very reliably reproduce the problem by

1. Boot in single user mode (/proc mounted etc.., but no other disturbing 
factors)
2. iptables -A INPUT -j REJECT

The sequence of events here is:

1. iptables modprobes the base modules (iptable_filter), then uploads the 
new iptable which referes to the REJECT match.

2. The kernel verifies the new iptable and triggers a request_module call
for the ipt_REJECT module and here the kernel gets completely stuck..

If all required modules are loaded manually avoiding the call to
request_module then there is no problem, but unfortunately this is not a
very feasible workaround here..

I do not think this is caused by any of our patches (plenty of netfilter
patches, upgraded to 2.4.26 etc etc..) but I will try building a "plain"  
UML kernel.  The earlier UML revision worked quite well except for that
klogd issue in SKAS mode which is a real pain as we have started to
migrate to SKAS in our development environment..

Ideas on how to approach problems like this are welcome.

Regards
Henrik



-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id=2562&alloc_id=6184&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] 2.4.23(24?)-2 TT mode hangs on request_module

[Robbie Dinn]

Henrik Nordstrom wrote:

> I can very reliably reproduce the problem by
> 
> 1. Boot in single user mode (/proc mounted etc.., but no other disturbing 
> factors)
> 2. iptables -A INPUT -j REJECT
> 

I think I may have seen this too, although I didn't understand what
was going on at the time (still don't). I am a little unsure what
versions of software I was using, because I wasn't working very
systematically. It might have been:
  host: 2.6.6 + skas3 patch or 2.6.6-rc3-mm2 + ska3 patch
	+ recent CVS netfilter p-o-m

  uml: 2.6.6 + blaisorblades unofficial patches
	and/or uml patches from suse kernel
	+ recent CVS netfilter p-o-m

I am just a monkey applying patches here, I don't claim to actually
understand the code.

I could get the UML machine to start in single user mode, but
when starting in run level 2 or 3, it would hang (eating lots of cpu
cycles) while starting the firewall.

I assumed that the hang was due to me failing to recompile and install
a fresh version of iptables on UML machine, but I don't have any good
reason to back up that belief. Does that sound plausible?
(I was using iptables-1.2.9 built against an older UML kernel, and had
the 1.2.10 sources waiting to be used).

> The sequence of events here is:
> 
> 1. iptables modprobes the base modules (iptable_filter), then uploads the 
> new iptable which referes to the REJECT match.
> 
> 2. The kernel verifies the new iptable and triggers a request_module call
> for the ipt_REJECT module and here the kernel gets completely stuck..
> 
> If all required modules are loaded manually avoiding the call to
> request_module then there is no problem, but unfortunately this is not a
> very feasible workaround here..
> 

The UML binary that I was using had all the iptables stuff statically
linked. I wasn't using modules at all.

If by so minor miracle I do manage to assemble something that works or
I find some useful information, I will post a follow up of course.

HTH

Robbie Dinn


-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id=2562&alloc_id=6184&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] 2.4.23(24?)-2 TT mode hangs on request_module

[Henrik Nordstrom]

On Mon, 17 May 2004, Robbie Dinn wrote:

> I think I may have seen this too, although I didn't understand what
> was going on at the time (still don't). I am a little unsure what
> versions of software I was using, because I wasn't working very
> systematically. It might have been:
>   host: 2.6.6 + skas3 patch or 2.6.6-rc3-mm2 + ska3 patch
> 	+ recent CVS netfilter p-o-m

If you are using SKAS and statically linked kernel (no modules) then your
problem is most likely different. The problem I am seeing is only when the
kernel tries to dynamically load a module. If code is statially linked or
manually loaded using modprobe then the problem does not occur, and also
if the host supports SKAS then the problem does not occur. The problem I
am seeing is only if using TT and allowing the kernel to dynamically
initiating loading of modules (not manually using modprobe or statically 
linked), has only been seen in the current CVS version of UML-2.4.

Please attach a debugger to your uml to see what it is doing. It may be
the same problem I am seeing, but most likely not.

If I were to guess what your problem is I would guess there is again some
problem with the vmalloc implementation in the version of UML you are
using.. there has historically been issues with vmalloc in UML and
iptables is one of very few users of the vmalloc kernel interface..

Regards
Henrik



-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id=2562&alloc_id=6184&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] 2.4.23(24?)-2 TT mode hangs on request_module

[Henrik Nordstrom]

On Mon, 17 May 2004, Henrik Nordstrom wrote:

> On all our boxes not having the SKAS patch recent UML 2.4 kernels has
> started hanging on the first request_module call from within the kernel.

Just rebuilt with uml-patch-2.4.26-1 and the problem remains, and in 
addition there is new problems.. now hostfs fails to mount manually 
specified directories again (root works, mount does not)

More info available tomorrow.

Regards
Henrik



-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id=2562&alloc_id=6184&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] 2.4.23(24?)-2 TT mode hangs on request_module

[Henrik Nordstrom]

On Mon, 17 May 2004, Henrik Nordstrom wrote:

> Just rebuilt with uml-patch-2.4.26-1 and the problem remains

Here is another simpler way to reproduce the same problem not involving
iptables:

1. Compile UML 2.4.26-1 with module support

2. Boot in TT mode (not SKAS)

3. ifconfig xyzzy (or any other non-existing device)

If there is any other out there using 2.4.26-1 in TT mode (not SKAS) can 
you please try if the above hangs your umls as well, or if there is 
something peculiar with my setup triggering this but I do not think there 
is anything special with my UML tree which can trigger this kind of 
problem.

Regards
Henrik



-------------------------------------------------------
This SF.Net email is sponsored by: SourceForge.net Broadband
Sign-up now for SourceForge Broadband and get the fastest
6.0/768 connection for only $19.95/mo for the first 3 months!
http://ads.osdn.com/?ad_id=2562&alloc_id=6184&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

[uml-devel] 2.4.26-1 hostfs very broken

[Henrik Nordstrom]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 730 bytes --]

hostfs is currently broken in very many ways

a) Mounting fails because it tries to open the hostfs filesystem directory 
as a file.

b) Any file I/O operations hang as their completion function is never 
called unless there is an error.

c) Even if the hostfs is mounted read-only all file operations is done 
read-write. Not sure if it at all cares about the read-only state..

d) Seem to create empty files on any attempt to access (open/stat etc) a 
non-existing file.

e) Mounting of directories ontop of the hostfs fails miserably (/proc 
mount disappeared, still showing the hostfs content)

Attached is a patch with my attempts to correct issues 'a' and 'b' by 
trying to compare with how humfs operates..

Regards
Henrik

[-- Attachment #2: Type: TEXT/PLAIN, Size: 1875 bytes --]

Binary files linux-2.4.26-1.orig/arch/um/fs/built-in.o and linux-2.4.26-1.hno/arch/um/fs/built-in.o differ
diff -ru linux-2.4.26-1.orig/arch/um/fs/hostfs/externfs.c linux-2.4.26-1.hno/arch/um/fs/hostfs/externfs.c
--- linux-2.4.26-1.orig/arch/um/fs/hostfs/externfs.c	Mon May 17 23:36:17 2004
+++ linux-2.4.26-1.hno/arch/um/fs/hostfs/externfs.c	Tue May 18 00:10:42 2004
@@ -1133,7 +1133,7 @@
 	if((mount_arg != NULL) && (*mount_arg != '\0'))
 		root = mount_arg;
 
-	return(uml_strdup(mount_arg));
+	return(uml_strdup(root));
 }
 
 struct super_block *hostfs_read_super(struct super_block *sb, void *data, 
Binary files linux-2.4.26-1.orig/arch/um/fs/hostfs/externfs.o and linux-2.4.26-1.hno/arch/um/fs/hostfs/externfs.o differ
diff -ru linux-2.4.26-1.orig/arch/um/fs/hostfs/host_fs.c linux-2.4.26-1.hno/arch/um/fs/hostfs/host_fs.c
--- linux-2.4.26-1.orig/arch/um/fs/hostfs/host_fs.c	Mon May 17 23:36:17 2004
+++ linux-2.4.26-1.hno/arch/um/fs/hostfs/host_fs.c	Tue May 18 01:12:55 2004
@@ -138,6 +138,10 @@
 		goto out;
 
 	err = host_open_file(path, 1, 1, fh);
+	if (err == -EISDIR) {
+	    kfree(fh);
+	    return NULL;
+	}
 	if(err)
 		goto out_free;
 
@@ -175,10 +179,9 @@
 				len - ignore_end);
 
  out:
-	if(err < 0)
-		(*completion)(buf, err, arg);
-	else err = 0;
-
+	(*completion)(buf, err, arg);
+	if (err > 0)
+	    err = 0;
 	return(err);
 }
 
@@ -191,8 +194,9 @@
 
 	err = write_file(fh, offset + start, buf + start, len);
 
-	if(err < 0)
-		(*completion)((char *) buf, err, arg);
+	(*completion)((char *) buf, err, arg);
+	if (err > 0)
+	    err = 0;
 	return(err);
 }
 
Binary files linux-2.4.26-1.orig/arch/um/fs/hostfs/host_fs.o and linux-2.4.26-1.hno/arch/um/fs/hostfs/host_fs.o differ
Binary files linux-2.4.26-1.orig/arch/um/fs/hostfs/hostfs.o and linux-2.4.26-1.hno/arch/um/fs/hostfs/hostfs.o differ

[uml-devel] [patch] 2.4.26-1 hostfs very broken

[Henrik Nordstrom]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 924 bytes --]

Attached is a revised patch to fix hostfs operation in 2.4.26-1. This 
replaces the earlier patch sent yesterday.

  + Correct default root path
  + Fix open flags when opening files
  + Don't create files when opening (only on create)
  + Set gid to 0 to not confuse with host permissions
  + Handle files not writable on the host
  + Signal completetion on I/O operations

This has been tested so far that

   - It boots my hostfs root read-only without destroying the contents
   - I can manually mount a directory from the host read-write and normal 
read/write operations seem to work (creation of files, writing etc)

The patch does change some functions in the common framework shared with 
humfs but from what I can tell the issues corrected in the common files 
should affect humfs as well and I do not expect these changes to have any 
ill effect on humfs, but I have not verified humfs operation.

Regards
Henrik

[-- Attachment #2: Type: TEXT/PLAIN, Size: 3004 bytes --]

diff -ru linux-2.4.26-1.orig/arch/um/fs/hostfs/externfs.c linux-2.4.26-1.hno/arch/um/fs/hostfs/externfs.c
--- linux-2.4.26-1.orig/arch/um/fs/hostfs/externfs.c	Mon May 17 23:36:17 2004
+++ linux-2.4.26-1.hno/arch/um/fs/hostfs/externfs.c	Tue May 18 00:10:42 2004
@@ -1133,7 +1133,7 @@
 	if((mount_arg != NULL) && (*mount_arg != '\0'))
 		root = mount_arg;
 
-	return(uml_strdup(mount_arg));
+	return(uml_strdup(root));
 }
 
 struct super_block *hostfs_read_super(struct super_block *sb, void *data, 
diff -ru linux-2.4.26-1.orig/arch/um/fs/hostfs/host_file.c linux-2.4.26-1.hno/arch/um/fs/hostfs/host_file.c
--- linux-2.4.26-1.orig/arch/um/fs/hostfs/host_file.c	Mon May 17 23:36:17 2004
+++ linux-2.4.26-1.hno/arch/um/fs/hostfs/host_file.c	Tue May 18 11:40:47 2004
@@ -72,28 +72,21 @@
 {
 	char tmp[HOSTFS_BUFSIZE], *file;
 	int mode = 0, err;
+	struct openflags flags = OPENFLAGS();
 
-	if(r && !w) 
-		mode = O_RDONLY;
-	else if(!r && w) 
-		mode = O_WRONLY;
-	else if(r && w) 
-		mode = O_RDWR;
-	else {
-	  	printk("Impossible mode in host_open_file - r = %d, w = %d", 
-		       r, w);
-		return(-EINVAL);
-	}
-
+	if (r)
+	    flags = of_read(flags);
+	if (w)
+	    flags = of_write(flags);
 	if(append)
-		mode |= O_APPEND;
+	    flags = of_append(flags);
 
 	err = -ENOMEM;
 	file = get_path(path, tmp, sizeof(tmp));
 	if(file == NULL)
 		goto out;
 	
-	err = open_filehandle(file, of_create(of_rdwr(OPENFLAGS())), mode, fh);
+	err = open_filehandle(file, flags, 0, fh);
  out:
 	free_path(file, tmp);
 	return(err);
diff -ru linux-2.4.26-1.orig/arch/um/fs/hostfs/host_fs.c linux-2.4.26-1.hno/arch/um/fs/hostfs/host_fs.c
--- linux-2.4.26-1.orig/arch/um/fs/hostfs/host_fs.c	Mon May 17 23:36:17 2004
+++ linux-2.4.26-1.hno/arch/um/fs/hostfs/host_fs.c	Tue May 18 11:38:51 2004
@@ -111,10 +111,14 @@
 {
 	const char *path[] = { jail_dir, mount, file, NULL };
 	int uid;
+	int gid;
+	int rc;
 
+	/* These should be mount flags from an internal "superblock" */
 	*uid_out = 0;
+	*gid_out = 0;
 	return(host_stat_file(path, dev_out, inode_out, mode_out, nlink_out, 
-			      &uid, gid_out, size_out, atime_out, mtime_out, 
+			      &uid, &gid, size_out, atime_out, mtime_out, 
 			      ctime_out, blksize_out, blocks_out));
 }
 
@@ -138,6 +142,12 @@
 		goto out;
 
 	err = host_open_file(path, 1, 1, fh);
+	if (err == -EISDIR) {
+	    kfree(fh);
+	    return NULL;
+	}
+	if (err == -EPERM)
+	    err = host_open_file(path, 1, 0, fh);
 	if(err)
 		goto out_free;
 
@@ -175,10 +185,9 @@
 				len - ignore_end);
 
  out:
-	if(err < 0)
-		(*completion)(buf, err, arg);
-	else err = 0;
-
+	(*completion)(buf, err, arg);
+	if (err > 0)
+	    err = 0;
 	return(err);
 }
 
@@ -191,8 +200,9 @@
 
 	err = write_file(fh, offset + start, buf + start, len);
 
-	if(err < 0)
-		(*completion)((char *) buf, err, arg);
+	(*completion)((char *) buf, err, arg);
+	if (err > 0)
+	    err = 0;
 	return(err);
 }
 

Re: [uml-devel] [patch] 2.4.26-1 hostfs very broken

[Jeff Dike]

uml@hno.marasystems.com said:
> Attached is a revised patch to fix hostfs operation in 2.4.26-1. This
> replaces the earlier patch sent yesterday. 

Applied, thanks.

		Jeff



-------------------------------------------------------
This SF.Net email is sponsored by the new InstallShield X.
From Windows to Linux, servers to mobile, InstallShield X is the one
installation-authoring solution that does it all. Learn more and
evaluate today! http://www.installshield.com/Dev2Dev/0504
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel