* Re: Virtual Console Hotplug Leads to OOM
2005-03-30 16:30 Virtual Console Hotplug Leads to OOM Prarit Bhargava
@ 2005-03-30 16:31 ` Prarit Bhargava
2005-04-01 0:17 ` Greg KH
2005-04-01 13:42 ` Prarit Bhargava
2 siblings, 0 replies; 4+ messages in thread
From: Prarit Bhargava @ 2005-03-30 16:31 UTC (permalink / raw)
To: linux-hotplug
[-- Attachment #1: Type: text/plain, Size: 1632 bytes --]
It helps to attach the test code :)
P.
Prarit Bhargava wrote:
> Colleagues,
>
> I've identified the following critical issue where the hotplug
> subsystem can cause OOM-killer to run. IMO this is a critical issue
> within hotplug -- in the case below, only the virtual console
> subsystem is responsible.
>
> The virtual consoles that are being spawned are responsible for 2
> hotplug events for every console opened -- one on the opening of the
> console, and one on the closing of the console.
>
> If you run the tty2_test which opens and closes ttys (attached to this
> email) you'll see THOUSANDS of hotplug events. After running for 1
> minute on a 16p ia64 I had well over 4000 hotplug events.
>
> If we're unlucky enough to fall below the low-water memory point at
> the same time tty2_test is spawning, the OOM-killer starts up. Memory
> is taken up by the hotplug events ... and we're off into the weeds.
>
> The issue with the virtual console code is that it uses the
> class_simple library. This is a problem as the library pre-defines a
> global kset parent as class_obj (or at least AFAICT). This parent is
> used by ALL subsystems using the class_simple, not just the virtual
> console subsystem.
>
> This parent has a pointer to allow hotplug slot operations for all
> it's children (including the virtual consoles). IMO zero-ing this
> pointer out is not a valid option as this would break hotplug op's for
> many other subystems.
>
> ** I did notice that the class struct has it's own hotplug value --
> can/should this be used to override the parent's hotplug operations? **
>
> P.
>
>
>
[-- Attachment #2: tty2_test.c --]
[-- Type: text/plain, Size: 1117 bytes --]
#include <sys/types.h>
#include <sys/stat.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <time.h>
#include <pthread.h>
#define NTHREADS 300
void *thread_function();
int open_fail_num;
int open_success;
int
main(int argc, char *argv[])
{
int i, j;
pthread_t thread_id[NTHREADS];
for(;;)
{
for(i=0; i < NTHREADS; i++)
{
pthread_create(&thread_id[i], NULL, &thread_function, NULL);
}
for(j=0; j < NTHREADS; j++)
{
pthread_join(thread_id[j], NULL);
}
printf("open failures: %i\n", open_fail_num);
printf("open success: %i\n", open_success);
}
}
void *thread_function()
{
int fd;
time_t t;
// fd = open("/dev/cua0", O_NONBLOCK);
// fd = open("/dev/cua0", O_RDWR);
fd = open("/dev/tty9", O_RDWR);
// if (fd == -1)
if (fd < 0)
{
open_fail_num++;
// perror("open");
// exit(1);
}
else
{
open_success++;
}
/* just waste some random time */
t = (time((time_t *)0) &31L) << 6;
while (t-- > 0)
(void)time((time_t *)0);
close(fd);
// _exit(0);
}
[-- Attachment #3: Makefile --]
[-- Type: text/plain, Size: 566 bytes --]
#
# make test command
#
NAME_SRC=tty2_test
CFLAGS= -O -g -Wall -I /usr/include/
LDFLAGS=-lpthread
CC=cc
GET=co
SRCS=$(NAME_SRC).c
OBJS=$(NAME_SRC).o
DEPEND= makedepend $(CFLAGS)
all: $(NAME_SRC)
# To make an executable
$(NAME_SRC): $(OBJS)
$(CC) $(LDFLAGS) -o $@ $(OBJS)
# what are the source dependencies
depend: $(SRCS)
$(DEPEND) $(SRCS)
# clean out the junk
clean:
-rm $(NAME_SRC) *~ *.o
# DO NOT DELETE THIS LINE -- make depend depends on it.
$(NAME_SRC).o: /usr/include/sys/types.h /usr/include/sys/stat.h \
/usr/include/fcntl.h /usr/include/pthread.h
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: Virtual Console Hotplug Leads to OOM
2005-03-30 16:30 Virtual Console Hotplug Leads to OOM Prarit Bhargava
2005-03-30 16:31 ` Prarit Bhargava
@ 2005-04-01 0:17 ` Greg KH
2005-04-01 13:42 ` Prarit Bhargava
2 siblings, 0 replies; 4+ messages in thread
From: Greg KH @ 2005-04-01 0:17 UTC (permalink / raw)
To: linux-hotplug
On Wed, Mar 30, 2005 at 11:30:23AM -0500, Prarit Bhargava wrote:
> Colleagues,
>
> I've identified the following critical issue where the hotplug subsystem
> can cause OOM-killer to run. IMO this is a critical issue within
> hotplug -- in the case below, only the virtual console subsystem is
> responsible.
>
> The virtual consoles that are being spawned are responsible for 2
> hotplug events for every console opened -- one on the opening of the
> console, and one on the closing of the console.
Yes.
> If you run the tty2_test which opens and closes ttys (attached to this
> email) you'll see THOUSANDS of hotplug events. After running for 1
> minute on a 16p ia64 I had well over 4000 hotplug events.
Guess you need a faster machine, my little x86-64 had way more than that :)
> If we're unlucky enough to fall below the low-water memory point at the
> same time tty2_test is spawning, the OOM-killer starts up. Memory is
> taken up by the hotplug events ... and we're off into the weeds.
Yup.
> The issue with the virtual console code is that it uses the class_simple
> library. This is a problem as the library pre-defines a global kset
> parent as class_obj (or at least AFAICT). This parent is used by ALL
> subsystems using the class_simple, not just the virtual console subsystem.
If you look at the -mm tree, class_simple is now gone.
> This parent has a pointer to allow hotplug slot operations for all it's
> children (including the virtual consoles). IMO zero-ing this pointer
> out is not a valid option as this would break hotplug op's for many
> other subystems.
Very true.
> ** I did notice that the class struct has it's own hotplug value --
> can/should this be used to override the parent's hotplug operations? **
Why would you want to override this? We need those hotplug events. All
this means is that an admin better be careful who he lets have access to
the tty group :)
thanks,
greg k-h
-------------------------------------------------------
This SF.net email is sponsored by Demarc:
A global provider of Threat Management Solutions.
Download our HomeAdmin security software for free today!
http://www.demarc.com/info/Sentarus/hamr30
_______________________________________________
Linux-hotplug-devel mailing list http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Virtual Console Hotplug Leads to OOM
2005-03-30 16:30 Virtual Console Hotplug Leads to OOM Prarit Bhargava
2005-03-30 16:31 ` Prarit Bhargava
2005-04-01 0:17 ` Greg KH
@ 2005-04-01 13:42 ` Prarit Bhargava
2 siblings, 0 replies; 4+ messages in thread
From: Prarit Bhargava @ 2005-04-01 13:42 UTC (permalink / raw)
To: linux-hotplug
Greg,
Thanks for the reply.
Greg KH wrote:
>>If you run the tty2_test which opens and closes ttys (attached to this
>>email) you'll see THOUSANDS of hotplug events. After running for 1
>>minute on a 16p ia64 I had well over 4000 hotplug events.
>>
>>
>
>Guess you need a faster machine, my little x86-64 had way more than that :)
>
>
(sheepish grin) I did put a few sleeps in there so I could actually read
the output streaming out :)
>
>
>>If we're unlucky enough to fall below the low-water memory point at the
>>same time tty2_test is spawning, the OOM-killer starts up. Memory is
>>taken up by the hotplug events ... and we're off into the weeds.
>>
>>
>
>Yup.
>
>
No offense, but I'm not sure we should be so cavalier on this issue --
I'm pointing out that we have a potential fork bomb within the kernel.
Aside from that, I'm think that hotplug is more of a victim here than
the cause. I'm also noticing that on this machine with 100,000
outstanding task_structs, approximately 99,000 of them are hotplug
task_structs. Something is definately very wrong here. These should
have been reaped. These tasks are not in the pid table but are left
lying around on the task_struct list.
>
>
>>The issue with the virtual console code is that it uses the class_simple
>>library. This is a problem as the library pre-defines a global kset
>>parent as class_obj (or at least AFAICT). This parent is used by ALL
>>subsystems using the class_simple, not just the virtual console subsystem.
>>
>>
>
>If you look at the -mm tree, class_simple is now gone.
>
>
Will do. Was there some signifigant reason for removing class_simple?
(Just curious ...)
>
>
>>This parent has a pointer to allow hotplug slot operations for all it's
>>children (including the virtual consoles). IMO zero-ing this pointer
>>out is not a valid option as this would break hotplug op's for many
>>other subystems.
>>
>>
>
>Very true.
>
>
>
>>** I did notice that the class struct has it's own hotplug value --
>>can/should this be used to override the parent's hotplug operations? **
>>
>>
>
>Why would you want to override this? We need those hotplug events. All
>this means is that an admin better be careful who he lets have access to
>the tty group :)
>
>
Absolutely correct -- but on some machines the group can be uucp ...
which could *potentially* cause a security hole. The tty2_test is
(very) contrived but still ... it should be something we're aware of as
it could cause problems on machines at critical stress points.
>thanks,
>
>greg k-h
>
>
>
-------------------------------------------------------
This SF.net email is sponsored by Demarc:
A global provider of Threat Management Solutions.
Download our HomeAdmin security software for free today!
http://www.demarc.com/info/Sentarus/hamr30
_______________________________________________
Linux-hotplug-devel mailing list http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel
^ permalink raw reply [flat|nested] 4+ messages in thread