Virtual Console Hotplug Leads to OOM

Virtual Console Hotplug Leads to OOM

[Prarit Bhargava]

Colleagues,

I've identified the following critical issue where the hotplug subsystem 
can cause OOM-killer to run.  IMO this is a critical issue within 
hotplug -- in the case below, only the virtual console subsystem is 
responsible.

The virtual consoles that are being spawned are responsible for 2 
hotplug events for every console opened -- one on the opening of the 
console, and one on the closing of the console.

If you run the tty2_test which opens and closes ttys (attached to this 
email) you'll see THOUSANDS of hotplug events.  After running for 1 
minute on a 16p ia64 I had well over 4000 hotplug events.

If we're unlucky enough to fall below the low-water memory point at the 
same time tty2_test is spawning, the OOM-killer starts up.  Memory is 
taken up by the hotplug events ... and we're off into the weeds.

The issue with the virtual console code is that it uses the class_simple 
library.  This is a problem as the library pre-defines a global kset 
parent as class_obj (or at least AFAICT).  This parent is used by ALL 
subsystems using the class_simple, not just the virtual console subsystem.

This parent has a pointer to allow hotplug slot operations for all it's 
children (including the virtual consoles).  IMO zero-ing this pointer 
out is not a valid option as this would break hotplug op's for many 
other subystems.

** I did notice that the class struct has it's own hotplug value -- 
can/should this be used to override the parent's hotplug operations? **

P.




-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_ide95&alloc_id\x14396&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Virtual Console Hotplug Leads to OOM

[Prarit Bhargava]

[-- Attachment #1: Type: text/plain, Size: 1632 bytes --]

It helps to attach the test code :)

P.

Prarit Bhargava wrote:

> Colleagues,
>
> I've identified the following critical issue where the hotplug 
> subsystem can cause OOM-killer to run.  IMO this is a critical issue 
> within hotplug -- in the case below, only the virtual console 
> subsystem is responsible.
>
> The virtual consoles that are being spawned are responsible for 2 
> hotplug events for every console opened -- one on the opening of the 
> console, and one on the closing of the console.
>
> If you run the tty2_test which opens and closes ttys (attached to this 
> email) you'll see THOUSANDS of hotplug events.  After running for 1 
> minute on a 16p ia64 I had well over 4000 hotplug events.
>
> If we're unlucky enough to fall below the low-water memory point at 
> the same time tty2_test is spawning, the OOM-killer starts up.  Memory 
> is taken up by the hotplug events ... and we're off into the weeds.
>
> The issue with the virtual console code is that it uses the 
> class_simple library.  This is a problem as the library pre-defines a 
> global kset parent as class_obj (or at least AFAICT).  This parent is 
> used by ALL subsystems using the class_simple, not just the virtual 
> console subsystem.
>
> This parent has a pointer to allow hotplug slot operations for all 
> it's children (including the virtual consoles).  IMO zero-ing this 
> pointer out is not a valid option as this would break hotplug op's for 
> many other subystems.
>
> ** I did notice that the class struct has it's own hotplug value -- 
> can/should this be used to override the parent's hotplug operations? **
>
> P.
>
>
>

[-- Attachment #2: tty2_test.c --]
[-- Type: text/plain, Size: 1117 bytes --]

#include <sys/types.h>
#include <sys/stat.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <time.h>
#include <pthread.h>

#define NTHREADS 300

void *thread_function();
int open_fail_num;
int open_success;

int
main(int argc, char *argv[])
{
  int i, j;
  pthread_t thread_id[NTHREADS];

  for(;;)
    {
      for(i=0; i < NTHREADS; i++)
	{
	  pthread_create(&thread_id[i], NULL, &thread_function, NULL);
	}
      for(j=0; j < NTHREADS; j++)
	{
	  pthread_join(thread_id[j], NULL); 
	}
      printf("open failures: %i\n", open_fail_num);
      printf("open success: %i\n", open_success);
    }
}

void *thread_function()
{
  int fd;
  time_t t;

  //  fd = open("/dev/cua0", O_NONBLOCK);
  //  fd = open("/dev/cua0", O_RDWR);
  fd = open("/dev/tty9", O_RDWR);

  //  if (fd == -1)
  if (fd < 0)
    {
      open_fail_num++;
      //      perror("open");
      //      exit(1);
    }
  else
    {
      open_success++;
    }
  /* just waste some random time */
  t = (time((time_t *)0) &31L) << 6;
  while (t-- > 0)
    (void)time((time_t *)0);
  close(fd);	
  //  _exit(0);
}

[-- Attachment #3: Makefile --]
[-- Type: text/plain, Size: 566 bytes --]

#
# make test command
#
NAME_SRC=tty2_test
CFLAGS= -O -g -Wall -I /usr/include/
LDFLAGS=-lpthread
CC=cc
GET=co
SRCS=$(NAME_SRC).c
OBJS=$(NAME_SRC).o
DEPEND= makedepend $(CFLAGS)
all:	$(NAME_SRC)

# To make an executable

$(NAME_SRC): $(OBJS)
	$(CC) $(LDFLAGS) -o $@ $(OBJS)

# what are the source dependencies
depend:	$(SRCS)
	$(DEPEND) $(SRCS)

# clean out the junk
clean:
	-rm $(NAME_SRC) *~ *.o

# DO NOT DELETE THIS LINE -- make depend depends on it.
$(NAME_SRC).o: /usr/include/sys/types.h /usr/include/sys/stat.h \
/usr/include/fcntl.h /usr/include/pthread.h

Re: Virtual Console Hotplug Leads to OOM

[Greg KH]

On Wed, Mar 30, 2005 at 11:30:23AM -0500, Prarit Bhargava wrote:
> Colleagues,
> 
> I've identified the following critical issue where the hotplug subsystem 
> can cause OOM-killer to run.  IMO this is a critical issue within 
> hotplug -- in the case below, only the virtual console subsystem is 
> responsible.
> 
> The virtual consoles that are being spawned are responsible for 2 
> hotplug events for every console opened -- one on the opening of the 
> console, and one on the closing of the console.

Yes.

> If you run the tty2_test which opens and closes ttys (attached to this 
> email) you'll see THOUSANDS of hotplug events.  After running for 1 
> minute on a 16p ia64 I had well over 4000 hotplug events.

Guess you need a faster machine, my little x86-64 had way more than that :)

> If we're unlucky enough to fall below the low-water memory point at the 
> same time tty2_test is spawning, the OOM-killer starts up.  Memory is 
> taken up by the hotplug events ... and we're off into the weeds.

Yup.

> The issue with the virtual console code is that it uses the class_simple 
> library.  This is a problem as the library pre-defines a global kset 
> parent as class_obj (or at least AFAICT).  This parent is used by ALL 
> subsystems using the class_simple, not just the virtual console subsystem.

If you look at the -mm tree, class_simple is now gone.

> This parent has a pointer to allow hotplug slot operations for all it's 
> children (including the virtual consoles).  IMO zero-ing this pointer 
> out is not a valid option as this would break hotplug op's for many 
> other subystems.

Very true.

> ** I did notice that the class struct has it's own hotplug value -- 
> can/should this be used to override the parent's hotplug operations? **

Why would you want to override this?  We need those hotplug events.  All
this means is that an admin better be careful who he lets have access to
the tty group :)

thanks,

greg k-h


-------------------------------------------------------
This SF.net email is sponsored by Demarc:
A global provider of Threat Management Solutions.
Download our HomeAdmin security software for free today!
http://www.demarc.com/info/Sentarus/hamr30
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Virtual Console Hotplug Leads to OOM

[Prarit Bhargava]

Greg,

Thanks for the reply.

Greg KH wrote:

>>If you run the tty2_test which opens and closes ttys (attached to this 
>>email) you'll see THOUSANDS of hotplug events.  After running for 1 
>>minute on a 16p ia64 I had well over 4000 hotplug events.
>>    
>>
>
>Guess you need a faster machine, my little x86-64 had way more than that :)
>  
>
(sheepish grin) I did put a few sleeps in there so I could actually read 
the output streaming out :)

>  
>
>>If we're unlucky enough to fall below the low-water memory point at the 
>>same time tty2_test is spawning, the OOM-killer starts up.  Memory is 
>>taken up by the hotplug events ... and we're off into the weeds.
>>    
>>
>
>Yup.
>  
>
No offense, but I'm not sure we should be so cavalier on this issue -- 
I'm pointing out that we have a potential fork bomb within the kernel.

Aside from that, I'm think that hotplug is more of a victim here than 
the cause.  I'm also noticing that on this machine with 100,000 
outstanding task_structs, approximately 99,000 of them are hotplug 
task_structs.  Something is definately very wrong here.  These should 
have been reaped.  These tasks are not in the pid table but are left 
lying around on the task_struct list.

>  
>
>>The issue with the virtual console code is that it uses the class_simple 
>>library.  This is a problem as the library pre-defines a global kset 
>>parent as class_obj (or at least AFAICT).  This parent is used by ALL 
>>subsystems using the class_simple, not just the virtual console subsystem.
>>    
>>
>
>If you look at the -mm tree, class_simple is now gone.
>  
>
Will do.  Was there some signifigant reason for removing class_simple? 
(Just curious ...)

>  
>
>>This parent has a pointer to allow hotplug slot operations for all it's 
>>children (including the virtual consoles).  IMO zero-ing this pointer 
>>out is not a valid option as this would break hotplug op's for many 
>>other subystems.
>>    
>>
>
>Very true.
>
>  
>
>>** I did notice that the class struct has it's own hotplug value -- 
>>can/should this be used to override the parent's hotplug operations? **
>>    
>>
>
>Why would you want to override this?  We need those hotplug events.  All
>this means is that an admin better be careful who he lets have access to
>the tty group :)
>  
>
Absolutely correct -- but on some machines the group can be uucp ... 
which could *potentially* cause a security hole.  The tty2_test is 
(very) contrived but still ... it should be something we're aware of as 
it could cause problems on machines at critical stress points.

>thanks,
>
>greg k-h
>
>  
>


-------------------------------------------------------
This SF.net email is sponsored by Demarc:
A global provider of Threat Management Solutions.
Download our HomeAdmin security software for free today!
http://www.demarc.com/info/Sentarus/hamr30
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel