Re: [PATCH] Software Suspend: Fix suspend when console is in VT_AUTO/KD_GRAPHICS mode

[Daniel Drake <dsd@gentoo.org>]

Andrew Johnson wrote:
> When the console is in VT_AUTO/KD_GRAPHICS mode, switching to the
> SUSPEND_CONSOLE fails, resulting in vt_waitactive() waiting indefinately
> or until the task is interrupted.  The following patch tests if a
> console switch can occur in set_console() and returns early if a console
> switch is not possible.
> 
> Signed-off-by: Andrew Johnson <ajohnson@intrinsyc.com>
> 
> diff -rup linux-2.6.20.1/drivers/char/vt.c linux/drivers/char/vt.c
> --- linux-2.6.20.1/drivers/char/vt.c	2007-02-19 22:34:32.000000000 -0800
> +++ linux/drivers/char/vt.c	2007-03-08 14:15:41.000000000 -0800
> @@ -2188,10 +2188,20 @@ static void console_callback(struct work
>  	release_console_sem();
>  }
>  
> -void set_console(int nr)
> +extern char vt_dont_switch;
> +
> +int set_console(int nr)
>  {
> +	struct vc_data *vc = vc_cons[fg_console].d;
> +
> +	if(!vc_cons_allocated(nr) || vt_dont_switch || vc->vc_mode ==
> KD_GRAPHICS) {
> +		return -EINVAL;
> +	}
> +
>  	want_console = nr;
>  	schedule_console_callback();
> +
> +	return 0;
>  }

I haven't tested, but I think the above -EINVAL return will break chvt. 
chvt uses the VT_ACTIVATE ioctl which calls set_console(), and it is 
valid for chvt to be used to change away from a graphics-mode console -- 
that shouldn't be an error condition.

Daniel