very poor performance in 2.5.66[-mm1]

very poor performance in 2.5.66[-mm1]

[David Mansfield]

Hi list.

After all of the rave reviews about the interactivity fixes (both regular 
and I/O scheduler related), I decided to give the 2.5.latest a try on my 
desktop machine (system described below)

I started X, everything seemed fine, maybe a bit faster.  I opened a 
'gnome-terminal' and typed 'ls -ltr'.  Wow, it was 20x slower.

Here are the timings for 'ls -ltr':

2.5.66-mm1:      'ls -ltr'         31 seconds
2.5.66-mm1:      'ls -ltr | cat'   2 seconds
2.4.18-rhlatest: 'ls -ltr'         1.14 seconds

I also tried 2.5.66 vanilla with the exact same results.  Everything else 
seemed fine about the system.

I'm running Red Hat 8.0 updated to the latest packages, and I know that 
the anti-aliasing that gnome-terminal does takes a ton of CPU time.  In 
fact, all of the 31 seconds above is used by X.  You can see that when the 
output of 'ls' is piped through 'cat' (essentially changing the output 
from line buffered to block buffered), the time went down dramatically.

I also have noticed in the past that under redhat 8.0, X runs at nice -10.  
With the 2.5.66 kernel it was 'nice 0' (i.e. not reniced).  I tried 
various renicing, both positive and negative (+/- 1), to no effect.  Also, 
renicing X to 0 in 2.4.18-rhlatest doesn't cause the above abnormality.

This must be some scheduler interaction issue that has changed the way X 
buffers it's redraw requests or something.  Seems like something that used 
to be asynchronous is now synchronous.  Just a guess.

System:

UP,Preempt Athlon 1.4 Ghz, 512MB ram, IDE, SiS chipset for ide and video

Kernel config:

CONFIG_X86=y
CONFIG_MMU=y
CONFIG_UID16=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_EXPERIMENTAL=y
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSCTL=y
CONFIG_LOG_BUF_SHIFT=14
CONFIG_X86_PC=y
CONFIG_MK7=y
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_USE_3DNOW=y
CONFIG_PREEMPT=y
CONFIG_X86_TSC=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_NONFATAL=y
CONFIG_MICROCODE=y
CONFIG_X86_MSR=y
CONFIG_X86_CPUID=y
CONFIG_HIGHMEM4G=y
CONFIG_HIGHMEM=y
CONFIG_MTRR=y
CONFIG_HAVE_DEC_LOCK=y
CONFIG_PCI=y
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_NAMES=y
CONFIG_ISA=y
CONFIG_KCORE_ELF=y
CONFIG_BINFMT_ELF=y
CONFIG_PARPORT=y
CONFIG_PARPORT_PC=y
CONFIG_PARPORT_PC_CML1=y
CONFIG_PARPORT_SERIAL=y
CONFIG_PARPORT_PC_FIFO=y
CONFIG_PARPORT_1284=y
CONFIG_BLK_DEV_FD=y
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_NBD=y
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_SIZE=4096
CONFIG_BLK_DEV_INITRD=y
CONFIG_IDE=y
CONFIG_BLK_DEV_IDE=y
CONFIG_BLK_DEV_IDEDISK=y
CONFIG_IDEDISK_MULTI_MODE=y
CONFIG_BLK_DEV_IDECD=y
CONFIG_BLK_DEV_IDESCSI=y
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_IDEPCI_SHARE_IRQ=y
CONFIG_BLK_DEV_IDEDMA_PCI=y
CONFIG_IDEDMA_PCI_AUTO=y
CONFIG_BLK_DEV_IDEDMA=y
CONFIG_BLK_DEV_ADMA=y
CONFIG_BLK_DEV_SIS5513=y
CONFIG_IDEDMA_AUTO=y
CONFIG_BLK_DEV_IDE_MODES=y
CONFIG_SCSI=y
CONFIG_BLK_DEV_SR=y
CONFIG_BLK_DEV_SR_VENDOR=y
CONFIG_CHR_DEV_SG=y
CONFIG_SCSI_REPORT_LUNS=y
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_LINEAR=y
CONFIG_MD_RAID0=y
CONFIG_MD_RAID1=y
CONFIG_MD_RAID5=y
CONFIG_BLK_DEV_DM=y
CONFIG_NET=y
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_NETLINK_DEV=y
CONFIG_UNIX=y
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_SYN_COOKIES=y
CONFIG_IPV6_SCTP__=y
CONFIG_NETDEVICES=y
CONFIG_DUMMY=y
CONFIG_TUN=y
CONFIG_NET_ETHERNET=y
CONFIG_NET_PCI=y
CONFIG_8139TOO=y
CONFIG_INPUT=y
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_INPUT_EVDEV=y
CONFIG_SOUND_GAMEPORT=y
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=y
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_UNIX98_PTYS=y
CONFIG_UNIX98_PTY_COUNT=2048
CONFIG_PRINTER=y
CONFIG_LP_CONSOLE=y
CONFIG_PPDEV=y
CONFIG_NVRAM=y
CONFIG_RTC=y
CONFIG_AGP=y
CONFIG_AGP_SIS=y
CONFIG_RAW_DRIVER=y
CONFIG_EXT2_FS=y
CONFIG_EXT3_FS=y
CONFIG_JBD=y
CONFIG_MINIX_FS=y
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_ZISOFS_FS=y
CONFIG_UDF_FS=y
CONFIG_FAT_FS=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_PROC_FS=y
CONFIG_DEVPTS_FS=y
CONFIG_TMPFS=y
CONFIG_RAMFS=y
CONFIG_NFS_FS=y
CONFIG_NFS_V3=y
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_SUNRPC=y
CONFIG_SMB_FS=y
CONFIG_PARTITION_ADVANCED=y
CONFIG_MSDOS_PARTITION=y
CONFIG_SMB_NLS=y
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_ISO8859_1=y
CONFIG_NLS_ISO8859_15=y
CONFIG_NLS_UTF8=y
CONFIG_VIDEO_SELECT=y
CONFIG_VGA_CONSOLE=y
CONFIG_DUMMY_CONSOLE=y
CONFIG_SOUND=y
CONFIG_SND=y
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=y
CONFIG_SND_PCM_OSS=y
CONFIG_SND_RTCTIMER=y
CONFIG_SND_TRIDENT=y
CONFIG_USB=y
CONFIG_USB_DEVICEFS=y
CONFIG_USB_OHCI_HCD=y
CONFIG_USB_AUDIO=y
CONFIG_USB_ACM=y
CONFIG_USB_PRINTER=y
CONFIG_USB_STORAGE=y
CONFIG_USB_STORAGE_DATAFAB=y
CONFIG_USB_STORAGE_FREECOM=y
CONFIG_USB_STORAGE_ISD200=y
CONFIG_USB_STORAGE_DPCM=y
CONFIG_USB_STORAGE_HP8200e=y
CONFIG_USB_STORAGE_SDDR09=y
CONFIG_USB_STORAGE_JUMPSHOT=y
CONFIG_USB_HID=y
CONFIG_USB_HIDINPUT=y
CONFIG_DEBUG_KERNEL=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_KALLSYMS=y
CONFIG_ZLIB_INFLATE=y
CONFIG_X86_BIOS_REBOOT=y

-- 
/==============================\
| David Mansfield              |
| lkml@dm.cobite.com           |
\==============================/

Re: very poor performance in 2.5.66[-mm1]

[Andrew Morton]

David Mansfield <lkml@dm.cobite.com> wrote:
>
> 
> Hi list.
> 
> After all of the rave reviews about the interactivity fixes (both regular 
> and I/O scheduler related), I decided to give the 2.5.latest a try on my 
> desktop machine (system described below)
> 
> I started X, everything seemed fine, maybe a bit faster.  I opened a 
> 'gnome-terminal' and typed 'ls -ltr'.  Wow, it was 20x slower.
> 
> Here are the timings for 'ls -ltr':
> 
> 2.5.66-mm1:      'ls -ltr'         31 seconds
> 2.5.66-mm1:      'ls -ltr | cat'   2 seconds
> 2.4.18-rhlatest: 'ls -ltr'         1.14 seconds

How many files were there?

My /usr/bin contains 3168 files.  An `ls -ltr' in gnome-terminal takes 9.6
seconds.  In rxvt it takes 0.5 seconds.  That's an 850MHz P3.

So gnome-terminal appears to be a pretty slow application.  My guess would be
that something in the 2.5 kernel has exposed a marginality or an outright
bug in it.

It would be interesting to edit include/asm-i386/param.h and set HZ to 100.

Re: very poor performance in 2.5.66[-mm1]

[David Mansfield]

On Fri, 28 Mar 2003, Andrew Morton wrote:
> > After all of the rave reviews about the interactivity fixes (both regular 
> > and I/O scheduler related), I decided to give the 2.5.latest a try on my 
> > desktop machine (system described below)
> > 
> > I started X, everything seemed fine, maybe a bit faster.  I opened a 
> > 'gnome-terminal' and typed 'ls -ltr'.  Wow, it was 20x slower.
> > 
> > Here are the timings for 'ls -ltr':
> > 
> > 2.5.66-mm1:      'ls -ltr'         31 seconds
> > 2.5.66-mm1:      'ls -ltr | cat'   2 seconds
> > 2.4.18-rhlatest: 'ls -ltr'         1.14 seconds
> 
> How many files were there?

1337 files.

> My /usr/bin contains 3168 files.  An `ls -ltr' in gnome-terminal takes 9.6
> seconds.  In rxvt it takes 0.5 seconds.  That's an 850MHz P3.
> 
> So gnome-terminal appears to be a pretty slow application.  My guess would be
> that something in the 2.5 kernel has exposed a marginality or an outright
> bug in it.

Yes. gnome-terminal is godawful slow on RHAT 8.0 (it does Xrender
alpha-channel crap for every character to get the anti-aliasing).  But I
think the problem has to do with the pipe/pty wakeups.  After 'ls' writes
a line to the pty, it seems as though the gnome-terminal is being woken up
(even though 'ls' has more to write), it's generating the Xrender 
X-command and sending it to X.  X is waking up and rendering it (which 
forces a complete update of the screen).

Under 2.4.18-whatever, it would seem as though 'ls' is generating a large
number of lines of output before the gnome-terminal is waking up, causing
a dramatically fewer number of redraws.

> It would be interesting to edit include/asm-i386/param.h and set HZ to 100.
> 

I'll try to check this out over the weekend.

David

-- 
/==============================\
| David Mansfield              |
| lkml@dm.cobite.com           |
\==============================/

Re: very poor performance in 2.5.66[-mm1]

[jjs]

Just out of curiosity, does it help to say:

export LANG=en_US

?


David Mansfield wrote:

>On Fri, 28 Mar 2003, Andrew Morton wrote:
>  
>
>>>After all of the rave reviews about the interactivity fixes (both regular 
>>>and I/O scheduler related), I decided to give the 2.5.latest a try on my 
>>>desktop machine (system described below)
>>>
>>>I started X, everything seemed fine, maybe a bit faster.  I opened a 
>>>'gnome-terminal' and typed 'ls -ltr'.  Wow, it was 20x slower.
>>>
>>>Here are the timings for 'ls -ltr':
>>>
>>>2.5.66-mm1:      'ls -ltr'         31 seconds
>>>2.5.66-mm1:      'ls -ltr | cat'   2 seconds
>>>2.4.18-rhlatest: 'ls -ltr'         1.14 seconds
>>>      
>>>
>>How many files were there?
>>    
>>
>
>1337 files.
>
>  
>
>>My /usr/bin contains 3168 files.  An `ls -ltr' in gnome-terminal takes 9.6
>>seconds.  In rxvt it takes 0.5 seconds.  That's an 850MHz P3.
>>
>>So gnome-terminal appears to be a pretty slow application.  My guess would be
>>that something in the 2.5 kernel has exposed a marginality or an outright
>>bug in it.
>>    
>>
>
>Yes. gnome-terminal is godawful slow on RHAT 8.0 (it does Xrender
>alpha-channel crap for every character to get the anti-aliasing).  But I
>think the problem has to do with the pipe/pty wakeups.  After 'ls' writes
>a line to the pty, it seems as though the gnome-terminal is being woken up
>(even though 'ls' has more to write), it's generating the Xrender 
>X-command and sending it to X.  X is waking up and rendering it (which 
>forces a complete update of the screen).
>
>Under 2.4.18-whatever, it would seem as though 'ls' is generating a large
>number of lines of output before the gnome-terminal is waking up, causing
>a dramatically fewer number of redraws.
>
>  
>
>>It would be interesting to edit include/asm-i386/param.h and set HZ to 100.
>>
>>    
>>
>
>I'll try to check this out over the weekend.
>
>David
>
>  
>

Re: very poor performance in 2.5.66[-mm1]

[Ingo Molnar]

On Fri, 28 Mar 2003, David Mansfield wrote:

> Yes. gnome-terminal is godawful slow on RHAT 8.0 (it does Xrender
> alpha-channel crap for every character to get the anti-aliasing).  But I
> think the problem has to do with the pipe/pty wakeups.  After 'ls'
> writes a line to the pty, it seems as though the gnome-terminal is being
> woken up (even though 'ls' has more to write), it's generating the
> Xrender X-command and sending it to X.  X is waking up and rendering it
> (which forces a complete update of the screen).

this is a known bug in vte, fixed in the rawhide vte package. (you might
need to upgrade other packages as well.) Eg. try another, non-gnome-vte
based terminal, such as xterm or konsole, it wont show this problem.

	Ingo

Re: very poor performance in 2.5.66[-mm1]

[P]

David Mansfield wrote:
> Hi list.
> 
> After all of the rave reviews about the interactivity fixes (both regular 
> and I/O scheduler related), I decided to give the 2.5.latest a try on my 
> desktop machine (system described below)
> 
> I started X, everything seemed fine, maybe a bit faster.  I opened a 
> 'gnome-terminal' and typed 'ls -ltr'.  Wow, it was 20x slower.
> 
> Here are the timings for 'ls -ltr':
> 
> 2.5.66-mm1:      'ls -ltr'         31 seconds
> 2.5.66-mm1:      'ls -ltr | cat'   2 seconds
> 2.4.18-rhlatest: 'ls -ltr'         1.14 seconds

I've noticed this on all kernels and it seems scheduling related
hence why the latest triggers it for you. As far as I can see
most times writing data to gnome-terminal WHICH CAUSES IT TO SCROLL
it takes a ridiculous amount of time. If I take some CPU time away
from gnome-terminal by the official "window wiggling" method it
runs much faster. Note it's not rendering (antialiasing) related
as I turned that off with the same effect.

Pádraig.