* Inconsistent timing results of multithreaded program on an SMP machine.
@ 2005-11-20 9:27 Marcel Zalmanovici
2005-11-20 9:54 ` Con Kolivas
2005-11-20 14:02 ` Paul Jackson
0 siblings, 2 replies; 17+ messages in thread
From: Marcel Zalmanovici @ 2005-11-20 9:27 UTC (permalink / raw)
To: linux-kernel
[-- Attachment #1: Type: text/plain, Size: 1252 bytes --]
Hi,
I am trying, as part of my thesis, to make some improvement to the linux
scheduler.
Therefore I've written a small multithreaded application and tested how
long on average it takes for it to complete.
The results were very surprising. I expected to see the completion time
vary 1 to 2 seconds at most.
Instead what I've got was an oscillation where the maximum time was twice
and more than the minimum!! For a short test results ranged ~7sec to ~16
sec, and the same happened to longer tests where the min was ~1min and the
max ~2:30min.
Does anyone have any clue as to what might happen ?
Is there anything I can do to get stable results ?
Here is a small test case program:
(See attached file: sched_test.c)
The test was always done on a pretty much empty machine. I've tried both
kernel 2.6.4-52 and 2.6.13.4 but the results were the same.
I'm working on a Xeon Intel machine, dual processor, hyperthreaded.
Below is the info from /proc/cpuinfo
(See attached file: cpuinfo.log)
The machine's configuration is as follows: (filtered with grep "=[y|m]" )
(See attached file: conf_filtered)
Please respond to either:
marcel@il.ibm.com
or
marcel.mz@gmail.com
as I am not subscribed to the mailing list.
Thank you for your help,
Marcel
[-- Attachment #2: sched_test.c --]
[-- Type: application/octet-stream, Size: 1477 bytes --]
// Compile with:
// gcc -Wall -g -o sched_bind schedBind.c -lpthread
#include <pthread.h> /* needs to be before stdio.h */
#include <stdlib.h>
#include <sys/errno.h>
//#include <sys/processor.h>
//#include <sys/thread.h>
#include <stdio.h>
#include <sys/syscall.h>
#include <sys/types.h>
#include <unistd.h>
#include <linux/unistd.h>
_syscall0(pid_t,gettid)
pid_t gettid(void);
#define SIZE 7000
#define TIMES 10000
int arr[4][SIZE];
int idx;
void* compute_thread(void*);
int main(void)
{
int i;
pthread_t tid[8]; /* thread ID structure */
pthread_attr_t attr; /* thread attributes */
char thread[4][4] ={"0", "1", "2", "3"};
/* Initialize thread attribute structure */
//pthread_attr_setscope(&attr, PTHREAD_SCOPE_SYSTEM); // default anyway.
for (idx = 0; idx < 8; idx++ ) {
pthread_attr_init(&attr);
pthread_create(&(tid[idx]), &attr, compute_thread, thread[(idx/2)]);
}
/* main thread */
for (i= 0; i<8; i++)
pthread_join(tid[i], NULL);
printf("Main exit ...\n");
exit(0);
}
/* The thread function to be run */
void* compute_thread(void* dummy)
{
int i,j;
int procNum;
// int res;
procNum = atoi((char *)dummy);
printf("Thread DS %d TID = %d\n", procNum, gettid());
for (j=0; j<TIMES; j++) {
for(i= 0; i< SIZE; i++)
arr[procNum][(i*2345) % SIZE] = i;
}
printf("Thread DS %d, TID = %d running ended \n", procNum, gettid() );
return(void *)0;
}
[-- Attachment #3: cpuinfo.log --]
[-- Type: application/octet-stream, Size: 1956 bytes --]
processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 2
model name : Intel(R) Xeon(TM) CPU 3.20GHz
stepping : 5
cpu MHz : 3189.871
cache size : 512 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 1
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe cid xtpr
bogomips : 6385.56
processor : 1
vendor_id : GenuineIntel
cpu family : 15
model : 2
model name : Intel(R) Xeon(TM) CPU 3.20GHz
stepping : 5
cpu MHz : 3189.871
cache size : 512 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 1
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe cid xtpr
bogomips : 6379.18
processor : 2
vendor_id : GenuineIntel
cpu family : 15
model : 2
model name : Intel(R) Xeon(TM) CPU 3.20GHz
stepping : 5
cpu MHz : 3189.871
cache size : 512 KB
physical id : 3
siblings : 2
core id : 3
cpu cores : 1
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe cid xtpr
bogomips : 6379.50
processor : 3
vendor_id : GenuineIntel
cpu family : 15
model : 2
model name : Intel(R) Xeon(TM) CPU 3.20GHz
stepping : 5
cpu MHz : 3189.871
cache size : 512 KB
physical id : 3
siblings : 2
core id : 3
cpu cores : 1
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe cid xtpr
bogomips : 6379.46
[-- Attachment #4: conf_filtered --]
[-- Type: application/octet-stream, Size: 38313 bytes --]
CONFIG_X86=y
CONFIG_MMU=y
CONFIG_UID16=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_EXPERIMENTAL=y
CONFIG_CLEAN_COMPILE=y
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_SYSCTL=y
CONFIG_HOTPLUG=y
CONFIG_EVLOG=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_DELAY_ACCT=y
CONFIG_KALLSYMS=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
CONFIG_OBSOLETE_MODPARM=y
CONFIG_MODVERSIONS=y
CONFIG_KMOD=y
CONFIG_STOP_MACHINE=y
CONFIG_CKRM=y
CONFIG_CKRM_RES_DUMMY=m
CONFIG_CKRM_RES_SOCKETAQ=m
CONFIG_X86_GENERICARCH=y
CONFIG_X86_CYCLONE_TIMER=y
CONFIG_M586=y
CONFIG_X86_GENERIC=y
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_X86_PPRO_FENCE=y
CONFIG_X86_F00F_BUG=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_ALIGNMENT_16=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_SMP=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_P4THERMAL=y
CONFIG_TOSHIBA=m
CONFIG_I8K=m
CONFIG_MICROCODE=m
CONFIG_X86_MSR=m
CONFIG_X86_CPUID=m
CONFIG_EDD=m
CONFIG_HIGHMEM4G=y
CONFIG_HIGHMEM=y
CONFIG_HIGHPTE=y
CONFIG_MTRR=y
CONFIG_EFI=y
CONFIG_IRQBALANCE=y
CONFIG_HAVE_DEC_LOCK=y
CONFIG_BOOT_IOREMAP=y
CONFIG_REGPARM=y
CONFIG_PROC_MM=y
CONFIG_PM=y
CONFIG_PM_DISK=y
CONFIG_ACPI=y
CONFIG_ACPI_BOOT=y
CONFIG_ACPI_INTERPRETER=y
CONFIG_ACPI_SLEEP=y
CONFIG_ACPI_SLEEP_PROC_FS=y
CONFIG_ACPI_AC=m
CONFIG_ACPI_BATTERY=m
CONFIG_ACPI_BUTTON=m
CONFIG_ACPI_FAN=m
CONFIG_ACPI_PROCESSOR=m
CONFIG_ACPI_THERMAL=m
CONFIG_ACPI_ASUS=m
CONFIG_ACPI_TOSHIBA=m
CONFIG_ACPI_BUS=y
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_PCI=y
CONFIG_ACPI_SYSTEM=y
CONFIG_X86_PM_TIMER=y
CONFIG_ACPI_INITRD=y
CONFIG_APM=y
CONFIG_APM_DO_ENABLE=y
CONFIG_APM_DISPLAY_BLANK=y
CONFIG_APM_ALLOW_INTS=y
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_PROC_INTF=y
CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=m
CONFIG_CPU_FREQ_GOV_USERSPACE=m
CONFIG_CPU_FREQ_TABLE=m
CONFIG_X86_ACPI_CPUFREQ=m
CONFIG_X86_POWERNOW_K6=m
CONFIG_X86_POWERNOW_K7=m
CONFIG_X86_POWERNOW_K8=m
CONFIG_X86_POWERNOW_K8_ACPI=y
CONFIG_X86_GX_SUSPMOD=m
CONFIG_X86_SPEEDSTEP_CENTRINO=m
CONFIG_X86_SPEEDSTEP_ICH=m
CONFIG_X86_SPEEDSTEP_SMI=m
CONFIG_X86_P4_CLOCKMOD=m
CONFIG_X86_SPEEDSTEP_LIB=m
CONFIG_X86_LONGRUN=m
CONFIG_X86_LONGHAUL=m
CONFIG_PCI=y
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_ISA=y
CONFIG_SCx200=m
CONFIG_PCMCIA=m
CONFIG_YENTA=m
CONFIG_CARDBUS=y
CONFIG_I82092=m
CONFIG_I82365=m
CONFIG_TCIC=m
CONFIG_PCMCIA_PROBE=y
CONFIG_HOTPLUG_PCI=m
CONFIG_HOTPLUG_PCI_FAKE=m
CONFIG_HOTPLUG_PCI_COMPAQ=m
CONFIG_HOTPLUG_PCI_COMPAQ_NVRAM=y
CONFIG_HOTPLUG_PCI_IBM=m
CONFIG_HOTPLUG_PCI_ACPI=m
CONFIG_HOTPLUG_PCI_CPCI=y
CONFIG_HOTPLUG_PCI_CPCI_ZT5550=m
CONFIG_HOTPLUG_PCI_CPCI_GENERIC=m
CONFIG_HOTPLUG_PCI_PCIE=m
CONFIG_HOTPLUG_PCI_SHPC=m
CONFIG_BINFMT_ELF=y
CONFIG_BINFMT_AOUT=m
CONFIG_BINFMT_MISC=m
CONFIG_FW_LOADER=m
CONFIG_MTD=m
CONFIG_MTD_PARTITIONS=m
CONFIG_MTD_CONCAT=m
CONFIG_MTD_REDBOOT_PARTS=m
CONFIG_MTD_CMDLINE_PARTS=m
CONFIG_MTD_CHAR=m
CONFIG_MTD_BLOCK=m
CONFIG_FTL=m
CONFIG_INFTL=m
CONFIG_MTD_CFI=m
CONFIG_MTD_JEDECPROBE=m
CONFIG_MTD_GEN_PROBE=m
CONFIG_MTD_CFI_ADV_OPTIONS=y
CONFIG_MTD_CFI_NOSWAP=y
CONFIG_MTD_CFI_INTELEXT=m
CONFIG_MTD_CFI_AMDSTD=m
CONFIG_MTD_CFI_STAA=m
CONFIG_MTD_ABSENT=m
CONFIG_MTD_OBSOLETE_CHIPS=y
CONFIG_MTD_AMDSTD=m
CONFIG_MTD_SHARP=m
CONFIG_MTD_JEDEC=m
CONFIG_MTD_COMPLEX_MAPPINGS=y
CONFIG_MTD_PHYSMAP=m
CONFIG_MTD_PNC2000=m
CONFIG_MTD_SC520CDP=m
CONFIG_MTD_NETSC520=m
CONFIG_MTD_SBC_GXX=m
CONFIG_MTD_ELAN_104NC=m
CONFIG_MTD_OCTAGON=m
CONFIG_MTD_VMAX=m
CONFIG_MTD_SCx200_DOCFLASH=m
CONFIG_MTD_AMD76XROM=m
CONFIG_MTD_ICH2ROM=m
CONFIG_MTD_SCB2_FLASH=m
CONFIG_MTD_NETtel=m
CONFIG_MTD_DILNETPC=m
CONFIG_MTD_L440GX=m
CONFIG_MTD_PCI=m
CONFIG_MTD_PMC551=m
CONFIG_MTD_PMC551_BUGFIX=y
CONFIG_MTD_SLRAM=m
CONFIG_MTD_MTDRAM=m
CONFIG_MTD_BLKMTD=m
CONFIG_MTD_DOC2000=m
CONFIG_MTD_DOC2001=m
CONFIG_MTD_DOC2001PLUS=m
CONFIG_MTD_DOCPROBE=m
CONFIG_MTD_DOCPROBE_ADVANCED=y
CONFIG_MTD_DOCPROBE_HIGH=y
CONFIG_MTD_DOCPROBE_55AA=y
CONFIG_MTD_NAND=m
CONFIG_MTD_NAND_IDS=m
CONFIG_PARPORT=m
CONFIG_PARPORT_PC=m
CONFIG_PARPORT_PC_CML1=m
CONFIG_PARPORT_SERIAL=m
CONFIG_PARPORT_PC_FIFO=y
CONFIG_PARPORT_PC_SUPERIO=y
CONFIG_PARPORT_PC_PCMCIA=m
CONFIG_PARPORT_OTHER=y
CONFIG_PARPORT_1284=y
CONFIG_PNP=y
CONFIG_ISAPNP=y
CONFIG_BLK_DEV_FD=y
CONFIG_BLK_DEV_XD=m
CONFIG_PARIDE=m
CONFIG_PARIDE_PARPORT=m
CONFIG_PARIDE_PD=m
CONFIG_PARIDE_PCD=m
CONFIG_PARIDE_PF=m
CONFIG_PARIDE_PT=m
CONFIG_PARIDE_PG=m
CONFIG_PARIDE_ATEN=m
CONFIG_PARIDE_BPCK=m
CONFIG_PARIDE_BPCK6=m
CONFIG_PARIDE_COMM=m
CONFIG_PARIDE_DSTR=m
CONFIG_PARIDE_FIT2=m
CONFIG_PARIDE_FIT3=m
CONFIG_PARIDE_EPAT=m
CONFIG_PARIDE_EPATC8=y
CONFIG_PARIDE_EPIA=m
CONFIG_PARIDE_FRIQ=m
CONFIG_PARIDE_FRPW=m
CONFIG_PARIDE_KBIC=m
CONFIG_PARIDE_KTTI=m
CONFIG_PARIDE_ON20=m
CONFIG_PARIDE_ON26=m
CONFIG_BLK_CPQ_DA=m
CONFIG_BLK_CPQ_CISS_DA=m
CONFIG_CISS_SCSI_TAPE=y
CONFIG_BLK_DEV_DAC960=m
CONFIG_BLK_DEV_UMEM=m
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_CRYPTOLOOP=m
CONFIG_BLK_DEV_NBD=m
CONFIG_BLK_DEV_CARMEL=m
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_LBD=y
CONFIG_CIPHER_TWOFISH=m
CONFIG_IDE=y
CONFIG_BLK_DEV_IDE=y
CONFIG_BLK_DEV_IDEDISK=y
CONFIG_IDEDISK_MULTI_MODE=y
CONFIG_IDEDISK_STROKE=y
CONFIG_BLK_DEV_IDECS=m
CONFIG_BLK_DEV_IDECD=m
CONFIG_BLK_DEV_IDETAPE=m
CONFIG_BLK_DEV_IDEFLOPPY=y
CONFIG_BLK_DEV_IDESCSI=m
CONFIG_IDE_GENERIC=y
CONFIG_BLK_DEV_CMD640=y
CONFIG_BLK_DEV_CMD640_ENHANCED=y
CONFIG_BLK_DEV_IDEPNP=y
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_IDEPCI_SHARE_IRQ=y
CONFIG_BLK_DEV_OFFBOARD=y
CONFIG_BLK_DEV_GENERIC=y
CONFIG_BLK_DEV_RZ1000=y
CONFIG_BLK_DEV_IDEDMA_PCI=y
CONFIG_IDEDMA_PCI_AUTO=y
CONFIG_IDEDMA_ONLYDISK=y
CONFIG_BLK_DEV_ADMA=y
CONFIG_BLK_DEV_AEC62XX=y
CONFIG_BLK_DEV_ALI15X3=y
CONFIG_BLK_DEV_AMD74XX=y
CONFIG_BLK_DEV_ATIIXP=y
CONFIG_BLK_DEV_CMD64X=y
CONFIG_BLK_DEV_TRIFLEX=y
CONFIG_BLK_DEV_CY82C693=y
CONFIG_BLK_DEV_CS5520=m
CONFIG_BLK_DEV_CS5530=m
CONFIG_BLK_DEV_HPT34X=y
CONFIG_HPT34X_AUTODMA=y
CONFIG_BLK_DEV_HPT366=y
CONFIG_BLK_DEV_SC1200=y
CONFIG_BLK_DEV_PIIX=y
CONFIG_BLK_DEV_NS87415=y
CONFIG_BLK_DEV_PDC202XX_OLD=y
CONFIG_PDC202XX_BURST=y
CONFIG_BLK_DEV_PDC202XX_NEW=y
CONFIG_PDC202XX_FORCE=y
CONFIG_BLK_DEV_SVWKS=y
CONFIG_BLK_DEV_SIIMAGE=y
CONFIG_BLK_DEV_SIS5513=y
CONFIG_BLK_DEV_SLC90E66=y
CONFIG_BLK_DEV_VIA82CXXX=y
CONFIG_IDE_CHIPSETS=y
CONFIG_BLK_DEV_4DRIVES=y
CONFIG_BLK_DEV_ALI14XX=y
CONFIG_BLK_DEV_DTC2278=y
CONFIG_BLK_DEV_HT6560B=y
CONFIG_BLK_DEV_QD65XX=y
CONFIG_BLK_DEV_UMC8672=y
CONFIG_BLK_DEV_IDEDMA=y
CONFIG_IDEDMA_AUTO=y
CONFIG_SCSI=m
CONFIG_SCSI_PROC_FS=y
CONFIG_BLK_DEV_SD=m
CONFIG_CHR_DEV_ST=m
CONFIG_CHR_DEV_OSST=m
CONFIG_BLK_DEV_SR=m
CONFIG_CHR_DEV_SG=m
CONFIG_CHR_DEV_SCH=m
CONFIG_SCSI_MULTI_LUN=y
CONFIG_SCSI_REPORT_LUNS=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
CONFIG_SCSI_SPI_ATTRS=m
CONFIG_SCSI_FC_ATTRS=m
CONFIG_BLK_DEV_3W_XXXX_RAID=m
CONFIG_SCSI_7000FASST=m
CONFIG_SCSI_ACARD=m
CONFIG_SCSI_AHA152X=m
CONFIG_SCSI_AHA1542=m
CONFIG_SCSI_AACRAID=m
CONFIG_SCSI_AIC7XXX=m
CONFIG_AIC7XXX_REG_PRETTY_PRINT=y
CONFIG_SCSI_AIC7XXX_OLD=m
CONFIG_SCSI_AIC79XX=m
CONFIG_AIC79XX_REG_PRETTY_PRINT=y
CONFIG_SCSI_ADVANSYS=m
CONFIG_SCSI_IN2000=m
CONFIG_SCSI_MEGARAID=m
CONFIG_SCSI_SATA=y
CONFIG_SCSI_SATA_SVW=m
CONFIG_SCSI_ATA_PIIX=m
CONFIG_SCSI_SATA_PROMISE=m
CONFIG_SCSI_SATA_SIL=m
CONFIG_SCSI_SATA_SIS=m
CONFIG_SCSI_SATA_VIA=m
CONFIG_SCSI_SATA_VITESSE=m
CONFIG_SCSI_BUSLOGIC=m
CONFIG_SCSI_DMX3191D=m
CONFIG_SCSI_DTC3280=m
CONFIG_SCSI_EATA=m
CONFIG_SCSI_EATA_TAGGED_QUEUE=y
CONFIG_SCSI_EATA_LINKED_COMMANDS=y
CONFIG_SCSI_EATA_PIO=m
CONFIG_SCSI_FUTURE_DOMAIN=m
CONFIG_SCSI_GDTH=m
CONFIG_SCSI_GENERIC_NCR5380=m
CONFIG_SCSI_GENERIC_NCR5380_MMIO=m
CONFIG_SCSI_GENERIC_NCR53C400=y
CONFIG_SCSI_IPS=m
CONFIG_SCSI_PPA=m
CONFIG_SCSI_IMM=m
CONFIG_SCSI_NCR53C406A=m
CONFIG_SCSI_SYM53C8XX_2=m
CONFIG_SCSI_LPFC=m
CONFIG_SCSI_IPR=m
CONFIG_SCSI_IPR_TRACE=y
CONFIG_SCSI_IPR_DUMP=y
CONFIG_SCSI_PAS16=m
CONFIG_SCSI_PSI240I=m
CONFIG_SCSI_QLOGIC_FAS=m
CONFIG_SCSI_QLOGIC_ISP=m
CONFIG_SCSI_QLOGIC_FC=m
CONFIG_SCSI_QLOGIC_FC_FIRMWARE=y
CONFIG_SCSI_QLOGIC_1280=m
CONFIG_SCSI_QLA2XXX=m
CONFIG_SCSI_QLA21XX=m
CONFIG_SCSI_QLA22XX=m
CONFIG_SCSI_QLA2300=m
CONFIG_SCSI_QLA2322=m
CONFIG_SCSI_QLA6312=m
CONFIG_SCSI_QLA6322=m
CONFIG_SCSI_QLA2XXX_FAILOVER=y
CONFIG_SCSI_SYM53C416=m
CONFIG_SCSI_DC395x=m
CONFIG_SCSI_DC390T=m
CONFIG_SCSI_T128=m
CONFIG_SCSI_U14_34F=m
CONFIG_SCSI_U14_34F_TAGGED_QUEUE=y
CONFIG_SCSI_U14_34F_LINKED_COMMANDS=y
CONFIG_SCSI_ULTRASTOR=m
CONFIG_SCSI_NSP32=m
CONFIG_SCSI_DEBUG=m
CONFIG_PCMCIA_AHA152X=m
CONFIG_PCMCIA_FDOMAIN=m
CONFIG_PCMCIA_NINJA_SCSI=m
CONFIG_PCMCIA_QLOGIC=m
CONFIG_CD_NO_IDESCSI=y
CONFIG_AZTCD=m
CONFIG_GSCD=m
CONFIG_MCD=m
CONFIG_MCDX=m
CONFIG_OPTCD=m
CONFIG_SJCD=m
CONFIG_ISP16_CDI=m
CONFIG_CDU535=m
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_LINEAR=m
CONFIG_MD_RAID0=m
CONFIG_MD_RAID1=m
CONFIG_MD_RAID5=m
CONFIG_MD_RAID6=m
CONFIG_MD_MULTIPATH=m
CONFIG_BLK_DEV_DM=m
CONFIG_DM_CRYPT=m
CONFIG_DM_MULTIPATH=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_MIRROR=m
CONFIG_DM_ZERO=m
CONFIG_DM_FLAKEY=m
CONFIG_FUSION=m
CONFIG_FUSION_ISENSE=m
CONFIG_FUSION_CTL=m
CONFIG_FUSION_LAN=m
CONFIG_IEEE1394=m
CONFIG_IEEE1394_EXTRA_CONFIG_ROMS=y
CONFIG_IEEE1394_CONFIG_ROM_IP1394=y
CONFIG_IEEE1394_PCILYNX=m
CONFIG_IEEE1394_OHCI1394=m
CONFIG_IEEE1394_VIDEO1394=m
CONFIG_IEEE1394_SBP2=m
CONFIG_IEEE1394_ETH1394=m
CONFIG_IEEE1394_DV1394=m
CONFIG_IEEE1394_RAWIO=m
CONFIG_IEEE1394_CMP=m
CONFIG_IEEE1394_AMDTP=m
CONFIG_I2O=m
CONFIG_I2O_PCI=m
CONFIG_I2O_BLOCK=m
CONFIG_I2O_SCSI=m
CONFIG_I2O_PROC=m
CONFIG_NET=y
CONFIG_PACKET=m
CONFIG_PACKET_MMAP=y
CONFIG_NETLINK_DEV=m
CONFIG_UNIX=y
CONFIG_NET_KEY=m
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_FWMARK=y
CONFIG_IP_ROUTE_NAT=y
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_TOS=y
CONFIG_IP_ROUTE_VERBOSE=y
CONFIG_NET_IPIP=m
CONFIG_NET_IPGRE=m
CONFIG_NET_IPGRE_BROADCAST=y
CONFIG_IP_MROUTE=y
CONFIG_IP_PIMSM_V1=y
CONFIG_IP_PIMSM_V2=y
CONFIG_SYN_COOKIES=y
CONFIG_INET_AH=m
CONFIG_INET_ESP=m
CONFIG_INET_IPCOMP=m
CONFIG_IP_VS=m
CONFIG_IP_VS_PROTO_TCP=y
CONFIG_IP_VS_PROTO_UDP=y
CONFIG_IP_VS_PROTO_ESP=y
CONFIG_IP_VS_PROTO_AH=y
CONFIG_IP_VS_RR=m
CONFIG_IP_VS_WRR=m
CONFIG_IP_VS_LC=m
CONFIG_IP_VS_WLC=m
CONFIG_IP_VS_LBLC=m
CONFIG_IP_VS_LBLCR=m
CONFIG_IP_VS_DH=m
CONFIG_IP_VS_SH=m
CONFIG_IP_VS_SED=m
CONFIG_IP_VS_NQ=m
CONFIG_IP_VS_FTP=m
CONFIG_IPV6=m
CONFIG_IPV6_SUBTREES=y
CONFIG_IPV6_PRIVACY=y
CONFIG_IPV6_NDISC_NEW=y
CONFIG_INET6_AH=m
CONFIG_INET6_ESP=m
CONFIG_INET6_IPCOMP=m
CONFIG_IPV6_TUNNEL=m
CONFIG_IPV6_MOBILITY=m
CONFIG_IPV6_MOBILITY_MN=m
CONFIG_IPV6_MOBILITY_HA=m
CONFIG_DECNET=m
CONFIG_DECNET_SIOCGIFCONF=y
CONFIG_BRIDGE=m
CONFIG_NETFILTER=y
CONFIG_BRIDGE_NETFILTER=y
CONFIG_IP_NF_CONNTRACK=m
CONFIG_IP_NF_FTP=m
CONFIG_IP_NF_IRC=m
CONFIG_IP_NF_TFTP=m
CONFIG_IP_NF_AMANDA=m
CONFIG_IP_NF_QUEUE=m
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_LIMIT=m
CONFIG_IP_NF_MATCH_IPRANGE=m
CONFIG_IP_NF_MATCH_MAC=m
CONFIG_IP_NF_MATCH_PKTTYPE=m
CONFIG_IP_NF_MATCH_MARK=m
CONFIG_IP_NF_MATCH_MULTIPORT=m
CONFIG_IP_NF_MATCH_TOS=m
CONFIG_IP_NF_MATCH_RECENT=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_DSCP=m
CONFIG_IP_NF_MATCH_AH_ESP=m
CONFIG_IP_NF_MATCH_LENGTH=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_MATCH_TCPMSS=m
CONFIG_IP_NF_MATCH_HELPER=m
CONFIG_IP_NF_MATCH_STATE=m
CONFIG_IP_NF_MATCH_CONNTRACK=m
CONFIG_IP_NF_MATCH_OWNER=m
CONFIG_IP_NF_MATCH_PHYSDEV=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_NAT=m
CONFIG_IP_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_IP_NF_TARGET_SAME=m
CONFIG_IP_NF_NAT_SNMP_BASIC=m
CONFIG_IP_NF_NAT_IRC=m
CONFIG_IP_NF_NAT_FTP=m
CONFIG_IP_NF_NAT_TFTP=m
CONFIG_IP_NF_NAT_AMANDA=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_TOS=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_DSCP=m
CONFIG_IP_NF_TARGET_MARK=m
CONFIG_IP_NF_TARGET_CLASSIFY=m
CONFIG_IP_NF_TARGET_LOG=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_IP_NF_TARGET_TCPMSS=m
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m
CONFIG_IP_NF_COMPAT_IPCHAINS=m
CONFIG_IP_NF_COMPAT_IPFWADM=m
CONFIG_IP6_NF_FTP=m
CONFIG_IP6_NF_QUEUE=m
CONFIG_IP6_NF_IPTABLES=m
CONFIG_IP6_NF_MATCH_LIMIT=m
CONFIG_IP6_NF_MATCH_MAC=m
CONFIG_IP6_NF_MATCH_RT=m
CONFIG_IP6_NF_MATCH_OPTS=m
CONFIG_IP6_NF_MATCH_FRAG=m
CONFIG_IP6_NF_MATCH_HL=m
CONFIG_IP6_NF_MATCH_MULTIPORT=m
CONFIG_IP6_NF_MATCH_OWNER=m
CONFIG_IP6_NF_MATCH_MARK=m
CONFIG_IP6_NF_MATCH_IPV6HEADER=m
CONFIG_IP6_NF_MATCH_AHESP=m
CONFIG_IP6_NF_MATCH_LENGTH=m
CONFIG_IP6_NF_MATCH_EUI64=m
CONFIG_IP6_NF_CONNTRACK=m
CONFIG_IP6_NF_MATCH_STATE=m
CONFIG_IP6_NF_FILTER=m
CONFIG_IP6_NF_TARGET_LOG=m
CONFIG_IP6_NF_TARGET_REJECT=m
CONFIG_IP6_NF_MANGLE=m
CONFIG_IP6_NF_TARGET_MARK=m
CONFIG_DECNET_NF_GRABULATOR=m
CONFIG_BRIDGE_NF_EBTABLES=m
CONFIG_BRIDGE_EBT_BROUTE=m
CONFIG_BRIDGE_EBT_T_FILTER=m
CONFIG_BRIDGE_EBT_T_NAT=m
CONFIG_BRIDGE_EBT_802_3=m
CONFIG_BRIDGE_EBT_AMONG=m
CONFIG_BRIDGE_EBT_ARP=m
CONFIG_BRIDGE_EBT_IP=m
CONFIG_BRIDGE_EBT_LIMIT=m
CONFIG_BRIDGE_EBT_MARK=m
CONFIG_BRIDGE_EBT_PKTTYPE=m
CONFIG_BRIDGE_EBT_STP=m
CONFIG_BRIDGE_EBT_VLAN=m
CONFIG_BRIDGE_EBT_ARPREPLY=m
CONFIG_BRIDGE_EBT_DNAT=m
CONFIG_BRIDGE_EBT_MARK_T=m
CONFIG_BRIDGE_EBT_REDIRECT=m
CONFIG_BRIDGE_EBT_SNAT=m
CONFIG_BRIDGE_EBT_LOG=m
CONFIG_XFRM=y
CONFIG_XFRM_USER=m
CONFIG_IP_SCTP=m
CONFIG_SCTP_HMAC_MD5=y
CONFIG_ATM=y
CONFIG_ATM_CLIP=y
CONFIG_ATM_CLIP_NO_ICMP=y
CONFIG_ATM_LANE=m
CONFIG_ATM_MPOA=m
CONFIG_ATM_BR2684=m
CONFIG_VLAN_8021Q=m
CONFIG_LLC=y
CONFIG_LLC2=m
CONFIG_IPX=m
CONFIG_ATALK=m
CONFIG_DEV_APPLETALK=y
CONFIG_LTPC=m
CONFIG_COPS=m
CONFIG_COPS_DAYNA=y
CONFIG_COPS_TANGENT=y
CONFIG_IPDDP=m
CONFIG_IPDDP_ENCAP=y
CONFIG_IPDDP_DECAP=y
CONFIG_X25=m
CONFIG_LAPB=m
CONFIG_ECONET=m
CONFIG_WAN_ROUTER=m
CONFIG_NET_SCHED=y
CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
CONFIG_NET_SCH_HFSC=m
CONFIG_NET_SCH_CSZ=m
CONFIG_NET_SCH_ATM=y
CONFIG_NET_SCH_PRIO=m
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
CONFIG_NET_SCH_DELAY=m
CONFIG_NET_SCH_INGRESS=m
CONFIG_NET_QOS=y
CONFIG_NET_ESTIMATOR=y
CONFIG_NET_CLS=y
CONFIG_NET_CLS_TCINDEX=m
CONFIG_NET_CLS_ROUTE4=m
CONFIG_NET_CLS_ROUTE=y
CONFIG_NET_CLS_FW=m
CONFIG_NET_CLS_U32=m
CONFIG_NET_CLS_RSVP=m
CONFIG_NET_CLS_RSVP6=m
CONFIG_NET_CLS_POLICE=y
CONFIG_NET_PKTGEN=m
CONFIG_NETDEVICES=y
CONFIG_ARCNET=m
CONFIG_ARCNET_1201=m
CONFIG_ARCNET_1051=m
CONFIG_ARCNET_RAW=m
CONFIG_ARCNET_COM90xx=m
CONFIG_ARCNET_COM90xxIO=m
CONFIG_ARCNET_RIM_I=m
CONFIG_DUMMY=m
CONFIG_BONDING=m
CONFIG_EQUALIZER=m
CONFIG_TUN=m
CONFIG_ETHERTAP=m
CONFIG_NET_SB1000=m
CONFIG_NET_ETHERNET=y
CONFIG_MII=m
CONFIG_HAPPYMEAL=m
CONFIG_SUNGEM=m
CONFIG_NET_VENDOR_3COM=y
CONFIG_EL1=m
CONFIG_EL2=m
CONFIG_ELPLUS=m
CONFIG_EL16=m
CONFIG_EL3=m
CONFIG_3C515=m
CONFIG_VORTEX=m
CONFIG_TYPHOON=m
CONFIG_LANCE=m
CONFIG_NET_VENDOR_SMC=y
CONFIG_WD80x3=m
CONFIG_ULTRA=m
CONFIG_SMC9194=m
CONFIG_NET_VENDOR_RACAL=y
CONFIG_NI52=m
CONFIG_NI65=m
CONFIG_NET_TULIP=y
CONFIG_DE2104X=m
CONFIG_TULIP=m
CONFIG_TULIP_NAPI=y
CONFIG_TULIP_NAPI_HW_MITIGATION=y
CONFIG_DE4X5=m
CONFIG_WINBOND_840=m
CONFIG_DM9102=m
CONFIG_PCMCIA_XIRCOM=m
CONFIG_AT1700=m
CONFIG_DEPCA=m
CONFIG_HP100=m
CONFIG_NET_ISA=y
CONFIG_E2100=m
CONFIG_EWRK3=m
CONFIG_EEXPRESS=m
CONFIG_EEXPRESS_PRO=m
CONFIG_HPLAN_PLUS=m
CONFIG_HPLAN=m
CONFIG_LP486E=m
CONFIG_ETH16I=m
CONFIG_NE2000=m
CONFIG_ZNET=m
CONFIG_SEEQ8005=m
CONFIG_NET_PCI=y
CONFIG_PCNET32=m
CONFIG_AMD8111_ETH=m
CONFIG_ADAPTEC_STARFIRE=m
CONFIG_ADAPTEC_STARFIRE_NAPI=y
CONFIG_AC3200=m
CONFIG_APRICOT=m
CONFIG_B44=m
CONFIG_FORCEDETH=m
CONFIG_CS89x0=m
CONFIG_DGRS=m
CONFIG_EEPRO100=m
CONFIG_E100=m
CONFIG_E100_NAPI=y
CONFIG_FEALNX=m
CONFIG_NATSEMI=m
CONFIG_NE2K_PCI=m
CONFIG_8139CP=m
CONFIG_8139TOO=m
CONFIG_8139TOO_8129=y
CONFIG_SIS900=m
CONFIG_EPIC100=m
CONFIG_SUNDANCE=m
CONFIG_TLAN=m
CONFIG_VIA_RHINE=m
CONFIG_NET_POCKET=y
CONFIG_ATP=m
CONFIG_DE600=m
CONFIG_DE620=m
CONFIG_ACENIC=m
CONFIG_DL2K=m
CONFIG_E1000=m
CONFIG_E1000_NAPI=y
CONFIG_NS83820=m
CONFIG_HAMACHI=m
CONFIG_YELLOWFIN=m
CONFIG_R8169=m
CONFIG_SIS190=m
CONFIG_SK98LIN=m
CONFIG_TIGON3=m
CONFIG_NET_BROADCOM=m
CONFIG_NET_BCM44=m
CONFIG_IXGB=m
CONFIG_IXGB_NAPI=y
CONFIG_S2IO=m
CONFIG_S2IO_NAPI=y
CONFIG_FDDI=y
CONFIG_SKFP=m
CONFIG_HIPPI=y
CONFIG_ROADRUNNER=m
CONFIG_ROADRUNNER_LARGE_RINGS=y
CONFIG_PLIP=m
CONFIG_PPP=m
CONFIG_PPP_MULTILINK=y
CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=m
CONFIG_PPP_SYNC_TTY=m
CONFIG_PPP_DEFLATE=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_MPPE=m
CONFIG_PPPOE=m
CONFIG_PPPOATM=m
CONFIG_SLIP=m
CONFIG_SLIP_COMPRESSED=y
CONFIG_SLIP_SMART=y
CONFIG_SLIP_MODE_SLIP6=y
CONFIG_NET_RADIO=y
CONFIG_STRIP=m
CONFIG_WAVELAN=m
CONFIG_PCMCIA_WAVELAN=m
CONFIG_PCMCIA_NETWAVE=m
CONFIG_PCMCIA_RAYCS=m
CONFIG_AIRO=m
CONFIG_HERMES=m
CONFIG_PLX_HERMES=m
CONFIG_TMD_HERMES=m
CONFIG_PCI_HERMES=m
CONFIG_ATMEL=m
CONFIG_PCI_ATMEL=m
CONFIG_PCMCIA_HERMES=m
CONFIG_AIRO_CS=m
CONFIG_PCMCIA_ATMEL=m
CONFIG_PCMCIA_WL3501=m
CONFIG_PRISM54=m
CONFIG_NET_WIRELESS=y
CONFIG_TR=y
CONFIG_IBMTR=m
CONFIG_IBMOL=m
CONFIG_IBMLS=m
CONFIG_3C359=m
CONFIG_TMS380TR=m
CONFIG_TMSPCI=m
CONFIG_SKISA=m
CONFIG_PROTEON=m
CONFIG_ABYSS=m
CONFIG_SMCTR=m
CONFIG_NET_FC=y
CONFIG_NET_LPFC=m
CONFIG_RCPCI=m
CONFIG_SHAPER=m
CONFIG_NETCONSOLE=m
CONFIG_WAN=y
CONFIG_HOSTESS_SV11=m
CONFIG_LANMEDIA=m
CONFIG_SEALEVEL_4021=m
CONFIG_SYNCLINK_SYNCPPP=m
CONFIG_HDLC=m
CONFIG_HDLC_RAW=y
CONFIG_HDLC_RAW_ETH=y
CONFIG_HDLC_CISCO=y
CONFIG_HDLC_FR=y
CONFIG_HDLC_PPP=y
CONFIG_HDLC_X25=y
CONFIG_PCI200SYN=m
CONFIG_WANXL=m
CONFIG_N2=m
CONFIG_C101=m
CONFIG_FARSYNC=m
CONFIG_DLCI=m
CONFIG_SDLA=m
CONFIG_LAPBETHER=m
CONFIG_X25_ASY=m
CONFIG_NET_PCMCIA=y
CONFIG_PCMCIA_3C589=m
CONFIG_PCMCIA_3C574=m
CONFIG_PCMCIA_FMVJ18X=m
CONFIG_PCMCIA_PCNET=m
CONFIG_PCMCIA_NMCLAN=m
CONFIG_PCMCIA_SMC91C92=m
CONFIG_PCMCIA_XIRC2PS=m
CONFIG_PCMCIA_AXNET=m
CONFIG_PCMCIA_IBMTR=m
CONFIG_ATM_TCP=m
CONFIG_ATM_LANAI=m
CONFIG_ATM_ENI=m
CONFIG_ATM_FIRESTREAM=m
CONFIG_ATM_ZATM=m
CONFIG_ATM_NICSTAR=m
CONFIG_ATM_NICSTAR_USE_SUNI=y
CONFIG_ATM_NICSTAR_USE_IDT77105=y
CONFIG_ATM_IDT77252=m
CONFIG_ATM_IDT77252_RCV_ALL=y
CONFIG_ATM_IDT77252_USE_SUNI=y
CONFIG_ATM_AMBASSADOR=m
CONFIG_ATM_HORIZON=m
CONFIG_ATM_IA=m
CONFIG_ATM_FORE200E_MAYBE=m
CONFIG_ATM_FORE200E_PCA=y
CONFIG_ATM_FORE200E_PCA_DEFAULT_FW=y
CONFIG_ATM_FORE200E=m
CONFIG_ATM_HE=m
CONFIG_ATM_HE_USE_SUNI=y
CONFIG_HAMRADIO=y
CONFIG_AX25=m
CONFIG_AX25_DAMA_SLAVE=y
CONFIG_NETROM=m
CONFIG_ROSE=m
CONFIG_BPQETHER=m
CONFIG_SCC=m
CONFIG_SCC_DELAY=y
CONFIG_SCC_TRXECHO=y
CONFIG_BAYCOM_SER_FDX=m
CONFIG_BAYCOM_SER_HDX=m
CONFIG_BAYCOM_PAR=m
CONFIG_BAYCOM_EPP=m
CONFIG_YAM=m
CONFIG_IRDA=m
CONFIG_IRLAN=m
CONFIG_IRNET=m
CONFIG_IRCOMM=m
CONFIG_IRDA_ULTRA=y
CONFIG_IRDA_CACHE_LAST_LSAP=y
CONFIG_IRTTY_SIR=m
CONFIG_DONGLE=y
CONFIG_ESI_DONGLE=m
CONFIG_ACTISYS_DONGLE=m
CONFIG_TEKRAM_DONGLE=m
CONFIG_LITELINK_DONGLE=m
CONFIG_MA600_DONGLE=m
CONFIG_GIRBIL_DONGLE=m
CONFIG_MCP2120_DONGLE=m
CONFIG_OLD_BELKIN_DONGLE=m
CONFIG_ACT200L_DONGLE=m
CONFIG_USB_IRDA=m
CONFIG_SIGMATEL_FIR=m
CONFIG_NSC_FIR=m
CONFIG_WINBOND_FIR=m
CONFIG_TOSHIBA_FIR=m
CONFIG_SMC_IRCC_FIR=m
CONFIG_ALI_FIR=m
CONFIG_VLSI_FIR=m
CONFIG_VIA_FIR=m
CONFIG_BT=m
CONFIG_BT_L2CAP=m
CONFIG_BT_SCO=m
CONFIG_BT_RFCOMM=m
CONFIG_BT_RFCOMM_TTY=y
CONFIG_BT_BNEP=m
CONFIG_BT_BNEP_MC_FILTER=y
CONFIG_BT_BNEP_PROTO_FILTER=y
CONFIG_BT_CMTP=m
CONFIG_BT_HCIUSB=m
CONFIG_BT_HCIUSB_SCO=y
CONFIG_BT_HCIUART=m
CONFIG_BT_HCIUART_H4=y
CONFIG_BT_HCIUART_BCSP=y
CONFIG_BT_HCIUART_BCSP_TXCRC=y
CONFIG_BT_HCIBCM203X=m
CONFIG_BT_HCIBFUSB=m
CONFIG_BT_HCIDTL1=m
CONFIG_BT_HCIBT3C=m
CONFIG_BT_HCIBLUECARD=m
CONFIG_BT_HCIBTUART=m
CONFIG_BT_HCIVHCI=m
CONFIG_NETPOLL=y
CONFIG_NETPOLL_RX=y
CONFIG_NETPOLL_TRAP=y
CONFIG_NET_POLL_CONTROLLER=y
CONFIG_ISDN=m
CONFIG_ISDN_I4L=m
CONFIG_ISDN_PPP=y
CONFIG_ISDN_PPP_VJ=y
CONFIG_ISDN_MPP=y
CONFIG_IPPP_FILTER=y
CONFIG_ISDN_PPP_BSDCOMP=m
CONFIG_ISDN_AUDIO=y
CONFIG_ISDN_TTY_FAX=y
CONFIG_ISDN_X25=y
CONFIG_ISDN_DRV_HISAX=m
CONFIG_HISAX_EURO=y
CONFIG_DE_AOC=y
CONFIG_HISAX_1TR6=y
CONFIG_HISAX_NI1=y
CONFIG_HISAX_16_0=y
CONFIG_HISAX_16_3=y
CONFIG_HISAX_TELESPCI=y
CONFIG_HISAX_S0BOX=y
CONFIG_HISAX_AVM_A1=y
CONFIG_HISAX_FRITZPCI=y
CONFIG_HISAX_AVM_A1_PCMCIA=y
CONFIG_HISAX_ELSA=y
CONFIG_HISAX_IX1MICROR2=y
CONFIG_HISAX_DIEHLDIVA=y
CONFIG_HISAX_ASUSCOM=y
CONFIG_HISAX_TELEINT=y
CONFIG_HISAX_HFCS=y
CONFIG_HISAX_SEDLBAUER=y
CONFIG_HISAX_SPORTSTER=y
CONFIG_HISAX_MIC=y
CONFIG_HISAX_NETJET=y
CONFIG_HISAX_NETJET_U=y
CONFIG_HISAX_NICCY=y
CONFIG_HISAX_ISURF=y
CONFIG_HISAX_HSTSAPHIR=y
CONFIG_HISAX_BKM_A4T=y
CONFIG_HISAX_SCT_QUADRO=y
CONFIG_HISAX_GAZEL=y
CONFIG_HISAX_HFC_PCI=y
CONFIG_HISAX_W6692=y
CONFIG_HISAX_HFC_SX=y
CONFIG_HISAX_ENTERNOW_PCI=y
CONFIG_HISAX_DEBUG=y
CONFIG_HISAX_ELSA_CS=m
CONFIG_HISAX_AVM_A1_CS=m
CONFIG_HISAX_TELES_CS=m
CONFIG_HISAX_ST5481=m
CONFIG_HISAX_HFCUSB=m
CONFIG_HISAX_FRITZ_PCIPNP=m
CONFIG_HISAX_HDLC=y
CONFIG_ISDN_DRV_ICN=m
CONFIG_ISDN_DRV_PCBIT=m
CONFIG_ISDN_DRV_SC=m
CONFIG_ISDN_DRV_ACT2000=m
CONFIG_ISDN_DRV_TPAM=m
CONFIG_ISDN_CAPI=m
CONFIG_ISDN_DRV_AVMB1_VERBOSE_REASON=y
CONFIG_ISDN_CAPI_MIDDLEWARE=y
CONFIG_ISDN_CAPI_CAPI20=m
CONFIG_ISDN_CAPI_CAPIFS_BOOL=y
CONFIG_ISDN_CAPI_CAPIFS=m
CONFIG_ISDN_CAPI_CAPIDRV=m
CONFIG_CAPI_AVM=y
CONFIG_ISDN_DRV_AVMB1_B1ISA=m
CONFIG_ISDN_DRV_AVMB1_B1PCI=m
CONFIG_ISDN_DRV_AVMB1_B1PCIV4=y
CONFIG_ISDN_DRV_AVMB1_T1ISA=m
CONFIG_ISDN_DRV_AVMB1_B1PCMCIA=m
CONFIG_ISDN_DRV_AVMB1_AVM_CS=m
CONFIG_ISDN_DRV_AVMB1_T1PCI=m
CONFIG_ISDN_DRV_AVMB1_C4=m
CONFIG_CAPI_EICON=y
CONFIG_ISDN_DIVAS=m
CONFIG_ISDN_DIVAS_BRIPCI=y
CONFIG_ISDN_DIVAS_PRIPCI=y
CONFIG_ISDN_DIVAS_DIVACAPI=m
CONFIG_ISDN_DIVAS_USERIDI=m
CONFIG_ISDN_DIVAS_MAINT=m
CONFIG_PHONE=m
CONFIG_PHONE_IXJ=m
CONFIG_PHONE_IXJ_PCMCIA=m
CONFIG_INPUT=y
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_JOYDEV=m
CONFIG_INPUT_TSDEV=m
CONFIG_INPUT_EVDEV=m
CONFIG_GAMEPORT=m
CONFIG_SOUND_GAMEPORT=m
CONFIG_GAMEPORT_NS558=m
CONFIG_GAMEPORT_L4=m
CONFIG_GAMEPORT_EMU10K1=m
CONFIG_GAMEPORT_VORTEX=m
CONFIG_GAMEPORT_FM801=m
CONFIG_GAMEPORT_CS461x=m
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=m
CONFIG_SERIO_CT82C710=m
CONFIG_SERIO_PARKBD=m
CONFIG_SERIO_PCIPS2=m
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
CONFIG_KEYBOARD_SUNKBD=m
CONFIG_KEYBOARD_XTKBD=m
CONFIG_KEYBOARD_NEWTON=m
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_MOUSE_SERIAL=m
CONFIG_MOUSE_INPORT=m
CONFIG_MOUSE_ATIXL=y
CONFIG_MOUSE_LOGIBM=m
CONFIG_MOUSE_PC110PAD=m
CONFIG_INPUT_JOYSTICK=y
CONFIG_JOYSTICK_ANALOG=m
CONFIG_JOYSTICK_A3D=m
CONFIG_JOYSTICK_ADI=m
CONFIG_JOYSTICK_COBRA=m
CONFIG_JOYSTICK_GF2K=m
CONFIG_JOYSTICK_GRIP=m
CONFIG_JOYSTICK_GRIP_MP=m
CONFIG_JOYSTICK_GUILLEMOT=m
CONFIG_JOYSTICK_INTERACT=m
CONFIG_JOYSTICK_SIDEWINDER=m
CONFIG_JOYSTICK_TMDC=m
CONFIG_JOYSTICK_IFORCE=m
CONFIG_JOYSTICK_IFORCE_USB=y
CONFIG_JOYSTICK_IFORCE_232=y
CONFIG_JOYSTICK_WARRIOR=m
CONFIG_JOYSTICK_MAGELLAN=m
CONFIG_JOYSTICK_SPACEORB=m
CONFIG_JOYSTICK_SPACEBALL=m
CONFIG_JOYSTICK_STINGER=m
CONFIG_JOYSTICK_TWIDDLER=m
CONFIG_JOYSTICK_DB9=m
CONFIG_JOYSTICK_GAMECON=m
CONFIG_JOYSTICK_TURBOGRAFX=m
CONFIG_INPUT_TOUCHSCREEN=y
CONFIG_TOUCHSCREEN_GUNZE=m
CONFIG_INPUT_MISC=y
CONFIG_INPUT_PCSPKR=y
CONFIG_INPUT_UINPUT=m
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_SERIAL_NONSTANDARD=y
CONFIG_ROCKETPORT=m
CONFIG_SYNCLINK=m
CONFIG_SYNCLINKMP=m
CONFIG_N_HDLC=m
CONFIG_STALDRV=y
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_SERIAL_8250_CS=m
CONFIG_SERIAL_8250_EXTENDED=y
CONFIG_SERIAL_8250_MANY_PORTS=y
CONFIG_SERIAL_8250_SHARE_IRQ=y
CONFIG_SERIAL_8250_MULTIPORT=y
CONFIG_SERIAL_8250_RSA=y
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_UNIX98_PTYS=y
CONFIG_LEGACY_PTYS=y
CONFIG_PRINTER=m
CONFIG_PPDEV=m
CONFIG_TIPAR=m
CONFIG_QIC02_TAPE=m
CONFIG_QIC02_DYNCONF=y
CONFIG_IPMI_HANDLER=m
CONFIG_IPMI_PANIC_EVENT=y
CONFIG_IPMI_PANIC_STRING=y
CONFIG_IPMI_DEVICE_INTERFACE=m
CONFIG_IPMI_KCS=m
CONFIG_IPMI_WATCHDOG=m
CONFIG_WATCHDOG=y
CONFIG_SOFT_WATCHDOG=m
CONFIG_ACQUIRE_WDT=m
CONFIG_ADVANTECH_WDT=m
CONFIG_ALIM1535_WDT=m
CONFIG_ALIM7101_WDT=m
CONFIG_AMD7XX_TCO=m
CONFIG_SC520_WDT=m
CONFIG_EUROTECH_WDT=m
CONFIG_IB700_WDT=m
CONFIG_WAFER_WDT=m
CONFIG_I8XX_TCO=m
CONFIG_SC1200_WDT=m
CONFIG_SCx200_WDT=m
CONFIG_60XX_WDT=m
CONFIG_CPU5_WDT=m
CONFIG_W83627HF_WDT=m
CONFIG_W83877F_WDT=m
CONFIG_MACHZ_WDT=m
CONFIG_PCWATCHDOG=m
CONFIG_MIXCOMWD=m
CONFIG_WDT=m
CONFIG_WDT_501=y
CONFIG_PCIPCWATCHDOG=m
CONFIG_WDTPCI=m
CONFIG_WDT_501_PCI=y
CONFIG_USBPCWATCHDOG=m
CONFIG_HW_RANDOM=m
CONFIG_NVRAM=m
CONFIG_RTC=y
CONFIG_DTLK=m
CONFIG_R3964=m
CONFIG_APPLICOM=m
CONFIG_SONYPI=m
CONFIG_AGP=m
CONFIG_AGP_ALI=m
CONFIG_AGP_ATI=m
CONFIG_AGP_AMD=m
CONFIG_AGP_AMD64=m
CONFIG_AGP_INTEL=m
CONFIG_AGP_NVIDIA=m
CONFIG_AGP_SIS=m
CONFIG_AGP_SWORKS=m
CONFIG_AGP_VIA=m
CONFIG_AGP_EFFICEON=m
CONFIG_SYNCLINK_CS=m
CONFIG_SCx200_GPIO=m
CONFIG_RAW_DRIVER=m
CONFIG_HANGCHECK_TIMER=m
CONFIG_LIRC_SUPPORT=m
CONFIG_LIRC_BT829=m
CONFIG_LIRC_IT87=m
CONFIG_LIRC_ATIUSB=m
CONFIG_LIRC_SERIAL=m
CONFIG_LIRC_SIR=m
CONFIG_I2C=m
CONFIG_I2C_CHARDEV=m
CONFIG_I2C_ALGOBIT=m
CONFIG_I2C_ALGOPCF=m
CONFIG_I2C_ALI1535=m
CONFIG_I2C_ALI15X3=m
CONFIG_I2C_AMD756=m
CONFIG_I2C_AMD8111=m
CONFIG_I2C_I801=m
CONFIG_I2C_I810=m
CONFIG_I2C_ISA=m
CONFIG_I2C_NFORCE2=m
CONFIG_I2C_PARPORT=m
CONFIG_I2C_PARPORT_LIGHT=m
CONFIG_I2C_PIIX4=m
CONFIG_I2C_PROSAVAGE=m
CONFIG_I2C_SAVAGE4=m
CONFIG_SCx200_I2C=m
CONFIG_SCx200_ACB=m
CONFIG_I2C_SIS5595=m
CONFIG_I2C_SIS630=m
CONFIG_I2C_SIS96X=m
CONFIG_I2C_VIA=m
CONFIG_I2C_VIAPRO=m
CONFIG_I2C_VOODOO3=m
CONFIG_I2C_SENSOR=m
CONFIG_SENSORS_ADM1021=m
CONFIG_SENSORS_ASB100=m
CONFIG_SENSORS_DS1621=m
CONFIG_SENSORS_FSCHER=m
CONFIG_SENSORS_GL518SM=m
CONFIG_SENSORS_IT87=m
CONFIG_SENSORS_LM75=m
CONFIG_SENSORS_LM78=m
CONFIG_SENSORS_LM80=m
CONFIG_SENSORS_LM83=m
CONFIG_SENSORS_LM85=m
CONFIG_SENSORS_LM90=m
CONFIG_SENSORS_VIA686A=m
CONFIG_SENSORS_W83781D=m
CONFIG_SENSORS_W83L785TS=m
CONFIG_SENSORS_W83627HF=m
CONFIG_SENSORS_EEPROM=m
CONFIG_IBM_ASM=m
CONFIG_VIDEO_DEV=m
CONFIG_VIDEO_BT848=m
CONFIG_VIDEO_PMS=m
CONFIG_VIDEO_BWQCAM=m
CONFIG_VIDEO_CQCAM=m
CONFIG_VIDEO_W9966=m
CONFIG_VIDEO_CPIA=m
CONFIG_VIDEO_CPIA_PP=m
CONFIG_VIDEO_CPIA_USB=m
CONFIG_VIDEO_SAA5246A=m
CONFIG_VIDEO_SAA5249=m
CONFIG_TUNER_3036=m
CONFIG_VIDEO_STRADIS=m
CONFIG_VIDEO_MEYE=m
CONFIG_VIDEO_SAA7134=m
CONFIG_VIDEO_MXB=m
CONFIG_VIDEO_DPC=m
CONFIG_VIDEO_HEXIUM_ORION=m
CONFIG_VIDEO_HEXIUM_GEMINI=m
CONFIG_VIDEO_CX88=m
CONFIG_RADIO_CADET=m
CONFIG_RADIO_RTRACK=m
CONFIG_RADIO_RTRACK2=m
CONFIG_RADIO_AZTECH=m
CONFIG_RADIO_GEMTEK=m
CONFIG_RADIO_GEMTEK_PCI=m
CONFIG_RADIO_MAXIRADIO=m
CONFIG_RADIO_MAESTRO=m
CONFIG_RADIO_MIROPCM20=m
CONFIG_RADIO_SF16FMI=m
CONFIG_RADIO_SF16FMR2=m
CONFIG_RADIO_TERRATEC=m
CONFIG_RADIO_TRUST=m
CONFIG_RADIO_TYPHOON=m
CONFIG_RADIO_TYPHOON_PROC_FS=y
CONFIG_RADIO_ZOLTRIX=m
CONFIG_DVB=y
CONFIG_DVB_CORE=m
CONFIG_DVB_TWINHAN_DST=m
CONFIG_DVB_STV0299=m
CONFIG_DVB_SP887X=m
CONFIG_DVB_ALPS_TDLB7=m
CONFIG_DVB_ALPS_TDMB7=m
CONFIG_DVB_ATMEL_AT76C651=m
CONFIG_DVB_CX24110=m
CONFIG_DVB_GRUNDIG_29504_491=m
CONFIG_DVB_GRUNDIG_29504_401=m
CONFIG_DVB_MT312=m
CONFIG_DVB_VES1820=m
CONFIG_DVB_VES1X93=m
CONFIG_DVB_TDA1004X=m
CONFIG_DVB_NXT6000=m
CONFIG_DVB_AV7110=m
CONFIG_DVB_AV7110_OSD=y
CONFIG_DVB_BUDGET=m
CONFIG_DVB_BUDGET_CI=m
CONFIG_DVB_BUDGET_AV=m
CONFIG_DVB_BUDGET_PATCH=m
CONFIG_DVB_TTUSB_BUDGET=m
CONFIG_DVB_TTUSB_DEC=m
CONFIG_DVB_B2C2_SKYSTAR=m
CONFIG_DVB_BT8XX=m
CONFIG_VIDEO_SAA7146=m
CONFIG_VIDEO_SAA7146_VV=m
CONFIG_VIDEO_VIDEOBUF=m
CONFIG_VIDEO_TUNER=m
CONFIG_VIDEO_BUF=m
CONFIG_VIDEO_BTCX=m
CONFIG_VIDEO_IR=m
CONFIG_FB=y
CONFIG_FB_PM2=m
CONFIG_FB_PM2_FIFO_DISCONNECT=y
CONFIG_FB_CYBER2000=m
CONFIG_FB_IMSTT=y
CONFIG_FB_VGA16=m
CONFIG_FB_VESA=y
CONFIG_VIDEO_SELECT=y
CONFIG_FB_HGA=m
CONFIG_FB_RIVA=m
CONFIG_FB_I810=m
CONFIG_FB_I810_GTF=y
CONFIG_FB_RADEON=m
CONFIG_FB_RADEON_I2C=y
CONFIG_FB_ATY=m
CONFIG_FB_ATY_CT=y
CONFIG_FB_ATY_GX=y
CONFIG_FB_ATY_XL_INIT=y
CONFIG_FB_SIS=m
CONFIG_FB_SIS_300=y
CONFIG_FB_SIS_315=y
CONFIG_FB_NEOMAGIC=m
CONFIG_FB_KYRO=m
CONFIG_FB_3DFX=m
CONFIG_FB_VOODOO1=m
CONFIG_FB_TRIDENT=m
CONFIG_VGA_CONSOLE=y
CONFIG_MDA_CONSOLE=m
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
CONFIG_PCI_CONSOLE=y
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
CONFIG_BOOTSPLASH=y
CONFIG_SOUND=m
CONFIG_SND=m
CONFIG_SND_TIMER=m
CONFIG_SND_PCM=m
CONFIG_SND_HWDEP=m
CONFIG_SND_RAWMIDI=m
CONFIG_SND_SEQUENCER=m
CONFIG_SND_SEQ_DUMMY=m
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=m
CONFIG_SND_PCM_OSS=m
CONFIG_SND_SEQUENCER_OSS=y
CONFIG_SND_RTCTIMER=m
CONFIG_SND_VERBOSE_PRINTK=y
CONFIG_SND_DEBUG=y
CONFIG_SND_DEBUG_MEMORY=y
CONFIG_SND_MPU401_UART=m
CONFIG_SND_OPL3_LIB=m
CONFIG_SND_OPL4_LIB=m
CONFIG_SND_VX_LIB=m
CONFIG_SND_DUMMY=m
CONFIG_SND_VIRMIDI=m
CONFIG_SND_MTPAV=m
CONFIG_SND_SERIAL_U16550=m
CONFIG_SND_MPU401=m
CONFIG_SND_AD1816A=m
CONFIG_SND_AD1848=m
CONFIG_SND_CS4231=m
CONFIG_SND_CS4232=m
CONFIG_SND_CS4236=m
CONFIG_SND_ES968=m
CONFIG_SND_ES1688=m
CONFIG_SND_ES18XX=m
CONFIG_SND_GUSCLASSIC=m
CONFIG_SND_GUSEXTREME=m
CONFIG_SND_GUSMAX=m
CONFIG_SND_INTERWAVE=m
CONFIG_SND_INTERWAVE_STB=m
CONFIG_SND_OPTI92X_AD1848=m
CONFIG_SND_OPTI92X_CS4231=m
CONFIG_SND_OPTI93X=m
CONFIG_SND_SB8=m
CONFIG_SND_SB16=m
CONFIG_SND_SBAWE=m
CONFIG_SND_SB16_CSP=y
CONFIG_SND_WAVEFRONT=m
CONFIG_SND_ALS100=m
CONFIG_SND_AZT2320=m
CONFIG_SND_CMI8330=m
CONFIG_SND_DT019X=m
CONFIG_SND_OPL3SA2=m
CONFIG_SND_SGALAXY=m
CONFIG_SND_SSCAPE=m
CONFIG_SND_AC97_CODEC=m
CONFIG_SND_ALI5451=m
CONFIG_SND_ATIIXP=m
CONFIG_SND_AU8810=m
CONFIG_SND_AU8820=m
CONFIG_SND_AU8830=m
CONFIG_SND_AZT3328=m
CONFIG_SND_BT87X=m
CONFIG_SND_CS46XX=m
CONFIG_SND_CS46XX_NEW_DSP=y
CONFIG_SND_CS4281=m
CONFIG_SND_EMU10K1=m
CONFIG_SND_KORG1212=m
CONFIG_SND_MIXART=m
CONFIG_SND_NM256=m
CONFIG_SND_RME32=m
CONFIG_SND_RME96=m
CONFIG_SND_RME9652=m
CONFIG_SND_HDSP=m
CONFIG_SND_TRIDENT=m
CONFIG_SND_YMFPCI=m
CONFIG_SND_ALS4000=m
CONFIG_SND_CMIPCI=m
CONFIG_SND_ENS1370=m
CONFIG_SND_ENS1371=m
CONFIG_SND_ES1938=m
CONFIG_SND_ES1968=m
CONFIG_SND_MAESTRO3=m
CONFIG_SND_FM801=m
CONFIG_SND_FM801_TEA575X=m
CONFIG_SND_ICE1712=m
CONFIG_SND_ICE1724=m
CONFIG_SND_INTEL8X0=m
CONFIG_SND_INTEL8X0M=m
CONFIG_SND_SONICVIBES=m
CONFIG_SND_VIA82XX=m
CONFIG_SND_VX222=m
CONFIG_SND_USB_AUDIO=m
CONFIG_SND_VXPOCKET=m
CONFIG_SND_VXP440=m
CONFIG_SND_PDAUDIOCF=m
CONFIG_SOUND_PRIME=m
CONFIG_SOUND_BT878=m
CONFIG_SOUND_CMPCI=m
CONFIG_SOUND_CMPCI_FM=y
CONFIG_SOUND_CMPCI_MIDI=y
CONFIG_SOUND_CMPCI_JOYSTICK=y
CONFIG_SOUND_CMPCI_CM8738=y
CONFIG_SOUND_CMPCI_SPDIFLOOP=y
CONFIG_SOUND_EMU10K1=m
CONFIG_MIDI_EMU10K1=y
CONFIG_SOUND_CS4281=m
CONFIG_SOUND_ES1370=m
CONFIG_SOUND_ES1371=m
CONFIG_SOUND_ESSSOLO1=m
CONFIG_SOUND_MAESTRO=m
CONFIG_SOUND_MAESTRO3=m
CONFIG_SOUND_ICH=m
CONFIG_SOUND_SONICVIBES=m
CONFIG_SOUND_TRIDENT=m
CONFIG_SOUND_VIA82CXXX=m
CONFIG_MIDI_VIA82CXXX=y
CONFIG_SOUND_OSS=m
CONFIG_SOUND_TRACEINIT=y
CONFIG_SOUND_DMAP=y
CONFIG_SOUND_AD1889=m
CONFIG_SOUND_SGALAXY=m
CONFIG_SOUND_ADLIB=m
CONFIG_SOUND_ACI_MIXER=m
CONFIG_SOUND_CS4232=m
CONFIG_SOUND_SSCAPE=m
CONFIG_SOUND_GUS=m
CONFIG_SOUND_GUSMAX=y
CONFIG_SOUND_VMIDI=m
CONFIG_SOUND_TRIX=m
CONFIG_SOUND_MSS=m
CONFIG_SOUND_MPU401=m
CONFIG_SOUND_NM256=m
CONFIG_SOUND_MAD16=m
CONFIG_MAD16_OLDCARD=y
CONFIG_SOUND_PAS=m
CONFIG_SOUND_PSS=m
CONFIG_PSS_MIXER=y
CONFIG_SOUND_SB=m
CONFIG_SOUND_WAVEFRONT=m
CONFIG_SOUND_MAUI=m
CONFIG_SOUND_YM3812=m
CONFIG_SOUND_OPL3SA1=m
CONFIG_SOUND_OPL3SA2=m
CONFIG_SOUND_YMFPCI=m
CONFIG_SOUND_YMFPCI_LEGACY=y
CONFIG_SOUND_UART6850=m
CONFIG_SOUND_AEDSP16=m
CONFIG_SC6600=y
CONFIG_SC6600_JOY=y
CONFIG_AEDSP16_MPU401=y
CONFIG_SOUND_TVMIXER=m
CONFIG_SOUND_KAHLUA=m
CONFIG_SOUND_ALI5455=m
CONFIG_SOUND_FORTE=m
CONFIG_SOUND_RME96XX=m
CONFIG_SOUND_AD1980=m
CONFIG_USB=m
CONFIG_USB_DEVICEFS=y
CONFIG_USB_EHCI_HCD=m
CONFIG_USB_EHCI_SPLIT_ISO=y
CONFIG_USB_OHCI_HCD=m
CONFIG_USB_UHCI_HCD=m
CONFIG_USB_AUDIO=m
CONFIG_USB_MIDI=m
CONFIG_USB_ACM=m
CONFIG_USB_PRINTER=m
CONFIG_USB_STORAGE=m
CONFIG_USB_STORAGE_DATAFAB=y
CONFIG_USB_STORAGE_FREECOM=y
CONFIG_USB_STORAGE_ISD200=y
CONFIG_USB_STORAGE_DPCM=y
CONFIG_USB_STORAGE_HP8200e=y
CONFIG_USB_STORAGE_SDDR09=y
CONFIG_USB_STORAGE_SDDR55=y
CONFIG_USB_STORAGE_JUMPSHOT=y
CONFIG_USB_HID=m
CONFIG_USB_HIDINPUT=y
CONFIG_HID_FF=y
CONFIG_HID_PID=y
CONFIG_LOGITECH_FF=y
CONFIG_THRUSTMASTER_FF=y
CONFIG_USB_HIDDEV=y
CONFIG_USB_AIPTEK=m
CONFIG_USB_WACOM=m
CONFIG_USB_KBTAB=m
CONFIG_USB_POWERMATE=m
CONFIG_USB_MTOUCH=m
CONFIG_USB_XPAD=m
CONFIG_USB_ATI_REMOTE=m
CONFIG_USB_MDC800=m
CONFIG_USB_MICROTEK=m
CONFIG_USB_HPUSBSCSI=m
CONFIG_USB_DABUSB=m
CONFIG_USB_VICAM=m
CONFIG_USB_DSBR=m
CONFIG_USB_IBMCAM=m
CONFIG_USB_KONICAWC=m
CONFIG_USB_OV511=m
CONFIG_USB_SE401=m
CONFIG_USB_STV680=m
CONFIG_USB_W9968CF=m
CONFIG_USB_CATC=m
CONFIG_USB_KAWETH=m
CONFIG_USB_PEGASUS=m
CONFIG_USB_RTL8150=m
CONFIG_USB_USBNET=m
CONFIG_USB_ALI_M5632=y
CONFIG_USB_AN2720=y
CONFIG_USB_BELKIN=y
CONFIG_USB_GENESYS=y
CONFIG_USB_NET1080=y
CONFIG_USB_PL2301=y
CONFIG_USB_ARMLINUX=y
CONFIG_USB_EPSON2888=y
CONFIG_USB_ZAURUS=y
CONFIG_USB_CDCETHER=y
CONFIG_USB_AX8817X=y
CONFIG_USB_USS720=m
CONFIG_USB_SERIAL=m
CONFIG_USB_SERIAL_GENERIC=y
CONFIG_USB_SERIAL_BELKIN=m
CONFIG_USB_SERIAL_DIGI_ACCELEPORT=m
CONFIG_USB_SERIAL_EMPEG=m
CONFIG_USB_SERIAL_FTDI_SIO=m
CONFIG_USB_SERIAL_VISOR=m
CONFIG_USB_SERIAL_IPAQ=m
CONFIG_USB_SERIAL_IR=m
CONFIG_USB_SERIAL_EDGEPORT=m
CONFIG_USB_SERIAL_EDGEPORT_TI=m
CONFIG_USB_SERIAL_KEYSPAN_PDA=m
CONFIG_USB_SERIAL_KEYSPAN=m
CONFIG_USB_SERIAL_KEYSPAN_MPR=y
CONFIG_USB_SERIAL_KEYSPAN_USA28=y
CONFIG_USB_SERIAL_KEYSPAN_USA28X=y
CONFIG_USB_SERIAL_KEYSPAN_USA28XA=y
CONFIG_USB_SERIAL_KEYSPAN_USA28XB=y
CONFIG_USB_SERIAL_KEYSPAN_USA19=y
CONFIG_USB_SERIAL_KEYSPAN_USA18X=y
CONFIG_USB_SERIAL_KEYSPAN_USA19W=y
CONFIG_USB_SERIAL_KEYSPAN_USA19QW=y
CONFIG_USB_SERIAL_KEYSPAN_USA19QI=y
CONFIG_USB_SERIAL_KEYSPAN_USA49W=y
CONFIG_USB_SERIAL_KEYSPAN_USA49WLC=y
CONFIG_USB_SERIAL_KLSI=m
CONFIG_USB_SERIAL_KOBIL_SCT=m
CONFIG_USB_SERIAL_MCT_U232=m
CONFIG_USB_SERIAL_PL2303=m
CONFIG_USB_SERIAL_SAFE=m
CONFIG_USB_SERIAL_SAFE_PADDED=y
CONFIG_USB_SERIAL_CYBERJACK=m
CONFIG_USB_SERIAL_XIRCOM=m
CONFIG_USB_SERIAL_OMNINET=m
CONFIG_USB_EZUSB=y
CONFIG_USB_EMI62=m
CONFIG_USB_EMI26=m
CONFIG_USB_TIGL=m
CONFIG_USB_AUERSWALD=m
CONFIG_USB_RIO500=m
CONFIG_USB_LEGOTOWER=m
CONFIG_USB_LCD=m
CONFIG_USB_LED=m
CONFIG_USB_SPEEDTOUCH=m
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
CONFIG_EXT3_FS=m
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
CONFIG_JBD=m
CONFIG_JBD_DEBUG=y
CONFIG_FS_MBCACHE=y
CONFIG_REISERFS_FS=m
CONFIG_REISERFS_FS_XATTR=y
CONFIG_REISERFS_FS_POSIX_ACL=y
CONFIG_REISERFS_FS_SECURITY=y
CONFIG_JFS_FS=m
CONFIG_JFS_POSIX_ACL=y
CONFIG_JFS_STATISTICS=y
CONFIG_FS_POSIX_ACL=y
CONFIG_XFS_FS=m
CONFIG_XFS_RT=y
CONFIG_XFS_QUOTA=m
CONFIG_XFS_SECURITY=y
CONFIG_XFS_POSIX_ACL=y
CONFIG_XFS_DMAPI=m
CONFIG_MINIX_FS=y
CONFIG_ROMFS_FS=m
CONFIG_QUOTA=y
CONFIG_QFMT_V1=m
CONFIG_QFMT_V2=m
CONFIG_QUOTACTL=y
CONFIG_AUTOFS_FS=m
CONFIG_AUTOFS4_FS=m
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_ZISOFS_FS=y
CONFIG_UDF_FS=m
CONFIG_FAT_FS=m
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=m
CONFIG_NTFS_FS=m
CONFIG_RCFS_FS=m
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_DEVPTS_FS_XATTR=y
CONFIG_DEVPTS_FS_SECURITY=y
CONFIG_TMPFS=y
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_RAMFS=y
CONFIG_RELAYFS_FS=m
CONFIG_KLOG_CHANNEL=y
CONFIG_ADFS_FS=m
CONFIG_AFFS_FS=m
CONFIG_HFS_FS=m
CONFIG_HFSPLUS_FS=m
CONFIG_BEFS_FS=m
CONFIG_BFS_FS=m
CONFIG_EFS_FS=m
CONFIG_JFFS_FS=m
CONFIG_JFFS2_FS=m
CONFIG_CRAMFS=m
CONFIG_VXFS_FS=m
CONFIG_HPFS_FS=m
CONFIG_QNX4FS_FS=m
CONFIG_SYSV_FS=m
CONFIG_UFS_FS=m
CONFIG_NFS_FS=y
CONFIG_NFS_V3=y
CONFIG_NFS_ACL=y
CONFIG_NFS_V4=y
CONFIG_NFS_DIRECTIO=y
CONFIG_NFSD=m
CONFIG_NFSD_V3=y
CONFIG_NFSD_ACL=y
CONFIG_NFS_ACL_SUPPORT=y
CONFIG_NFSD_V4=y
CONFIG_NFSD_TCP=y
CONFIG_LOCKD=y
CONFIG_STATD=y
CONFIG_LOCKD_V4=y
CONFIG_EXPORTFS=m
CONFIG_SUNRPC=y
CONFIG_SUNRPC_GSS=y
CONFIG_RPCSEC_GSS_KRB5=y
CONFIG_SMB_FS=m
CONFIG_SMB_NLS_DEFAULT=y
CONFIG_CIFS=m
CONFIG_NCP_FS=m
CONFIG_NCPFS_PACKET_SIGNING=y
CONFIG_NCPFS_IOCTL_LOCKING=y
CONFIG_NCPFS_STRONG=y
CONFIG_NCPFS_NFS_NS=y
CONFIG_NCPFS_OS2_NS=y
CONFIG_NCPFS_SMALLDOS=y
CONFIG_NCPFS_NLS=y
CONFIG_NCPFS_EXTRAS=y
CONFIG_CODA_FS=m
CONFIG_INTERMEZZO_FS=m
CONFIG_AFS_FS=m
CONFIG_RXRPC=m
CONFIG_PARTITION_ADVANCED=y
CONFIG_OSF_PARTITION=y
CONFIG_ATARI_PARTITION=y
CONFIG_MAC_PARTITION=y
CONFIG_MSDOS_PARTITION=y
CONFIG_BSD_DISKLABEL=y
CONFIG_SOLARIS_X86_PARTITION=y
CONFIG_UNIXWARE_DISKLABEL=y
CONFIG_LDM_PARTITION=y
CONFIG_NEC98_PARTITION=y
CONFIG_ULTRIX_PARTITION=y
CONFIG_SUN_PARTITION=y
CONFIG_EFI_PARTITION=y
CONFIG_NLS=y
CONFIG_NLS_CODEPAGE_437=m
CONFIG_NLS_CODEPAGE_737=m
CONFIG_NLS_CODEPAGE_775=m
CONFIG_NLS_CODEPAGE_850=m
CONFIG_NLS_CODEPAGE_852=m
CONFIG_NLS_CODEPAGE_855=m
CONFIG_NLS_CODEPAGE_857=m
CONFIG_NLS_CODEPAGE_860=m
CONFIG_NLS_CODEPAGE_861=m
CONFIG_NLS_CODEPAGE_862=m
CONFIG_NLS_CODEPAGE_863=m
CONFIG_NLS_CODEPAGE_864=m
CONFIG_NLS_CODEPAGE_865=m
CONFIG_NLS_CODEPAGE_866=m
CONFIG_NLS_CODEPAGE_869=m
CONFIG_NLS_CODEPAGE_936=m
CONFIG_NLS_CODEPAGE_950=m
CONFIG_NLS_CODEPAGE_932=m
CONFIG_NLS_CODEPAGE_949=m
CONFIG_NLS_CODEPAGE_874=m
CONFIG_NLS_ISO8859_8=m
CONFIG_NLS_CODEPAGE_1250=m
CONFIG_NLS_CODEPAGE_1251=m
CONFIG_NLS_ISO8859_1=m
CONFIG_NLS_ISO8859_2=m
CONFIG_NLS_ISO8859_3=m
CONFIG_NLS_ISO8859_4=m
CONFIG_NLS_ISO8859_5=m
CONFIG_NLS_ISO8859_6=m
CONFIG_NLS_ISO8859_7=m
CONFIG_NLS_ISO8859_9=m
CONFIG_NLS_ISO8859_13=m
CONFIG_NLS_ISO8859_14=m
CONFIG_NLS_ISO8859_15=m
CONFIG_NLS_KOI8_R=m
CONFIG_NLS_KOI8_U=m
CONFIG_NLS_UTF8=m
CONFIG_PROFILING=y
CONFIG_OPROFILE=m
CONFIG_CRASH_DUMP=m
CONFIG_KERNTYPES=y
CONFIG_CRASH_DUMP_BLOCKDEV=m
CONFIG_CRASH_DUMP_NETDEV=m
CONFIG_CRASH_DUMP_COMPRESS_RLE=m
CONFIG_CRASH_DUMP_COMPRESS_GZIP=m
CONFIG_DEBUG_KERNEL=y
CONFIG_EARLY_PRINTK=y
CONFIG_DEBUG_STACK_USAGE=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y
CONFIG_SECURITY=y
CONFIG_SECURITY_NETWORK=y
CONFIG_SECURITY_CAPABILITIES=y
CONFIG_SECURITY_CAPABILITIES_BOOTPARAM=y
CONFIG_SECURITY_ROOTPLUG=m
CONFIG_SECURITY_SELINUX=y
CONFIG_SECURITY_SELINUX_BOOTPARAM=y
CONFIG_SECURITY_SELINUX_DEVELOP=y
CONFIG_CRYPTO=y
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_NULL=m
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_SHA1=m
CONFIG_CRYPTO_SHA256=m
CONFIG_CRYPTO_SHA512=m
CONFIG_CRYPTO_DES=y
CONFIG_CRYPTO_BLOWFISH=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_SERPENT=m
CONFIG_CRYPTO_AES=m
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST6=m
CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_DEFLATE=m
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_TEST=m
CONFIG_CRC32=y
CONFIG_QSORT=y
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=m
CONFIG_SUSE_KERNEL=y
CONFIG_X86_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_X86_TRAMPOLINE=y
CONFIG_PC=y
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Inconsistent timing results of multithreaded program on an SMP machine.
2005-11-20 9:27 Marcel Zalmanovici
@ 2005-11-20 9:54 ` Con Kolivas
2005-11-20 10:18 ` Marcel Zalmanovici
2005-11-20 14:02 ` Paul Jackson
1 sibling, 1 reply; 17+ messages in thread
From: Con Kolivas @ 2005-11-20 9:54 UTC (permalink / raw)
To: Marcel Zalmanovici; +Cc: linux-kernel
On Sun, 20 Nov 2005 20:27, Marcel Zalmanovici wrote:
> Hi,
>
> I am trying, as part of my thesis, to make some improvement to the linux
> scheduler.
> Therefore I've written a small multithreaded application and tested how
> long on average it takes for it to complete.
>
> The results were very surprising. I expected to see the completion time
> vary 1 to 2 seconds at most.
> Instead what I've got was an oscillation where the maximum time was twice
> and more than the minimum!! For a short test results ranged ~7sec to ~16
> sec, and the same happened to longer tests where the min was ~1min and the
> max ~2:30min.
>
> Does anyone have any clue as to what might happen ?
> Is there anything I can do to get stable results ?
>
> Here is a small test case program:
>
> (See attached file: sched_test.c)
>
> The test was always done on a pretty much empty machine. I've tried both
> kernel 2.6.4-52 and 2.6.13.4 but the results were the same.
>
> I'm working on a Xeon Intel machine, dual processor, hyperthreaded.
^^^^^^^^^^^^^^
I suspect what you're seeing is the random nature of threads to bind either to
a hyperthread sibling or a real core. If all your threads land on only one
physical cpu and both hyperthread siblings it will run much slower than if
half land on one physical cpu and half on the other physical cpu. To confirm
this, try setting cpu affinity just for one logical core of each phyiscal cpu
(like affinity for cpu 1 and 3 only for example) or disabling hyperthread in
the bios.
Cheers,
Con
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Inconsistent timing results of multithreaded program on an SMP machine.
2005-11-20 9:54 ` Con Kolivas
@ 2005-11-20 10:18 ` Marcel Zalmanovici
2005-11-20 10:28 ` Con Kolivas
0 siblings, 1 reply; 17+ messages in thread
From: Marcel Zalmanovici @ 2005-11-20 10:18 UTC (permalink / raw)
To: Con Kolivas; +Cc: linux-kernel
Hi Con,
Thanks for the quick reply.
I suspected hyperthreading might cause some these results, therefore, I
have also tried disabling hyperthreading altogether.
The results haven't changed.
I've also tried entering in single-user mode, no effect there either.
Marcel
Con Kolivas
<kernel@kolivas.o To: Marcel Zalmanovici/Haifa/IBM@IBMIL
rg> cc: linux-kernel@vger.kernel.org
Subject: Re: Inconsistent timing results of multithreaded program on an SMP
20/11/2005 11:54 machine.
On Sun, 20 Nov 2005 20:27, Marcel Zalmanovici wrote:
> Hi,
>
> I am trying, as part of my thesis, to make some improvement to the linux
> scheduler.
> Therefore I've written a small multithreaded application and tested how
> long on average it takes for it to complete.
>
> The results were very surprising. I expected to see the completion time
> vary 1 to 2 seconds at most.
> Instead what I've got was an oscillation where the maximum time was twice
> and more than the minimum!! For a short test results ranged ~7sec to ~16
> sec, and the same happened to longer tests where the min was ~1min and
the
> max ~2:30min.
>
> Does anyone have any clue as to what might happen ?
> Is there anything I can do to get stable results ?
>
> Here is a small test case program:
>
> (See attached file: sched_test.c)
>
> The test was always done on a pretty much empty machine. I've tried both
> kernel 2.6.4-52 and 2.6.13.4 but the results were the same.
>
> I'm working on a Xeon Intel machine, dual processor, hyperthreaded.
^^^^^^^^^^^^^^
I suspect what you're seeing is the random nature of threads to bind either
to
a hyperthread sibling or a real core. If all your threads land on only one
physical cpu and both hyperthread siblings it will run much slower than if
half land on one physical cpu and half on the other physical cpu. To
confirm
this, try setting cpu affinity just for one logical core of each phyiscal
cpu
(like affinity for cpu 1 and 3 only for example) or disabling hyperthread
in
the bios.
Cheers,
Con
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Inconsistent timing results of multithreaded program on an SMP machine.
2005-11-20 10:18 ` Marcel Zalmanovici
@ 2005-11-20 10:28 ` Con Kolivas
2005-11-20 10:35 ` Muli Ben-Yehuda
2005-11-20 10:43 ` Marcel Zalmanovici
0 siblings, 2 replies; 17+ messages in thread
From: Con Kolivas @ 2005-11-20 10:28 UTC (permalink / raw)
To: Marcel Zalmanovici; +Cc: linux-kernel
On Sun, 20 Nov 2005 21:18, Marcel Zalmanovici wrote:
> Hi Con,
>
> Thanks for the quick reply.
No problems.
Please reply below what you are replying to so we can keep track of email
threads.
> <kernel@kolivas.o To: Marcel
> Zalmanovici/Haifa/IBM@IBMIL rg> cc:
> linux-kernel@vger.kernel.org Subject: Re: Inconsistent timing results of
> multithreaded program on an SMP 20/11/2005 11:54 machine.
>
> On Sun, 20 Nov 2005 20:27, Marcel Zalmanovici wrote:
> > Hi,
> >
> > I am trying, as part of my thesis, to make some improvement to the linux
> > scheduler.
> > Therefore I've written a small multithreaded application and tested how
> > long on average it takes for it to complete.
> >
> > The results were very surprising. I expected to see the completion time
> > vary 1 to 2 seconds at most.
> > Instead what I've got was an oscillation where the maximum time was twice
> > and more than the minimum!! For a short test results ranged ~7sec to ~16
> > sec, and the same happened to longer tests where the min was ~1min and
>
> the
>
> > max ~2:30min.
> >
> > Does anyone have any clue as to what might happen ?
> > Is there anything I can do to get stable results ?
> >
> > Here is a small test case program:
> >
> > (See attached file: sched_test.c)
> >
> > The test was always done on a pretty much empty machine. I've tried both
> > kernel 2.6.4-52 and 2.6.13.4 but the results were the same.
> >
> > I'm working on a Xeon Intel machine, dual processor, hyperthreaded.
>
> ^^^^^^^^^^^^^^
>
> I suspect what you're seeing is the random nature of threads to bind either
> to
> a hyperthread sibling or a real core. If all your threads land on only one
> physical cpu and both hyperthread siblings it will run much slower than if
> half land on one physical cpu and half on the other physical cpu. To
> confirm
> this, try setting cpu affinity just for one logical core of each phyiscal
> cpu
> (like affinity for cpu 1 and 3 only for example) or disabling hyperthread
> in
> the bios.
Ok I've had a look at the actual program now ;) Are you timing the time it
takes to completion of everything?
This part of your program:
for (i= 0; i<8; i++)
pthread_join(tid[i], NULL);
Cares about the order the threads finish. Do you think this might be affecting
your results?
Cheers,
Con
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Inconsistent timing results of multithreaded program on an SMP machine.
2005-11-20 10:28 ` Con Kolivas
@ 2005-11-20 10:35 ` Muli Ben-Yehuda
2005-11-20 10:39 ` Con Kolivas
2005-11-20 10:43 ` Marcel Zalmanovici
1 sibling, 1 reply; 17+ messages in thread
From: Muli Ben-Yehuda @ 2005-11-20 10:35 UTC (permalink / raw)
To: Con Kolivas; +Cc: Marcel Zalmanovici, linux-kernel
On Sun, Nov 20, 2005 at 09:28:13PM +1100, Con Kolivas wrote:
> Ok I've had a look at the actual program now ;) Are you timing the time it
> takes to completion of everything?
>
> This part of your program:
> for (i= 0; i<8; i++)
> pthread_join(tid[i], NULL);
>
> Cares about the order the threads finish. Do you think this might be affecting
> your results?
I don't see why it should matter. Depending on the order the threads
finish, we will always wait in pthread_join until the last one
finishes, and then do between 0 and 7 more pthread_joins that should
return immediately (since the last one has already finished).
Cheers,
Muli
--
Muli Ben-Yehuda
http://www.mulix.org | http://mulix.livejournal.com/
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Inconsistent timing results of multithreaded program on an SMP machine.
2005-11-20 10:35 ` Muli Ben-Yehuda
@ 2005-11-20 10:39 ` Con Kolivas
2005-11-20 10:43 ` Muli Ben-Yehuda
2005-11-20 10:50 ` Marcel Zalmanovici
0 siblings, 2 replies; 17+ messages in thread
From: Con Kolivas @ 2005-11-20 10:39 UTC (permalink / raw)
To: Muli Ben-Yehuda; +Cc: Marcel Zalmanovici, linux-kernel
On Sun, 20 Nov 2005 21:35, Muli Ben-Yehuda wrote:
> On Sun, Nov 20, 2005 at 09:28:13PM +1100, Con Kolivas wrote:
> > Ok I've had a look at the actual program now ;) Are you timing the time
> > it takes to completion of everything?
> >
> > This part of your program:
> > for (i= 0; i<8; i++)
> > pthread_join(tid[i], NULL);
> >
> > Cares about the order the threads finish. Do you think this might be
> > affecting your results?
>
> I don't see why it should matter. Depending on the order the threads
> finish, we will always wait in pthread_join until the last one
> finishes, and then do between 0 and 7 more pthread_joins that should
> return immediately (since the last one has already finished).
If it was instant it shouldn't matter. I'm aware of that in theory, but there
have certainly been reports of pthread_join taking quite a while happening in
a sort of lazy/sloppy way. I don't know why this is the case but I wondered
if it was showing up here.
Cheers,
Con
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Inconsistent timing results of multithreaded program on an SMP machine.
2005-11-20 10:39 ` Con Kolivas
@ 2005-11-20 10:43 ` Muli Ben-Yehuda
2005-11-20 10:50 ` Marcel Zalmanovici
1 sibling, 0 replies; 17+ messages in thread
From: Muli Ben-Yehuda @ 2005-11-20 10:43 UTC (permalink / raw)
To: Con Kolivas; +Cc: Marcel Zalmanovici, linux-kernel
On Sun, Nov 20, 2005 at 09:39:00PM +1100, Con Kolivas wrote:
> On Sun, 20 Nov 2005 21:35, Muli Ben-Yehuda wrote:
> If it was instant it shouldn't matter. I'm aware of that in theory, but there
> have certainly been reports of pthread_join taking quite a while happening in
> a sort of lazy/sloppy way. I don't know why this is the case but I wondered
> if it was showing up here.
I see, I didn't realize pthread_join might not return immediately even
if the thread already finished (I would actually consider that a
bug...). Marcel, could you please check?
Cheers,
Muli
--
Muli Ben-Yehuda
http://www.mulix.org | http://mulix.livejournal.com/
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Inconsistent timing results of multithreaded program on an SMP machine.
2005-11-20 10:28 ` Con Kolivas
2005-11-20 10:35 ` Muli Ben-Yehuda
@ 2005-11-20 10:43 ` Marcel Zalmanovici
1 sibling, 0 replies; 17+ messages in thread
From: Marcel Zalmanovici @ 2005-11-20 10:43 UTC (permalink / raw)
To: Con Kolivas; +Cc: linux-kernel
Con Kolivas
<kernel@kolivas.o To: Marcel Zalmanovici/Haifa/IBM@IBMIL
rg> cc: linux-kernel@vger.kernel.org
Subject: Re: Inconsistent timing results of multithreaded program on an SMP
20/11/2005 12:28 machine.
On Sun, 20 Nov 2005 21:18, Marcel Zalmanovici wrote:
> Hi Con,
>
> Thanks for the quick reply.
No problems.
Please reply below what you are replying to so we can keep track of email
threads.
> <kernel@kolivas.o To: Marcel
> Zalmanovici/Haifa/IBM@IBMIL rg> cc:
> linux-kernel@vger.kernel.org Subject: Re: Inconsistent timing results of
> multithreaded program on an SMP 20/11/2005 11:54 machine.
>
> On Sun, 20 Nov 2005 20:27, Marcel Zalmanovici wrote:
> > Hi,
> >
> > I am trying, as part of my thesis, to make some improvement to the
linux
> > scheduler.
> > Therefore I've written a small multithreaded application and tested how
> > long on average it takes for it to complete.
> >
> > The results were very surprising. I expected to see the completion time
> > vary 1 to 2 seconds at most.
> > Instead what I've got was an oscillation where the maximum time was
twice
> > and more than the minimum!! For a short test results ranged ~7sec to
~16
> > sec, and the same happened to longer tests where the min was ~1min and
>
> the
>
> > max ~2:30min.
> >
> > Does anyone have any clue as to what might happen ?
> > Is there anything I can do to get stable results ?
> >
> > Here is a small test case program:
> >
> > (See attached file: sched_test.c)
> >
> > The test was always done on a pretty much empty machine. I've tried
both
> > kernel 2.6.4-52 and 2.6.13.4 but the results were the same.
> >
> > I'm working on a Xeon Intel machine, dual processor, hyperthreaded.
>
> ^^^^^^^^^^^^^^
>
> I suspect what you're seeing is the random nature of threads to bind
either
> to
> a hyperthread sibling or a real core. If all your threads land on only
one
> physical cpu and both hyperthread siblings it will run much slower than
if
> half land on one physical cpu and half on the other physical cpu. To
> confirm
> this, try setting cpu affinity just for one logical core of each phyiscal
> cpu
> (like affinity for cpu 1 and 3 only for example) or disabling hyperthread
> in
> the bios.
Ok I've had a look at the actual program now ;) Are you timing the time it
takes to completion of everything?
Yes. I'm timing the whole program.
This part of your program:
for (i= 0; i<8; i++)
pthread_join(tid[i], NULL);
Cares about the order the threads finish. Do you think this might be
affecting
your results?
I highly doubt that. The worst case scenario would have thread 0 finish
last right ? but then, all other queries would simply return without
waiting.
Cheers,
Con
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Inconsistent timing results of multithreaded program on an SMP machine.
2005-11-20 10:39 ` Con Kolivas
2005-11-20 10:43 ` Muli Ben-Yehuda
@ 2005-11-20 10:50 ` Marcel Zalmanovici
2005-11-20 10:50 ` Con Kolivas
1 sibling, 1 reply; 17+ messages in thread
From: Marcel Zalmanovici @ 2005-11-20 10:50 UTC (permalink / raw)
To: Con Kolivas; +Cc: linux-kernel, Muli Ben-Yehuda
Con Kolivas
<kernel@kolivas.o To: Muli Ben-Yehuda <mulix@mulix.org>
rg> cc: Marcel Zalmanovici/Haifa/IBM@IBMIL, linux-kernel@vger.kernel.org
Subject: Re: Inconsistent timing results of multithreaded program on an SMP
20/11/2005 12:39 machine.
On Sun, 20 Nov 2005 21:35, Muli Ben-Yehuda wrote:
> On Sun, Nov 20, 2005 at 09:28:13PM +1100, Con Kolivas wrote:
> > Ok I've had a look at the actual program now ;) Are you timing the time
> > it takes to completion of everything?
> >
> > This part of your program:
> > for (i= 0; i<8; i++)
> > pthread_join(tid[i], NULL);
> >
> > Cares about the order the threads finish. Do you think this might be
> > affecting your results?
>
> I don't see why it should matter. Depending on the order the threads
> finish, we will always wait in pthread_join until the last one
> finishes, and then do between 0 and 7 more pthread_joins that should
> return immediately (since the last one has already finished).
If it was instant it shouldn't matter. I'm aware of that in theory, but
there
have certainly been reports of pthread_join taking quite a while happening
in
a sort of lazy/sloppy way. I don't know why this is the case but I wondered
if it was showing up here.
I've looked through the detailed run results and found this:
Thread DS 0, TID = 7646
Thread DS 0, TID = 7645
Thread DS 1, TID = 7648
Thread DS 1, TID = 7650
Thread DS 2, TID = 7651
Thread DS 2, TID = 7652
Thread DS 3, TID = 7653
Thread DS 3, TID = 7654
Main exit ...
real 23.25
As you can see except the first 2 threads all finished in order they were
created. With the average being at about 16.5 this is a high result and
thread order was almost ideal.
Cheers,
Con
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Inconsistent timing results of multithreaded program on an SMP machine.
2005-11-20 10:50 ` Marcel Zalmanovici
@ 2005-11-20 10:50 ` Con Kolivas
0 siblings, 0 replies; 17+ messages in thread
From: Con Kolivas @ 2005-11-20 10:50 UTC (permalink / raw)
To: Marcel Zalmanovici; +Cc: linux-kernel, Muli Ben-Yehuda
On Sun, 20 Nov 2005 21:50, Marcel Zalmanovici wrote:
> On Sun, 20 Nov 2005 21:35, Muli Ben-Yehuda wrote:
> > On Sun, Nov 20, 2005 at 09:28:13PM +1100, Con Kolivas wrote:
> > > Ok I've had a look at the actual program now ;) Are you timing the time
> > > it takes to completion of everything?
> > >
> > > This part of your program:
> > > for (i= 0; i<8; i++)
> > > pthread_join(tid[i], NULL);
> > >
> > > Cares about the order the threads finish. Do you think this might be
> > > affecting your results?
> >
> > I don't see why it should matter. Depending on the order the threads
> > finish, we will always wait in pthread_join until the last one
> > finishes, and then do between 0 and 7 more pthread_joins that should
> > return immediately (since the last one has already finished).
>
> If it was instant it shouldn't matter. I'm aware of that in theory, but
> there
> have certainly been reports of pthread_join taking quite a while happening
> in
> a sort of lazy/sloppy way. I don't know why this is the case but I wondered
>
> if it was showing up here.
>
> I've looked through the detailed run results and found this:
>
> Thread DS 0, TID = 7646
> Thread DS 0, TID = 7645
> Thread DS 1, TID = 7648
> Thread DS 1, TID = 7650
> Thread DS 2, TID = 7651
> Thread DS 2, TID = 7652
> Thread DS 3, TID = 7653
> Thread DS 3, TID = 7654
> Main exit ...
> real 23.25
>
> As you can see except the first 2 threads all finished in order they were
> created. With the average being at about 16.5 this is a high result and
> thread order was almost ideal.
Ok, but how long does pthread_join actually take to complete?
Cheers,
Con
P.S. Lotus notes is terrible for a mailer isn't it?
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Inconsistent timing results of multithreaded program on an SMP machine.
2005-11-20 9:27 Marcel Zalmanovici
2005-11-20 9:54 ` Con Kolivas
@ 2005-11-20 14:02 ` Paul Jackson
2005-11-24 9:50 ` Marcel Zalmanovici
1 sibling, 1 reply; 17+ messages in thread
From: Paul Jackson @ 2005-11-20 14:02 UTC (permalink / raw)
To: Marcel Zalmanovici; +Cc: linux-kernel
Marcel wrote:
> Instead what I've got was an oscillation where the maximum time was twice
> and more than the minimum!! For a short test results ranged ~7sec to ~16 ...
Just for grins, try displaying which cpu each thread runs on. Display the
return from "latestcpu(getpid())" in the two per-thread printf's, to display
the thread's cpu at the beginning and end of each compute_thread(). Perhaps
you will notice that the per-thread cpu correlates with the test times.
/*
* int latestcpu(pid_t pid)
*
* Copyright (C) 2005 Silicon Graphics, Inc.
* This code is subject to the terms and conditions of the
* GNU General Public License.
*
* Return most recent CPU on which task pid executed.
*
* The last used CPU is visible for a given pid as field #39
* (starting with #1) in the file /proc/<pid>/stat. Currently
* this file has 41 fields, in which case this is the 3rd to
* the last field.
*/
#include <stdio.h>
#include <fcntl.h>
#include <sys/types.h>
#include <limits.h>
#include <linux/limits.h>
int latestcpu(pid_t pid)
{
char buf[PATH_MAX];
int fd;
int cpu = -1;
snprintf(buf, sizeof(buf), "/proc/%d/stat", pid);
fd = open(buf, O_RDONLY);
buf[0] = 0; /* in case fd < 0 and read() is no-op */
read(fd, buf, sizeof(buf));
close(fd);
sscanf(buf, "%*u %*s %*s %*u %*u %*u %*u %*u %*u %*u "
"%*u %*u %*u %*u %*u %*u %*u %*u %*u %*u "
"%*u %*u %*u %*u %*u %*u %*u %*u %*u %*u "
"%*u %*u %*u %*u %*u %*u %*u %*u %u", /* 39th field */
&cpu);
return cpu;
}
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <pj@sgi.com> 1.925.600.0401
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Inconsistent timing results of multithreaded program on an SMP machine.
[not found] <OF507D27BA.6B51F19A-ONC22570C3.002E62D2-C22570C3.002F0C99@il.ibm.com>
@ 2005-11-24 9:40 ` Con Kolivas
2005-11-24 10:00 ` Marcel Zalmanovici
0 siblings, 1 reply; 17+ messages in thread
From: Con Kolivas @ 2005-11-24 9:40 UTC (permalink / raw)
To: Marcel Zalmanovici; +Cc: linux-kernel, Muli Ben-Yehuda
On Thu, 24 Nov 2005 19:33, Marcel Zalmanovici wrote:
> Hi Con,
>
> I've tried the pthread_join theory.
> pthread_join completes very fast, no evidence on it being the perpetrator
> here.
>
> I've ran your ps -... idea in a different term window every ms while the
> test was running. It created a large file so I won't mail it to you, but
> both me and Muli observed that once the thread ends it quickly disappears
> from the ps list.
>
> I've also ran Muli's idea and added gettimeofday calls.
> Here's the altered code and output:
>
> (See attached file: idle.log)(See attached file: sched_test.c)
>
> As you can see, if a thread already finished pthread_join returns in a
> split second.
>
> Any other ideas are welcome :-)
Profile the actual application?
> P.S. - I agree that Lotus isn't ideal for this kind of conversations, but
> that's what IBM is using.
It was tongue in cheek ;) Well that should still not stop you from replying
below the original thread.
Cheers,
Con
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Inconsistent timing results of multithreaded program on an SMP machine.
2005-11-20 14:02 ` Paul Jackson
@ 2005-11-24 9:50 ` Marcel Zalmanovici
0 siblings, 0 replies; 17+ messages in thread
From: Marcel Zalmanovici @ 2005-11-24 9:50 UTC (permalink / raw)
To: Paul Jackson; +Cc: linux-kernel, Muli Ben-Yehuda, Shlomit Pinter
[-- Attachment #1: Type: text/plain, Size: 4186 bytes --]
Hi Paul,
I've tried your suggestion.
I tried first adding latestcpu( getpid() ), like you've said, but as you
can see from the log this did not yield anything interesting.
I've then tried with latestcpu( gettid() ). The results there surprised my
a bit:
I would've expected to see low times whenever the 2 threads using a DS are
on the same CPU and high times when they are placed on different CPUs.
Also lower times would be expected when the threads aer divided 4 to CPU 0
and 4 to CPU 1 (I've disabled hyperthreading for now).
If you look at the run that took 14.78sec you'll see that only the threads
of 1 out of 4 DS are on same CPU, but result is low. Conversly, look at the
19.55sec reults; same thing happens there.
Pretty much the same can be seen looking at the 14.62 and 19.57 results,
except there the threads of 2 out of 4 DS match.
As for the 2nd presumption, look at times 14.64 and 18.90 - threads are
divided 4 and 4 on both ocassions.
Here are the log files:
(See attached file: latest.tid.log)(See attached file: latest.pid.log)
Any other ideas are highly welcomed.
Marcel
Paul Jackson
<pj@sgi.com> To: Marcel Zalmanovici/Haifa/IBM@IBMIL
cc: linux-kernel@vger.kernel.org
20/11/2005 16:02 Subject: Re: Inconsistent timing results of multithreaded program on an SMP
machine.
Marcel wrote:
> Instead what I've got was an oscillation where the maximum time was twice
> and more than the minimum!! For a short test results ranged ~7sec to ~16
...
Just for grins, try displaying which cpu each thread runs on. Display the
return from "latestcpu(getpid())" in the two per-thread printf's, to
display
the thread's cpu at the beginning and end of each compute_thread().
Perhaps
you will notice that the per-thread cpu correlates with the test times.
/*
* int latestcpu(pid_t pid)
*
* Copyright (C) 2005 Silicon Graphics, Inc.
* This code is subject to the terms and conditions of the
* GNU General Public License.
*
* Return most recent CPU on which task pid executed.
*
* The last used CPU is visible for a given pid as field #39
* (starting with #1) in the file /proc/<pid>/stat. Currently
* this file has 41 fields, in which case this is the 3rd to
* the last field.
*/
#include <stdio.h>
#include <fcntl.h>
#include <sys/types.h>
#include <limits.h>
#include <linux/limits.h>
int latestcpu(pid_t pid)
{
char buf[PATH_MAX];
int fd;
int cpu = -1;
snprintf(buf, sizeof(buf), "/proc/%d/stat", pid);
fd = open(buf, O_RDONLY);
buf[0] = 0; /* in case fd < 0 and read() is no-op */
read(fd, buf, sizeof(buf));
close(fd);
sscanf(buf, "%*u %*s %*s %*u %*u %*u %*u %*u %*u %*u "
"%*u %*u %*u %*u %*u %*u %*u %*u %*u %*u "
"%*u %*u %*u %*u %*u %*u %*u %*u %*u %*u "
"%*u %*u %*u %*u %*u %*u %*u %*u %u", /* 39th
field */
&cpu);
return cpu;
}
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <pj@sgi.com> 1.925.600.0401
[-- Attachment #2: latest.tid.log --]
[-- Type: application/octet-stream, Size: 17766 bytes --]
Linux suse-marcel 2.6.4-Marcel-1 #122 SMP Sat Nov 12 21:24:16 IST 2005 i686 i686 i386 GNU/Linux
Thu Nov 24 11:15:55 IST 2005
Thread DS 0 TID = 23546, latest CPU = 0
Thread DS 1 TID = 23548, latest CPU = 1
Thread DS 0 TID = 23547, latest CPU = 0
Thread DS 1 TID = 23549, latest CPU = 1
Thread DS 2 TID = 23550, latest CPU = 1
Thread DS 2 TID = 23551, latest CPU = 1
Thread DS 3 TID = 23552, latest CPU = 1
Thread DS 3 TID = 23553, latest CPU = 1
Thread DS 0, TID = 23546 running ended, latest CPU = 0
Thread DS 0, TID = 23547 running ended, latest CPU = 0
Thread DS 1, TID = 23548 running ended, latest CPU = 1
Thread DS 3, TID = 23552 running ended, latest CPU = 0
Thread DS 3, TID = 23553 running ended, latest CPU = 0
Thread DS 2, TID = 23550 running ended, latest CPU = 0
Thread DS 2, TID = 23551 running ended, latest CPU = 1
Thread DS 1, TID = 23549 running ended, latest CPU = 0
Main exit ...
real 14.49
user 28.98
sys 0.00
CPU 199%
(major 0 + minor 187) page faults
swaps 0s vol 1446 invol 7
Thread DS 0 TID = 23556, latest CPU = 1
Thread DS 0 TID = 23557, latest CPU = 0
Thread DS 1 TID = 23558, latest CPU = 0
Thread DS 1 TID = 23559, latest CPU = 0
Thread DS 2 TID = 23560, latest CPU = 0
Thread DS 2 TID = 23561, latest CPU = 0
Thread DS 3 TID = 23562, latest CPU = 0
Thread DS 3 TID = 23563, latest CPU = 0
Thread DS 3, TID = 23562 running ended, latest CPU = 1
Thread DS 1, TID = 23558 running ended, latest CPU = 0
Thread DS 1, TID = 23559 running ended, latest CPU = 0
Thread DS 0, TID = 23556 running ended, latest CPU = 1
Thread DS 3, TID = 23563 running ended, latest CPU = 1
Thread DS 2, TID = 23561 running ended, latest CPU = 1
Thread DS 0, TID = 23557 running ended, latest CPU = 1
Thread DS 2, TID = 23560 running ended, latest CPU = 1
Main exit ...
real 15.18
user 30.29
sys 0.00
CPU 199%
(major 0 + minor 187) page faults
swaps 0s vol 1470 invol 4
Thread DS 0 TID = 23566, latest CPU = 0
Thread DS 0 TID = 23567, latest CPU = 1
Thread DS 1 TID = 23568, latest CPU = 1
Thread DS 1 TID = 23569, latest CPU = 1
Thread DS 2 TID = 23570, latest CPU = 1
Thread DS 2 TID = 23571, latest CPU = 1
Thread DS 3 TID = 23572, latest CPU = 1
Thread DS 3 TID = 23573, latest CPU = 1
Thread DS 0, TID = 23566 running ended, latest CPU = 0
Thread DS 0, TID = 23567 running ended, latest CPU = 1
Thread DS 3, TID = 23573 running ended, latest CPU = 1
Thread DS 3, TID = 23572 running ended, latest CPU = 1
Thread DS 2, TID = 23570 running ended, latest CPU = 0
Thread DS 1, TID = 23568 running ended, latest CPU = 1
Thread DS 2, TID = 23571 running ended, latest CPU = 0
Thread DS 1, TID = 23569 running ended, latest CPU = 0
Main exit ...
real 14.64
user 29.22
sys 0.00
CPU 199%
(major 0 + minor 189) page faults
swaps 0s vol 1429 invol 7
Thread DS 0 TID = 23576, latest CPU = 1
Thread DS 0 TID = 23577, latest CPU = 0
Thread DS 1 TID = 23578, latest CPU = 0
Thread DS 1 TID = 23579, latest CPU = 0
Thread DS 2 TID = 23580, latest CPU = 0
Thread DS 2 TID = 23581, latest CPU = 0
Thread DS 3 TID = 23582, latest CPU = 0
Thread DS 3 TID = 23583, latest CPU = 0
Thread DS 0, TID = 23576 running ended, latest CPU = 1
Thread DS 2, TID = 23581 running ended, latest CPU = 0
Thread DS 3, TID = 23583 running ended, latest CPU = 0
Thread DS 3, TID = 23582 running ended, latest CPU = 0
Thread DS 1, TID = 23578 running ended, latest CPU = 0
Thread DS 0, TID = 23577 running ended, latest CPU = 1
Thread DS 2, TID = 23580 running ended, latest CPU = 1
Thread DS 1, TID = 23579 running ended, latest CPU = 0
Main exit ...
real 14.65
user 29.29
sys 0.00
CPU 199%
(major 0 + minor 187) page faults
swaps 0s vol 1438 invol 4
Thread DS 0 TID = 23586, latest CPU = 0
Thread DS 0 TID = 23587, latest CPU = 1
Thread DS 1 TID = 23588, latest CPU = 1
Thread DS 1 TID = 23589, latest CPU = 1
Thread DS 2 TID = 23590, latest CPU = 1
Thread DS 2 TID = 23591, latest CPU = 1
Thread DS 3 TID = 23592, latest CPU = 1
Thread DS 3 TID = 23593, latest CPU = 1
Thread DS 1, TID = 23588 running ended, latest CPU = 1
Thread DS 2, TID = 23590 running ended, latest CPU = 1
Thread DS 3, TID = 23593 running ended, latest CPU = 1
Thread DS 0, TID = 23587 running ended, latest CPU = 0
Thread DS 0, TID = 23586 running ended, latest CPU = 0
Thread DS 2, TID = 23591 running ended, latest CPU = 1
Thread DS 1, TID = 23589 running ended, latest CPU = 0
Thread DS 3, TID = 23592 running ended, latest CPU = 1
Main exit ...
real 14.61
user 29.18
sys 0.00
CPU 199%
(major 0 + minor 187) page faults
swaps 0s vol 1444 invol 4
Thread DS 0 TID = 23596, latest CPU = 1
Thread DS 0 TID = 23597, latest CPU = 0
Thread DS 1 TID = 23598, latest CPU = 0
Thread DS 1 TID = 23599, latest CPU = 0
Thread DS 2 TID = 23600, latest CPU = 0
Thread DS 2 TID = 23601, latest CPU = 0
Thread DS 3 TID = 23602, latest CPU = 0
Thread DS 3 TID = 23603, latest CPU = 0
Thread DS 2, TID = 23601 running ended, latest CPU = 1
Thread DS 2, TID = 23600 running ended, latest CPU = 1
Thread DS 1, TID = 23598 running ended, latest CPU = 0
Thread DS 1, TID = 23599 running ended, latest CPU = 0
Thread DS 0, TID = 23596 running ended, latest CPU = 1
Thread DS 3, TID = 23602 running ended, latest CPU = 1
Thread DS 0, TID = 23597 running ended, latest CPU = 1
Thread DS 3, TID = 23603 running ended, latest CPU = 1
Main exit ...
real 18.89
user 37.74
sys 0.00
CPU 199%
(major 0 + minor 187) page faults
swaps 0s vol 1857 invol 4
Thread DS 0 TID = 23606, latest CPU = 0
Thread DS 0 TID = 23607, latest CPU = 1
Thread DS 1 TID = 23608, latest CPU = 1
Thread DS 1 TID = 23609, latest CPU = 1
Thread DS 2 TID = 23610, latest CPU = 1
Thread DS 2 TID = 23611, latest CPU = 1
Thread DS 3 TID = 23612, latest CPU = 1
Thread DS 3 TID = 23613, latest CPU = 1
Thread DS 0, TID = 23606 running ended, latest CPU = 0
Thread DS 2, TID = 23610 running ended, latest CPU = 1
Thread DS 2, TID = 23611 running ended, latest CPU = 1
Thread DS 0, TID = 23607 running ended, latest CPU = 1
Thread DS 1, TID = 23609 running ended, latest CPU = 0
Thread DS 1, TID = 23608 running ended, latest CPU = 0
Thread DS 3, TID = 23613 running ended, latest CPU = 0
Thread DS 3, TID = 23612 running ended, latest CPU = 1
Main exit ...
real 14.62
user 29.21
sys 0.00
CPU 199%
(major 0 + minor 187) page faults
swaps 0s vol 1435 invol 5
Thread DS 0 TID = 23616, latest CPU = 1
Thread DS 1 TID = 23618, latest CPU = 0
Thread DS 1 TID = 23619, latest CPU = 0
Thread DS 2 TID = 23620, latest CPU = 0
Thread DS 2 TID = 23621, latest CPU = 0
Thread DS 3 TID = 23622, latest CPU = 0
Thread DS 0 TID = 23617, latest CPU = 1
Thread DS 3 TID = 23623, latest CPU = 0
Thread DS 0, TID = 23616 running ended, latest CPU = 1
Thread DS 0, TID = 23617 running ended, latest CPU = 1
Thread DS 3, TID = 23623 running ended, latest CPU = 1
Thread DS 1, TID = 23619 running ended, latest CPU = 0
Thread DS 2, TID = 23621 running ended, latest CPU = 0
Thread DS 3, TID = 23622 running ended, latest CPU = 1
Thread DS 2, TID = 23620 running ended, latest CPU = 0
Thread DS 1, TID = 23618 running ended, latest CPU = 1
Main exit ...
real 14.51
user 28.99
sys 0.00
CPU 199%
(major 0 + minor 187) page faults
swaps 0s vol 1452 invol 4
Thread DS 0 TID = 23626, latest CPU = 0
Thread DS 0 TID = 23627, latest CPU = 1
Thread DS 1 TID = 23628, latest CPU = 1
Thread DS 1 TID = 23629, latest CPU = 1
Thread DS 2 TID = 23630, latest CPU = 1
Thread DS 2 TID = 23631, latest CPU = 1
Thread DS 3 TID = 23632, latest CPU = 1
Thread DS 3 TID = 23633, latest CPU = 1
Thread DS 0, TID = 23626 running ended, latest CPU = 0
Thread DS 2, TID = 23630 running ended, latest CPU = 0
Thread DS 2, TID = 23631 running ended, latest CPU = 0
Thread DS 3, TID = 23633 running ended, latest CPU = 0
Thread DS 3, TID = 23632 running ended, latest CPU = 1
Thread DS 1, TID = 23628 running ended, latest CPU = 1
Thread DS 1, TID = 23629 running ended, latest CPU = 1
Thread DS 0, TID = 23627 running ended, latest CPU = 0
Main exit ...
real 14.54
user 29.06
sys 0.00
CPU 199%
(major 0 + minor 187) page faults
swaps 0s vol 1450 invol 3
Thread DS 0 TID = 23636, latest CPU = 1
Thread DS 0 TID = 23637, latest CPU = 0
Thread DS 1 TID = 23638, latest CPU = 0
Thread DS 1 TID = 23639, latest CPU = 0
Thread DS 2 TID = 23640, latest CPU = 0
Thread DS 2 TID = 23641, latest CPU = 0
Thread DS 3 TID = 23642, latest CPU = 0
Thread DS 3 TID = 23643, latest CPU = 0
Thread DS 0, TID = 23636 running ended, latest CPU = 1
Thread DS 2, TID = 23641 running ended, latest CPU = 1
Thread DS 3, TID = 23643 running ended, latest CPU = 1
Thread DS 3, TID = 23642 running ended, latest CPU = 1
Thread DS 1, TID = 23639 running ended, latest CPU = 1
Thread DS 2, TID = 23640 running ended, latest CPU = 1
Thread DS 1, TID = 23638 running ended, latest CPU = 0
Thread DS 0, TID = 23637 running ended, latest CPU = 0
Main exit ...
real 14.73
user 29.32
sys 0.00
CPU 198%
(major 0 + minor 187) page faults
swaps 0s vol 1474 invol 3
Thread DS 0 TID = 23646, latest CPU = 0
Thread DS 0 TID = 23647, latest CPU = 1
Thread DS 1 TID = 23648, latest CPU = 1
Thread DS 1 TID = 23649, latest CPU = 1
Thread DS 2 TID = 23650, latest CPU = 1
Thread DS 2 TID = 23651, latest CPU = 1
Thread DS 3 TID = 23652, latest CPU = 1
Thread DS 3 TID = 23653, latest CPU = 1
Thread DS 0, TID = 23646 running ended, latest CPU = 0
Thread DS 1, TID = 23648 running ended, latest CPU = 1
Thread DS 0, TID = 23647 running ended, latest CPU = 1
Thread DS 3, TID = 23652 running ended, latest CPU = 1
Thread DS 3, TID = 23653 running ended, latest CPU = 1
Thread DS 1, TID = 23649 running ended, latest CPU = 0
Thread DS 2, TID = 23651 running ended, latest CPU = 1
Thread DS 2, TID = 23650 running ended, latest CPU = 0
Main exit ...
real 14.78
user 29.55
sys 0.00
CPU 199%
(major 0 + minor 187) page faults
swaps 0s vol 1443 invol 5
Thread DS 0 TID = 23656, latest CPU = 1
Thread DS 0 TID = 23657, latest CPU = 0
Thread DS 1 TID = 23658, latest CPU = 0
Thread DS 1 TID = 23659, latest CPU = 0
Thread DS 2 TID = 23660, latest CPU = 0
Thread DS 2 TID = 23661, latest CPU = 0
Thread DS 3 TID = 23662, latest CPU = 0
Thread DS 3 TID = 23663, latest CPU = 0
Thread DS 1, TID = 23659 running ended, latest CPU = 1
Thread DS 1, TID = 23658 running ended, latest CPU = 1
Thread DS 3, TID = 23663 running ended, latest CPU = 1
Thread DS 2, TID = 23661 running ended, latest CPU = 0
Thread DS 3, TID = 23662 running ended, latest CPU = 0
Thread DS 2, TID = 23660 running ended, latest CPU = 0
Thread DS 0, TID = 23656 running ended, latest CPU = 1
Thread DS 0, TID = 23657 running ended, latest CPU = 0
Main exit ...
real 18.90
user 37.34
sys 0.00
CPU 197%
(major 0 + minor 187) page faults
swaps 0s vol 1602 invol 3
Thread DS 0 TID = 23666, latest CPU = 0
Thread DS 0 TID = 23667, latest CPU = 1
Thread DS 1 TID = 23668, latest CPU = 1
Thread DS 1 TID = 23669, latest CPU = 1
Thread DS 2 TID = 23670, latest CPU = 1
Thread DS 2 TID = 23671, latest CPU = 1
Thread DS 3 TID = 23672, latest CPU = 1
Thread DS 3 TID = 23673, latest CPU = 1
Thread DS 0, TID = 23666 running ended, latest CPU = 0
Thread DS 3, TID = 23672 running ended, latest CPU = 0
Thread DS 1, TID = 23668 running ended, latest CPU = 1
Thread DS 3, TID = 23673 running ended, latest CPU = 1
Thread DS 2, TID = 23671 running ended, latest CPU = 1
Thread DS 1, TID = 23669 running ended, latest CPU = 0
Thread DS 0, TID = 23667 running ended, latest CPU = 0
Thread DS 2, TID = 23670 running ended, latest CPU = 1
Main exit ...
real 14.57
user 29.14
sys 0.00
CPU 199%
(major 0 + minor 187) page faults
swaps 0s vol 1458 invol 4
Thread DS 0 TID = 23676, latest CPU = 1
Thread DS 0 TID = 23677, latest CPU = 0
Thread DS 1 TID = 23678, latest CPU = 0
Thread DS 1 TID = 23679, latest CPU = 0
Thread DS 2 TID = 23680, latest CPU = 0
Thread DS 2 TID = 23681, latest CPU = 0
Thread DS 3 TID = 23682, latest CPU = 0
Thread DS 3 TID = 23683, latest CPU = 0
Thread DS 0, TID = 23676 running ended, latest CPU = 1
Thread DS 2, TID = 23681 running ended, latest CPU = 0
Thread DS 1, TID = 23678 running ended, latest CPU = 1
Thread DS 1, TID = 23679 running ended, latest CPU = 0
Thread DS 2, TID = 23680 running ended, latest CPU = 1
Thread DS 0, TID = 23677 running ended, latest CPU = 1
Thread DS 3, TID = 23683 running ended, latest CPU = 0
Thread DS 3, TID = 23682 running ended, latest CPU = 0
Main exit ...
real 17.07
user 34.00
sys 0.00
CPU 199%
(major 0 + minor 187) page faults
swaps 0s vol 1568 invol 4
Thread DS 0 TID = 23687, latest CPU = 0
Thread DS 0 TID = 23688, latest CPU = 1
Thread DS 1 TID = 23689, latest CPU = 1
Thread DS 1 TID = 23690, latest CPU = 1
Thread DS 2 TID = 23691, latest CPU = 1
Thread DS 2 TID = 23692, latest CPU = 1
Thread DS 3 TID = 23693, latest CPU = 1
Thread DS 3 TID = 23694, latest CPU = 1
Thread DS 0, TID = 23687 running ended, latest CPU = 0
Thread DS 0, TID = 23688 running ended, latest CPU = 1
Thread DS 3, TID = 23694 running ended, latest CPU = 1
Thread DS 3, TID = 23693 running ended, latest CPU = 1
Thread DS 1, TID = 23690 running ended, latest CPU = 1
Thread DS 1, TID = 23689 running ended, latest CPU = 1
Thread DS 2, TID = 23691 running ended, latest CPU = 0
Thread DS 2, TID = 23692 running ended, latest CPU = 1
Main exit ...
real 14.56
user 29.11
sys 0.00
CPU 199%
(major 0 + minor 187) page faults
swaps 0s vol 1437 invol 6
Thread DS 0 TID = 23697, latest CPU = 1
Thread DS 2 TID = 23701, latest CPU = 1
Thread DS 1 TID = 23699, latest CPU = 0
Thread DS 0 TID = 23698, latest CPU = 1
Thread DS 1 TID = 23700, latest CPU = 0
Thread DS 2 TID = 23702, latest CPU = 1
Thread DS 3 TID = 23703, latest CPU = 1
Thread DS 3 TID = 23704, latest CPU = 1
Thread DS 1, TID = 23700 running ended, latest CPU = 0
Thread DS 0, TID = 23698 running ended, latest CPU = 1
Thread DS 1, TID = 23699 running ended, latest CPU = 0
Thread DS 2, TID = 23701 running ended, latest CPU = 1
Thread DS 3, TID = 23704 running ended, latest CPU = 0
Thread DS 0, TID = 23697 running ended, latest CPU = 0
Thread DS 3, TID = 23703 running ended, latest CPU = 0
Thread DS 2, TID = 23702 running ended, latest CPU = 1
Main exit ...
real 14.49
user 28.98
sys 0.00
CPU 199%
(major 0 + minor 191) page faults
swaps 0s vol 1435 invol 11
Thread DS 0 TID = 23707, latest CPU = 0
Thread DS 0 TID = 23708, latest CPU = 1
Thread DS 1 TID = 23709, latest CPU = 1
Thread DS 1 TID = 23710, latest CPU = 1
Thread DS 2 TID = 23711, latest CPU = 1
Thread DS 2 TID = 23712, latest CPU = 1
Thread DS 3 TID = 23713, latest CPU = 1
Thread DS 3 TID = 23714, latest CPU = 1
Thread DS 1, TID = 23709 running ended, latest CPU = 1
Thread DS 1, TID = 23710 running ended, latest CPU = 1
Thread DS 3, TID = 23713 running ended, latest CPU = 0
Thread DS 3, TID = 23714 running ended, latest CPU = 0
Thread DS 2, TID = 23711 running ended, latest CPU = 1
Thread DS 2, TID = 23712 running ended, latest CPU = 0
Thread DS 0, TID = 23707 running ended, latest CPU = 0
Thread DS 0, TID = 23708 running ended, latest CPU = 1
Main exit ...
real 17.80
user 35.45
sys 0.00
CPU 199%
(major 0 + minor 187) page faults
swaps 0s vol 1623 invol 6
Thread DS 0 TID = 23717, latest CPU = 1
Thread DS 0 TID = 23718, latest CPU = 0
Thread DS 1 TID = 23719, latest CPU = 0
Thread DS 1 TID = 23720, latest CPU = 0
Thread DS 2 TID = 23721, latest CPU = 0
Thread DS 2 TID = 23722, latest CPU = 0
Thread DS 3 TID = 23723, latest CPU = 0
Thread DS 3 TID = 23724, latest CPU = 0
Thread DS 1, TID = 23719 running ended, latest CPU = 0
Thread DS 0, TID = 23718 running ended, latest CPU = 1
Thread DS 2, TID = 23722 running ended, latest CPU = 1
Thread DS 2, TID = 23721 running ended, latest CPU = 1
Thread DS 3, TID = 23724 running ended, latest CPU = 1
Thread DS 3, TID = 23723 running ended, latest CPU = 1
Thread DS 0, TID = 23717 running ended, latest CPU = 0
Thread DS 1, TID = 23720 running ended, latest CPU = 1
Main exit ...
real 14.62
user 29.19
sys 0.00
CPU 199%
(major 0 + minor 187) page faults
swaps 0s vol 1426 invol 3
Thread DS 0 TID = 23727, latest CPU = 0
Thread DS 0 TID = 23728, latest CPU = 1
Thread DS 1 TID = 23729, latest CPU = 1
Thread DS 1 TID = 23730, latest CPU = 1
Thread DS 2 TID = 23731, latest CPU = 1
Thread DS 2 TID = 23732, latest CPU = 1
Thread DS 3 TID = 23733, latest CPU = 1
Thread DS 3 TID = 23734, latest CPU = 1
Thread DS 2, TID = 23732 running ended, latest CPU = 0
Thread DS 2, TID = 23731 running ended, latest CPU = 1
Thread DS 1, TID = 23729 running ended, latest CPU = 1
Thread DS 1, TID = 23730 running ended, latest CPU = 0
Thread DS 0, TID = 23727 running ended, latest CPU = 0
Thread DS 3, TID = 23733 running ended, latest CPU = 0
Thread DS 0, TID = 23728 running ended, latest CPU = 1
Thread DS 3, TID = 23734 running ended, latest CPU = 0
Main exit ...
real 19.55
user 39.06
sys 0.00
CPU 199%
(major 0 + minor 190) page faults
swaps 0s vol 1937 invol 7
Thread DS 0 TID = 23737, latest CPU = 1
Thread DS 0 TID = 23738, latest CPU = 0
Thread DS 1 TID = 23739, latest CPU = 0
Thread DS 1 TID = 23740, latest CPU = 0
Thread DS 2 TID = 23741, latest CPU = 0
Thread DS 2 TID = 23742, latest CPU = 0
Thread DS 3 TID = 23743, latest CPU = 0
Thread DS 3 TID = 23744, latest CPU = 0
Thread DS 1, TID = 23739 running ended, latest CPU = 1
Thread DS 3, TID = 23744 running ended, latest CPU = 1
Thread DS 3, TID = 23743 running ended, latest CPU = 1
Thread DS 1, TID = 23740 running ended, latest CPU = 0
Thread DS 2, TID = 23741 running ended, latest CPU = 0
Thread DS 2, TID = 23742 running ended, latest CPU = 0
Thread DS 0, TID = 23737 running ended, latest CPU = 1
Thread DS 0, TID = 23738 running ended, latest CPU = 0
Main exit ...
real 19.57
user 38.75
sys 0.00
CPU 197%
(major 0 + minor 187) page faults
swaps 0s vol 1606 invol 3
[-- Attachment #3: latest.pid.log --]
[-- Type: application/octet-stream, Size: 17765 bytes --]
Linux suse-marcel 2.6.4-Marcel-1 #122 SMP Sat Nov 12 21:24:16 IST 2005 i686 i686 i386 GNU/Linux
Thu Nov 24 11:09:21 IST 2005
Thread DS 0 TID = 23309, latest CPU = 1
Thread DS 0 TID = 23310, latest CPU = 0
Thread DS 1 TID = 23311, latest CPU = 0
Thread DS 1 TID = 23312, latest CPU = 0
Thread DS 2 TID = 23313, latest CPU = 0
Thread DS 2 TID = 23314, latest CPU = 0
Thread DS 3 TID = 23315, latest CPU = 0
Thread DS 3 TID = 23316, latest CPU = 0
Thread DS 0, TID = 23309 running ended, latest CPU = 0
Thread DS 3, TID = 23316 running ended, latest CPU = 0
Thread DS 2, TID = 23313 running ended, latest CPU = 0
Thread DS 1, TID = 23312 running ended, latest CPU = 0
Thread DS 3, TID = 23315 running ended, latest CPU = 0
Thread DS 1, TID = 23311 running ended, latest CPU = 0
Thread DS 2, TID = 23314 running ended, latest CPU = 0
Thread DS 0, TID = 23310 running ended, latest CPU = 0
Main exit ...
real 14.59
user 29.17
sys 0.00
CPU 199%
(major 0 + minor 187) page faults
swaps 0s vol 1390 invol 7
Thread DS 0 TID = 23319, latest CPU = 0
Thread DS 0 TID = 23320, latest CPU = 1
Thread DS 1 TID = 23321, latest CPU = 1
Thread DS 1 TID = 23322, latest CPU = 1
Thread DS 2 TID = 23323, latest CPU = 1
Thread DS 2 TID = 23324, latest CPU = 1
Thread DS 3 TID = 23325, latest CPU = 1
Thread DS 3 TID = 23326, latest CPU = 1
Thread DS 0, TID = 23319 running ended, latest CPU = 1
Thread DS 2, TID = 23324 running ended, latest CPU = 1
Thread DS 3, TID = 23326 running ended, latest CPU = 1
Thread DS 2, TID = 23323 running ended, latest CPU = 1
Thread DS 1, TID = 23322 running ended, latest CPU = 1
Thread DS 1, TID = 23321 running ended, latest CPU = 1
Thread DS 3, TID = 23325 running ended, latest CPU = 1
Thread DS 0, TID = 23320 running ended, latest CPU = 1
Main exit ...
real 14.69
user 29.33
sys 0.00
CPU 199%
(major 0 + minor 187) page faults
swaps 0s vol 1394 invol 3
Thread DS 0 TID = 23329, latest CPU = 1
Thread DS 0 TID = 23330, latest CPU = 0
Thread DS 1 TID = 23331, latest CPU = 0
Thread DS 1 TID = 23332, latest CPU = 0
Thread DS 2 TID = 23333, latest CPU = 0
Thread DS 2 TID = 23334, latest CPU = 0
Thread DS 3 TID = 23335, latest CPU = 0
Thread DS 3 TID = 23336, latest CPU = 0
Thread DS 2, TID = 23334 running ended, latest CPU = 0
Thread DS 2, TID = 23333 running ended, latest CPU = 0
Thread DS 1, TID = 23331 running ended, latest CPU = 0
Thread DS 1, TID = 23332 running ended, latest CPU = 0
Thread DS 0, TID = 23329 running ended, latest CPU = 0
Thread DS 3, TID = 23335 running ended, latest CPU = 0
Thread DS 0, TID = 23330 running ended, latest CPU = 0
Thread DS 3, TID = 23336 running ended, latest CPU = 0
Main exit ...
real 20.08
user 40.06
sys 0.00
CPU 199%
(major 0 + minor 187) page faults
swaps 0s vol 1904 invol 4
Thread DS 0 TID = 23339, latest CPU = 0
Thread DS 0 TID = 23340, latest CPU = 1
Thread DS 1 TID = 23341, latest CPU = 1
Thread DS 1 TID = 23342, latest CPU = 1
Thread DS 2 TID = 23343, latest CPU = 1
Thread DS 2 TID = 23344, latest CPU = 1
Thread DS 3 TID = 23345, latest CPU = 1
Thread DS 3 TID = 23346, latest CPU = 1
Thread DS 0, TID = 23339 running ended, latest CPU = 1
Thread DS 0, TID = 23340 running ended, latest CPU = 1
Thread DS 3, TID = 23346 running ended, latest CPU = 1
Thread DS 3, TID = 23345 running ended, latest CPU = 1
Thread DS 1, TID = 23342 running ended, latest CPU = 1
Thread DS 1, TID = 23341 running ended, latest CPU = 1
Thread DS 2, TID = 23343 running ended, latest CPU = 1
Thread DS 2, TID = 23344 running ended, latest CPU = 1
Main exit ...
real 14.57
user 29.12
sys 0.00
CPU 199%
(major 0 + minor 187) page faults
swaps 0s vol 1396 invol 6
Thread DS 0 TID = 23349, latest CPU = 1
Thread DS 0 TID = 23350, latest CPU = 0
Thread DS 1 TID = 23351, latest CPU = 0
Thread DS 1 TID = 23352, latest CPU = 0
Thread DS 2 TID = 23353, latest CPU = 0
Thread DS 2 TID = 23354, latest CPU = 0
Thread DS 3 TID = 23355, latest CPU = 0
Thread DS 3 TID = 23356, latest CPU = 0
Thread DS 0, TID = 23349 running ended, latest CPU = 0
Thread DS 3, TID = 23355 running ended, latest CPU = 0
Thread DS 3, TID = 23356 running ended, latest CPU = 0
Thread DS 2, TID = 23354 running ended, latest CPU = 0
Thread DS 0, TID = 23350 running ended, latest CPU = 0
Thread DS 2, TID = 23353 running ended, latest CPU = 0
Thread DS 1, TID = 23351 running ended, latest CPU = 0
Thread DS 1, TID = 23352 running ended, latest CPU = 0
Main exit ...
real 17.14
user 34.24
sys 0.00
CPU 199%
(major 0 + minor 187) page faults
swaps 0s vol 1469 invol 5
Thread DS 0 TID = 23359, latest CPU = 0
Thread DS 0 TID = 23360, latest CPU = 1
Thread DS 1 TID = 23361, latest CPU = 1
Thread DS 1 TID = 23362, latest CPU = 1
Thread DS 2 TID = 23363, latest CPU = 1
Thread DS 2 TID = 23364, latest CPU = 1
Thread DS 3 TID = 23365, latest CPU = 1
Thread DS 3 TID = 23366, latest CPU = 1
Thread DS 0, TID = 23359 running ended, latest CPU = 1
Thread DS 1, TID = 23362 running ended, latest CPU = 1
Thread DS 0, TID = 23360 running ended, latest CPU = 1
Thread DS 2, TID = 23363 running ended, latest CPU = 1
Thread DS 3, TID = 23365 running ended, latest CPU = 1
Thread DS 2, TID = 23364 running ended, latest CPU = 1
Thread DS 1, TID = 23361 running ended, latest CPU = 1
Thread DS 3, TID = 23366 running ended, latest CPU = 1
Main exit ...
real 14.59
user 29.16
sys 0.00
CPU 199%
(major 0 + minor 187) page faults
swaps 0s vol 1394 invol 5
Thread DS 0 TID = 23369, latest CPU = 0
Thread DS 0 TID = 23370, latest CPU = 1
Thread DS 1 TID = 23371, latest CPU = 1
Thread DS 1 TID = 23372, latest CPU = 1
Thread DS 2 TID = 23373, latest CPU = 1
Thread DS 2 TID = 23374, latest CPU = 1
Thread DS 3 TID = 23375, latest CPU = 1
Thread DS 3 TID = 23376, latest CPU = 1
Thread DS 0, TID = 23370 running ended, latest CPU = 1
Thread DS 2, TID = 23374 running ended, latest CPU = 1
Thread DS 2, TID = 23373 running ended, latest CPU = 1
Thread DS 1, TID = 23372 running ended, latest CPU = 1
Thread DS 3, TID = 23376 running ended, latest CPU = 1
Thread DS 1, TID = 23371 running ended, latest CPU = 1
Thread DS 0, TID = 23369 running ended, latest CPU = 1
Thread DS 3, TID = 23375 running ended, latest CPU = 1
Main exit ...
real 14.68
user 29.35
sys 0.00
CPU 199%
(major 0 + minor 187) page faults
swaps 0s vol 1396 invol 3
Thread DS 0 TID = 23379, latest CPU = 0
Thread DS 0 TID = 23380, latest CPU = 1
Thread DS 1 TID = 23381, latest CPU = 1
Thread DS 1 TID = 23382, latest CPU = 1
Thread DS 2 TID = 23383, latest CPU = 1
Thread DS 2 TID = 23384, latest CPU = 1
Thread DS 3 TID = 23385, latest CPU = 1
Thread DS 3 TID = 23386, latest CPU = 1
Thread DS 1, TID = 23381 running ended, latest CPU = 1
Thread DS 1, TID = 23382 running ended, latest CPU = 1
Thread DS 3, TID = 23386 running ended, latest CPU = 1
Thread DS 3, TID = 23385 running ended, latest CPU = 1
Thread DS 0, TID = 23379 running ended, latest CPU = 1
Thread DS 0, TID = 23380 running ended, latest CPU = 1
Thread DS 2, TID = 23383 running ended, latest CPU = 1
Thread DS 2, TID = 23384 running ended, latest CPU = 1
Main exit ...
real 15.60
user 31.11
sys 0.00
CPU 199%
(major 0 + minor 187) page faults
swaps 0s vol 1477 invol 5
Thread DS 0 TID = 23389, latest CPU = 0
Thread DS 0 TID = 23390, latest CPU = 1
Thread DS 1 TID = 23391, latest CPU = 1
Thread DS 1 TID = 23392, latest CPU = 1
Thread DS 2 TID = 23393, latest CPU = 1
Thread DS 2 TID = 23394, latest CPU = 1
Thread DS 3 TID = 23395, latest CPU = 1
Thread DS 3 TID = 23396, latest CPU = 1
Thread DS 2, TID = 23394 running ended, latest CPU = 1
Thread DS 3, TID = 23395 running ended, latest CPU = 1
Thread DS 0, TID = 23390 running ended, latest CPU = 1
Thread DS 1, TID = 23392 running ended, latest CPU = 1
Thread DS 2, TID = 23393 running ended, latest CPU = 1
Thread DS 0, TID = 23389 running ended, latest CPU = 1
Thread DS 3, TID = 23396 running ended, latest CPU = 1
Thread DS 1, TID = 23391 running ended, latest CPU = 1
Main exit ...
real 14.61
user 29.21
sys 0.00
CPU 199%
(major 0 + minor 187) page faults
swaps 0s vol 1381 invol 3
Thread DS 0 TID = 23399, latest CPU = 1
Thread DS 0 TID = 23400, latest CPU = 0
Thread DS 1 TID = 23401, latest CPU = 0
Thread DS 1 TID = 23402, latest CPU = 0
Thread DS 2 TID = 23403, latest CPU = 0
Thread DS 2 TID = 23404, latest CPU = 0
Thread DS 3 TID = 23405, latest CPU = 0
Thread DS 3 TID = 23406, latest CPU = 0
Thread DS 1, TID = 23402 running ended, latest CPU = 0
Thread DS 1, TID = 23401 running ended, latest CPU = 0
Thread DS 3, TID = 23406 running ended, latest CPU = 0
Thread DS 2, TID = 23404 running ended, latest CPU = 0
Thread DS 3, TID = 23405 running ended, latest CPU = 0
Thread DS 2, TID = 23403 running ended, latest CPU = 0
Thread DS 0, TID = 23399 running ended, latest CPU = 0
Thread DS 0, TID = 23400 running ended, latest CPU = 0
Main exit ...
real 19.12
user 37.78
sys 0.00
CPU 197%
(major 0 + minor 187) page faults
swaps 0s vol 1530 invol 3
Thread DS 0 TID = 23409, latest CPU = 0
Thread DS 0 TID = 23410, latest CPU = 1
Thread DS 1 TID = 23411, latest CPU = 1
Thread DS 1 TID = 23412, latest CPU = 1
Thread DS 2 TID = 23413, latest CPU = 1
Thread DS 2 TID = 23414, latest CPU = 1
Thread DS 3 TID = 23415, latest CPU = 1
Thread DS 3 TID = 23416, latest CPU = 1
Thread DS 0, TID = 23409 running ended, latest CPU = 1
Thread DS 0, TID = 23410 running ended, latest CPU = 1
Thread DS 2, TID = 23413 running ended, latest CPU = 1
Thread DS 3, TID = 23415 running ended, latest CPU = 1
Thread DS 3, TID = 23416 running ended, latest CPU = 1
Thread DS 2, TID = 23414 running ended, latest CPU = 1
Thread DS 1, TID = 23412 running ended, latest CPU = 1
Thread DS 1, TID = 23411 running ended, latest CPU = 1
Main exit ...
real 17.10
user 34.17
sys 0.00
CPU 199%
(major 0 + minor 187) page faults
swaps 0s vol 1481 invol 4
Thread DS 0 TID = 23419, latest CPU = 1
Thread DS 0 TID = 23420, latest CPU = 0
Thread DS 1 TID = 23421, latest CPU = 0
Thread DS 1 TID = 23422, latest CPU = 0
Thread DS 2 TID = 23423, latest CPU = 0
Thread DS 2 TID = 23424, latest CPU = 0
Thread DS 3 TID = 23425, latest CPU = 0
Thread DS 3 TID = 23426, latest CPU = 0
Thread DS 0, TID = 23419 running ended, latest CPU = 0
Thread DS 1, TID = 23421 running ended, latest CPU = 0
Thread DS 3, TID = 23425 running ended, latest CPU = 0
Thread DS 3, TID = 23426 running ended, latest CPU = 0
Thread DS 2, TID = 23424 running ended, latest CPU = 0
Thread DS 0, TID = 23420 running ended, latest CPU = 0
Thread DS 1, TID = 23422 running ended, latest CPU = 0
Thread DS 2, TID = 23423 running ended, latest CPU = 0
Main exit ...
real 14.62
user 29.18
sys 0.00
CPU 199%
(major 0 + minor 187) page faults
swaps 0s vol 1382 invol 6
Thread DS 0 TID = 23429, latest CPU = 0
Thread DS 0 TID = 23430, latest CPU = 1
Thread DS 1 TID = 23431, latest CPU = 1
Thread DS 1 TID = 23432, latest CPU = 1
Thread DS 2 TID = 23433, latest CPU = 1
Thread DS 2 TID = 23434, latest CPU = 1
Thread DS 3 TID = 23435, latest CPU = 1
Thread DS 3 TID = 23436, latest CPU = 1
Thread DS 3, TID = 23435 running ended, latest CPU = 1
Thread DS 1, TID = 23431 running ended, latest CPU = 1
Thread DS 1, TID = 23432 running ended, latest CPU = 1
Thread DS 2, TID = 23433 running ended, latest CPU = 1
Thread DS 3, TID = 23436 running ended, latest CPU = 1
Thread DS 2, TID = 23434 running ended, latest CPU = 1
Thread DS 0, TID = 23429 running ended, latest CPU = 1
Thread DS 0, TID = 23430 running ended, latest CPU = 1
Main exit ...
real 15.68
user 31.17
sys 0.00
CPU 198%
(major 0 + minor 187) page faults
swaps 0s vol 1422 invol 3
Thread DS 0 TID = 23439, latest CPU = 1
Thread DS 0 TID = 23440, latest CPU = 0
Thread DS 1 TID = 23441, latest CPU = 0
Thread DS 1 TID = 23442, latest CPU = 0
Thread DS 2 TID = 23443, latest CPU = 0
Thread DS 2 TID = 23444, latest CPU = 0
Thread DS 3 TID = 23445, latest CPU = 0
Thread DS 3 TID = 23446, latest CPU = 0
Thread DS 2, TID = 23443 running ended, latest CPU = 0
Thread DS 1, TID = 23441 running ended, latest CPU = 0
Thread DS 3, TID = 23446 running ended, latest CPU = 0
Thread DS 3, TID = 23445 running ended, latest CPU = 0
Thread DS 1, TID = 23442 running ended, latest CPU = 0
Thread DS 2, TID = 23444 running ended, latest CPU = 0
Thread DS 0, TID = 23439 running ended, latest CPU = 0
Thread DS 0, TID = 23440 running ended, latest CPU = 0
Main exit ...
real 19.91
user 39.41
sys 0.00
CPU 197%
(major 0 + minor 192) page faults
swaps 0s vol 1503 invol 5
Thread DS 0 TID = 23449, latest CPU = 1
Thread DS 0 TID = 23450, latest CPU = 0
Thread DS 1 TID = 23451, latest CPU = 0
Thread DS 1 TID = 23452, latest CPU = 0
Thread DS 2 TID = 23453, latest CPU = 0
Thread DS 2 TID = 23454, latest CPU = 0
Thread DS 3 TID = 23455, latest CPU = 0
Thread DS 3 TID = 23456, latest CPU = 0
Thread DS 1, TID = 23451 running ended, latest CPU = 0
Thread DS 0, TID = 23449 running ended, latest CPU = 0
Thread DS 3, TID = 23456 running ended, latest CPU = 0
Thread DS 3, TID = 23455 running ended, latest CPU = 0
Thread DS 0, TID = 23450 running ended, latest CPU = 0
Thread DS 2, TID = 23453 running ended, latest CPU = 0
Thread DS 2, TID = 23454 running ended, latest CPU = 0
Thread DS 1, TID = 23452 running ended, latest CPU = 0
Main exit ...
real 14.55
user 29.10
sys 0.00
CPU 199%
(major 0 + minor 187) page faults
swaps 0s vol 1427 invol 4
Thread DS 0 TID = 23459, latest CPU = 0
Thread DS 0 TID = 23460, latest CPU = 1
Thread DS 1 TID = 23461, latest CPU = 1
Thread DS 1 TID = 23462, latest CPU = 1
Thread DS 2 TID = 23463, latest CPU = 1
Thread DS 2 TID = 23464, latest CPU = 1
Thread DS 3 TID = 23465, latest CPU = 1
Thread DS 3 TID = 23466, latest CPU = 1
Thread DS 1, TID = 23461 running ended, latest CPU = 1
Thread DS 1, TID = 23462 running ended, latest CPU = 1
Thread DS 3, TID = 23466 running ended, latest CPU = 1
Thread DS 3, TID = 23465 running ended, latest CPU = 1
Thread DS 0, TID = 23459 running ended, latest CPU = 1
Thread DS 0, TID = 23460 running ended, latest CPU = 1
Thread DS 2, TID = 23463 running ended, latest CPU = 1
Thread DS 2, TID = 23464 running ended, latest CPU = 1
Main exit ...
real 19.61
user 39.14
sys 0.00
CPU 199%
(major 0 + minor 187) page faults
swaps 0s vol 1835 invol 5
Thread DS 0 TID = 23469, latest CPU = 1
Thread DS 0 TID = 23470, latest CPU = 0
Thread DS 1 TID = 23471, latest CPU = 0
Thread DS 1 TID = 23472, latest CPU = 0
Thread DS 2 TID = 23473, latest CPU = 0
Thread DS 2 TID = 23474, latest CPU = 0
Thread DS 3 TID = 23475, latest CPU = 0
Thread DS 3 TID = 23476, latest CPU = 0
Thread DS 3, TID = 23476 running ended, latest CPU = 0
Thread DS 3, TID = 23475 running ended, latest CPU = 0
Thread DS 2, TID = 23474 running ended, latest CPU = 0
Thread DS 2, TID = 23473 running ended, latest CPU = 0
Thread DS 1, TID = 23472 running ended, latest CPU = 0
Thread DS 1, TID = 23471 running ended, latest CPU = 0
Thread DS 0, TID = 23469 running ended, latest CPU = 0
Thread DS 0, TID = 23470 running ended, latest CPU = 0
Main exit ...
real 21.46
user 42.56
sys 0.00
CPU 198%
(major 0 + minor 187) page faults
swaps 0s vol 1612 invol 3
Thread DS 0 TID = 23479, latest CPU = 0
Thread DS 0 TID = 23480, latest CPU = 1
Thread DS 1 TID = 23481, latest CPU = 1
Thread DS 1 TID = 23482, latest CPU = 1
Thread DS 2 TID = 23483, latest CPU = 1
Thread DS 2 TID = 23484, latest CPU = 1
Thread DS 3 TID = 23485, latest CPU = 1
Thread DS 3 TID = 23486, latest CPU = 1
Thread DS 0, TID = 23479 running ended, latest CPU = 1
Thread DS 1, TID = 23482 running ended, latest CPU = 1
Thread DS 2, TID = 23483 running ended, latest CPU = 1
Thread DS 3, TID = 23485 running ended, latest CPU = 1
Thread DS 1, TID = 23481 running ended, latest CPU = 1
Thread DS 3, TID = 23486 running ended, latest CPU = 1
Thread DS 2, TID = 23484 running ended, latest CPU = 1
Thread DS 0, TID = 23480 running ended, latest CPU = 1
Main exit ...
real 14.55
user 29.09
sys 0.00
CPU 199%
(major 0 + minor 187) page faults
swaps 0s vol 1377 invol 3
Thread DS 0 TID = 23489, latest CPU = 1
Thread DS 0 TID = 23490, latest CPU = 0
Thread DS 1 TID = 23491, latest CPU = 0
Thread DS 1 TID = 23492, latest CPU = 0
Thread DS 2 TID = 23493, latest CPU = 0
Thread DS 2 TID = 23494, latest CPU = 0
Thread DS 3 TID = 23495, latest CPU = 0
Thread DS 3 TID = 23496, latest CPU = 0
Thread DS 0, TID = 23489 running ended, latest CPU = 0
Thread DS 3, TID = 23496 running ended, latest CPU = 0
Thread DS 3, TID = 23495 running ended, latest CPU = 0
Thread DS 0, TID = 23490 running ended, latest CPU = 0
Thread DS 1, TID = 23491 running ended, latest CPU = 0
Thread DS 1, TID = 23492 running ended, latest CPU = 0
Thread DS 2, TID = 23494 running ended, latest CPU = 0
Thread DS 2, TID = 23493 running ended, latest CPU = 0
Main exit ...
real 14.56
user 29.12
sys 0.00
CPU 199%
(major 0 + minor 187) page faults
swaps 0s vol 1396 invol 6
Thread DS 0 TID = 23499, latest CPU = 0
Thread DS 0 TID = 23500, latest CPU = 0
Thread DS 1 TID = 23501, latest CPU = 0
Thread DS 1 TID = 23502, latest CPU = 0
Thread DS 2 TID = 23503, latest CPU = 0
Thread DS 2 TID = 23504, latest CPU = 0
Thread DS 3 TID = 23505, latest CPU = 0
Thread DS 3 TID = 23506, latest CPU = 0
Thread DS 1, TID = 23501 running ended, latest CPU = 0
Thread DS 0, TID = 23499 running ended, latest CPU = 0
Thread DS 0, TID = 23500 running ended, latest CPU = 0
Thread DS 3, TID = 23506 running ended, latest CPU = 0
Thread DS 1, TID = 23502 running ended, latest CPU = 0
Thread DS 2, TID = 23504 running ended, latest CPU = 0
Thread DS 3, TID = 23505 running ended, latest CPU = 0
Thread DS 2, TID = 23503 running ended, latest CPU = 0
Main exit ...
real 14.72
user 29.43
sys 0.00
CPU 199%
(major 0 + minor 187) page faults
swaps 0s vol 1382 invol 5
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Inconsistent timing results of multithreaded program on an SMP machine.
2005-11-24 9:40 ` Inconsistent timing results of multithreaded program on an SMP machine Con Kolivas
@ 2005-11-24 10:00 ` Marcel Zalmanovici
2005-11-24 12:43 ` Paul Jackson
0 siblings, 1 reply; 17+ messages in thread
From: Marcel Zalmanovici @ 2005-11-24 10:00 UTC (permalink / raw)
To: Con Kolivas; +Cc: linux-kernel, Muli Ben-Yehuda
Con Kolivas
<kernel@kolivas.o To: Marcel Zalmanovici/Haifa/IBM@IBMIL
rg> cc: linux-kernel@vger.kernel.org, Muli Ben-Yehuda <mulix@mulix.org>
Subject: Re: Inconsistent timing results of multithreaded program on an SMP
24/11/2005 11:40 machine.
On Thu, 24 Nov 2005 19:33, Marcel Zalmanovici wrote:
> Hi Con,
>
> I've tried the pthread_join theory.
> pthread_join completes very fast, no evidence on it being the perpetrator
> here.
>
> I've ran your ps -... idea in a different term window every ms while the
> test was running. It created a large file so I won't mail it to you, but
> both me and Muli observed that once the thread ends it quickly disappears
> from the ps list.
>
> I've also ran Muli's idea and added gettimeofday calls.
> Here's the altered code and output:
>
> (See attached file: idle.log)(See attached file: sched_test.c)
>
> As you can see, if a thread already finished pthread_join returns in a
> split second.
>
> Any other ideas are welcome :-)
Profile the actual application?
Well, technically, this IS the actual application. Ultimately, I would like
to make some changes to the kernel in order to reduce the cache misses and
maybe improve run time as a consequence.
Having results spread out as much as I get, it would be very difficult, if
not impossible for me to estimate if my algorithm conveyed any real
results.
I have profiled (time ticks and L2 cache misses) the example I've posted
and a few other variations on the example, but haven't seen nothing out of
the ordinary.
Around 98-99% of time and misses are at the inner loop of the example.
Is there anything else I can check with profile that may help me understand
the phenomenon ?
> P.S. - I agree that Lotus isn't ideal for this kind of conversations, but
> that's what IBM is using.
It was tongue in cheek ;) Well that should still not stop you from replying
below the original thread.
Cheers,
Con
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Inconsistent timing results of multithreaded program on an SMP machine.
2005-11-24 10:00 ` Marcel Zalmanovici
@ 2005-11-24 12:43 ` Paul Jackson
2005-12-04 15:26 ` Marcel Zalmanovici
0 siblings, 1 reply; 17+ messages in thread
From: Paul Jackson @ 2005-11-24 12:43 UTC (permalink / raw)
To: Marcel Zalmanovici; +Cc: kernel, linux-kernel, mulix
Marcel,
I condensed the results you attached to another reply earlier
today on this query into a little table, showing for each of
8 threads, in order, which of the two CPUs 0 or 1 they finished
on, and the real (elapsed) time:
0 0 0 1 0 1 0 0 14.49
1 1 0 0 1 1 1 1 15.18
0 1 0 1 0 0 1 1 14.64
1 1 0 0 0 1 0 0 14.65
0 0 0 1 1 1 1 1 14.61
1 1 0 0 1 1 1 1 18.89
0 1 0 0 1 1 0 1 14.62
1 1 0 1 0 0 1 1 14.51
0 0 1 1 0 0 0 1 14.54
0 1 0 1 1 1 1 1 14.73
0 1 0 1 0 1 1 1 14.78
0 1 1 1 0 0 0 1 18.90
0 0 0 1 1 1 0 1 14.57
1 1 0 1 0 1 0 0 17.07
0 1 1 1 0 1 1 1 14.56
0 1 0 0 1 1 0 0 14.49
0 1 1 1 0 1 0 0 17.80
0 1 0 1 1 1 1 1 14.62
0 1 0 1 0 1 0 0 19.55
0 1 0 1 0 0 1 1 19.57
I notice that the deviation of the faster runs is very low,
with the best 13 of 20 runs all finishing in times between
14.49 and 14.78 seconds, but the slower runs have a higher
deviation, with the worst 7 of 20 runs finishing in times
between 15.18 and 19.57 seconds, a wider range of times.
That is, the best 13 runs differ in time by only 0.29 secs,
but the worst 7 runs differ in time by 4.39 seconds, a much
wider range.
I agree with you that I don't see a pattern in which CPU
the threads finished on or any relation between that and
the real times.
However this skewed distribution of times does suggest that
something is intruding - something that only happens perhaps
a third of the time is running or making things worse.
What I might suggest next is that you look for nice ways
that fit on a page or less and condense out all nonessential
detail to present these timing results. People have a
tendency to dump a big attachment of raw output data onto
email lists, which almost no one ever reads closely,
especially if it is in some homebrew format perculiar to
that particular inquiry. Time spent cleaning up the show
of data is often well spent, because then others might
actually look at the data and recognize a pattern that
they have something useful to say about.
Then after you have it collapsed to a nice table format,
go even the next step and describe in words any patterns
that are apparent, which will catch another chunk of
potential readers, who will skim over a data table without
really thinking. Then fine tune that wording, until it
would be understood by someone, the first time they heard
it, standing at the coffee urn, discussing sports. Often
times, by the time you get that far, you will have an
'aha' moment, leading to further experiments and insights,
before you even get a chance to ask someone else.
Question - how do you know its the multithreading that
is causing the variance? Try this with single threads,
just one per cpu, with or without hyperthread perhaps.
Never assume that you know the cause of something until
you have bracketed the tests with demonstrations that
all else being equal (or as close as can be) just adding
or removing the one suspect element causes the observed
affect to come and go.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <pj@sgi.com> 1.925.600.0401
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Inconsistent timing results of multithreaded program on an SMP machine.
2005-11-24 12:43 ` Paul Jackson
@ 2005-12-04 15:26 ` Marcel Zalmanovici
2005-12-04 19:54 ` Paul Jackson
0 siblings, 1 reply; 17+ messages in thread
From: Marcel Zalmanovici @ 2005-12-04 15:26 UTC (permalink / raw)
To: Paul Jackson; +Cc: kernel, linux-kernel, mulix
Hi Paul,
I've tried (again) running on a single CPU just to reassure myself that
everything is stable there. The differences were negligible.
Also, I've ran the test on a Regatta machine with 2 hyperthreaded CPUs
(just like my Intel Xeon). The results surprised me somewhat since they
were VERY stable; less than a second between the shortest and longest runs.
In addition to that, other users were connected to the machine at that
time.
I've converted the *writes* to the array tp *reads* from it and ran this
test on my machine.
The results were even more stable than I could possible hope for; maybe 0.2
sec between extremities.
I think that the conclusion from this test is that the write-back algorithm
might be responsible for the oscillating results.
Unfortunetaly, after looking at the hardware configuration I'm more
puzzled; it has write back and write through with the former being set as
default.
Write back is supposed to yield better results since data is written back
only when needed. Not seeing a correlation between run times and CPU on
which the threads run puzzles me even more now.
Any ideas come to mind ? cause I'm running out of.
Marcel
Paul Jackson
<pj@sgi.com> To: Marcel Zalmanovici/Haifa/IBM@IBMIL
cc: kernel@kolivas.org, linux-kernel@vger.kernel.org, mulix@mulix.org
24/11/2005 14:43 Subject: Re: Inconsistent timing results of multithreaded program on an SMP machine.
Marcel,
I condensed the results you attached to another reply earlier
today on this query into a little table, showing for each of
8 threads, in order, which of the two CPUs 0 or 1 they finished
on, and the real (elapsed) time:
0 0 0 1 0 1 0 0 14.49
1 1 0 0 1 1 1 1 15.18
0 1 0 1 0 0 1 1 14.64
1 1 0 0 0 1 0 0 14.65
0 0 0 1 1 1 1 1 14.61
1 1 0 0 1 1 1 1 18.89
0 1 0 0 1 1 0 1 14.62
1 1 0 1 0 0 1 1 14.51
0 0 1 1 0 0 0 1 14.54
0 1 0 1 1 1 1 1 14.73
0 1 0 1 0 1 1 1 14.78
0 1 1 1 0 0 0 1 18.90
0 0 0 1 1 1 0 1 14.57
1 1 0 1 0 1 0 0 17.07
0 1 1 1 0 1 1 1 14.56
0 1 0 0 1 1 0 0 14.49
0 1 1 1 0 1 0 0 17.80
0 1 0 1 1 1 1 1 14.62
0 1 0 1 0 1 0 0 19.55
0 1 0 1 0 0 1 1 19.57
I notice that the deviation of the faster runs is very low,
with the best 13 of 20 runs all finishing in times between
14.49 and 14.78 seconds, but the slower runs have a higher
deviation, with the worst 7 of 20 runs finishing in times
between 15.18 and 19.57 seconds, a wider range of times.
That is, the best 13 runs differ in time by only 0.29 secs,
but the worst 7 runs differ in time by 4.39 seconds, a much
wider range.
I agree with you that I don't see a pattern in which CPU
the threads finished on or any relation between that and
the real times.
However this skewed distribution of times does suggest that
something is intruding - something that only happens perhaps
a third of the time is running or making things worse.
What I might suggest next is that you look for nice ways
that fit on a page or less and condense out all nonessential
detail to present these timing results. People have a
tendency to dump a big attachment of raw output data onto
email lists, which almost no one ever reads closely,
especially if it is in some homebrew format perculiar to
that particular inquiry. Time spent cleaning up the show
of data is often well spent, because then others might
actually look at the data and recognize a pattern that
they have something useful to say about.
Then after you have it collapsed to a nice table format,
go even the next step and describe in words any patterns
that are apparent, which will catch another chunk of
potential readers, who will skim over a data table without
really thinking. Then fine tune that wording, until it
would be understood by someone, the first time they heard
it, standing at the coffee urn, discussing sports. Often
times, by the time you get that far, you will have an
'aha' moment, leading to further experiments and insights,
before you even get a chance to ask someone else.
Question - how do you know its the multithreading that
is causing the variance? Try this with single threads,
just one per cpu, with or without hyperthread perhaps.
Never assume that you know the cause of something until
you have bracketed the tests with demonstrations that
all else being equal (or as close as can be) just adding
or removing the one suspect element causes the observed
affect to come and go.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <pj@sgi.com> 1.925.600.0401
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Inconsistent timing results of multithreaded program on an SMP machine.
2005-12-04 15:26 ` Marcel Zalmanovici
@ 2005-12-04 19:54 ` Paul Jackson
0 siblings, 0 replies; 17+ messages in thread
From: Paul Jackson @ 2005-12-04 19:54 UTC (permalink / raw)
To: Marcel Zalmanovici; +Cc: kernel, linux-kernel, mulix
Marcel wrote:
> I think that the conclusion from this test is that the write-back algorithm
> might be responsible for the oscillating results.
Well ... if you suspect your storage/hardware, then try a different
file system, such as network files or /dev/ramdisk.
It's a few seconds worth of difference you're dealing with, so try
playing around with printf's in the user code and printk's in the
kernel at key points in the sequence of events, so you can see what
happens when.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <pj@sgi.com> 1.925.600.0401
^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2005-12-04 19:55 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <OF507D27BA.6B51F19A-ONC22570C3.002E62D2-C22570C3.002F0C99@il.ibm.com>
2005-11-24 9:40 ` Inconsistent timing results of multithreaded program on an SMP machine Con Kolivas
2005-11-24 10:00 ` Marcel Zalmanovici
2005-11-24 12:43 ` Paul Jackson
2005-12-04 15:26 ` Marcel Zalmanovici
2005-12-04 19:54 ` Paul Jackson
2005-11-20 9:27 Marcel Zalmanovici
2005-11-20 9:54 ` Con Kolivas
2005-11-20 10:18 ` Marcel Zalmanovici
2005-11-20 10:28 ` Con Kolivas
2005-11-20 10:35 ` Muli Ben-Yehuda
2005-11-20 10:39 ` Con Kolivas
2005-11-20 10:43 ` Muli Ben-Yehuda
2005-11-20 10:50 ` Marcel Zalmanovici
2005-11-20 10:50 ` Con Kolivas
2005-11-20 10:43 ` Marcel Zalmanovici
2005-11-20 14:02 ` Paul Jackson
2005-11-24 9:50 ` Marcel Zalmanovici
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.