46% performance drop with change in glibc

46% performance drop with change in glibc

[Andrew Theurer]

I have been trying to track down a xen performance regression from Fedora 
core3 to core4 i386 and I think I might have narrowed it down.  First some 
background info:  I ran the SDET benchmark [1] and discovered that we have 
about a 45% regression after upgrading to Fedora core 4.  I noticed 
xenoprofile showed ~20% in error_code on FC4, but only ~3.5% in error_code 
(should we even have 3%?!?) on FC3.  At first I thought the change to gcc-4 
might have something to do with it, so I tried gcc4 on FC3 -no regression.  
I noticed glibc was also different, so I upgraded FC3 from 2.3.3 to 2.3.5 
-bingo.  I am not sure what about glibc is causing this, but here are the 
two profiles:

fc3 with glic-2.3.3:

samples  %        app name                 symbol name
56097    22.8770  cc1                      (no symbols)
22780     9.2900  troff                    (no symbols)
8646      3.5259  xen-unstable-syms        error_code
7822      3.1899  grotty                   (no symbols)
5283      2.1545  libc-2.3.3.so            _int_malloc
4914      2.0040  vmlinux-2.6.12-xen0-up   buffered_rmqueue
4453      1.8160  bash                     (no symbols)
4372      1.7830  xen-unstable-syms        get_page_from_l1e
3492      1.4241  xen-unstable-syms        put_page_from_l1e
3338      1.3613  vmlinux-2.6.12-xen0-up   system_call
3040      1.2397  xen-unstable-syms        hypercall
2801      1.1423  xen-unstable-syms        alloc_page_type
2799      1.1415  vmlinux-2.6.12-xen0-up   zap_pte_range
2798      1.1411  libstdc++.so.6.0.3       (no symbols)
2761      1.1260  libc-2.3.3.so            vfprintf
2637      1.0754  vmlinux-2.6.12-xen0-up   do_no_page
2555      1.0420  libc-2.3.3.so            __i686.get_pc_thunk.bx
2548      1.0391  vmlinux-2.6.12-xen0-up   page_fault

fc3 with glibc-2.3.5:

samples  %        app name                 symbol name
133404   22.7469  xen-unstable-syms        error_code
31411     5.3559  libc-2.3.5.so            malloc
30316     5.1692  troff                    (no symbols)
20632     3.5180  libc-2.3.5.so            vfprintf
16626     2.8349  xen-unstable-syms        gpf_emulate_4gb
10639     1.8141  xen-unstable-syms        do_general_protection
10105     1.7230  grotty                   (no symbols)
9837      1.6773  libc-2.3.5.so            getwchar
8057      1.3738  xen-unstable-syms        decode_register
7347      1.2527  libc-2.3.5.so            iswgraph
7114      1.2130  vmlinux-2.6.12-xen0-up   buffered_rmqueue
6926      1.1810  cc1                      is_attribute_p
6679      1.1388  libc-2.3.5.so            _int_malloc
6350      1.0827  xen-unstable-syms        fixup_seg
5673      0.9673  xen-unstable-syms        get_baselimit
5039      0.8592  bash                     (no symbols)
4957      0.8452  xen-unstable-syms        get_page_from_l1e
4908      0.8369  xen-unstable-syms        put_page_from_l1e
4576      0.7803  ld-2.3.5.so              do_lookup_x
4322      0.7370  xen-unstable-syms        hypercall
3915      0.6676  vmlinux-2.6.12-xen0-up   system_call

I assume something related to gpf_emulate_4gb...
Anyone have an idea why this would cause such a regression?

-Andrew


[1] SDET described here:
http://www.spec.org/osg/sdm91/

Re: 46% performance drop with change in glibc

[Kip Macy]

[-- Attachment #1.1: Type: text/plain, Size: 3249 bytes --]

The segment offset handling emulation is expensive. You need to either
rename your TLS libraries away or use a glibc with the xen TLS patch.

-Kip

On 11/2/05, Andrew Theurer <habanero@us.ibm.com> wrote:
>
> I have been trying to track down a xen performance regression from Fedora
> core3 to core4 i386 and I think I might have narrowed it down. First some
> background info: I ran the SDET benchmark [1] and discovered that we have
> about a 45% regression after upgrading to Fedora core 4. I noticed
> xenoprofile showed ~20% in error_code on FC4, but only ~3.5% in error_code
> (should we even have 3%?!?) on FC3. At first I thought the change to gcc-4
> might have something to do with it, so I tried gcc4 on FC3 -no regression.
> I noticed glibc was also different, so I upgraded FC3 from 2.3.3 to 2.3.5
> -bingo. I am not sure what about glibc is causing this, but here are the
> two profiles:
>
> fc3 with glic-2.3.3:
>
> samples % app name symbol name
> 56097 22.8770 cc1 (no symbols)
> 22780 9.2900 troff (no symbols)
> 8646 3.5259 xen-unstable-syms error_code
> 7822 3.1899 grotty (no symbols)
> 5283 2.1545 libc-2.3.3.so <http://libc-2.3.3.so> _int_malloc
> 4914 2.0040 vmlinux-2.6.12-xen0-up buffered_rmqueue
> 4453 1.8160 bash (no symbols)
> 4372 1.7830 xen-unstable-syms get_page_from_l1e
> 3492 1.4241 xen-unstable-syms put_page_from_l1e
> 3338 1.3613 vmlinux-2.6.12-xen0-up system_call
> 3040 1.2397 xen-unstable-syms hypercall
> 2801 1.1423 xen-unstable-syms alloc_page_type
> 2799 1.1415 vmlinux-2.6.12-xen0-up zap_pte_range
> 2798 1.1411 libstdc++.so.6.0.3 (no symbols)
> 2761 1.1260 libc-2.3.3.so <http://libc-2.3.3.so> vfprintf
> 2637 1.0754 vmlinux-2.6.12-xen0-up do_no_page
> 2555 1.0420 libc-2.3.3.so <http://libc-2.3.3.so> __i686.get_pc_thunk.bx
> 2548 1.0391 vmlinux-2.6.12-xen0-up page_fault
>
> fc3 with glibc-2.3.5:
>
> samples % app name symbol name
> 133404 22.7469 xen-unstable-syms error_code
> 31411 5.3559 libc-2.3.5.so <http://libc-2.3.5.so> malloc
> 30316 5.1692 troff (no symbols)
> 20632 3.5180 libc-2.3.5.so <http://libc-2.3.5.so> vfprintf
> 16626 2.8349 xen-unstable-syms gpf_emulate_4gb
> 10639 1.8141 xen-unstable-syms do_general_protection
> 10105 1.7230 grotty (no symbols)
> 9837 1.6773 libc-2.3.5.so <http://libc-2.3.5.so> getwchar
> 8057 1.3738 xen-unstable-syms decode_register
> 7347 1.2527 libc-2.3.5.so <http://libc-2.3.5.so> iswgraph
> 7114 1.2130 vmlinux-2.6.12-xen0-up buffered_rmqueue
> 6926 1.1810 cc1 is_attribute_p
> 6679 1.1388 libc-2.3.5.so <http://libc-2.3.5.so> _int_malloc
> 6350 1.0827 xen-unstable-syms fixup_seg
> 5673 0.9673 xen-unstable-syms get_baselimit
> 5039 0.8592 bash (no symbols)
> 4957 0.8452 xen-unstable-syms get_page_from_l1e
> 4908 0.8369 xen-unstable-syms put_page_from_l1e
> 4576 0.7803 ld-2.3.5.so <http://ld-2.3.5.so> do_lookup_x
> 4322 0.7370 xen-unstable-syms hypercall
> 3915 0.6676 vmlinux-2.6.12-xen0-up system_call
>
> I assume something related to gpf_emulate_4gb...
> Anyone have an idea why this would cause such a regression?
>
> -Andrew
>
>
> [1] SDET described here:
> http://www.spec.org/osg/sdm91/
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

[-- Attachment #1.2: Type: text/html, Size: 8348 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: 46% performance drop with change in glibc

[Andrew Theurer]

I had thought I moved tls to tls.disabled, but I'll check again.

-Andrew

On Wednesday 02 November 2005 13:17, Kip Macy wrote:
> The segment offset handling emulation is expensive. You need to either
> rename your TLS libraries away or use a glibc with the xen TLS patch.
>
> -Kip
>
> On 11/2/05, Andrew Theurer <habanero@us.ibm.com> wrote:
> > I have been trying to track down a xen performance regression from
> > Fedora core3 to core4 i386 and I think I might have narrowed it down.
> > First some background info: I ran the SDET benchmark [1] and discovered
> > that we have about a 45% regression after upgrading to Fedora core 4. I
> > noticed xenoprofile showed ~20% in error_code on FC4, but only ~3.5% in
> > error_code (should we even have 3%?!?) on FC3. At first I thought the
> > change to gcc-4 might have something to do with it, so I tried gcc4 on
> > FC3 -no regression. I noticed glibc was also different, so I upgraded
> > FC3 from 2.3.3 to 2.3.5 -bingo. I am not sure what about glibc is
> > causing this, but here are the two profiles:
> >
> > fc3 with glic-2.3.3:
> >
> > samples % app name symbol name
> > 56097 22.8770 cc1 (no symbols)
> > 22780 9.2900 troff (no symbols)
> > 8646 3.5259 xen-unstable-syms error_code
> > 7822 3.1899 grotty (no symbols)
> > 5283 2.1545 libc-2.3.3.so <http://libc-2.3.3.so> _int_malloc
> > 4914 2.0040 vmlinux-2.6.12-xen0-up buffered_rmqueue
> > 4453 1.8160 bash (no symbols)
> > 4372 1.7830 xen-unstable-syms get_page_from_l1e
> > 3492 1.4241 xen-unstable-syms put_page_from_l1e
> > 3338 1.3613 vmlinux-2.6.12-xen0-up system_call
> > 3040 1.2397 xen-unstable-syms hypercall
> > 2801 1.1423 xen-unstable-syms alloc_page_type
> > 2799 1.1415 vmlinux-2.6.12-xen0-up zap_pte_range
> > 2798 1.1411 libstdc++.so.6.0.3 (no symbols)
> > 2761 1.1260 libc-2.3.3.so <http://libc-2.3.3.so> vfprintf
> > 2637 1.0754 vmlinux-2.6.12-xen0-up do_no_page
> > 2555 1.0420 libc-2.3.3.so <http://libc-2.3.3.so> __i686.get_pc_thunk.bx
> > 2548 1.0391 vmlinux-2.6.12-xen0-up page_fault
> >
> > fc3 with glibc-2.3.5:
> >
> > samples % app name symbol name
> > 133404 22.7469 xen-unstable-syms error_code
> > 31411 5.3559 libc-2.3.5.so <http://libc-2.3.5.so> malloc
> > 30316 5.1692 troff (no symbols)
> > 20632 3.5180 libc-2.3.5.so <http://libc-2.3.5.so> vfprintf
> > 16626 2.8349 xen-unstable-syms gpf_emulate_4gb
> > 10639 1.8141 xen-unstable-syms do_general_protection
> > 10105 1.7230 grotty (no symbols)
> > 9837 1.6773 libc-2.3.5.so <http://libc-2.3.5.so> getwchar
> > 8057 1.3738 xen-unstable-syms decode_register
> > 7347 1.2527 libc-2.3.5.so <http://libc-2.3.5.so> iswgraph
> > 7114 1.2130 vmlinux-2.6.12-xen0-up buffered_rmqueue
> > 6926 1.1810 cc1 is_attribute_p
> > 6679 1.1388 libc-2.3.5.so <http://libc-2.3.5.so> _int_malloc
> > 6350 1.0827 xen-unstable-syms fixup_seg
> > 5673 0.9673 xen-unstable-syms get_baselimit
> > 5039 0.8592 bash (no symbols)
> > 4957 0.8452 xen-unstable-syms get_page_from_l1e
> > 4908 0.8369 xen-unstable-syms put_page_from_l1e
> > 4576 0.7803 ld-2.3.5.so <http://ld-2.3.5.so> do_lookup_x
> > 4322 0.7370 xen-unstable-syms hypercall
> > 3915 0.6676 vmlinux-2.6.12-xen0-up system_call
> >
> > I assume something related to gpf_emulate_4gb...
> > Anyone have an idea why this would cause such a regression?
> >
> > -Andrew
> >
> >
> > [1] SDET described here:
> > http://www.spec.org/osg/sdm91/
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel

Re: 46% performance drop with change in glibc

[Andrew Theurer]

Well, I thought I was losing my mind, but hopefully not.  When I upgraded 
glibc on FC3, I did forget to move tls to tls.disabled.  So, I moved it to 
tls.disabled, and the problem goes away.  HOWEVER, on FC4, I always had tls 
moved to tls.disabled, and I am still getting this problem.  So, on FC4, why 
would these libs not be in /lib/tls in the first place?

/lib/tls[.disabled] on fc3:

i486  i686           libc.so.6      libm.so.6            libpthread.so.0  
librt.so.1           libthread_db.so.1
i586  libc-2.3.3.so  libm-2.3.3.so  libpthread-2.3.3.so  librt-2.3.3.so   
libthread_db-1.0.so

/lib/tls[.disabled] on fc4:

i486  i586  i686



-Andrew

On Wednesday 02 November 2005 13:23, Andrew Theurer wrote:
> I had thought I moved tls to tls.disabled, but I'll check again.
>
> -Andrew
>
> On Wednesday 02 November 2005 13:17, Kip Macy wrote:
> > The segment offset handling emulation is expensive. You need to either
> > rename your TLS libraries away or use a glibc with the xen TLS patch.
> >
> > -Kip
> >
> > On 11/2/05, Andrew Theurer <habanero@us.ibm.com> wrote:
> > > I have been trying to track down a xen performance regression from
> > > Fedora core3 to core4 i386 and I think I might have narrowed it down.
> > > First some background info: I ran the SDET benchmark [1] and
> > > discovered that we have about a 45% regression after upgrading to
> > > Fedora core 4. I noticed xenoprofile showed ~20% in error_code on FC4,
> > > but only ~3.5% in error_code (should we even have 3%?!?) on FC3. At
> > > first I thought the change to gcc-4 might have something to do with
> > > it, so I tried gcc4 on FC3 -no regression. I noticed glibc was also
> > > different, so I upgraded FC3 from 2.3.3 to 2.3.5 -bingo. I am not sure
> > > what about glibc is causing this, but here are the two profiles:
> > >
> > > fc3 with glic-2.3.3:
> > >
> > > samples % app name symbol name
> > > 56097 22.8770 cc1 (no symbols)
> > > 22780 9.2900 troff (no symbols)
> > > 8646 3.5259 xen-unstable-syms error_code
> > > 7822 3.1899 grotty (no symbols)
> > > 5283 2.1545 libc-2.3.3.so <http://libc-2.3.3.so> _int_malloc
> > > 4914 2.0040 vmlinux-2.6.12-xen0-up buffered_rmqueue
> > > 4453 1.8160 bash (no symbols)
> > > 4372 1.7830 xen-unstable-syms get_page_from_l1e
> > > 3492 1.4241 xen-unstable-syms put_page_from_l1e
> > > 3338 1.3613 vmlinux-2.6.12-xen0-up system_call
> > > 3040 1.2397 xen-unstable-syms hypercall
> > > 2801 1.1423 xen-unstable-syms alloc_page_type
> > > 2799 1.1415 vmlinux-2.6.12-xen0-up zap_pte_range
> > > 2798 1.1411 libstdc++.so.6.0.3 (no symbols)
> > > 2761 1.1260 libc-2.3.3.so <http://libc-2.3.3.so> vfprintf
> > > 2637 1.0754 vmlinux-2.6.12-xen0-up do_no_page
> > > 2555 1.0420 libc-2.3.3.so <http://libc-2.3.3.so>
> > > __i686.get_pc_thunk.bx 2548 1.0391 vmlinux-2.6.12-xen0-up page_fault
> > >
> > > fc3 with glibc-2.3.5:
> > >
> > > samples % app name symbol name
> > > 133404 22.7469 xen-unstable-syms error_code
> > > 31411 5.3559 libc-2.3.5.so <http://libc-2.3.5.so> malloc
> > > 30316 5.1692 troff (no symbols)
> > > 20632 3.5180 libc-2.3.5.so <http://libc-2.3.5.so> vfprintf
> > > 16626 2.8349 xen-unstable-syms gpf_emulate_4gb
> > > 10639 1.8141 xen-unstable-syms do_general_protection
> > > 10105 1.7230 grotty (no symbols)
> > > 9837 1.6773 libc-2.3.5.so <http://libc-2.3.5.so> getwchar
> > > 8057 1.3738 xen-unstable-syms decode_register
> > > 7347 1.2527 libc-2.3.5.so <http://libc-2.3.5.so> iswgraph
> > > 7114 1.2130 vmlinux-2.6.12-xen0-up buffered_rmqueue
> > > 6926 1.1810 cc1 is_attribute_p
> > > 6679 1.1388 libc-2.3.5.so <http://libc-2.3.5.so> _int_malloc
> > > 6350 1.0827 xen-unstable-syms fixup_seg
> > > 5673 0.9673 xen-unstable-syms get_baselimit
> > > 5039 0.8592 bash (no symbols)
> > > 4957 0.8452 xen-unstable-syms get_page_from_l1e
> > > 4908 0.8369 xen-unstable-syms put_page_from_l1e
> > > 4576 0.7803 ld-2.3.5.so <http://ld-2.3.5.so> do_lookup_x
> > > 4322 0.7370 xen-unstable-syms hypercall
> > > 3915 0.6676 vmlinux-2.6.12-xen0-up system_call
> > >
> > > I assume something related to gpf_emulate_4gb...
> > > Anyone have an idea why this would cause such a regression?
> > >
> > > -Andrew
> > >
> > >
> > > [1] SDET described here:
> > > http://www.spec.org/osg/sdm91/
> > >
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xensource.com
> > > http://lists.xensource.com/xen-devel
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: 46% performance drop with change in glibc

[Anthony Liguori]

Hi Andrew,

My immediate guess would be TLS emulation.

Do you not disable TLS on FC4?

Regards,

Anthony Liguori

Andrew Theurer wrote:

>I have been trying to track down a xen performance regression from Fedora 
>core3 to core4 i386 and I think I might have narrowed it down.  First some 
>background info:  I ran the SDET benchmark [1] and discovered that we have 
>about a 45% regression after upgrading to Fedora core 4.  I noticed 
>xenoprofile showed ~20% in error_code on FC4, but only ~3.5% in error_code 
>(should we even have 3%?!?) on FC3.  At first I thought the change to gcc-4 
>might have something to do with it, so I tried gcc4 on FC3 -no regression.  
>I noticed glibc was also different, so I upgraded FC3 from 2.3.3 to 2.3.5 
>-bingo.  I am not sure what about glibc is causing this, but here are the 
>two profiles:
>
>fc3 with glic-2.3.3:
>
>samples  %        app name                 symbol name
>56097    22.8770  cc1                      (no symbols)
>22780     9.2900  troff                    (no symbols)
>8646      3.5259  xen-unstable-syms        error_code
>7822      3.1899  grotty                   (no symbols)
>5283      2.1545  libc-2.3.3.so            _int_malloc
>4914      2.0040  vmlinux-2.6.12-xen0-up   buffered_rmqueue
>4453      1.8160  bash                     (no symbols)
>4372      1.7830  xen-unstable-syms        get_page_from_l1e
>3492      1.4241  xen-unstable-syms        put_page_from_l1e
>3338      1.3613  vmlinux-2.6.12-xen0-up   system_call
>3040      1.2397  xen-unstable-syms        hypercall
>2801      1.1423  xen-unstable-syms        alloc_page_type
>2799      1.1415  vmlinux-2.6.12-xen0-up   zap_pte_range
>2798      1.1411  libstdc++.so.6.0.3       (no symbols)
>2761      1.1260  libc-2.3.3.so            vfprintf
>2637      1.0754  vmlinux-2.6.12-xen0-up   do_no_page
>2555      1.0420  libc-2.3.3.so            __i686.get_pc_thunk.bx
>2548      1.0391  vmlinux-2.6.12-xen0-up   page_fault
>
>fc3 with glibc-2.3.5:
>
>samples  %        app name                 symbol name
>133404   22.7469  xen-unstable-syms        error_code
>31411     5.3559  libc-2.3.5.so            malloc
>30316     5.1692  troff                    (no symbols)
>20632     3.5180  libc-2.3.5.so            vfprintf
>16626     2.8349  xen-unstable-syms        gpf_emulate_4gb
>10639     1.8141  xen-unstable-syms        do_general_protection
>10105     1.7230  grotty                   (no symbols)
>9837      1.6773  libc-2.3.5.so            getwchar
>8057      1.3738  xen-unstable-syms        decode_register
>7347      1.2527  libc-2.3.5.so            iswgraph
>7114      1.2130  vmlinux-2.6.12-xen0-up   buffered_rmqueue
>6926      1.1810  cc1                      is_attribute_p
>6679      1.1388  libc-2.3.5.so            _int_malloc
>6350      1.0827  xen-unstable-syms        fixup_seg
>5673      0.9673  xen-unstable-syms        get_baselimit
>5039      0.8592  bash                     (no symbols)
>4957      0.8452  xen-unstable-syms        get_page_from_l1e
>4908      0.8369  xen-unstable-syms        put_page_from_l1e
>4576      0.7803  ld-2.3.5.so              do_lookup_x
>4322      0.7370  xen-unstable-syms        hypercall
>3915      0.6676  vmlinux-2.6.12-xen0-up   system_call
>
>I assume something related to gpf_emulate_4gb...
>Anyone have an idea why this would cause such a regression?
>
>-Andrew
>
>
>[1] SDET described here:
>http://www.spec.org/osg/sdm91/
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>
>  
>

Re: 46% performance drop with change in glibc

[Ulrich Drepper]

[-- Attachment #1.1: Type: text/plain, Size: 625 bytes --]

Andrew Theurer wrote:
> So, on FC4, why 
> would these libs not be in /lib/tls in the first place?

Too l33t to read release notes?

There is no tls subdir since NPTL is the default (and LT is completely
gone in FC5).  The code in /lib is the TLS code using the correct ABI.

The crippled ABI is supported by the code in /lib/i686/nosegneg.  This
code is *automatically* used if the kernel is compiled to match the
glibc version.  This happens using the

linux-2.6-xen-vdso-note.patch

patch in Red Hat's kernel RPMs.

-- 
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: 46% performance drop with change in glibc

[Ian Pratt]

 
> There is no tls subdir since NPTL is the default (and LT is 
> completely gone in FC5).  The code in /lib is the TLS code 
> using the correct ABI.
> 
> The crippled ABI is supported by the code in 
> /lib/i686/nosegneg.  This code is *automatically* used if the 
> kernel is compiled to match the glibc version.  This happens using the
> 
> linux-2.6-xen-vdso-note.patch

I believe we have this patch in the xen repo, but I haven't ever
verified whether the right thing happens under FC4.

Andrew: please could you check you have the xen glibc rpm installed and
verify that the glibc selection is working using one of our kernels. 

Thanks,
Ian

Re: 46% performance drop with change in glibc

[Andrew Theurer]

On Friday 04 November 2005 10:08, you wrote:
> > There is no tls subdir since NPTL is the default (and LT is
> > completely gone in FC5).  The code in /lib is the TLS code
> > using the correct ABI.
> >
> > The crippled ABI is supported by the code in
> > /lib/i686/nosegneg.  This code is *automatically* used if the
> > kernel is compiled to match the glibc version.  This happens using the
> >
> > linux-2.6-xen-vdso-note.patch
>
> I believe we have this patch in the xen repo, but I haven't ever
> verified whether the right thing happens under FC4.

I am not seeing this patch in ./patches/linux-2.6.12 directory.  Is it 
somewhere else?

> Andrew: please could you check you have the xen glibc rpm installed and
> verify that the glibc selection is working using one of our kernels.

I am pretty sure I have the xen glibc, but a I can't verify it's working 
until I get a clean build of a patched xen kernel.  I suspect we need a 
little more than just the linux-2.6-xen-vdso-note patch to build cleanly:

kernel/built-in.o(.text+0x2671f): In function `kallsyms_lookup':
: undefined reference to `in_gate_area_no_task'
mm/built-in.o(.text+0x12f5f): In function `get_user_pages':
: undefined reference to `in_gate_area'
mm/built-in.o(.text+0x12f7b): In function `get_user_pages':
: undefined reference to `get_gate_vma'
fs/built-in.o(.text+0x34608): In function `show_map':
task_mmu.c: undefined reference to `get_gate_vma'
fs/built-in.o(.text+0x34785): In function `m_start':
task_mmu.c: undefined reference to `get_gate_vma'
fs/built-in.o(.text+0x34887): In function `m_stop':
task_mmu.c: undefined reference to `get_gate_vma'
fs/built-in.o(.text+0x348e6): In function `m_next':
task_mmu.c: undefined reference to `get_gate_vma'

RE: 46% performance drop with change in glibc

[Ian Pratt]

> > > The crippled ABI is supported by the code in /lib/i686/nosegneg.  
> > > This code is *automatically* used if the kernel is 
> compiled to match 
> > > the glibc version.  This happens using the
> > >
> > > linux-2.6-xen-vdso-note.patch
> >
> > I believe we have this patch in the xen repo, but I haven't ever 
> > verified whether the right thing happens under FC4.
> 
> I am not seeing this patch in ./patches/linux-2.6.12 
> directory.  Is it somewhere else?

It should be in the main tree. The appended patch was included some time
ago, but I've never tested it to see whether it works since I don't
generally run bleeding edge Fedora.

Is there some piece of the patch missing?

Thanks,
Ian


--- linux-2.6/arch/xen/i386/kernel/vsyscall-note.S
+++ linux-2.6/arch/xen/i386/kernel/vsyscall-note.S
@@ -0,0 +1,32 @@
+/*
+ * This supplies .note.* sections to go into the PT_NOTE inside the
vDSO text.
+ * Here we can supply some information useful to userland.
+ * First we get the vanilla i386 note that supplies the kernel version
info.
+ */
+
+#include "../../../i386/kernel/vsyscall-note.S"
+
+/*
+ * Now we add a special note telling glibc's dynamic linker a fake 
+hardware
+ * flavor that it will use to choose the search path for libraries in 
+the
+ * same way it uses real hardware capabilities like "mmx".
+ * We supply "nosegneg" as the fake capability, to indicate that we
+ * do not like negative offsets in instructions using segment 
+overrides,
+ * since we implement those inefficiently.  This makes it possible to
+ * install libraries optimized to avoid those access patterns in 
+someplace
+ * like /lib/i686/tls/nosegneg.  Note that an /etc/ld.so.conf.d/file
+ * corresponding to the bits here is needed to make ldconfig work
right.
+ * It should contain:
+ *	hwcap 0 nosegneg
+ * to match the mapping of bit to name that we give here.
+ */
+#define NOTE_KERNELCAP_BEGIN(ncaps, mask) \
+	ASM_ELF_NOTE_BEGIN(".note.kernelcap", "a", "GNU", 2) \
+	.long ncaps, mask
+#define NOTE_KERNELCAP(bit, name) \
+	.byte bit; .asciz name
+#define NOTE_KERNELCAP_END ASM_ELF_NOTE_END
+
+NOTE_KERNELCAP_BEGIN(1, 1)
+NOTE_KERNELCAP(1, "nosegneg")  /* Change 1 back to 0 when glibc is 
+fixed! */ NOTE_KERNELCAP_END

Re: 46% performance drop with change in glibc

[Andrew Theurer]

Ian Pratt wrote:

>>>>The crippled ABI is supported by the code in /lib/i686/nosegneg.  
>>>>This code is *automatically* used if the kernel is 
>>>>
>>compiled to match 
>>
>>>>the glibc version.  This happens using the
>>>>
>>>>linux-2.6-xen-vdso-note.patch
>>>>
>>>I believe we have this patch in the xen repo, but I haven't ever 
>>>verified whether the right thing happens under FC4.
>>>
>>I am not seeing this patch in ./patches/linux-2.6.12 
>>directory.  Is it somewhere else?
>>
>
>It should be in the main tree. The appended patch was included some time
>ago, but I've never tested it to see whether it works since I don't
>generally run bleeding edge Fedora.
>
>Is there some piece of the patch missing?
>
No, it's all there.  Only part I needed was the change to ld.so.conf.  
So, anyone with FC4, you might want to make sure you have the entry 
"hwcap 0 nosegneg".  It looks like the performance regression has 
disappeared now.  Thanks all for you help.

-Andrew

RE: 46% performance drop with change in glibc

[Ian Pratt]

 > >It should be in the main tree. The appended patch was included some 
> >time ago, but I've never tested it to see whether it works since I 
> >don't generally run bleeding edge Fedora.
> >
> >Is there some piece of the patch missing?
> >
> No, it's all there.  Only part I needed was the change to 
> ld.so.conf.  
> So, anyone with FC4, you might want to make sure you have the 
> entry "hwcap 0 nosegneg".  It looks like the performance 
> regression has disappeared now.  Thanks all for you help.

Perhaps we should have the install script insert a file called
/etc/ld.so.conf.d/xen containing "hwcap 0 nosegneg" ?

Ian

Re: 46% performance drop with change in glibc

[Andrew Theurer]

On Tuesday 08 November 2005 07:17, Ian Pratt wrote:
>  > >It should be in the main tree. The appended patch was included some
> > >
> > >time ago, but I've never tested it to see whether it works since I
> > >don't generally run bleeding edge Fedora.
> > >
> > >Is there some piece of the patch missing?
> >
> > No, it's all there.  Only part I needed was the change to
> > ld.so.conf.
> > So, anyone with FC4, you might want to make sure you have the
> > entry "hwcap 0 nosegneg".  It looks like the performance
> > regression has disappeared now.  Thanks all for you help.
>
> Perhaps we should have the install script insert a file called
> /etc/ld.so.conf.d/xen containing "hwcap 0 nosegneg" ?

Assuming most/all distros are doing it this way.  I do recall a beta version 
of openSUSE just moved the tls dir in an rc script, but maybe they are doing 
something different now.  Anyone involved with Linux distributions care to 
comment?

-Andrew

Re: 46% performance drop with change in glibc

[Ulrich Drepper]

Andrew Theurer wrote:
>> Perhaps we should have the install script insert a file called
>> /etc/ld.so.conf.d/xen containing "hwcap 0 nosegneg" ?
> 
> Anyone involved with Linux distributions care to comment?

Installing the xen kernel RPM from FC automatically adds the appropriate 
line to ld.so.conf.

-- 
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖

Re: 46% performance drop with change in glibc

[Kurt Garloff]

[-- Attachment #1.1: Type: text/plain, Size: 713 bytes --]

Hi,

On Tue, Nov 08, 2005 at 09:57:05AM -0500, Andrew Theurer wrote:
> Assuming most/all distros are doing it this way.  I do recall a beta version 
> of openSUSE just moved the tls dir in an rc script, 

Yes, indeed, that was done prior to SUSE Linux 10.0: just detect whether 
you're running under Xen and if so move away /lib/tls.

> but maybe they are doing something different now.

In openSUSE Linux 10.0, we just yanked the negative segment references
out of gcc + glibc.
The effort to have two versions just does not justify the non-measurable
performance / code-density gains.
My recommendation would be to do that everywhere.

Best,
-- 
Kurt Garloff, Director SUSE Labs, Novell Inc.

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: 46% performance drop with change in glibc

[Jeremy Katz]

On Tue, 2005-11-08 at 13:17 +0000, Ian Pratt wrote:
>  > >It should be in the main tree. The appended patch was included some 
> > >time ago, but I've never tested it to see whether it works since I 
> > >don't generally run bleeding edge Fedora.
> > >
> > >Is there some piece of the patch missing?
> > >
> > No, it's all there.  Only part I needed was the change to 
> > ld.so.conf.  
> > So, anyone with FC4, you might want to make sure you have the 
> > entry "hwcap 0 nosegneg".  It looks like the performance 
> > regression has disappeared now.  Thanks all for you help.
> 
> Perhaps we should have the install script insert a file called
> /etc/ld.so.conf.d/xen containing "hwcap 0 nosegneg" ?

Sounds sane.  As Uli said, this mirrors what happens when our xen kernel
package is installed.

Jeremy