spinlock in centos 6.4 and redhat enterprise 6 using chcon

spinlock in centos 6.4 and redhat enterprise 6 using chcon

[Stefano Borini]

Good morning,

I am encountering what seems to be a spinlock with the chcon utility 
trying the following operation

chcon -t texrel_shlib_t /tmp/subdir/withheldpath

where withheld path is a .so that is going to be accessed with dlopen.
I am not invoking the chcon command directly nor performing the dlopen, 
a closed-source library does that, apparently to prepare the .so for 
dlopening.

Note that if I try the same operation from the command line, even while 
the spinlock is in progress, no lock occurs.

I am unable to understand the details of what may cause this spinlock. 
This is the backtrace of chcon, apparently involving some thread local 
storage

#0  0x0000003e3ea00b64 in rtld_lock_default_lock_recursive () from 
/lib64/ld-linux-x86-64.so.2
#1  0x0000003e3ea11257 in tls_get_addr_tail () from 
/lib64/ld-linux-x86-64.so.2
#2  0x0000003e3ea11660 in __tls_get_addr () from /lib64/ld-linux-x86-64.so.2
#3  0x0000003e40a14334 in selinux_raw_to_trans_context () from 
/lib64/libselinux.so.1
#4  0x0000003e40a0ca7a in getfilecon () from /lib64/libselinux.so.1

Checking the tls_get_addr_tail function, it is apparently stuck in the 
again: loop

http://code.woboq.org/userspace/glibc/elf/dl-tls.c.html#742

I have only access to the centos 6.4 and can run additional 
non-destructive tests if needed. It's a customer machine so I am unable 
to say if modifications have been done to it when it comes to security, 
although I suspect it's a standard centos6.4 installation with selinux 
enabled.

The current ls -Z of /tmp gives

system_u:object_r:tmp_t:s0

of subdir and of the so file is

unconfined_u:object_r:user_tmp_t:s0

Thank you for your help.


-- 
Stefano Borini
QuantumWise A/S

Re: spinlock in centos 6.4 and redhat enterprise 6 using chcon

[Stephen Smalley]

On 01/29/2015 05:14 AM, Stefano Borini wrote:
> Good morning,
> 
> I am encountering what seems to be a spinlock with the chcon utility
> trying the following operation
> 
> chcon -t texrel_shlib_t /tmp/subdir/withheldpath
> 
> where withheld path is a .so that is going to be accessed with dlopen.
> I am not invoking the chcon command directly nor performing the dlopen,
> a closed-source library does that, apparently to prepare the .so for
> dlopening.
> 
> Note that if I try the same operation from the command line, even while
> the spinlock is in progress, no lock occurs.
> 
> I am unable to understand the details of what may cause this spinlock.
> This is the backtrace of chcon, apparently involving some thread local
> storage
> 
> #0  0x0000003e3ea00b64 in rtld_lock_default_lock_recursive () from
> /lib64/ld-linux-x86-64.so.2
> #1  0x0000003e3ea11257 in tls_get_addr_tail () from
> /lib64/ld-linux-x86-64.so.2
> #2  0x0000003e3ea11660 in __tls_get_addr () from
> /lib64/ld-linux-x86-64.so.2
> #3  0x0000003e40a14334 in selinux_raw_to_trans_context () from
> /lib64/libselinux.so.1
> #4  0x0000003e40a0ca7a in getfilecon () from /lib64/libselinux.so.1
> 
> Checking the tls_get_addr_tail function, it is apparently stuck in the
> again: loop
> 
> http://code.woboq.org/userspace/glibc/elf/dl-tls.c.html#742
> 
> I have only access to the centos 6.4 and can run additional
> non-destructive tests if needed. It's a customer machine so I am unable
> to say if modifications have been done to it when it comes to security,
> although I suspect it's a standard centos6.4 installation with selinux
> enabled.
> 
> The current ls -Z of /tmp gives
> 
> system_u:object_r:tmp_t:s0
> 
> of subdir and of the so file is
> 
> unconfined_u:object_r:user_tmp_t:s0
> 
> Thank you for your help.

libselinux version?

Re: spinlock in centos 6.4 and redhat enterprise 6 using chcon

[Stephen Smalley]

On 01/29/2015 05:14 AM, Stefano Borini wrote:
> Good morning,
> 
> I am encountering what seems to be a spinlock with the chcon utility
> trying the following operation
> 
> chcon -t texrel_shlib_t /tmp/subdir/withheldpath
> 
> where withheld path is a .so that is going to be accessed with dlopen.
> I am not invoking the chcon command directly nor performing the dlopen,
> a closed-source library does that, apparently to prepare the .so for
> dlopening.

While this obviously shouldn't hang, it is definitely wrong for this
library to be invoking chcon on the .so file.  The label should be set
when the .so file is first installed, preferably by rpm itself by adding
a file_contexts entry via semanage fcontext -a followed by a restorecon
call in the %post scriptlet.  Can you bug the author of the
closed-source library to fix their package?

> 
> Note that if I try the same operation from the command line, even while
> the spinlock is in progress, no lock occurs.
> 
> I am unable to understand the details of what may cause this spinlock.
> This is the backtrace of chcon, apparently involving some thread local
> storage
> 
> #0  0x0000003e3ea00b64 in rtld_lock_default_lock_recursive () from
> /lib64/ld-linux-x86-64.so.2
> #1  0x0000003e3ea11257 in tls_get_addr_tail () from
> /lib64/ld-linux-x86-64.so.2
> #2  0x0000003e3ea11660 in __tls_get_addr () from
> /lib64/ld-linux-x86-64.so.2
> #3  0x0000003e40a14334 in selinux_raw_to_trans_context () from
> /lib64/libselinux.so.1
> #4  0x0000003e40a0ca7a in getfilecon () from /lib64/libselinux.so.1
> 
> Checking the tls_get_addr_tail function, it is apparently stuck in the
> again: loop
> 
> http://code.woboq.org/userspace/glibc/elf/dl-tls.c.html#742
> 
> I have only access to the centos 6.4 and can run additional
> non-destructive tests if needed. It's a customer machine so I am unable
> to say if modifications have been done to it when it comes to security,
> although I suspect it's a standard centos6.4 installation with selinux
> enabled.
> 
> The current ls -Z of /tmp gives
> 
> system_u:object_r:tmp_t:s0
> 
> of subdir and of the so file is
> 
> unconfined_u:object_r:user_tmp_t:s0
> 
> Thank you for your help.
> 
> 

Re: spinlock in centos 6.4 and redhat enterprise 6 using chcon

[Stefano Borini]

On 01/30/2015 05:15 PM, Stephen Smalley wrote:
> While this obviously shouldn't hang, it is definitely wrong for this
> library to be invoking chcon on the .so file.  The label should be set
> when the .so file is first installed, preferably by rpm itself by adding
> a file_contexts entry via semanage fcontext -a followed by a restorecon
> call in the %post scriptlet.  Can you bug the author of the
> closed-source library to fix their package?

I mailed them and waiting for an answer, but I guess that they are doing 
so as a workaround because they need to dlopen it and they are unable to 
do so.

The version of selinux is the default provided by centos6.4. I'll write 
back the specific detail on Monday. I don't have access to the machine 
outside of office hours.

I tried to produce some code that simulate what I think it might happen 
in the closed source library, but I was unable to reproduce the problem. 
My assumption was that a separate thread was issuing a dlopen and then 
the chcon, but besides the fact that I don't see how this may lead to 
chcon hanging, it failed to produce any problem.

I also tried to reproduce the issue on another centos6.4 installation 
without success. However, we already encountered this hang condition in 
two unrelated customers, so it's not a random fluke.




-- 
Stefano Borini
QuantumWise A/S

Re: spinlock in centos 6.4 and redhat enterprise 6 using chcon

[Stephen Smalley]

On 01/30/2015 04:25 PM, Stefano Borini wrote:
> On 01/30/2015 05:15 PM, Stephen Smalley wrote:
>> While this obviously shouldn't hang, it is definitely wrong for this
>> library to be invoking chcon on the .so file.  The label should be set
>> when the .so file is first installed, preferably by rpm itself by adding
>> a file_contexts entry via semanage fcontext -a followed by a restorecon
>> call in the %post scriptlet.  Can you bug the author of the
>> closed-source library to fix their package?
> 
> I mailed them and waiting for an answer, but I guess that they are doing
> so as a workaround because they need to dlopen it and they are unable to
> do so.
> 
> The version of selinux is the default provided by centos6.4. I'll write
> back the specific detail on Monday. I don't have access to the machine
> outside of office hours.
> 
> I tried to produce some code that simulate what I think it might happen
> in the closed source library, but I was unable to reproduce the problem.
> My assumption was that a separate thread was issuing a dlopen and then
> the chcon, but besides the fact that I don't see how this may lead to
> chcon hanging, it failed to produce any problem.
> 
> I also tried to reproduce the issue on another centos6.4 installation
> without success. However, we already encountered this hang condition in
> two unrelated customers, so it's not a random fluke.

I'm wondering if it might be a bug in glibc in that centos release
rather than in libselinux.  I don't see any relevant difference in
libselinux/src/setrans_client.c between the .src.rpm for centos 6.4 and
current master to explain it, so if it is truly a bug in libselinux, it
would seem to still be present.

Also, it looks like 6.4 is long since obsolete, so upgrading would be
advisable regardless.

Re: spinlock in centos 6.4 and redhat enterprise 6 using chcon

[Stefano Borini]

> I'm wondering if it might be a bug in glibc in that centos release
> rather than in libselinux.  I don't see any relevant difference in
> libselinux/src/setrans_client.c between the .src.rpm for centos 6.4 and
> current master to explain it, so if it is truly a bug in libselinux, it
> would seem to still be present.


Could be. I will try to send them a mail as well, maybe they have
some ideas. I'll keep you guys posted if I learn something useful
for selinux. At this point, I am mostly trying to find a workaround
that doesn't require excessive tinkering from my client. Also,
personal curiosity.


> Also, it looks like 6.4 is long since obsolete, so upgrading would be
> advisable regardless.


It's a business customer and I can't propose that as a solution.


-- 
Stefano Borini
QuantumWise A/S

Re: spinlock in centos 6.4 and redhat enterprise 6 using chcon

[Stefano Borini]

On 02/01/2015 12:17 PM, Stefano Borini wrote:
>
>> I'm wondering if it might be a bug in glibc in that centos release
>> rather than in libselinux.  I don't see any relevant difference in
>> libselinux/src/setrans_client.c between the .src.rpm for centos 6.4 and
>> current master to explain it, so if it is truly a bug in libselinux, it
>> would seem to still be present.

I found the source of the problem.

Due to other constraints, we are forced to LD_PRELOAD libGL. This in 
turn preloads libnvidia-tls, which apparently messes up something with 
the glibc tls support.

Removing the libGL preloading fixes the issue, although it introduces 
another, but I'll figure something out.

Thanks for the help!


-- 
Stefano Borini
QuantumWise A/S