iptables 1.3.1 crashes on iptables-restore

iptables 1.3.1 crashes on iptables-restore

[Alexander Samad]

[-- Attachment #1: Type: text/plain, Size: 459 bytes --]

Hi

I have just build 1.3.1 (svn version 3805)  with these patches
 policy 
TARPIT iprange raw dstlimit time IPMARK CLASSIFY
dropped-table pool realm TCPLAG ULOG account nf_conntrack nth owner-
socketlookup ownercmd conntrack-cacheline-opt osf mport tcp-window-
tracking quota from POM-NG


when I try to do a iptables-restore with about 800 lines I get a
segmentation fault.

What is the most expediant way to track this problem down ?

Alex

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: iptables 1.3.1 crashes on iptables-restore

[Roberto Nibali]

> when I try to do a iptables-restore with about 800 lines I get a
> segmentation fault.
> What is the most expediant way to track this problem down ?

o Compile and link iptables with debugging info.
o Before you run the command make sure you have the core file size
   settings to unlimited (ulimit -c unlimited).
o Run your command.
o You should have a core file in your directory.
o gdb -q core /path/to/iptables-restore
o bt

Send the output to this list. Hope that is expedient enough ;).

Cheers,
Roberto Nibali, ratz
-- 
echo 
'[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq' | dc

Re: iptables 1.3.1 crashes on iptables-restore

[Amin Azez]

Roberto Nibali wrote:
>> when I try to do a iptables-restore with about 800 lines I get a
>> segmentation fault.
>> What is the most expediant way to track this problem down ?
> 
> 
> o Compile and link iptables with debugging info.
> o Before you run the command make sure you have the core file size
>   settings to unlimited (ulimit -c unlimited).
> o Run your command.
> o You should have a core file in your directory.
> o gdb -q core /path/to/iptables-restore
> o bt
> 
> Send the output to this list. Hope that is expedient enough ;).

I haven't seen any follow-up here but I am also having problems with
hanging and segfaulting with iptables-restore 1.3.1

If I re-order the rules, the problem changes or goes away.

With one combination of rules, I get a sefault with a backtrace as shown
here.

(I am using layer7 in some of my rules)

#0  0x4207418b in _int_malloc () from /lib/tls/libc.so.6
#1  0x42073a5e in calloc () from /lib/tls/libc.so.6
#2  0x0804aa19 in fw_calloc (count=1, size=8484) at iptables.c:514
#3  0x0804d4ce in do_command (argc=13, argv=0x8057660, table=0x8057668,
handle=0xbfffcdec) at iptables.c:2096
#4  0x0804a0fb in main (argc=-1073753484, argv=0xbffffa94) at
iptables-restore.c:401
#5  0x42015574 in __libc_start_main () from /lib/tls/libc.so.6

In this case the segfault occured while processing this rule (mangle table):
-A staticentries -m layer7 --l7proto html -j MARK --set-mark 0x1

but does not occur if I remove this rule which is processed much earlier on:
-A layer7entries -m layer7 --l7proto msnmessenger -j RETURN

To me this indicates memory corruption at some point early on, I would
blame layer7 apart from the report here that I am responding to.


In another case where iptables-restore hangs, strace shows:
...
open("/etc/l7-protocols/protocols/rdp.pat", O_RDONLY) = 5
fstat64(5, {st_mode=S_IFREG|0644, st_size=661, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7ede000
read(5, "# RDP - Remote Desktop Protocol "..., 4096) = 661
futex(0x83e2b98, FUTEX_WAKE, 1, ptrace: umoven: Input/output error
{...})  = 0
futex(0x83e2b98, FUTEX_WAIT, -7, NULL

The interesting point being the blocked futex which to me indicates
(guessing) that the kernel was sent some dodgy iptables soup and is
choking on it.


I'm debugging further to see where this is happening but thought it post
here with what I have.

iptables 1.3.3 doesn't segfault but does hang.

Sam

Re: iptables 1.3.1 crashes on iptables-restore

[Amin Azez]

I'm now on iptables 1.3.3 because 1.3.1 with pablo's opt_merge patch
results in access of unallocated memory, 1.3.3 does not.

1.3.3 still hangs but does not segfault.
1.3.3 with simple rule sets has a clean valgrind report but this sort of
stuff for layer7

==30940== 8729 bytes in 115 blocks are definitely lost in loss record 6 of 7
==30940==    at 0x1B903D1C: malloc (vg_replace_malloc.c:131)
==30940==    by 0x1BC56C33: pre_process (libipt_layer7.c:142)
==30940==    by 0x1BC5702F: parse_layer7_protocol (libipt_layer7.c:267)
==30940==    by 0x1BC57142: parse (libipt_layer7.c:281)
==30940==
==30940==
==30940== 58880 bytes in 115 blocks are definitely lost in loss record 7
of 7
==30940==    at 0x1B903D1C: malloc (vg_replace_malloc.c:131)
==30940==    by 0x1BC56D8D: readl7dir (libipt_layer7.c:171)
==30940==    by 0x1BC56F9A: parse_layer7_protocol (libipt_layer7.c:234)
==30940==    by 0x1BC57142: parse (libipt_layer7.c:281)


So I will use that as my clues from now on, but it looks like iptables
1.3.3 is "clean" and layer7 needs looking at.

Sam

Fixed Re: iptables 1.3.1 crashes on iptables-restore

[Amin Azez]

The futex hanging was in glibc during fclose() (!!!)

The fix was as simple as:

LD_ASSUME_KERNEL=2.2.5

environment variable.

I guess my kernel version is too high for my glibc version or something
like that!

This also stops various segfaults too.

Sam

Re: Fixed Re: iptables 1.3.1 crashes on iptables-restore

[Roberto Nibali]

Amin Azez wrote:
> The futex hanging was in glibc during fclose() (!!!)

Strange. So you run a 2.6.x kernel.

> The fix was as simple as:
> 
> LD_ASSUME_KERNEL=2.2.5
> 
> environment variable.

Which would disable TLS.

> I guess my kernel version is too high for my glibc version or something
> like that!

What's the output /lib/libc.so.6? Just out of curiosity. And maybe if 
you set LD_DEBUG=all and run iptables? Also the ldd /sbin/iptables and 
LD_TRACE_LOADED_OBJECTS=1 setting may give you more input.

> This also stops various segfaults too.

Interesting.

Cheers,
Roberto Nibali, ratz
-- 
echo 
'[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq' | dc

Re: Fixed Re: iptables 1.3.1 crashes on iptables-restore

[Amin Azez]

Roberto Nibali wrote:

> Amin Azez wrote:
>
>> The futex hanging was in glibc during fclose() (!!!)
>
>
> Strange. So you run a 2.6.x kernel.
>
Yes, I need the latest and greatest conntrack stuff, naturally ;-)

>> The fix was as simple as:
>>
>> LD_ASSUME_KERNEL=2.2.5
>>
>> environment variable.
>
>
> Which would disable TLS.
>
>> I guess my kernel version is too high for my glibc version or something
>> like that!
>
>
> What's the output /lib/libc.so.6? 

I'm not sure what you mean here, but ls -l shows:
/lib/libc.so.6 -> libc-2.3.2.so

> Just out of curiosity. And maybe if you set LD_DEBUG=all and run iptables?

It doesn't show anything interesting, iptables-restore does when
restoring L7 rules that use fopen/fclose, they show:
...
...
     25033:     binding file /lib/libnss_files.so.2 to
/lib/tls/libc.so.6: normal symbol `fclose' [GLIBC_2.1]
     25033:     file=/lib/iptables/libipt_tcp.so;  generating link map
     25033:       dynamic: 0xb7fd727c  base: 0xb7fd4000   size: 0x0000335c
     25033:         entry: 0xb7fd4788  phdr: 0xb7fd4034  phnum:          3
...
...

I have to say I'm not sure what libnss* is about here, I thought libnss
stuff was for yp/niss/ldap lookup and the like for getXXXbyYYY libc
calls, libnss_files.so starts to look like a filesystem redirector for
user-filesystems which I thought glibc folk were dead against.  Anyway....

> Also the ldd /sbin/iptables 

ldd iptables shows:
        libdl.so.2 => /lib/libdl.so.2 (0xb7fe1000)
        libc.so.6 => /lib/tls/libc.so.6 (0x42000000)
        /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0xb7feb000)


> and LD_TRACE_LOADED_OBJECTS=1 setting may give you more input.

LD_DEBUG=all LD_TRACE_LOADED_OBJECTS=1 iptables-restore < ~/IPTABLES.BAD

     25325:
     25325:     file=libdl.so.2;  needed by iptables-restore
     25325:     find library=libdl.so.2; searching
     25325:      search cache=/etc/ld.so.cache
     25325:       trying file=/lib/libdl.so.2
     25325:
     25325:     file=libdl.so.2;  generating link map
     25325:       dynamic: 0xb7fe3008  base: 0xb7fe1000   size: 0x000021ac
     25325:         entry: 0xb7fe26f0  phdr: 0xb7fe1034  phnum:          6
     25325:
     25325:
     25325:     file=libc.so.6;  needed by iptables-restore
     25325:     find library=libc.so.6; searching
     25325:      search cache=/etc/ld.so.cache
     25325:       trying file=/lib/tls/libc.so.6
     25325:
     25325:     file=libc.so.6;  generating link map
     25325:       dynamic: 0x42130920  base: 0x00000000   size: 0x00132f08
     25325:         entry: 0x42015660  phdr: 0x42000034  phnum:          8
     25325:
     25325:     checking for version `GLIBC_2.1' in file /lib/libdl.so.2
required by file iptables-restore
     25325:     checking for version `GLIBC_2.0' in file /lib/libdl.so.2
required by file iptables-restore
     25325:     checking for version `GLIBC_2.1' in file
/lib/tls/libc.so.6 required by file iptables-restore
     25325:     checking for version `GLIBC_2.3' in file
/lib/tls/libc.so.6 required by file iptables-restore
     25325:     checking for version `GLIBC_2.0' in file
/lib/tls/libc.so.6 required by file iptables-restore
     25325:     checking for version `GLIBC_PRIVATE' in file
/lib/ld-linux.so.2 required by file /lib/libdl.so.2
     25325:     checking for version `GLIBC_2.3' in file
/lib/tls/libc.so.6 required by file /lib/libdl.so.2
     25325:     checking for version `GLIBC_2.1.3' in file
/lib/tls/libc.so.6 required by file /lib/libdl.so.2
     25325:     checking for version `GLIBC_2.1' in file
/lib/tls/libc.so.6 required by file /lib/libdl.so.2
     25325:     checking for version `GLIBC_PRIVATE' in file
/lib/tls/libc.so.6 required by file /lib/libdl.so.2
     25325:     checking for version `GLIBC_2.0' in file
/lib/tls/libc.so.6 required by file /lib/libdl.so.2
     25325:     checking for version `GLIBC_2.3' in file
/lib/ld-linux.so.2 required by file /lib/tls/libc.so.6
     25325:     checking for version `GLIBC_2.1' in file
/lib/ld-linux.so.2 required by file /lib/tls/libc.so.6
     25325:     checking for version `GLIBC_2.0' in file
/lib/ld-linux.so.2 required by file /lib/tls/libc.so.6
     25325:     checking for version `GLIBC_PRIVATE' in file
/lib/ld-linux.so.2 required by file /lib/tls/libc.so.6
        libdl.so.2 => /lib/libdl.so.2 (0xb7fe1000)
        libc.so.6 => /lib/tls/libc.so.6 (0x42000000)
        /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0xb7feb000)



>
>> This also stops various segfaults too.
>
>
> Interesting.

worrying

Sam

Re: Fixed Re: iptables 1.3.1 crashes on iptables-restore

[Roberto Nibali]

> Yes, I need the latest and greatest conntrack stuff, naturally ;-)

So you run a 2.6.14 kernel :)

>>What's the output /lib/libc.so.6? 
>  
> I'm not sure what you mean here, but ls -l shows:
> /lib/libc.so.6 -> libc-2.3.2.so

Invoke /lib/libc.so.6 like a command. It won't destroy your system but
it will display how your libc was crafted.

>>Just out of curiosity. And maybe if you set LD_DEBUG=all and run iptables?
>  
> It doesn't show anything interesting, iptables-restore does when
> restoring L7 rules that use fopen/fclose, they show:
> ...
> ...
>      25033:     binding file /lib/libnss_files.so.2 to
> /lib/tls/libc.so.6: normal symbol `fclose' [GLIBC_2.1]
>      25033:     file=/lib/iptables/libipt_tcp.so;  generating link map
>      25033:       dynamic: 0xb7fd727c  base: 0xb7fd4000   size: 0x0000335c
>      25033:         entry: 0xb7fd4788  phdr: 0xb7fd4034  phnum:          3
> ...
> ...
> 
> I have to say I'm not sure what libnss* is about here, I thought libnss
> stuff was for yp/niss/ldap lookup and the like for getXXXbyYYY libc
> calls, libnss_files.so starts to look like a filesystem redirector for
> user-filesystems which I thought glibc folk were dead against.  Anyway....

libnss is a long and sad story in glibc development, especially when it
comes to linking ...

>>Also the ldd /sbin/iptables 
> 
> ldd iptables shows:
>         libdl.so.2 => /lib/libdl.so.2 (0xb7fe1000)
>         libc.so.6 => /lib/tls/libc.so.6 (0x42000000)
>         /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0xb7feb000)
>  
>>and LD_TRACE_LOADED_OBJECTS=1 setting may give you more input.
> 
> 
> LD_DEBUG=all LD_TRACE_LOADED_OBJECTS=1 iptables-restore < ~/IPTABLES.BAD

I don't see anything obviously strange, so I can't help you here, sorry.

Regards,
Roberto Nibali, ratz
-- 
-------------------------------------------------------------
addr://Rathausgasse 31, CH-5001 Aarau  tel://++41 62 823 9355
http://www.terreactive.com             fax://++41 62 823 9356
-------------------------------------------------------------
terreActive AG                       Wir sichern Ihren Erfolg
-------------------------------------------------------------

Re: Fixed Re: iptables 1.3.1 crashes on iptables-restore

[Amin Azez]

Roberto Nibali wrote:

>>Yes, I need the latest and greatest conntrack stuff, naturally ;-)
>>    
>>
>
>So you run a 2.6.14 kernel :)
>  
>
I'm waiting for all the fancy goods to get merged before I take that one.

>
>Invoke /lib/libc.so.6 like a command. It won't destroy your system but
>it will display how your libc was crafted.
>  
>
GNU C Library stable release version 2.3.2, by Roland McGrath et al.
Copyright (C) 2003 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 3.2.2 20030222 (Red Hat Linux 3.2.2-5).
Compiled on a Linux 2.4.20 system on 2003-03-13.
Available extensions:
        GNU libio by Per Bothner
        crypt add-on version 2.1 by Michael Glad and others
        linuxthreads-0.10 by Xavier Leroy
        BIND-8.2.3-T5B
        libthread_db work sponsored by Alpha Processor Inc
        NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
Thread-local storage support included.
Report bugs using the `glibcbug' script to <bugs@gnu.org>.

Sam

Re: Fixed Re: iptables 1.3.1 crashes on iptables-restore

[Roberto Nibali]

> I'm waiting for all the fancy goods to get merged before I take that one.

They get merged into the next mm round I reckon, Linus won't pull until
2.6.13 is out the doors.

>>Invoke /lib/libc.so.6 like a command. It won't destroy your system but
>>it will display how your libc was crafted.
>> 
> GNU C Library stable release version 2.3.2, by Roland McGrath et al.
> Copyright (C) 2003 Free Software Foundation, Inc.
> This is free software; see the source for copying conditions.
> There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
> PARTICULAR PURPOSE.
> Compiled by GNU CC version 3.2.2 20030222 (Red Hat Linux 3.2.2-5).
> Compiled on a Linux 2.4.20 system on 2003-03-13.
> Available extensions:
>         GNU libio by Per Bothner
>         crypt add-on version 2.1 by Michael Glad and others
>         linuxthreads-0.10 by Xavier Leroy
>         BIND-8.2.3-T5B
>         libthread_db work sponsored by Alpha Processor Inc
>         NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
> Thread-local storage support included.

I'm by no means a libc expert but how did you get infected with this
libc version? :). I was under the impression that TLS support wasn't
functional until NPTL was introduced in the 2.6.x era. To my knowledge
this could explain the TLS related crashes you were seeing. Any chance
of upgrading this libc of yours?

I could of course be totally wrong.

Best regards,
Roberto Nibali, ratz
-- 
-------------------------------------------------------------
addr://Rathausgasse 31, CH-5001 Aarau  tel://++41 62 823 9355
http://www.terreactive.com             fax://++41 62 823 9356
-------------------------------------------------------------
terreActive AG                       Wir sichern Ihren Erfolg
-------------------------------------------------------------

Re: Fixed Re: iptables 1.3.1 crashes on iptables-restore

[Amin Azez]

Roberto Nibali wrote:

>>I'm waiting for all the fancy goods to get merged before I take that one.
>>    
>>
>They get merged into the next mm round I reckon, Linus won't pull until
>2.6.13 is out the doors.
>  
>
>>>Invoke /lib/libc.so.6 like a command. It won't destroy your system but
>>>it will display how your libc was crafted.
>>>
>>>      
>>>
>>GNU C Library stable release version 2.3.2, by Roland McGrath et al.
>>Copyright (C) 2003 Free Software Foundation, Inc.
>>This is free software; see the source for copying conditions.
>>There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
>>PARTICULAR PURPOSE.
>>Compiled by GNU CC version 3.2.2 20030222 (Red Hat Linux 3.2.2-5).
>>Compiled on a Linux 2.4.20 system on 2003-03-13.
>>Available extensions:
>>        GNU libio by Per Bothner
>>        crypt add-on version 2.1 by Michael Glad and others
>>        linuxthreads-0.10 by Xavier Leroy
>>        BIND-8.2.3-T5B
>>        libthread_db work sponsored by Alpha Processor Inc
>>        NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
>>Thread-local storage support included.
>>    
>>
>
>I'm by no means a libc expert but how did you get infected with this
>libc version? :). 
>
er.. I inherited it on my development box.

>I was under the impression that TLS support wasn't
>functional until NPTL was introduced in the 2.6.x era. 
>
I'm not sure that this glibc is providing TLS, can you see that it is?
Certainly I can't see that iptables needs any, although maybe some of
the libc stuff does(all the getnameby stuff maybe wants per-thread
static buffers instead of per-process?)

>To my knowledge
>this could explain the TLS related crashes you were seeing. Any chance
>of upgrading this libc of yours?
>  
>
I hope so. Are you sure they are TLS related?

>I could of course be totally wrong.
>
>  
>
It's more than I've had to go on so far, so keep talking.

Sam

>Best regards,
>Roberto Nibali, ratz
>  
>

Re: Fixed Re: iptables 1.3.1 crashes on iptables-restore

[Roberto Nibali]

>>>GNU C Library stable release version 2.3.2, by Roland McGrath et al.
>>>Copyright (C) 2003 Free Software Foundation, Inc.
>>>This is free software; see the source for copying conditions.
>>>There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
>>>PARTICULAR PURPOSE.
>>>Compiled by GNU CC version 3.2.2 20030222 (Red Hat Linux 3.2.2-5).
>>>Compiled on a Linux 2.4.20 system on 2003-03-13.
>>>Available extensions:
>>>       GNU libio by Per Bothner
>>>       crypt add-on version 2.1 by Michael Glad and others
>>>       linuxthreads-0.10 by Xavier Leroy
>>>       BIND-8.2.3-T5B
>>>       libthread_db work sponsored by Alpha Processor Inc
>>>       NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
>>>Thread-local storage support included.
   ^^^^^^^^^^^^^^^^^^^^
To me this indicates that TLS is included which is strange. The libc I
use for our 2.4.x kernel stuff looks as follows:

GNU C Library stable release version 2.3.5, by Roland McGrath et al.
Copyright (C) 2005 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 3.4.4.
Compiled on a Linux 2.6.11 system on 2005-07-26.
Available extensions:
        GNU libio by Per Bothner
        crypt add-on version 2.1 by Michael Glad and others
        linuxthreads-0.10 by Xavier Leroy
        BIND-8.2.3-T5B
        libthread_db work sponsored by Alpha Processor Inc
        NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>.

The one I use on 2.6.x kernels is as follows:

GNU C Library stable release version 2.3.2 (20030827), by Roland McGrath
et al.
Copyright (C) 2003 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Configured for i686-suse-linux.
Compiled by GNU CC version 3.3.1 (SuSE Linux).
Compiled on a Linux 2.6.0-test3 system on 2003-09-23.
Available extensions:
        GNU libio by Per Bothner
        crypt add-on version 2.1 by Michael Glad and others
        linuxthreads-0.10 by Xavier Leroy
        NoVersion patch for broken glibc 2.0 binaries
        BIND-8.2.3-T5B
        libthread_db work sponsored by Alpha Processor Inc
        NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
Thread-local storage support included.
Report bugs using the `glibcbug' script to <bugs@gnu.org>.

So I wonder how TLS has slipped into your libc.

>>
>>I'm by no means a libc expert but how did you get infected with this
>>libc version? :). 
>> 
> er.. I inherited it on my development box.

Ok. I see that's a RH box so I presume they certainly know what they
did. I vaguely rememeber that someone of RH once backported the NPTL
stuff to 2.4.x kernels, so this would explain your crippled libc :).

>>I was under the impression that TLS support wasn't
>>functional until NPTL was introduced in the 2.6.x era. 
>> 
> I'm not sure that this glibc is providing TLS, can you see that it is?

The output of /lib/libc.so.6 tells me.

> Certainly I can't see that iptables needs any, although maybe some of
> the libc stuff does(all the getnameby stuff maybe wants per-thread
> static buffers instead of per-process?)

iptables has no influence on this, it's how the linker will put it
together. Obviously it has added the TLS enabled c-library to your
iptables binary which could have provoked this issue. Also when you
LD_ASSUME*'d iptables it wouldn't show the defect anymore. So the
problem must be related to your libc or the toolchain you use to compile
binaries.

>>To my knowledge
>>this could explain the TLS related crashes you were seeing. Any chance
>>of upgrading this libc of yours?
>> 
>>
> I hope so. Are you sure they are TLS related?

No, I'm afraid. In my normal life I thankfully almost never have to deal
with libc issues.

>>I could of course be totally wrong.
>>
> It's more than I've had to go on so far, so keep talking.

I'm off now and I think we should continue this discussion privately,
since the added value for netfilter developers is getting slimmer with
each reply :). We can always post the result of our findings back to
this list, IMHO.

Cheers,
Roberto Nibali, ratz
-- 
-------------------------------------------------------------
addr://Rathausgasse 31, CH-5001 Aarau  tel://++41 62 823 9355
http://www.terreactive.com             fax://++41 62 823 9356
-------------------------------------------------------------
terreActive AG                       Wir sichern Ihren Erfolg
-------------------------------------------------------------