[RFC dwarves 0/3] Explore using address-based DWARF->ELF

[RFC dwarves 0/3] Explore using address-based DWARF->ELF

[Alan Maguire]

This work-in-progress series is an initial exploration of using
function addresses to match between DWARF and ELF to improve matching
accuracy, and as a prelude to potentially representing address info
in BTF itself.

It is missing ~1000 functions in the resultant generated vmlinux BTF
where we currently cannot find an associated address in the DWARF;
there may be further enhancements needed in our DWARF function address
association, or we may ultimately need to use a hybrid address/name
scheme, falling back to names where DWARF addresses are not present.

When we come to encode addresses in BTF, they will be likely 32-bit
section relative.  This will allow for kernels that randomize the kernel
base address.  But that is not tackled yet; these initial changes are
focused on improving the BTF generated using address-based matching,
but not augmenting BTF with addresses yet.

Would be great if folks could see if there are any potential
improvements to make the matching process more complete, or just want
to try out the series and compare before/after BTF encoding to see what
is missing. Thanks!

Alan Maguire (3):
  btf_encoder: collect ELF function addresses
  dwarf_loader: make more effort to retrieve a function address
  btf_encoder: use function address to match ELF -> DWARF

 btf_encoder.c  | 78 ++++++++++++++++++++++++++------------------------
 dwarf_loader.c | 29 ++++++++++++++++---
 dwarves.h      |  3 +-
 3 files changed, 68 insertions(+), 42 deletions(-)

-- 
2.39.3

[RFC dwarves 1/3] btf_encoder: collect ELF function addresses

[Alan Maguire]

When collecting immutable ELF function info, store addresses
of functions as these will be used for better DWARF/ELF correspondence
matching.

Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
---
 btf_encoder.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/btf_encoder.c b/btf_encoder.c
index 0bc2334..b1e64d1 100644
--- a/btf_encoder.c
+++ b/btf_encoder.c
@@ -98,6 +98,7 @@ struct elf_function {
 	size_t		prefixlen;
 	bool		kfunc;
 	uint32_t	kfunc_flags;
+	uint64_t	addr;
 };
 
 struct elf_secinfo {
@@ -1461,6 +1462,7 @@ static void elf_functions__collect_function(struct elf_functions *functions, GEl
 
 	func = &functions->entries[functions->cnt];
 	func->name = name;
+	func->addr = elf_sym__value(sym);
 	if (strchr(name, '.')) {
 		const char *suffix = strchr(name, '.');
 
-- 
2.39.3

[RFC dwarves 2/3] dwarf_loader: make more effort to retrieve a function address

[Alan Maguire]

Many DW_AT_subroutine DIEs do not have address info associated
directly; some use DW_AT_ranges, and some have address info
stored with the associated abstract origin.  Use these information
sources to improve address retrieval for "struct function"
representations.

Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
---
 dwarf_loader.c | 29 +++++++++++++++++++++++++----
 dwarves.h      |  3 ++-
 2 files changed, 27 insertions(+), 5 deletions(-)

diff --git a/dwarf_loader.c b/dwarf_loader.c
index e1ba7bc..d46e82a 100644
--- a/dwarf_loader.c
+++ b/dwarf_loader.c
@@ -1519,6 +1519,17 @@ static struct function *function__new(Dwarf_Die *die, struct cu *cu, struct conf
 		func->vtable_entry    = -1;
 		if (dwarf_hasattr(die, DW_AT_vtable_elem_location))
 			func->vtable_entry = attr_offset(die, DW_AT_vtable_elem_location);
+		/* some functions will have entry address in the start address
+		 * in DW_AT_ranges.
+		 */
+		if (dwarf_hasattr(die, DW_AT_ranges)) {
+			Dwarf_Addr base, start, high_pc;
+			ptrdiff_t offset = 0;
+
+			dwarf_ranges(die, offset, &base, &start, &high_pc);
+			if (start)
+				func->addr = (uint64_t)start;
+		}
 		func->cu_total_size_inline_expansions = 0;
 		func->cu_total_nr_inline_expansions = 0;
 		func->priv = NULL;
@@ -2805,7 +2816,7 @@ static int tag__recode_dwarf_type(struct tag *tag, struct cu *cu)
 	case DW_TAG_subprogram: {
 		struct function *fn = tag__function(tag);
 
-		if (fn->name == 0)  {
+		if (fn->name == 0 || function__addr(fn) == 0) {
 			if (dtag->abstract_origin == 0 &&
 			    dtag->specification == 0) {
 				/*
@@ -2818,9 +2829,19 @@ static int tag__recode_dwarf_type(struct tag *tag, struct cu *cu)
 			dtype = dwarf_cu__find_tag_by_ref(cu->priv, dtag, abstract_origin);
 			if (dtype == NULL)
 				dtype = dwarf_cu__find_tag_by_ref(cu->priv, dtag, specification);
-			if (dtype != NULL)
-				fn->name = tag__function(dtag__tag(dtype))->name;
-			else {
+			if (dtype != NULL) {
+				struct function *dfn = tag__function(dtag__tag(dtype));
+				if (fn->name == 0)
+					fn->name = dfn->name;
+				/* share function address info between tag and
+				 * abstract origin.
+				 */
+				if (!function__addr(fn))
+					fn->addr = function__addr(dfn);
+				else if (!function__addr(dfn))
+					dfn->addr = function__addr(fn);
+			}
+			if (fn->name == 0) {
 				fprintf(stderr,
 					"%s: couldn't find name for "
 					"function %#llx, abstract_origin=%#llx,"
diff --git a/dwarves.h b/dwarves.h
index 36c6898..847e13d 100644
--- a/dwarves.h
+++ b/dwarves.h
@@ -1081,6 +1081,7 @@ struct function {
 	const char	 *name;
 	const char	 *linkage_name;
 	const char	 *alias;	/* name.isra.0 */
+	uint64_t	 addr;
 	uint32_t	 cu_total_size_inline_expansions;
 	uint16_t	 cu_total_nr_inline_expansions;
 	uint8_t		 inlined:2;
@@ -1146,7 +1147,7 @@ const char *function__prototype_conf(const struct function *func,
 
 static __pure inline uint64_t function__addr(const struct function *func)
 {
-	return func->lexblock.ip.addr;
+	return func->lexblock.ip.addr ?: func->addr;
 }
 
 static __pure inline uint32_t function__size(const struct function *func)
-- 
2.39.3

[RFC dwarves 3/3] btf_encoder: use function address to match ELF -> DWARF

[Alan Maguire]

Currently we use function names (or prefixes in the case of
foo.isra.0) to match betwen ELF symtab entries and DWARF
representations.  This can lead to wrong matches, especially
where optimized function representations are concerned.  Instead
sort and search ELF functions by address, and use the retrieved
"struct function" address to carry out DWARF->ELF matches.

Note this is work-in-progress and many functions are missing as
many functions do not have - or at least we have not retrieved -
address info associated with their DWARF representations.

As things stand, there are exactly 1000 functions missing from
BTF encoded using the address-based approach, since we skip functions
for which we have no address info.  This approach actually adds
63 functions, so there are effectively 1063 missing functions.

485 of these missing functions are __probestub functions we do not need, i.e.

[66116] FUNC '__probestub_xhci_setup_device' type_id=61452 linkage=static
[61452] FUNC_PROTO '(anon)' ret_type_id=0 vlen=2
        '__data' type_id=108
        'vdev' type_id=44186

The real function is:

[147543] FUNC 'xhci_setup_device' type_id=147542 linkage=static
[147542] FUNC_PROTO '(anon)' ret_type_id=21 vlen=4
        'hcd' type_id=37057
        'udev' type_id=37029
        'setup' type_id=44209
        'timeout_ms' type_id=9

This leaves us with a mismatch of 578 functions.  These include
140 missing __bpf_trace_ functions, which are definitely needed.

So perhaps we can fix up our DWARF representation to find associated
addresses for some/all of these, but we may end up having to fall
back to name-based association for some cases.

Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
---
 btf_encoder.c | 78 ++++++++++++++++++++++++++-------------------------
 1 file changed, 40 insertions(+), 38 deletions(-)

diff --git a/btf_encoder.c b/btf_encoder.c
index b1e64d1..f4fd937 100644
--- a/btf_encoder.c
+++ b/btf_encoder.c
@@ -1344,12 +1344,7 @@ static int functions_cmp(const void *_a, const void *_b)
 	const struct elf_function *a = _a;
 	const struct elf_function *b = _b;
 
-	/* if search key allows prefix match, verify target has matching
-	 * prefix len and prefix matches.
-	 */
-	if (a->prefixlen && a->prefixlen == b->prefixlen)
-		return strncmp(a->name, b->name, b->prefixlen);
-	return strcmp(a->name, b->name);
+	return a->addr - b->addr;
 }
 
 #ifndef max
@@ -1358,10 +1353,15 @@ static int functions_cmp(const void *_a, const void *_b)
 
 static int saved_functions_cmp(const void *_a, const void *_b)
 {
-	const struct btf_encoder_func_state *a = _a;
-	const struct btf_encoder_func_state *b = _b;
+	const struct elf_function *a = ((const struct btf_encoder_func_state *)_a)->elf;
+	const struct elf_function *b = ((const struct btf_encoder_func_state *)_b)->elf;
 
-	return functions_cmp(a->elf, b->elf);
+	/* if search key allows prefix match, verify target has matching
+	 * prefix len and prefix matches.
+	 */
+	if (a->prefixlen && a->prefixlen == b->prefixlen)
+		return strncmp(a->name, b->name, b->prefixlen);
+	return strcmp(a->name, b->name);
 }
 
 static int saved_functions_combine(struct btf_encoder_func_state *a, struct btf_encoder_func_state *b)
@@ -1457,12 +1457,14 @@ static void elf_functions__collect_function(struct elf_functions *functions, GEl
 		return;
 
 	name = elf_sym__name(sym, functions->symtab);
-	if (!name)
+	if (!name || strstr(name, ".cold."))
 		return;
 
 	func = &functions->entries[functions->cnt];
-	func->name = name;
 	func->addr = elf_sym__value(sym);
+	if (!func->addr)
+		return;
+	func->name = name;
 	if (strchr(name, '.')) {
 		const char *suffix = strchr(name, '.');
 
@@ -1493,10 +1495,10 @@ static struct elf_functions *btf_encoder__elf_functions(struct btf_encoder *enco
 }
 
 static struct elf_function *btf_encoder__find_function(const struct btf_encoder *encoder,
-						       const char *name, size_t prefixlen)
+						       uint64_t addr) 
 {
 	struct elf_functions *funcs = elf_functions__find(encoder->cu->elf, &encoder->elf_functions_list);
-	struct elf_function key = { .name = name, .prefixlen = prefixlen };
+	struct elf_function key = { .addr = addr };
 
 	return bsearch(&key, funcs->entries, funcs->cnt, sizeof(key), functions_cmp);
 }
@@ -2019,6 +2021,7 @@ static int btf_encoder__collect_kfuncs(struct btf_encoder *encoder)
 		struct elf_function *elf_fn;
 		unsigned int ranges_cnt;
 		char *func, *name;
+		uint64_t addr;
 		ptrdiff_t off;
 		GElf_Sym sym;
 		bool found;
@@ -2036,6 +2039,9 @@ static int btf_encoder__collect_kfuncs(struct btf_encoder *encoder)
 		func = get_func_name(name);
 		if (!func)
 			continue;
+		addr = elf_sym__value(&sym);
+		if (!addr)
+			continue;
 
 		/* Check if function belongs to a kfunc set */
 		ranges = gobuffer__entries(&btf_kfunc_ranges);
@@ -2062,7 +2068,7 @@ static int btf_encoder__collect_kfuncs(struct btf_encoder *encoder)
 			continue;
 		}
 
-		elf_fn = btf_encoder__find_function(encoder, func, 0);
+		elf_fn = btf_encoder__find_function(encoder, addr);
 		if (elf_fn) {
 			elf_fn->kfunc = true;
 			elf_fn->kfunc_flags = pair->flags;
@@ -2658,35 +2664,31 @@ int btf_encoder__encode_cu(struct btf_encoder *encoder, struct cu *cu, struct co
 			continue;
 		if (funcs->cnt) {
 			const char *name;
+			uint64_t addr;
 
 			name = function__name(fn);
 			if (!name)
 				continue;
+			addr = function__addr(fn);
+			if (encoder->verbose)
+				printf("function '%s' addr 0x%lx\n",
+				       name, addr);
+			if (!addr)
+				continue;
 
-			/* prefer exact function name match... */
-			func = btf_encoder__find_function(encoder, name, 0);
-			if (!func && funcs->suffix_cnt &&
-			    conf_load->btf_gen_optimized) {
-				/* falling back to name.isra.0 match if no exact
-				 * match is found; only bother if we found any
-				 * .suffix function names.  The function
-				 * will be saved and added once we ensure
-				 * it does not have optimized-out parameters
-				 * in any cu.
-				 */
-				func = btf_encoder__find_function(encoder, name,
-								  strlen(name));
-				if (func) {
-					if (encoder->verbose)
-						printf("matched function '%s' with '%s'%s\n",
-						       name, func->name,
-						       fn->proto.optimized_parms ?
-						       ", has optimized-out parameters" :
-						       fn->proto.unexpected_reg ? ", has unexpected register use by params" :
-						       "");
-					if (!func->alias)
-						func->alias = strdup(name);
-				}
+			func = btf_encoder__find_function(encoder, addr);
+			if (func && strchr(func->name, '.')) {
+				if (!conf_load->btf_gen_optimized) {
+					continue;
+				} else if (encoder->verbose)
+					printf("matched function '%s' with '%s'%s\n",
+					       name, func->name,
+					       fn->proto.optimized_parms ?
+					       ", has optimized-out parameters" :
+					       fn->proto.unexpected_reg ? ", has unexpected register use by params" :
+					       "");
+				if (!func->alias)
+					func->alias = strdup(name);
 			}
 		} else {
 			if (!fn->external)
-- 
2.39.3

Re: [RFC dwarves 3/3] btf_encoder: use function address to match ELF -> DWARF

[Eduard Zingerman]

On Thu, 2025-05-01 at 15:56 +0100, Alan Maguire wrote:
> Currently we use function names (or prefixes in the case of
> foo.isra.0) to match betwen ELF symtab entries and DWARF
> representations.  This can lead to wrong matches, especially
> where optimized function representations are concerned.  Instead
> sort and search ELF functions by address, and use the retrieved
> "struct function" address to carry out DWARF->ELF matches.
> 
> Note this is work-in-progress and many functions are missing as
> many functions do not have - or at least we have not retrieved -
> address info associated with their DWARF representations.
> 
> As things stand, there are exactly 1000 functions missing from
> BTF encoded using the address-based approach, since we skip functions
> for which we have no address info.  This approach actually adds
> 63 functions, so there are effectively 1063 missing functions.
> 
> 485 of these missing functions are __probestub functions we do not need, i.e.
> 
> [66116] FUNC '__probestub_xhci_setup_device' type_id=61452 linkage=static
> [61452] FUNC_PROTO '(anon)' ret_type_id=0 vlen=2
>         '__data' type_id=108
>         'vdev' type_id=44186
> 
> The real function is:
> 
> [147543] FUNC 'xhci_setup_device' type_id=147542 linkage=static
> [147542] FUNC_PROTO '(anon)' ret_type_id=21 vlen=4
>         'hcd' type_id=37057
>         'udev' type_id=37029
>         'setup' type_id=44209
>         'timeout_ms' type_id=9
> 
> This leaves us with a mismatch of 578 functions.  These include
> 140 missing __bpf_trace_ functions, which are definitely needed.
> 
> So perhaps we can fix up our DWARF representation to find associated
> addresses for some/all of these, but we may end up having to fall
> back to name-based association for some cases.
> 
> Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
> ---

Hi Alan,

The change makes sense to me, the code updates look reasonable.
Interestingly enough, I observe much smaller discrepancies, when using
llvm (version 19) for kernel compilation:
a. functions detected by dwarves/next but not detected with patch: 56
b. functions detected with this patch but not detected by dwarves/next: 70.

I only investigated group (b) and noticed two oddities, there are probably other.
- function "kmem_cache_release" is discarded from BTF by dwarves/next
  with the following log:

    kmem_cache_release (kmem_cache_release): skipping BTF encoding of function due to param type mismatch for param#1 s != k
  
  while it is present with this patch.
  Debugging a bit I can see that btf_encoder__save_func() is called
  for this function only once with patch but twice by dwarves/next.
  I suspect this happens because of how btf_encoder__encode_cu() looks after this patch:

    int btf_encoder__encode_cu(struct btf_encoder *encoder, struct cu *cu, struct conf_load *conf_load)
    {
        ...
        cu__for_each_function(cu, core_id, fn) {
            ...
            if (...) {
                ...
                func = btf_encoder__find_function(encoder, addr);
                ...
            } else {
                if (!fn->external)
                    continue;
            }
            if (!func)
                continue;
    
            err = btf_encoder__save_func(encoder, fn, func);
            if (err)
                goto out;
        }
        ...
    }

  Previously find function call used name: `btf_encoder__find_function(encoder, name, strlen(name))`,
  Because now it uses address specified in DWARF I suspect that:
  - The function is inlined or something and has different addresses
    encoded in DWARF but only one address encoded in ELF symbol table.
    (There is an inlined instance of the `kmem_cache_release` in DWARF).
  - `func` is NULL for one of two DWARF instances
    of this function and `btf_encoder__save_func` is not called.

- another oddity is about functions with aliases, here is an example
  from `thermal_netlink.c`:
  
    static int thermal_genl_event_threshold_up(struct param *p) { ... }
    ...
    int thermal_genl_event_threshold_down(struct param *p)
        __attribute__((alias("thermal_genl_event_threshold_up")));

  In symbol table it is encoded as:
  
    238180: ffffffff82b2d590   611 FUNC    GLOBAL DEFAULT     1 thermal_genl_event_threshold_down

  While in DWARF it is encoded as:

    DW_TAG_subprogram
      DW_AT_low_pc    (0xffffffff82b2d590)
      DW_AT_high_pc   (0xffffffff82b2d7f3)
      DW_AT_frame_base        (DW_OP_reg6 RBP)
      DW_AT_call_all_calls    (true)
      DW_AT_name      ("thermal_genl_event_threshold_up")
      DW_AT_decl_file ("/home/eddy/work/bpf-next/drivers/thermal/thermal_netlink.c")
      DW_AT_decl_line (263)
      DW_AT_prototyped        (true)
      DW_AT_type      (0x059ec28b "int")

  And I assume that it is not in the BTF generated by dwarves/next
  because of the same `btf_encoder__find_function` check.
  
Thanks,
Eduard

Re: [RFC dwarves 3/3] btf_encoder: use function address to match ELF -> DWARF

[Alan Maguire]

On 02/05/2025 00:25, Eduard Zingerman wrote:
> On Thu, 2025-05-01 at 15:56 +0100, Alan Maguire wrote:
>> Currently we use function names (or prefixes in the case of
>> foo.isra.0) to match betwen ELF symtab entries and DWARF
>> representations.  This can lead to wrong matches, especially
>> where optimized function representations are concerned.  Instead
>> sort and search ELF functions by address, and use the retrieved
>> "struct function" address to carry out DWARF->ELF matches.
>>
>> Note this is work-in-progress and many functions are missing as
>> many functions do not have - or at least we have not retrieved -
>> address info associated with their DWARF representations.
>>
>> As things stand, there are exactly 1000 functions missing from
>> BTF encoded using the address-based approach, since we skip functions
>> for which we have no address info.  This approach actually adds
>> 63 functions, so there are effectively 1063 missing functions.
>>
>> 485 of these missing functions are __probestub functions we do not need, i.e.
>>
>> [66116] FUNC '__probestub_xhci_setup_device' type_id=61452 linkage=static
>> [61452] FUNC_PROTO '(anon)' ret_type_id=0 vlen=2
>>         '__data' type_id=108
>>         'vdev' type_id=44186
>>
>> The real function is:
>>
>> [147543] FUNC 'xhci_setup_device' type_id=147542 linkage=static
>> [147542] FUNC_PROTO '(anon)' ret_type_id=21 vlen=4
>>         'hcd' type_id=37057
>>         'udev' type_id=37029
>>         'setup' type_id=44209
>>         'timeout_ms' type_id=9
>>
>> This leaves us with a mismatch of 578 functions.  These include
>> 140 missing __bpf_trace_ functions, which are definitely needed.
>>
>> So perhaps we can fix up our DWARF representation to find associated
>> addresses for some/all of these, but we may end up having to fall
>> back to name-based association for some cases.
>>
>> Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
>> ---
> 
> Hi Alan,
> 
> The change makes sense to me, the code updates look reasonable.
> Interestingly enough, I observe much smaller discrepancies, when using
> llvm (version 19) for kernel compilation:
> a. functions detected by dwarves/next but not detected with patch: 56
> b. functions detected with this patch but not detected by dwarves/next: 70.
> 

Interesting! Thanks for testing this! Good news the difference is much
smaller here.

> I only investigated group (b) and noticed two oddities, there are probably other.
> - function "kmem_cache_release" is discarded from BTF by dwarves/next
>   with the following log:
> 
>     kmem_cache_release (kmem_cache_release): skipping BTF encoding of function due to param type mismatch for param#1 s != k
>   
>   while it is present with this patch.
>   Debugging a bit I can see that btf_encoder__save_func() is called
>   for this function only once with patch but twice by dwarves/next.
>   I suspect this happens because of how btf_encoder__encode_cu() looks after this patch:
> 
>     int btf_encoder__encode_cu(struct btf_encoder *encoder, struct cu *cu, struct conf_load *conf_load)
>     {
>         ...
>         cu__for_each_function(cu, core_id, fn) {
>             ...
>             if (...) {
>                 ...
>                 func = btf_encoder__find_function(encoder, addr);
>                 ...
>             } else {
>                 if (!fn->external)
>                     continue;
>             }
>             if (!func)
>                 continue;
>     
>             err = btf_encoder__save_func(encoder, fn, func);
>             if (err)
>                 goto out;
>         }
>         ...
>     }
> 
>   Previously find function call used name: `btf_encoder__find_function(encoder, name, strlen(name))`,
>   Because now it uses address specified in DWARF I suspect that:
>   - The function is inlined or something and has different addresses
>     encoded in DWARF but only one address encoded in ELF symbol table.
>     (There is an inlined instance of the `kmem_cache_release` in DWARF).
>   - `func` is NULL for one of two DWARF instances
>     of this function and `btf_encoder__save_func` is not called.
> 

I think I smuggled in another change here; skipping ".cold"
compiler-generated "function" addresses when collecting ELF functions.
Is there a kmem_cache_release.cold.0 or similar perhaps? With this
change we don't consider it as a candidate match sice a .cold is really
the function, it's just an infrequently-executed cold path subportion.
Such .cold compiler-generated functions don't usually match the function
signature, but we were incorrectly comparing across them too; Alexei saw
a few kfuncs getting left out of BTF as a result of this.

> - another oddity is about functions with aliases, here is an example
>   from `thermal_netlink.c`:
>   
>     static int thermal_genl_event_threshold_up(struct param *p) { ... }
>     ...
>     int thermal_genl_event_threshold_down(struct param *p)
>         __attribute__((alias("thermal_genl_event_threshold_up")));
> 
>   In symbol table it is encoded as:
>   
>     238180: ffffffff82b2d590   611 FUNC    GLOBAL DEFAULT     1 thermal_genl_event_threshold_down
> 
>   While in DWARF it is encoded as:
> 
>     DW_TAG_subprogram
>       DW_AT_low_pc    (0xffffffff82b2d590)
>       DW_AT_high_pc   (0xffffffff82b2d7f3)
>       DW_AT_frame_base        (DW_OP_reg6 RBP)
>       DW_AT_call_all_calls    (true)
>       DW_AT_name      ("thermal_genl_event_threshold_up")
>       DW_AT_decl_file ("/home/eddy/work/bpf-next/drivers/thermal/thermal_netlink.c")
>       DW_AT_decl_line (263)
>       DW_AT_prototyped        (true)
>       DW_AT_type      (0x059ec28b "int")
> 
>   And I assume that it is not in the BTF generated by dwarves/next
>   because of the same `btf_encoder__find_function` check.
>

Ah, good to know. Will need to think about aliases too. I'm hoping to
improve the number of matches in gcc, but it may be the case that we
have to fall back on name-based matching if there is no address
associated with DWARF. Hopefully in such cases there are no ambiguities.
Will investigate further and see if we can do a bit better with
address-based matching first. Thanks again!

Alan

> Thanks,
> Eduard
>