Strange segfault on native Go binaries

Strange segfault on native Go binaries

[Otavio Salvador]

Hello all,

I am trying to debug a strange issue on native Go binaries. We're
seeing it on 'glide' for example.

If I build 'glide-native' and I go to its tmp/work/.../image/bin/glide
it works just fine. For example, running it we see the expected error
as I am running inside a clean environment without any extra tools:

otavio@yocto-env ~/.../build %
./tmp/work/x86_64-linux/glide-native/0.13.3-r0/build/bin/glide
[ERROR] Error getting version: exec: "go": executable file not found in $PATH.

Now, if we run it from the sysroot-native, it segfaults:

otavio@yocto-env ~/.../build %
./tmp/work/cortexa9hf-neon-oel-linux-gnueabi/updatehub/1.1.1-r0/recipe-sysroot-native/usr/bin/glide
[1]    26085 segmentation fault (core dumped)

And this segfault is also happening inside the build. I ran the
readelf on them and it seems to have been modified to use the
uninative libc loader but I am in doubt how if this is the culprit.
Can someone shed any light on this?

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9 9981-7854          Mobile: +1 (347) 903-9750

Re: [OE-core] Strange segfault on native Go binaries

[Khem Raj]

On Sat, Dec 12, 2020 at 3:52 PM Otavio Salvador
<otavio.salvador@ossystems.com.br> wrote:
>
> Hello all,
>
> I am trying to debug a strange issue on native Go binaries. We're
> seeing it on 'glide' for example.
>
> If I build 'glide-native' and I go to its tmp/work/.../image/bin/glide
> it works just fine. For example, running it we see the expected error
> as I am running inside a clean environment without any extra tools:
>
> otavio@yocto-env ~/.../build %
> ./tmp/work/x86_64-linux/glide-native/0.13.3-r0/build/bin/glide
> [ERROR] Error getting version: exec: "go": executable file not found in $PATH.
>
> Now, if we run it from the sysroot-native, it segfaults:
>
> otavio@yocto-env ~/.../build %
> ./tmp/work/cortexa9hf-neon-oel-linux-gnueabi/updatehub/1.1.1-r0/recipe-sysroot-native/usr/bin/glide
> [1]    26085 segmentation fault (core dumped)
>
> And this segfault is also happening inside the build. I ran the
> readelf on them and it seems to have been modified to use the
> uninative libc loader but I am in doubt how if this is the culprit.
> Can someone shed any light on this?
>

it seems relocation is not working in some cases. whats your host
distro, are you using gold linker by any chance.

> --
> Otavio Salvador                             O.S. Systems
> http://www.ossystems.com.br        http://code.ossystems.com.br
> Mobile: +55 (53) 9 9981-7854          Mobile: +1 (347) 903-9750
>
> 
>

Re: [OE-core] Strange segfault on native Go binaries

[Otavio Salvador]

Hello Khem,

Em sáb., 12 de dez. de 2020 às 23:30, Khem Raj <raj.khem@gmail.com> escreveu:
> On Sat, Dec 12, 2020 at 3:52 PM Otavio Salvador
> <otavio.salvador@ossystems.com.br> wrote:
> > And this segfault is also happening inside the build. I ran the
> > readelf on them and it seems to have been modified to use the
> > uninative libc loader but I am in doubt how if this is the culprit.
> > Can someone shed any light on this?
>
> it seems relocation is not working in some cases. whats your host
> distro, are you using gold linker by any chance.

Ubuntu 20.04 inside Docker

The features are: DISTRO_FEATURES = argp ipv4 ipv6 systemd ldconfig



-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9 9981-7854          Mobile: +1 (347) 903-9750

Re: [OE-core] Strange segfault on native Go binaries

[Khem Raj]

On Sun, Dec 13, 2020 at 6:52 AM Otavio Salvador
<otavio.salvador@ossystems.com.br> wrote:
>
> Hello Khem,
>
> Em sáb., 12 de dez. de 2020 às 23:30, Khem Raj <raj.khem@gmail.com> escreveu:
> > On Sat, Dec 12, 2020 at 3:52 PM Otavio Salvador
> > <otavio.salvador@ossystems.com.br> wrote:
> > > And this segfault is also happening inside the build. I ran the
> > > readelf on them and it seems to have been modified to use the
> > > uninative libc loader but I am in doubt how if this is the culprit.
> > > Can someone shed any light on this?
> >
> > it seems relocation is not working in some cases. whats your host
> > distro, are you using gold linker by any chance.
>
> Ubuntu 20.04 inside Docker
>
> The features are: DISTRO_FEATURES = argp ipv4 ipv6 systemd ldconfig
>

Does it happen outside docker ?

>
>
> --
> Otavio Salvador                             O.S. Systems
> http://www.ossystems.com.br        http://code.ossystems.com.br
> Mobile: +55 (53) 9 9981-7854          Mobile: +1 (347) 903-9750

Re: [OE-core] Strange segfault on native Go binaries

[Otavio Salvador]

Hello Khem,

Em seg., 14 de dez. de 2020 às 00:49, Khem Raj <raj.khem@gmail.com> escreveu:
> On Sun, Dec 13, 2020 at 6:52 AM Otavio Salvador
> <otavio.salvador@ossystems.com.br> wrote:
> > Em sáb., 12 de dez. de 2020 às 23:30, Khem Raj <raj.khem@gmail.com> escreveu:
> > > On Sat, Dec 12, 2020 at 3:52 PM Otavio Salvador
> > > <otavio.salvador@ossystems.com.br> wrote:
> > > > And this segfault is also happening inside the build. I ran the
> > > > readelf on them and it seems to have been modified to use the
> > > > uninative libc loader but I am in doubt how if this is the culprit.
> > > > Can someone shed any light on this?
> > >
> > > it seems relocation is not working in some cases. whats your host
> > > distro, are you using gold linker by any chance.
> >
> > Ubuntu 20.04 inside Docker
> >
> > The features are: DISTRO_FEATURES = argp ipv4 ipv6 systemd ldconfig
>
> Does it happen outside docker ?

Yes as it fails on our CI as well; however in this specific host I
cannot avoid the Docker use as my host is NixOS.

How can I debug it?

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9 9981-7854          Mobile: +1 (347) 903-9750

Re: [OE-core] Strange segfault on native Go binaries

[Konrad Weihmann]

I'm experiencing the same on plain Ubuntu 20.04 since 1.15.x update on 
master (dunfell works like a charm even without any hacks) - but also 
Ubuntu 18.04 and 16.04 based containers do show the same behavior.
This affects only the native variants.
As a partial workaround I disabled GO_DYNLINK for native - so I kind of 
second the impression that relocation is buggy in here.

On 14.12.20 12:31, Otavio Salvador wrote:
> Hello Khem,
> 
> Em seg., 14 de dez. de 2020 às 00:49, Khem Raj <raj.khem@gmail.com> escreveu:
>> On Sun, Dec 13, 2020 at 6:52 AM Otavio Salvador
>> <otavio.salvador@ossystems.com.br> wrote:
>>> Em sáb., 12 de dez. de 2020 às 23:30, Khem Raj <raj.khem@gmail.com> escreveu:
>>>> On Sat, Dec 12, 2020 at 3:52 PM Otavio Salvador
>>>> <otavio.salvador@ossystems.com.br> wrote:
>>>>> And this segfault is also happening inside the build. I ran the
>>>>> readelf on them and it seems to have been modified to use the
>>>>> uninative libc loader but I am in doubt how if this is the culprit.
>>>>> Can someone shed any light on this?
>>>>
>>>> it seems relocation is not working in some cases. whats your host
>>>> distro, are you using gold linker by any chance.
>>>
>>> Ubuntu 20.04 inside Docker
>>>
>>> The features are: DISTRO_FEATURES = argp ipv4 ipv6 systemd ldconfig
>>
>> Does it happen outside docker ?
> 
> Yes as it fails on our CI as well; however in this specific host I
> cannot avoid the Docker use as my host is NixOS.
> 
> How can I debug it?
> 
> 
> 
> 
> 

Re: [OE-core] Strange segfault on native Go binaries

[Otavio Salvador]

Hello Konrad,

Em seg., 14 de dez. de 2020 às 09:51, Konrad Weihmann
<kweihmann@outlook.com> escreveu:
> I'm experiencing the same on plain Ubuntu 20.04 since 1.15.x update on
> master (dunfell works like a charm even without any hacks) - but also
> Ubuntu 18.04 and 16.04 based containers do show the same behavior.
> This affects only the native variants.

Indeed; reverting to 1.14 "fixes" it here as well.

> As a partial workaround I disabled GO_DYNLINK for native - so I kind of
> second the impression that relocation is buggy in here.

Where you did it? I tried in the 'glide-native' case but it did not work out.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9 9981-7854          Mobile: +1 (347) 903-9750

Re: [OE-core] Strange segfault on native Go binaries

[Konrad Weihmann]

Had that at a bunch of my tools in meta-sca - as I couldn't rule out my 
own fault, I help myself with the static linking workaround

On 14.12.20 15:03, Otavio Salvador wrote:
> Hello Konrad,
> 
> Em seg., 14 de dez. de 2020 às 09:51, Konrad Weihmann
> <kweihmann@outlook.com> escreveu:
>> I'm experiencing the same on plain Ubuntu 20.04 since 1.15.x update on
>> master (dunfell works like a charm even without any hacks) - but also
>> Ubuntu 18.04 and 16.04 based containers do show the same behavior.
>> This affects only the native variants.
> 
> Indeed; reverting to 1.14 "fixes" it here as well.
> 
>> As a partial workaround I disabled GO_DYNLINK for native - so I kind of
>> second the impression that relocation is buggy in here.
> 
> Where you did it? I tried in the 'glide-native' case but it did not work out.
> 

Re: [OE-core] Strange segfault on native Go binaries

[Otavio Salvador]

Hello Konrad,

Em seg., 14 de dez. de 2020 às 11:06, Konrad Weihmann
<kweihmann@outlook.com> escreveu:
> Had that at a bunch of my tools in meta-sca - as I couldn't rule out my
> own fault, I help myself with the static linking workaround
>
> On 14.12.20 15:03, Otavio Salvador wrote:
> >> As a partial workaround I disabled GO_DYNLINK for native - so I kind of
> >> second the impression that relocation is buggy in here.
> >
> > Where you did it? I tried in the 'glide-native' case but it did not work out.

Adding:

# Disable dnylinking
GO_DYNLINK_class-native = "1"

"solves" it, indeed.

Richard and Khem, it seems to be related to Go itself. I'd like to
propose we add this to the class so  we workaround it for now until we
fix it properly.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9 9981-7854          Mobile: +1 (347) 903-9750

Re: [OE-core] Strange segfault on native Go binaries

[Konrad Weihmann]

On 14.12.20 16:25, Otavio Salvador wrote:
> Hello Konrad,
> 
> Em seg., 14 de dez. de 2020 às 11:06, Konrad Weihmann
> <kweihmann@outlook.com> escreveu:
>> Had that at a bunch of my tools in meta-sca - as I couldn't rule out my
>> own fault, I help myself with the static linking workaround
>>
>> On 14.12.20 15:03, Otavio Salvador wrote:
>>>> As a partial workaround I disabled GO_DYNLINK for native - so I kind of
>>>> second the impression that relocation is buggy in here.
>>>
>>> Where you did it? I tried in the 'glide-native' case but it did not work out.
> 
> Adding:
> 
> # Disable dnylinking
> GO_DYNLINK_class-native = "1"
> 
> "solves" it, indeed.
> 
> Richard and Khem, it seems to be related to Go itself. I'd like to
> propose we add this to the class so  we workaround it for now until we
> fix it properly.
> 

I don't know if that a good idea - I had at least one tool that wasn't 
even producing binaries with this hack (gosec to be exact) - so if that 
would be added, then only as a very temporary fix. I would expect some 
newly introduced *security* feature in go linking to cause it, but I 
haven't found the time yet to check that in detail

Re: [OE-core] Strange segfault on native Go binaries

[Khem Raj]

On Sat, Dec 12, 2020 at 08:52:34PM -0300, Otavio Salvador wrote:
> Hello all,
> 
> I am trying to debug a strange issue on native Go binaries. We're
> seeing it on 'glide' for example.
> 
> If I build 'glide-native' and I go to its tmp/work/.../image/bin/glide
> it works just fine. For example, running it we see the expected error
> as I am running inside a clean environment without any extra tools:
> 
> otavio@yocto-env ~/.../build %
> ./tmp/work/x86_64-linux/glide-native/0.13.3-r0/build/bin/glide
> [ERROR] Error getting version: exec: "go": executable file not found in $PATH.
> 
> Now, if we run it from the sysroot-native, it segfaults:
> 
> otavio@yocto-env ~/.../build %
> ./tmp/work/cortexa9hf-neon-oel-linux-gnueabi/updatehub/1.1.1-r0/recipe-sysroot-native/usr/bin/glide
> [1]    26085 segmentation fault (core dumped)
> 
> And this segfault is also happening inside the build. I ran the
> readelf on them and it seems to have been modified to use the
> uninative libc loader but I am in doubt how if this is the culprit.
> Can someone shed any light on this?
> 

It looks like a known regression in 1.15.5

There was a linker regression which crept into go compiler in 1.15.5
which was immediately patched and we also backported that to OE-core
however, recently we have switched to using go-binary-native to
bootstrap the toolchain which means the bug was still not plugged in the
native version even though we patched our go versions, we cant help the
prebuilts.

Luckily we now have .6 release which came out last week, which means we
can upgrade to it and that should take care of the segfault, I have
improvised on the patch which was sent for upgrades and sent a v2 [1]

Please try it out and let me know.

[1] https://lists.openembedded.org/g/openembedded-core/topic/patch_v2_go_update_1_15_5/78964072?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,78964072

> -- 
> Otavio Salvador                             O.S. Systems
> http://www.ossystems.com.br        http://code.ossystems.com.br
> Mobile: +55 (53) 9 9981-7854          Mobile: +1 (347) 903-9750

> 
> 
> 

Re: [OE-core] Strange segfault on native Go binaries

[Otavio Salvador]

Em seg., 14 de dez. de 2020 às 19:59, Khem Raj <raj.khem@gmail.com> escreveu:
> On Sat, Dec 12, 2020 at 08:52:34PM -0300, Otavio Salvador wrote:
> > Hello all,
> >
> > I am trying to debug a strange issue on native Go binaries. We're
> > seeing it on 'glide' for example.
> >
> > If I build 'glide-native' and I go to its tmp/work/.../image/bin/glide
> > it works just fine. For example, running it we see the expected error
> > as I am running inside a clean environment without any extra tools:
> >
> > otavio@yocto-env ~/.../build %
> > ./tmp/work/x86_64-linux/glide-native/0.13.3-r0/build/bin/glide
> > [ERROR] Error getting version: exec: "go": executable file not found in $PATH.
> >
> > Now, if we run it from the sysroot-native, it segfaults:
> >
> > otavio@yocto-env ~/.../build %
> > ./tmp/work/cortexa9hf-neon-oel-linux-gnueabi/updatehub/1.1.1-r0/recipe-sysroot-native/usr/bin/glide
> > [1]    26085 segmentation fault (core dumped)
> >
> > And this segfault is also happening inside the build. I ran the
> > readelf on them and it seems to have been modified to use the
> > uninative libc loader but I am in doubt how if this is the culprit.
> > Can someone shed any light on this?
> >
>
> It looks like a known regression in 1.15.5
>
> There was a linker regression which crept into go compiler in 1.15.5
> which was immediately patched and we also backported that to OE-core
> however, recently we have switched to using go-binary-native to
> bootstrap the toolchain which means the bug was still not plugged in the
> native version even though we patched our go versions, we cant help the
> prebuilts.
>
> Luckily we now have .6 release which came out last week, which means we
> can upgrade to it and that should take care of the segfault, I have
> improvised on the patch which was sent for upgrades and sent a v2 [1]
>
> Please try it out and let me know.
>
> [1] https://lists.openembedded.org/g/openembedded-core/topic/patch_v2_go_update_1_15_5/78964072?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,78964072

I tried it but it did not solve the problem. The behavior is the same
for this specific issue. The cgo issue documented was indeed fixed but
the relocation one is still happening :(


-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9 9981-7854          Mobile: +1 (347) 903-9750

Re: [OE-core] Strange segfault on native Go binaries

[Khem Raj]

On Mon, Dec 14, 2020 at 3:50 PM Otavio Salvador
<otavio.salvador@ossystems.com.br> wrote:
>
> Em seg., 14 de dez. de 2020 às 19:59, Khem Raj <raj.khem@gmail.com> escreveu:
> > On Sat, Dec 12, 2020 at 08:52:34PM -0300, Otavio Salvador wrote:
> > > Hello all,
> > >
> > > I am trying to debug a strange issue on native Go binaries. We're
> > > seeing it on 'glide' for example.
> > >
> > > If I build 'glide-native' and I go to its tmp/work/.../image/bin/glide
> > > it works just fine. For example, running it we see the expected error
> > > as I am running inside a clean environment without any extra tools:
> > >
> > > otavio@yocto-env ~/.../build %
> > > ./tmp/work/x86_64-linux/glide-native/0.13.3-r0/build/bin/glide
> > > [ERROR] Error getting version: exec: "go": executable file not found in $PATH.
> > >
> > > Now, if we run it from the sysroot-native, it segfaults:
> > >
> > > otavio@yocto-env ~/.../build %
> > > ./tmp/work/cortexa9hf-neon-oel-linux-gnueabi/updatehub/1.1.1-r0/recipe-sysroot-native/usr/bin/glide
> > > [1]    26085 segmentation fault (core dumped)
> > >
> > > And this segfault is also happening inside the build. I ran the
> > > readelf on them and it seems to have been modified to use the
> > > uninative libc loader but I am in doubt how if this is the culprit.
> > > Can someone shed any light on this?
> > >
> >
> > It looks like a known regression in 1.15.5
> >
> > There was a linker regression which crept into go compiler in 1.15.5
> > which was immediately patched and we also backported that to OE-core
> > however, recently we have switched to using go-binary-native to
> > bootstrap the toolchain which means the bug was still not plugged in the
> > native version even though we patched our go versions, we cant help the
> > prebuilts.
> >
> > Luckily we now have .6 release which came out last week, which means we
> > can upgrade to it and that should take care of the segfault, I have
> > improvised on the patch which was sent for upgrades and sent a v2 [1]
> >
> > Please try it out and let me know.
> >
> > [1] https://lists.openembedded.org/g/openembedded-core/topic/patch_v2_go_update_1_15_5/78964072?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,78964072
>
> I tried it but it did not solve the problem. The behavior is the same
> for this specific issue. The cgo issue documented was indeed fixed but
> the relocation one is still happening :(

The issue seems to be in stripped glide binary in sysroot but normal
unstripped binary is ok. So the issue is indeed orthogonal. I am
looking into what changes during strip
step here

>
>
> --
> Otavio Salvador                             O.S. Systems
> http://www.ossystems.com.br        http://code.ossystems.com.br
> Mobile: +55 (53) 9 9981-7854          Mobile: +1 (347) 903-9750

Re: [OE-core] Strange segfault on native Go binaries

[Khem Raj]

On 12/14/20 4:12 PM, Khem Raj wrote:
> On Mon, Dec 14, 2020 at 3:50 PM Otavio Salvador
> <otavio.salvador@ossystems.com.br> wrote:
>>
>> Em seg., 14 de dez. de 2020 às 19:59, Khem Raj <raj.khem@gmail.com> escreveu:
>>> On Sat, Dec 12, 2020 at 08:52:34PM -0300, Otavio Salvador wrote:
>>>> Hello all,
>>>>
>>>> I am trying to debug a strange issue on native Go binaries. We're
>>>> seeing it on 'glide' for example.
>>>>
>>>> If I build 'glide-native' and I go to its tmp/work/.../image/bin/glide
>>>> it works just fine. For example, running it we see the expected error
>>>> as I am running inside a clean environment without any extra tools:
>>>>
>>>> otavio@yocto-env ~/.../build %
>>>> ./tmp/work/x86_64-linux/glide-native/0.13.3-r0/build/bin/glide
>>>> [ERROR] Error getting version: exec: "go": executable file not found in $PATH.
>>>>
>>>> Now, if we run it from the sysroot-native, it segfaults:
>>>>
>>>> otavio@yocto-env ~/.../build %
>>>> ./tmp/work/cortexa9hf-neon-oel-linux-gnueabi/updatehub/1.1.1-r0/recipe-sysroot-native/usr/bin/glide
>>>> [1]    26085 segmentation fault (core dumped)
>>>>
>>>> And this segfault is also happening inside the build. I ran the
>>>> readelf on them and it seems to have been modified to use the
>>>> uninative libc loader but I am in doubt how if this is the culprit.
>>>> Can someone shed any light on this?
>>>>
>>>
>>> It looks like a known regression in 1.15.5
>>>
>>> There was a linker regression which crept into go compiler in 1.15.5
>>> which was immediately patched and we also backported that to OE-core
>>> however, recently we have switched to using go-binary-native to
>>> bootstrap the toolchain which means the bug was still not plugged in the
>>> native version even though we patched our go versions, we cant help the
>>> prebuilts.
>>>
>>> Luckily we now have .6 release which came out last week, which means we
>>> can upgrade to it and that should take care of the segfault, I have
>>> improvised on the patch which was sent for upgrades and sent a v2 [1]
>>>
>>> Please try it out and let me know.
>>>
>>> [1] https://lists.openembedded.org/g/openembedded-core/topic/patch_v2_go_update_1_15_5/78964072?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,78964072
>>
>> I tried it but it did not solve the problem. The behavior is the same
>> for this specific issue. The cgo issue documented was indeed fixed but
>> the relocation one is still happening :(
> 
> The issue seems to be in stripped glide binary in sysroot but normal
> unstripped binary is ok. So the issue is indeed orthogonal. I am
> looking into what changes during strip
> step here
> 

The problem seems to obscure editing of the glide binary which is 
resulting in a single byte in ELF header being modified between 
do_install and do_populate_sysroot

easy reproducer is

bitbake -cinstall glide-native

then make a copy of glide installed in image/ dir ( where do_install 
puts it ) then run

bitbake -cpopulate_sysroot glide-native

% cmp 
../../image/mnt/b/yoe/master/build/tmp/work/x86_64-linux/glide-native/0.13.3-r0/recipe-sysroot-native/usr/bin/glide 
glide

so they are identical.

and now compare the installed binary and the saved copy.

% cmp 
../../image/mnt/b/yoe/master/build/tmp/work/x86_64-linux/glide-native/0.13.3-r0/recipe-sysroot-native/usr/bin/glide 
glide
../../image/mnt/b/yoe/master/build/tmp/work/x86_64-linux/glide-native/0.13.3-r0/recipe-sysroot-native/usr/bin/glide 
glide differ: byte 57, line 1

it seems installed binary is fine until do_install but then gets edited 
during populate_sysroot

I have shunted many functions, including patchelf operations and the 
binary still gets modified in place

I wonder what other operations could be put under scanner here, Let me 
know if anyone has ideas..



>>
>>
>> --
>> Otavio Salvador                             O.S. Systems
>> http://www.ossystems.com.br        http://code.ossystems.com.br
>> Mobile: +55 (53) 9 9981-7854          Mobile: +1 (347) 903-9750

Re: [OE-core] Strange segfault on native Go binaries

[Richard Purdie]

On Mon, 2020-12-14 at 21:18 -0800, Khem Raj wrote:
> The problem seems to obscure editing of the glide binary which is 
> resulting in a single byte in ELF header being modified between 
> do_install and do_populate_sysroot
> 
> easy reproducer is
> 
> bitbake -cinstall glide-native
> 
> then make a copy of glide installed in image/ dir ( where do_install 
> puts it ) then run
> 
> bitbake -cpopulate_sysroot glide-native
> 
> % cmp 
> ../../image/mnt/b/yoe/master/build/tmp/work/x86_64-linux/glide-
> native/0.13.3-r0/recipe-sysroot-native/usr/bin/glide 
> glide
> 
> so they are identical.
> 
> and now compare the installed binary and the saved copy.
> 
> % cmp 
> ../../image/mnt/b/yoe/master/build/tmp/work/x86_64-linux/glide-
> native/0.13.3-r0/recipe-sysroot-native/usr/bin/glide 
> glide
> ../../image/mnt/b/yoe/master/build/tmp/work/x86_64-linux/glide-
> native/0.13.3-r0/recipe-sysroot-native/usr/bin/glide 
> glide differ: byte 57, line 1
> 
> it seems installed binary is fine until do_install but then gets
> edited 
> during populate_sysroot
> 
> I have shunted many functions, including patchelf operations and the 
> binary still gets modified in place
> 
> I wonder what other operations could be put under scanner here, Let
> me 
> know if anyone has ideas..

Is it from chrpath.bbclass? or the patchelf call in uninative.bbclass?

Cheers,

Richard