[Qemu-devel] NSIS - including DLL dependencies for Windows

[Qemu-devel] NSIS - including DLL dependencies for Windows

[Adam Baxter]

Hi,
How do I gather and include the required DLLs (SDL, zlib etc) in the NSIS
installer? (and also in a standalone build to be zipped up, but that's less
important)

I noticed the wiki doesn't really mention bundling dependencies anywhere.

The official unofficial windows builds are documented at
https://qemu.weilnetz.de/FAQ but this doesn't cover building the actual
installer.

Regards,
Adam

Re: [Qemu-devel] NSIS - including DLL dependencies for Windows

[Daniel P. Berrangé]

On Thu, Jan 31, 2019 at 02:45:13PM +1100, Adam Baxter wrote:
> Hi,
> How do I gather and include the required DLLs (SDL, zlib etc) in the NSIS
> installer? (and also in a standalone build to be zipped up, but that's less
> important)
> 
> I noticed the wiki doesn't really mention bundling dependencies anywhere.
> 
> The official unofficial windows builds are documented at
> https://qemu.weilnetz.de/FAQ but this doesn't cover building the actual
> installer.

Yeah, this is a pretty major ommision in QEMU's build rules for the NSIS
installer making it pretty much useless as is. We really need to expand
it so that it can resolve the dlls that qemu .exe's need, locate them
on the host and bundle them into the installer automatically.

It seems we can get a list of deps for an .exe using 

$ winedump  -j import /path/to/binary.exe  | grep offset | grep dll | awk '{print $3}'

We would need to run that for each .exe we're bundling. I think we might
need to also then run that recursively for each .dll to get the transitive
set of required dlls.

Even this is not entirely satisfactory though, as it is blindly assuming
that the only thing we need in the QEMU installer is the dlls themselves.
GTK at least has other resources used at runtime besides the dll that
should be bundled.

This kind of problem is why the GNOME project created the msitools project
for creating .msi installers, to relpace nsis. With msitools there is a
metadata file for each package listing all the files that need it needs
to have bundled into an installer:

  https://gitlab.gnome.org/GNOME/msitools/tree/master/data/wixl

This is not without its own downsides though. It bundles all the .wxi
files itself, with their content based on current mingw packages present
in the most recent Fedora release. This means it won't work on any other
distro :-(

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] NSIS - including DLL dependencies for Windows

[Adam Baxter]

My extremely embarassing script for getting the DLLs is as follows,
but at least it avoids a WINE dependency:

STAGING=/tmp/myqemu/
make install prefix=$STAGING
#todo: probably better to walk these dependencies in a loop, and not
use grep (but seriously, where's the mingw dumpbin)
FIRST=$(strings $STAGING/*.exe | grep '\.dll' | sort -u | xargs -I{}
readlink -e /usr/x86_64-w64-mingw32/sys-root/mingw/bin/{})
SECOND=$(for d in $FIRST; do strings $d | grep '\.dll' | sort -u |
xargs -I{} readlink -e /usr/x86_64-w64-mingw32/sys-root/mingw/bin/{};
done)
THIRD=$(for d in $SECOND; do strings $d | grep '\.dll' | sort -u |
xargs -I{} readlink -e /usr/x86_64-w64-mingw32/sys-root/mingw/bin/{};
done)
echo $FIRST $SECOND $THIRD | sed 's/ /\n/g' | sort -u | xargs -I{} cp
-v {} $STAGING

--Adam

On Thu, 31 Jan 2019 at 21:49, Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Thu, Jan 31, 2019 at 02:45:13PM +1100, Adam Baxter wrote:
> > Hi,
> > How do I gather and include the required DLLs (SDL, zlib etc) in the NSIS
> > installer? (and also in a standalone build to be zipped up, but that's less
> > important)
> >
> > I noticed the wiki doesn't really mention bundling dependencies anywhere.
> >
> > The official unofficial windows builds are documented at
> > https://qemu.weilnetz.de/FAQ but this doesn't cover building the actual
> > installer.
>
> Yeah, this is a pretty major ommision in QEMU's build rules for the NSIS
> installer making it pretty much useless as is. We really need to expand
> it so that it can resolve the dlls that qemu .exe's need, locate them
> on the host and bundle them into the installer automatically.
>
> It seems we can get a list of deps for an .exe using
>
> $ winedump  -j import /path/to/binary.exe  | grep offset | grep dll | awk '{print $3}'
>
> We would need to run that for each .exe we're bundling. I think we might
> need to also then run that recursively for each .dll to get the transitive
> set of required dlls.
>
> Even this is not entirely satisfactory though, as it is blindly assuming
> that the only thing we need in the QEMU installer is the dlls themselves.
> GTK at least has other resources used at runtime besides the dll that
> should be bundled.
>
> This kind of problem is why the GNOME project created the msitools project
> for creating .msi installers, to relpace nsis. With msitools there is a
> metadata file for each package listing all the files that need it needs
> to have bundled into an installer:
>
>   https://gitlab.gnome.org/GNOME/msitools/tree/master/data/wixl
>
> This is not without its own downsides though. It bundles all the .wxi
> files itself, with their content based on current mingw packages present
> in the most recent Fedora release. This means it won't work on any other
> distro :-(
>
> Regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] NSIS - including DLL dependencies for Windows

[Peter Maydell]

On Thu, 31 Jan 2019 at 10:50, Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Thu, Jan 31, 2019 at 02:45:13PM +1100, Adam Baxter wrote:
> > Hi,
> > How do I gather and include the required DLLs (SDL, zlib etc) in the NSIS
> > installer? (and also in a standalone build to be zipped up, but that's less
> > important)
> >
> > I noticed the wiki doesn't really mention bundling dependencies anywhere.
> >
> > The official unofficial windows builds are documented at
> > https://qemu.weilnetz.de/FAQ but this doesn't cover building the actual
> > installer.
>
> Yeah, this is a pretty major ommision in QEMU's build rules for the NSIS
> installer making it pretty much useless as is. We really need to expand
> it so that it can resolve the dlls that qemu .exe's need, locate them
> on the host and bundle them into the installer automatically.

I think my concern here potentially is a licensing one -- if we don't
actually ship installer binaries then that's fine for us, but an
installer script that automatically pulls in DLLs for glib and others
is a bit of a footgun for anybody that uses it since it will very likely
create an installer binary that is very difficult to ship to anybody
whilst remaining compliant with the licenses for our dependencies.

thanks
-- PMM

Re: [Qemu-devel] NSIS - including DLL dependencies for Windows

[Daniel P. Berrangé]

On Thu, Jan 31, 2019 at 11:19:04AM +0000, Peter Maydell wrote:
> On Thu, 31 Jan 2019 at 10:50, Daniel P. Berrangé <berrange@redhat.com> wrote:
> >
> > On Thu, Jan 31, 2019 at 02:45:13PM +1100, Adam Baxter wrote:
> > > Hi,
> > > How do I gather and include the required DLLs (SDL, zlib etc) in the NSIS
> > > installer? (and also in a standalone build to be zipped up, but that's less
> > > important)
> > >
> > > I noticed the wiki doesn't really mention bundling dependencies anywhere.
> > >
> > > The official unofficial windows builds are documented at
> > > https://qemu.weilnetz.de/FAQ but this doesn't cover building the actual
> > > installer.
> >
> > Yeah, this is a pretty major ommision in QEMU's build rules for the NSIS
> > installer making it pretty much useless as is. We really need to expand
> > it so that it can resolve the dlls that qemu .exe's need, locate them
> > on the host and bundle them into the installer automatically.
> 
> I think my concern here potentially is a licensing one -- if we don't
> actually ship installer binaries then that's fine for us, but an
> installer script that automatically pulls in DLLs for glib and others
> is a bit of a footgun for anybody that uses it since it will very likely
> create an installer binary that is very difficult to ship to anybody
> whilst remaining compliant with the licenses for our dependencies.

I don't see this situation as much different in terms of licensing risk
from building & hosting / distributing a docker container image. In both
cases the person is bundling up a huge set of dependancies and need to
ensure they are compliant with licenses. The only difference is the file
format that the bundled content is in.

Essentially it is relying on the fact that the distro used as input for
the docker image (or nsis installer) is providing full source + build
tool chain, as the way to achieve license compliance.

So if anything I'd say that providing a fully scripted way to pick up
the DLLs from distro packages is a better way to help users achieve
license compliance than letting them copy around DLLs manually from
unknown sources. 

I can understand that QEMU project doesn't want to get into the business
of providing binary packages as it certainly simplifies our life from the
POV of licensing by not having to think about it at all. I think it is
reasonable to allow users to have an easy way to build an installer for
their own usage though.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] NSIS - including DLL dependencies for Windows

[Stefan Weil]

On 31.01.19 12:28, Daniel P. Berrangé wrote:
> On Thu, Jan 31, 2019 at 11:19:04AM +0000, Peter Maydell wrote:
>> On Thu, 31 Jan 2019 at 10:50, Daniel P. Berrangé <berrange@redhat.com> wrote:
>>>
>>> On Thu, Jan 31, 2019 at 02:45:13PM +1100, Adam Baxter wrote:
>>>> Hi,
>>>> How do I gather and include the required DLLs (SDL, zlib etc) in the NSIS
>>>> installer? (and also in a standalone build to be zipped up, but that's less
>>>> important)
>>>>
>>>> I noticed the wiki doesn't really mention bundling dependencies anywhere.
>>>>
>>>> The official unofficial windows builds are documented at
>>>> https://qemu.weilnetz.de/FAQ but this doesn't cover building the actual
>>>> installer.
>>>
>>> Yeah, this is a pretty major ommision in QEMU's build rules for the NSIS
>>> installer making it pretty much useless as is. We really need to expand
>>> it so that it can resolve the dlls that qemu .exe's need, locate them
>>> on the host and bundle them into the installer automatically.


The current code already automatically includes all DLL files found in
dll/w32 or dll/w64 in the root source directory.

I had chosen that very simple solution because I am afraid that a
general rule to find the right DLL files for Windows native and cross
builds with all kinds of cross build hosts (Debian, Fedora, ...) is
impossible to maintain.

So currently every developer has to select and provide the DLL files
needed. That's how I‌ do it for my installers, too. Each time when I
update Debian or the Cygwin packages which I use for the cross build, I
also have to update the DLL set.

Regards,
Stefan Weil

Re: [Qemu-devel] NSIS - including DLL dependencies for Windows

[Daniel P. Berrangé]

On Thu, Jan 31, 2019 at 10:49:11AM +0000, Daniel P. Berrangé wrote:
> On Thu, Jan 31, 2019 at 02:45:13PM +1100, Adam Baxter wrote:
> > Hi,
> > How do I gather and include the required DLLs (SDL, zlib etc) in the NSIS
> > installer? (and also in a standalone build to be zipped up, but that's less
> > important)
> > 
> > I noticed the wiki doesn't really mention bundling dependencies anywhere.
> > 
> > The official unofficial windows builds are documented at
> > https://qemu.weilnetz.de/FAQ but this doesn't cover building the actual
> > installer.
> 
> Yeah, this is a pretty major ommision in QEMU's build rules for the NSIS
> installer making it pretty much useless as is. We really need to expand
> it so that it can resolve the dlls that qemu .exe's need, locate them
> on the host and bundle them into the installer automatically.
> 
> It seems we can get a list of deps for an .exe using 
> 
> $ winedump  -j import /path/to/binary.exe  | grep offset | grep dll | awk '{print $3}'

FYI, I re-discovered a way to do this without needing wine, just plain
binutils

  x86_64-w64-mingw32-objdump -p /path/to/binary.exe | grep "DLL Name" | awk '{print $3}'


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] NSIS - including DLL dependencies for Windows

[Stefan Weil]

On 13.02.19 10:32, Daniel P. Berrangé wrote:
> FYI, I re-discovered a way to do this without needing wine, just plain
> binutils
> 
>   x86_64-w64-mingw32-objdump -p /path/to/binary.exe | grep "DLL Name" | awk '{print $3}'


See also https://github.com/gsauthof/pe-util. The command line tool
peldd displays the DLLs recursively.

Regards
Stefan