Using b43-fwcutter on Windows drivers

Using b43-fwcutter on Windows drivers

[João Paulo Rechi Vita]

Hello b43 developers,

Is it possible to use b43-fwcutter to extract firmware from Windows
drivers? The manpage says "Currently b43-fwcutter supports Apple MacOS
X, Microsoft Windows and Linux drivers, but keep in mind that
b43-fwcutter doesn't support all driver versions.", but looking at
fwcutter_list.h I don't see entries for the windows drivers. Are
offsets for windows drivers available somewhere?

Thanks,

......................................................................................
Jo?o Paulo Rechi Vita  |  +1.415.851.5778  |  Endless

Using b43-fwcutter on Windows drivers

[Larry Finger]

On 06/06/2017 08:02 PM, Jo?o Paulo Rechi Vita wrote:
> Hello b43 developers,
> 
> Is it possible to use b43-fwcutter to extract firmware from Windows
> drivers? The manpage says "Currently b43-fwcutter supports Apple MacOS
> X, Microsoft Windows and Linux drivers, but keep in mind that
> b43-fwcutter doesn't support all driver versions.", but looking at
> fwcutter_list.h I don't see entries for the windows drivers. Are
> offsets for windows drivers available somewhere?

The reason that you do not see any entries for Windows drivers is because those 
drivers are stripped such that we can get the offsets, but not the description 
for that particular firmware entry. One could go through each one and identify 
the correspondence with other firmware sources, but that would be very laborious 
and error prone. I have chosen not to try that approach.

What do you hope to gain from Windows firmware? Extracting firmware from the 
Linux drivers should yield better firmware if the two operating systems have 
different requirements.

Larry

Using b43-fwcutter on Windows drivers

[João Paulo Rechi Vita]

On Tue, Jun 6, 2017 at 6:41 PM, Larry Finger <Larry.Finger@lwfinger.net> wrote:
> On 06/06/2017 08:02 PM, Jo?o Paulo Rechi Vita wrote:
>>
>> Hello b43 developers,
>>
>> Is it possible to use b43-fwcutter to extract firmware from Windows
>> drivers? The manpage says "Currently b43-fwcutter supports Apple MacOS
>> X, Microsoft Windows and Linux drivers, but keep in mind that
>> b43-fwcutter doesn't support all driver versions.", but looking at
>> fwcutter_list.h I don't see entries for the windows drivers. Are
>> offsets for windows drivers available somewhere?
>
>
> The reason that you do not see any entries for Windows drivers is because
> those drivers are stripped such that we can get the offsets, but not the
> description for that particular firmware entry. One could go through each
> one and identify the correspondence with other firmware sources, but that
> would be very laborious and error prone. I have chosen not to try that
> approach.
>
> What do you hope to gain from Windows firmware? Extracting firmware from the
> Linux drivers should yield better firmware if the two operating systems have
> different requirements.
>

Thanks for your reply, Larry.

The idea is to help non-techy users on dual-boot installations, when
they have no other means to connect their Linux system to the
internet. On Endless we support a very specific Windows+Endless
dual-boot layout, so it is easy for us to find the Windows partition.
We want grab the Windows drivers and extract the firmware files from
them automatically when a compatible card is detected and the user is
offline.

I understand that there is some manual work involved on this, but are
there any pointers on how to get the offsets from the Windows drivers?
I could check if the extracted data matches anything we already know
from the wl firmware.

Also, you mention that firmware may be different due to different OS
requirements -- I expected the hw (thus, also the firmware) to behave
the same way despite which OS is running on the CPU. Maybe I'm being
naive here?

......................................................................................
Jo?o Paulo Rechi Vita  |  +1.415.851.5778  |  Endless

Using b43-fwcutter on Windows drivers

[Larry Finger]

On 06/07/2017 07:08 PM, Jo?o Paulo Rechi Vita wrote:
> On Tue, Jun 6, 2017 at 6:41 PM, Larry Finger <Larry.Finger@lwfinger.net> wrote:
>> On 06/06/2017 08:02 PM, Jo?o Paulo Rechi Vita wrote:
>>>
>>> Hello b43 developers,
>>>
>>> Is it possible to use b43-fwcutter to extract firmware from Windows
>>> drivers? The manpage says "Currently b43-fwcutter supports Apple MacOS
>>> X, Microsoft Windows and Linux drivers, but keep in mind that
>>> b43-fwcutter doesn't support all driver versions.", but looking at
>>> fwcutter_list.h I don't see entries for the windows drivers. Are
>>> offsets for windows drivers available somewhere?
>>
>>
>> The reason that you do not see any entries for Windows drivers is because
>> those drivers are stripped such that we can get the offsets, but not the
>> description for that particular firmware entry. One could go through each
>> one and identify the correspondence with other firmware sources, but that
>> would be very laborious and error prone. I have chosen not to try that
>> approach.
>>
>> What do you hope to gain from Windows firmware? Extracting firmware from the
>> Linux drivers should yield better firmware if the two operating systems have
>> different requirements.
>>
> 
> Thanks for your reply, Larry.
> 
> The idea is to help non-techy users on dual-boot installations, when
> they have no other means to connect their Linux system to the
> internet. On Endless we support a very specific Windows+Endless
> dual-boot layout, so it is easy for us to find the Windows partition.
> We want grab the Windows drivers and extract the firmware files from
> them automatically when a compatible card is detected and the user is
> offline.
> 
> I understand that there is some manual work involved on this, but are
> there any pointers on how to get the offsets from the Windows drivers?
> I could check if the extracted data matches anything we already know
> from the wl firmware.
> 
> Also, you mention that firmware may be different due to different OS
> requirements -- I expected the hw (thus, also the firmware) to behave
> the same way despite which OS is running on the CPU. Maybe I'm being
> naive here?

In general, the firmware should be the same, but as I have not seen any 
specifics on Windows drivers, there could be differences. The reverse 
engineering started with MIPS drivers for Linux. We never looked at any RE for 
Windows drivers.

A potential solution to your problem might be to use The method employed in 
http://www.lwfinger.com/b43-firmware/no_net_install_bcm43xx_firmware.tar.bz2. 
That tar file contains Broadcom object files with firmware for b43 and b43legacy 
with a script that extracts the firmware files. It meets Broadcom restrictions 
in that it only redistributes drivers, not firmware files. That particular tar 
file contains 5.100.138 b43 firmware; however, it would be easy to package a 
different driver, and adjust the extraction script accordingly. If it is not 
possible to package the tar file as part of your Windows installation, at least 
you could have your users download that file from a Windows boot, and then run 
the firmware extraction from a Linux boot.

The only thing I ask is that you host the download on your server. I do not 
promise to host those files in perpetuity. I wrote part of the script file, and 
you have my permission to use it as you please as long as you preserve my 
copyright. The rest of the script is part of the openSUSE package.

Larry

Using b43-fwcutter on Windows drivers

[João Paulo Rechi Vita]

On Thu, Jun 8, 2017 at 12:48 AM, Larry Finger <Larry.Finger@lwfinger.net> wrote:
> On 06/07/2017 07:08 PM, Jo?o Paulo Rechi Vita
> A potential solution to your problem might be to use The method employed in
> http://www.lwfinger.com/b43-firmware/no_net_install_bcm43xx_firmware.tar.bz2.
> That tar file contains Broadcom object files with firmware for b43 and
> b43legacy with a script that extracts the firmware files. It meets Broadcom
> restrictions in that it only redistributes drivers, not firmware files. That
> particular tar file contains 5.100.138 b43 firmware; however, it would be
> easy to package a different driver, and adjust the extraction script
> accordingly. If it is not possible to package the tar file as part of your
> Windows installation, at least you could have your users download that file
> from a Windows boot, and then run the firmware extraction from a Linux boot.
>
> The only thing I ask is that you host the download on your server. I do not
> promise to host those files in perpetuity. I wrote part of the script file,
> and you have my permission to use it as you please as long as you preserve
> my copyright. The rest of the script is part of the openSUSE package.
>

We already ship a similar script based on the debian package for
b43-fwcutter (with some adaptations for our rootfs layout), that would
download the drivers, extract and install firmware for the user, so we
could just ship the files as well. What I was trying to avoid is
shipping those files, and that's where extracting from the Windows
drivers would come in handy.

Thanks,

......................................................................................
Jo?o Paulo Rechi Vita  |  +1.415.851.5778  |  Endless

Using b43-fwcutter on Windows drivers

[Michael Büsch]

On Mon, 12 Jun 2017 12:28:09 -0400
Jo?o Paulo Rechi Vita <jprvita@endlessm.com> wrote:

> We already ship a similar script based on the debian package for
> b43-fwcutter (with some adaptations for our rootfs layout), that would
> download the drivers, extract and install firmware for the user, so we
> could just ship the files as well. What I was trying to avoid is
> shipping those files, and that's where extracting from the Windows
> drivers would come in handy.


Well, the problem is that it is very hard to get the names for the
firmware images found in the binary.

What you could do is try to find a Linux driver and a Windows driver
with exactly matching FW versions and then use the names from the Linux
driver to extract the blobs from the Windows driver.
It would be pretty easy to write a script that takes a Linux driver and
its matching extract-offsets and extract-names and then create the
corresponding offsets for the Windows binary.
Please feel free to create such a script and I would be happy to
include it in b43-fwcutter.

A script to auto-generate the extract-info from a Linux driver is
already included with b43-fwcutter.

-- 
Michael
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.infradead.org/pipermail/b43-dev/attachments/20170612/a26f16b7/attachment.sig>

Using b43-fwcutter on Windows drivers

[João Paulo Rechi Vita]

Hello again,

On Mon, Jun 12, 2017 at 12:59 PM, Michael B?sch <m@bues.ch> wrote:
> On Mon, 12 Jun 2017 12:28:09 -0400
> Jo?o Paulo Rechi Vita <jprvita@endlessm.com> wrote:
>
>> We already ship a similar script based on the debian package for
>> b43-fwcutter (with some adaptations for our rootfs layout), that would
>> download the drivers, extract and install firmware for the user, so we
>> could just ship the files as well. What I was trying to avoid is
>> shipping those files, and that's where extracting from the Windows
>> drivers would come in handy.
>
>
> Well, the problem is that it is very hard to get the names for the
> firmware images found in the binary.
>
> What you could do is try to find a Linux driver and a Windows driver
> with exactly matching FW versions and then use the names from the Linux
> driver to extract the blobs from the Windows driver.

We've been trying to do some binary grepping for the first 4 bytes of
the firmware files in the windows driver (grep -obUaP
"\x75\x01\x00\x00" BCMWL664.SYS), but although there are some matching
occurrences, after that the bytes start to differ. Any other
suggestions on trying to match the firmware files with windows
drivers?

> It would be pretty easy to write a script that takes a Linux driver and
> its matching extract-offsets and extract-names and then create the
> corresponding offsets for the Windows binary.
> Please feel free to create such a script and I would be happy to
> include it in b43-fwcutter.
>
> A script to auto-generate the extract-info from a Linux driver is
> already included with b43-fwcutter.
>

Yep, I have looked at that, and tried to objdump the windows driver,
but as you guys said, the symbol names are stripped out of it.

On a another topic, is there a license file available somewhere for
the wl MIPS drivers? I've downloaded the Asus/Netgear tarballs that
they were originally extracted from, but couldn't find a license for
the drivers there, just a general release notes file on the top dir,
and some un-related licenses for different projects used in the
router's rootfs image.

Thanks,

......................................................................................
Jo?o Paulo Rechi Vita  |  +1.415.851.5778  |  Endless

Using b43-fwcutter on Windows drivers

[Michael Büsch]

On Thu, 15 Jun 2017 18:21:40 -0400
Jo?o Paulo Rechi Vita <jprvita@endlessm.com> wrote:

> We've been trying to do some binary grepping for the first 4 bytes of
> the firmware files in the windows driver (grep -obUaP
> "\x75\x01\x00\x00" BCMWL664.SYS), but although there are some matching
> occurrences, after that the bytes start to differ. Any other
> suggestions on trying to match the firmware files with windows
> drivers?

Well, then you most likely have differing fw versions.
Try to find other binaries.

> On a another topic, is there a license file available somewhere for
> the wl MIPS drivers? I've downloaded the Asus/Netgear tarballs that
> they were originally extracted from, but couldn't find a license for
> the drivers there, just a general release notes file on the top dir,
> and some un-related licenses for different projects used in the
> router's rootfs image.

I don't know anything about the licensing.

-- 
Michael
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.infradead.org/pipermail/b43-dev/attachments/20170616/061c5758/attachment.sig>

Using b43-fwcutter on Windows drivers

[Robert McQueen]

Hi Larry,

On 08/06/17 05:48, Larry Finger wrote:
> A potential solution to your problem might be to use The method employed
> in
> http://www.lwfinger.com/b43-firmware/no_net_install_bcm43xx_firmware.tar.bz2.
> That tar file contains Broadcom object files with firmware for b43 and
> b43legacy with a script that extracts the firmware files. It meets
> Broadcom restrictions in that it only redistributes drivers, not
> firmware files. That particular tar file contains 5.100.138 b43
> firmware; however, it would be easy to package a different driver, and
> adjust the extraction script accordingly. If it is not possible to
> package the tar file as part of your Windows installation, at least you
> could have your users download that file from a Windows boot, and then
> run the firmware extraction from a Linux boot.

Thanks for this, I work with Joao and I've been investigating this
avenue of automatically fetching the drivers and then unpacking the
firmware when needed with a modprobe install hook, so we can get these
b43 cards working as seamlessly as possible in Endless when udev/etc
finds a supported card. I don't have any relevant hardware to test
unfortunately, but at least "on paper" it looks like it's working
nicely. Before we press on and enable it in the distro, I had a few
questions.

1. It looks like brcmsmac supports three devices which are all also
claimed by b43. I imagine that, because b43 usually requires manual
interaction with fwcutter, it's rare that users would get b43 working
unless they knew they needed it - so in a sense brcmsmac is the
"default" driver for these devices. However if we get b43 its firmware
automatically, does it become non-deterministic which driver will bind?

If so it feels to me (although I've got no evidence) that this could
expose some differences in behaviour / capabilities and could confuse
userland stuff like NetworkManager which uses the wireless extensions
extensively, so settings might not be applied correctly, etc. My
instinct is to just make this choice one way or another by either having
b43 not register for these three devices in our kernel, or blacklist
brcmsmac. If both drivers are available, do you have any suggestions
over which one would/should be used?

2. Your script uses the 5.100.138 driver, but 6.30.163.46 is available.
Is there any reason to not use the newer one? I had naively assumed
newer is better but there is some anecdotal evidence that some versions
work better for some cards - Arch Linux goes to the effort of packaging
two different firmware versions:
 https://wiki.archlinux.org/index.php/broadcom_wireless#b43

Is there any truth/merit in this? Which firmware would you use in an
optimal configuration? Do I need to switch the version based on the
device, or get the firmware for some cards from an older version and
some from a newer one?

3. I've not been able to find the MIPS wl_apsta drivers for download
directly from Broadcom, and every other source (wifi router source
drops, OpenWRT, etc) I've found seems not to have a LICENSE file along
with the .o. It seems most likely to me that the same Broadcom license
as applied to the PC STA wl drivers applies to these ones - ie like
http://metadata.ftp-master.debian.org/changelogs/non-free/b/broadcom-sta/broadcom-sta_6.30.223.271-7_copyright
- but I have no proof of that. Does anyone have any pointers to a more
definitive statement of the license the wl_apsta.o drivers are under?

> Larry

Cheers,
Rob

........................................................................

Robert McQueen  |  +1.415.413.4159  |  Endless <http://endlessm.com/>

Using b43-fwcutter on Windows drivers

[Larry Finger]

On 08/11/2017 07:48 AM, Robert McQueen wrote:
> Thanks for this, I work with Joao and I've been investigating this
> avenue of automatically fetching the drivers and then unpacking the
> firmware when needed with a modprobe install hook, so we can get these
> b43 cards working as seamlessly as possible in Endless when udev/etc
> finds a supported card. I don't have any relevant hardware to test
> unfortunately, but at least "on paper" it looks like it's working
> nicely. Before we press on and enable it in the distro, I had a few
> questions.
> 
> 1. It looks like brcmsmac supports three devices which are all also
> claimed by b43. I imagine that, because b43 usually requires manual
> interaction with fwcutter, it's rare that users would get b43 working
> unless they knew they needed it - so in a sense brcmsmac is the
> "default" driver for these devices. However if we get b43 its firmware
> automatically, does it become non-deterministic which driver will bind?

I would expect that the connection would not be deterministic for those devices. 
The safest thing would be for you to create a special kernel patch that would 
comment out the appropriate lines in the PCI device table found in 
drivers/bcma/host_pci.c. That would be the entries with device ID of 0x0576, 
0x4353, 0x4357, and 43224, although that last one does not seem to be in my 
table. With those changes, there would be no contention.

> If so it feels to me (although I've got no evidence) that this could
> expose some differences in behaviour / capabilities and could confuse
> userland stuff like NetworkManager which uses the wireless extensions
> extensively, so settings might not be applied correctly, etc. My
> instinct is to just make this choice one way or another by either having
> b43 not register for these three devices in our kernel, or blacklist
> brcmsmac. If both drivers are available, do you have any suggestions
> over which one would/should be used?

I suspect that the Broadcom version would be better. They have access to the 
data sheets for the device, which the b43 authors do not.

> 2. Your script uses the 5.100.138 driver, but 6.30.163.46 is available.
> Is there any reason to not use the newer one? I had naively assumed
> newer is better but there is some anecdotal evidence that some versions
> work better for some cards - Arch Linux goes to the effort of packaging
> two different firmware versions:
>   https://wiki.archlinux.org/index.php/broadcom_wireless#b43
> 
> Is there any truth/merit in this? Which firmware would you use in an
> optimal configuration? Do I need to switch the version based on the
> device, or get the firmware for some cards from an older version and
> some from a newer one?

The newer version may be better. We have no documentation regarding what changes 
may have been made. You get to make your choice.

> 3. I've not been able to find the MIPS wl_apsta drivers for download
> directly from Broadcom, and every other source (wifi router source
> drops, OpenWRT, etc) I've found seems not to have a LICENSE file along
> with the .o. It seems most likely to me that the same Broadcom license
> as applied to the PC STA wl drivers applies to these ones - ie like
> http://metadata.ftp-master.debian.org/changelogs/non-free/b/broadcom-sta/broadcom-sta_6.30.223.271-7_copyright
> - but I have no proof of that. Does anyone have any pointers to a more
> definitive statement of the license the wl_apsta.o drivers are under?

You will only find those wl_apsta drivers contained within the open-source 
packages for the various AP systems. That is where you would find any license 
statements made by Broadcom.

Larry