[PATCH] firmware: Automatically pull missing FW files

[PATCH] firmware: Automatically pull missing FW files

[Tadeusz Struk]

Hi,
When a binary FW file is not present in the firmware dir then the build
will fail. This patch enables auto-download any missing FW file for both shipped
and external targets.
Generated against linux-next.

Signed-off-by: Tadeusz Struk <tadeusz.struk@intel.com>
---
 firmware/Makefile |   25 +++++++++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)

diff --git a/firmware/Makefile b/firmware/Makefile
index 5747417..3ad12f6 100644
--- a/firmware/Makefile
+++ b/firmware/Makefile
@@ -178,10 +178,31 @@ wordsize_deps := $(wildcard include/config/64bit.h include/config/32bit.h \
 		include/config/superh32.h include/config/superh64.h \
 		include/config/x86_32.h include/config/x86_64.h)
 
-$(patsubst %,$(obj)/%.gen.S, $(fw-shipped-y)): %: $(wordsize_deps)
+FW_URL := git.kernel.org/cgit/linux/kernel/git/firmware/linux-firmware.git/plain
+
+.NOTPARALLEL: get_missing_external $(patsubst %,$(obj)/%.gen.S, $(fw-external-y))
+.NOTPARALLEL: get_missing_shipped $(patsubst %,$(obj)/%.gen.S, $(fw-shipped-y))
+
+get_missing_shipped:
+	@for file in $(fw-shipped-y) $(fw-shipped-m); do \
+		if [ ! -f $(obj)/$$file ]; then \
+			wget $(FW_URL)/$$file -o /dev/null -O $(obj)/$$file; \
+		fi; \
+	done;
+
+get_missing_external:
+	@for file in $(fw-external-y); do \
+		if [ ! -f $(obj)/$$file ]; then \
+			wget $(FW_URL)/$$file -o /dev/null -O $(obj)/$$file; \
+		fi; \
+	done;
+
+$(patsubst %,$(obj)/%.gen.S, $(fw-shipped-y)): %: $(wordsize_deps) \
+		get_missing_shipped
 	$(call cmd,fwbin,$(patsubst %.gen.S,%,$@))
+
 $(patsubst %,$(obj)/%.gen.S, $(fw-external-y)): %: $(wordsize_deps) \
-		include/config/extra/firmware/dir.h
+		include/config/extra/firmware/dir.h get_missing_external
 	$(call cmd,fwbin,$(fwabs)/$(patsubst $(obj)/%.gen.S,%,$@))
 
 # The .o files depend on the binaries directly; the .S files don't.

Re: [PATCH] firmware: Automatically pull missing FW files

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 500 bytes --]

On Wed, 2014-08-20 at 11:21 -0700, Tadeusz Struk wrote:
> Hi,
> When a binary FW file is not present in the firmware dir then the
> build will fail. 

I'm not sure I understand. Precisely what fails?

I don't like this patch very much. We should be removing the legacy
firmware/ directory entirely, not patching it up.

Userspace is responsible for providing the firmware, and it should
generally come from an entirely separate checkout of the linux-firmware
repository.

-- 
dwmw2


[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5745 bytes --]

Re: [PATCH] firmware: Automatically pull missing FW files

[Tadeusz Struk]

Hi David,
On 08/20/2014 11:34 AM, David Woodhouse wrote:
> I'm not sure I understand. Precisely what fails?

I clone a subsystem, configure it to use
CONFIG_EXTRA_FIRMWARE="qat_895xcc.bin", type make && make install and get:

MK_FW   firmware/qat_895xxc.bin.gen.S
make[1]: *** No rule to make target `firmware/qat_895xxc.bin', needed by
`firmware/qat_895xxc.bin.gen.o'.  Stop.

I thought it might be useful if it would pull whatever FW it needs and
not just give up. It also might be useful if one wants to "refresh" the
FW binaries. In this case one can do rm firmware/*.[bin|ihex] && make &&
make install

> 
> I don't like this patch very much. We should be removing the legacy
> firmware/ directory entirely, not patching it up.
> 
> Userspace is responsible for providing the firmware, and it should
> generally come from an entirely separate checkout of the linux-firmware
> repository.

Yes, if you use udev helper. When you want to compile in the blobs to
your kernel it is needed in build time, right?
In both cases you need the binary anyway so you can copy it manually
from linux-firmware or use this nice feature to do it for you.
If you want to remove firmware/ directory entirely then this makefile
will be gone as well so what's the problem?

Re: [PATCH] firmware: Automatically pull missing FW files

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1013 bytes --]

On Wed, 20 Aug 2014 12:21:23 -0700, Tadeusz Struk said:
> Hi David,
> On 08/20/2014 11:34 AM, David Woodhouse wrote:
> > I'm not sure I understand. Precisely what fails?
>
> I clone a subsystem, configure it to use
> CONFIG_EXTRA_FIRMWARE="qat_895xcc.bin", type make && make install and get:
>
> MK_FW   firmware/qat_895xxc.bin.gen.S
> make[1]: *** No rule to make target `firmware/qat_895xxc.bin', needed by
> `firmware/qat_895xxc.bin.gen.o'.  Stop.

Is this what you were looking for?

config EXTRA_FIRMWARE_DIR
        string "Firmware blobs root directory"
        depends on EXTRA_FIRMWARE != ""
        default "firmware"
        help
          This option controls the directory in which the kernel build system
          looks for the firmware files listed in the EXTRA_FIRMWARE option.
          The default is firmware/ in the kernel source tree, but by changing
          this option you can point it elsewhere, such as /lib/firmware/ or
          some other directory containing the firmware files.



[-- Attachment #2: Type: application/pgp-signature, Size: 848 bytes --]

Re: [PATCH] firmware: Automatically pull missing FW files

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 847 bytes --]

On Wed, 2014-08-20 at 12:21 -0700, Tadeusz Struk wrote:
> Hi David,
> On 08/20/2014 11:34 AM, David Woodhouse wrote:
> > I'm not sure I understand. Precisely what fails?
> 
> I clone a subsystem, configure it to use
> CONFIG_EXTRA_FIRMWARE="qat_895xcc.bin", type make && make install and get:
> 
> MK_FW   firmware/qat_895xxc.bin.gen.S
> make[1]: *** No rule to make target `firmware/qat_895xxc.bin', needed by
> `firmware/qat_895xxc.bin.gen.o'.  Stop.

Can't you already just use CONFIG_EXTRA_FIRMWARE_DIR ? 

> Yes, if you use udev helper. When you want to compile in the blobs to
> your kernel it is needed in build time, right?

Yes. But seriously: don't do that. Let firmware get loaded from
userspace the normal way. Don't build kernel images that you can't distribute
because they include non-GPL parts.

-- 
dwmw2


[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5745 bytes --]

Re: [PATCH] firmware: Automatically pull missing FW files

[Tadeusz Struk]

On 08/20/2014 01:39 PM, David Woodhouse wrote:
> On Wed, 2014-08-20 at 12:21 -0700, Tadeusz Struk wrote:
>> Hi David,
>> On 08/20/2014 11:34 AM, David Woodhouse wrote:
>>> I'm not sure I understand. Precisely what fails?
>>
>> I clone a subsystem, configure it to use
>> CONFIG_EXTRA_FIRMWARE="qat_895xcc.bin", type make && make install and get:
>>
>> MK_FW   firmware/qat_895xxc.bin.gen.S
>> make[1]: *** No rule to make target `firmware/qat_895xxc.bin', needed by
>> `firmware/qat_895xxc.bin.gen.o'.  Stop.
> 
> Can't you already just use CONFIG_EXTRA_FIRMWARE_DIR ? 


What for? The point is that you don't have the FW binary file and you
have to download it, which can happen automatically and this is what
this patch does. The director where it will be stored doesn't really
matter here.

> 
>> Yes, if you use udev helper. When you want to compile in the blobs to
>> your kernel it is needed in build time, right?
> 
> Yes. But seriously: don't do that. Let firmware get loaded from
> userspace the normal way. Don't build kernel images that you can't distribute
> because they include non-GPL parts.
> 

Re: [PATCH] firmware: Automatically pull missing FW files

[Henrique de Moraes Holschuh]

On Wed, 20 Aug 2014, Tadeusz Struk wrote:
> What for? The point is that you don't have the FW binary file and you
> have to download it, which can happen automatically and this is what
> this patch does. The director where it will be stored doesn't really
> matter here.

"git pull" will at least tell you something is seriously wrong (if you know
the firmware tree should never require a forced update, that is).

I'd really appreciate if the firmware tree had signed tags or signed
commits, though.  Those at least provide decent end-to-end security (both at
rest, and while in transit) for data once it has been "git push"'d.

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh