[bnx2] [Regression 4.8] Driver loading fails without firmware

[bnx2] [Regression 4.8] Driver loading fails without firmware

[Paul Menzel]

Dear Linux folks,


A server with the Broadcom devices below, fails to load the drivers 
because of missing firmware.

 > $ lspci -nn
 > […]
> 01:00.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet [14e4:1639] (rev 20)
> 01:00.1 Ethernet controller [0200]: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet [14e4:1639] (rev 20)
> 02:00.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet [14e4:1639] (rev 20)
> 02:00.1 Ethernet controller [0200]: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet [14e4:1639] (rev 20)
> […]

Here are the error messages from Linux 4.8.4.

> [   10.365839] bnx2: QLogic bnx2 Gigabit Ethernet Driver v2.2.6 (January 29, 2014)
> [   10.366989] bnx2 0000:01:00.0: Direct firmware load for bnx2/bnx2-mips-09-6.2.1b.fw failed with error -2
> [   10.367384] bnx2: Can't load firmware file "bnx2/bnx2-mips-09-6.2.1b.fw"
> [   10.367719] bnx2: probe of 0000:01:00.0 failed with error -2
> [   10.368616] bnx2 0000:01:00.1: Direct firmware load for bnx2/bnx2-mips-09-6.2.1b.fw failed with error -2
> [   10.369015] bnx2: Can't load firmware file "bnx2/bnx2-mips-09-6.2.1b.fw"
> [   10.369292] bnx2: probe of 0000:01:00.1 failed with error -2
> [   10.370194] bnx2 0000:02:00.0: Direct firmware load for bnx2/bnx2-mips-09-6.2.1b.fw failed with error -2
> [   10.374460] bnx2: Can't load firmware file "bnx2/bnx2-mips-09-6.2.1b.fw"
> [   10.374743] bnx2: probe of 0000:02:00.0 failed with error -2
> [   10.375619] bnx2 0000:02:00.1: Direct firmware load for bnx2/bnx2-mips-09-6.2.1b.fw failed with error -2
> [   10.376024] bnx2: Can't load firmware file "bnx2/bnx2-mips-09-6.2.1b.fw"
> [   10.376303] bnx2: probe of 0000:02:00.1 failed with error -2
> [   10.376589] bnx2x: QLogic 5771x/578xx 10/20-Gigabit Ethernet Driver bnx2x 1.712.30-0 (2014/02/10)
> [   10.509323] bnx2: QLogic bnx2 Gigabit Ethernet Driver v2.2.6 (January 29, 2014)
> [   10.510467] bnx2 0000:01:00.0: Direct firmware load for bnx2/bnx2-mips-09-6.2.1b.fw failed with error -2
> [   10.510855] bnx2: Can't load firmware file "bnx2/bnx2-mips-09-6.2.1b.fw"
> [   10.511188] bnx2: probe of 0000:01:00.0 failed with error -2
> [   10.512087] bnx2 0000:01:00.1: Direct firmware load for bnx2/bnx2-mips-09-6.2.1b.fw failed with error -2
> [   10.512492] bnx2: Can't load firmware file "bnx2/bnx2-mips-09-6.2.1b.fw"
> [   10.512773] bnx2: probe of 0000:01:00.1 failed with error -2
> [   10.513648] bnx2 0000:02:00.0: Direct firmware load for bnx2/bnx2-mips-09-6.2.1b.fw failed with error -2
> [   10.517861] bnx2: Can't load firmware file "bnx2/bnx2-mips-09-6.2.1b.fw"
> [   10.518182] bnx2: probe of 0000:02:00.0 failed with error -2
> [   10.519071] bnx2 0000:02:00.1: Direct firmware load for bnx2/bnx2-mips-09-6.2.1b.fw failed with error -2
> [   10.519466] bnx2: Can't load firmware file "bnx2/bnx2-mips-09-6.2.1b.fw"
> [   10.519740] bnx2: probe of 0000:02:00.1 failed with error -2
> [   10.520039] bnx2x: QLogic 5771x/578xx 10/20-Gigabit Ethernet Driver bnx2x 1.712.30-0 (2014/02/10)

The system boots fine with the *same* initramfs and Linux 4.4.27, and 
Linux 4.7.10.

> $ dmesg | grep bnx # Linux 4.7.10
> [    9.875426] bnx2: QLogic bnx2 Gigabit Ethernet Driver v2.2.6 (January 29, 2014)
> [    9.876766] bnx2 0000:01:00.0 eth0: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem e6000000, IRQ 43, node addr 14:fe:b5:c6:81:01
> [    9.878065] bnx2 0000:01:00.1 eth1: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem e8000000, IRQ 44, node addr 14:fe:b5:c6:81:03
> [    9.879357] bnx2 0000:02:00.0 eth2: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem ea000000, IRQ 45, node addr 14:fe:b5:c6:81:05
> [    9.880630] bnx2 0000:02:00.1 eth3: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem ec000000, IRQ 46, node addr 14:fe:b5:c6:81:07
> [    9.881129] bnx2x: QLogic 5771x/578xx 10/20-Gigabit Ethernet Driver bnx2x 1.712.30-0 (2014/02/10)
> [   27.692521] bnx2 0000:01:00.0 net00: renamed from eth0
> [   27.702703] bnx2 0000:01:00.1 net01: renamed from eth1
> [   27.719779] bnx2 0000:02:00.0 net02: renamed from eth2
> [   27.768563] bnx2 0000:02:00.1 net03: renamed from eth3
> [   28.146396] bnx2 0000:01:00.0 net00: using MSIX
> [   31.294206] bnx2 0000:01:00.0 net00: NIC Copper Link is Up, 1000 Mbps full duplex

There were not many commits between Linux 4.7 and Linux 4.8.

Could the commit below cause this regression?

> $ git log --oneline v4.7...v4.8 -- drivers/net/ethernet/broadcom/bnx2.c
> 3e1be7a bnx2: Reset device during driver initialization
> commit 3e1be7ad2d38c6bd6aeef96df9bd0a7822f4e51c
> Author: Baoquan He <bhe@redhat.com>
> Date:   Fri Sep 9 22:43:12 2016 +0800
>
>     bnx2: Reset device during driver initialization
>
>     When system enters into kdump kernel because of kernel panic, it won't
>     shutdown devices. On-flight DMA will continue transferring data until
>     device driver initializes. All devices are supposed to reset during
>     driver initialization. And this property is used to fix the kdump
>     failure in system with intel iommu. Other systems with hardware iommu
>     should be similar. Please check commit 091d42e ("iommu/vt-d: Copy
>     translation tables from old kernel") and those commits around.
>
>     But bnx2 driver doesn't reset device during driver initialization. The
>     device resetting is deferred to net device up stage. This will cause
>     hardware iommu handling failure on bnx2 device. And its resetting relies
>     on firmware. So in this patch move the firmware requesting code to earlier
>     bnx2_init_one(), then next call bnx2_reset_chip to reset device.
>
>     Signed-off-by: Baoquan He <bhe@redhat.com>
>     Signed-off-by: David S. Miller <davem@davemloft.net>
>
> diff --git a/drivers/net/ethernet/broadcom/bnx2.c b/drivers/net/ethernet/broadcom/bnx2.c
> index 8fc3f3c..505ceaf 100644
> --- a/drivers/net/ethernet/broadcom/bnx2.c
> +++ b/drivers/net/ethernet/broadcom/bnx2.c
> @@ -6356,10 +6356,6 @@ bnx2_open(struct net_device *dev)
>         struct bnx2 *bp = netdev_priv(dev);
>         int rc;
>
> -       rc = bnx2_request_firmware(bp);
> -       if (rc < 0)
> -               goto out;
> -
>         netif_carrier_off(dev);
>
>         bnx2_disable_int(bp);
> @@ -6428,7 +6424,6 @@ bnx2_open(struct net_device *dev)
>         bnx2_free_irq(bp);
>         bnx2_free_mem(bp);
>         bnx2_del_napi(bp);
> -       bnx2_release_firmware(bp);
>         goto out;
>  }
>
> @@ -8575,6 +8570,12 @@ bnx2_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
>
>         pci_set_drvdata(pdev, dev);
>
> +       rc = bnx2_request_firmware(bp);
> +       if (rc < 0)
> +               goto error;
> +
> +
> +       bnx2_reset_chip(bp, BNX2_DRV_MSG_CODE_RESET);
>         memcpy(dev->dev_addr, bp->mac_addr, ETH_ALEN);
>
>         dev->hw_features = NETIF_F_IP_CSUM | NETIF_F_SG |
> @@ -8607,6 +8608,7 @@ bnx2_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
>         return 0;
>
>  error:
> +       bnx2_release_firmware(bp);
>         pci_iounmap(pdev, bp->regview);
>         pci_release_regions(pdev);
>         pci_disable_device(pdev);


Kind regards,

Paul

Re: [bnx2] [Regression 4.8] Driver loading fails without firmware

[Rick Jones]

On 10/25/2016 07:33 AM, Paul Menzel wrote:
> Dear Linux folks,
>
> A server with the Broadcom devices below, fails to load the drivers
> because of missing firmware.

I have run into the same sort of issue from time to time when going to a 
newer kernel.  A newer version of the driver wants a newer version of 
the firmware.  Usually, finding a package "out there" with the newer 
version of the firmware, and installing it onto the system is sufficient.

happy benchmarking,

rick jones

Re: [bnx2] [Regression 4.8] Driver loading fails without firmware

[Paul Menzel]

Dear Rick,


On 10/25/16 17:24, Rick Jones wrote:
> On 10/25/2016 07:33 AM, Paul Menzel wrote:

>> A server with the Broadcom devices below, fails to load the drivers
>> because of missing firmware.
>
> I have run into the same sort of issue from time to time when going to a
> newer kernel.  A newer version of the driver wants a newer version of
> the firmware.  Usually, finding a package "out there" with the newer
> version of the firmware, and installing it onto the system is sufficient.

To my knowledge, the firmware files haven’t changed since years [1].


Kind regards,

Paul


[1] 
https://git.kernel.org/cgit/linux/kernel/git/firmware/linux-firmware.git/tree/bnx2

Re: [bnx2] [Regression 4.8] Driver loading fails without firmware

[Rick Jones]

On 10/25/2016 08:31 AM, Paul Menzel wrote:
> To my knowledge, the firmware files haven’t changed since years [1].

Indeed - it looks like I read "bnx2" and thought "bnx2x"  Must remember 
to hold-off on replying until after the morning orange juice is consumed :)

rick

Re: [bnx2] [Regression 4.8] Driver loading fails without firmware

[Paul Menzel]

Dear Baoquan, Linux folks,


I am sorry for Thunderbird’s autowrapping messing up the message formatting.


On 10/25/16 16:33, Paul Menzel wrote:

> A server with the Broadcom devices below, fails to load the drivers
> because of missing firmware.
>
>> $ lspci -nn
>> […]
>> 01:00.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme II
>> BCM5709 Gigabit Ethernet [14e4:1639] (rev 20)
>> 01:00.1 Ethernet controller [0200]: Broadcom Corporation NetXtreme II
>> BCM5709 Gigabit Ethernet [14e4:1639] (rev 20)
>> 02:00.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme II
>> BCM5709 Gigabit Ethernet [14e4:1639] (rev 20)
>> 02:00.1 Ethernet controller [0200]: Broadcom Corporation NetXtreme II
>> BCM5709 Gigabit Ethernet [14e4:1639] (rev 20)
>> […]
>
> Here are the error messages from Linux 4.8.4.
>
>> [   10.365839] bnx2: QLogic bnx2 Gigabit Ethernet Driver v2.2.6
>> (January 29, 2014)
>> [   10.366989] bnx2 0000:01:00.0: Direct firmware load for
>> bnx2/bnx2-mips-09-6.2.1b.fw failed with error -2
>> [   10.367384] bnx2: Can't load firmware file
>> "bnx2/bnx2-mips-09-6.2.1b.fw"
>> [   10.367719] bnx2: probe of 0000:01:00.0 failed with error -2
>> [   10.368616] bnx2 0000:01:00.1: Direct firmware load for
>> bnx2/bnx2-mips-09-6.2.1b.fw failed with error -2
>> [   10.369015] bnx2: Can't load firmware file
>> "bnx2/bnx2-mips-09-6.2.1b.fw"
>> [   10.369292] bnx2: probe of 0000:01:00.1 failed with error -2
>> [   10.370194] bnx2 0000:02:00.0: Direct firmware load for
>> bnx2/bnx2-mips-09-6.2.1b.fw failed with error -2
>> [   10.374460] bnx2: Can't load firmware file
>> "bnx2/bnx2-mips-09-6.2.1b.fw"
>> [   10.374743] bnx2: probe of 0000:02:00.0 failed with error -2
>> [   10.375619] bnx2 0000:02:00.1: Direct firmware load for
>> bnx2/bnx2-mips-09-6.2.1b.fw failed with error -2
>> [   10.376024] bnx2: Can't load firmware file
>> "bnx2/bnx2-mips-09-6.2.1b.fw"
>> [   10.376303] bnx2: probe of 0000:02:00.1 failed with error -2
>> [   10.376589] bnx2x: QLogic 5771x/578xx 10/20-Gigabit Ethernet Driver
>> bnx2x 1.712.30-0 (2014/02/10)
>> [   10.509323] bnx2: QLogic bnx2 Gigabit Ethernet Driver v2.2.6
>> (January 29, 2014)
>> [   10.510467] bnx2 0000:01:00.0: Direct firmware load for
>> bnx2/bnx2-mips-09-6.2.1b.fw failed with error -2
>> [   10.510855] bnx2: Can't load firmware file
>> "bnx2/bnx2-mips-09-6.2.1b.fw"
>> [   10.511188] bnx2: probe of 0000:01:00.0 failed with error -2
>> [   10.512087] bnx2 0000:01:00.1: Direct firmware load for
>> bnx2/bnx2-mips-09-6.2.1b.fw failed with error -2
>> [   10.512492] bnx2: Can't load firmware file
>> "bnx2/bnx2-mips-09-6.2.1b.fw"
>> [   10.512773] bnx2: probe of 0000:01:00.1 failed with error -2
>> [   10.513648] bnx2 0000:02:00.0: Direct firmware load for
>> bnx2/bnx2-mips-09-6.2.1b.fw failed with error -2
>> [   10.517861] bnx2: Can't load firmware file
>> "bnx2/bnx2-mips-09-6.2.1b.fw"
>> [   10.518182] bnx2: probe of 0000:02:00.0 failed with error -2
>> [   10.519071] bnx2 0000:02:00.1: Direct firmware load for
>> bnx2/bnx2-mips-09-6.2.1b.fw failed with error -2
>> [   10.519466] bnx2: Can't load firmware file
>> "bnx2/bnx2-mips-09-6.2.1b.fw"
>> [   10.519740] bnx2: probe of 0000:02:00.1 failed with error -2
>> [   10.520039] bnx2x: QLogic 5771x/578xx 10/20-Gigabit Ethernet Driver
>> bnx2x 1.712.30-0 (2014/02/10)
>
> The system boots fine with the *same* initramfs and Linux 4.4.27, and
> Linux 4.7.10.
>
>> $ dmesg | grep bnx # Linux 4.7.10
>> [    9.875426] bnx2: QLogic bnx2 Gigabit Ethernet Driver v2.2.6
>> (January 29, 2014)
>> [    9.876766] bnx2 0000:01:00.0 eth0: Broadcom NetXtreme II BCM5709
>> 1000Base-T (C0) PCI Express found at mem e6000000, IRQ 43, node addr
>> 14:fe:b5:c6:81:01
>> [    9.878065] bnx2 0000:01:00.1 eth1: Broadcom NetXtreme II BCM5709
>> 1000Base-T (C0) PCI Express found at mem e8000000, IRQ 44, node addr
>> 14:fe:b5:c6:81:03
>> [    9.879357] bnx2 0000:02:00.0 eth2: Broadcom NetXtreme II BCM5709
>> 1000Base-T (C0) PCI Express found at mem ea000000, IRQ 45, node addr
>> 14:fe:b5:c6:81:05
>> [    9.880630] bnx2 0000:02:00.1 eth3: Broadcom NetXtreme II BCM5709
>> 1000Base-T (C0) PCI Express found at mem ec000000, IRQ 46, node addr
>> 14:fe:b5:c6:81:07
>> [    9.881129] bnx2x: QLogic 5771x/578xx 10/20-Gigabit Ethernet Driver
>> bnx2x 1.712.30-0 (2014/02/10)
>> [   27.692521] bnx2 0000:01:00.0 net00: renamed from eth0
>> [   27.702703] bnx2 0000:01:00.1 net01: renamed from eth1
>> [   27.719779] bnx2 0000:02:00.0 net02: renamed from eth2
>> [   27.768563] bnx2 0000:02:00.1 net03: renamed from eth3
>> [   28.146396] bnx2 0000:01:00.0 net00: using MSIX
>> [   31.294206] bnx2 0000:01:00.0 net00: NIC Copper Link is Up, 1000
>> Mbps full duplex
>
> There were not many commits between Linux 4.7 and Linux 4.8.
>
> Could the commit below cause this regression?
>
>> $ git log --oneline v4.7...v4.8 -- drivers/net/ethernet/broadcom/bnx2.c
>> 3e1be7a bnx2: Reset device during driver initialization
>> commit 3e1be7ad2d38c6bd6aeef96df9bd0a7822f4e51c
>> Author: Baoquan He <bhe@redhat.com>
>> Date:   Fri Sep 9 22:43:12 2016 +0800
>>
>>     bnx2: Reset device during driver initialization
>>
>>     When system enters into kdump kernel because of kernel panic, it
>> won't
>>     shutdown devices. On-flight DMA will continue transferring data until
>>     device driver initializes. All devices are supposed to reset during
>>     driver initialization. And this property is used to fix the kdump
>>     failure in system with intel iommu. Other systems with hardware iommu
>>     should be similar. Please check commit 091d42e ("iommu/vt-d: Copy
>>     translation tables from old kernel") and those commits around.
>>
>>     But bnx2 driver doesn't reset device during driver initialization.
>> The
>>     device resetting is deferred to net device up stage. This will cause
>>     hardware iommu handling failure on bnx2 device. And its resetting
>> relies
>>     on firmware. So in this patch move the firmware requesting code to
>> earlier
>>     bnx2_init_one(), then next call bnx2_reset_chip to reset device.
>>
>>     Signed-off-by: Baoquan He <bhe@redhat.com>
>>     Signed-off-by: David S. Miller <davem@davemloft.net>
>>
>> diff --git a/drivers/net/ethernet/broadcom/bnx2.c
>> b/drivers/net/ethernet/broadcom/bnx2.c
>> index 8fc3f3c..505ceaf 100644
>> --- a/drivers/net/ethernet/broadcom/bnx2.c
>> +++ b/drivers/net/ethernet/broadcom/bnx2.c
>> @@ -6356,10 +6356,6 @@ bnx2_open(struct net_device *dev)
>>         struct bnx2 *bp = netdev_priv(dev);
>>         int rc;
>>
>> -       rc = bnx2_request_firmware(bp);
>> -       if (rc < 0)
>> -               goto out;
>> -
>>         netif_carrier_off(dev);
>>
>>         bnx2_disable_int(bp);
>> @@ -6428,7 +6424,6 @@ bnx2_open(struct net_device *dev)
>>         bnx2_free_irq(bp);
>>         bnx2_free_mem(bp);
>>         bnx2_del_napi(bp);
>> -       bnx2_release_firmware(bp);
>>         goto out;
>>  }
>>
>> @@ -8575,6 +8570,12 @@ bnx2_init_one(struct pci_dev *pdev, const
>> struct pci_device_id *ent)
>>
>>         pci_set_drvdata(pdev, dev);
>>
>> +       rc = bnx2_request_firmware(bp);
>> +       if (rc < 0)
>> +               goto error;
>> +
>> +
>> +       bnx2_reset_chip(bp, BNX2_DRV_MSG_CODE_RESET);
>>         memcpy(dev->dev_addr, bp->mac_addr, ETH_ALEN);
>>
>>         dev->hw_features = NETIF_F_IP_CSUM | NETIF_F_SG |
>> @@ -8607,6 +8608,7 @@ bnx2_init_one(struct pci_dev *pdev, const struct
>> pci_device_id *ent)
>>         return 0;
>>
>>  error:
>> +       bnx2_release_firmware(bp);
>>         pci_iounmap(pdev, bp->regview);
>>         pci_release_regions(pdev);
>>         pci_disable_device(pdev);

Baoquan, could you please fix this regression. My suggestion is, that 
you add the old code back, but check if the firmware has been loaded. If 
it hasn’t, load it again.

That way, people can update their Linux kernel, and it continues working 
without changing the initramfs, or anything else.


Kind regards,

Paul

Re: [bnx2] [Regression 4.8] Driver loading fails without firmware

[Baoquan He]

Hi Paul,

Sorry for this.


On 10/26/16 at 12:31pm, Paul Menzel wrote:
> > >         dev->hw_features = NETIF_F_IP_CSUM | NETIF_F_SG |
> > > @@ -8607,6 +8608,7 @@ bnx2_init_one(struct pci_dev *pdev, const struct
> > > pci_device_id *ent)
> > >         return 0;
> > > 
> > >  error:
> > > +       bnx2_release_firmware(bp);
> > >         pci_iounmap(pdev, bp->regview);
> > >         pci_release_regions(pdev);
> > >         pci_disable_device(pdev);
> 
> Baoquan, could you please fix this regression. My suggestion is, that you
> add the old code back, but check if the firmware has been loaded. If it
> hasn’t, load it again.
> 
> That way, people can update their Linux kernel, and it continues working
> without changing the initramfs, or anything else.

I saw your mail but I am also not familiar with bnx2 driver. As the
commit log says I just tried to make bnx2 driver reset itself earlier.

So you did a git bisect and found this commit caused the regression,
right? If yes, and network developers have no action, I will look into
the code and see if I have idea to fix it.

Thanks
Baoquan

Re: [bnx2] [Regression 4.8] Driver loading fails without firmware

[Paul Menzel]

[-- Attachment #1: Type: text/plain, Size: 1536 bytes --]

Dear Baoquan,


On 10/26/16 14:00, Baoquan He wrote:

> On 10/26/16 at 12:31pm, Paul Menzel wrote:
>>>>         dev->hw_features = NETIF_F_IP_CSUM | NETIF_F_SG |
>>>> @@ -8607,6 +8608,7 @@ bnx2_init_one(struct pci_dev *pdev, const struct
>>>> pci_device_id *ent)
>>>>         return 0;
>>>>
>>>>  error:
>>>> +       bnx2_release_firmware(bp);
>>>>         pci_iounmap(pdev, bp->regview);
>>>>         pci_release_regions(pdev);
>>>>         pci_disable_device(pdev);
>>
>> Baoquan, could you please fix this regression. My suggestion is, that you
>> add the old code back, but check if the firmware has been loaded. If it
>> hasn’t, load it again.
>>
>> That way, people can update their Linux kernel, and it continues working
>> without changing the initramfs, or anything else.
>
> I saw your mail but I am also not familiar with bnx2 driver. As the
> commit log says I just tried to make bnx2 driver reset itself earlier.
>
> So you did a git bisect and found this commit caused the regression,
> right? If yes, and network developers have no action, I will look into
> the code and see if I have idea to fix it.

Well, I looked through the commits and found that one, which would 
explain the changed behavior.

To be sure, and to follow your request, I took Linux 4.8.4 and reverted 
your commit (attached). Then I deleted the firmware again from the 
initramfs, and rebooted. The devices showed up just fine as before.

So to summarize, the commit is indeed the culprit.

Thank you for looking into this.


Kind regards,

Paul

[-- Attachment #2: 0001-Revert-bnx2-Reset-device-during-driver-initializatio.patch --]
[-- Type: text/x-patch, Size: 1579 bytes --]

>From 61b8dac8796343a797858b4a2eb0a59a0cfcd735 Mon Sep 17 00:00:00 2001
From: Paul Menzel <pmenzel@molgen.mpg.de>
Date: Thu, 27 Oct 2016 11:34:52 +0200
Subject: [PATCH] Revert "bnx2: Reset device during driver initialization"

This reverts commit 3e1be7ad2d38c6bd6aeef96df9bd0a7822f4e51c.
---
 drivers/net/ethernet/broadcom/bnx2.c | 12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)

diff --git a/drivers/net/ethernet/broadcom/bnx2.c b/drivers/net/ethernet/broadcom/bnx2.c
index 27f11a5..ecd357d 100644
--- a/drivers/net/ethernet/broadcom/bnx2.c
+++ b/drivers/net/ethernet/broadcom/bnx2.c
@@ -6356,6 +6356,10 @@ bnx2_open(struct net_device *dev)
 	struct bnx2 *bp = netdev_priv(dev);
 	int rc;
 
+	rc = bnx2_request_firmware(bp);
+	if (rc < 0)
+		goto out;
+
 	netif_carrier_off(dev);
 
 	bnx2_disable_int(bp);
@@ -6424,6 +6428,7 @@ bnx2_open(struct net_device *dev)
 	bnx2_free_irq(bp);
 	bnx2_free_mem(bp);
 	bnx2_del_napi(bp);
+	bnx2_release_firmware(bp);
 	goto out;
 }
 
@@ -8570,12 +8575,6 @@ bnx2_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 
 	pci_set_drvdata(pdev, dev);
 
-	rc = bnx2_request_firmware(bp);
-	if (rc < 0)
-		goto error;
-
-
-	bnx2_reset_chip(bp, BNX2_DRV_MSG_CODE_RESET);
 	memcpy(dev->dev_addr, bp->mac_addr, ETH_ALEN);
 
 	dev->hw_features = NETIF_F_IP_CSUM | NETIF_F_SG |
@@ -8608,7 +8607,6 @@ bnx2_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 	return 0;
 
 error:
-	bnx2_release_firmware(bp);
 	pci_iounmap(pdev, bp->regview);
 	pci_release_regions(pdev);
 	pci_disable_device(pdev);
-- 
2.4.1

RE: [bnx2] [Regression 4.8] Driver loading fails without firmware

[Mody, Rasesh]

> From: dept_hsg_linux_nic_dev-bounces@qlclistserver.qlogic.com
> [mailto:dept_hsg_linux_nic_dev-bounces@qlclistserver.qlogic.com] On
> Behalf Of Paul Menzel
> 
> Dear Baoquan,
> 
> 
> On 10/26/16 14:00, Baoquan He wrote:
> 
> > On 10/26/16 at 12:31pm, Paul Menzel wrote:
> >>>>         dev->hw_features = NETIF_F_IP_CSUM | NETIF_F_SG | @@
> >>>> -8607,6 +8608,7 @@ bnx2_init_one(struct pci_dev *pdev, const struct
> >>>> pci_device_id *ent)
> >>>>         return 0;
> >>>>
> >>>>  error:
> >>>> +       bnx2_release_firmware(bp);
> >>>>         pci_iounmap(pdev, bp->regview);
> >>>>         pci_release_regions(pdev);
> >>>>         pci_disable_device(pdev);
> >>
> >> Baoquan, could you please fix this regression. My suggestion is, that
> >> you add the old code back, but check if the firmware has been loaded.
> >> If it hasn’t, load it again.
> >>
> >> That way, people can update their Linux kernel, and it continues
> >> working without changing the initramfs, or anything else.
> >
> > I saw your mail but I am also not familiar with bnx2 driver. As the
> > commit log says I just tried to make bnx2 driver reset itself earlier.
> >
> > So you did a git bisect and found this commit caused the regression,
> > right? If yes, and network developers have no action, I will look into
> > the code and see if I have idea to fix it.
> 
> Well, I looked through the commits and found that one, which would explain
> the changed behavior.
> 
> To be sure, and to follow your request, I took Linux 4.8.4 and reverted your
> commit (attached). Then I deleted the firmware again from the initramfs,
> and rebooted. The devices showed up just fine as before.

Thanks Paul!
Acked-by: Rasesh Mody <Rasesh.Mody@cavium.com>

> So to summarize, the commit is indeed the culprit.
> 
> Thank you for looking into this.
> 
> 
> Kind regards,
> 
> Paul

Re: [bnx2] [Regression 4.8] Driver loading fails without firmware

[Baoquan He]

On 10/27/16 at 03:21pm, Paul Menzel wrote:
> Dear Baoquan,
> > > Baoquan, could you please fix this regression. My suggestion is, that you
> > > add the old code back, but check if the firmware has been loaded. If it
> > > hasn’t, load it again.
> > > 
> > > That way, people can update their Linux kernel, and it continues working
> > > without changing the initramfs, or anything else.
> > 
> > I saw your mail but I am also not familiar with bnx2 driver. As the
> > commit log says I just tried to make bnx2 driver reset itself earlier.
> > 
> > So you did a git bisect and found this commit caused the regression,
> > right? If yes, and network developers have no action, I will look into
> > the code and see if I have idea to fix it.
> 
> Well, I looked through the commits and found that one, which would explain
> the changed behavior.
> 
> To be sure, and to follow your request, I took Linux 4.8.4 and reverted your
> commit (attached). Then I deleted the firmware again from the initramfs, and
> rebooted. The devices showed up just fine as before.
> 
> So to summarize, the commit is indeed the culprit.

Hi Paul,

Sorry for this.

Could you tell the steps to reproduce? I will find a machine with bnx2
NIC and check if there's other ways.

Thanks
Baoquan

> From 61b8dac8796343a797858b4a2eb0a59a0cfcd735 Mon Sep 17 00:00:00 2001
> From: Paul Menzel <pmenzel@molgen.mpg.de>
> Date: Thu, 27 Oct 2016 11:34:52 +0200
> Subject: [PATCH] Revert "bnx2: Reset device during driver initialization"
> 
> This reverts commit 3e1be7ad2d38c6bd6aeef96df9bd0a7822f4e51c.
> ---
>  drivers/net/ethernet/broadcom/bnx2.c | 12 +++++-------
>  1 file changed, 5 insertions(+), 7 deletions(-)
> 
> diff --git a/drivers/net/ethernet/broadcom/bnx2.c b/drivers/net/ethernet/broadcom/bnx2.c
> index 27f11a5..ecd357d 100644
> --- a/drivers/net/ethernet/broadcom/bnx2.c
> +++ b/drivers/net/ethernet/broadcom/bnx2.c
> @@ -6356,6 +6356,10 @@ bnx2_open(struct net_device *dev)
>  	struct bnx2 *bp = netdev_priv(dev);
>  	int rc;
>  
> +	rc = bnx2_request_firmware(bp);
> +	if (rc < 0)
> +		goto out;
> +
>  	netif_carrier_off(dev);
>  
>  	bnx2_disable_int(bp);
> @@ -6424,6 +6428,7 @@ bnx2_open(struct net_device *dev)
>  	bnx2_free_irq(bp);
>  	bnx2_free_mem(bp);
>  	bnx2_del_napi(bp);
> +	bnx2_release_firmware(bp);
>  	goto out;
>  }
>  
> @@ -8570,12 +8575,6 @@ bnx2_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
>  
>  	pci_set_drvdata(pdev, dev);
>  
> -	rc = bnx2_request_firmware(bp);
> -	if (rc < 0)
> -		goto error;
> -
> -
> -	bnx2_reset_chip(bp, BNX2_DRV_MSG_CODE_RESET);
>  	memcpy(dev->dev_addr, bp->mac_addr, ETH_ALEN);
>  
>  	dev->hw_features = NETIF_F_IP_CSUM | NETIF_F_SG |
> @@ -8608,7 +8607,6 @@ bnx2_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
>  	return 0;
>  
>  error:
> -	bnx2_release_firmware(bp);
>  	pci_iounmap(pdev, bp->regview);
>  	pci_release_regions(pdev);
>  	pci_disable_device(pdev);
> -- 
> 2.4.1
> 

Re: [bnx2] [Regression 4.8] Driver loading fails without firmware

[Paul Menzel]

Dear Baoquan,


Am Samstag, den 29.10.2016, 10:55 +0800 schrieb Baoquan He:
> On 10/27/16 at 03:21pm, Paul Menzel wrote:

> > > > Baoquan, could you please fix this regression. My suggestion is, that you
> > > > add the old code back, but check if the firmware has been loaded. If it
> > > > hasn’t, load it again.
> > > > 
> > > > That way, people can update their Linux kernel, and it continues working
> > > > without changing the initramfs, or anything else.
> > > 
> > > I saw your mail but I am also not familiar with bnx2 driver. As the
> > > commit log says I just tried to make bnx2 driver reset itself earlier.
> > > 
> > > So you did a git bisect and found this commit caused the regression,
> > > right? If yes, and network developers have no action, I will look into
> > > the code and see if I have idea to fix it.
> > 
> > Well, I looked through the commits and found that one, which would explain
> > the changed behavior.
> > 
> > To be sure, and to follow your request, I took Linux 4.8.4 and reverted your
> > commit (attached). Then I deleted the firmware again from the initramfs, and
> > rebooted. The devices showed up just fine as before.
> > 
> > So to summarize, the commit is indeed the culprit.

> Sorry for this.
> 
> Could you tell the steps to reproduce? I will find a machine with bnx2
> NIC and check if there's other ways.

Well, delete the bnx2 firmware files from the initramfs, and start the
system.

Did you read my proposal, to try to load the firmware twice, that means,
basically revert only the deleted lines of your commit, and add an
additional check?


Kind regards,

Paul

Re: [bnx2] [Regression 4.8] Driver loading fails without firmware

[Baoquan He]

Hi Paul,

On 10/30/16 at 12:05pm, Paul Menzel wrote:
> Dear Baoquan,
> 
> 
> Am Samstag, den 29.10.2016, 10:55 +0800 schrieb Baoquan He:
> > On 10/27/16 at 03:21pm, Paul Menzel wrote:
> 
> > > > > Baoquan, could you please fix this regression. My suggestion is, that you
> > > > > add the old code back, but check if the firmware has been loaded. If it
> > > > > hasn’t, load it again.
> > > > > 
> > > > > That way, people can update their Linux kernel, and it continues working
> > > > > without changing the initramfs, or anything else.
> > > > 
> > > > I saw your mail but I am also not familiar with bnx2 driver. As the
> > > > commit log says I just tried to make bnx2 driver reset itself earlier.
> > > > 
> > > > So you did a git bisect and found this commit caused the regression,
> > > > right? If yes, and network developers have no action, I will look into
> > > > the code and see if I have idea to fix it.
> > > 
> > > Well, I looked through the commits and found that one, which would explain
> > > the changed behavior.
> > > 
> > > To be sure, and to follow your request, I took Linux 4.8.4 and reverted your
> > > commit (attached). Then I deleted the firmware again from the initramfs, and
> > > rebooted. The devices showed up just fine as before.
> > > 
> > > So to summarize, the commit is indeed the culprit.
> 
> > Sorry for this.
> > 
> > Could you tell the steps to reproduce? I will find a machine with bnx2
> > NIC and check if there's other ways.
> 
> Well, delete the bnx2 firmware files from the initramfs, and start the
> system.
> 
> Did you read my proposal, to try to load the firmware twice, that means,
> basically revert only the deleted lines of your commit, and add an
> additional check?

Thanks for your information!

I got a x86_64 system with bnx2 NIC, and clone Linus's git tree into
that system. Then building a new kernel 4.9.0-rc3+ with new initramfs.
But when I uncompressed the new initramfs, didn't find bnx2 related
firmware, no bnx2 files under lib/firmware of uncompressed initramfs
folder. While I did see them in /lib/firmware/bnx2/bnx2-xxxxx.fw. Could
you please say it more specifically how I should do to reproduce the
failure you encountered? I think your proposal looks good, just need a
test before post.

Thanks
Baoquan

Re: [bnx2] [Regression 4.8] Driver loading fails without firmware

[Baoquan He]

On 10/31/16 at 11:59am, Baoquan He wrote:
> Hi Paul,
> 
> On 10/30/16 at 12:05pm, Paul Menzel wrote:
> > Dear Baoquan,
> > 
> > 
> > Am Samstag, den 29.10.2016, 10:55 +0800 schrieb Baoquan He:
> > > On 10/27/16 at 03:21pm, Paul Menzel wrote:
> > 
> > > > > > Baoquan, could you please fix this regression. My suggestion is, that you
> > > > > > add the old code back, but check if the firmware has been loaded. If it
> > > > > > hasn’t, load it again.
> > > > > > 
> > > > > > That way, people can update their Linux kernel, and it continues working
> > > > > > without changing the initramfs, or anything else.
> > > > > 
> > > > > I saw your mail but I am also not familiar with bnx2 driver. As the
> > > > > commit log says I just tried to make bnx2 driver reset itself earlier.
> > > > > 
> > > > > So you did a git bisect and found this commit caused the regression,
> > > > > right? If yes, and network developers have no action, I will look into
> > > > > the code and see if I have idea to fix it.
> > > > 
> > > > Well, I looked through the commits and found that one, which would explain
> > > > the changed behavior.
> > > > 
> > > > To be sure, and to follow your request, I took Linux 4.8.4 and reverted your
> > > > commit (attached). Then I deleted the firmware again from the initramfs, and
> > > > rebooted. The devices showed up just fine as before.
> > > > 
> > > > So to summarize, the commit is indeed the culprit.
> > 
> > > Sorry for this.
> > > 
> > > Could you tell the steps to reproduce? I will find a machine with bnx2
> > > NIC and check if there's other ways.
> > 
> > Well, delete the bnx2 firmware files from the initramfs, and start the
> > system.
> > 
> > Did you read my proposal, to try to load the firmware twice, that means,
> > basically revert only the deleted lines of your commit, and add an
> > additional check?
> 
Please ignore this one, I have reproduced it. Will post a fix after
test.
> 
> I got a x86_64 system with bnx2 NIC, and clone Linus's git tree into
> that system. Then building a new kernel 4.9.0-rc3+ with new initramfs.
> But when I uncompressed the new initramfs, didn't find bnx2 related
> firmware, no bnx2 files under lib/firmware of uncompressed initramfs
> folder. While I did see them in /lib/firmware/bnx2/bnx2-xxxxx.fw. Could
> you please say it more specifically how I should do to reproduce the
> failure you encountered? I think your proposal looks good, just need a
> test before post.
> 
> Thanks
> Baoquan

Re: [bnx2] [Regression 4.8] Driver loading fails without firmware

[Baoquan He]

Hi Paul,

On 10/26/16 at 12:31pm, Paul Menzel wrote:
> Baoquan, could you please fix this regression. My suggestion is, that you
> add the old code back, but check if the firmware has been loaded. If it
> hasn’t, load it again.
> 
> That way, people can update their Linux kernel, and it continues working
> without changing the initramfs, or anything else.

I checked code and this looks good to me. I can post a patch with this
change to upstream, see what maintainers and other reviewers say.

The thing is I don't understand quite well about your requirement. With
my understanding, you just didn't add bnx2 firmware into initramfs, but
later opening the interface can still request that firmware with "ifup
eth-xxx" command. Is that correct? If yes, requeting firmware twice in
probing path and opening path looks good.


However I am wondering what's your exact steps to do this. 

What I tried to do is I execute command "dracut --add-drivers bnx2 -f
/boot/initramfs-4.9.0-rc3+.img 4.9.0-rc3+" to build a new initramfs,
meanwhile make sure bnx2.ko is included, then uncompressed initramfs and
deleted bnx2 folder under lib/firmware/ of uncompressed initramfs. Then
pack them to be /boot/initramfs-4.9.0-rc3+.img and restart. I did saw
below failure message. But later how did you really make the bnx2
network interface up? Could you say it more specifically?

[    7.364186] bnx2: QLogic bnx2 Gigabit Ethernet Driver v2.2.6 (January 29, 2014)
[    7.371706] ACPI: PCI Interrupt Link [LN44] enabled at IRQ 44
[    7.378128] bnx2 0000:01:00.0: Direct firmware load for bnx2/bnx2-mips-09-6.2.1b.fw failed with error -2
[    7.387619] bnx2: Can't load firmware file "bnx2/bnx2-mips-09-6.2.1b.fw"
[    7.387888] bnx2: probe of 0000:01:00.0 failed with error -2
[    7.388990] ACPI: PCI Interrupt Link [LN45] enabled at IRQ 45
[    7.389370] bnx2 0000:01:00.1: Direct firmware load for bnx2/bnx2-mips-09-6.2.1b.fw failed with error -2
[    7.389371] bnx2: Can't load firmware file "bnx2/bnx2-mips-09-6.2.1b.fw"
[    7.389475] bnx2: probe of 0000:01:00.1 failed with error -2

Thanks
Baoquan

Re: [bnx2] [Regression 4.8] Driver loading fails without firmware

[Paul Menzel]

Dear Baoquan,


On 10/31/16 11:09, Baoquan He wrote:

> On 10/26/16 at 12:31pm, Paul Menzel wrote:
>> Baoquan, could you please fix this regression. My suggestion is, that you
>> add the old code back, but check if the firmware has been loaded. If it
>> hasn’t, load it again.
>>
>> That way, people can update their Linux kernel, and it continues working
>> without changing the initramfs, or anything else.
>
> I checked code and this looks good to me. I can post a patch with this
> change to upstream, see what maintainers and other reviewers say.
>
> The thing is I don't understand quite well about your requirement. With
> my understanding, you just didn't add bnx2 firmware into initramfs, but
> later opening the interface can still request that firmware with "ifup
> eth-xxx" command. Is that correct? If yes, requesting firmware twice in
> probing path and opening path looks good.
>
> However I am wondering what's your exact steps to do this.
>
> What I tried to do is I execute command "dracut --add-drivers bnx2 -f
> /boot/initramfs-4.9.0-rc3+.img 4.9.0-rc3+" to build a new initramfs,
> meanwhile make sure bnx2.ko is included, then uncompressed initramfs and
> deleted bnx2 folder under lib/firmware/ of uncompressed initramfs. Then
> pack them to be /boot/initramfs-4.9.0-rc3+.img and restart. I did saw
> below failure message. But later how did you really make the bnx2
> network interface up? Could you say it more specifically?
>
> [    7.364186] bnx2: QLogic bnx2 Gigabit Ethernet Driver v2.2.6 (January 29, 2014)
> [    7.371706] ACPI: PCI Interrupt Link [LN44] enabled at IRQ 44
> [    7.378128] bnx2 0000:01:00.0: Direct firmware load for bnx2/bnx2-mips-09-6.2.1b.fw failed with error -2
> [    7.387619] bnx2: Can't load firmware file "bnx2/bnx2-mips-09-6.2.1b.fw"
> [    7.387888] bnx2: probe of 0000:01:00.0 failed with error -2
> [    7.388990] ACPI: PCI Interrupt Link [LN45] enabled at IRQ 45
> [    7.389370] bnx2 0000:01:00.1: Direct firmware load for bnx2/bnx2-mips-09-6.2.1b.fw failed with error -2
> [    7.389371] bnx2: Can't load firmware file "bnx2/bnx2-mips-09-6.2.1b.fw"
> [    7.389475] bnx2: probe of 0000:01:00.1 failed with error -2

Hopefully I understood your questions correctly, so that my answers make 
sense.

First, sorry for not saying that earlier, on our system the driver is 
built into the Linux kernel.

```
$ grep BNX2 /boot/config-4.8.4.mx64.112
# CONFIG_SCSI_BNX2_ISCSI is not set
CONFIG_BNX2=y
CONFIG_BNX2X=y
CONFIG_BNX2X_SRIOV=y
```

Second, the filesystem driver is also built into the Linux kernel.

On our system, there is a systemd service unit, which sets up the 
network device.

```
$ more /etc/systemd/system/network.service
[Unit]
Description=Network Connectivity
DefaultDependencies=no

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/mxnetctl start
ExecStart=/sbin/ip addr add XXX broadcast XXX dev net00
ExecStart=/sbin/ip link set up dev net00
ExecStop=/sbin/ip addr del XXX dev net00
StandardOutput=syslog

[Install]
WantedBy=network.target
```

During that time, the hard drive has been detected, and the filesystem 
has been mounted.


Kind regards,

Paul

Re: [bnx2] [Regression 4.8] Driver loading fails without firmware

[Baoquan He]

On 10/31/16 at 11:43am, Paul Menzel wrote:
> Dear Baoquan,
> 
> 
> On 10/31/16 11:09, Baoquan He wrote:
> 
> > On 10/26/16 at 12:31pm, Paul Menzel wrote:
> > > Baoquan, could you please fix this regression. My suggestion is, that you
> > > add the old code back, but check if the firmware has been loaded. If it
> > > hasn’t, load it again.
> > > 
> > > That way, people can update their Linux kernel, and it continues working
> > > without changing the initramfs, or anything else.
> > 
> > I checked code and this looks good to me. I can post a patch with this
> > change to upstream, see what maintainers and other reviewers say.
> > 
> > The thing is I don't understand quite well about your requirement. With
> > my understanding, you just didn't add bnx2 firmware into initramfs, but
> > later opening the interface can still request that firmware with "ifup
> > eth-xxx" command. Is that correct? If yes, requesting firmware twice in
> > probing path and opening path looks good.
> > 
> > However I am wondering what's your exact steps to do this.
> > 
> > What I tried to do is I execute command "dracut --add-drivers bnx2 -f
> > /boot/initramfs-4.9.0-rc3+.img 4.9.0-rc3+" to build a new initramfs,
> > meanwhile make sure bnx2.ko is included, then uncompressed initramfs and
> > deleted bnx2 folder under lib/firmware/ of uncompressed initramfs. Then
> > pack them to be /boot/initramfs-4.9.0-rc3+.img and restart. I did saw
> > below failure message. But later how did you really make the bnx2
> > network interface up? Could you say it more specifically?
> > 
> > [    7.364186] bnx2: QLogic bnx2 Gigabit Ethernet Driver v2.2.6 (January 29, 2014)
> > [    7.371706] ACPI: PCI Interrupt Link [LN44] enabled at IRQ 44
> > [    7.378128] bnx2 0000:01:00.0: Direct firmware load for bnx2/bnx2-mips-09-6.2.1b.fw failed with error -2
> > [    7.387619] bnx2: Can't load firmware file "bnx2/bnx2-mips-09-6.2.1b.fw"
> > [    7.387888] bnx2: probe of 0000:01:00.0 failed with error -2
> > [    7.388990] ACPI: PCI Interrupt Link [LN45] enabled at IRQ 45
> > [    7.389370] bnx2 0000:01:00.1: Direct firmware load for bnx2/bnx2-mips-09-6.2.1b.fw failed with error -2
> > [    7.389371] bnx2: Can't load firmware file "bnx2/bnx2-mips-09-6.2.1b.fw"
> > [    7.389475] bnx2: probe of 0000:01:00.1 failed with error -2
> 
> Hopefully I understood your questions correctly, so that my answers make
> sense.
> 
> First, sorry for not saying that earlier, on our system the driver is built
> into the Linux kernel.

It's clear. So you built bnx2 driver into kernel and that makes bnx2
probe earlier be executed in do_initcalls() when initramfs has not been
uncompressed and mounted, surely requesting firmware will fail. While
building it as kernel module makes bnx2 probe be executed after
initramfs has been uncompressed and mounted, and firmware can be
detected.

Well, sometime code change is very easy, but it's very difficult to
write patch log to make maintainers and reviewers easily understand why
this patch is acceptable and user case is reasonalbe.

I will post a patch to add the firmware requesting code back in
bnx2_open.

Thanks for your detailed information, Paul.

> 
> ```
> $ grep BNX2 /boot/config-4.8.4.mx64.112
> # CONFIG_SCSI_BNX2_ISCSI is not set
> CONFIG_BNX2=y
> CONFIG_BNX2X=y
> CONFIG_BNX2X_SRIOV=y
> ```
> 
> Second, the filesystem driver is also built into the Linux kernel.
> 
> On our system, there is a systemd service unit, which sets up the network
> device.
> 
> ```
> $ more /etc/systemd/system/network.service
> [Unit]
> Description=Network Connectivity
> DefaultDependencies=no
> 
> [Service]
> Type=oneshot
> RemainAfterExit=yes
> ExecStart=/usr/sbin/mxnetctl start
> ExecStart=/sbin/ip addr add XXX broadcast XXX dev net00
> ExecStart=/sbin/ip link set up dev net00
> ExecStop=/sbin/ip addr del XXX dev net00
> StandardOutput=syslog
> 
> [Install]
> WantedBy=network.target
> ```
> 
> During that time, the hard drive has been detected, and the filesystem has
> been mounted.
> 
> 
> Kind regards,
> 
> Paul