tg3 'No PHY devices' loading issue

tg3 'No PHY devices' loading issue

[Josh Boyer]

Hi Matt and Michael,

I'm seeing an odd issue with the tg3 driver on one of my development
machines.  I've tried kernels 3.2.10, 3.3.0, 3.3.1, 3.3.2 and 3.4-rc3
and they all seem to exhibit this issue now.  When the machine boots
and the tg3 driver is loaded, it fails to find a PHY and then reports
'Problem fetching invariants of chip'.  If I do a rmmod/modprobe of
tg3 after login, the probe seems to work fine and ethernet works as
expected.  You can see this in the dmesg below:

[jwboyer@vader ~]$ dmesg | grep tg3
[    2.084969] tg3.c:v3.122 (December 7, 2011)
[    2.093511] tg3 mdio bus: probed
[    2.093513] tg3 0000:03:00.0: No PHY devices
[    2.093531] tg3 0000:03:00.0: Problem fetching invariants of chip, aborting
[   90.824697] tg3.c:v3.122 (December 7, 2011)
[   90.857068] tg3 mdio bus: probed
[   90.862540] tg3 0000:03:00.0: eth0: Tigon3 [partno(BCM57788) rev 57780001] (PCI Express) MAC address 18:03:73:e6:01:88
[   90.862547] tg3 0000:03:00.0: eth0: attached PHY driver [Broadcom BCM57780] (mii_bus:phy_addr=300:01)
[   90.862552] tg3 0000:03:00.0: eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1]
[   90.862557] tg3 0000:03:00.0: eth0: dma_rwctrl[76180000] dma_mask[64-bit]
[   90.919961] tg3 0000:03:00.0: irq 47 for MSI/MSI-X
[   91.863311] tg3 0000:03:00.0: p3p1: Link is down
[   92.864348] tg3 0000:03:00.0: p3p1: Link is up at 100 Mbps, full duplex
[   92.864352] tg3 0000:03:00.0: p3p1: Flow control is on for TX and on for RX

It has worked on some of the older kernels without the need for the
manual rmmod/modprobe step, so it seems to be somewhat timing related.
I'm not sure if there is a module load ordering issue, but that doesn't
seem to be the case.  I can't explain why a later modprobe would work
just fine though.

Do you have any thoughts on how to go about debugging/fixing this?  I'd
be happy to test and provide whatever information you need.

josh

Re: tg3 'No PHY devices' loading issue

[Matt Carlson]

On Tue, Apr 17, 2012 at 10:18:57AM -0400, Josh Boyer wrote:
> Hi Matt and Michael,
> 
> I'm seeing an odd issue with the tg3 driver on one of my development
> machines.  I've tried kernels 3.2.10, 3.3.0, 3.3.1, 3.3.2 and 3.4-rc3
> and they all seem to exhibit this issue now.  When the machine boots
> and the tg3 driver is loaded, it fails to find a PHY and then reports
> 'Problem fetching invariants of chip'.  If I do a rmmod/modprobe of
> tg3 after login, the probe seems to work fine and ethernet works as
> expected.  You can see this in the dmesg below:
> 
> [jwboyer@vader ~]$ dmesg | grep tg3
> [    2.084969] tg3.c:v3.122 (December 7, 2011)
> [    2.093511] tg3 mdio bus: probed
> [    2.093513] tg3 0000:03:00.0: No PHY devices
> [    2.093531] tg3 0000:03:00.0: Problem fetching invariants of chip, aborting
> [   90.824697] tg3.c:v3.122 (December 7, 2011)
> [   90.857068] tg3 mdio bus: probed
> [   90.862540] tg3 0000:03:00.0: eth0: Tigon3 [partno(BCM57788) rev 57780001] (PCI Express) MAC address 18:03:73:e6:01:88
> [   90.862547] tg3 0000:03:00.0: eth0: attached PHY driver [Broadcom BCM57780] (mii_bus:phy_addr=300:01)
> [   90.862552] tg3 0000:03:00.0: eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1]
> [   90.862557] tg3 0000:03:00.0: eth0: dma_rwctrl[76180000] dma_mask[64-bit]
> [   90.919961] tg3 0000:03:00.0: irq 47 for MSI/MSI-X
> [   91.863311] tg3 0000:03:00.0: p3p1: Link is down
> [   92.864348] tg3 0000:03:00.0: p3p1: Link is up at 100 Mbps, full duplex
> [   92.864352] tg3 0000:03:00.0: p3p1: Flow control is on for TX and on for RX
> 
> It has worked on some of the older kernels without the need for the
> manual rmmod/modprobe step, so it seems to be somewhat timing related.
> I'm not sure if there is a module load ordering issue, but that doesn't
> seem to be the case.  I can't explain why a later modprobe would work
> just fine though.
> 
> Do you have any thoughts on how to go about debugging/fixing this?  I'd
> be happy to test and provide whatever information you need.

The 57788 uses the broadcom phylib module.  For some reason, it isn't
available the first module load attempt.  A while ago, code was added to
phylib to request modules from userspace if the particular phy support
code wasn't already loaded.  It looks like this mechanism isn't working
too well the first time through.

Can you reproduce the problem if you run 'rmmod broadcom' and then
reload tg3?

Re: tg3 'No PHY devices' loading issue

[Ben Hutchings]

On Tue, Apr 17, 2012 at 11:50:07AM -0700, Matt Carlson wrote:
> On Tue, Apr 17, 2012 at 10:18:57AM -0400, Josh Boyer wrote:
> > Hi Matt and Michael,
> > 
> > I'm seeing an odd issue with the tg3 driver on one of my development
> > machines.  I've tried kernels 3.2.10, 3.3.0, 3.3.1, 3.3.2 and 3.4-rc3
> > and they all seem to exhibit this issue now.  When the machine boots
> > and the tg3 driver is loaded, it fails to find a PHY and then reports
> > 'Problem fetching invariants of chip'.  If I do a rmmod/modprobe of
> > tg3 after login, the probe seems to work fine and ethernet works as
> > expected.  You can see this in the dmesg below:
[...]
> The 57788 uses the broadcom phylib module.  For some reason, it isn't
> available the first module load attempt.  A while ago, code was added to
> phylib to request modules from userspace if the particular phy support
> code wasn't already loaded.  It looks like this mechanism isn't working
> too well the first time through.
[...]

At a guess, tg3 is in the initramfs but broadcom isn't.

Ben.

-- 
Ben Hutchings
We get into the habit of living before acquiring the habit of thinking.
                                                              - Albert Camus

Re: tg3 'No PHY devices' loading issue

[Josh Boyer]

On Tue, Apr 17, 2012 at 11:50:07AM -0700, Matt Carlson wrote:
> On Tue, Apr 17, 2012 at 10:18:57AM -0400, Josh Boyer wrote:
> > It has worked on some of the older kernels without the need for the
> > manual rmmod/modprobe step, so it seems to be somewhat timing related.
> > I'm not sure if there is a module load ordering issue, but that doesn't
> > seem to be the case.  I can't explain why a later modprobe would work
> > just fine though.
> > 
> > Do you have any thoughts on how to go about debugging/fixing this?  I'd
> > be happy to test and provide whatever information you need.
> 
> The 57788 uses the broadcom phylib module.  For some reason, it isn't
> available the first module load attempt.  A while ago, code was added to
> phylib to request modules from userspace if the particular phy support
> code wasn't already loaded.  It looks like this mechanism isn't working
> too well the first time through.
> 
> Can you reproduce the problem if you run 'rmmod broadcom' and then
> reload tg3?

I'll give it a shot, but that sounds like a quite plausible scenario to
me.  Oddly, even after the manual modprobe, the broadcom phy module
doesn't show an increased use count.  There's also no dependency on
'broadcom' listed for tg3 by modinfo.

josh

Re: tg3 'No PHY devices' loading issue

[Josh Boyer]

On Tue, Apr 17, 2012 at 08:10:42PM +0100, Ben Hutchings wrote:
> On Tue, Apr 17, 2012 at 11:50:07AM -0700, Matt Carlson wrote:
> > On Tue, Apr 17, 2012 at 10:18:57AM -0400, Josh Boyer wrote:
> > > Hi Matt and Michael,
> > > 
> > > I'm seeing an odd issue with the tg3 driver on one of my development
> > > machines.  I've tried kernels 3.2.10, 3.3.0, 3.3.1, 3.3.2 and 3.4-rc3
> > > and they all seem to exhibit this issue now.  When the machine boots
> > > and the tg3 driver is loaded, it fails to find a PHY and then reports
> > > 'Problem fetching invariants of chip'.  If I do a rmmod/modprobe of
> > > tg3 after login, the probe seems to work fine and ethernet works as
> > > expected.  You can see this in the dmesg below:
> [...]
> > The 57788 uses the broadcom phylib module.  For some reason, it isn't
> > available the first module load attempt.  A while ago, code was added to
> > phylib to request modules from userspace if the particular phy support
> > code wasn't already loaded.  It looks like this mechanism isn't working
> > too well the first time through.
> [...]
> 
> At a guess, tg3 is in the initramfs but broadcom isn't.

Good guess.  At least on one of the initramfs' I checked.  I'll verify
the others, but it seems likely they match.  As I just said in my other
reply, I would have expected tg3 to have a module dep on it to get it
included in the initramfs built, but it seems that isn't the case.

josh

Re: tg3 'No PHY devices' loading issue

[Ben Hutchings]

On Tue, Apr 17, 2012 at 03:40:30PM -0400, Josh Boyer wrote:
> On Tue, Apr 17, 2012 at 08:10:42PM +0100, Ben Hutchings wrote:
> > On Tue, Apr 17, 2012 at 11:50:07AM -0700, Matt Carlson wrote:
> > > On Tue, Apr 17, 2012 at 10:18:57AM -0400, Josh Boyer wrote:
> > > > Hi Matt and Michael,
> > > > 
> > > > I'm seeing an odd issue with the tg3 driver on one of my development
> > > > machines.  I've tried kernels 3.2.10, 3.3.0, 3.3.1, 3.3.2 and 3.4-rc3
> > > > and they all seem to exhibit this issue now.  When the machine boots
> > > > and the tg3 driver is loaded, it fails to find a PHY and then reports
> > > > 'Problem fetching invariants of chip'.  If I do a rmmod/modprobe of
> > > > tg3 after login, the probe seems to work fine and ethernet works as
> > > > expected.  You can see this in the dmesg below:
> > [...]
> > > The 57788 uses the broadcom phylib module.  For some reason, it isn't
> > > available the first module load attempt.  A while ago, code was added to
> > > phylib to request modules from userspace if the particular phy support
> > > code wasn't already loaded.  It looks like this mechanism isn't working
> > > too well the first time through.
> > [...]
> > 
> > At a guess, tg3 is in the initramfs but broadcom isn't.
> 
> Good guess.  At least on one of the initramfs' I checked.  I'll verify
> the others, but it seems likely they match.  As I just said in my other
> reply, I would have expected tg3 to have a module dep on it to get it
> included in the initramfs built, but it seems that isn't the case.
 
tg3 doesn't use any symbols from broadcom, which is the only way
depmod can find dependencies.

In initramfs-tools (used in Debian and its derivatives) we just copy
all PHY drivers into the initramfs by default.  dracut should probably
do the same, at least when phylib is in the initramfs (or built into
the kernel).  (tg3 does use symbols from phylib.)

Ben.

-- 
Ben Hutchings
We get into the habit of living before acquiring the habit of thinking.
                                                              - Albert Camus

Re: tg3 'No PHY devices' loading issue

[Josh Boyer]

On Tue, Apr 17, 2012 at 03:39:08PM -0400, Josh Boyer wrote:
> On Tue, Apr 17, 2012 at 11:50:07AM -0700, Matt Carlson wrote:
> > On Tue, Apr 17, 2012 at 10:18:57AM -0400, Josh Boyer wrote:
> > > It has worked on some of the older kernels without the need for the
> > > manual rmmod/modprobe step, so it seems to be somewhat timing related.
> > > I'm not sure if there is a module load ordering issue, but that doesn't
> > > seem to be the case.  I can't explain why a later modprobe would work
> > > just fine though.
> > > 
> > > Do you have any thoughts on how to go about debugging/fixing this?  I'd
> > > be happy to test and provide whatever information you need.
> > 
> > The 57788 uses the broadcom phylib module.  For some reason, it isn't
> > available the first module load attempt.  A while ago, code was added to
> > phylib to request modules from userspace if the particular phy support
> > code wasn't already loaded.  It looks like this mechanism isn't working
> > too well the first time through.
> > 
> > Can you reproduce the problem if you run 'rmmod broadcom' and then
> > reload tg3?
> 
> I'll give it a shot, but that sounds like a quite plausible scenario to
> me.  Oddly, even after the manual modprobe, the broadcom phy module
> doesn't show an increased use count.  There's also no dependency on
> 'broadcom' listed for tg3 by modinfo.

If I rmmod both tg3 and broadcom and then just load tg3 (either via
insmod or modprobe), it works fine.  It does so because whatever you did
to make it request the phylib module from userspace also worked because
the broadcom module is loaded as well.

So it seems my issue is basically because the initramfs isn't built with
the broadcom module included.  I'll have to figure out a way to teach
dracut what to do here, since it doesn't seem to be automatically
picking it up (Harald CC'd).

Thanks much for the very quick response from everyone.

josh

Re: tg3 'No PHY devices' loading issue

[Josh Boyer]

On Tue, Apr 17, 2012 at 09:59:55PM +0100, Ben Hutchings wrote:
> On Tue, Apr 17, 2012 at 03:40:30PM -0400, Josh Boyer wrote:
> > On Tue, Apr 17, 2012 at 08:10:42PM +0100, Ben Hutchings wrote:
> > > On Tue, Apr 17, 2012 at 11:50:07AM -0700, Matt Carlson wrote:
> > > > On Tue, Apr 17, 2012 at 10:18:57AM -0400, Josh Boyer wrote:
> > > > > Hi Matt and Michael,
> > > > > 
> > > > > I'm seeing an odd issue with the tg3 driver on one of my development
> > > > > machines.  I've tried kernels 3.2.10, 3.3.0, 3.3.1, 3.3.2 and 3.4-rc3
> > > > > and they all seem to exhibit this issue now.  When the machine boots
> > > > > and the tg3 driver is loaded, it fails to find a PHY and then reports
> > > > > 'Problem fetching invariants of chip'.  If I do a rmmod/modprobe of
> > > > > tg3 after login, the probe seems to work fine and ethernet works as
> > > > > expected.  You can see this in the dmesg below:
> > > [...]
> > > > The 57788 uses the broadcom phylib module.  For some reason, it isn't
> > > > available the first module load attempt.  A while ago, code was added to
> > > > phylib to request modules from userspace if the particular phy support
> > > > code wasn't already loaded.  It looks like this mechanism isn't working
> > > > too well the first time through.
> > > [...]
> > > 
> > > At a guess, tg3 is in the initramfs but broadcom isn't.
> > 
> > Good guess.  At least on one of the initramfs' I checked.  I'll verify
> > the others, but it seems likely they match.  As I just said in my other
> > reply, I would have expected tg3 to have a module dep on it to get it
> > included in the initramfs built, but it seems that isn't the case.
>  
> tg3 doesn't use any symbols from broadcom, which is the only way
> depmod can find dependencies.
> 
> In initramfs-tools (used in Debian and its derivatives) we just copy
> all PHY drivers into the initramfs by default.  dracut should probably
> do the same, at least when phylib is in the initramfs (or built into
> the kernel).  (tg3 does use symbols from phylib.)

Heh.  Yep, that's basically what I just concluded as well in my other
reply.  I'll work with Harald and get that included or something similar
to it.

josh

Re: tg3 'No PHY devices' loading issue

[Harald Hoyer]

Am 17.04.2012 23:05, schrieb Josh Boyer:
> On Tue, Apr 17, 2012 at 03:39:08PM -0400, Josh Boyer wrote:
>> On Tue, Apr 17, 2012 at 11:50:07AM -0700, Matt Carlson wrote:
>>> On Tue, Apr 17, 2012 at 10:18:57AM -0400, Josh Boyer wrote:
>>>> It has worked on some of the older kernels without the need for the
>>>> manual rmmod/modprobe step, so it seems to be somewhat timing related.
>>>> I'm not sure if there is a module load ordering issue, but that doesn't
>>>> seem to be the case.  I can't explain why a later modprobe would work
>>>> just fine though.
>>>>
>>>> Do you have any thoughts on how to go about debugging/fixing this?  I'd
>>>> be happy to test and provide whatever information you need.
>>>
>>> The 57788 uses the broadcom phylib module.  For some reason, it isn't
>>> available the first module load attempt.  A while ago, code was added to
>>> phylib to request modules from userspace if the particular phy support
>>> code wasn't already loaded.  It looks like this mechanism isn't working
>>> too well the first time through.
>>>
>>> Can you reproduce the problem if you run 'rmmod broadcom' and then
>>> reload tg3?
>>
>> I'll give it a shot, but that sounds like a quite plausible scenario to
>> me.  Oddly, even after the manual modprobe, the broadcom phy module
>> doesn't show an increased use count.  There's also no dependency on
>> 'broadcom' listed for tg3 by modinfo.
> 
> If I rmmod both tg3 and broadcom and then just load tg3 (either via
> insmod or modprobe), it works fine.  It does so because whatever you did
> to make it request the phylib module from userspace also worked because
> the broadcom module is loaded as well.
> 
> So it seems my issue is basically because the initramfs isn't built with
> the broadcom module included.  I'll have to figure out a way to teach
> dracut what to do here, since it doesn't seem to be automatically
> picking it up (Harald CC'd).
> 
> Thanks much for the very quick response from everyone.
> 
> josh

Ok, so we have to include all kernel/drivers/net/phy/*.ko in the initramfs?

Re: tg3 'No PHY devices' loading issue

[Josh Boyer]

On Wed, Apr 18, 2012 at 10:14:47AM +0200, Harald Hoyer wrote:
> Am 17.04.2012 23:05, schrieb Josh Boyer:
> > So it seems my issue is basically because the initramfs isn't built with
> > the broadcom module included.  I'll have to figure out a way to teach
> > dracut what to do here, since it doesn't seem to be automatically
> > picking it up (Harald CC'd).
> > 
> > Thanks much for the very quick response from everyone.
> > 
> > josh
> 
> Ok, so we have to include all kernel/drivers/net/phy/*.ko in the initramfs?

I believe that is probably the safest and most expedient solution, yes.
Ben said the Debian initramfs tool is doing just that.

josh