forcedeth net driver: reverse mac address after pxe boot

forcedeth net driver: reverse mac address after pxe boot

[Alex Owen]

I an issue with the forcedeth driver when used after the BIOS PXE routines.
When booting directly from disk the ethernet MAC address is normal.
eg: 00:16:17:xx:yy:zz
But is the BIOS PXE boot stack loads pxelinux which then boots the
local disk, or if pxelinux loads a kernel/initrd, then the MAC address
detected by the forcedeth linux driver is reversed.
eg zz:yy:xx:17:16:00

This is obviously causes me a problem with automated installs started
via PXE boot as the installed cannot DHCP as the MAC address is wrong.

I have read some of the bug reports and LKML threads on WOL and
forcedeth and I have looked at the code of the driver... most closely
forcedeth v57 as per comment #22 of
http://bugzilla.kernel.org/show_bug.cgi?id=6604

My understanding of the code is that the driver determines the cards
MAC address by reading from registers in the ethernet controller, but
that for reasons best known to nvidia this  address backwards and so
the driver "fixes" this by itself reversing the read values and
writing them back to the controller.

This normally works ok and there has been some work to put the old
"wrong" MAC address back at close down to get WOL to work to.

Enter PXE... Booting from PXE (in BIOS) seems to "fixup" the "wrong"
MAC address so when the driver determines the cards MAC address by
reading from registers in the ethernet controller then MAC address
there is now CORRECT. The driver however assumes it is reversed and in
trying to "fix" the MAC address is infact writes back the
revesed/broken MAC back to the controller.

The obvious fix for this is to try and read the MAC address from the
canonical location... ie where is the source of the address writen
into the controlers registers at power on? But do we know where that
may be?

The other solution would be unconditionally reset the controler to
it's power on state then use the current logic? can we reset the
controller via software?
There does seem to be an nv_mac_reset function... and this does seem
to be called if the card has a capability DEV_HAS_POWER_CONTROL but it
is called in nv_open() while the MAC is read in nv_probe().

Perhaps we need to unconditionally run nv_mac_reset just before
reading the MAC in nv_probe() ?

Anyway I hope that someone who knows kernel internals and this driver
inparticular feels the urge to look at this!

Thanks for reading this far!
Alex Owen

Re: forcedeth net driver: reverse mac address after pxe boot

[H. Peter Anvin]

Alex Owen wrote:
> 
> This is obviously causes me a problem with automated installs started
> via PXE boot as the installed cannot DHCP as the MAC address is wrong.
> 

I have a forcedeth system (ASUS A8N-E) which can't use either the Linux 
driver *or* the standard Windows driver *at all* after booting PXE and 
then exiting the PXE stack for local boot.

> The obvious fix for this is to try and read the MAC address from the
> canonical location... ie where is the source of the address writen
> into the controlers registers at power on? But do we know where that
> may be?
> 
> The other solution would be unconditionally reset the controler to
> it's power on state then use the current logic? can we reset the
> controller via software?
> There does seem to be an nv_mac_reset function... and this does seem
> to be called if the card has a capability DEV_HAS_POWER_CONTROL but it
> is called in nv_open() while the MAC is read in nv_probe().

Doing a proper reset of the hardware would be the right thing, assuming 
that this is safe/possible to do.

	-hpa

Re: forcedeth net driver: reverse mac address after pxe boot

[Alan Cox]

Ar Mer, 2006-10-04 am 17:19 +0100, ysgrifennodd Alex Owen:
> The obvious fix for this is to try and read the MAC address from the
> canonical location... ie where is the source of the address writen
> into the controlers registers at power on? But do we know where that
> may be?

Why not check if the first or last 3 bytes are the Nvidia owner bits.
The only card that will misdetect is 

00:16:17:17:16:00

which doesn't matter anyway

Alan

RE: forcedeth net driver: reverse mac address after pxe boot

[Ayaz Abdulla]

This is a known issue and has been fixed in the latest PXE code. I will
have to find that information for you guys. Can one of you open a bug to
formally keep track of it.

Thanks,
Ayaz

-----Original Message-----
From: Alan Cox [mailto:alan@lxorguk.ukuu.org.uk] 
Sent: Wednesday, October 04, 2006 10:31 AM
To: Alex Owen
Cc: linux-kernel@vger.kernel.org; c-d.hailfinger.kernel.2004@gmx.net;
Ayaz Abdulla
Subject: Re: forcedeth net driver: reverse mac address after pxe boot


Ar Mer, 2006-10-04 am 17:19 +0100, ysgrifennodd Alex Owen:
> The obvious fix for this is to try and read the MAC address from the
> canonical location... ie where is the source of the address writen
> into the controlers registers at power on? But do we know where that
> may be?

Why not check if the first or last 3 bytes are the Nvidia owner bits.
The only card that will misdetect is 

00:16:17:17:16:00

which doesn't matter anyway

Alan

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

Re: forcedeth net driver: reverse mac address after pxe boot

[Alex Owen]

[-- Attachment #1: Type: text/plain, Size: 988 bytes --]

This is a patch against the RHEL4_U3 forcedeth.c source as distributed
by nvidia in the package
http://download.nvidia.com/XFree86/nforce/1.11/NFORCE-Linux-x86-1.11.zip

It performs the test suggested by Alan Cox and reveses the MAC address
as needed.
It is not pretty but is solves my issue while I am waiting for my PC
vendor to get a new BIOS sorted with the upgraded bootagent that fixes
the problem.
As it may help others I'm posting it here!!!

Alex Owen

On 04/10/06, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
> Ar Mer, 2006-10-04 am 17:19 +0100, ysgrifennodd Alex Owen:
> > The obvious fix for this is to try and read the MAC address from the
> > canonical location... ie where is the source of the address writen
> > into the controlers registers at power on? But do we know where that
> > may be?
>
> Why not check if the first or last 3 bytes are the Nvidia owner bits.
> The only card that will misdetect is
>
> 00:16:17:17:16:00
>
> which doesn't matter anyway
>
> Alan
>
>

[-- Attachment #2: forcedeth-pxe-reversed-mac.patch --]
[-- Type: text/x-patch, Size: 2418 bytes --]

--- forcedeth.c.orig	2006-06-19 13:10:32.000000000 +0100
+++ forcedeth.c	2006-10-20 17:52:33.000000000 +0100
@@ -4784,12 +4784,39 @@
 	np->orig_mac[0] = readl(base + NvRegMacAddrA);
 	np->orig_mac[1] = readl(base + NvRegMacAddrB);
 
+/* RAO FIXME - if mac address is reversed (test oid) then flip into dev_addr then strait copy to orig mac... THEN continue */
+
+        if ( ( (np->orig_mac[0] >> 16) & 0xff ) == 0x17 &&
+             ( (np->orig_mac[0] >>  8) & 0xff ) == 0x16 &&
+             ( (np->orig_mac[0] >>  0) & 0xff ) == 0x00 )
+                { /* mac address is reveresed (PXE bug) so copy into dev_addr */
+                        dev->dev_addr[0] = (np->orig_mac[1] >>  8) & 0xff;
+                        dev->dev_addr[1] = (np->orig_mac[1] >>  0) & 0xff;
+                        dev->dev_addr[2] = (np->orig_mac[0] >> 24) & 0xff;
+                        dev->dev_addr[3] = (np->orig_mac[0] >> 16) & 0xff;
+                        dev->dev_addr[4] = (np->orig_mac[0] >>  8) & 0xff;
+                        dev->dev_addr[5] = (np->orig_mac[0] >>  0) & 0xff;
+                  /* Now copy "reveresed" into orig_mac */
+                        np->orig_mac[0] = (dev->dev_addr[0] << 0) + (dev->dev_addr[1] << 8) +
+                                          (dev->dev_addr[2] << 16) + (dev->dev_addr[3] << 24);
+                        np->orig_mac[1] = (dev->dev_addr[4] << 0) + (dev->dev_addr[5] << 8);
+
+                        printk(KERN_ERR "forcedeth: REVERSED Mac address detected: %02x:%02x:%02x:%02x:%02x:%02x\n",
+                        dev->dev_addr[0], dev->dev_addr[1], dev->dev_addr[2],
+                        dev->dev_addr[3], dev->dev_addr[4], dev->dev_addr[5]);
+	                printk(KERN_ERR "Fixing MAC address.\n");
+                }
+
 	dev->dev_addr[0] = (np->orig_mac[1] >>  8) & 0xff;
 	dev->dev_addr[1] = (np->orig_mac[1] >>  0) & 0xff;
 	dev->dev_addr[2] = (np->orig_mac[0] >> 24) & 0xff;
 	dev->dev_addr[3] = (np->orig_mac[0] >> 16) & 0xff;
 	dev->dev_addr[4] = (np->orig_mac[0] >>  8) & 0xff;
 	dev->dev_addr[5] = (np->orig_mac[0] >>  0) & 0xff;
+       	printk(KERN_ERR "forcedeth: using Mac address: %02x:%02x:%02x:%02x:%02x:%02x\n",
+			dev->dev_addr[0], dev->dev_addr[1], dev->dev_addr[2],
+			dev->dev_addr[3], dev->dev_addr[4], dev->dev_addr[5]);
+
 #if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,13)
 	memcpy(dev->perm_addr, dev->dev_addr, dev->addr_len);
 

Re: forcedeth net driver: reverse mac address after pxe boot

[Denis Vlasenko]

On Wednesday 04 October 2006 18:19, Alex Owen wrote:
> I an issue with the forcedeth driver when used after the BIOS PXE routines.
> When booting directly from disk the ethernet MAC address is normal.
> eg: 00:16:17:xx:yy:zz
> But is the BIOS PXE boot stack loads pxelinux which then boots the
> local disk, or if pxelinux loads a kernel/initrd, then the MAC address
> detected by the forcedeth linux driver is reversed.
> eg zz:yy:xx:17:16:00
> 
> This is obviously causes me a problem with automated installs started
> via PXE boot as the installed cannot DHCP as the MAC address is wrong.
> 
> I have read some of the bug reports and LKML threads on WOL and
> forcedeth and I have looked at the code of the driver... most closely
> forcedeth v57 as per comment #22 of
> http://bugzilla.kernel.org/show_bug.cgi?id=6604
> 
> My understanding of the code is that the driver determines the cards
> MAC address by reading from registers in the ethernet controller, but
> that for reasons best known to nvidia this  address backwards and so
> the driver "fixes" this by itself reversing the read values and
> writing them back to the controller.
> 
> This normally works ok and there has been some work to put the old
> "wrong" MAC address back at close down to get WOL to work to.
> 
> Enter PXE... Booting from PXE (in BIOS) seems to "fixup" the "wrong"
> MAC address so when the driver determines the cards MAC address by
> reading from registers in the ethernet controller then MAC address
> there is now CORRECT. The driver however assumes it is reversed and in
> trying to "fix" the MAC address is infact writes back the
> revesed/broken MAC back to the controller.
> 
> The obvious fix for this is to try and read the MAC address from the
> canonical location... ie where is the source of the address writen
> into the controlers registers at power on? But do we know where that
> may be?
> 
> The other solution would be unconditionally reset the controler to
> it's power on state then use the current logic? can we reset the
> controller via software?
> There does seem to be an nv_mac_reset function... and this does seem
> to be called if the card has a capability DEV_HAS_POWER_CONTROL but it
> is called in nv_open() while the MAC is read in nv_probe().
> 
> Perhaps we need to unconditionally run nv_mac_reset just before
> reading the MAC in nv_probe() ?
> 
> Anyway I hope that someone who knows kernel internals and this driver
> inparticular feels the urge to look at this!

Also MAC addresses cannot be arbitrary. If nothing else would work,
we can add OUI numbers (3 most significant bytes of MAC)
of known suppliers of nvidia eth and check whether they are
in lower bytes or in higher ones.
--
vda

Re: forcedeth net driver: reverse mac address after pxe boot

[John W. Linville]

On Wed, Oct 04, 2006 at 05:19:20PM +0100, Alex Owen wrote:

> The obvious fix for this is to try and read the MAC address from the
> canonical location... ie where is the source of the address writen
> into the controlers registers at power on? But do we know where that
> may be?

This seems like The Right Thing (TM) to me, but we need someone from
NVidia(?) to provide that information.  Ayaz?

> The other solution would be unconditionally reset the controler to
> it's power on state then use the current logic? can we reset the
> controller via software?

This seems like a plausible alternative.

The MAC address validation schemes suggested by others would probably
"work", but they would be a bit fragile.  For example, every new vendor
of forcedeth hardware would have a new OUI to be added to the list.

John
-- 
John W. Linville
linville@tuxdriver.com

Re: forcedeth net driver: reverse mac address after pxe boot

[Carl-Daniel Hailfinger]

John W. Linville wrote:
> On Wed, Oct 04, 2006 at 05:19:20PM +0100, Alex Owen wrote:
> 
>> The obvious fix for this is to try and read the MAC address from the
>> canonical location... ie where is the source of the address writen
>> into the controlers registers at power on? But do we know where that
>> may be?
> 
> This seems like The Right Thing (TM) to me, but we need someone from
> NVidia(?) to provide that information.  Ayaz?

The canonical location of the "original" MAC address is where we write
back the reversed MAC address. So that won't work.

>> The other solution would be unconditionally reset the controler to
>> it's power on state then use the current logic? can we reset the
>> controller via software?
> 
> This seems like a plausible alternative.

AFAIK there is no way to do that (except powering off the machine).

> The MAC address validation schemes suggested by others would probably
> "work", but they would be a bit fragile.  For example, every new vendor
> of forcedeth hardware would have a new OUI to be added to the list.

Nooooo! That would be a nightmare. Especially because some BIOSes
allow users to set the MAC address. All those setups would be broken
instantly by that solution. (Yes, you can probably be clever, but
then you don't have a chance to fix this generically.)


Regards,
Carl-Daniel
-- 
http://www.hailfinger.org/

Re: forcedeth net driver: reverse mac address after pxe boot

[John W. Linville]

On Thu, Oct 05, 2006 at 08:35:05PM +0200, Carl-Daniel Hailfinger wrote:
> John W. Linville wrote:
> > On Wed, Oct 04, 2006 at 05:19:20PM +0100, Alex Owen wrote:
> > 
> >> The obvious fix for this is to try and read the MAC address from the
> >> canonical location... ie where is the source of the address writen
> >> into the controlers registers at power on? But do we know where that
> >> may be?
> > 
> > This seems like The Right Thing (TM) to me, but we need someone from
> > NVidia(?) to provide that information.  Ayaz?
> 
> The canonical location of the "original" MAC address is where we write
> back the reversed MAC address. So that won't work.

I think you misunderstand the suggestion (which is admittedly based
on supposition).

On most devices, the MAC address is programmed into a register at
runtime.  It is not "burned-in" to the device itself.  Instead it is
usually stored in some sort of eeprom/nvram/flash/whatever.  The driver
retrieves it at runtime from the nvram and programs it into the device.

In this case, the forcedeth driver is retrieving the MAC address from
a register, reversing it, and writing it back to the _same_ register.
Experience suggests that this register is unlikely to have "magically"
received that information.  The supposition is that instead some
firmware has pre-loaded the register, perhaps at IPL?

It is possible that the device itself is loading the MAC address
from e.g. a serial eeprom tied to the device and inaccessible to
the CPU.  If that is the case, there may be no other solution than
the current silliness.  Since the driver is reversed engineered,
the current silliness (and a prayer for fixed PXE firmware) is the
only good alternative we have.

John
-- 
John W. Linville
linville@tuxdriver.com

RE: forcedeth net driver: reverse mac address after pxe boot

[Ayaz Abdulla]

The BIOS will write to the mac address register. The address will be
written in reverse order. 

This bug has already been fixed in the PXE code. Can you let me know
what version of PXE you are running (it should display the version on
the screen during boot up)?

Thanks,
Ayaz


-----Original Message-----
From: John W. Linville [mailto:linville@tuxdriver.com] 
Sent: Thursday, October 05, 2006 12:31 PM
To: Carl-Daniel Hailfinger
Cc: Alex Owen; linux-kernel@vger.kernel.org; Ayaz Abdulla; H. Peter
Anvin; Alan Cox
Subject: Re: forcedeth net driver: reverse mac address after pxe boot


On Thu, Oct 05, 2006 at 08:35:05PM +0200, Carl-Daniel Hailfinger wrote:
> John W. Linville wrote:
> > On Wed, Oct 04, 2006 at 05:19:20PM +0100, Alex Owen wrote:
> > 
> >> The obvious fix for this is to try and read the MAC address from
the
> >> canonical location... ie where is the source of the address writen
> >> into the controlers registers at power on? But do we know where
that
> >> may be?
> > 
> > This seems like The Right Thing (TM) to me, but we need someone from
> > NVidia(?) to provide that information.  Ayaz?
> 
> The canonical location of the "original" MAC address is where we write
> back the reversed MAC address. So that won't work.

I think you misunderstand the suggestion (which is admittedly based
on supposition).

On most devices, the MAC address is programmed into a register at
runtime.  It is not "burned-in" to the device itself.  Instead it is
usually stored in some sort of eeprom/nvram/flash/whatever.  The driver
retrieves it at runtime from the nvram and programs it into the device.

In this case, the forcedeth driver is retrieving the MAC address from
a register, reversing it, and writing it back to the _same_ register.
Experience suggests that this register is unlikely to have "magically"
received that information.  The supposition is that instead some
firmware has pre-loaded the register, perhaps at IPL?

It is possible that the device itself is loading the MAC address
from e.g. a serial eeprom tied to the device and inaccessible to
the CPU.  If that is the case, there may be no other solution than
the current silliness.  Since the driver is reversed engineered,
the current silliness (and a prayer for fixed PXE firmware) is the
only good alternative we have.

John
-- 
John W. Linville
linville@tuxdriver.com
-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

Re: forcedeth net driver: reverse mac address after pxe boot

[Alex Owen]

On 05/10/06, Ayaz Abdulla <AAbdulla@nvidia.com> wrote:
> The BIOS will write to the mac address register. The address will be
> written in reverse order.
(Why does the BIOS have to write it out in reverse order? Seems a bit
odd to my simple mind!)

> This bug has already been fixed in the PXE code. Can you let me know
> what version of PXE you are running (it should display the version on
> the screen during boot up)?
I was booting with "Nvidia Boot Agent 240.0532" when I encountered this issue.

Alex Owen

RE: forcedeth net driver: reverse mac address after pxe boot

[Ayaz Abdulla]

This has been fixed in version "243.0537". You will have to request an
updated BIOS from your board vendor.

Regards,
Ayaz


-----Original Message-----
From: Alex Owen [mailto:r.alex.owen@gmail.com] 
Sent: Friday, October 06, 2006 7:37 AM
To: Ayaz Abdulla
Cc: linux-kernel@vger.kernel.org
Subject: Re: forcedeth net driver: reverse mac address after pxe boot


On 05/10/06, Ayaz Abdulla <AAbdulla@nvidia.com> wrote:
> The BIOS will write to the mac address register. The address will be
> written in reverse order.
(Why does the BIOS have to write it out in reverse order? Seems a bit
odd to my simple mind!)

> This bug has already been fixed in the PXE code. Can you let me know
> what version of PXE you are running (it should display the version on
> the screen during boot up)?
I was booting with "Nvidia Boot Agent 240.0532" when I encountered this
issue.

Alex Owen
-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

Re: forcedeth net driver: reverse mac address after pxe boot

[John W. Linville]

On Fri, Oct 06, 2006 at 10:29:04AM -0700, Ayaz Abdulla wrote:
> This has been fixed in version "243.0537". You will have to request an
> updated BIOS from your board vendor.

Ayaz,

Can you explain the whole "reverse-order MAC address" thing?
Why/how does it end-up in that register backwards in the first place?
Does it serve some purpose that way?  Or is it just a bug that we
have to live with?

John
-- 
John W. Linville
linville@tuxdriver.com

RE: forcedeth net driver: reverse mac address after pxe boot

[Ayaz Abdulla]

It is just a legacy bug that we have to live with. Newer chipsets will
have the correct format in the BIOS.

With the latest forcedeth driver and latest PXE you should not have any
issues.

Regards,
Ayaz


-----Original Message-----
From: John W. Linville [mailto:linville@tuxdriver.com] 
Sent: Friday, October 06, 2006 12:12 PM
To: Ayaz Abdulla
Cc: Alex Owen; linux-kernel@vger.kernel.org
Subject: Re: forcedeth net driver: reverse mac address after pxe boot


On Fri, Oct 06, 2006 at 10:29:04AM -0700, Ayaz Abdulla wrote:
> This has been fixed in version "243.0537". You will have to request an
> updated BIOS from your board vendor.

Ayaz,

Can you explain the whole "reverse-order MAC address" thing?
Why/how does it end-up in that register backwards in the first place?
Does it serve some purpose that way?  Or is it just a bug that we
have to live with?

John
-- 
John W. Linville
linville@tuxdriver.com
-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

Re: forcedeth net driver: reverse mac address after pxe boot

[Andrew de Quincey]

On Thursday 05 October 2006 20:31, John W. Linville wrote:
> On Thu, Oct 05, 2006 at 08:35:05PM +0200, Carl-Daniel Hailfinger wrote:
> > John W. Linville wrote:
> > > On Wed, Oct 04, 2006 at 05:19:20PM +0100, Alex Owen wrote:
> > >> The obvious fix for this is to try and read the MAC address from the
> > >> canonical location... ie where is the source of the address writen
> > >> into the controlers registers at power on? But do we know where that
> > >> may be?
> > >
> > > This seems like The Right Thing (TM) to me, but we need someone from
> > > NVidia(?) to provide that information.  Ayaz?
> >
> > The canonical location of the "original" MAC address is where we write
> > back the reversed MAC address. So that won't work.
>
> I think you misunderstand the suggestion (which is admittedly based
> on supposition).
>
> On most devices, the MAC address is programmed into a register at
> runtime.  It is not "burned-in" to the device itself.  Instead it is
> usually stored in some sort of eeprom/nvram/flash/whatever.  The driver
> retrieves it at runtime from the nvram and programs it into the device.
>
> In this case, the forcedeth driver is retrieving the MAC address from
> a register, reversing it, and writing it back to the _same_ register.
> Experience suggests that this register is unlikely to have "magically"
> received that information.  The supposition is that instead some
> firmware has pre-loaded the register, perhaps at IPL?
>
> It is possible that the device itself is loading the MAC address
> from e.g. a serial eeprom tied to the device and inaccessible to
> the CPU.  If that is the case, there may be no other solution than
> the current silliness.  Since the driver is reversed engineered,
> the current silliness (and a prayer for fixed PXE firmware) is the
> only good alternative we have.

I think my conclusion was that the BIOS was responsible for retrieving the MAC 
from "somewhere" (perhaps a BIOS specific location) and writing it into the 
register that forcedeth users. 

AFAIR, the binary driver only ever accessed the location forcedeth is 
accessing, and it also did the same byte swapping process. I never bothered 
to take the BIOS itelf apart to find out where it was getting the MAC from to 
begin with.