Intel Microcode loader, tg3 driver, and the -rc8-mmotd New World Order firmware...

Intel Microcode loader, tg3 driver, and the -rc8-mmotd New World Order firmware...

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1071 bytes --]

I built the -rc8-mmotd kernel, and built it with 'CONFIG_FIRMWARE_IN_KERNEL=n'.
Lo and behold, the microcode.ko was now doing a request_firmware for
'intel-ucode/06-0f-06' (which makes sense, the Core2 Duo in this laptop is
family 6, model 15, stepping 6).  However, what I had in /lib/firmware was
the Intel-distributed 'microcode.dat' with updates for all the CPUs (which
used to work in times past).

What's the magic incantation to take the microcode.dat and create something
that the firmware driver is willing to use, or is this all borked up and
I need to do a major rethink or fix my config?

Another minor annoyance - the tg3 driver, when builtin to the kernel, wouldn't
load the microcode in this config. It complained it couldn't get 'tigon/tg3_tos.bin',
but that's almost certainly an issue with Fedora's 'nash' firmware support and/or
my understanding of it - I got *that* part working by dropping the file into
/lib/firmware/tigon and building the driver as a module. Fortunately, I don't
need the tg3 driver to boot far enough to get a full udev running.


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Intel Microcode loader, tg3 driver, and the -rc8-mmotd New World Order firmware...

[Andrew Morton]

On Wed, 02 Jul 2008 21:32:38 -0400 Valdis.Kletnieks@vt.edu wrote:

> I built the -rc8-mmotd kernel, and built it with 'CONFIG_FIRMWARE_IN_KERNEL=n'.
> Lo and behold, the microcode.ko was now doing a request_firmware for
> 'intel-ucode/06-0f-06' (which makes sense, the Core2 Duo in this laptop is
> family 6, model 15, stepping 6).  However, what I had in /lib/firmware was
> the Intel-distributed 'microcode.dat' with updates for all the CPUs (which
> used to work in times past).
> 
> What's the magic incantation to take the microcode.dat and create something
> that the firmware driver is willing to use, or is this all borked up and
> I need to do a major rethink or fix my config?
> 
> Another minor annoyance - the tg3 driver, when builtin to the kernel, wouldn't
> load the microcode in this config. It complained it couldn't get 'tigon/tg3_tos.bin',
> but that's almost certainly an issue with Fedora's 'nash' firmware support and/or
> my understanding of it - I got *that* part working by dropping the file into
> /lib/firmware/tigon and building the driver as a module. Fortunately, I don't
> need the tg3 driver to boot far enough to get a full udev running.
> 

(cc dwmw2)

Re: Intel Microcode loader, tg3 driver, and the -rc8-mmotd New World Order firmware...

[Tigran Aivazian]

Hi Valdis,

On Wed, 2 Jul 2008 Valdis.Kletnieks@vt.edu wrote:

> I built the -rc8-mmotd kernel, and built it with 'CONFIG_FIRMWARE_IN_KERNEL=n'.
> Lo and behold, the microcode.ko was now doing a request_firmware for
> 'intel-ucode/06-0f-06' (which makes sense, the Core2 Duo in this laptop is
> family 6, model 15, stepping 6).  However, what I had in /lib/firmware was
> the Intel-distributed 'microcode.dat' with updates for all the CPUs (which
> used to work in times past).
>
> What's the magic incantation to take the microcode.dat and create something
> that the firmware driver is willing to use, or is this all borked up and
> I need to do a major rethink or fix my config?

that's because it expects the Intel-supplied microcode data and you are 
using the old style microcode.dat data.

Kind regards
Tigran

>
> Another minor annoyance - the tg3 driver, when builtin to the kernel, wouldn't
> load the microcode in this config. It complained it couldn't get 'tigon/tg3_tos.bin',
> but that's almost certainly an issue with Fedora's 'nash' firmware support and/or
> my understanding of it - I got *that* part working by dropping the file into
> /lib/firmware/tigon and building the driver as a module. Fortunately, I don't
> need the tg3 driver to boot far enough to get a full udev running.
>
>

Re: Intel Microcode loader, tg3 driver, and the -rc8-mmotd New World Order firmware...

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1960 bytes --]

On Thu, 03 Jul 2008 07:17:16 BST, Tigran Aivazian said:
> Hi Valdis,
> 
> On Wed, 2 Jul 2008 Valdis.Kletnieks@vt.edu wrote:
> 
> > I built the -rc8-mmotd kernel, and built it with 'CONFIG_FIRMWARE_IN_KERNEL
=n'.
> > Lo and behold, the microcode.ko was now doing a request_firmware for
> > 'intel-ucode/06-0f-06' (which makes sense, the Core2 Duo in this laptop is
> > family 6, model 15, stepping 6).  However, what I had in /lib/firmware was
> > the Intel-distributed 'microcode.dat' with updates for all the CPUs (which
> > used to work in times past).
> >
> > What's the magic incantation to take the microcode.dat and create something
> > that the firmware driver is willing to use, or is this all borked up and
> > I need to do a major rethink or fix my config?
> 
> that's because it expects the Intel-supplied microcode data and you are 
> using the old style microcode.dat data.

I fed it the stuff I downloaded today from this URL:

http://downloadcenter.intel.com/filter_results.aspx?strTypes=all&ProductID=2643&OSFullName=Linux*&lang=eng&strOSs=39&submit=Go!

which gets me a microcode-20080401.dat that does the same thing.  Is there
some *other* Intel-supplied microcode data I should be getting instead?

(If I should be looking elsewhere, can somebody fix http://www.urbanmyth.org/microcode/
to point somewhere other than http://downloadcenter.intel.com/default.aspx so
people go to the right place?)
> 
> Kind regards
> Tigran
> 
> >
> > Another minor annoyance - the tg3 driver, when builtin to the kernel, would
n't
> > load the microcode in this config. It complained it couldn't get 'tigon/tg3
_tos.bin',
> > but that's almost certainly an issue with Fedora's 'nash' firmware support 
and/or
> > my understanding of it - I got *that* part working by dropping the file int
o
> > /lib/firmware/tigon and building the driver as a module. Fortunately, I don
't
> > need the tg3 driver to boot far enough to get a full udev running.
> >
> >
> 


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Intel Microcode loader, tg3 driver, and the -rc8-mmotd New World Order firmware...

[Tigran Aivazian]

On Thu, 3 Jul 2008 Valdis.Kletnieks@vt.edu wrote:

> On Thu, 03 Jul 2008 07:17:16 BST, Tigran Aivazian said:
>> Hi Valdis,
>>
>> On Wed, 2 Jul 2008 Valdis.Kletnieks@vt.edu wrote:
>>
>>> I built the -rc8-mmotd kernel, and built it with 'CONFIG_FIRMWARE_IN_KERNEL
> =n'.
>>> Lo and behold, the microcode.ko was now doing a request_firmware for
>>> 'intel-ucode/06-0f-06' (which makes sense, the Core2 Duo in this laptop is
>>> family 6, model 15, stepping 6).  However, what I had in /lib/firmware was
>>> the Intel-distributed 'microcode.dat' with updates for all the CPUs (which
>>> used to work in times past).
>>>
>>> What's the magic incantation to take the microcode.dat and create something
>>> that the firmware driver is willing to use, or is this all borked up and
>>> I need to do a major rethink or fix my config?
>>
>> that's because it expects the Intel-supplied microcode data and you are
>> using the old style microcode.dat data.
>
> I fed it the stuff I downloaded today from this URL:
>
> http://downloadcenter.intel.com/filter_results.aspx?strTypes=all&ProductID=2643&OSFullName=Linux*&lang=eng&strOSs=39&submit=Go!
>
> which gets me a microcode-20080401.dat that does the same thing.  Is there
> some *other* Intel-supplied microcode data I should be getting instead?

Oh, sorry, I assumed that Intel distribute the data in the format that 
driver expects.

Kind regards
Tigran

Re: Intel Microcode loader, tg3 driver, and the -rc8-mmotd New World Order firmware...

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 2077 bytes --]

On Thu, 2008-07-03 at 07:44 +0100, Tigran Aivazian wrote:
> On Thu, 3 Jul 2008 Valdis.Kletnieks@vt.edu wrote:
> 
> > On Thu, 03 Jul 2008 07:17:16 BST, Tigran Aivazian said:
> >> Hi Valdis,
> >>
> >> On Wed, 2 Jul 2008 Valdis.Kletnieks@vt.edu wrote:
> >>
> >>> I built the -rc8-mmotd kernel, and built it with 'CONFIG_FIRMWARE_IN_KERNEL
> > =n'.
> >>> Lo and behold, the microcode.ko was now doing a request_firmware for
> >>> 'intel-ucode/06-0f-06' (which makes sense, the Core2 Duo in this laptop is
> >>> family 6, model 15, stepping 6).  However, what I had in /lib/firmware was
> >>> the Intel-distributed 'microcode.dat' with updates for all the CPUs (which
> >>> used to work in times past).
> >>>
> >>> What's the magic incantation to take the microcode.dat and create something
> >>> that the firmware driver is willing to use, or is this all borked up and
> >>> I need to do a major rethink or fix my config?
> >>
> >> that's because it expects the Intel-supplied microcode data and you are
> >> using the old style microcode.dat data.
> >
> > I fed it the stuff I downloaded today from this URL:
> >
> > http://downloadcenter.intel.com/filter_results.aspx?strTypes=all&ProductID=2643&OSFullName=Linux*〈=eng&strOSs=39&submit=Go!
> >
> > which gets me a microcode-20080401.dat that does the same thing.  Is there
> > some *other* Intel-supplied microcode data I should be getting instead?
> 
> Oh, sorry, I assumed that Intel distribute the data in the format that 
> driver expects.

I think the kernel _does_ manage to extract just the part it wants from
the data, but it expects the data in binary form. Drop the attached
files in /lib/udev/microcode.sh and /etc/udev/rules.d/51-microcode.rules
respectively. (There's probably a better way to handle this kind of
thing by putting hooks in firmware.sh rather than using a special rule
which overrides it?)

The recent firmware changes haven't modified this. The important change
seems to have been here (in 2006):

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=a30a6a2c

-- 
dwmw2

[-- Attachment #2: microcode.sh --]
[-- Type: application/x-shellscript, Size: 548 bytes --]

[-- Attachment #3: 51-microcode.rules --]
[-- Type: text/plain, Size: 158 bytes --]

# firmware requests for intel-ucode/* are satisfied by microcode_ctl
SUBSYSTEM=="firmware", ENV{FIRMWARE}=="intel-ucode/*", ACTION=="add", RUN="microcode.sh"

Re: Intel Microcode loader, tg3 driver, and the -rc8-mmotd New World Order firmware...

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1085 bytes --]

On Thu, 03 Jul 2008 10:23:48 BST, David Woodhouse said:

> The recent firmware changes haven't modified this. The important change
> seems to have been here (in 2006):
> 
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=a30a6a2c

Aha. That's the part I was missing. :)

>From the changelog for that commit, Shaohua Li wrote:

"with the changes, we should put all intel-ucode/xx-xx-xx microcode files
into the firmware dir (I had a tool to split previous big data file into
small one and later we will release new style data file).  The init script
should be changed to ..."

And apparently I got stuck between the unreleased tool to split the file,
and the release of the new style data file.

Anyhow, it appears the firmware_request() was just a bullet loaded in the
chamber waiting for me to pull the trigger 2 years later by setting
CONFIG_FIRMWARE_IN_KERNEL=n :)

The behavior is explained, and presumably Intel will eventually release
a method of getting the new-format bits, and all will be right with the
world (or at least this part of it.. ;)

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Intel Microcode loader, tg3 driver, and the -rc8-mmotd New World Order firmware...

[David Woodhouse]

On Thu, 2008-07-03 at 06:17 -0400, Valdis.Kletnieks@vt.edu wrote:
> On Thu, 03 Jul 2008 10:23:48 BST, David Woodhouse said:
> 
> > The recent firmware changes haven't modified this. The important change
> > seems to have been here (in 2006):
> > 
> > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=a30a6a2c
> 
> Aha. That's the part I was missing. :)
> 
> From the changelog for that commit, Shaohua Li wrote:
> 
> "with the changes, we should put all intel-ucode/xx-xx-xx microcode files
> into the firmware dir (I had a tool to split previous big data file into
> small one and later we will release new style data file).  The init script
> should be changed to ..."
> 
> And apparently I got stuck between the unreleased tool to split the file,
> and the release of the new style data file.

That 'new style' data file is just the binary output of the
microcode_ctl tool. The kernel is still capable of finding the section
it requires, when fed the whole blob.

> Anyhow, it appears the firmware_request() was just a bullet loaded in the
> chamber waiting for me to pull the trigger 2 years later by setting
> CONFIG_FIRMWARE_IN_KERNEL=n :)

No, the recent firmware changes haven't modified this. The
FIRMWARE_IN_KERNEL option makes no difference here.

> The behavior is explained, and presumably Intel will eventually release
> a method of getting the new-format bits, and all will be right with the
> world (or at least this part of it.. ;)

Some would say that Intel already released a method for making it work,
in the form of the email to which you're replying... does that udev
configuration not work for you?

-- 
dwmw2

Re: Intel Microcode loader, tg3 driver, and the -rc8-mmotd New World Order firmware...

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 930 bytes --]

On Thu, 03 Jul 2008 11:30:29 BST, David Woodhouse said:

> > Anyhow, it appears the firmware_request() was just a bullet loaded in the
> > chamber waiting for me to pull the trigger 2 years later by setting
> > CONFIG_FIRMWARE_IN_KERNEL=n :)
> 
> No, the recent firmware changes haven't modified this. The
> FIRMWARE_IN_KERNEL option makes no difference here.

"D'Oh!" -- H. Simpson.
"Nevermind" - Emily Litella.

Actually, FIRMWARE_IN_KERNEL *did* make a difference - it busticated the tg3
driver and while fighting with the 'firmware: requesting tigon/tg3_tso.bin'
errors I had just inflicted on myself, I found the 'requesting intel-ucode/060f06'
messages as well - and totally failed to notice that the *previous* kernel
and initscripts had *all along* been doing a modprobe, which generated 2 requests
(one per CPU) which failed, and then the initscript ran microcode_ctl *anyhow*,
papering over the two failed requests.. ;)

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Intel Microcode loader, tg3 driver, and the -rc8-mmotd New World Order firmware...

[David Woodhouse]

On Thu, 2008-07-03 at 07:34 -0400, Valdis.Kletnieks@vt.edu wrote:
> Actually, FIRMWARE_IN_KERNEL *did* make a difference - it busticated the tg3
> driver and while fighting with the 'firmware: requesting tigon/tg3_tso.bin'
> errors I had just inflicted on myself, 

Just to confirm: that works either with CONFIG_FIRMWARE_IN_KERNEL=y or
if you run 'make INSTALL_FW_PATH=/lib/firmware firmware_install', right?

I'm not entirely sure whether we should make 'make firmware_install'
install to /lib/firmware by default, or to $(O)/usr/lib/firmware as it
does at the moment.

The former is consistent with 'make modules_install', and the latter is
consistent with 'make headers_install'. Both of which have good reasons
for being the way they are.

-- 
dwmw2

Re: Intel Microcode loader, tg3 driver, and the -rc8-mmotd New World Order firmware...

[Kay Sievers]

On Thu, 2008-07-03 at 13:21 +0100, David Woodhouse wrote:
> On Thu, 2008-07-03 at 07:34 -0400, Valdis.Kletnieks@vt.edu wrote:
> > Actually, FIRMWARE_IN_KERNEL *did* make a difference - it busticated the tg3
> > driver and while fighting with the 'firmware: requesting tigon/tg3_tso.bin'
> > errors I had just inflicted on myself, 
> 
> Just to confirm: that works either with CONFIG_FIRMWARE_IN_KERNEL=y or
> if you run 'make INSTALL_FW_PATH=/lib/firmware firmware_install', right?
> 
> I'm not entirely sure whether we should make 'make firmware_install'
> install to /lib/firmware by default, or to $(O)/usr/lib/firmware as it
> does at the moment.
> 
> The former is consistent with 'make modules_install', and the latter is
> consistent with 'make headers_install'. Both of which have good reasons
> for being the way they are.

Headers are not needed for bootup, firmware might be. :) But /usr might
be on a different partition/disk/storage at the time we need it, right?

I would say /usr/lib/firmware should not even exist, udev does
intentionally not even look there.

Thanks,
Kay

Re: Intel Microcode loader, tg3 driver, and the -rc8-mmotd New World Order firmware...

[David Woodhouse]

On Thu, 2008-07-03 at 14:41 +0200, Kay Sievers wrote:
> Headers are not needed for bootup, firmware might be. :) But /usr
> might be on a different partition/disk/storage at the time we need it,
> right?
> 
> I would say /usr/lib/firmware should not even exist, udev does
> intentionally not even look there.

Not /usr/lib/firmware. Currently, when you run 'make firmware_install',
it gets installed to usr/lib/firmware _within_ the kernel build
directory. And you're expected to copy it from there (or override with
'make INSTALL_FW_PATH=/lib/firmware firmware_install')

-- 
dwmw2

Re: Intel Microcode loader, tg3 driver, and the -rc8-mmotd New World Order firmware...

[Kay Sievers]

On Thu, 2008-07-03 at 13:45 +0100, David Woodhouse wrote:
> On Thu, 2008-07-03 at 14:41 +0200, Kay Sievers wrote:
> > Headers are not needed for bootup, firmware might be. :) But /usr
> > might be on a different partition/disk/storage at the time we need it,
> > right?
> > 
> > I would say /usr/lib/firmware should not even exist, udev does
> > intentionally not even look there.
> 
> Not /usr/lib/firmware. Currently, when you run 'make firmware_install',
> it gets installed to usr/lib/firmware _within_ the kernel build
> directory. And you're expected to copy it from there (or override with
> 'make INSTALL_FW_PATH=/lib/firmware firmware_install')

Ah, ok. So by default "make modules_install" installs in the rootfs, but
"make firmware_install" installs in the build directory? Hmm ...

Thanks,
Kay

Re: Intel Microcode loader, tg3 driver, and the -rc8-mmotd New World Order firmware...

[David Woodhouse]

On Thu, 2008-07-03 at 15:03 +0200, Kay Sievers wrote:
> Ah, ok. So by default "make modules_install" installs in the rootfs,
> but "make firmware_install" installs in the build directory?

Right. And 'make headers_install' installs in the build directory, as I
said. And I'm not entirely _which_ one 'firmware_install' should do.

-- 
dwmw2

Re: Intel Microcode loader, tg3 driver, and the -rc8-mmotd New World Order firmware...

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1109 bytes --]

On Thu, 03 Jul 2008 15:03:32 +0200, Kay Sievers said:
> On Thu, 2008-07-03 at 13:45 +0100, David Woodhouse wrote:
> > On Thu, 2008-07-03 at 14:41 +0200, Kay Sievers wrote:
> > > Headers are not needed for bootup, firmware might be. :) But /usr
> > > might be on a different partition/disk/storage at the time we need it,
> > > right?
> > > 
> > > I would say /usr/lib/firmware should not even exist, udev does
> > > intentionally not even look there.
> > 
> > Not /usr/lib/firmware. Currently, when you run 'make firmware_install',
> > it gets installed to usr/lib/firmware _within_ the kernel build
> > directory. And you're expected to copy it from there (or override with
> > 'make INSTALL_FW_PATH=/lib/firmware firmware_install')
> 
> Ah, ok. So by default "make modules_install" installs in the rootfs, but
> "make firmware_install" installs in the build directory? Hmm ...

The *odd* part is that make firmware_install isn't symmetric - it drops it
in $O/usr/lib/firmware, but you don't want the final result in /usr/lib,
you want it in /lib.  Maybe it should be dropping it in $O/lib/firmware
instead?

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Intel Microcode loader, tg3 driver, and the -rc8-mmotd New World Order firmware...

[David Woodhouse]

On Thu, 2008-07-03 at 09:54 -0400, Valdis.Kletnieks@vt.edu wrote:
> The *odd* part is that make firmware_install isn't symmetric - it drops it
> in $O/usr/lib/firmware, but you don't want the final result in /usr/lib,
> you want it in /lib.  Maybe it should be dropping it in $O/lib/firmware
> instead?

Not sure that makes sense. $(O)/usr is already used for userspace stuff.
It doesn't necessarily correspond to /usr.

-- 
dwmw2

Re: Intel Microcode loader, tg3 driver, and the -rc8-mmotd New World Order firmware...

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1908 bytes --]

On Thu, 03 Jul 2008 13:21:35 BST, David Woodhouse said:
> On Thu, 2008-07-03 at 07:34 -0400, Valdis.Kletnieks@vt.edu wrote:
> > Actually, FIRMWARE_IN_KERNEL *did* make a difference - it busticated the tg
3
> > driver and while fighting with the 'firmware: requesting tigon/tg3_tso.bin'
> > errors I had just inflicted on myself, 
> 
> Just to confirm: that works either with CONFIG_FIRMWARE_IN_KERNEL=y or
> if you run 'make INSTALL_FW_PATH=/lib/firmware firmware_install', right?

The old config was FIRMWARE_IN_KERNEL=y, and that worked fine because the
microcode was compiled into tg3.o and All Was Good.

My current, *also* working config has FIRMWARE_IN_KERNEL=n, and CONFIG_TIGON3=m.
System comes up, initrd runs, the real root gets loaded, udev gets started,
and when the modprobe happens, /lib/firmware/tigon is there to handle the
request_firmware that gets launched when the module loads.

The *broken* config was FIRMWARE_IN_KERNEL=n, CONFIG_TIGON3=y.  System comes up,
the request_firmware pops *very* early (as in while we're still on the initrd),
and I couldn't figure out how to get Fedora/nash/etc to do the firmware load
from the initrd (nash seems to want it in /firmware/tigon3, but putting it
there didn't help any).  By the time udev gets launched, that request is DOA
and the interface doesn't come up.  But as I said, that *seems* to be just a
lack of clue on my part about what nash wants.

> I'm not entirely sure whether we should make 'make firmware_install'
> install to /lib/firmware by default, or to $(O)/usr/lib/firmware as it
> does at the moment.
> 
> The former is consistent with 'make modules_install', and the latter is
> consistent with 'make headers_install'. Both of which have good reasons
> for being the way they are.

Somebody else can decide that one.  Kay does make a point in another mail
that you *really* want it to end up in /lib/firmware for good reasons.

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Intel Microcode loader, tg3 driver, and the -rc8-mmotd New World Order firmware...

[Kay Sievers]

On Thu, 2008-07-03 at 10:23 +0100, David Woodhouse wrote:
> On Thu, 2008-07-03 at 07:44 +0100, Tigran Aivazian wrote:
> > On Thu, 3 Jul 2008 Valdis.Kletnieks@vt.edu wrote:
> > 
> > > On Thu, 03 Jul 2008 07:17:16 BST, Tigran Aivazian said:
> > >> Hi Valdis,
> > >>
> > >> On Wed, 2 Jul 2008 Valdis.Kletnieks@vt.edu wrote:
> > >>
> > >>> I built the -rc8-mmotd kernel, and built it with 'CONFIG_FIRMWARE_IN_KERNEL
> > > =n'.
> > >>> Lo and behold, the microcode.ko was now doing a request_firmware for
> > >>> 'intel-ucode/06-0f-06' (which makes sense, the Core2 Duo in this laptop is
> > >>> family 6, model 15, stepping 6).  However, what I had in /lib/firmware was
> > >>> the Intel-distributed 'microcode.dat' with updates for all the CPUs (which
> > >>> used to work in times past).
> > >>>
> > >>> What's the magic incantation to take the microcode.dat and create something
> > >>> that the firmware driver is willing to use, or is this all borked up and
> > >>> I need to do a major rethink or fix my config?
> > >>
> > >> that's because it expects the Intel-supplied microcode data and you are
> > >> using the old style microcode.dat data.
> > >
> > > I fed it the stuff I downloaded today from this URL:
> > >
> > > http://downloadcenter.intel.com/filter_results.aspx?strTypes=all&ProductID=2643&OSFullName=Linux*〈=eng&strOSs=39&submit=Go!
> > >
> > > which gets me a microcode-20080401.dat that does the same thing.  Is there
> > > some *other* Intel-supplied microcode data I should be getting instead?
> > 
> > Oh, sorry, I assumed that Intel distribute the data in the format that 
> > driver expects.
> 
> I think the kernel _does_ manage to extract just the part it wants from
> the data, but it expects the data in binary form. Drop the attached
> files in /lib/udev/microcode.sh and /etc/udev/rules.d/51-microcode.rules
> respectively. (There's probably a better way to handle this kind of
> thing by putting hooks in firmware.sh rather than using a special rule
> which overrides it?)
> 
> The recent firmware changes haven't modified this. The important change
> seems to have been here (in 2006):
> 
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=a30a6a2c

This commit states:
  "so we don't need the application 'microcode_ctl' to assist".
Seems the kernel code extracts the right section, in the all-in-one
file, for the actual CPU:
  while ((offset = get_next_ucode_from_buffer(&mc, buf, size, offset))

So, maybe a:
  # rewrite firmware file name to all-in-one Intel CPU microcode data file
  SUBSYSTEM=="firmware", ENV{FIRMWARE}=="intel-ucode/*", ENV{FIRMWARE}="intel-ucode/microcode.dat"
would be enough?

Thanks,
Kay

Re: Intel Microcode loader, tg3 driver, and the -rc8-mmotd New World Order firmware...

[David Woodhouse]

On Thu, 2008-07-03 at 12:22 +0200, Kay Sievers wrote:
> So, maybe a:
>   # rewrite firmware file name to all-in-one Intel CPU microcode data file
>   SUBSYSTEM=="firmware", ENV{FIRMWARE}=="intel-ucode/*", ENV{FIRMWARE}="intel-ucode/microcode.dat"
> would be enough?

Nah, it needs the binary form. The actual 'microcode.dat' just looks
like this...

0x00000001,     0x00000013,     0x02062001,     0x00000683,
0x2f0da1b0,     0x00000001,     0x00000001,     0x00000000,
0x00000000,     0x00000000,     0x00000000,     0x00000000,
0xbf5ad468,     0xc79f5237,     0xbd53889e,     0x896bfd13,
0x7adc0c8f,     0x44e9e0bc,     0x1a331fc9,     0x00b0f479,

That's all microcode_ctl actually does; read in the text file and output
the (complete) binary. Hence:
      /sbin/microcode_ctl -f "$DIR/microcode.dat" -d /sys$DEVPATH/data

At a later date, we could make userspace output only the part for the
desired CPU, and rip out the kernel-side code which searches for it
within the big blob. That was presumably the intention in the commit
which changed things. But then again, the kernel-side code still needs
to do a certain amount of sanity checking on what it receives -- so I'm
not sure we gain a huge amount by trying to remove that functionality
from the kernel (not that I've looked hard at the line count).

If we're _not_ going to make userspace provide only what's required, and
we keep the selection code in the kernel, then I don't see the point in
having separate firmware filenames for different cpus. We could just
change the driver to call request_firmware("intel-microcode.bin") and do
this one-off conversion:
	touch /lib/firmware/intel-microcode.bin
	microcode_ctl -d /lib/firmware/intel-microcode.bin

-- 
dwmw2

Re: Intel Microcode loader, tg3 driver, and the -rc8-mmotd New World Order firmware...

[Kay Sievers]

On Thu, 2008-07-03 at 11:42 +0100, David Woodhouse wrote:
> On Thu, 2008-07-03 at 12:22 +0200, Kay Sievers wrote:
> > So, maybe a:
> >   # rewrite firmware file name to all-in-one Intel CPU microcode data file
> >   SUBSYSTEM=="firmware", ENV{FIRMWARE}=="intel-ucode/*", ENV{FIRMWARE}="intel-ucode/microcode.dat"
> > would be enough?
> 
> Nah, it needs the binary form. The actual 'microcode.dat' just looks
> like this...

Ah, I see.

> 0x00000001,     0x00000013,     0x02062001,     0x00000683,
> 0x2f0da1b0,     0x00000001,     0x00000001,     0x00000000,
> 0x00000000,     0x00000000,     0x00000000,     0x00000000,
> 0xbf5ad468,     0xc79f5237,     0xbd53889e,     0x896bfd13,
> 0x7adc0c8f,     0x44e9e0bc,     0x1a331fc9,     0x00b0f479,
> 
> That's all microcode_ctl actually does; read in the text file and output
> the (complete) binary. Hence:
>       /sbin/microcode_ctl -f "$DIR/microcode.dat" -d /sys$DEVPATH/data
> 
> At a later date, we could make userspace output only the part for the
> desired CPU, and rip out the kernel-side code which searches for it
> within the big blob. That was presumably the intention in the commit
> which changed things. But then again, the kernel-side code still needs
> to do a certain amount of sanity checking on what it receives -- so I'm
> not sure we gain a huge amount by trying to remove that functionality
> from the kernel (not that I've looked hard at the line count).

I think, the in-kernel selection logic is nice, people can still
provide a custom .bin file which contains only the needed code for the
actual CPU, and it will not really do any "search", right?

> If we're _not_ going to make userspace provide only what's required, and
> we keep the selection code in the kernel, then I don't see the point in
> having separate firmware filenames for different cpus. We could just
> change the driver to call request_firmware("intel-microcode.bin") and do
> this one-off conversion:
> 	touch /lib/firmware/intel-microcode.bin
> 	microcode_ctl -d /lib/firmware/intel-microcode.bin

Sounds fine, yeah. With the in-kernel search, we should make the file
just a name without the CPU id. This would also make it easier to find
and include the fw into the kernel, right?

We could always export the CPU id somewhere else, so "advanced" tools
could look it up in sysfs and upload in a "custom" way. I don't think
anybody will really need that. :)

Thanks,
Kay

Re: Intel Microcode loader, tg3 driver, and the -rc8-mmotd New World Order firmware...

[Giacomo A. Catenazzi]

[ Added Arjan and the relevant Intel contact]

Tigran Aivazian wrote:
> On Thu, 3 Jul 2008 Valdis.Kletnieks@vt.edu wrote:
> 
>> On Thu, 03 Jul 2008 07:17:16 BST, Tigran Aivazian said:
>>> Hi Valdis,
>>>
>>> On Wed, 2 Jul 2008 Valdis.Kletnieks@vt.edu wrote:
>>>
>>>> I built the -rc8-mmotd kernel, and built it with 
>>>> 'CONFIG_FIRMWARE_IN_KERNEL
>> =n'.
>>>> Lo and behold, the microcode.ko was now doing a request_firmware for
>>>> 'intel-ucode/06-0f-06' (which makes sense, the Core2 Duo in this 
>>>> laptop is
>>>> family 6, model 15, stepping 6).  However, what I had in 
>>>> /lib/firmware was
>>>> the Intel-distributed 'microcode.dat' with updates for all the CPUs 
>>>> (which
>>>> used to work in times past).
>>>>
>>>> What's the magic incantation to take the microcode.dat and create 
>>>> something
>>>> that the firmware driver is willing to use, or is this all borked up 
>>>> and
>>>> I need to do a major rethink or fix my config?
>>>
>>> that's because it expects the Intel-supplied microcode data and you are
>>> using the old style microcode.dat data.
>>
>> I fed it the stuff I downloaded today from this URL:
>>
>> http://downloadcenter.intel.com/filter_results.aspx?strTypes=all&ProductID=2643&OSFullName=Linux*&lang=eng&strOSs=39&submit=Go! 
>>
>>
>> which gets me a microcode-20080401.dat that does the same thing.  Is 
>> there
>> some *other* Intel-supplied microcode data I should be getting instead?
> 
> Oh, sorry, I assumed that Intel distribute the data in the format that 
> driver expects.

There are two format of Intel CPU microcode and two methods to load it.
- old: the microcodes are in a big file, which include multiple
   microcodes (for multiple CPU).  The driver require a char device
   and a user space loader ("microcode_ctl")
- new: one microcode per file, using the 'request_firmware'
   infrastructure. No user space support needed.

Actually Intel provides only the old methods.
There was talks with Arjan and Intel about the distribution format
for the new methods. But I don't have any new.
I think that when the new format is fully specified (directory
structure, tar, gzip,...) Intel will distribute the microcodes
in the new form.

ciao
	cate

Re: Intel Microcode loader, tg3 driver, and the -rc8-mmotd New World Order firmware...

[Arjan van de Ven]

On Thu, 03 Jul 2008 11:24:34 +0200
"Giacomo A. Catenazzi" <cate@debian.org> wrote:

> There are two format of Intel CPU microcode and two methods to load
> it.
> - old: the microcodes are in a big file, which include multiple
>    microcodes (for multiple CPU).  The driver require a char device
>    and a user space loader ("microcode_ctl")
> - new: one microcode per file, using the 'request_firmware'
>    infrastructure. No user space support needed.
> 
> Actually Intel provides only the old methods.
> There was talks with Arjan and Intel about the distribution format
> for the new methods. But I don't have any new.
> I think that when the new format is fully specified (directory
> structure, tar, gzip,...) Intel will distribute the microcodes
> in the new form.


we hope to switch to the new form but there's the small case of
"installed base" using ancient kernels, and it's kind of not nice to
have to release 2 sets. At some point we will switch over though.

Re: Intel Microcode loader, tg3 driver, and the -rc8-mmotd New World Order firmware...

[David Woodhouse]

On Thu, 2008-07-03 at 06:56 -0700, Arjan van de Ven wrote:
> On Thu, 03 Jul 2008 11:24:34 +0200
> "Giacomo A. Catenazzi" <cate@debian.org> wrote:
> 
> > There are two format of Intel CPU microcode and two methods to load
> > it.
> > - old: the microcodes are in a big file, which include multiple
> >    microcodes (for multiple CPU).  The driver require a char device
> >    and a user space loader ("microcode_ctl")
> > - new: one microcode per file, using the 'request_firmware'
> >    infrastructure. No user space support needed.

Actually there are three:

1. The text format 'microcode.dat', including updates for all CPUs.
2. The binary format output by microcode_ctl, still including all CPUs.
3. The smaller files with just the relevant subset of #2.

The kernel (since 2006) can actually take either #2 or #3. The udev
scripts I just posted will use microcode_ctl to feed it #2, when they
find #1 on the file system.

A small amount of extra work in the userspace tool would let those udev
scripts feed #3 to the kernel, and then the code in the kernel to select
the appropriate update could be removed.

> > Actually Intel provides only the old methods.
> > There was talks with Arjan and Intel about the distribution format
> > for the new methods. But I don't have any new.
> > I think that when the new format is fully specified (directory
> > structure, tar, gzip,...) Intel will distribute the microcodes
> > in the new form.

Doesn't the "new format" (#3) involve hard links too, since there are
some cases where the same microcode update applies to more than one CPU
revision?

> we hope to switch to the new form but there's the small case of
> "installed base" using ancient kernels, and it's kind of not nice to
> have to release 2 sets. At some point we will switch over though.

Do we really need to _ship_ it in a different form? It's not exactly
hard to convert from the text form (#1) to either of the other two --
either on the fly in udev scripts, or at installation time.

-- 
dwmw2

Re: Intel Microcode loader, tg3 driver, and the -rc8-mmotd New World Order firmware...

[Giacomo A. Catenazzi]

[recipient clean-up and adding Simon]

David Woodhouse wrote:
> On Thu, 2008-07-03 at 06:56 -0700, Arjan van de Ven wrote:
>> On Thu, 03 Jul 2008 11:24:34 +0200
>> "Giacomo A. Catenazzi" <cate@debian.org> wrote:

> Actually there are three:
> 
> 1. The text format 'microcode.dat', including updates for all CPUs.
> 2. The binary format output by microcode_ctl, still including all CPUs.
> 3. The smaller files with just the relevant subset of #2.

we are diverging :-(

The #2 is not on upstream microcode_ctl. BTW Debian/Ubuntu already
diverge a lot in respect of upstream.
Further, IIRC, Simon slowed/stopped developing microcode_ctl, because
he think (IIRC) that distributions used other solutions or going to
use #3.

So if we need #2, I really want Simon (or someone) that take care
of continuing microcode_ctl upstream and merging distribution
patches.

> The kernel (since 2006) can actually take either #2 or #3. The udev
> scripts I just posted will use microcode_ctl to feed it #2, when they
> find #1 on the file system.


#1 and #2 have some problem in Debian/Ubuntu:
- microcode_ctl is in /usr
- it require a module, so we load the module and we wait that
   the device is created (it is asynchronous)
- initram would require an additional program, to an early load
- split on architectures [32/64bit CPU] (but this is an internal
   autobuild / non-free Debian problem)

> 
> A small amount of extra work in the userspace tool would let those udev
> scripts feed #3 to the kernel, and then the code in the kernel to select
> the appropriate update could be removed.

I don't think Intel want this solution.
Earlier Intel preferred to give us two version of microcode
(new and old kernel) instead of let us to filter out one
short microcode, which trigged a kernel bug.

I'm ready to do such script (really I've already
a similar tool)


> 
>>> Actually Intel provides only the old methods.
>>> There was talks with Arjan and Intel about the distribution format
>>> for the new methods. But I don't have any new.
>>> I think that when the new format is fully specified (directory
>>> structure, tar, gzip,...) Intel will distribute the microcodes
>>> in the new form.
> 
> Doesn't the "new format" (#3) involve hard links too, since there are
> some cases where the same microcode update applies to more than one CPU
> revision?

no, I don't think. Intel documentation on BIOS writing, document
that microcode should be discarded if the header information don't
match with current CPU.  I didn't checked if some microcode
contains same data with different header (which anyway is not feeded
to the CPU)

> 
>> we hope to switch to the new form but there's the small case of
>> "installed base" using ancient kernels, and it's kind of not nice to
>> have to release 2 sets. At some point we will switch over though.
> 
> Do we really need to _ship_ it in a different form? It's not exactly
> hard to convert from the text form (#1) to either of the other two --
> either on the fly in udev scripts, or at installation time.

It was a personal interpretation of Intel wishes.

ciao
	cate