[lm-sensors] i801_smbus 0000:00:1f.3: Bus collision!

[lm-sensors] i801_smbus 0000:00:1f.3: Bus collision!

[Richard Hirst]

This is on an Intel motherboard running an FC3 2.6.10-1.766.FC3smp
kernel with these additional patches:

linux-ipmi-2.6.10-base.diff
linux-i2c-2.6.10-nonblock.diff
linux-i2c-2.6.10-i801_nonblock.diff
linux-ipmi-2.6.10-smb.diff
patch-linux-2.6.11.5-bmcsensors.diff

The board has an mBMC which is basically working in that I can read
the sensors either by 'ipmitool' or 'sensors'.

However, round about every 10 reboots or so, I get the Bus collision
message and the system locks up solid during boot after outputting a
few messages such as

i801_smbus 0000:00:1f.3: Bus collision!
i801_smbus 0000:00:1f.3: Reset failed! (01)
i801_smbus 0000:00:1f.3: Reset failed! (01)
i801_smbus 0000:00:1f.3: Reset failed! (01)
bmcsensors.o: Error 0xff on cmd 0xa/0x23; state = 2; probably fatal.
i801_smbus 0000:00:1f.3: Reset failed! (01)
i801_smbus 0000:00:1f.3: Reset failed! (01)
i801_smbus 0000:00:1f.3: Reset failed! (01)


I'm assuming this indicates that two things have tried to use the
i2c bus at the same time, and I guess one of them is the bmcsensors
code.

I also tried adding code to check 'd->in_use' at the beginning of
i801_start() because it looked to me like in_use should perhaps
normally be zero at that point.  Don't know if that is valid, but
I did get a few indications of i801_start() getting called with
d->in_use non-zero.

It's a single cpu box with hyperthreading, running an SMP kernel.

Anyway,

a) has anyone else seen problems like this?

b) is it a known problem that is likely fixed in later code?

c) could it be a bug triggered by the SMP kernel?

d) any suggestions as to where I go from here ;-)


Thanks,
  Richard

[lm-sensors] i801_smbus 0000:00:1f.3: Bus collision!

[Mark M. Hoffman]

Hello Richard:

* Richard Hirst <rhirst@levanta.com> [2005-09-07 22:36:03 +0100]:
> This is on an Intel motherboard running an FC3 2.6.10-1.766.FC3smp
> kernel with these additional patches:
> 
> linux-ipmi-2.6.10-base.diff
> linux-i2c-2.6.10-nonblock.diff
> linux-i2c-2.6.10-i801_nonblock.diff
> linux-ipmi-2.6.10-smb.diff
> patch-linux-2.6.11.5-bmcsensors.diff

I assume these patches came from here:
http://openipmi.sourceforge.net/

> The board has an mBMC which is basically working in that I can read
> the sensors either by 'ipmitool' or 'sensors'.
> 
> However, round about every 10 reboots or so, I get the Bus collision
> message and the system locks up solid during boot after outputting a
> few messages such as
> 
> i801_smbus 0000:00:1f.3: Bus collision!
> i801_smbus 0000:00:1f.3: Reset failed! (01)
> i801_smbus 0000:00:1f.3: Reset failed! (01)
> i801_smbus 0000:00:1f.3: Reset failed! (01)
> bmcsensors.o: Error 0xff on cmd 0xa/0x23; state = 2; probably fatal.
> i801_smbus 0000:00:1f.3: Reset failed! (01)
> i801_smbus 0000:00:1f.3: Reset failed! (01)
> i801_smbus 0000:00:1f.3: Reset failed! (01)
> 
> 
> I'm assuming this indicates that two things have tried to use the
> i2c bus at the same time, and I guess one of them is the bmcsensors
> code.
> 
> I also tried adding code to check 'd->in_use' at the beginning of
> i801_start() because it looked to me like in_use should perhaps
> normally be zero at that point.  Don't know if that is valid, but
> I did get a few indications of i801_start() getting called with
> d->in_use non-zero.

I looked at the patch *very* briefly... I don't think 'd->in_use' is used to
prevent concurrent accesses.  It looks like a kind of adapter ref-count to me.

I would suggest you use either of sensors or ipmitool, but not both.  If you're
using ipmitool, perhaps make sure you're *not* loading any of the sensors drivers:
eeprom, lm78, etc.

> It's a single cpu box with hyperthreading, running an SMP kernel.
> 
> Anyway,
> 
> a) has anyone else seen problems like this?
> 
> b) is it a known problem that is likely fixed in later code?
> 
> c) could it be a bug triggered by the SMP kernel?
> 
> d) any suggestions as to where I go from here ;-)

Have you tried the openipmi mailing list?
http://lists.sourceforge.net/lists/listinfo/openipmi-developer

Regards,

-- 
Mark M. Hoffman
mhoffman@lightlink.com

[lm-sensors] i801_smbus 0000:00:1f.3: Bus collision!

[Richard Hirst]

On Wed, Sep 07, 2005 at 08:05:53PM -0400, Mark M. Hoffman wrote:
> Hello Richard:
> 
> * Richard Hirst <rhirst@levanta.com> [2005-09-07 22:36:03 +0100]:
> > This is on an Intel motherboard running an FC3 2.6.10-1.766.FC3smp
> > kernel with these additional patches:
> > 
> > linux-ipmi-2.6.10-base.diff
> > linux-i2c-2.6.10-nonblock.diff
> > linux-i2c-2.6.10-i801_nonblock.diff
> > linux-ipmi-2.6.10-smb.diff
> > patch-linux-2.6.11.5-bmcsensors.diff
> 
> I assume these patches came from here:
> http://openipmi.sourceforge.net/

Yes.

> > The board has an mBMC which is basically working in that I can read
> > the sensors either by 'ipmitool' or 'sensors'.
> > 
> > However, round about every 10 reboots or so, I get the Bus collision
> > message and the system locks up solid during boot after outputting a
> > few messages such as
> > 
> > i801_smbus 0000:00:1f.3: Bus collision!
> > i801_smbus 0000:00:1f.3: Reset failed! (01)
> > i801_smbus 0000:00:1f.3: Reset failed! (01)
> > i801_smbus 0000:00:1f.3: Reset failed! (01)
> > bmcsensors.o: Error 0xff on cmd 0xa/0x23; state = 2; probably fatal.
> > i801_smbus 0000:00:1f.3: Reset failed! (01)
> > i801_smbus 0000:00:1f.3: Reset failed! (01)
> > i801_smbus 0000:00:1f.3: Reset failed! (01)
> > 
> > 
> > I'm assuming this indicates that two things have tried to use the
> > i2c bus at the same time, and I guess one of them is the bmcsensors
> > code.
> > 
> > I also tried adding code to check 'd->in_use' at the beginning of
> > i801_start() because it looked to me like in_use should perhaps
> > normally be zero at that point.  Don't know if that is valid, but
> > I did get a few indications of i801_start() getting called with
> > d->in_use non-zero.
> 
> I looked at the patch *very* briefly... I don't think 'd->in_use' is used to
> prevent concurrent accesses.  It looks like a kind of adapter ref-count to me.

Yeah, it could be, but it isn't very clear at all.  It looked like
i801_finish() sets d->finished = 1, and then the next call to
i801_poll() sets in_use = 0.  d->in_use certainly isn't used to
prevent concurrent access, I guess something higher up should be
preventing that.

> I would suggest you use either of sensors or ipmitool, but not both.  If you're
> using ipmitool, perhaps make sure you're *not* loading any of the sensors drivers:
> eeprom, lm78, etc.

OK; at the point where it dies I certianly havn't used ipmitool.
Not sure if startup scripts have used sensors or not.

> > It's a single cpu box with hyperthreading, running an SMP kernel.
> > 
> > Anyway,
> > 
> > a) has anyone else seen problems like this?
> > 
> > b) is it a known problem that is likely fixed in later code?
> > 
> > c) could it be a bug triggered by the SMP kernel?
> > 
> > d) any suggestions as to where I go from here ;-)
> 
> Have you tried the openipmi mailing list?
> http://lists.sourceforge.net/lists/listinfo/openipmi-developer

Yeah, that might be a good place to ask, thanks!

I did get another failure where the console output ended in

i801_smbus 0000:00:1f.3: Reset failed! (01)
i801_smbus 0000:00:1f.3: Reset failed! (01)
i801_smbus 0000:00:1f.3: Reset failed! (01)
i80<4>do_IRQ: stack overflow: 116

which gives me a few more ideas of things to look at.

Thanks,
  Richard

[lm-sensors] i801_smbus 0000:00:1f.3: Bus collision!

[Yani Ioannou]

On 9/8/05, Richard Hirst <rhirst@levanta.com> wrote:

> > > The board has an mBMC which is basically working in that I can read
> > > the sensors either by 'ipmitool' or 'sensors'.
> > >
> > > However, round about every 10 reboots or so, I get the Bus collision
> > > message and the system locks up solid during boot after outputting a
> > > few messages such as
> > >
> > > i801_smbus 0000:00:1f.3: Bus collision!
> > > i801_smbus 0000:00:1f.3: Reset failed! (01)
> > > i801_smbus 0000:00:1f.3: Reset failed! (01)
> > > i801_smbus 0000:00:1f.3: Reset failed! (01)
> > > bmcsensors.o: Error 0xff on cmd 0xa/0x23; state = 2; probably fatal.
> > > i801_smbus 0000:00:1f.3: Reset failed! (01)
> > > i801_smbus 0000:00:1f.3: Reset failed! (01)
> > > i801_smbus 0000:00:1f.3: Reset failed! (01)

The bus collisions are expected - both the IPMI BMC (not bmcsensors
directly) and the i2c modules you have loaded are competing for the
smbus. However although I've done this (loaded both bmcsensors and i2c
modules and got bus collisions) its never resulted in a lockup.

> > > I'm assuming this indicates that two things have tried to use the
> > > i2c bus at the same time, and I guess one of them is the bmcsensors
> > > code.
No, not directly. bmcsensors uses the IPMI messaging interface to send
out queries to the IPMI BMC which then reads the sensors readings over
the i2c/smbus (which is sometimes shared with the system).

> > > I also tried adding code to check 'd->in_use' at the beginning of
> > > i801_start() because it looked to me like in_use should perhaps
> > > normally be zero at that point.  Don't know if that is valid, but
> > > I did get a few indications of i801_start() getting called with
> > > d->in_use non-zero.

The IPMI BMC won't be using the i801 interface driver...its basically
another computer on your motherboard :), I'd suspect the i2c drivers
or i801 are causing the system lockup, as Mark says though you
shouldn't be using both (but it shouldn't lockup IHMO).
 
> OK; at the point where it dies I certianly havn't used ipmitool.
> Not sure if startup scripts have used sensors or not.

You will encounter the same problems if you use ipmitool as frequently
while using sensors.

> > > c) could it be a bug triggered by the SMP kernel?
All my IPMI machines (and I venture to guess the vast majority of IPMI
boxes) are SMP so I doubt its an SMP problem.

> > >
> > > d) any suggestions as to where I go from here ;-)


> > Have you tried the openipmi mailing list?
> > http://lists.sourceforge.net/lists/listinfo/openipmi-developer
> 
> Yeah, that might be a good place to ask, thanks!

I don't think your problem has anything to do with the IPMI subsystem,
so that isn't a good place to ask.

Yani

[lm-sensors] i801_smbus 0000:00:1f.3: Bus collision!

[Richard Hirst]

On Thu, Sep 08, 2005 at 09:56:29AM -0400, Yani Ioannou wrote:
> On 9/8/05, Richard Hirst <rhirst@levanta.com> wrote:
> 
> > > > The board has an mBMC which is basically working in that I can read
> > > > the sensors either by 'ipmitool' or 'sensors'.
> > > >
> > > > However, round about every 10 reboots or so, I get the Bus collision
> > > > message and the system locks up solid during boot after outputting a
> > > > few messages such as
> > > >
> > > > i801_smbus 0000:00:1f.3: Bus collision!
> > > > i801_smbus 0000:00:1f.3: Reset failed! (01)
> > > > i801_smbus 0000:00:1f.3: Reset failed! (01)
> > > > i801_smbus 0000:00:1f.3: Reset failed! (01)
> > > > bmcsensors.o: Error 0xff on cmd 0xa/0x23; state = 2; probably fatal.
> > > > i801_smbus 0000:00:1f.3: Reset failed! (01)
> > > > i801_smbus 0000:00:1f.3: Reset failed! (01)
> > > > i801_smbus 0000:00:1f.3: Reset failed! (01)
> 
> The bus collisions are expected - both the IPMI BMC (not bmcsensors
> directly) and the i2c modules you have loaded are competing for the
> smbus. However although I've done this (loaded both bmcsensors and i2c
> modules and got bus collisions) its never resulted in a lockup.

The lockup seems to be as a result of stack recursion; I modified
the "Reset failed!" message to report the address of a local stack
variable and after about 5 such messages the variable address starts
working down the stack until it hangs.  Trying to get a decent
stack trace now.

I currently have these modules loaded:

Module                  Size  Used by
...
bmcsensors             53861  0 
eeprom                 11353  0 
i2c_sensor              7361  1 eeprom
i2c_ipmi                8397  0 
ipmi_msghandler        41509  1 i2c_ipmi
i2c_dev                13249  0 
i2c_i801               13053  0 
...
i2c_piix4              12113  0 
i2c_core               29889  7 bmcsensors,eeprom,i2c_sensor,i2c_ipmi,i2c_dev,i2c_i801,i2c_piix4
...

The system is actually only interested in getting sensor values
by reading files in /sys/class/i2c-dev/... (I didn't write that bit).

> No, not directly. bmcsensors uses the IPMI messaging interface to send
> out queries to the IPMI BMC which then reads the sensors readings over
> the i2c/smbus (which is sometimes shared with the system).
...
> The IPMI BMC won't be using the i801 interface driver...its basically
> another computer on your motherboard :), I'd suspect the i2c drivers
> or i801 are causing the system lockup, as Mark says though you
> shouldn't be using both (but it shouldn't lockup IHMO).

So, out of the above modules, which can I do without?  eeprom, at
least; any others?

My understanding is that I'm talking over the i2c to the mBMC, and it
is then talking over (a different?) i2c to the sensors.  So the collision
is presumably effectively between the bmcsensors and the eeprom code.

You can get an idea of the h/w from a previous post, if you have
the time:

<http://lists.lm-sensors.org/pipermail/lm-sensors/2005-June/012847.html>

Thanks for the helpful response,

Richard

[lm-sensors] i801_smbus 0000:00:1f.3: Bus collision!

[Richard Hirst]

On Thu, Sep 08, 2005 at 09:56:29AM -0400, Yani Ioannou wrote:
> The bus collisions are expected - both the IPMI BMC (not bmcsensors
> directly) and the i2c modules you have loaded are competing for the
> smbus. However although I've done this (loaded both bmcsensors and i2c
> modules and got bus collisions) its never resulted in a lockup.

The code does this in i801_check_hststs():

        } else if (d->hststs & 0x08) {
                entry->result = -EIO;
                dev_err(&I801_dev->dev, "Bus collision!\n");
                /* Clock stops and slave is stuck in mid-transmission */
        } else if (d->hststs & 0x04) {

and on return from i801_check_hststs() can continue to poke at the
hardware.  That presumably breaks both this transaction and whatever
one it clashed with.  I'd taken the "slave is stuck" comment to be
pretty serious, and assumed this was a "shouldn't happen" error
condition.  Wouldn't it make more sense to use a semaphore or similar
to avoid even attempting the second transaction when the first is
in progress?

Richard

[lm-sensors] i801_smbus 0000:00:1f.3: Bus collision!

[Mark M. Hoffman]

Hi Yani:

I wrote:
> > > Have you tried the openipmi mailing list?
> > > http://lists.sourceforge.net/lists/listinfo/openipmi-developer

> On 9/8/05, Richard Hirst <rhirst@levanta.com> wrote:
> > Yeah, that might be a good place to ask, thanks!

* Yani Ioannou <yani.ioannou@gmail.com> [2005-09-08 09:56:29 -0400]:
> I don't think your problem has anything to do with the IPMI subsystem,
> so that isn't a good place to ask.

I disagree: Richard is running with large & nontrivial patches to both
i2c-i801 and i2c-core that are maintained there, at least for now.

Regards,

-- 
Mark M. Hoffman
mhoffman@lightlink.com

[lm-sensors] i801_smbus 0000:00:1f.3: Bus collision!

[Yani Ioannou]

On 9/8/05, Mark M. Hoffman <mhoffman@lightlink.com> wrote:
> Hi Yani:
> 
> I wrote:
> > > > Have you tried the openipmi mailing list?
> > > > http://lists.sourceforge.net/lists/listinfo/openipmi-developer
> 
> > On 9/8/05, Richard Hirst <rhirst@levanta.com> wrote:
> > > Yeah, that might be a good place to ask, thanks!
> 
> * Yani Ioannou <yani.ioannou@gmail.com> [2005-09-08 09:56:29 -0400]:
> > I don't think your problem has anything to do with the IPMI subsystem,
> > so that isn't a good place to ask.
> 
> I disagree: Richard is running with large & nontrivial patches to both
> i2c-i801 and i2c-core that are maintained there, at least for now.

Ah, sorry it makes sense to ask there then :).

Yani

[lm-sensors] i801_smbus 0000:00:1f.3: Bus collision!

[Mark M. Hoffman]

Hi Yani:

* Yani Ioannou <yani.ioannou@gmail.com> [2005-09-08 09:56:29 -0400]:
> The bus collisions are expected - both the IPMI BMC (not bmcsensors
> directly) and the i2c modules you have loaded are competing for the
> smbus. However although I've done this (loaded both bmcsensors and i2c
> modules and got bus collisions) its never resulted in a lockup.

I should also point out: I've been able to get my own ICH5/i801 SMBus 
locked up so hard that it takes a complete power cycle for the machine
to even POST correctly.  Ultimately, that's the fault of an extremely
unforgiving SMBus client chip somewhere (to say the least) - and there's
absolutely nothing the I2C/SMBus driver can do about that.

*However* even when this happens for me, *Linux* still runs fine.  It's
only the SMBus itself that gets locked up.  In my case, I can still 
attempt to read sensors or eeprom or whatever, and of course that fails
with tons of collisions - but it doesn't kill the whole kernel.

Richard's symptoms seem to implicate the i2c-i801 and i2c-core patches.

Regards,

-- 
Mark M. Hoffman
mhoffman@lightlink.com

[lm-sensors] i801_smbus 0000:00:1f.3: Bus collision!

[Richard Hirst]

On Thu, Sep 08, 2005 at 01:12:34PM -0400, Mark M. Hoffman wrote:
> Richard's symptoms seem to implicate the i2c-i801 and i2c-core patches.

Yeah, got a stack trace at the point where it starts recursing
down the stack:

Call Trace:
 [<f89e4bb7>] i801_start+0x2fd/0x375 [i2c_i801]
 [<f8a0ede6>] i2c_start_entry+0x28/0x52 [i2c_core]
 [<f8a1010c>] i2c_non_blocking_op+0xbb/0xcf [i2c_core]
 [<f8ea5960>] start_resend+0x7d/0x99 [ipmi_smb]
 [<f8ea52d9>] msg_done_handler+0x57/0x532 [ipmi_smb]
 [<c011e09c>] printk+0xe/0x11
 [<f89e4680>] i801_block_next_byte+0xb7/0xfd [i2c_i801]
 [<f8a0eec0>] i2c_entry_put+0x58/0x7c [i2c_core]
 [<f89e4c1e>] i801_start+0x364/0x375 [i2c_i801]
 [<f8a0ede6>] i2c_start_entry+0x28/0x52 [i2c_core]
 [<f8a1010c>] i2c_non_blocking_op+0xbb/0xcf [i2c_core]
 [<f8ea525a>] retry_timeout+0x47/0x6f [ipmi_smb]
 [<f8ea5213>] retry_timeout+0x0/0x6f [ipmi_smb]
 [<c012548a>] run_timer_softirq+0x123/0x145
 [<c0122094>] __do_softirq+0x4c/0xb1
 [<c0105db7>] do_softirq+0x41/0x48


so it looks like there is now a path where a request can fail
in i2c_start() which causes it to start the next request
which calls i2c_start() ... etc.

Pretty clear this is most likely an issue with the patches I'm
using.  Thanks to you both for the helpful responses.

Richard