RE: [PATCH v4] ntb: ntb perf tool

["Allen Hubbe" <Allen.Hubbe@emc.com>]

From: Allen Hubbe
> From: Jiang, Dave <dave.jiang@intel.com>
> > On Jan 12, 2016, at 7:08 AM, Hubbe, Allen <Allen.Hubbe@emc.com>
> > > From: Dave Jiang
> > >> Providing raw performance data via a tool that directly access data
> > from
> > >

> There might also be a race in link-up.  I got stuck with a lot of this,
> once, and no link related messages on the other side:
> [60025.597128] ntb_perf: ntb_hw_intel 0000:00:03.0: Remote version = 0x0
> [60025.607119] ntb_perf: ntb_hw_intel 0000:00:03.0: perf_link_work

I think the race has is between clearing local spads in link cleanup, and the peer writing the same spads in link work.

Suppose B is already loaded, and then A is loaded.
B: link was down - link cleanup - clear spads on B.

A: modprobe ntb_perf.
A: ntb_link_enable - tells the ntb hw to enable, but link up is async.
A: ntb_link_event - trigger a link event, even if there was no link state change.

Race #1: is the link up fast enough on A?
A: perf_link_event - link state is down, schedule link cleanup.

A & B: perf_link_event - link state is now up, schedule link work.

Race #2: A clears good data written by B, AND B reads good data written by A?
B: link work - write good data to spads on A, continue below...
A: link cleanup - clear spads on A (oops!).
B: link work continued - read spads on B have good data, work is done!

Result:
A: link work - write good data to spads on B, read spads on A are zero, reschedule... forever.

This would be a very short window to race, so there might be something else, too.  I don't recall seeing link work run on B at all, and I missed the opportunity to save that log.  Now it's lost among all the others in dmesg, and I can't distinguish "no output" from "maybe these messages" that far back in the log.  If a banner was printed each time the module is loaded, that would help.

In ntrdma, I avoid this race by /not/ clearing the local registers.  To avoid reading bad data, the peer updates a counter as long as the data is valid.  Observing a change in that counter value indicates good data.  That avoids a race between A and B each writing the same registers.

Allen