[parisc-linux] Virtual mapping of IO cards

[parisc-linux] Virtual mapping of IO cards

[John Marvin]

Note: I've changed the subject, because I think this issue is mostly
separate from the Linux syscall issue.

> An HPMC may be delayed, relative to the instruction that caused it.  The
> worst case is that a context switch _could_ occur before the HPMC occurs
> (and yes, we did see this problem with our HP-UX and HP-RT VME systems
> when a VME time-out was long enough).  This can make it more difficult to
> figure out what instruction was issued to cause the HPMC.  The advantage
> of the page fault is that you know exactly what instruction caused the
> fault.

But this doesn't help make your case.  You got the HPMC anyway because the
card failed to respond in time.  The virtual mapping didn't help you.

Virtually mapping the card doesn't help with HPMC's caused by dma buffer
mismanagement (i.e. the card causes the HPMC while mastering the bus).

Drivers for memory mapped register only cards, i.e. cards without any
type of onboard memory (i.e. framebuffers, script memory, etc.) are
not very likely to ever run into a bug of this type, since register
pointers are usually set up once and never changed.

In my experience, the majority of HPMC's have been caused by VM errors.
Then comes the two cases mentioned above (card not responding, dma
errors).  In my experience, the majority of driver page faults were caused
by memory references (i.e. mismanaging memory buffers), not IO space
references.  I can't say I've ever seen a driver page fault bug (i.e. one
that would have HPMC'd in the current Linux implementation) that was
caused by the driver mismanaging a pointer to its virtually mapped card
space.  That is my experience.  YMMV.

What percentage of the bugs that are caused by a driver mismanaging a
pointer to its card space would be significantly helped by page faulting,
rather than HPMC'ing? Although an HPMC can be delayed, I've found in
the majority of the cases it was either right on, or one instruction off.

I'm not trying to argue against virtually mapping the card (although I
would be all for avoiding mapping a large graphics frame buffer in the
kernel address space). I just want to be sure we do it for the right
reasons.

John Marvin
jsm@fc.hp.com

Re: [parisc-linux] Virtual mapping of IO cards

[Grant Grundler]

John Marvin wrote:
...
> Virtually mapping the card doesn't help with HPMC's caused by dma buffer
> mismanagement (i.e. the card causes the HPMC while mastering the bus).

But it generally helps one tell the difference.

> 
> Drivers for memory mapped register only cards, i.e. cards without any
> type of onboard memory (i.e. framebuffers, script memory, etc.) are
> not very likely to ever run into a bug of this type, since register
> pointers are usually set up once and never changed.

Agreed. "not very likely" and "never" are not the same. When bringing
up new drivers, it is in fact more likely than for a mostly working
driver. So while I agree with the statement, I don't agree with the
conclusion.


> In my experience, the majority of HPMC's have been caused by VM errors.

That's cuz you work mostly on VM! :^)

For me, it's been mostly device driver bugs (drivers loosing DMA addresses)
and some I/O MMU code bugs.

> Then comes the two cases mentioned above (card not responding, dma
> errors).  In my experience, the majority of driver page faults were caused
> by memory references (i.e. mismanaging memory buffers), not IO space
> references.

Agreed.

> I can't say I've ever seen a driver page fault bug (i.e. one
> that would have HPMC'd in the current Linux implementation) that was
> caused by the driver mismanaging a pointer to its virtually mapped card
> space.  That is my experience.  YMMV.

I have. The driver control structure got corrupted.
I've worked on so many memory corruption problems.
I can't imagine debugging them if an errant driver were
scribbling in real mode to random places in memory.
AFAICT, the whole point of virtual mode is to avoid this.

> What percentage of the bugs that are caused by a driver mismanaging a
> pointer to its card space would be significantly helped by page faulting,
> rather than HPMC'ing? Although an HPMC can be delayed, I've found in
> the majority of the cases it was either right on, or one instruction off.

Very small. But I am still convinced it's worth it.

AFAIK, the processor can only HPMC if a LOAD times out.
What about processor STORE?

I know stores to I/O devices which have "gone fatal" will complete.
I don't know about stores to "unused" I/O space (no device response).
I assume stores to physical memory while in real mode will complete too.
I'd really like to avoid that. Stores from virtual mode are likely
to data page fault (not always) while in real mode I had the impression
they never would. Is this correct?

> 
> I'm not trying to argue against virtually mapping the card (although I
> would be all for avoiding mapping a large graphics frame buffer in the
> kernel address space). I just want to be sure we do it for the right
> reasons.

ok. Fair enough.

grant

Grant Grundler
Unix Development Lab
+1.408.447.7253

Re: [parisc-linux] Virtual mapping of IO cards

[Frank Rowand]

Grant Grundler wrote:
> 

< stuff deleted >

> AFAIK, the processor can only HPMC if a LOAD times out.
> What about processor STORE?

Yes, a bad store can HPMC.  The reason for this is that not being able to write
data successfully means that you have possible data corruption (for
example, data that you had expected to be in persistent storage is not there).

< stuff deleted >

> grant
> 
> Grant Grundler

-Frank
-- 
MontaVista Software, Inc
frank_rowand@mvista.com

Re: [parisc-linux] Virtual mapping of IO cards

[Frank Rowand]

John Marvin wrote:
> 
> Note: I've changed the subject, because I think this issue is mostly
> separate from the Linux syscall issue.
> 
> > An HPMC may be delayed, relative to the instruction that caused it.  The
> > worst case is that a context switch _could_ occur before the HPMC occurs
> > (and yes, we did see this problem with our HP-UX and HP-RT VME systems
> > when a VME time-out was long enough).  This can make it more difficult to
> > figure out what instruction was issued to cause the HPMC.  The advantage
> > of the page fault is that you know exactly what instruction caused the
> > fault.
> 
> But this doesn't help make your case.  You got the HPMC anyway because the
> card failed to respond in time.  The virtual mapping didn't help you.
> 
> Virtually mapping the card doesn't help with HPMC's caused by dma buffer
> mismanagement (i.e. the card causes the HPMC while mastering the bus).
> 
> Drivers for memory mapped register only cards, i.e. cards without any
> type of onboard memory (i.e. framebuffers, script memory, etc.) are
> not very likely to ever run into a bug of this type, since register
> pointers are usually set up once and never changed.
> 
> In my experience, the majority of HPMC's have been caused by VM errors.
> Then comes the two cases mentioned above (card not responding, dma
> errors).  In my experience, the majority of driver page faults were caused
> by memory references (i.e. mismanaging memory buffers), not IO space
> references.  I can't say I've ever seen a driver page fault bug (i.e. one
> that would have HPMC'd in the current Linux implementation) that was
> caused by the driver mismanaging a pointer to its virtually mapped card
> space.  That is my experience.  YMMV.
> 
> What percentage of the bugs that are caused by a driver mismanaging a
> pointer to its card space would be significantly helped by page faulting,
> rather than HPMC'ing? Although an HPMC can be delayed, I've found in
> the majority of the cases it was either right on, or one instruction off.


Yes, an HPMC caused by a read will behave as you describe.  It's the
case of a write that may be delayed by passing through one or more queues.


> I'm not trying to argue against virtually mapping the card (although I
> would be all for avoiding mapping a large graphics frame buffer in the
> kernel address space). I just want to be sure we do it for the right
> reasons.
> 
> John Marvin
> jsm@fc.hp.com
> 
> ---------------------------------------------------------------------------
> To unsubscribe: send e-mail to parisc-linux-request@thepuffingroup.com with
> `unsubscribe' as the subject.

-Frank
-- 
MontaVista Software, Inc
frank_rowand@mvista.com

Re: [parisc-linux] Virtual mapping of IO cards

[John Marvin]

> >
> > Drivers for memory mapped register only cards, i.e. cards without any
> > type of onboard memory (i.e. framebuffers, script memory, etc.) are
> > not very likely to ever run into a bug of this type, since register
> pointers are usually set up once and never changed.
>
> Agreed. "not very likely" and "never" are not the same. When bringing
> up new drivers, it is in fact more likely than for a mostly working
>  driver. So while I agree with the statement, I don't agree with the
> conclusion.

I guess I was mostly talking from a released version perspective, not a
development perspective.  During development you would actually get even
more debug information if you just put driver specific range checking into
a private version of the read/write macro's (i.e. you could get source
code line numbers, etc.)

I really am not trying to argue strongly against mapping the cards. I
see only two performance issues:

    1) PCXL2 processors require that the U bit be set for IO mappings.
    As far as I can tell, this is the only processor that requires this.
    The TLB miss handlers don't currently support this. We would have
    to add support for it in the D miss handler. Since the U bit is
    in an inconvenient location it would require a separate test and
    deposit, making all code pay the penalty of the additional
    instructions. Perhaps we would eventually want to add the U bit
    support anyway, but not all processors support it (the
    PCXS and PCXT processors do not support it).

    2) I really would like to avoid mapping the framebuffer into
    the kernel address space. Perhaps maybe that is not really an
    issue, since after boot the user will bring up X windows and
    the kernel mapping won't actually be actively used (I'm assuming
    that graphics on Linux maps the framebuffer into the user address
    space, but I haven't checked to be sure).

So, if you want to map the cards, I'm not going to oppose that.

John