Who is stomping PCI config space?

Who is stomping PCI config space?

[Jon Smirl]

I have two video cards:
PCI Rage128
AGP Radeon

X is loaded on the Rage128 on VT7
X is not configured to support the radeon

I switch to VT1 and load radeonfb
This works fine.

Now I switch to VT7 and dump PCI config space for the radeon

[root@jonsmirl 0000:01:00.0]# hexdump config >/foo
[root@jonsmirl 0000:01:00.0]# diff /foo /bar
1c1
< 0000000 1002 4966 0080 02b0 0001 0300 4010 0080
---
> 0000000 1002 4966 0083 02b0 0001 0300 4010 0080
4c4
< 0000030 0000 fea0 0058 0000 0000 0000 010b 0008
---
> 0000030 0000 fe90 0058 0000 0000 0000 010b 0008
[root@jonsmirl 0000:01:00.0]#

Somebody is changing PCI command from 83 to 80 and disabling my
radeon's memory and iospace. Who is doing this? It has to be X since
it doesn't happen if I switch to VT6 or VT8.

Why is X mucking with a card it doesn't have a driver loaded for? 
Where is this happening in the X code?

-- 
Jon Smirl
jonsmirl@gmail.com

Re: Who is stomping PCI config space?

[Benjamin Herrenschmidt]

On Thu, 2005-03-03 at 14:03 -0500, Jon Smirl wrote:
> I have two video cards:
> PCI Rage128
> AGP Radeon
> 
> X is loaded on the Rage128 on VT7
> X is not configured to support the radeon
> 
> I switch to VT1 and load radeonfb
> This works fine.
> 
> Now I switch to VT7 and dump PCI config space for the radeon
> 
> [root@jonsmirl 0000:01:00.0]# hexdump config >/foo
> [root@jonsmirl 0000:01:00.0]# diff /foo /bar
> 1c1
> < 0000000 1002 4966 0080 02b0 0001 0300 4010 0080
> ---
> > 0000000 1002 4966 0083 02b0 0001 0300 4010 0080
> 4c4
> < 0000030 0000 fea0 0058 0000 0000 0000 010b 0008
> ---
> > 0000030 0000 fe90 0058 0000 0000 0000 010b 0008
> [root@jonsmirl 0000:01:00.0]#
> 
> Somebody is changing PCI command from 83 to 80 and disabling my
> radeon's memory and iospace. Who is doing this? It has to be X since
> it doesn't happen if I switch to VT6 or VT8.
> 
> Why is X mucking with a card it doesn't have a driver loaded for? 
> Where is this happening in the X code?

I had a report of something similar with X and a mach64 ... I haven't
yet investigated, but basically, X is using "fbdev" (so no specific
driver) and seem to be disabling memory access to the card, thus blowing
up the kernel fbdev.

Ben.

Re: Who is stomping PCI config space?

[Jon Smirl]

My understanding of the X source code is limited, can anyone tell me
where X is turning off secondary cards on VT switch?

The patch looks like it keeps X from getting secondary cards onto the
list of cards to begin with.

Can anyone explain why X is disabling cards on VT switch that it does
not have drivers loaded for? This behavior just toasts my fbdev
driver. My driver is getting interrupts, now I switch VTs to X and the
card's MMIO window disappears. Of course my driver messes up, it go
into a hard loop servicing an interrupt it can't acknowledge.


On Fri, 04 Mar 2005 10:07:25 +1100, Benjamin Herrenschmidt
<benh@kernel.crashing.org> wrote:
> On Thu, 2005-03-03 at 14:03 -0500, Jon Smirl wrote:
> > I have two video cards:
> > PCI Rage128
> > AGP Radeon
> >
> > X is loaded on the Rage128 on VT7
> > X is not configured to support the radeon
> >
> > I switch to VT1 and load radeonfb
> > This works fine.
> >
> > Now I switch to VT7 and dump PCI config space for the radeon
> >
> > [root@jonsmirl 0000:01:00.0]# hexdump config >/foo
> > [root@jonsmirl 0000:01:00.0]# diff /foo /bar
> > 1c1
> > < 0000000 1002 4966 0080 02b0 0001 0300 4010 0080
> > ---
> > > 0000000 1002 4966 0083 02b0 0001 0300 4010 0080
> > 4c4
> > < 0000030 0000 fea0 0058 0000 0000 0000 010b 0008
> > ---
> > > 0000030 0000 fe90 0058 0000 0000 0000 010b 0008
> > [root@jonsmirl 0000:01:00.0]#
> >
> > Somebody is changing PCI command from 83 to 80 and disabling my
> > radeon's memory and iospace. Who is doing this? It has to be X since
> > it doesn't happen if I switch to VT6 or VT8.
> >
> > Why is X mucking with a card it doesn't have a driver loaded for?
> > Where is this happening in the X code?
> 
> I had a report of something similar with X and a mach64 ... I haven't
> yet investigated, but basically, X is using "fbdev" (so no specific
> driver) and seem to be disabling memory access to the card, thus blowing
> up the kernel fbdev.
> 
> Ben.
> 
> 


-- 
Jon Smirl
jonsmirl@gmail.com

Re: Who is stomping PCI config space?

[Jon Smirl]

Hopefully someone who knows what is going on with VT switching and how
hardware gets enabled will respond and we can get this fixed in the
server.  I see Zoltan's patch but we shouldn't have to tell X to leave
hardware alone that doesn't belong to it. X just has no business
messing with cards it does not own.

Meanwhile I am forced to write to PCI config space and reenable IO
access from inside my interrupt handler. Yuck, yuck, yuck!!!

-- 
Jon Smirl
jonsmirl@gmail.com

Re: [Linux-fbdev-devel] Re: Who is stomping PCI config space?

[Benjamin Herrenschmidt]

On Thu, 2005-03-03 at 22:03 -0500, Jon Smirl wrote:
> Hopefully someone who knows what is going on with VT switching and how
> hardware gets enabled will respond and we can get this fixed in the
> server.  I see Zoltan's patch but we shouldn't have to tell X to leave
> hardware alone that doesn't belong to it. X just has no business
> messing with cards it does not own.
> 
> Meanwhile I am forced to write to PCI config space and reenable IO
> access from inside my interrupt handler. Yuck, yuck, yuck!!!

Well, that shows why we need this arbitration for who gets the VGA
enable bits in the kernel :)

X disables any other VGA card IO/MEM in the system so that at one given
point in time, only one of them will decode VGA cycles. Wether it has
those cards to drive in it's config or not doesn't matter, the problem
at the bus level is the same.

Ben.

Re: [Linux-fbdev-devel] Re: Who is stomping PCI config space?

[Egbert Eich]

Benjamin Herrenschmidt writes:
 > On Thu, 2005-03-03 at 22:03 -0500, Jon Smirl wrote:
 > > Hopefully someone who knows what is going on with VT switching and how
 > > hardware gets enabled will respond and we can get this fixed in the
 > > server.  I see Zoltan's patch but we shouldn't have to tell X to leave
 > > hardware alone that doesn't belong to it. X just has no business
 > > messing with cards it does not own.
 > > 
 > > Meanwhile I am forced to write to PCI config space and reenable IO
 > > access from inside my interrupt handler. Yuck, yuck, yuck!!!
 > 
 > Well, that shows why we need this arbitration for who gets the VGA
 > enable bits in the kernel :)

It needs to have it in some central place which doesn't necessarily
have to be the kernel. 
The point is: if Jon needs these registers in an interrupt handler
he may have to tweak PCI config space form there anyway since another 
card may currently have VGA routed.

 > 
 > X disables any other VGA card IO/MEM in the system so that at one given
 > point in time, only one of them will decode VGA cycles. Wether it has
 > those cards to drive in it's config or not doesn't matter, the problem
 > at the bus level is the same.

Right. It however should only do so if one of the cards it is driving 
itself requires VGA registers for its mode of operation.

Egbert.

Re: Re: Who is stomping PCI config space?

[Jon Smirl]

On Fri, 4 Mar 2005 13:02:34 +0100, Egbert Eich <eich@suse.de> wrote:
> you know well that we are not living in the same time zone, so you
> may want to give me some time to answer.

I was getting upset that when I did a VT switch my machine was hard
locking and I had to keep rebooting all of the time to debug the
problem. I had spent many hours trying to find the problem in my code
and it wasn't there. I was not expecting X to alter the state of
hardware it did not have a driver loaded for.

> It needs to have it in some central place which doesn't necessarily
> have to be the kernel.
> The point is: if Jon needs these registers in an interrupt handler
> he may have to tweak PCI config space form there anyway since another
> card may currently have VGA routed.
> 
>  >
>  > X disables any other VGA card IO/MEM in the system so that at one given
>  > point in time, only one of them will decode VGA cycles. Wether it has
>  > those cards to drive in it's config or not doesn't matter, the problem
>  > at the bus level is the same.
> 
> Right. It however should only do so if one of the cards it is driving
> itself requires VGA registers for its mode of operation.

Ok, now I see what X is doing. It is making sure there is only one
active VGA device in the system.  However, my framebuffer driver is
not using VGA mode.

I have similar code in my VGA support patch but I only shut down the
card mem/io long enough to turn off VGA support. I don't leave the
whole card turned off. I also see now that I should shut interrupts
off while doing this.

pci_read_config_word(pdev, PCI_COMMAND, &command);
pci_write_config_word(pdev, PCI_COMMAND, command | PCI_COMMAND_IO |
PCI_COMMAND_MEMORY);

vga_io_w(0x3C3, ~0x01 & vga_io_r(0x3C3));
vga_io_w(0x46e8, ~0x08 & vga_io_r(0x46e8));
vga_io_w(0x102, ~0x01 & vga_io_r(0x102));

pci_write_config_word(pdev, PCI_COMMAND, command);

If we leave the whole card turned off I can't access the interrupt
status registers to acknowledge the interrupt and shut it off.

Does this approach work for X? Where is the code that does this at VT
switch time?

On VT enter X would need to:
1) shut off interrupts
2) disable IO/MEM on all VGA cards - remember IO/mem state
3) turn the cards on one at a time and disable VGA - remember VGA enable state
4) restore IO/MEM state to all cards
5) turn interrupts back on

On VT exit:
1) shut off interrupts
2) disable IO/MEM on all VGA cards - remember IO/mem state
3) restore VGA enable state
4) restore IO/MEM state to all cards
5) turn interrupts back on

-- 
Jon Smirl
jonsmirl@gmail.com


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Re: [Linux-fbdev-devel] Re: Who is stomping PCI config space?

[Benjamin Herrenschmidt]

> Ok, now I see what X is doing. It is making sure there is only one
> active VGA device in the system.  However, my framebuffer driver is
> not using VGA mode.
> 
> I have similar code in my VGA support patch but I only shut down the
> card mem/io long enough to turn off VGA support. I don't leave the
> whole card turned off. I also see now that I should shut interrupts
> off while doing this.

Shut interrupts off never works. It only affects one CPU, you can still
take the interrupt on the other CPU. Unless you switch interrupt
emission on the card itself, or use some kind of spinlock to put IRQs
"on hold" while you are doing the shutdown...
 
> pci_read_config_word(pdev, PCI_COMMAND, &command);
> pci_write_config_word(pdev, PCI_COMMAND, command | PCI_COMMAND_IO |
> PCI_COMMAND_MEMORY);
> 
> vga_io_w(0x3C3, ~0x01 & vga_io_r(0x3C3));
> vga_io_w(0x46e8, ~0x08 & vga_io_r(0x46e8));
> vga_io_w(0x102, ~0x01 & vga_io_r(0x102));
> 
> pci_write_config_word(pdev, PCI_COMMAND, command);
> 
> If we leave the whole card turned off I can't access the interrupt
> status registers to acknowledge the interrupt and shut it off.

Also, some cards won't let you switch VGA access decoding off.

> Does this approach work for X? Where is the code that does this at VT
> switch time?
> 
> On VT enter X would need to:
> 1) shut off interrupts

Not possible (system interrupts). Unless it does it on the card itself,
which is a per-driver thing.

> 2) disable IO/MEM on all VGA cards - remember IO/mem state
> 3) turn the cards on one at a time and disable VGA - remember VGA enable state
> 4) restore IO/MEM state to all cards
> 5) turn interrupts back on
> 
> On VT exit:
> 1) shut off interrupts
> 2) disable IO/MEM on all VGA cards - remember IO/mem state
> 3) restore VGA enable state
> 4) restore IO/MEM state to all cards
> 5) turn interrupts back on

Ben.

Re: [Linux-fbdev-devel] Re: Who is stomping PCI config space?

[Egbert Eich]

Benjamin Herrenschmidt writes:
 > > 
 > > If we leave the whole card turned off I can't access the interrupt
 > > status registers to acknowledge the interrupt and shut it off.
 > 
 > Also, some cards won't let you switch VGA access decoding off.

This is what I suspect. I had some success with the generic VGA disable
code but our mileage may vary.

 > 
 > > Does this approach work for X? Where is the code that does this at VT
 > > switch time?
 > > 
 > > On VT enter X would need to:
 > > 1) shut off interrupts
 > 
 > Not possible (system interrupts). Unless it does it on the card itself,
 > which is a per-driver thing.

How do we deal with interrupts from VGA cards that we don't have a 
driver for? We don't service interrupts in this case either.
Can't we unregister the interrupt handler until everything
is set up right? Maybe a dump idea...
Also there is a generic VGA register to control interrupts.


Egbert.

Re: [Linux-fbdev-devel] Re: Who is stomping PCI config space?

[Benjamin Herrenschmidt]

> How do we deal with interrupts from VGA cards that we don't have a 
> driver for? 

We don't get any, do we ? If we get an interrupt we don't have a driver
for, the system dies usually ;) Well, unless it's an edge interrupt, in
which case we can just drop it on the floor, but PCI interrupts are
level.

> We don't service interrupts in this case either.
> Can't we unregister the interrupt handler until everything
> is set up right? Maybe a dump idea...
> Also there is a generic VGA register to control interrupts.

We have reached the end of my VGA knowledge, but I seriously doubt a VGA
card will issue interrupts if not previously programmed to do so... So
we probably don't have to bother. We only care about interrupts that we
have a use for, that is interrupts triggered by cards upon request of
their driver, for things like DMA/AGP command buffer operations, vblank,
monitor detection, ...

Ben.

Re: [Linux-fbdev-devel] Re: Who is stomping PCI config space?

[Egbert Eich]

Benjamin Herrenschmidt writes:
 > 
 > > How do we deal with interrupts from VGA cards that we don't have a 
 > > driver for? 
 > 
 > We don't get any, do we ? If we get an interrupt we don't have a driver
 > for, the system dies usually ;) Well, unless it's an edge interrupt, in
 > which case we can just drop it on the floor, but PCI interrupts are
 > level.

Usually VGA chips are capable of sending interrupts. At least the VGA
specs had a bit where interrupts can be enabled/disabled.
I'm surprised that this has not been an issue yet as I'm not sure if
the BIOS POST always leaves them disabled. For PCI chips they probalbly
do.

 > 
 > > We don't service interrupts in this case either.
 > > Can't we unregister the interrupt handler until everything
 > > is set up right? Maybe a dump idea...
 > > Also there is a generic VGA register to control interrupts.
 > 
 > We have reached the end of my VGA knowledge, but I seriously doubt a VGA
 > card will issue interrupts if not previously programmed to do so... So
 > we probably don't have to bother. We only care about interrupts that we

Possibly.

 > have a use for, that is interrupts triggered by cards upon request of
 > their driver, for things like DMA/AGP command buffer operations, vblank,
 > monitor detection, ...

The classic use for interrupts has been vblank. We have never used it
and I've heard reports from some people who have played around with this
that on some VGA chips it did what it was supposed to on others it didn't.

Cheers,
	Egbert.

Re: [Linux-fbdev-devel] Re: Who is stomping PCI config space?

[Benjamin Herrenschmidt]

On Mon, 2005-03-07 at 12:19 +0100, Egbert Eich wrote:
> Benjamin Herrenschmidt writes:
>  > 
>  > > How do we deal with interrupts from VGA cards that we don't have a 
>  > > driver for? 
>  > 
>  > We don't get any, do we ? If we get an interrupt we don't have a driver
>  > for, the system dies usually ;) Well, unless it's an edge interrupt, in
>  > which case we can just drop it on the floor, but PCI interrupts are
>  > level.
> 
> Usually VGA chips are capable of sending interrupts. At least the VGA
> specs had a bit where interrupts can be enabled/disabled.
> I'm surprised that this has not been an issue yet as I'm not sure if
> the BIOS POST always leaves them disabled. For PCI chips they probalbly
> do.

I would be surprised if a BIOS left it enabled ... it would be a really
bad thing to do...

Re: [Linux-fbdev-devel] Re: Who is stomping PCI config space?

[Egbert Eich]

Jon Smirl writes:
 > On Fri, 4 Mar 2005 13:02:34 +0100, Egbert Eich <eich@suse.de> wrote:
 > > you know well that we are not living in the same time zone, so you
 > > may want to give me some time to answer.
 > 
 > I was getting upset that when I did a VT switch my machine was hard
 > locking and I had to keep rebooting all of the time to debug the
 > problem. I had spent many hours trying to find the problem in my code
 > and it wasn't there. I was not expecting X to alter the state of
 > hardware it did not have a driver loaded for.
 > 

I understand that this upsets you. But from my explanations you
should see that there are reasons for this behavior.


 > > It needs to have it in some central place which doesn't necessarily
 > > have to be the kernel.
 > > The point is: if Jon needs these registers in an interrupt handler
 > > he may have to tweak PCI config space form there anyway since another
 > > card may currently have VGA routed.
 > > 
 > >  >
 > >  > X disables any other VGA card IO/MEM in the system so that at one given
 > >  > point in time, only one of them will decode VGA cycles. Wether it has
 > >  > those cards to drive in it's config or not doesn't matter, the problem
 > >  > at the bus level is the same.
 > > 
 > > Right. It however should only do so if one of the cards it is driving
 > > itself requires VGA registers for its mode of operation.
 > 
 > Ok, now I see what X is doing. It is making sure there is only one
 > active VGA device in the system.  However, my framebuffer driver is
 > not using VGA mode.

Right, but X may be using a VGA register for something - maybe for
setting fb start (panning) or setting the HW cursor or whatever
else. 
If your hardware decodes VGA it would also decode these register
accesses. Without knowing details about your card it would be hard
for X to know if it does or not.

 > 
 > I have similar code in my VGA support patch but I only shut down the
 > card mem/io long enough to turn off VGA support. I don't leave the
 > whole card turned off. I also see now that I should shut interrupts
 > off while doing this.

To shut off VGA you'd need a driver. But as you said the driver isn't
loaded. 
When I designed the interface in X I was thinking about 'background
drivers' which would be able to set this up. Unfortunately this
issue had never been persued any further at the time and has been
forgotten since then.
I don't recall how much of this I actually implemented.

 > 
 > pci_read_config_word(pdev, PCI_COMMAND, &command);
 > pci_write_config_word(pdev, PCI_COMMAND, command | PCI_COMMAND_IO |
 > PCI_COMMAND_MEMORY);
 > 
 > vga_io_w(0x3C3, ~0x01 & vga_io_r(0x3C3));
 > vga_io_w(0x46e8, ~0x08 & vga_io_r(0x46e8));
 > vga_io_w(0x102, ~0x01 & vga_io_r(0x102));
 > 
 > pci_write_config_word(pdev, PCI_COMMAND, command);
 > 
 > If we leave the whole card turned off I can't access the interrupt
 > status registers to acknowledge the interrupt and shut it off.

Right.
 > 
 > Does this approach work for X? Where is the code that does this at VT
 > switch time?

We use this approach in the int10 code to get around the problem that
some BIOSes stomp over other VGA cards during POSTing - I discovered
this with Tseng BIOSes which should not be in use much today any more. 
Therefore I cannot say if these remedies work on all cards, though.
What we can say I think is that these calls don't hurt as we have used
them for years without issues.

 > 
 > On VT enter X would need to:
 > 1) shut off interrupts
 > 2) disable IO/MEM on all VGA cards - remember IO/mem state
 > 3) turn the cards on one at a time and disable VGA - remember VGA enable state
 > 4) restore IO/MEM state to all cards
 > 5) turn interrupts back on
 > 
 > On VT exit:
 > 1) shut off interrupts
 > 2) disable IO/MEM on all VGA cards - remember IO/mem state
 > 3) restore VGA enable state
 > 4) restore IO/MEM state to all cards
 > 5) turn interrupts back on

Right. Let me see how I can integrate this into my 
resource management in X. It shouldn't be too hard
to do and would be a remedy until we find a better
solution. 
I would need help from the kernel people to do the
interrupts. Also I may make this job configurable 
as I'm not sure if these VGA disable registers
are universally used.

Currently X has a heuristic that looks for enabled
VGA to identify the primary (ie. POSTed) video card.

With more than one graphics card enabled we will
mess up this heuristic. I need to look into this.

Egbert.

Re: [Linux-fbdev-devel] Re: Who is stomping PCI config space?

[Jon Smirl]

In the past Jesse Barnes has suggested a completely different approach
to VGA sharing. Instead of each VT trying to control which VGA device
is enabled we keep them all turned off. Then on each access we turn
VGA support on just long enough for the access and then turn it back
off.

To make this work we have to have a kernel based 'token' for who has
the VGA at the moment. This scheme lets multiuser systems work. For
big IA64 machines which can support multiple simultaneous VGAs we just
have multiple tokens.

We would just ban VGA access from interrupt context, if you really
needed VGA access you would use a workqueue which can be scheduled
from the interrupt. None of my driver work needs VGA access from
interrupt context, is this true for all drivers? I need access to the
video hardware, but not access to the VGA support.

-- 
Jon Smirl
jonsmirl@gmail.com

Re: [Linux-fbdev-devel] Re: Who is stomping PCI config space?

[Benjamin Herrenschmidt]

On Fri, 2005-03-04 at 12:58 -0500, Jon Smirl wrote:
> In the past Jesse Barnes has suggested a completely different approach
> to VGA sharing. Instead of each VT trying to control which VGA device
> is enabled we keep them all turned off. Then on each access we turn
> VGA support on just long enough for the access and then turn it back
> off.

That's what I had in mind too.

> To make this work we have to have a kernel based 'token' for who has
> the VGA at the moment. This scheme lets multiuser systems work. For
> big IA64 machines which can support multiple simultaneous VGAs we just
> have multiple tokens.

That means that the arbitrer must need to know how to enable/disable VGA
decoding on a per-card basis (when it's possible at all), or active
collaboration with the kernel driver. Also, you can't prevent the
interrupts from happening unless you also disable them on your card.

> We would just ban VGA access from interrupt context, if you really
> needed VGA access you would use a workqueue which can be scheduled
> from the interrupt. None of my driver work needs VGA access from
> interrupt context, is this true for all drivers? I need access to the
> video hardware, but not access to the VGA support.

It's not enough.

There is a variety of cases where the only way to turn off VGA access is
to turn off IO & MEM decoding completely on the card...

Ben.

Re: [Linux-fbdev-devel] Re: Who is stomping PCI config space?

[Egbert Eich]

Benjamin Herrenschmidt writes:
 > On Fri, 2005-03-04 at 12:58 -0500, Jon Smirl wrote:
 > > In the past Jesse Barnes has suggested a completely different approach
 > > to VGA sharing. Instead of each VT trying to control which VGA device
 > > is enabled we keep them all turned off. Then on each access we turn
 > > VGA support on just long enough for the access and then turn it back
 > > off.
 > 
 > That's what I had in mind too.

If we can control this.
However if we can there is not much difference which apporach we take.

 > 
 > > To make this work we have to have a kernel based 'token' for who has
 > > the VGA at the moment. This scheme lets multiuser systems work. For
 > > big IA64 machines which can support multiple simultaneous VGAs we just
 > > have multiple tokens.
 > 
 > That means that the arbitrer must need to know how to enable/disable VGA
 > decoding on a per-card basis (when it's possible at all), or active
 > collaboration with the kernel driver. Also, you can't prevent the
 > interrupts from happening unless you also disable them on your card.
 > 
 > > We would just ban VGA access from interrupt context, if you really
 > > needed VGA access you would use a workqueue which can be scheduled
 > > from the interrupt. None of my driver work needs VGA access from
 > > interrupt context, is this true for all drivers? I need access to the
 > > video hardware, but not access to the VGA support.
 > 
 > It's not enough.
 > 
 > There is a variety of cases where the only way to turn off VGA access is
 > to turn off IO & MEM decoding completely on the card...

That's the apporach we took when we implemented this in X. 
It was the safest bet.

Egbert.

Re: [Linux-fbdev-devel] Re: Who is stomping PCI config space?

[Benjamin Herrenschmidt]

On Sat, 2005-03-05 at 20:07 +0100, Egbert Eich wrote:

> That's the apporach we took when we implemented this in X. 
> It was the safest bet.

Yes, whatever mecanism I want to provide at the kernel level will do
that bet by default, unless it's informed by the card's driver that
legacy decoding was disabled for the card. That is, we bet the worse,
and only the guy who "knows" (ie, the card specific driver) can tell us
that life is actually much brighter ;)

Ben.

Re: [Linux-fbdev-devel] Re: Who is stomping PCI config space?

[Benjamin Herrenschmidt]

> It needs to have it in some central place which doesn't necessarily
> have to be the kernel. 

I think it has to be the kernel so we can include kernel services, like
kernel fbdev's or vgacon in the loop.

> The point is: if Jon needs these registers in an interrupt handler
> he may have to tweak PCI config space form there anyway since another 
> card may currently have VGA routed.

The problem is that a card that has VGA support _and_ MMIO registers.
Even if nobody uses VGA on it, if a different VGA card on the same bus
is active, X will disable IO decoding on the first one (to disable VGA
too).

There isn't a simple way to solve that that I know, except if the card
can be configured to totally ignore VGA accesses, in which case it
doesn't need to be disabled, but X doesn't know it...  (though if we
have a central "arbibrer", the driver for the card can tell it to remove
that card from the arbitration).

In the normal case tho, I don't really see how to deal with that other
than, indeed, switching the enables at interrupt time, when the IRQ gets
in, which will obviously conflict with a concurrent server working on
the "other" card at the same time....

So even with a kernel based arbitrer, the irq scenario isn't possible to
deal with properly, unless VGA decoding can be completely disabled on a
given card, or nobody uses VGA memory accesses on any card. (Most modern
drivers only use non-VGA memory/IO, or even IO are remapped to some
different addresses with some cards).

As soon as two cards are in the system on the same bus segment, with one
of the them needing legacy VGA accesses to the low PCI memory/IO ports,
and the other one potentially decoding that space, there is simply no
solution that I can see with interrupts. 
>  > 
>  > X disables any other VGA card IO/MEM in the system so that at one given
>  > point in time, only one of them will decode VGA cycles. Wether it has
>  > those cards to drive in it's config or not doesn't matter, the problem
>  > at the bus level is the same.
> 
> Right. It however should only do so if one of the cards it is driving 
> itself requires VGA registers for its mode of operation.

Ben.

Re: [Linux-fbdev-devel] Re: Who is stomping PCI config space?

[Egbert Eich]

Ben,

Benjamin Herrenschmidt writes:
 > > The point is: if Jon needs these registers in an interrupt handler
 > > he may have to tweak PCI config space form there anyway since another 
 > > card may currently have VGA routed.
 > 
 > The problem is that a card that has VGA support _and_ MMIO registers.
 > Even if nobody uses VGA on it, if a different VGA card on the same bus
 > is active, X will disable IO decoding on the first one (to disable VGA
 > too).

Right. And without any driver support it is difficult to know if the
card decodes legacy resources or if this decoding is turned off.
I can do waht Jon suggested and use the 'industry standard' VGA disable
registers. 
Depending on the card those registers may disable more than just VGA.

 > 
 > There isn't a simple way to solve that that I know, except if the card
 > can be configured to totally ignore VGA accesses, in which case it
 > doesn't need to be disabled, but X doesn't know it...  (though if we
 > have a central "arbibrer", the driver for the card can tell it to remove
 > that card from the arbitration).

Right. But we don't load the driver.

 > 
 > In the normal case tho, I don't really see how to deal with that other
 > than, indeed, switching the enables at interrupt time, when the IRQ gets
 > in, which will obviously conflict with a concurrent server working on
 > the "other" card at the same time....

On a multi-CPU system this can be a problem as one CPU may work on
the previously enabled VGA card while the other one tries to service
the VGA card that has sent the interrupt.

 > 
 > So even with a kernel based arbitrer, the irq scenario isn't possible to
 > deal with properly, unless VGA decoding can be completely disabled on a
 > given card, or nobody uses VGA memory accesses on any card. (Most modern
 > drivers only use non-VGA memory/IO, or even IO are remapped to some
 > different addresses with some cards).

On a lot of cards there may be situations where VGA registers need to
be accessed for specific purposes.
Furthermore it must not be expected that nobody uses older cards which
still require VGA registers.

Is there a way to halt a process that is using VGA registers for the
time the interrupt gets serviced? For example sending a signal to 
this process so that it suspends norml execution waits in a signal 
handler?
All this would only matter for systems with more than one CPU.

Cheers,
	Egbert.

Re: [Linux-fbdev-devel] Re: Who is stomping PCI config space?

[Benjamin Herrenschmidt]

> Right. And without any driver support it is difficult to know if the
> card decodes legacy resources or if this decoding is turned off.
> I can do waht Jon suggested and use the 'industry standard' VGA disable
> registers. 
> Depending on the card those registers may disable more than just VGA.
> 
>  > 
>  > There isn't a simple way to solve that that I know, except if the card
>  > can be configured to totally ignore VGA accesses, in which case it
>  > doesn't need to be disabled, but X doesn't know it...  (though if we
>  > have a central "arbibrer", the driver for the card can tell it to remove
>  > that card from the arbitration).
> 
> Right. But we don't load the driver.

Yes, you can assume the worst case at first, and later on have the
driver disable decoding of legacy stuff and inform your arbiter to stop
caring about that specific card, no ?

> On a multi-CPU system this can be a problem as one CPU may work on
> the previously enabled VGA card while the other one tries to service
> the VGA card that has sent the interrupt.

Yes. Fancy things like local disabling of irqs etc... aren't an option.
>  > 
>  > So even with a kernel based arbitrer, the irq scenario isn't possible to
>  > deal with properly, unless VGA decoding can be completely disabled on a
>  > given card, or nobody uses VGA memory accesses on any card. (Most modern
>  > drivers only use non-VGA memory/IO, or even IO are remapped to some
>  > different addresses with some cards).
> 
> On a lot of cards there may be situations where VGA registers need to
> be accessed for specific purposes.

Yup, though "modern" cards usually don't. A radeon operates 100% without
access to any legacy resource lately (except the console restore on
console switch, but that's fairly isolated and I even removed it on ppc
as it breaks things on setups where the card was boostrapped in "no VGA"
mode. I personally think it should be done by the kernel). I think
nvidia's "remap" the VGA registes somewhere in normal MMIO space, at
least I think I remember seeing something like that.

> Furthermore it must not be expected that nobody uses older cards which
> still require VGA registers.

Agreed.

> Is there a way to halt a process that is using VGA registers for the
> time the interrupt gets serviced? For example sending a signal to 
> this process so that it suspends norml execution waits in a signal 
> handler?

Not really. Well, something could be hacked, that is, the current CPU is
obviously handling the IRQ, so the process "below" has been halted, but
it would require some kind of IPI tricks to also halt other CPUs, quite
heavy handed I think.

> All this would only matter for systems with more than one CPU.

Yes, but those are becoming quite common. Also multicore is coming ;)

Ben.

Re: [Linux-fbdev-devel] Re: Who is stomping PCI config space?

[Egbert Eich]

Benjamin Herrenschmidt writes:
 > 
 > Yes, you can assume the worst case at first, and later on have the
 > driver disable decoding of legacy stuff and inform your arbiter to stop
 > caring about that specific card, no ?

Right. Actually we had different mechanisms implemented and I also
had the concept of 'background' driver which would just do this
without driving any card.
Unfortunately this has never really been employed much and I would
have to look how this was designed in detail.

 > > 
 > > On a lot of cards there may be situations where VGA registers need to
 > > be accessed for specific purposes.
 > 
 > Yup, though "modern" cards usually don't. A radeon operates 100% without
 > access to any legacy resource lately (except the console restore on
 > console switch, but that's fairly isolated and I even removed it on ppc
 > as it breaks things on setups where the card was boostrapped in "no VGA"
 > mode. I personally think it should be done by the kernel). I think
 > nvidia's "remap" the VGA registes somewhere in normal MMIO space, at
 > least I think I remember seeing something like that.

How about POSTing? Doesn't the Radeon card have to be POSTed?

 > 
 > > Furthermore it must not be expected that nobody uses older cards which
 > > still require VGA registers.
 > 
 > Agreed.
 > 
 > > Is there a way to halt a process that is using VGA registers for the
 > > time the interrupt gets serviced? For example sending a signal to 
 > > this process so that it suspends norml execution waits in a signal 
 > > handler?
 > 
 > Not really. Well, something could be hacked, that is, the current CPU is
 > obviously handling the IRQ, so the process "below" has been halted, but
 > it would require some kind of IPI tricks to also halt other CPUs, quite
 > heavy handed I think.

Yes, that's what I was asking. Would there be a way to avoid too
much overhead in the kernel?
But maybe your suggestion how to handle the situation where interrupt
handling that required VGA resources could be dealt with outside the 
interrupt handler would be sufficient for all cases where this may
matter.

 > 
 > > All this would only matter for systems with more than one CPU.
 > 
 > Yes, but those are becoming quite common. Also multicore is coming ;)
 > 

I agree.

Cheers,
	Egbert.

Re: Who is stomping PCI config space?

[Egbert Eich]

Jon,

you know well that we are not living in the same time zone, so you
may want to give me some time to answer.

Jon Smirl writes:
 > Hopefully someone who knows what is going on with VT switching and how
 > hardware gets enabled will respond and we can get this fixed in the
 > server.  I see Zoltan's patch but we shouldn't have to tell X to leave
 > hardware alone that doesn't belong to it. X just has no business
 > messing with cards it does not own.

Err. We can reenable the card if no other card is interested in this
kind of legacy resource (we distinguish between mem and io resources
as we can disable them separately). And as far as I remember X does
this already when it is not in setup mode.
However if your X driver says it needs these resources - during 
'normal' operation, not setup - then there is no way around turning
of other cards.

 > 
 > Meanwhile I am forced to write to PCI config space and reenable IO
 > access from inside my interrupt handler. Yuck, yuck, yuck!!!

Even with a central resource manager (regardless if you have it inside
the kernel or not) your interrupt code would have to do this - as 
you don't know if the code you are coming from needed to access these 
these ports on another card.

Egbert.

Re: Who is stomping PCI config space?

[Egbert Eich]

Benjamin Herrenschmidt writes:
 > 
 > I had a report of something similar with X and a mach64 ... I haven't
 > yet investigated, but basically, X is using "fbdev" (so no specific
 > driver) and seem to be disabling memory access to the card, thus blowing
 > up the kernel fbdev.
 > 

Benjamin,

I faintly remember adding code to the Xserver that it wouldn't do this
when you only used fbdev drivers. 
It may do this for setup (ie. probing) as it has no idea about the drivers
it is going to encounter. With the new interface I've added we may actually
change this. I'm already doing this to determine if I need to enable PIO.

I need to look at this code again, though.

Egbert.

Re: [Linux-fbdev-devel] Re: Who is stomping PCI config space?

[Benjamin Herrenschmidt]

On Fri, 2005-03-04 at 12:25 +0100, Egbert Eich wrote:
> Benjamin Herrenschmidt writes:
>  > 
>  > I had a report of something similar with X and a mach64 ... I haven't
>  > yet investigated, but basically, X is using "fbdev" (so no specific
>  > driver) and seem to be disabling memory access to the card, thus blowing
>  > up the kernel fbdev.
>  > 
> 
> Benjamin,
> 
> I faintly remember adding code to the Xserver that it wouldn't do this
> when you only used fbdev drivers. 
> It may do this for setup (ie. probing) as it has no idea about the drivers
> it is going to encounter. With the new interface I've added we may actually
> change this. I'm already doing this to determine if I need to enable PIO.
> 
> I need to look at this code again, though.

Ok. In this case, the fbdev only uses mmio. Anyway, I'll investigate as
soon as I can install the repro case (FC3 distro) on an mach64 machine,
probably early next week.

Re: [Linux-fbdev-devel] Re: Who is stomping PCI config space?

[Egbert Eich]

Benjamin Herrenschmidt writes:
 > 
 > Ok. In this case, the fbdev only uses mmio. Anyway, I'll investigate as
 > soon as I can install the repro case (FC3 distro) on an mach64 machine,
 > probably early next week.
 > 

Ben,

I'll also revisit the special fbdev code that I wrote as soon as I 
get some time.

Cheers,
	Egbert.

Re: Who is stomping PCI config space?

[Egbert Eich]

Jon Smirl writes:
 > 
 > Somebody is changing PCI command from 83 to 80 and disabling my
 > radeon's memory and iospace. Who is doing this? It has to be X since
 > it doesn't happen if I switch to VT6 or VT8.
 > 
 > Why is X mucking with a card it doesn't have a driver loaded for? 
 > Where is this happening in the X code?
 > 

Jon,

Yes, X is donig this as it is assuming that it is in 
charge of all the graphics hardware in the machine.

It has to make this assumption as it cannot be sure that 
the card has *no* legacy resources open. Without a driver 
loaded it has no way of knowing any differently.

It doesn't matter if these legacy ports are used on this 
card, it only matters these resources are used by another 
card for setup purposes. 

A lot of cards do have legacy ports open - even if nobody 
uses them and without the driver the Xserver cannot really 
do much about this.

At the time this was programmed this assumption was 
necessary for the fast majority of cards.
I claim it is still required for a large number of 
cards today as when the card has been POSTed the 
BIOS usually leaves the legacy ports open.

So far we have no central agent available to schedule
access to shared resources.

If you want to access these cards from inside your 
kernel you could just check if they are enabled before 
you do and enable them if they aren't. 
This works as long as you can be sure nobody else will 
access legacy resources from anywhere else - this may 
be difficult if you have a machine with more than one 
CPU, if you have more than one userland process that 
may want to access the HW.
You should make sure to restore the state before you 
return.

As a note on the side: 
The Xserver doesn't 'stomp' over PCI config space any 
more without telling the kernel: The code which does 
direct PIO banging for config space access doesn't 
get used any more unless the user explicitely *asks*
for it. 
The option to do so is well hidden from the user.
This has been this way since X.Org 6.8.0 - I didn't 
change this for 6.7.0 for lack of testing.

Egbert.

Re: Who is stomping PCI config space?

[Benjamin Herrenschmidt]

> If you want to access these cards from inside your 
> kernel you could just check if they are enabled before 
> you do and enable them if they aren't. 
> This works as long as you can be sure nobody else will 
> access legacy resources from anywhere else - this may 
> be difficult if you have a machine with more than one 
> CPU, if you have more than one userland process that 
> may want to access the HW.
> You should make sure to restore the state before you 
> return.

Saving/restoring state doesn't help because of the SMP case.

> As a note on the side: 
> The Xserver doesn't 'stomp' over PCI config space any 
> more without telling the kernel: The code which does 
> direct PIO banging for config space access doesn't 
> get used any more unless the user explicitely *asks*
> for it. 
> The option to do so is well hidden from the user.
> This has been this way since X.Org 6.8.0 - I didn't 
> change this for 6.7.0 for lack of testing.
> 
> Egbert.
> _______________________________________________
> xorg mailing list
> xorg@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/xorg
-- 
Benjamin Herrenschmidt <benh@kernel.crashing.org>

Re: Who is stomping PCI config space?

[Egbert Eich]

Zoltan Boszormenyi writes:
 > Hm. Someone other than a Linux-ruby user finally complaining. :-)
 > Try patching XFree or XOrg with the proper patch
 > 
 > http://www.ltn.lv/~aivils/files/080_pci_isolate_device_feature.diff
 > 
 > for XFree, or
 > 
 > http://www.ltn.lv/~aivils/files/Xorg-6.7.0-isolate_device.patch
 > 
 > for XOrg, it's applicable for XOrg-6.8.x, too.
 > 
 > Then add
 >           Option         "SingleCard" "on"
 > to your Section "ServerLayout".
 > 

This patch adds another wart to the resource broker ;-)
But it may actually help the people who need this advanced 
feature and know what they are doing.
I have thought about putting something like this in but was
discouraged when I saw users trying out every single option
and then complaining that it didn't work.

I would rather go with something 'automatic'. This would be
a resource broker outside of X.

Still I'm inclined to adding something like this to the code
however I'm reluctant to document it so well ;-)

Cheers,
	Egbert.