Some discussion points open source friendly graphics [was: HARDWARE: Open-Source-Friendly Graphics Cards -- Viable?]

Some discussion points open source friendly graphics [was: HARDWARE: Open-Source-Friendly Graphics Cards -- Viable?]

[Timothy Miller]

I'm still trying to digest all the feedback I've been getting.  It's 
overwhelming and gratifying, and I want to offer my gratitude for all 
the discussion and ideas.

It's been pointed out that some of this discussion may be getting 
off-topic for LKML.  My addition to that is that soon, it will get very 
specialized in graphics math, and it would be best to move to a list 
where people actually talk about this on a regular basis.

To that end, I'd appreciate it people could point me to some other 
appropriate mailing lists.  Not just the names, but URL's to the FAQ's 
which explain exactly how to subscribe, please.

Also, I'm thinking of starting my own yahoo groups list specifically for 
this chip.  Is that a good idea?

Next, I'm getting lots of ideas from people.  Some of them are core to 
the product, and some of them would be nice for follow-on products.  For 
instance, dual-video would not be on the first model released.  However, 
it is important that analog output always have crisp rise and fall times 
and be free of noise in order to maximize display quality.

The reprogramability of the FPGA has many advantages, but 
reprogramability is not its primary purpose.  The primary reason to use 
an FPGA is to minimize NRE for manufacturing.  However, as a result, 
users will be able to download updates.  Additionally, those who are 
dedicated enough to reprogram it completely will find the necessary 
documentation to do so.  Finally, it is my desire that we would release 
the source code to the FPGA for obsoleted products, however, it's too 
early to make promises.


Ok, now on to some design stuff:

The picture I have in my head at this time expands on the idea of the 
setup engine seen in most GPU's.  What I'm thinking is that the setup 
engine will be general-purpose-ish CPU with special vector and matrix 
instructions.  This way, the transformation stage will occur in 
"software" executed by a specialized processor.  Additionally, the 
lighting phase might be done here as well.

The setup engine would produce triangle parameters which are fed to a 
rasterizer which does Gouraud shading and texture-mapping.  That feeds 
pixels into something that handles antialiasing and alpha blending, etc.

The advantages are:

- The community can customize the setup engine as they please, just by 
writing code.
- This also includes the 2D emulation
- Anything "missing" can be emulated.

The disadvantages are:

- Triangle rate limited by speed of processor
- T&L is serialized, rather than being parallelized in dedicated hardware
- Phong shading and bump mapping may be impossible or too slow

It's been a while since I read about phong shading and bump mapping.  As 
I understand it, some of the lighting phase is pushed into the 
rasterizer.  With gouraud shading, colors are interpolated between the 
virtexes.  With phong shading, the surface normals are interpolated 
across the triangle, and that's used to compute lighting.  Bump mapping 
is like phong shading where the normals are specified in a bitmap.

What I don't know is how important bump-mapping and phong shading are.

Re: Some discussion points open source friendly graphics

[karl.vogel]

> It's been pointed out that some of this discussion may be getting
> off-topic for LKML.  My addition to that is that soon, it will get
> very specialized in graphics math, and it would be best to move to a
> list where people actually talk about this on a regular basis.
>
> To that end, I'd appreciate it people could point me to some other
> appropriate mailing lists.  Not just the names, but URL's to the FAQ's
> which explain exactly how to subscribe, please.

Something like this?
  http://www.opencores.org/forums.cgi/video/

Re: Some discussion points open source friendly graphics [was: HARDWARE: Open-Source-Friendly Graphics Cards -- Viable?]

[Jeff Garzik]

Timothy Miller wrote:
> Also, I'm thinking of starting my own yahoo groups list specifically for 
> this chip.  Is that a good idea?

In theory it's useful; in reality at you'll have to balance what's 
public and what's not, and it's easy to allow yourself to get bogged 
down into "digesting", so much so that the "doing" is put off.

My only request is that you (a) make the graphics interface as simple 
and high level as feasible and (b) you publish the hardware interface 
specification as soon as possible, so that driver work can occur in 
parallel.


> Next, I'm getting lots of ideas from people.  Some of them are core to 
> the product, and some of them would be nice for follow-on products.  For 
> instance, dual-video would not be on the first model released.  However, 
> it is important that analog output always have crisp rise and fall times 
> and be free of noise in order to maximize display quality.

Once you have a core design, it's easier to dicker about specific 
features.  I would put this stuff on the "worry about it later" list.


> The reprogramability of the FPGA has many advantages, but 
> reprogramability is not its primary purpose.  The primary reason to use 
> an FPGA is to minimize NRE for manufacturing.  However, as a result, 
> users will be able to download updates.  Additionally, those who are 

Will the capability to apply these updates be included with the base card?
Will users need to purchase additional "update FPGA" hardware to do the 
reprogramming?


> Ok, now on to some design stuff:
> 
> The picture I have in my head at this time expands on the idea of the 
> setup engine seen in most GPU's.  What I'm thinking is that the setup 
> engine will be general-purpose-ish CPU with special vector and matrix 
> instructions.  This way, the transformation stage will occur in 
> "software" executed by a specialized processor.  Additionally, the 
> lighting phase might be done here as well.
> 
> The setup engine would produce triangle parameters which are fed to a 
> rasterizer which does Gouraud shading and texture-mapping.  That feeds 
> pixels into something that handles antialiasing and alpha blending, etc.
> 
> The advantages are:
> 
> - The community can customize the setup engine as they please, just by 
> writing code.
> - This also includes the 2D emulation
> - Anything "missing" can be emulated.
> 
> The disadvantages are:
> 
> - Triangle rate limited by speed of processor
> - T&L is serialized, rather than being parallelized in dedicated hardware
> - Phong shading and bump mapping may be impossible or too slow

Well, I certainly like it :)

I think that a more generic approach allows you to recognize performance 
bottlenecks, and update the IP core to support instructions specific to 
(say) triangles.

Random closing notes:

* data movement will be an everpresent issue

* in graphics you really have a number of data types (16-bit float, 
etc.) that are specific to graphics.  Supporting these natively should 
be quite helpful, if not an already-obvious prerequisite

	Jeff

Re: Some discussion points open source friendly graphics [was: HARDWARE: Open-Source-Friendly Graphics Cards -- Viable?]

[Timothy Miller]

Jeff Garzik wrote:
> Timothy Miller wrote:
> 
>> Also, I'm thinking of starting my own yahoo groups list specifically 
>> for this chip.  Is that a good idea?
> 
> 
> In theory it's useful; in reality at you'll have to balance what's 
> public and what's not, and it's easy to allow yourself to get bogged 
> down into "digesting", so much so that the "doing" is put off.

Right.  I was just discussing this with the CEO, and he pointed out that 
I would get bogged down with too much online discussion.  At FIRST, I 
might be able to handle an open list, but pretty quickly, I'd have to 
create an invite-only list where a select group who reads the other list 
would digest things for me.

Yeah, if I'm going to design a large chunk of the chip and write a large 
chunk of the software, I can't spend all day reading a mailing list.  :)

> 
> My only request is that you (a) make the graphics interface as simple 
> and high level as feasible and (b) you publish the hardware interface 
> specification as soon as possible, so that driver work can occur in 
> parallel.

Yes, and what I need to do is find the right developers to talk to so 
that I can given them interfaces that they actually NEED, because right 
now, I'm just guessing.

> 
> 
>> Next, I'm getting lots of ideas from people.  Some of them are core to 
>> the product, and some of them would be nice for follow-on products.  
>> For instance, dual-video would not be on the first model released.  
>> However, it is important that analog output always have crisp rise and 
>> fall times and be free of noise in order to maximize display quality.
> 
> 
> Once you have a core design, it's easier to dicker about specific 
> features.  I would put this stuff on the "worry about it later" list.

Definately.  There's loads of features that are NICE, but not useful to 
most people.  I need to make the lowest common demoninator first.

> 
>> The reprogramability of the FPGA has many advantages, but 
>> reprogramability is not its primary purpose.  The primary reason to 
>> use an FPGA is to minimize NRE for manufacturing.  However, as a 
>> result, users will be able to download updates.  Additionally, those 
>> who are 
> 
> 
> Will the capability to apply these updates be included with the base card?
> Will users need to purchase additional "update FPGA" hardware to do the 
> reprogramming?

My intention is to include a bit of logic in the FPGA which can be used 
to reprogram the prom.  You would then cycle power to get the FPGA to 
load the new code.  Only if it gets corrupted would you have to solder 
on a header and reprogram it another way.

Also, if someone decides to reprogram it completely, and they haven't 
included a host interface and prom programmer in their design, then, 
well, they're on their own.  :)

> 
>> Ok, now on to some design stuff:
>>
>> The picture I have in my head at this time expands on the idea of the 
>> setup engine seen in most GPU's.  What I'm thinking is that the setup 
>> engine will be general-purpose-ish CPU with special vector and matrix 
>> instructions.  This way, the transformation stage will occur in 
>> "software" executed by a specialized processor.  Additionally, the 
>> lighting phase might be done here as well.
>>
>> The setup engine would produce triangle parameters which are fed to a 
>> rasterizer which does Gouraud shading and texture-mapping.  That feeds 
>> pixels into something that handles antialiasing and alpha blending, etc.
>>
>> The advantages are:
>>
>> - The community can customize the setup engine as they please, just by 
>> writing code.
>> - This also includes the 2D emulation
>> - Anything "missing" can be emulated.
>>
>> The disadvantages are:
>>
>> - Triangle rate limited by speed of processor
>> - T&L is serialized, rather than being parallelized in dedicated hardware
>> - Phong shading and bump mapping may be impossible or too slow
> 
> 
> Well, I certainly like it :)
> 
> I think that a more generic approach allows you to recognize performance 
> bottlenecks, and update the IP core to support instructions specific to 
> (say) triangles.
> 
> Random closing notes:
> 
> * data movement will be an everpresent issue
> 
> * in graphics you really have a number of data types (16-bit float, 
> etc.) that are specific to graphics.  Supporting these natively should 
> be quite helpful, if not an already-obvious prerequisite


Yeah, for the transformation phase, you need 32-bit floating point. 
After that, you can switch to 16.16 fixed point and get perfect results.

Re: Some discussion points open source friendly graphics [was: HARDWARE: Open-Source-Friendly Graphics Cards -- Viable?]

[Daniel Phillips]

On Monday 25 October 2004 18:45, Timothy Miller wrote:
> My intention is to include a bit of logic in the FPGA which can be used
> to reprogram the prom.  You would then cycle power to get the FPGA to
> load the new code.

Power cycle the whole machine, or software-reset the card?

Regards,

Daniel

Re: Some discussion points open source friendly graphics [was: HARDWARE: Open-Source-Friendly Graphics Cards -- Viable?]

[linux-os]

On Wed, 27 Oct 2004, Daniel Phillips wrote:

> On Monday 25 October 2004 18:45, Timothy Miller wrote:
>> My intention is to include a bit of logic in the FPGA which can be used
>> to reprogram the prom.  You would then cycle power to get the FPGA to
>> load the new code.
>
> Power cycle the whole machine, or software-reset the card?
>
> Regards,
>
> Daniel

The FPGA requires a physical reset-pin toggle to load new
bits into the gate-array. This could be software-toggled,
but that would require at least one external gate. This
is because the power-reset needs to reset the chip before
its bits are loaded plus some pin, after loading, needs to
be an output to its reset pin also. Therefore, you need
the external gate that is always present.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.9 on an i686 machine (5537.79 BogoMips).
  Notice : All mail here is now cached and reviewed by John Ashcroft.
                  98.36% of all statistics are fiction.

Re: Some discussion points open source friendly graphics [was: HARDWARE: Open-Source-Friendly Graphics Cards -- Viable?]

[Timothy Miller]

linux-os wrote:
> On Wed, 27 Oct 2004, Daniel Phillips wrote:
> 
>> On Monday 25 October 2004 18:45, Timothy Miller wrote:
>>
>>> My intention is to include a bit of logic in the FPGA which can be used
>>> to reprogram the prom.  You would then cycle power to get the FPGA to
>>> load the new code.
>>
>>
>> Power cycle the whole machine, or software-reset the card?
>>
>> Regards,
>>
>> Daniel
> 
> 
> The FPGA requires a physical reset-pin toggle to load new
> bits into the gate-array. This could be software-toggled,
> but that would require at least one external gate. This
> is because the power-reset needs to reset the chip before
> its bits are loaded plus some pin, after loading, needs to
> be an output to its reset pin also. Therefore, you need
> the external gate that is always present.


Generally speaking, more chips == bad.  If reprogramming the FPGA were a 
regular event, I'd see the point, but I hope that MOST users would NEVER 
have to reprogram it.

Re: Some discussion points open source friendly graphics [was: HARDWARE: Open-Source-Friendly Graphics Cards -- Viable?]

[Timothy Miller]

Daniel Phillips wrote:
> On Monday 25 October 2004 18:45, Timothy Miller wrote:
> 
>>My intention is to include a bit of logic in the FPGA which can be used
>>to reprogram the prom.  You would then cycle power to get the FPGA to
>>load the new code.
> 
> 
> Power cycle the whole machine, or software-reset the card?

I think you'll have to power-cycle the whole machine.

1) You have to get the FPGA's loader logic to reload the bitstream. 
While there are external signals to get the FPGA to reload, there may be 
a chicken&egg problem with getting soft logic in the FPGA to tell the 
FPGA to reload.

2) When you have reloaded the FPGA, registers are in an unknown state. 
One important thing that is lost if the PCI config space info.  That 
would have to be restored by software.


Is it impossible to soft-reset it?  Maybe not.  But so much is lost and 
broken in the process that it may be highly impractical.

Nevertheless, if there's a way to do it, and the community wants to 
develop it, I'm all in favor.

Re: Some discussion points open source friendly graphics [was: HARDWARE: Open-Source-Friendly Graphics Cards -- Viable?]

[Giuseppe Bilotta]

Timothy Miller wrote:
> > The reprogramability of the FPGA has many advantages, but 
> > reprogramability is not its primary purpose.  The primary reason to use 
> > an FPGA is to minimize NRE for manufacturing.  However, as a result, 
> > users will be able to download updates.  Additionally, those who are 

Jeff Garzik wrote:
> Will the capability to apply these updates be included with the base card?
> Will users need to purchase additional "update FPGA" hardware to do the 
> reprogramming?

Also, what if the reprogramming goes wrong? Do I just throw the 
card away or will there be some form of recovery possible?

-- 
Giuseppe "Oblomov" Bilotta

Can't you see
It all makes perfect sense
Expressed in dollar and cents
Pounds shillings and pence
                  (Roger Waters)

Re: Some discussion points open source friendly graphics [was: HARDWARE: Open-Source-Friendly Graphics Cards -- Viable?]

[Timothy Miller]

Giuseppe Bilotta wrote:
> Timothy Miller wrote:
> 
>>>The reprogramability of the FPGA has many advantages, but 
>>>reprogramability is not its primary purpose.  The primary reason to use 
>>>an FPGA is to minimize NRE for manufacturing.  However, as a result, 
>>>users will be able to download updates.  Additionally, those who are 
> 
> 
> Jeff Garzik wrote:
> 
>>Will the capability to apply these updates be included with the base card?
>>Will users need to purchase additional "update FPGA" hardware to do the 
>>reprogramming?
> 
> 
> Also, what if the reprogramming goes wrong? Do I just throw the 
> card away or will there be some form of recovery possible?
> 


For those who are taking the risk of reprogramming it completely, 
they'll already have read the schematics and instructions for using an 
external device to program the PROM.

For everyone else, it's the same problem you get when programming a 
motherboard goes awry.  When the BIOS is hosed, you can't use the MB 
until you replace the chip.

For cost reasons, we likely wouldn't socket the chip, so you'd probably 
have to send it in for an RMA.  We'd reprogram it, and send it back.  Or 
if you have a friend with the right tools, they can do it.

Re: Some discussion points open source friendly graphics [was: HARDWARE: Open-Source-Friendly Graphics Cards -- Viable?]

[linux-os]

On Tue, 26 Oct 2004, Timothy Miller wrote:

>
>
> Giuseppe Bilotta wrote:
> > Timothy Miller wrote:
> >
> >>>The reprogramability of the FPGA has many advantages, but
> >>>reprogramability is not its primary purpose.  The primary reason to use
> >>>an FPGA is to minimize NRE for manufacturing.  However, as a result,
> >>>users will be able to download updates.  Additionally, those who are
> >
> >
> > Jeff Garzik wrote:
> >
> >>Will the capability to apply these updates be included with the base card?
> >>Will users need to purchase additional "update FPGA" hardware to do the
> >>reprogramming?
> >
> >
> > Also, what if the reprogramming goes wrong? Do I just throw the
> > card away or will there be some form of recovery possible?
> >
>
>
> For those who are taking the risk of reprogramming it completely,
> they'll already have read the schematics and instructions for using an
> external device to program the PROM.
>
> For everyone else, it's the same problem you get when programming a
> motherboard goes awry.  When the BIOS is hosed, you can't use the MB
> until you replace the chip.
>
> For cost reasons, we likely wouldn't socket the chip, so you'd probably
> have to send it in for an RMA.  We'd reprogram it, and send it back.  Or
> if you have a friend with the right tools, they can do it.
>

Normally you use a boundary-scan (JTAG) serial header so you can program,
reprogram, debug the chip. FPGA development tools expect (require)
this.

Check out http:/www.macraigor.com/full_gnu.htm for their GNU tools
and devices, designed for Linux (and M$).


Cheers,
Dick Johnson
Penguin : Linux version 2.6.9 on an i686 machine (5537.79 GrumpyMips).
                 98.36% of all statistics are fiction.

Re: Some discussion points open source friendly graphics [was: HARDWARE: Open-Source-Friendly Graphics Cards -- Viable?]

[Timothy Miller]

linux-os wrote:
> On Tue, 26 Oct 2004, Timothy Miller wrote:
> 

>>For cost reasons, we likely wouldn't socket the chip, so you'd probably
>>have to send it in for an RMA.  We'd reprogram it, and send it back.  Or
>>if you have a friend with the right tools, they can do it.
>>
> 
> 
> Normally you use a boundary-scan (JTAG) serial header so you can program,
> reprogram, debug the chip. FPGA development tools expect (require)
> this.
> 
> Check out http:/www.macraigor.com/full_gnu.htm for their GNU tools
> and devices, designed for Linux (and M$).

Sounds cool.  The easier we can make it, the better.

Re: Some discussion points open source friendly graphics [was: HARDWARE: Open-Source-Friendly Graphics Cards -- Viable?]

[Timothy Miller]

pbecke wrote:
> In many of the FPGA designs I have done, I have always tried to use some 
> kind of parallel host interface port, that are built into many FPGAs to 
> program the FPGA.  This allows the FPGA to be loaded as if it is a 
> memory device, Yes, it does mean that some external glue logic is 
> usually required to decode address lines and such, but, it means that 
> you can eliminate the eeprom, and usually the loading to the devices it 
> self if much faster.  I have more experience with the ram based FPGAs 
> from Xilinx, then the eprom based ones from Actel, so it may not be 
> feasible with the eprom based ones.  But I really think that in system 
> configuration is an advantage, that opens some possibilities that are 
> much more difficult to implement if the design relies on a serial prom 
> for the source of the FPGA configuration data.

My first desire is to minimize chip count, which means one FPGA with as 
much of the logic in it as possible.  If we do multi-core designs, it 
would be logical to be able to reprogram each of the chips separately. 
I would bundle the PCI controller with the PROM reprogrammer and any 
other logic which doesn't change much and put the rest into the other 
chip(s).  This way, you could reprogram the core without risking the 
whole thing.

> 
> By the way have you considered opencores.org's VGA controller as a 
> starting point for a design?

I am aware of it, but I've hesitated to even look at it.  I respect the 
author's wish to put it under a GPL-like license, and I don't want to 
"cheat".  (The issue here isn't what's legal but what's ethical. 
Companies do bad things all the time which are perfectly legal, and I 
want to avoid that.)  The author of that VGA core posted to the 
kerneltrap thread.  Perhaps I can make some sort of arrangement with 
him.  For instance, a friend of mine sold a commercial program which had 
some GPL'd components, but he first asked the authors to relicense under 
LGPL.  I'd be happy to release my improvements to any PIECES that I got 
from elsewhere, but I want to have the option of not releasing the WHOLE 
thing.

Need help and advice... [was: HARDWARE: Open-Source-Friendly Graphics Cards -- Viable?]

[Timothy Miller]

I'm probably doing this too early, but it's been pointed out in so many 
words that I won't scale.  I've spent two full work days digesting all 
of the discussions on LKML, kerneltrap, and slashdot.  That's been as 
fun as you can imagine, but when the time comes for me to start working 
on the design, I can't do that at all anymore.

What I'll need are representatives of the community who work with me to 
make sure that my design does what everyone needs it to do, and I need 
representatives who engage in mailing lists and extract for me 
information that I should see.

I'm sure there are lots of people who would love to have that kind of 
influence over a company whose hardware they want to buy.  I need the 
subset of those people who have expertise in 3D graphics, most of whom 
I'm sure are on OpenGL and X.org mailing lists right now.  If there's 
anyone here who can get me connected with those people, I would 
appreciate it very much.


I feel like I'm being presumptuous.  Although everyone at Tech Source is 
excited by the idea, and there seems to be a ground-swell of interest, 
the economic feasibility of this is still in the air.  I don't want to 
waste anyone's time.  However, a lot of excellent ideas have been put 
forth, and I'd like discussion to continue, but in a way that I can handle.

I humbly request advice on how I might go about handling all of this in 
the best way.

I also think the discussion should transition off LKML and onto its own 
forum.

Also, I want to thank everyone for talking to me about this.  Members of 
this list and people posting to kerneltrap and slashdot have been very 
informative and helpful.  I also want to mention Jeremy Andrews of 
kerneltrap.org who first turned my posting into news.  I've wanted to do 
a project like this for a long time, but the past few days of online 
discussion have really helped push it towards becoming reality.

Re: Some discussion points open source friendly graphics [was: HARDWARE: Open-Source-Friendly Graphics Cards -- Viable?]

[Daniel Phillips]

On Monday 25 October 2004 11:54, Timothy Miller wrote:
> The reprogramability of the FPGA has many advantages, but
> reprogramability is not its primary purpose.

But it might turn out to be a reason for it turning into a geek trophy, if the 
price is not enormously higher than closed-spec cards.  You could for 
example, program real-time sound effects processing into the FPGA and output 
the samples through a standard sound card.

The enthusiast market is a big market these days.

> The picture I have in my head at this time expands on the idea of the
> setup engine seen in most GPU's.  What I'm thinking is that the setup
> engine will be general-purpose-ish CPU with special vector and matrix
> instructions.  This way, the transformation stage will occur in
> "software" executed by a specialized processor.  Additionally, the
> lighting phase might be done here as well.
>
> The setup engine would produce triangle parameters which are fed to a
> rasterizer which does Gouraud shading and texture-mapping.  That feeds
> pixels into something that handles antialiasing and alpha blending, etc.

I hope you're planning to have a divider available to the rasterizer for 
perspective interpolation, particularly of textures.

Regards,

Daniel

Re: Some discussion points open source friendly graphics [was: HARDWARE: Open-Source-Friendly Graphics Cards -- Viable?]

[Timothy Miller]

Daniel Phillips wrote:
> On Monday 25 October 2004 11:54, Timothy Miller wrote:
> 
>>The reprogramability of the FPGA has many advantages, but
>>reprogramability is not its primary purpose.
> 
> 
> But it might turn out to be a reason for it turning into a geek trophy, if the 
> price is not enormously higher than closed-spec cards.  You could for 
> example, program real-time sound effects processing into the FPGA and output 
> the samples through a standard sound card.
> 
> The enthusiast market is a big market these days.
> 
> 
>>The picture I have in my head at this time expands on the idea of the
>>setup engine seen in most GPU's.  What I'm thinking is that the setup
>>engine will be general-purpose-ish CPU with special vector and matrix
>>instructions.  This way, the transformation stage will occur in
>>"software" executed by a specialized processor.  Additionally, the
>>lighting phase might be done here as well.
>>
>>The setup engine would produce triangle parameters which are fed to a
>>rasterizer which does Gouraud shading and texture-mapping.  That feeds
>>pixels into something that handles antialiasing and alpha blending, etc.
> 
> 
> I hope you're planning to have a divider available to the rasterizer for 
> perspective interpolation, particularly of textures.


My plans are not quite that specific at this time.  For instance, I'm 
not entirely sure, yet, how perspective correction is done.