RE: Generic Disk Driver in Linux

RE: Generic Disk Driver in Linux

[Aleksey Gorelov]

>From: linux-kernel-owner@vger.kernel.org 
>[mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Jan Engelhardt
>>
>> I was curious that can we develop a generic disk driver that could
>> handle all the kinds of hard drives - IDE, SCSI, RAID et al?
>
>ide_generic
>sd_mod
>
>All there, what more do you want?

Unfortunately, not _all_. DMRAID does not support all fake raids yet. Moreover, there is usually
some gap for bleeding edge hw support.

>
>> I thought we could use the BIOS interrupt 13H for this purpose,
>>
>I fail to see a BIOS on non-x86 computers.
>
>> but ran into a LOT of real mode / protected mode issues.
>>
>Sure. We are not real mode.
>Ever heard of BIOS limitations? If no, first check out 
>http://www.pcguide.com/ref/hdd/bios/sizeGB8-c.html
  This is not really relevant. They actually answer it right in the article - BIOS extensions, aka
EDD.

  I'd say the performance issue is more in int13 non-dma vs. dma data transfers, and tricks
associated with vm86. Although most 'on board' BIOSes support dma just fine for single drives,
faik raid and 'plug-in' card option ROMS usually lacks any dma support. Plus, there is no way to
tell whether BIOS will do dma or pio upfront.
  Another problem here is that BIOSes usually support legacy PIC interrupt model, and modern OSes
use IO APIC and resteer the interrups.

Aleks.

>
>
>
>Jan Engelhardt
>-- 
>-
>To unsubscribe from this list: send the line "unsubscribe 
>linux-kernel" in

RE: Generic Disk Driver in Linux

[Arjan van de Ven]

On Thu, 2006-08-24 at 11:19 -0700, Aleksey Gorelov wrote:
> >From: linux-kernel-owner@vger.kernel.org 
> >[mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Jan Engelhardt
> >>
> >> I was curious that can we develop a generic disk driver that could
> >> handle all the kinds of hard drives - IDE, SCSI, RAID et al?
> >
> >ide_generic
> >sd_mod
> >
> >All there, what more do you want?
> 
> Unfortunately, not _all_. DMRAID does not support all fake raids yet. 

Hi,

it'll be easier and quicker to rev engineer 5 more formats than it will
be to get the bios thing working ;) And the performance of the bios
thing will be really really bad... (hint: real mode can access only 1Mb
of memory, so you will bounce buffer all IO's)

Greetings,
   Arjan van de Ven

RE: Generic Disk Driver in Linux

[Aleksey Gorelov]

--- Arjan van de Ven <arjan@infradead.org> wrote:

> On Thu, 2006-08-24 at 11:19 -0700, Aleksey Gorelov wrote:
> > >From: linux-kernel-owner@vger.kernel.org 
> > >[mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Jan Engelhardt
> > >>
> > >> I was curious that can we develop a generic disk driver that could
> > >> handle all the kinds of hard drives - IDE, SCSI, RAID et al?
> > >
> > >ide_generic
> > >sd_mod
> > >
> > >All there, what more do you want?
> > 
> > Unfortunately, not _all_. DMRAID does not support all fake raids yet. 
> 
> Hi,
> 
> it'll be easier and quicker to rev engineer 5 more formats than it will
> be to get the bios thing working ;) And the performance of the bios
probably true - I'm actually not great fan of originally proposed approach. But, 
unfortunately, manufactures and vendors still look more to MS. Until market 
situation changes, there is always a gap...

> thing will be really really bad... (hint: real mode can access only 1Mb
> of memory, so you will bounce buffer all IO's)
This is true for non-dma case only. As I already mentioned before, most BIOSes 
support dma, and there is no 1Mb limit for that (at least on modern hw).

Aleks.

> 
> Greetings,
>    Arjan van de Ven
> 

RE: Generic Disk Driver in Linux

[Arjan van de Ven]

On Thu, 2006-08-24 at 12:40 -0700, Aleksey Gorelov wrote:
> 
> --- Arjan van de Ven <arjan@infradead.org> wrote:
> 
> > On Thu, 2006-08-24 at 11:19 -0700, Aleksey Gorelov wrote:
> > > >From: linux-kernel-owner@vger.kernel.org 
> > > >[mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Jan Engelhardt
> > > >>
> > > >> I was curious that can we develop a generic disk driver that could
> > > >> handle all the kinds of hard drives - IDE, SCSI, RAID et al?
> > > >
> > > >ide_generic
> > > >sd_mod
> > > >
> > > >All there, what more do you want?
> > > 
> > > Unfortunately, not _all_. DMRAID does not support all fake raids yet. 
> > 
> > Hi,
> > 
> > it'll be easier and quicker to rev engineer 5 more formats than it will
> > be to get the bios thing working ;) And the performance of the bios
> probably true - I'm actually not great fan of originally proposed approach. But, 
> unfortunately, manufactures and vendors still look more to MS. Until market 
> situation changes, there is always a gap...

there are only so many different ways to describe raid0.
And those companies aren't going to keep changing that "just because",
changing costs them money, so there is an incentive for them to keep it
as is

> 
> > thing will be really really bad... (hint: real mode can access only 1Mb
> > of memory, so you will bounce buffer all IO's)
> This is true for non-dma case only. As I already mentioned before, most BIOSes 
> support dma, and there is no 1Mb limit for that (at least on modern hw).

but only DMA to lower regions usually

RE: Generic Disk Driver in Linux

[Aleksey Gorelov]

--- Arjan van de Ven <arjan@infradead.org> wrote:


> > > 
> > > it'll be easier and quicker to rev engineer 5 more formats than it will
> > > be to get the bios thing working ;) And the performance of the bios
> > probably true - I'm actually not great fan of originally proposed approach. But, 
> > unfortunately, manufactures and vendors still look more to MS. Until market 
> > situation changes, there is always a gap...
> 
> there are only so many different ways to describe raid0.
> And those companies aren't going to keep changing that "just because",
> changing costs them money, so there is an incentive for them to keep it
> as is

Bottom line is - today there is lack of support for it, and for some companies it 
means loosing money. Tomorrow there will be support for all raid0 variants, but 
as well some kind of new unsupported hw/technology - hence gap is still there, 
spiral thing :-( Nothing we can do here but to improve support in Linux today in
a hope it will take bigger market share tomorrow.

> 
> > 
> > > thing will be really really bad... (hint: real mode can access only 1Mb
> > > of memory, so you will bounce buffer all IO's)
> > This is true for non-dma case only. As I already mentioned before, most BIOSes 
> > support dma, and there is no 1Mb limit for that (at least on modern hw).
> 
> but only DMA to lower regions usually

  It depends on your memory manager. Virtual DMA services (aka VDS) are there and being
used from 1992 (for instance, in emm386 - Linux's memory management is more powerfull than that).

RE: Generic Disk Driver in Linux

[Arjan van de Ven]

On Thu, 2006-08-24 at 15:21 -0700, Aleksey Gorelov wrote:
> 
> --- Arjan van de Ven <arjan@infradead.org> wrote:
> 
> 
> > > > 
> > > > it'll be easier and quicker to rev engineer 5 more formats than it will
> > > > be to get the bios thing working ;) And the performance of the bios
> > > probably true - I'm actually not great fan of originally proposed approach. But, 
> > > unfortunately, manufactures and vendors still look more to MS. Until market 
> > > situation changes, there is always a gap...
> > 
> > there are only so many different ways to describe raid0.
> > And those companies aren't going to keep changing that "just because",
> > changing costs them money, so there is an incentive for them to keep it
> > as is
> 
> Bottom line is - today there is lack of support for it, 

can you name one?
(so far all you did was claim this but not name even one)

RE: Generic Disk Driver in Linux

[Aleksey Gorelov]

--- Arjan van de Ven <arjan@infradead.org> wrote:

> On Thu, 2006-08-24 at 15:21 -0700, Aleksey Gorelov wrote:
> > 
> > --- Arjan van de Ven <arjan@infradead.org> wrote:
> > 
> > 
> > > > > 
> > > > > it'll be easier and quicker to rev engineer 5 more formats than it will
> > > > > be to get the bios thing working ;) And the performance of the bios
> > > > probably true - I'm actually not great fan of originally proposed approach. But, 
> > > > unfortunately, manufactures and vendors still look more to MS. Until market 
> > > > situation changes, there is always a gap...
> > > 
> > > there are only so many different ways to describe raid0.
> > > And those companies aren't going to keep changing that "just because",
> > > changing costs them money, so there is an incentive for them to keep it
> > > as is
> > 
> > Bottom line is - today there is lack of support for it, 
> 
> can you name one?
> (so far all you did was claim this but not name even one)
> 
Here is a good support status
http://linuxmafia.com/faq/Hardware/sata.html
you can start with those who has proprietary binary only drivers.

I should admit though, that dmraid support has improved lately. For instance, 
they recently added Adaptec support, which has not been there before.

Re: Generic Disk Driver in Linux

[Phillip Susi]

The int 13 calls to the bios can only accept addresses within the first 
1 MB of memory, and the calls are synchronous, so DMA really doesn't 
matter as the cpu will be busy waiting anyhow while the IO takes place, 
which will wreak all kinds of hell on the rest of the running system, 
including other hardware ISRs.


Aleksey Gorelov wrote:
>> thing will be really really bad... (hint: real mode can access only 1Mb
>> of memory, so you will bounce buffer all IO's)
> This is true for non-dma case only. As I already mentioned before, most BIOSes 
> support dma, and there is no 1Mb limit for that (at least on modern hw).
> 
> Aleks.

Re: Generic Disk Driver in Linux

[Aleksey Gorelov]

--- Phillip Susi <psusi@cfl.rr.com> wrote:

> The int 13 calls to the bios can only accept addresses within the first 
> 1 MB of memory, and the calls are synchronous, so DMA really doesn't 
> matter as the cpu will be busy waiting anyhow while the IO takes place, 
> which will wreak all kinds of hell on the rest of the running system, 
> including other hardware ISRs.
> 
  Not that easy. Once you start int13 transfer, your vm86 thread may be 
preempted to allow cpu doing something useful, including other ISRs processing.

Re: Generic Disk Driver in Linux

[Helge Hafting]

Aleksey Gorelov wrote:
>> From: linux-kernel-owner@vger.kernel.org 
>> [mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Jan Engelhardt
>>     
>>> I was curious that can we develop a generic disk driver that could
>>> handle all the kinds of hard drives - IDE, SCSI, RAID et al?
>>>       
>> ide_generic
>> sd_mod
>>
>> All there, what more do you want?
>>     
>
> Unfortunately, not _all_. DMRAID does not support all fake raids yet. Moreover, there is usually
> some gap for bleeding edge hw support.
>   
Nobody will want to use bleeding edge hardware with an int13 driver,
because the performance will necessarily be much worse than using more
moderate hardware with the generic IDE driver.

Int13 for new hardware makes no sense to me.  The same goes for
fake raids - linux will usually be able to see the disks as single
disks, and you can then use them as such with a plain
software raid on top.  The fakeraid functionality is not needed,
it has no performance edge precicely because it is fake.

If fakeraid support is necessary to read some existing filesystem,
then implement support in md or dmraid or even as a userspace
filesystem.  Either choice will perform better, and also be
easier to do than making a working and stable int13 driver.


Helge Hafting

Re: Generic Disk Driver in Linux

[Aleksey Gorelov]

--- Helge Hafting <helge.hafting@aitel.hist.no> wrote:

> Aleksey Gorelov wrote:
> >> From: linux-kernel-owner@vger.kernel.org 
> >> [mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Jan Engelhardt
> >>     
> >>> I was curious that can we develop a generic disk driver that could
> >>> handle all the kinds of hard drives - IDE, SCSI, RAID et al?
> >>>       
> >> ide_generic
> >> sd_mod
> >>
> >> All there, what more do you want?
> >>     
> >
> > Unfortunately, not _all_. DMRAID does not support all fake raids yet. Moreover, there is
> usually
> > some gap for bleeding edge hw support.
> >   
> Nobody will want to use bleeding edge hardware with an int13 driver,
> because the performance will necessarily be much worse than using more
> moderate hardware with the generic IDE driver.
If some one wants Linux server - I totally agree. I would probably even avoid relying purely on
generic IDE and instead use chipset specific variant or libata.
But if someone wants to access already installed & working other OS stuff - that's a different
story. Bad performance is still better than no support at all.

> 
> Int13 for new hardware makes no sense to me.  The same goes for
> fake raids - linux will usually be able to see the disks as single
> disks, and you can then use them as such with a plain
> software raid on top.  The fakeraid functionality is not needed,
> it has no performance edge precicely because it is fake.
> 
> If fakeraid support is necessary to read some existing filesystem,
> then implement support in md or dmraid or even as a userspace
> filesystem.  Either choice will perform better, and also be
> easier to do than making a working and stable int13 driver.
> 
> 
> Helge Hafting
> 

You are right here. I only see the possibility of int13 driver as some 'backup' plan for HW that
not yet have enough support via native drivers, or if someone wants occasional access and does not
want to upgrade to latest kernels for some reason.

Re: Generic Disk Driver in Linux

[Greg Freemyer]

On 8/25/06, Aleksey Gorelov <dared1st@yahoo.com> wrote:
>
>
> --- Helge Hafting <helge.hafting@aitel.hist.no> wrote:
>
> > Aleksey Gorelov wrote:
> > >> From: linux-kernel-owner@vger.kernel.org
> > >> [mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Jan Engelhardt
> > >>
> > >>> I was curious that can we develop a generic disk driver that could
> > >>> handle all the kinds of hard drives - IDE, SCSI, RAID et al?
> > >>>
> > >> ide_generic
> > >> sd_mod
> > >>
> > >> All there, what more do you want?
> > >>
> > >
> > > Unfortunately, not _all_. DMRAID does not support all fake raids yet. Moreover, there is
> > usually
> > > some gap for bleeding edge hw support.
> > >
> > Nobody will want to use bleeding edge hardware with an int13 driver,
> > because the performance will necessarily be much worse than using more
> > moderate hardware with the generic IDE driver.
> If some one wants Linux server - I totally agree. I would probably even avoid relying purely on
> generic IDE and instead use chipset specific variant or libata.
> But if someone wants to access already installed & working other OS stuff - that's a different
> story. Bad performance is still better than no support at all.
>

I'll add a vote for this.  Not sure if the user base is large enough
to justify it being added to the mainline kernel, but Computer
Forensic professionals often need to image computers with Raid setups.
 (image == dd copy of logical volume)

For many configurations, the current choices are sub-optimal.  It
would be nice to have a boot disk that could capture any and all raid
volumes.

FYI: There are already many linux boot CDs that specifically target
Computer Forensics tasks, so there is a bigger user community than
many might assume.  Examples incude SPADA (law enforcement only),
smart, Farmer Dude's boot cd, FCCU GNU/Linux Forensic Boot CD, Helix,
Penguin Sleuth.

All of the above would benefit from a generic, but slow raid driver.
For this use it would be great if both real raid and fake raid were
supported.  Not sure if fake raid comes with int13 support or not.

Greg
-- 
Greg Freemyer
The Norcross Group
Forensics for the 21st Century

Re: Generic Disk Driver in Linux

[Helge Hafting]

Aleksey Gorelov wrote:
> --- Helge Hafting <helge.hafting@aitel.hist.no> wrote:
>
>   
>> Aleksey Gorelov wrote:
>>     
>>>> From: linux-kernel-owner@vger.kernel.org 
>>>> [mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Jan Engelhardt
>>>>     
>>>>         
>>>>> I was curious that can we develop a generic disk driver that could
>>>>> handle all the kinds of hard drives - IDE, SCSI, RAID et al?
>>>>>       
>>>>>           
>>>> ide_generic
>>>> sd_mod
>>>>
>>>> All there, what more do you want?
>>>>     
>>>>         
>>> Unfortunately, not _all_. DMRAID does not support all fake raids yet. Moreover, there is
>>>       
>> usually
>>     
>>> some gap for bleeding edge hw support.
>>>   
>>>       
>> Nobody will want to use bleeding edge hardware with an int13 driver,
>> because the performance will necessarily be much worse than using more
>> moderate hardware with the generic IDE driver.
>>     
> If some one wants Linux server - I totally agree. I would probably even avoid relying purely on
> generic IDE and instead use chipset specific variant or libata.
> But if someone wants to access already installed & working other OS stuff - that's a different
> story. Bad performance is still better than no support at all.
>   
Yes, there is forensics, and there is the copying of data when
migrating a server to linux from some other OS. 

Perhaps I worry too much - I imagine salesmen claiming linux
support for their raid cards, because the int13 driver works. 
Just like they claim "linux support" with abominations like ndiswrapper.
The bad effects may be carefully hidden with "disk" tests that are
smaller than main memory in some huge-memory machine.

Then linux gets a bad reputation among ignorant purchasers,
performance went so much up with some other os .  . .

Helge Hafting

Re: Generic Disk Driver in Linux

[Robert Hancock]

Daniel Rodrick wrote:
> Hi List,
> 
> I was curious that can we develop a generic disk driver that could
> handle all the kinds of hard drives - IDE, SCSI, RAID et al?
> 
> I thought we could use the BIOS interrupt 13H for this purpose, but
> ran into a LOT of real mode / protected mode issues.
> 
> Any other ideas?

Aside from the real/protected mode issues (which are pretty much a 
show-stopper right there), the performance would be horrible. Usually 
the Interrupt 13 code on especially the more advanced controllers like 
hardware RAID, etc. is designed solely to function, with little 
attention to speed, and the performance tends to be horrible - look at 
versions of Ghost that run into DOS and look how long it takes to image 
a drive connected to an IBM ServeRAID controller..

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

Generic Disk Driver in Linux

[Daniel Rodrick]

Hi List,

I was curious that can we develop a generic disk driver that could
handle all the kinds of hard drives - IDE, SCSI, RAID et al?

I thought we could use the BIOS interrupt 13H for this purpose, but
ran into a LOT of real mode / protected mode issues.

Any other ideas?

Thanks,

Dan.

Re: Generic Disk Driver in Linux

[Jan Engelhardt]

>
> I was curious that can we develop a generic disk driver that could
> handle all the kinds of hard drives - IDE, SCSI, RAID et al?

ide_generic
sd_mod

All there, what more do you want?

> I thought we could use the BIOS interrupt 13H for this purpose,
>
I fail to see a BIOS on non-x86 computers.

> but ran into a LOT of real mode / protected mode issues.
>
Sure. We are not real mode.
Ever heard of BIOS limitations? If no, first check out 
http://www.pcguide.com/ref/hdd/bios/sizeGB8-c.html



Jan Engelhardt
-- 

Re: Generic Disk Driver in Linux

[Ralf Baechle]

On Thu, Aug 24, 2006 at 06:17:59PM +0530, Daniel Rodrick wrote:

> I was curious that can we develop a generic disk driver that could
> handle all the kinds of hard drives - IDE, SCSI, RAID et al?
> 
> I thought we could use the BIOS interrupt 13H for this purpose, but
> ran into a LOT of real mode / protected mode issues.

A BIOS only contains a better than nothing quality driver, take the VESA
VLB driver as an example.  And lacks portability, upgrading is a pain.

  Ralf