megaraid_sata vs megasr

megaraid_sata vs megasr

[Maarten Vanraes]

Hello, (can you CC me? i'm not on the list)

I have a LSI MegaRAID SAS 8204ELP card, and since it wasn't recognized by any 
driver, i tried using the megasr driver supplied by LSI (it also supplies 
open_source code with it), i had to modify the code a bit to get it to boot 
with kernel-2.6.21 . but there's a very high iowait on it. i'm wondering if 
the megaraid_sas driver is meant to be the driver for the same hardware as 
megasr? would it help if i just added another PCI ID to the module and try 
that one?

Sincerely,

Maarten Vanraes

[] # lspci -s 08:00.0 -vvvn
08:00.0 0100: 1000:0059 (rev 08)
        Subsystem: 1000:3002
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ 
Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
<TAbort- <MAbort- >SERR+ <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        Interrupt: pin A routed to IRQ 18
        Region 0: I/O ports at 3000 [size=256]
        Region 1: Memory at d8410000 (64-bit, non-prefetchable) [size=16K]
        Region 3: Memory at d8400000 (64-bit, non-prefetchable) [size=64K]
        [virtual] Expansion ROM at d8000000 [disabled] [size=2M]
        Capabilities: [50] Power Management version 2
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA 
PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [68] Express (v1) Endpoint, MSI 00
                DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, 
L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE- FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- 
Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 4096 bytes
                DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- 
TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x8, ASPM L0s L1, Latency 
L0 <64ns, L1 <1us
                        ClockPM- Suprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- 
CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk- 
DLActive- BWMgmt- ABWMgmt-
        Capabilities: [98] Message Signalled Interrupts: Mask- 64bit+ 
Queue=0/0 Enable-
                Address: 0000000000000000  Data: 0000
        Capabilities: [b0] MSI-X: Enable- Mask- TabSize=1
                Vector table: BAR=1 offset=00002000
                PBA: BAR=1 offset=00003000
        Capabilities: [100] Advanced Error Reporting <?>
        Kernel driver in use: megasr


-- 
Maarten Vanraes
BA NV: IT & Security

Re: megaraid_sata vs megasr

[Jeff Garzik]

Maarten Vanraes wrote:
> Hello, (can you CC me? i'm not on the list)
> 
> I have a LSI MegaRAID SAS 8204ELP card, and since it wasn't recognized by any 
> driver, i tried using the megasr driver supplied by LSI (it also supplies 
> open_source code with it), i had to modify the code a bit to get it to boot 
> with kernel-2.6.21 . but there's a very high iowait on it. i'm wondering if 
> the megaraid_sas driver is meant to be the driver for the same hardware as 
> megasr? would it help if i just added another PCI ID to the module and try 
> that one?
> 
> Sincerely,
> 
> Maarten Vanraes
> 
> [] # lspci -s 08:00.0 -vvvn
> 08:00.0 0100: 1000:0059 (rev 08)
>         Subsystem: 1000:3002
>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ 
> Stepping- SERR+ FastB2B- DisINTx-
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
> <TAbort- <MAbort- >SERR+ <PERR- INTx-
>         Latency: 0, Cache Line Size: 32 bytes
>         Interrupt: pin A routed to IRQ 18
>         Region 0: I/O ports at 3000 [size=256]
>         Region 1: Memory at d8410000 (64-bit, non-prefetchable) [size=16K]
>         Region 3: Memory at d8400000 (64-bit, non-prefetchable) [size=64K]
>         [virtual] Expansion ROM at d8000000 [disabled] [size=2M]
>         Capabilities: [50] Power Management version 2
>         Capabilities: [68] Express (v1) Endpoint, MSI 00
>         Capabilities: [98] Message Signalled Interrupts: Mask- 64bit+ 
>         Capabilities: [b0] MSI-X: Enable- Mask- TabSize=1
>         Capabilities: [100] Advanced Error Reporting <?>

Yeah you can try drivers/scsi/megaraid/megaraid_sas.c or 
drivers/message/fusion/mptsas.c.

A URL to the megasr driver would be nice.

Also, please keep linux-scsi in CC (I added it), that's where this sort 
of things is usually discussed.

	Jeff

Re: megaraid_sata vs megasr

[Maarten Vanraes]

[-- Attachment #1: Type: text/plain, Size: 3425 bytes --]

Hi,

link where i downloaded the driver:

http://www.lsi.com/storage_home/products_home/internal_raid/megaraid_sas/megaraid_sas_8204elp/index.html?locale=EN&remote=1

i downloaded the 09.32_RedHat5.zip , there are, next to the binaries, also an 
open_source directory, where you can compile the driver. however, this driver 
doesn't work on recent kernels. I made a patch to let it work on 2.6.21 (it 
may not be clean).

there is a CPU soft lockup warning at boot time, when the module is used, (but 
that was there even on RH5 with the old kernel and binary driver. so, i doubt 
it was my doing.)

In the megaraid_sata (i've looked at it), there's 2 versions: xscale and ppc 
or something; which should i use?

and lastly: mpt fusion? that sounds like a totally different driver, any 
reason as why to use that one?

We have 2 identical machines to be used as xen hosts, which both have the same 
card. the first one is already in production, since we saw the iowait problem 
much later, the second one is empty for now, and i keep it that way for now, 
so i can test out on it. Of course, time is ticking away, since there are 
virtual machines being sold :-/ . You know how management works...

Sincerely,

Maarten Vanraes
PS: i just saw that there is a newer driver, but i suspect the source didn't 
change, it's probably a recompile for the new RH5 kernel. However, i'll see 
if there are any diff in it, and if so, i'll test it out this week.

Op maandag 26 mei 2008 19:11, schreef Jeff Garzik:
> Maarten Vanraes wrote:
> > Hello, (can you CC me? i'm not on the list)
> >
> > I have a LSI MegaRAID SAS 8204ELP card, and since it wasn't recognized by
> > any driver, i tried using the megasr driver supplied by LSI (it also
> > supplies open_source code with it), i had to modify the code a bit to get
> > it to boot with kernel-2.6.21 . but there's a very high iowait on it. i'm
> > wondering if the megaraid_sas driver is meant to be the driver for the
> > same hardware as megasr? would it help if i just added another PCI ID to
> > the module and try that one?
> >
> > Sincerely,
> >
> > Maarten Vanraes
> >
> > [] # lspci -s 08:00.0 -vvvn
> > 08:00.0 0100: 1000:0059 (rev 08)
> >         Subsystem: 1000:3002
> >         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> > ParErr+ Stepping- SERR+ FastB2B- DisINTx-
> >         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> > <TAbort- <MAbort- >SERR+ <PERR- INTx-
> >         Latency: 0, Cache Line Size: 32 bytes
> >         Interrupt: pin A routed to IRQ 18
> >         Region 0: I/O ports at 3000 [size=256]
> >         Region 1: Memory at d8410000 (64-bit, non-prefetchable)
> > [size=16K] Region 3: Memory at d8400000 (64-bit, non-prefetchable)
> > [size=64K] [virtual] Expansion ROM at d8000000 [disabled] [size=2M]
> > Capabilities: [50] Power Management version 2
> >         Capabilities: [68] Express (v1) Endpoint, MSI 00
> >         Capabilities: [98] Message Signalled Interrupts: Mask- 64bit+
> >         Capabilities: [b0] MSI-X: Enable- Mask- TabSize=1
> >         Capabilities: [100] Advanced Error Reporting <?>
>
> Yeah you can try drivers/scsi/megaraid/megaraid_sas.c or
> drivers/message/fusion/mptsas.c.
>
> A URL to the megasr driver would be nice.
>
> Also, please keep linux-scsi in CC (I added it), that's where this sort
> of things is usually discussed.
>
> 	Jeff

-- 
Maarten Vanraes
BA NV: IT & Security

[-- Attachment #2: kernel_2.6.20.patch --]
[-- Type: text/x-diff, Size: 3264 bytes --]

--- open_source/osl/lin/linux_osl.c	2008-02-07 17:23:36.000000000 +0000
+++ open_source/osl/lin/linux_osl.c	2008-04-10 14:13:10.000000000 +0000
@@ -66,7 +66,7 @@
 
 
 #if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,0)
- module_param(dbglvl_g, uint, DRIVER_DBG_LEVEL);
+ module_param(dbglvl_g, uint, 0);
 #else 
 #ifdef MODULE
  MODULE_PARM			(dbglvl_g, "i");
@@ -90,7 +90,7 @@
 #endif
 
 /* Function Prototypes */
-static LSI_RAID_IRQRETURN_T lsraid_isr(int irq, void *devp, struct pt_regs *regs);
+static LSI_RAID_IRQRETURN_T lsraid_isr(int irq, void *devp);
 
 /* helper functions */
 static int __devinit lsraid_probe_one(struct pci_dev*, 
@@ -556,9 +556,10 @@
  * This is a tasklet, Linux guarantees, we are not running on more than one CPU at the
  * same time.
  **/
-static void megasr_linux_build_io(void *devp)
+
+static void megasr_linux_build_io(struct work_struct *work)
 {
-	linux_adp_t				*linux_adp = (linux_adp_t *) devp;
+	linux_adp_t				*linux_adp = container_of(work, linux_adp_t, work_build_io_in);
 	rcl_packet_public_t			*rclp;
 	rcl_packet_public_t			*rclp_tmp;
 	struct scsi_cmnd		*scp;
@@ -634,9 +635,9 @@
  * counterpart for the "sleepy" build IO routine. We complete the scsi commands to the mid-layer
  * These commands are those for which virtual mapping, using kmap(), was obtained.
  **/
-static void megasr_linux_dpc_complete_cmd(void *devp)
+static void megasr_linux_dpc_complete_cmd(struct work_struct *work)
 {
-	linux_adp_t				*linux_adp = (linux_adp_t *)devp;
+	linux_adp_t				*linux_adp = container_of(work, linux_adp_t, work_build_io_out);
 	rcl_packet_public_t			*rclp;
 	rcl_packet_public_t			*rclp_tmp;
 	struct scsi_cmnd		*scp;
@@ -961,8 +962,7 @@
  *@remarks
  * This is a driver entry function 
  */
-static LSI_RAID_IRQRETURN_T lsraid_isr(int irq, void* os_cxt, 
-		struct pt_regs* regs)
+static LSI_RAID_IRQRETURN_T lsraid_isr(int irq, void* os_cxt)
 {
 	int				ret_val = LSI_FALSE;
 	unsigned long	flags;
@@ -1073,9 +1073,8 @@
 	// setup work queues for DPC
 	megasr_clist_setup(&lxadp->build_io_list_in);
 	megasr_clist_setup(&lxadp->build_io_list_out);
-	INIT_WORK(&lxadp->work_build_io_in, megasr_linux_build_io, lxadp);
-	INIT_WORK(&lxadp->work_build_io_out, megasr_linux_dpc_complete_cmd,
-	lxadp);
+	INIT_WORK(&lxadp->work_build_io_in, megasr_linux_build_io);
+	INIT_WORK(&lxadp->work_build_io_out, megasr_linux_dpc_complete_cmd);
 #endif
 
 #ifdef __LP64__
--- open_source/osl/lin/linux_osl.h	2008-02-07 17:23:36.000000000 +0000
+++ open_source/osl/lin/linux_osl.h	2008-04-10 12:47:36.000000000 +0000
@@ -111,7 +111,8 @@
 #define LSI_RAID_SCP2_LUN(scp)		(scp)->device->lun		/* LUN */
 #define LSI_RAID_IRQRETURN_T 		irqreturn_t
 
-#define LSI_RAID_HOST_SET_LOCK(host, lock) scsi_assign_lock(host, lock)
+//#define LSI_RAID_HOST_SET_LOCK(host, lock) scsi_assign_lock(host, lock)
+#define LSI_RAID_HOST_SET_LOCK(host, lock) host->host_lock = lock;
 #define LSI_RAID_CURRENT_TIME_IN_SEC (CURRENT_TIME.tv_sec)
 
 #else
Alleen in test/open_source/osl/lin: linux_osl.o
Alleen in test/open_source/osl/lin: .linux_osl.o.cmd
Alleen in test/open_source/osl/lin: osl_char.o
Alleen in test/open_source/osl/lin: .osl_char.o.cmd
Alleen in test/open_source/oss/lin: lin_oss.o
Alleen in test/open_source/oss/lin: .lin_oss.o.cmd

RE: megaraid_sata vs megasr

[Moore, Eric]

On Tuesday, May 27, 2008 3:56 AM,  Maarten Vanraes wrote:
> and lastly: mpt fusion? that sounds like a totally different 
> driver, any 
> reason as why to use that one?

No, mpt fusion is not for megaraid controllers.    The megasr driver is
the "software raid" driver stack over mpt fusion controller,  which is
closed source, you should contact the megaraid team to get the entire
driver.   The megaraid team also have their own bios to be used with
these controllers, this bios will create a default raid volume when no
one is present, thus corrupting the data,  hense you don't want to use
the mpt fusion drivers for that megaraid controller.  The megasas driver
is full hardware raid solution, like raid levels 0, 1, 5, 6, 10, 50, and
60.

Eric

Re: megaraid_sata vs megasr

[Maarten Vanraes]

So in effect, you are telling me that we bought fakeraid cards... since this 
is the "recommended driver"? is the card a hardware raid? or software.

We had specifically asked for hardware raid... :-(

Sincerely,

Maarten

Op Tuesday 27 May 2008 22:32:55 schreef Moore, Eric:
> On Tuesday, May 27, 2008 3:56 AM,  Maarten Vanraes wrote:
> > and lastly: mpt fusion? that sounds like a totally different
> > driver, any
> > reason as why to use that one?
>
> No, mpt fusion is not for megaraid controllers.    The megasr driver is
> the "software raid" driver stack over mpt fusion controller,  which is
> closed source, you should contact the megaraid team to get the entire
> driver.   The megaraid team also have their own bios to be used with
> these controllers, this bios will create a default raid volume when no
> one is present, thus corrupting the data,  hense you don't want to use
> the mpt fusion drivers for that megaraid controller.  The megasas driver
> is full hardware raid solution, like raid levels 0, 1, 5, 6, 10, 50, and
> 60.
>
> Eric

RE: megaraid_sata vs megasr

[Moore, Eric]

On Wednesday, May 28, 2008 8:18 AM, Maarten Vanraes wrote:
> 
> So in effect, you are telling me that we bought fakeraid 
> cards... since this 
> is the "recommended driver"? is the card a hardware raid? or software.
> 
> We had specifically asked for hardware raid... :-(
> 

I worked on megaraid team some five years ago, now I work on mpt fusion.
I can't tell you what the recommended megaraid cards, as they have
changed since then from U320.   The mpt fusion cards come with Raid
levels 0, 1, and 1 enhanced, which is implemented in hardware/firmware.


You could try looking here:  
http://www.lsi.com/storage_home/products_home/internal_raid/index.html

Re: megaraid_sata vs megasr

[Maarten Vanraes]

Op Wednesday 28 May 2008 17:44:50 schreef Moore, Eric:
> On Wednesday, May 28, 2008 8:18 AM, Maarten Vanraes wrote:
> > So in effect, you are telling me that we bought fakeraid
> > cards... since this
> > is the "recommended driver"? is the card a hardware raid? or software.
> >
> > We had specifically asked for hardware raid... :-(
>
> I worked on megaraid team some five years ago, now I work on mpt fusion.
> I can't tell you what the recommended megaraid cards, as they have
> changed since then from U320.   The mpt fusion cards come with Raid
> levels 0, 1, and 1 enhanced, which is implemented in hardware/firmware.
>
>
> You could try looking here:
> http://www.lsi.com/storage_home/products_home/internal_raid/index.html

i think it's this one: 
http://www.lsi.com/storage_home/products_home/internal_raid/megaraid_sas/megaraid_sas_8208elp/index.html

in any case: LSI support site directs me to the megasr driver for my card. But 
there is something wrong in either case:

on machine 2: i did an rsync from machine A array 2 to machine B array 2 (SATA 
disks of 1T in raid 1): speed is around 20MB/s probably due to load on 
machine A, or whatever (machine B has nothing running whatsoever.)

hdparm -tT /dev/sdb shows between 50 and 80MB/s

the moment i do an scp or rsync from something else, the first rsync drops 
immediately to around 5MB/s, while the second doesn't do more than 5MB/s 
(often one would show around 2MB/s.)

at that time, the iowait (dual quad core) is above 90% ...

Since both identical machines behave in the same matter, i believe there is 
something seriously wrong with the megasr driver in the first place, and 
tried to get support, but they request me to supply a serial number of the 
card. (Both machines are already running in a colocation room).

I had to patch it to work with 2.6.21, i'm wondering if i did anything stupid 
in the patch. can you check that maybe? (it's a small patch)

on the other hand, if there is an driver that works well, but just needs a 
little work, i'll be happy to test this out.

I'm just not a kernel devel, and don't really know how to solve this.

the problem is just that i want to be helped one way or another.

IF in your opinion the cards are not the best solution and i would have to 
upgrade it, i can do that; but i suspect the megasr driver.

am i wrong in my thinking?

Sincerely,

Maarten