[Qemu-devel] [QEMU-KVM]: Megasas + TCM_Loop + SG_IO into Windows XP guests

[Qemu-devel] [QEMU-KVM]: Megasas + TCM_Loop + SG_IO into Windows XP guests

[Nicholas A. Bellinger]

Greetings Hannes and co,

I have been spending a bit of time trying Megasas HBA emulation +
TCM_Loop + SG_IO with a Windows XP SP2 KVM guests..  So far, I noticed
that hw/scsi-generic.c:execute_command_run() using bdev_aio_ioctl()
appears to be broken for XP guests, which causes the first 36-byte
INQUIRY sent via SG_IO to never make it back to QEMU and results in the
win32 LSI drive taking the LUN offline, et al.  Note that everything
does appear to be functioning as expected in kernel space for the first
INQUIRY with the TCM_Loop LLD and Linux/SCSI code (AFAICT) and Linux KVM
guests using megasas emulation are still working.

So, I ended up needing requiring the following quick hack for
hw/scsi-generic.c:execute_command_run() to make SG_IO function
synchronously using bdrv_ioctl(), which at least gets LUN registration
and basic control path CDBs working for the XP guest.

Here is how it looks in action on a v2.6.34-rc7 host so far:

http://www.linux-iscsi.org/images/TCM-KVM-megasas-XP-05132010.png


diff --git a/hw/scsi-generic.c b/hw/scsi-generic.c
index 6c58742..aa1eb83 100644
--- a/hw/scsi-generic.c
+++ b/hw/scsi-generic.c
@@ -140,6 +140,7 @@ static int execute_command_run(SCSIGenericReq *r,
 {
     BlockDriverState *bdrv = r->req.dev->conf.dinfo->bdrv;
     SCSIGenericState *s = DO_UPCAST(SCSIGenericState, qdev, r->req.dev);
+    int ret;
 
     r->io_header.interface_id = 'S';
     r->io_header.dxfer_direction = sgdir[r->req.cmd.mode];
@@ -161,11 +162,16 @@ static int execute_command_run(SCSIGenericReq *r,
     printf("\n");
     }
 #endif
+#if 0
     r->req.aiocb = bdrv_aio_ioctl(bdrv, SG_IO, &r->io_header, complete, r);
     if (r->req.aiocb == NULL) {
         BADF("execute_command: read failed !\n");
         return -1;
     }
+#else
+    ret = bdrv_ioctl(bdrv, SG_IO, &r->io_header);
+    complete((void *)r, ret);
+#endif
 
      *      return 0;
 }


Beyond the initial LUN registration and control CDB parts, doing bulk
DATA_SG_IO traffic is completing successfully (and everything looks sane
with TCM_Loop and Linux/SCSI) but it appears that the correct blocks are
not actually getting written/read by megasas.  This appears to be the
case with both hw/scsi-generic.c and hw/scsi-disk.c modes of operation
for megasas with the win32 XP guest.

So I was wondering if anyone aware of known issues with QEMU
asynchronous SG_IO into MSFT KVM guests with virtio or hw/lsi53c895a.c,
or would this be something specific to megasas HBA emulation and XP
guests..?

Hannes, which MSFT guest + driver did you get work stable with bulk
DATA_SG_IO and hw/scsi-disk.c..?

Best,

--nab

[Qemu-devel] Re: [QEMU-KVM]: Megasas + TCM_Loop + SG_IO into Windows XP guests

[Hannes Reinecke]

Nicholas A. Bellinger wrote:
> Greetings Hannes and co,
> 
> I have been spending a bit of time trying Megasas HBA emulation +
> TCM_Loop + SG_IO with a Windows XP SP2 KVM guests..  So far, I noticed
> that hw/scsi-generic.c:execute_command_run() using bdev_aio_ioctl()
> appears to be broken for XP guests, which causes the first 36-byte
> INQUIRY sent via SG_IO to never make it back to QEMU and results in the
> win32 LSI drive taking the LUN offline, et al.  Note that everything
> does appear to be functioning as expected in kernel space for the first
> INQUIRY with the TCM_Loop LLD and Linux/SCSI code (AFAICT) and Linux KVM
> guests using megasas emulation are still working.
> 
Now that is really odd. Have you checked if it works with the
'normal' KVM disk backend?

> So, I ended up needing requiring the following quick hack for
> hw/scsi-generic.c:execute_command_run() to make SG_IO function
> synchronously using bdrv_ioctl(), which at least gets LUN registration
> and basic control path CDBs working for the XP guest.
> 
> Here is how it looks in action on a v2.6.34-rc7 host so far:
> 
> http://www.linux-iscsi.org/images/TCM-KVM-megasas-XP-05132010.png
> 
> 
> diff --git a/hw/scsi-generic.c b/hw/scsi-generic.c
> index 6c58742..aa1eb83 100644
> --- a/hw/scsi-generic.c
> +++ b/hw/scsi-generic.c
> @@ -140,6 +140,7 @@ static int execute_command_run(SCSIGenericReq *r,
>  {
>      BlockDriverState *bdrv = r->req.dev->conf.dinfo->bdrv;
>      SCSIGenericState *s = DO_UPCAST(SCSIGenericState, qdev, r->req.dev);
> +    int ret;
>  
>      r->io_header.interface_id = 'S';
>      r->io_header.dxfer_direction = sgdir[r->req.cmd.mode];
> @@ -161,11 +162,16 @@ static int execute_command_run(SCSIGenericReq *r,
>      printf("\n");
>      }
>  #endif
> +#if 0
>      r->req.aiocb = bdrv_aio_ioctl(bdrv, SG_IO, &r->io_header, complete, r);
>      if (r->req.aiocb == NULL) {
>          BADF("execute_command: read failed !\n");
>          return -1;
>      }
> +#else
> +    ret = bdrv_ioctl(bdrv, SG_IO, &r->io_header);
> +    complete((void *)r, ret);
> +#endif
>  
>       *      return 0;
>  }
> 
> 
> Beyond the initial LUN registration and control CDB parts, doing bulk
> DATA_SG_IO traffic is completing successfully (and everything looks sane
> with TCM_Loop and Linux/SCSI) but it appears that the correct blocks are
> not actually getting written/read by megasas.  This appears to be the
> case with both hw/scsi-generic.c and hw/scsi-disk.c modes of operation
> for megasas with the win32 XP guest.
> 
Oh. Hmm.

> So I was wondering if anyone aware of known issues with QEMU
> asynchronous SG_IO into MSFT KVM guests with virtio or hw/lsi53c895a.c,
> or would this be something specific to megasas HBA emulation and XP
> guests..?
> 
> Hannes, which MSFT guest + driver did you get work stable with bulk
> DATA_SG_IO and hw/scsi-disk.c..?
> 
Well, I have two more patches for megasas.
The one is just a cleanup to remove duplicate definitions, but the
other contains a real issue with a misjudged cast in megasas_enqueue_frame().
Not sure if that helps here, but it's worth a try nevertheless.

I'll be sending them with separate mails.

Let's see if I can find some time working on the megasas emulation.
Maybe I find something.
Last time I checked it was with a Windows7 build, but I didn't do
any real tests there. Basically just checking if the system boots up :-)

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)

[Qemu-devel] Re: [QEMU-KVM]: Megasas + TCM_Loop + SG_IO into Windows XP guests

[Nicholas A. Bellinger]

On Fri, 2010-05-14 at 09:22 +0200, Hannes Reinecke wrote:
> Nicholas A. Bellinger wrote:
> > Greetings Hannes and co,
> > 
<SNIP>
> 
> > So, I ended up needing requiring the following quick hack for
> > hw/scsi-generic.c:execute_command_run() to make SG_IO function
> > synchronously using bdrv_ioctl(), which at least gets LUN registration
> > and basic control path CDBs working for the XP guest.
> > 
> > Here is how it looks in action on a v2.6.34-rc7 host so far:
> > 
> > http://www.linux-iscsi.org/images/TCM-KVM-megasas-XP-05132010.png
> > 
> > 
> > diff --git a/hw/scsi-generic.c b/hw/scsi-generic.c
> > index 6c58742..aa1eb83 100644
> > --- a/hw/scsi-generic.c
> > +++ b/hw/scsi-generic.c
> > @@ -140,6 +140,7 @@ static int execute_command_run(SCSIGenericReq *r,
> >  {
> >      BlockDriverState *bdrv = r->req.dev->conf.dinfo->bdrv;
> >      SCSIGenericState *s = DO_UPCAST(SCSIGenericState, qdev, r->req.dev);
> > +    int ret;
> >  
> >      r->io_header.interface_id = 'S';
> >      r->io_header.dxfer_direction = sgdir[r->req.cmd.mode];
> > @@ -161,11 +162,16 @@ static int execute_command_run(SCSIGenericReq *r,
> >      printf("\n");
> >      }
> >  #endif
> > +#if 0
> >      r->req.aiocb = bdrv_aio_ioctl(bdrv, SG_IO, &r->io_header, complete, r);
> >      if (r->req.aiocb == NULL) {
> >          BADF("execute_command: read failed !\n");
> >          return -1;
> >      }
> > +#else
> > +    ret = bdrv_ioctl(bdrv, SG_IO, &r->io_header);
> > +    complete((void *)r, ret);
> > +#endif
> >  
> >       *      return 0;
> >  }
> > 
> > 
> > Beyond the initial LUN registration and control CDB parts, doing bulk
> > DATA_SG_IO traffic is completing successfully (and everything looks sane
> > with TCM_Loop and Linux/SCSI) but it appears that the correct blocks are
> > not actually getting written/read by megasas.  This appears to be the
> > case with both hw/scsi-generic.c and hw/scsi-disk.c modes of operation
> > for megasas with the win32 XP guest.
> > 
> Oh. Hmm.
>
> > So I was wondering if anyone aware of known issues with QEMU
> > asynchronous SG_IO into MSFT KVM guests with virtio or hw/lsi53c895a.c,
> > or would this be something specific to megasas HBA emulation and XP
> > guests..?
> > 
> > Hannes, which MSFT guest + driver did you get work stable with bulk
> > DATA_SG_IO and hw/scsi-disk.c..?
> > 
> Well, I have two more patches for megasas.
> The one is just a cleanup to remove duplicate definitions, but the
> other contains a real issue with a misjudged cast in megasas_enqueue_frame().
> Not sure if that helps here, but it's worth a try nevertheless.
> 
> I'll be sending them with separate mails.
> 

Thanks, applied both patches to the megasas friendly qemu-kvm.git tree.

> Let's see if I can find some time working on the megasas emulation.
> Maybe I find something.
> Last time I checked it was with a Windows7 build, but I didn't do
> any real tests there. Basically just checking if the system boots up :-)
> 

Nothing fancy just yet.  This is involving a normal NTFS filesystem
format on a small TCM/FILEIO LUN using scsi-generic and a userspace
FILEIO with scsi-disk.

This involves the XP guest waiting until the very last READ_10 once the
format has completed (eg: all WRITE and VERIFY CDBs complete with GOOD
status AFAICT) before announcing that mkfs.ntfs failed without any
helpful exception message (due to missing metadata of some sort I would
assume..?)

So perhaps dumping QEMU and TCM_Loop SCSI payloads to determine if any
correct blocks from megasas_handle_io() are actually making it out to
KVM host is going to be my next option.  ;)

I might try with a newer version (and 64-bit) version of windows server
and see if that can produce some manner of more useful information for
us.

Best,

--nab

[Qemu-devel] Re: [QEMU-KVM]: Megasas + TCM_Loop + SG_IO into Windows XP guests

[Nicholas A. Bellinger]

On Fri, 2010-05-14 at 02:42 -0700, Nicholas A. Bellinger wrote:
> On Fri, 2010-05-14 at 09:22 +0200, Hannes Reinecke wrote:
> > Nicholas A. Bellinger wrote:
> > > Greetings Hannes and co,
> > > 
> <SNIP>
> > Let's see if I can find some time working on the megasas emulation.
> > Maybe I find something.
> > Last time I checked it was with a Windows7 build, but I didn't do
> > any real tests there. Basically just checking if the system boots up :-)
> > 
> 
> Nothing fancy just yet.  This is involving a normal NTFS filesystem
> format on a small TCM/FILEIO LUN using scsi-generic and a userspace
> FILEIO with scsi-disk.
> 
> This involves the XP guest waiting until the very last READ_10 once the
> format has completed (eg: all WRITE and VERIFY CDBs complete with GOOD
> status AFAICT) before announcing that mkfs.ntfs failed without any
> helpful exception message (due to missing metadata of some sort I would
> assume..?)
> 
> So perhaps dumping QEMU and TCM_Loop SCSI payloads to determine if any
> correct blocks from megasas_handle_io() are actually making it out to
> KVM host is going to be my next option.  ;)
> 

Greetings Hannes,

So I spent some more time with XP guests this weekend, and I noticed two
things immediately when using hw/lsi53c895a.c instead of hw/megasas.c
with the same two TCM_Loop SAS LUNs via SG_IO from last week:

1) With lsi53c895a, XP guests are able to boot successfully w/ out the
synchronous SG_IO hack that is currently required to get past the first
36-byte INQUIRY for megasas + XP SP2

2) With lsi53c895a, XP is able to successfully create and mount a NTFS
filesystem, reboot, and read blocks appear to be functioning properly.
FYI I have not run any 'write known pattern then read-back and compare
blocks' data integrity tests from with in the XP guests just yet, but I
am confident that TCM scatterlist -> se_mem_t mapping is working as
expected on the KVM Host.

Futhermore, after formatting a 5 GB TCM/FILEIO LUN with lsi53c895a, and
then rebooting with megasas with the same two configured TCM_Loop SG_IO
devices, it appears to be able to mount and read blocks successfully.
Attempting to write new blocks on the mounted filesystem also appears to
work to some degree, but throughput slows down to a crawl during XP
guest buffer cache flush, which is likely attributed to the use of my
quick SYNC SG_IO hack.

So it appears that there are two seperate issues here, and AFAICT they
both look to be XP and megasas specific.  For #2, it may be something
about the format of the incoming scatterlists generated during XP's
mkfs.ntfs that is causing some issues.  While watching output during fs
creation, I noticed the following WRITE_10s with a starting 4088 byte
scatterlist and a trailing 8 byte scatterlist:

megasas: writel mmio 40: 2b0b003
megasas: Found mapped frame 2 context 82b0b000 pa 2b0b000
megasas: Enqueue frame context 82b0b000 tail 493 busy 1
megasas: LD SCSI dev 2 lun 0 sdev 0xdc0230 xfer 16384
scsi-generic: Using cur_addr: 0x000000000ff6c008 cur_len: 0x0000000000000ff8
scsi-generic: Adding iovec for mem: 0x7f1783b96008 len: 0x0000000000000ff8
scsi-generic: Using cur_addr: 0x000000000fd6e000 cur_len: 0x0000000000001000
scsi-generic: Adding iovec for mem: 0x7f1783998000 len: 0x0000000000001000
scsi-generic: Using cur_addr: 0x000000000fe2f000 cur_len: 0x0000000000001000
scsi-generic: Adding iovec for mem: 0x7f1783a59000 len: 0x0000000000001000
scsi-generic: Using cur_addr: 0x000000000fdf0000 cur_len: 0x0000000000001000
scsi-generic: Adding iovec for mem: 0x7f1783a1a000 len: 0x0000000000001000
scsi-generic: Using cur_addr: 0x000000000fded000 cur_len: 0x0000000000000008
scsi-generic: Adding iovec for mem: 0x7f1783a17000 len: 0x0000000000000008
scsi-generic: execute IOV: iovec_count: 5, dxferp: 0xd92420, dxfer_len: 16384
scsi-generic: -----------------------> Issuing SG_IO CDB len 10: 0x2a 00 00 00 fa be 00 00 20 00 
scsi-generic: scsi_write_complete() ret = 0
scsi-generic: Command complete 0x0xd922c0 tag=0x82b0b000 status=0
megasas: LD SCSI req 0xd922c0 cmd 0xda92c0 lun 0xdc0230 finished with status 0 len 16384
megasas: Complete frame context 82b0b000 tail 493 busy 0 doorbell 0

Also, the final READ_10 that produces the 'could not create filesystem'
exception is for LBA 63 and XP looking for the first FS blocks after
GPT.

Could there be some breakage in megasas with a length < PAGE_SIZE for
the scatterlist..?    As lsi53c895a seems to work OK for this case, is
there something about the logic of parsing the incoming struct
scatterlists that is different between the two HBA drivers..?  AFAICT
both are using Gerd's common code in hw/scsi-bus.c, unless there is
something about megasas_map_sgl() that is causing issues with the
above..?

Best,

--nab

[Qemu-devel] Re: [QEMU-KVM]: Megasas + TCM_Loop + SG_IO into Windows XP guests

[Hannes Reinecke]

Nicholas A. Bellinger wrote:
> On Fri, 2010-05-14 at 02:42 -0700, Nicholas A. Bellinger wrote:
>> On Fri, 2010-05-14 at 09:22 +0200, Hannes Reinecke wrote:
>>> Nicholas A. Bellinger wrote:
>>>> Greetings Hannes and co,
>>>>
>> <SNIP>
>>> Let's see if I can find some time working on the megasas emulation.
>>> Maybe I find something.
>>> Last time I checked it was with a Windows7 build, but I didn't do
>>> any real tests there. Basically just checking if the system boots up :-)
>>>
>> Nothing fancy just yet.  This is involving a normal NTFS filesystem
>> format on a small TCM/FILEIO LUN using scsi-generic and a userspace
>> FILEIO with scsi-disk.
>>
>> This involves the XP guest waiting until the very last READ_10 once the
>> format has completed (eg: all WRITE and VERIFY CDBs complete with GOOD
>> status AFAICT) before announcing that mkfs.ntfs failed without any
>> helpful exception message (due to missing metadata of some sort I would
>> assume..?)
>>
>> So perhaps dumping QEMU and TCM_Loop SCSI payloads to determine if any
>> correct blocks from megasas_handle_io() are actually making it out to
>> KVM host is going to be my next option.  ;)
>>
> 
> Greetings Hannes,
> 
> So I spent some more time with XP guests this weekend, and I noticed two
> things immediately when using hw/lsi53c895a.c instead of hw/megasas.c
> with the same two TCM_Loop SAS LUNs via SG_IO from last week:
> 
> 1) With lsi53c895a, XP guests are able to boot successfully w/ out the
> synchronous SG_IO hack that is currently required to get past the first
> 36-byte INQUIRY for megasas + XP SP2
> 
> 2) With lsi53c895a, XP is able to successfully create and mount a NTFS
> filesystem, reboot, and read blocks appear to be functioning properly.
> FYI I have not run any 'write known pattern then read-back and compare
> blocks' data integrity tests from with in the XP guests just yet, but I
> am confident that TCM scatterlist -> se_mem_t mapping is working as
> expected on the KVM Host.
> 
> Futhermore, after formatting a 5 GB TCM/FILEIO LUN with lsi53c895a, and
> then rebooting with megasas with the same two configured TCM_Loop SG_IO
> devices, it appears to be able to mount and read blocks successfully.
> Attempting to write new blocks on the mounted filesystem also appears to
> work to some degree, but throughput slows down to a crawl during XP
> guest buffer cache flush, which is likely attributed to the use of my
> quick SYNC SG_IO hack.
> 
> So it appears that there are two seperate issues here, and AFAICT they
> both look to be XP and megasas specific.  For #2, it may be something
> about the format of the incoming scatterlists generated during XP's
> mkfs.ntfs that is causing some issues.  While watching output during fs
> creation, I noticed the following WRITE_10s with a starting 4088 byte
> scatterlist and a trailing 8 byte scatterlist:
> 
> megasas: writel mmio 40: 2b0b003
> megasas: Found mapped frame 2 context 82b0b000 pa 2b0b000
> megasas: Enqueue frame context 82b0b000 tail 493 busy 1
> megasas: LD SCSI dev 2 lun 0 sdev 0xdc0230 xfer 16384
> scsi-generic: Using cur_addr: 0x000000000ff6c008 cur_len: 0x0000000000000ff8
> scsi-generic: Adding iovec for mem: 0x7f1783b96008 len: 0x0000000000000ff8
> scsi-generic: Using cur_addr: 0x000000000fd6e000 cur_len: 0x0000000000001000
> scsi-generic: Adding iovec for mem: 0x7f1783998000 len: 0x0000000000001000
> scsi-generic: Using cur_addr: 0x000000000fe2f000 cur_len: 0x0000000000001000
> scsi-generic: Adding iovec for mem: 0x7f1783a59000 len: 0x0000000000001000
> scsi-generic: Using cur_addr: 0x000000000fdf0000 cur_len: 0x0000000000001000
> scsi-generic: Adding iovec for mem: 0x7f1783a1a000 len: 0x0000000000001000
> scsi-generic: Using cur_addr: 0x000000000fded000 cur_len: 0x0000000000000008
> scsi-generic: Adding iovec for mem: 0x7f1783a17000 len: 0x0000000000000008
> scsi-generic: execute IOV: iovec_count: 5, dxferp: 0xd92420, dxfer_len: 16384
> scsi-generic: -----------------------> Issuing SG_IO CDB len 10: 0x2a 00 00 00 fa be 00 00 20 00 
> scsi-generic: scsi_write_complete() ret = 0
> scsi-generic: Command complete 0x0xd922c0 tag=0x82b0b000 status=0
> megasas: LD SCSI req 0xd922c0 cmd 0xda92c0 lun 0xdc0230 finished with status 0 len 16384
> megasas: Complete frame context 82b0b000 tail 493 busy 0 doorbell 0
> 
> Also, the final READ_10 that produces the 'could not create filesystem'
> exception is for LBA 63 and XP looking for the first FS blocks after
> GPT.
> 
> Could there be some breakage in megasas with a length < PAGE_SIZE for
> the scatterlist..?    As lsi53c895a seems to work OK for this case, is
> there something about the logic of parsing the incoming struct
> scatterlists that is different between the two HBA drivers..?  AFAICT
> both are using Gerd's common code in hw/scsi-bus.c, unless there is
> something about megasas_map_sgl() that is causing issues with the
> above..?
> 

The usual disclaimer here: I'm less than happy with the current SCSI disk handling.
Currently we have the two options:
- Using 'scsi-disk', which will _emulate_ a SCSI disk internally, but allow to use
  asynchronous I/O using normal read/write syscalls
- Using 'scsi-generic', which will allow you to pass-through any SCSI device, but
  disallow asynchronous I/O and requires you to use the SG_IO interface.
The latter also implies that the host will mark _all_ I/O commands as 'block_pc',
so the code path within the kernel is quite different from those taken by I/Os
coming in via the 'scsi-disk' emulation.
Guess it's time to have a 'scsi-passthrough' device ...

Other than that: Think we have to investigate.
If you could send me a quite setup guide on how to configure TCM_Loop for an
existing device I'd give it a go ...

Thanks,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)

[Qemu-devel] Re: [QEMU-KVM]: Megasas + TCM_Loop + SG_IO into Windows XP guests

[Nicholas A. Bellinger]

On Tue, 2010-05-18 at 11:43 +0200, Hannes Reinecke wrote:
> Nicholas A. Bellinger wrote:
> > On Fri, 2010-05-14 at 02:42 -0700, Nicholas A. Bellinger wrote:
> >> On Fri, 2010-05-14 at 09:22 +0200, Hannes Reinecke wrote:
> >>> Nicholas A. Bellinger wrote:
> >>>> Greetings Hannes and co,
> >>>>
> >> <SNIP>
> >>> Let's see if I can find some time working on the megasas emulation.
> >>> Maybe I find something.
> >>> Last time I checked it was with a Windows7 build, but I didn't do
> >>> any real tests there. Basically just checking if the system boots up :-)
> >>>
> >> Nothing fancy just yet.  This is involving a normal NTFS filesystem
> >> format on a small TCM/FILEIO LUN using scsi-generic and a userspace
> >> FILEIO with scsi-disk.
> >>
> >> This involves the XP guest waiting until the very last READ_10 once the
> >> format has completed (eg: all WRITE and VERIFY CDBs complete with GOOD
> >> status AFAICT) before announcing that mkfs.ntfs failed without any
> >> helpful exception message (due to missing metadata of some sort I would
> >> assume..?)
> >>
> >> So perhaps dumping QEMU and TCM_Loop SCSI payloads to determine if any
> >> correct blocks from megasas_handle_io() are actually making it out to
> >> KVM host is going to be my next option.  ;)
> >>
> > 
> > Greetings Hannes,
> > 
> > So I spent some more time with XP guests this weekend, and I noticed two
> > things immediately when using hw/lsi53c895a.c instead of hw/megasas.c
> > with the same two TCM_Loop SAS LUNs via SG_IO from last week:
> > 
> > 1) With lsi53c895a, XP guests are able to boot successfully w/ out the
> > synchronous SG_IO hack that is currently required to get past the first
> > 36-byte INQUIRY for megasas + XP SP2
> > 
> > 2) With lsi53c895a, XP is able to successfully create and mount a NTFS
> > filesystem, reboot, and read blocks appear to be functioning properly.
> > FYI I have not run any 'write known pattern then read-back and compare
> > blocks' data integrity tests from with in the XP guests just yet, but I
> > am confident that TCM scatterlist -> se_mem_t mapping is working as
> > expected on the KVM Host.
> > 
> > Futhermore, after formatting a 5 GB TCM/FILEIO LUN with lsi53c895a, and
> > then rebooting with megasas with the same two configured TCM_Loop SG_IO
> > devices, it appears to be able to mount and read blocks successfully.
> > Attempting to write new blocks on the mounted filesystem also appears to
> > work to some degree, but throughput slows down to a crawl during XP
> > guest buffer cache flush, which is likely attributed to the use of my
> > quick SYNC SG_IO hack.
> > 
> > So it appears that there are two seperate issues here, and AFAICT they
> > both look to be XP and megasas specific.  For #2, it may be something
> > about the format of the incoming scatterlists generated during XP's
> > mkfs.ntfs that is causing some issues.  While watching output during fs
> > creation, I noticed the following WRITE_10s with a starting 4088 byte
> > scatterlist and a trailing 8 byte scatterlist:
> > 
> > megasas: writel mmio 40: 2b0b003
> > megasas: Found mapped frame 2 context 82b0b000 pa 2b0b000
> > megasas: Enqueue frame context 82b0b000 tail 493 busy 1
> > megasas: LD SCSI dev 2 lun 0 sdev 0xdc0230 xfer 16384
> > scsi-generic: Using cur_addr: 0x000000000ff6c008 cur_len: 0x0000000000000ff8
> > scsi-generic: Adding iovec for mem: 0x7f1783b96008 len: 0x0000000000000ff8
> > scsi-generic: Using cur_addr: 0x000000000fd6e000 cur_len: 0x0000000000001000
> > scsi-generic: Adding iovec for mem: 0x7f1783998000 len: 0x0000000000001000
> > scsi-generic: Using cur_addr: 0x000000000fe2f000 cur_len: 0x0000000000001000
> > scsi-generic: Adding iovec for mem: 0x7f1783a59000 len: 0x0000000000001000
> > scsi-generic: Using cur_addr: 0x000000000fdf0000 cur_len: 0x0000000000001000
> > scsi-generic: Adding iovec for mem: 0x7f1783a1a000 len: 0x0000000000001000
> > scsi-generic: Using cur_addr: 0x000000000fded000 cur_len: 0x0000000000000008
> > scsi-generic: Adding iovec for mem: 0x7f1783a17000 len: 0x0000000000000008
> > scsi-generic: execute IOV: iovec_count: 5, dxferp: 0xd92420, dxfer_len: 16384
> > scsi-generic: -----------------------> Issuing SG_IO CDB len 10: 0x2a 00 00 00 fa be 00 00 20 00 
> > scsi-generic: scsi_write_complete() ret = 0
> > scsi-generic: Command complete 0x0xd922c0 tag=0x82b0b000 status=0
> > megasas: LD SCSI req 0xd922c0 cmd 0xda92c0 lun 0xdc0230 finished with status 0 len 16384
> > megasas: Complete frame context 82b0b000 tail 493 busy 0 doorbell 0
> > 
> > Also, the final READ_10 that produces the 'could not create filesystem'
> > exception is for LBA 63 and XP looking for the first FS blocks after
> > GPT.
> > 
> > Could there be some breakage in megasas with a length < PAGE_SIZE for
> > the scatterlist..?    As lsi53c895a seems to work OK for this case, is
> > there something about the logic of parsing the incoming struct
> > scatterlists that is different between the two HBA drivers..?  AFAICT
> > both are using Gerd's common code in hw/scsi-bus.c, unless there is
> > something about megasas_map_sgl() that is causing issues with the
> > above..?
> > 
> 
> The usual disclaimer here: I'm less than happy with the current SCSI disk handling.
> Currently we have the two options:
> - Using 'scsi-disk', which will _emulate_ a SCSI disk internally, but allow to use
>   asynchronous I/O using normal read/write syscalls
> - Using 'scsi-generic', which will allow you to pass-through any SCSI device, but
>   disallow asynchronous I/O and requires you to use the SG_IO interface.

Well, this is only true so far for the SYNC SG_IO patch with KVM XP
guests.  The asynchronous I/O still works as expected for Linux KVM
guests for 10 Gb/sec sec throughput.

> The latter also implies that the host will mark _all_ I/O commands as 'block_pc',
> so the code path within the kernel is quite different from those taken by I/Os
> coming in via the 'scsi-disk' emulation.
> Guess it's time to have a 'scsi-passthrough' device ...

Currently with QEMU-KVM hw/scsi-generic.c and STGT usr/bs_sg.c we are
expecting driver/scsi/sg.c:sg_start_req() to the passed return
hp->iov_count..

> 
> Other than that: Think we have to investigate.
> If you could send me a quite setup guide on how to configure TCM_Loop for an
> existing device I'd give it a go ...
> 

Sure, the setup for a TCM/IBLOCK device with the TCM_Loop fabric module
is:

tcm_node --block <$HBA/$DEV> <$UDEV_PATH>

and then setup the TCM_Loop virtual SAS endpoint LUN=0 with TCM/LIO 4.0
with a nexus and LUN=0 with:

tcm_loop --createnexus 1
tcm_loop --addlun <$SAS_TARGET_PORT> 1 0 $HBA/$DEV

Best,

--nab

[Qemu-devel] Re: [QEMU-KVM]: Megasas + TCM_Loop + SG_IO into Windows XP guests

[Nicholas A. Bellinger]

On Tue, 2010-05-18 at 04:18 -0700, Nicholas A. Bellinger wrote:
> On Tue, 2010-05-18 at 11:43 +0200, Hannes Reinecke wrote:
> > Nicholas A. Bellinger wrote:
> > > On Fri, 2010-05-14 at 02:42 -0700, Nicholas A. Bellinger wrote:
> > > Greetings Hannes,
> > > 
> > > So I spent some more time with XP guests this weekend, and I noticed two
> > > things immediately when using hw/lsi53c895a.c instead of hw/megasas.c
> > > with the same two TCM_Loop SAS LUNs via SG_IO from last week:
> > > 
> > > 1) With lsi53c895a, XP guests are able to boot successfully w/ out the
> > > synchronous SG_IO hack that is currently required to get past the first
> > > 36-byte INQUIRY for megasas + XP SP2
> > > 
> > > 2) With lsi53c895a, XP is able to successfully create and mount a NTFS
> > > filesystem, reboot, and read blocks appear to be functioning properly.
> > > FYI I have not run any 'write known pattern then read-back and compare
> > > blocks' data integrity tests from with in the XP guests just yet, but I
> > > am confident that TCM scatterlist -> se_mem_t mapping is working as
> > > expected on the KVM Host.
> > > 
> > > Futhermore, after formatting a 5 GB TCM/FILEIO LUN with lsi53c895a, and
> > > then rebooting with megasas with the same two configured TCM_Loop SG_IO
> > > devices, it appears to be able to mount and read blocks successfully.
> > > Attempting to write new blocks on the mounted filesystem also appears to
> > > work to some degree, but throughput slows down to a crawl during XP
> > > guest buffer cache flush, which is likely attributed to the use of my
> > > quick SYNC SG_IO hack.
> > > 
> > > So it appears that there are two seperate issues here, and AFAICT they
> > > both look to be XP and megasas specific.  For #2, it may be something
> > > about the format of the incoming scatterlists generated during XP's
> > > mkfs.ntfs that is causing some issues.  While watching output during fs
> > > creation, I noticed the following WRITE_10s with a starting 4088 byte
> > > scatterlist and a trailing 8 byte scatterlist:
> > > 
> > > megasas: writel mmio 40: 2b0b003
> > > megasas: Found mapped frame 2 context 82b0b000 pa 2b0b000
> > > megasas: Enqueue frame context 82b0b000 tail 493 busy 1
> > > megasas: LD SCSI dev 2 lun 0 sdev 0xdc0230 xfer 16384
> > > scsi-generic: Using cur_addr: 0x000000000ff6c008 cur_len: 0x0000000000000ff8
> > > scsi-generic: Adding iovec for mem: 0x7f1783b96008 len: 0x0000000000000ff8
> > > scsi-generic: Using cur_addr: 0x000000000fd6e000 cur_len: 0x0000000000001000
> > > scsi-generic: Adding iovec for mem: 0x7f1783998000 len: 0x0000000000001000
> > > scsi-generic: Using cur_addr: 0x000000000fe2f000 cur_len: 0x0000000000001000
> > > scsi-generic: Adding iovec for mem: 0x7f1783a59000 len: 0x0000000000001000
> > > scsi-generic: Using cur_addr: 0x000000000fdf0000 cur_len: 0x0000000000001000
> > > scsi-generic: Adding iovec for mem: 0x7f1783a1a000 len: 0x0000000000001000
> > > scsi-generic: Using cur_addr: 0x000000000fded000 cur_len: 0x0000000000000008
> > > scsi-generic: Adding iovec for mem: 0x7f1783a17000 len: 0x0000000000000008
> > > scsi-generic: execute IOV: iovec_count: 5, dxferp: 0xd92420, dxfer_len: 16384
> > > scsi-generic: -----------------------> Issuing SG_IO CDB len 10: 0x2a 00 00 00 fa be 00 00 20 00 
> > > scsi-generic: scsi_write_complete() ret = 0
> > > scsi-generic: Command complete 0x0xd922c0 tag=0x82b0b000 status=0
> > > megasas: LD SCSI req 0xd922c0 cmd 0xda92c0 lun 0xdc0230 finished with status 0 len 16384
> > > megasas: Complete frame context 82b0b000 tail 493 busy 0 doorbell 0
> > > 
> > > Also, the final READ_10 that produces the 'could not create filesystem'
> > > exception is for LBA 63 and XP looking for the first FS blocks after
> > > GPT.
> > > 
> > > Could there be some breakage in megasas with a length < PAGE_SIZE for
> > > the scatterlist..?    As lsi53c895a seems to work OK for this case, is
> > > there something about the logic of parsing the incoming struct
> > > scatterlists that is different between the two HBA drivers..?  AFAICT
> > > both are using Gerd's common code in hw/scsi-bus.c, unless there is
> > > something about megasas_map_sgl() that is causing issues with the
> > > above..?
> > > 
> > 
> > The usual disclaimer here: I'm less than happy with the current SCSI disk handling.
> > Currently we have the two options:
> > - Using 'scsi-disk', which will _emulate_ a SCSI disk internally, but allow to use
> >   asynchronous I/O using normal read/write syscalls
> > - Using 'scsi-generic', which will allow you to pass-through any SCSI device, but
> >   disallow asynchronous I/O and requires you to use the SG_IO interface.
> 
> Well, this is only true so far for the SYNC SG_IO patch with KVM XP
> guests.  The asynchronous I/O still works as expected for Linux KVM
> guests for 10 Gb/sec sec throughput.
> 
> > The latter also implies that the host will mark _all_ I/O commands as 'block_pc',
> > so the code path within the kernel is quite different from those taken by I/Os
> > coming in via the 'scsi-disk' emulation.
> > Guess it's time to have a 'scsi-passthrough' device ...
> 
> Currently with QEMU-KVM hw/scsi-generic.c and STGT usr/bs_sg.c we are
> expecting driver/scsi/sg.c:sg_start_req() to the passed return
> hp->iov_count..
> 

Greetings Hannes and Gerd,

Just a quick update on this one..  After giving MEGASAS with shot with
Windows 7 x64, I am able to successfully format, mount and copy blocks
to a TCM/FILEIO LUN backstore and TCM_Loop SAS target port with SG_IO:

http://www.linux-iscsi.org/index.php/Image:TCM-KVM-Megasas-8708EM2-Windows7-x64.png

The 8708EM2 driver for the emulated LSI DFI PowerPC RAID Core was
automatically configured and did not require any extra setup once
megasas emulation was enabled from the QEMU CLI.  Seriously, a very
splended job Dr. Hannes.  8-)

Also it's worth mentioning that I am still running C: on QEMU IDE
emulation, as I could not quite figure out how to boot for a megasas LD
with option-rom.  What QEMU CLI ops where requried to make this work
again..?

Anyways, the issue remains in the megasas friendly qemu-kvm.git tree and
still appears to be specific to hw/megasas.c LDs for a v2.6.34 KVM
x86_64 host into 32-bit XP guests with both scsi-disk and scsi-generic
QEMU backstores.  I am thinking about having a look with the
i386-softmmu target and see if that makes any difference..  What do you
think..?

Best,

--nab

[Qemu-devel] Re: [QEMU-KVM]: Megasas + TCM_Loop + SG_IO into Windows XP guests

[Gerd Hoffmann]

   Hi,

> Also it's worth mentioning that I am still running C: on QEMU IDE
> emulation, as I could not quite figure out how to boot for a megasas LD
> with option-rom.  What QEMU CLI ops where requried to make this work
> again..?

-option-rom $file

or

-device megasas,romfile=$file

cheers,
   Gerd

[Qemu-devel] Re: [QEMU-KVM]: Megasas + TCM_Loop + SG_IO into Windows XP guests

[Alexander Graf]

Am 31.05.2010 um 11:52 schrieb Gerd Hoffmann <kraxel@redhat.com>:

>  Hi,
>
>> Also it's worth mentioning that I am still running C: on QEMU IDE
>> emulation, as I could not quite figure out how to boot for a  
>> megasas LD
>> with option-rom.  What QEMU CLI ops where requried to make this work
>> again..?
>
> -option-rom $file
>
> or
>
> -device megasas,romfile=$file

Or -drive ...,boot=on if you're using qemu-kvm.

Alex