can't access LUN >2TB with aic7xxx, Infortrend EonStor and 2.6.31-rc1

can't access LUN >2TB with aic7xxx, Infortrend EonStor and 2.6.31-rc1

[Tomasz Torcz]

  Hello,

  recently I've acquired Infortrend EonStor A16U-G2421 array
with 16x 500GB SATA drives inside. It is connected via:
04:00.0 SCSI storage controller: Adaptec AHA-2940U2/U2W
to host running Fedora 12 with following kernel:
2.6.31-0.33.rc1.git2.fc12.x86_64

  When I export LUN >2TB from array I got following errors:

scsi 5:0:0:1: Direct-Access     VW       VRU1610          347G PQ: 0 ANSI: 5
scsi target5:0:0: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)
scsi5:A:0:1: Tagged Queuing enabled.  Depth 4
sd 5:0:0:1: Attached scsi generic sg3 type 0
sd 5:0:0:1: [sdd] READ CAPACITY(16) failed
sd 5:0:0:1: [sdd] Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
sd 5:0:0:1: [sdd] Sense not available.
sd 5:0:0:1: [sdd] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB)
sd 5:0:0:1: [sdd] Write Protect is off
sd 5:0:0:1: [sdd] Mode Sense: 9b 00 00 08
sd 5:0:0:1: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 5:0:0:1: [sdd] READ CAPACITY(16) failed
sd 5:0:0:1: [sdd] Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
sd 5:0:0:1: [sdd] Sense not available.
 sdd: unknown partition table
sd 5:0:0:1: [sdd] READ CAPACITY(16) failed
sd 5:0:0:1: [sdd] Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
sd 5:0:0:1: [sdd] Sense not available.
sd 5:0:0:1: [sdd] Attached SCSI disk

  According to manufacturer, this array supports LUNs up to 64TB, 
although >2TB LUNs require enigmatic ,,support from OS side''.
I was under impression that aic7xxx supports them, but it seems
not true. What can I tweak on the Linux side to use 6TB volumes?

(please keep CCs to me on reply)
-- 
Tomasz Torcz                 "God, root, what's the difference?"
xmpp: zdzichubg@chrome.pl         "God is more forgiving."

Re: can't access LUN >2TB with aic7xxx, Infortrend EonStor and 2.6.31-rc1

[Matthew Wilcox]

On Mon, Jun 29, 2009 at 03:08:35PM +0200, Tomasz Torcz wrote:
>   recently I've acquired Infortrend EonStor A16U-G2421 array
> with 16x 500GB SATA drives inside. It is connected via:
> 04:00.0 SCSI storage controller: Adaptec AHA-2940U2/U2W
> to host running Fedora 12 with following kernel:
> 2.6.31-0.33.rc1.git2.fc12.x86_64
> 
>   When I export LUN >2TB from array I got following errors:
> 
> scsi 5:0:0:1: Direct-Access     VW       VRU1610          347G PQ: 0 ANSI: 5
> scsi target5:0:0: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)
> scsi5:A:0:1: Tagged Queuing enabled.  Depth 4
> sd 5:0:0:1: Attached scsi generic sg3 type 0
> sd 5:0:0:1: [sdd] READ CAPACITY(16) failed
> sd 5:0:0:1: [sdd] Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
> sd 5:0:0:1: [sdd] Sense not available.
> 
>   According to manufacturer, this array supports LUNs up to 64TB, 
> although >2TB LUNs require enigmatic ,,support from OS side''.
> I was under impression that aic7xxx supports them, but it seems
> not true. What can I tweak on the Linux side to use 6TB volumes?

SCSI commands come in different sizes.  Drivers are assumed to support
commands up to 12 bytes long, unless they report otherwise by setting
their max_cmd_len parameter.  Unfortunately, the command that reports
the device capacity for devices over 2TB is a 16 byte command, and the
aic7xxx driver doesn't claim it supports 16 byte commands, so we refuse
to even send it to the driver.

It's possible the aic7xxx hardware does support 16 byte commands.  If so,
this is an easy fix.  If it doesn't, you could try replacing the card
with one that does (for example, the LSI 8xx and 10xx cards all support
16 byte commands).

You could also go for a configuration fix where you reconfigure the array
with LUNs below 2TB and then use MD or DM to stitch them back together
into a single device.

Let's hope someone who understands the aic7xxx hardware can tell us
whether it supports 16 byte commands or not.

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: can't access LUN >2TB with aic7xxx, Infortrend EonStor and 2.6.31-rc1

[Drew]

> You could also go for a configuration fix where you reconfigure the array
> with LUNs below 2TB and then use MD or DM to stitch them back together
> into a single device.

Would you not also have to partition the drive using GPT partitions?
LVM aside, I thought any partition over 2TB had to be GPT.


-- 
Drew

"Nothing in life is to be feared. It is only to be understood."
--Marie Curie

Re: can't access LUN >2TB with aic7xxx, Infortrend EonStor and 2.6.31-rc1

[Matthew Wilcox]

On Mon, Jun 29, 2009 at 06:40:21AM -0700, Drew wrote:
> > You could also go for a configuration fix where you reconfigure the array
> > with LUNs below 2TB and then use MD or DM to stitch them back together
> > into a single device.
> 
> Would you not also have to partition the drive using GPT partitions?
> LVM aside, I thought any partition over 2TB had to be GPT.

If you choose to partition the device at all -- I often create filesystems
directly on the block device.  Regardless, you need the SCSI command
to be able to get to the device before you can start worrying about
partitioning schemes.

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: can't access LUN >2TB with aic7xxx, Infortrend EonStor and 2.6.31-rc1

[Drew]

>> Would you not also have to partition the drive using GPT partitions?
>> LVM aside, I thought any partition over 2TB had to be GPT.
>
> If you choose to partition the device at all -- I often create filesystems
> directly on the block device.  Regardless, you need the SCSI command
> to be able to get to the device before you can start worrying about
> partitioning schemes.

That's a given. I threw out the GPT issue because once the OP gets the
hardware seeing the drive/lun they will, if they weren't already aware
of it, run smack into the 2TB partition limit. Just trying to think a
couple of steps ahead.


-- 
Drew

"Nothing in life is to be feared. It is only to be understood."
--Marie Curie
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: can't access LUN >2TB with aic7xxx, Infortrend EonStor and 2.6.31-rc1

[Matthew Wilcox]

On Mon, Jun 29, 2009 at 07:34:31AM -0600, Matthew Wilcox wrote:
> SCSI commands come in different sizes.  Drivers are assumed to support
> commands up to 12 bytes long, unless they report otherwise by setting
> their max_cmd_len parameter.  Unfortunately, the command that reports
> the device capacity for devices over 2TB is a 16 byte command, and the
> aic7xxx driver doesn't claim it supports 16 byte commands, so we refuse
> to even send it to the driver.
> 
> It's possible the aic7xxx hardware does support 16 byte commands.  If so,
> this is an easy fix.  If it doesn't, you could try replacing the card
> with one that does (for example, the LSI 8xx and 10xx cards all support
> 16 byte commands).
> 
> You could also go for a configuration fix where you reconfigure the array
> with LUNs below 2TB and then use MD or DM to stitch them back together
> into a single device.
> 
> Let's hope someone who understands the aic7xxx hardware can tell us
> whether it supports 16 byte commands or not.

I should have read the driver more thoroughly before writing that email.

It seems the driver does support larger commands.  Here's the hint:

        hscb->cdb_len = cmd->cmd_len;
        if (hscb->cdb_len <= 12) {
                memcpy(hscb->shared_data.cdb, cmd->cmnd, hscb->cdb_len);
        } else {
                memcpy(hscb->cdb32, cmd->cmnd, hscb->cdb_len);
                scb->flags |= SCB_CDB32_PTR;
        }

The aic79xx is a little more tricky ... it seems it has different cases to
support 12, 16 and >16 byte CDBs, but it's not clear that the >16 byte CDB case is implemented.  Let's be conservative for now and bump it up to 16 bytes instead of 32.


diff --git a/drivers/scsi/aic7xxx/aic79xx_osm.c b/drivers/scsi/aic7xxx/aic79xx_osm.c
index 75b2331..3853f33 100644
--- a/drivers/scsi/aic7xxx/aic79xx_osm.c
+++ b/drivers/scsi/aic7xxx/aic79xx_osm.c
@@ -1241,6 +1241,7 @@ ahd_linux_register_host(struct ahd_softc *ahd, struct scsi_host_template *templa
 	host->max_lun = AHD_NUM_LUNS;
 	host->max_channel = 0;
 	host->sg_tablesize = AHD_NSEG;
+	host->max_cmd_len = 16;
 	ahd_lock(ahd, &s);
 	ahd_set_unit(ahd, ahd_linux_unit++);
 	ahd_unlock(ahd, &s);
diff --git a/drivers/scsi/aic7xxx/aic7xxx_osm.c b/drivers/scsi/aic7xxx/aic7xxx_osm.c
index fd2b978..2d069de 100644
--- a/drivers/scsi/aic7xxx/aic7xxx_osm.c
+++ b/drivers/scsi/aic7xxx/aic7xxx_osm.c
@@ -1115,6 +1115,7 @@ ahc_linux_register_host(struct ahc_softc *ahc, struct scsi_host_template *templa
 	host->max_lun = AHC_NUM_LUNS;
 	host->max_channel = (ahc->features & AHC_TWIN) ? 1 : 0;
 	host->sg_tablesize = AHC_NSEG;
+	host->max_cmd_len = 32;
 	ahc_lock(ahc, &s);
 	ahc_set_unit(ahc, ahc_linux_unit++);
 	ahc_unlock(ahc, &s);

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: can't access LUN >2TB with aic7xxx, Infortrend EonStor and 2.6.31-rc1

[Tomasz Torcz]

On Mon, Jun 29, 2009 at 07:46:03AM -0600, Matthew Wilcox wrote:
> > 
> > It's possible the aic7xxx hardware does support 16 byte commands.  If so,
> > this is an easy fix.  If it doesn't, you could try replacing the card
> > with one that does (for example, the LSI 8xx and 10xx cards all support
> > 16 byte commands).
> 
> The aic79xx is a little more tricky ... it seems it has different cases to
> support 12, 16 and >16 byte CDBs, but it's not clear that the >16 byte CDB case is implemented.  Let's be conservative for now and bump it up to 16 bytes instead of 32.
> 
> 
> diff --git a/drivers/scsi/aic7xxx/aic79xx_osm.c b/drivers/scsi/aic7xxx/aic79xx_osm.c
> index 75b2331..3853f33 100644
> --- a/drivers/scsi/aic7xxx/aic79xx_osm.c
> +++ b/drivers/scsi/aic7xxx/aic79xx_osm.c


  Thanks for the patch, I will rebuild driver and test tomorrow.


-- 
Tomasz Torcz                "Funeral in the morning, IDE hacking
xmpp: zdzichubg@chrome.pl    in the afternoon and evening." - Alan Cox

Re: can't access LUN >2TB with aic7xxx, Infortrend EonStor and 2.6.31-rc1

[Tomasz Torcz]

On Mon, Jun 29, 2009 at 07:46:03AM -0600, Matthew Wilcox wrote:
> > whether it supports 16 byte commands or not.
> 
> I should have read the driver more thoroughly before writing that email.
> 
> It seems the driver does support larger commands.  Here's the hint:
> 
>         hscb->cdb_len = cmd->cmd_len;
>         if (hscb->cdb_len <= 12) {
>                 memcpy(hscb->shared_data.cdb, cmd->cmnd, hscb->cdb_len);
>         } else {
>                 memcpy(hscb->cdb32, cmd->cmnd, hscb->cdb_len);
>                 scb->flags |= SCB_CDB32_PTR;
>         }
> 
> The aic79xx is a little more tricky ... it seems it has different cases to
> support 12, 16 and >16 byte CDBs, but it's not clear that the >16 byte CDB case is implemented.  Let's be conservative for now and bump it up to 16 bytes instead of 32.
> 
> 
> diff --git a/drivers/scsi/aic7xxx/aic79xx_osm.c b/drivers/scsi/aic7xxx/aic79xx_osm.c
> index 75b2331..3853f33 100644
> --- a/drivers/scsi/aic7xxx/aic79xx_osm.c
> +++ b/drivers/scsi/aic7xxx/aic79xx_osm.c
> @@ -1241,6 +1241,7 @@ ahd_linux_register_host(struct ahd_softc *ahd, struct scsi_host_template *templa
>  	host->max_lun = AHD_NUM_LUNS;
>  	host->max_channel = 0;
>  	host->sg_tablesize = AHD_NSEG;
> +	host->max_cmd_len = 16;
>  	ahd_lock(ahd, &s);
>  	ahd_set_unit(ahd, ahd_linux_unit++);
>  	ahd_unlock(ahd, &s);
> diff --git a/drivers/scsi/aic7xxx/aic7xxx_osm.c b/drivers/scsi/aic7xxx/aic7xxx_osm.c
> index fd2b978..2d069de 100644
> --- a/drivers/scsi/aic7xxx/aic7xxx_osm.c
> +++ b/drivers/scsi/aic7xxx/aic7xxx_osm.c
> @@ -1115,6 +1115,7 @@ ahc_linux_register_host(struct ahc_softc *ahc, struct scsi_host_template *templa
>  	host->max_lun = AHC_NUM_LUNS;
>  	host->max_channel = (ahc->features & AHC_TWIN) ? 1 : 0;
>  	host->sg_tablesize = AHC_NSEG;
> +	host->max_cmd_len = 32;
>  	ahc_lock(ahc, &s);
>  	ahc_set_unit(ahc, ahc_linux_unit++);
>  	ahc_unlock(ahc, &s);


  I did a rebuild and driver is not happy with above changes. Take a look
(I have one RAID5 LUN exported which consist 8x500GB, so about 3.5 TB big):

csi5 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 7.0
        <Adaptec 2940 Ultra2 SCSI adapter>
        aic7890/91: Ultra2 Wide Channel A, SCSI Id=7, 32/253 SCBs

scsi 5:0:0:0: Direct-Access     VW       VRU1610          347G PQ: 0 ANSI: 5
scsi5:A:0:0: Tagged Queuing enabled.  Depth 4
scsi target5:0:0: Beginning Domain Validation
scsi target5:0:0: wide asynchronous
scsi target5:0:0: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)
scsi target5:0:0: Domain Validation skipping write tests
scsi target5:0:0: Ending Domain Validation
scsi: waiting for bus probes to complete ...
sd 5:0:0:0: Attached scsi generic sg2 type 0
(scsi5:A:0:0): No or incomplete CDB sent to device.
(scsi5:A:0:0): Protocol violation in Message-in phase.  Attempting to abort.
(scsi5:A:0:0): Abort Tag Message Sent
(scsi5:A:0:0): SCB 2 - Abort Tag Completed.
(scsi5:A:0:0): No or incomplete CDB sent to device.
(scsi5:A:0:0): Protocol violation in Message-in phase.  Attempting to abort.
(scsi5:A:0:0): Abort Tag Message Sent
(scsi5:A:0:0): SCB 3 - Abort Tag Completed.
(scsi5:A:0:0): No or incomplete CDB sent to device.
(scsi5:A:0:0): Protocol violation in Message-in phase.  Attempting to abort.
(scsi5:A:0:0): Abort Tag Message Sent

[ sequence of SCB 2 and SCB 3 aborts repeats 7 more times ]

(scsi5:A:0:0): SCB 2 - Abort Tag Completed.
(scsi5:A:0:0): No or incomplete CDB sent to device.
(scsi5:A:0:0): Protocol violation in Message-in phase.  Attempting to abort.
(scsi5:A:0:0): Abort Tag Message Sent
(scsi5:A:0:0): SCB 3 - Abort Tag Completed.
sd 5:0:0:0: [sdc] READ CAPACITY(16) failed
sd 5:0:0:0: [sdc] Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
sd 5:0:0:0: [sdc] Sense not available.
sd 5:0:0:0: [sdc] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB)
sd 5:0:0:0: [sdc] Write Protect is off
sd 5:0:0:0: [sdc] Mode Sense: 9b 00 00 08
sd 5:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
(scsi5:A:0:0): No or incomplete CDB sent to device.
(scsi5:A:0:0): Protocol violation in Message-in phase.  Attempting to abort.
(scsi5:A:0:0): Abort Tag Message Sent
(scsi5:A:0:0): SCB 3 - Abort Tag Completed.
(scsi5:A:0:0): No or incomplete CDB sent to device.
(scsi5:A:0:0): Protocol violation in Message-in phase.  Attempting to abort.
(scsi5:A:0:0): Abort Tag Message Sent
(scsi5:A:0:0): SCB 2 - Abort Tag Completed.
(scsi5:A:0:0): No or incomplete CDB sent to device.

[ another 7 repeats snipped ]

(scsi5:A:0:0): Protocol violation in Message-in phase.  Attempting to abort.
(scsi5:A:0:0): Abort Tag Message Sent
(scsi5:A:0:0): SCB 3 - Abort Tag Completed.
(scsi5:A:0:0): No or incomplete CDB sent to device.
(scsi5:A:0:0): Protocol violation in Message-in phase.  Attempting to abort.
(scsi5:A:0:0): Abort Tag Message Sent
(scsi5:A:0:0): SCB 2 - Abort Tag Completed.
sd 5:0:0:0: [sdc] READ CAPACITY(16) failed
sd 5:0:0:0: [sdc] Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
sd 5:0:0:0: [sdc] Sense not available.
 sdc: unknown partition table
(scsi5:A:0:0): No or incomplete CDB sent to device.
(scsi5:A:0:0): Protocol violation in Message-in phase.  Attempting to abort.
(scsi5:A:0:0): Abort Tag Message Sent
(scsi5:A:0:0): SCB 2 - Abort Tag Completed.
(scsi5:A:0:0): No or incomplete CDB sent to device.
(scsi5:A:0:0): Protocol violation in Message-in phase.  Attempting to abort.
(scsi5:A:0:0): Abort Tag Message Sent
(scsi5:A:0:0): SCB 3 - Abort Tag Completed.
(scsi5:A:0:0): No or incomplete CDB sent to device.
(scsi5:A:0:0): Protocol violation in Message-in phase.  Attempting to abort.
(scsi5:A:0:0): Abort Tag Message Sent
(scsi5:A:0:0): SCB 2 - Abort Tag Completed.
(scsi5:A:0:0): No or incomplete CDB sent to device.
(scsi5:A:0:0): Protocol violation in Message-in phase.  Attempting to abort.
(scsi5:A:0:0): Abort Tag Message Sent

[ note SCB 1 (not 2 or 3) below ]

(scsi5:A:0:0): SCB 1 - Abort Tag Completed.
(scsi5:A:0:0): No or incomplete CDB sent to device.
(scsi5:A:0:0): Protocol violation in Message-in phase.  Attempting to abort.
(scsi5:A:0:0): Abort Tag Message Sent

[ snipped every group of 3 messages below ]

(scsi5:A:0:0): SCB 3 - Abort Tag Completed.

(scsi5:A:0:0): SCB 2 - Abort Tag Completed.

(scsi5:A:0:0): SCB 2 - Abort Tag Completed.

(scsi5:A:0:0): SCB 3 - Abort Tag Completed.

(scsi5:A:0:0): SCB 2 - Abort Tag Completed.

(scsi5:A:0:0): SCB 3 - Abort Tag Completed.

(scsi5:A:0:0): SCB 2 - Abort Tag Completed.

(scsi5:A:0:0): SCB 3 - Abort Tag Completed.

(scsi5:A:0:0): SCB 2 - Abort Tag Completed.

(scsi5:A:0:0): SCB 1 - Abort Tag Completed.

(scsi5:A:0:0): SCB 3 - Abort Tag Completed.

(scsi5:A:0:0): SCB 2 - Abort Tag Completed.

(scsi5:A:0:0): SCB 1 - Abort Tag Completed.

(scsi5:A:0:0): SCB 3 - Abort Tag Completed.

sd 5:0:0:0: [sdc] READ CAPACITY(16) failed
sd 5:0:0:0: [sdc] Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
sd 5:0:0:0: [sdc] Sense not available.
sd 5:0:0:0: [sdc] Attached SCSI disk


And whole LUN is not visible:
# grep sdc /proc/partitions 
   8       32 2147483648 sdc


-- 
Tomasz Torcz                                                       72->|   80->|
xmpp: zdzichubg@chrome.pl                                          72->|   80->|

Re: can't access LUN >2TB with aic7xxx, Infortrend EonStor and 2.6.31-rc1

[James Bottomley]

On Tue, 2009-06-30 at 11:40 +0200, Tomasz Torcz wrote:
> On Mon, Jun 29, 2009 at 07:46:03AM -0600, Matthew Wilcox wrote:
> > > whether it supports 16 byte commands or not.
> > 
> > I should have read the driver more thoroughly before writing that email.
> > 
> > It seems the driver does support larger commands.  Here's the hint:
> > 
> >         hscb->cdb_len = cmd->cmd_len;
> >         if (hscb->cdb_len <= 12) {
> >                 memcpy(hscb->shared_data.cdb, cmd->cmnd, hscb->cdb_len);
> >         } else {
> >                 memcpy(hscb->cdb32, cmd->cmnd, hscb->cdb_len);
> >                 scb->flags |= SCB_CDB32_PTR;
> >         }
> > 
> > The aic79xx is a little more tricky ... it seems it has different cases to
> > support 12, 16 and >16 byte CDBs, but it's not clear that the >16 byte CDB case is implemented.  Let's be conservative for now and bump it up to 16 bytes instead of 32.
> > 
> > 
> > diff --git a/drivers/scsi/aic7xxx/aic79xx_osm.c b/drivers/scsi/aic7xxx/aic79xx_osm.c
> > index 75b2331..3853f33 100644
> > --- a/drivers/scsi/aic7xxx/aic79xx_osm.c
> > +++ b/drivers/scsi/aic7xxx/aic79xx_osm.c
> > @@ -1241,6 +1241,7 @@ ahd_linux_register_host(struct ahd_softc *ahd, struct scsi_host_template *templa
> >  	host->max_lun = AHD_NUM_LUNS;
> >  	host->max_channel = 0;
> >  	host->sg_tablesize = AHD_NSEG;
> > +	host->max_cmd_len = 16;
> >  	ahd_lock(ahd, &s);
> >  	ahd_set_unit(ahd, ahd_linux_unit++);
> >  	ahd_unlock(ahd, &s);
> > diff --git a/drivers/scsi/aic7xxx/aic7xxx_osm.c b/drivers/scsi/aic7xxx/aic7xxx_osm.c
> > index fd2b978..2d069de 100644
> > --- a/drivers/scsi/aic7xxx/aic7xxx_osm.c
> > +++ b/drivers/scsi/aic7xxx/aic7xxx_osm.c
> > @@ -1115,6 +1115,7 @@ ahc_linux_register_host(struct ahc_softc *ahc, struct scsi_host_template *templa
> >  	host->max_lun = AHC_NUM_LUNS;
> >  	host->max_channel = (ahc->features & AHC_TWIN) ? 1 : 0;
> >  	host->sg_tablesize = AHC_NSEG;
> > +	host->max_cmd_len = 32;
> >  	ahc_lock(ahc, &s);
> >  	ahc_set_unit(ahc, ahc_linux_unit++);
> >  	ahc_unlock(ahc, &s);
> 
> 
>   I did a rebuild and driver is not happy with above changes. Take a look
> (I have one RAID5 LUN exported which consist 8x500GB, so about 3.5 TB big):
> 
> csi5 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 7.0
>         <Adaptec 2940 Ultra2 SCSI adapter>
>         aic7890/91: Ultra2 Wide Channel A, SCSI Id=7, 32/253 SCBs
> 
> scsi 5:0:0:0: Direct-Access     VW       VRU1610          347G PQ: 0 ANSI: 5
> scsi5:A:0:0: Tagged Queuing enabled.  Depth 4
> scsi target5:0:0: Beginning Domain Validation
> scsi target5:0:0: wide asynchronous
> scsi target5:0:0: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)
> scsi target5:0:0: Domain Validation skipping write tests
> scsi target5:0:0: Ending Domain Validation
> scsi: waiting for bus probes to complete ...
> sd 5:0:0:0: Attached scsi generic sg2 type 0
> (scsi5:A:0:0): No or incomplete CDB sent to device.
> (scsi5:A:0:0): Protocol violation in Message-in phase.  Attempting to abort.
> (scsi5:A:0:0): Abort Tag Message Sent
> (scsi5:A:0:0): SCB 2 - Abort Tag Completed.
> (scsi5:A:0:0): No or incomplete CDB sent to device.
> (scsi5:A:0:0): Protocol violation in Message-in phase.  Attempting to abort.
> (scsi5:A:0:0): Abort Tag Message Sent
> (scsi5:A:0:0): SCB 3 - Abort Tag Completed.
> (scsi5:A:0:0): No or incomplete CDB sent to device.
> (scsi5:A:0:0): Protocol violation in Message-in phase.  Attempting to abort.
> (scsi5:A:0:0): Abort Tag Message Sent

That's unfortunate ... it basically means that the untested >12 byte CDB
code path isn't working.

The slight problem is debugging this:  I don't have storage anywhere
approaching 2TB.  However, I might be able to rig the kernel to accept
16byte CDBs for <2TB ... however, I suspect I don't have any drives
which will accept them either.

James

Re: can't access LUN >2TB with aic7xxx, Infortrend EonStor and 2.6.31-rc1

[James Bottomley]

On Tue, 2009-06-30 at 11:40 +0200, Tomasz Torcz wrote:
>   I did a rebuild and driver is not happy with above changes. Take a look
> (I have one RAID5 LUN exported which consist 8x500GB, so about 3.5 TB big):

Actually, looks like a simple thinko in the aic7xxx core code ... it
should be indexing of the saved tag not the original tag.  Can you try
this as a fresh patch?

James

---

diff --git a/drivers/scsi/aic7xxx/aic7xxx_core.c b/drivers/scsi/aic7xxx/aic7xxx_core.c
index e6f2bb7..56e431e 100644
--- a/drivers/scsi/aic7xxx/aic7xxx_core.c
+++ b/drivers/scsi/aic7xxx/aic7xxx_core.c
@@ -593,7 +593,7 @@ ahc_swap_with_next_hscb(struct ahc_softc *ahc, struct scb *scb)
 	memcpy(q_hscb, scb->hscb, sizeof(*scb->hscb));
 	if ((scb->flags & SCB_CDB32_PTR) != 0) {
 		q_hscb->shared_data.cdb_ptr =
-		    ahc_htole32(ahc_hscb_busaddr(ahc, q_hscb->tag)
+		    ahc_htole32(ahc_hscb_busaddr(ahc, saved_tag)
 			      + offsetof(struct hardware_scb, cdb32));
 	}
 	q_hscb->tag = saved_tag;
diff --git a/drivers/scsi/aic7xxx/aic7xxx_osm.c b/drivers/scsi/aic7xxx/aic7xxx_osm.c
index fd2b978..2d069de 100644
--- a/drivers/scsi/aic7xxx/aic7xxx_osm.c
+++ b/drivers/scsi/aic7xxx/aic7xxx_osm.c
@@ -1115,6 +1115,7 @@ ahc_linux_register_host(struct ahc_softc *ahc, struct scsi_host_template *templa
 	host->max_lun = AHC_NUM_LUNS;
 	host->max_channel = (ahc->features & AHC_TWIN) ? 1 : 0;
 	host->sg_tablesize = AHC_NSEG;
+	host->max_cmd_len = 32;
 	ahc_lock(ahc, &s);
 	ahc_set_unit(ahc, ahc_linux_unit++);
 	ahc_unlock(ahc, &s);

Re: can't access LUN >2TB with aic7xxx, Infortrend EonStor and 2.6.31-rc1

[Tomasz Torcz]

On Tue, Jun 30, 2009 at 03:48:19PM +0000, James Bottomley wrote:
> On Tue, 2009-06-30 at 11:40 +0200, Tomasz Torcz wrote:
> >   I did a rebuild and driver is not happy with above changes. Take a look
> > (I have one RAID5 LUN exported which consist 8x500GB, so about 3.5 TB big):
> 
> Actually, looks like a simple thinko in the aic7xxx core code ... it
> should be indexing of the saved tag not the original tag.  Can you try
> this as a fresh patch?
> 
> James
> 
> ---
> 
> diff --git a/drivers/scsi/aic7xxx/aic7xxx_core.c b/drivers/scsi/aic7xxx/aic7xxx_core.c
> index e6f2bb7..56e431e 100644
> --- a/drivers/scsi/aic7xxx/aic7xxx_core.c
> +++ b/drivers/scsi/aic7xxx/aic7xxx_core.c
> @@ -593,7 +593,7 @@ ahc_swap_with_next_hscb(struct ahc_softc *ahc, struct scb *scb)
>  	memcpy(q_hscb, scb->hscb, sizeof(*scb->hscb));
>  	if ((scb->flags & SCB_CDB32_PTR) != 0) {
>  		q_hscb->shared_data.cdb_ptr =
> -		    ahc_htole32(ahc_hscb_busaddr(ahc, q_hscb->tag)
> +		    ahc_htole32(ahc_hscb_busaddr(ahc, saved_tag)
>  			      + offsetof(struct hardware_scb, cdb32));
>  	}
>  	q_hscb->tag = saved_tag;
> diff --git a/drivers/scsi/aic7xxx/aic7xxx_osm.c b/drivers/scsi/aic7xxx/aic7xxx_osm.c
> index fd2b978..2d069de 100644
> --- a/drivers/scsi/aic7xxx/aic7xxx_osm.c
> +++ b/drivers/scsi/aic7xxx/aic7xxx_osm.c
> @@ -1115,6 +1115,7 @@ ahc_linux_register_host(struct ahc_softc *ahc, struct scsi_host_template *templa
>  	host->max_lun = AHC_NUM_LUNS;
>  	host->max_channel = (ahc->features & AHC_TWIN) ? 1 : 0;
>  	host->sg_tablesize = AHC_NSEG;
> +	host->max_cmd_len = 32;
>  	ahc_lock(ahc, &s);
>  	ahc_set_unit(ahc, ahc_linux_unit++);
>  	ahc_unlock(ahc, &s);


  Still not good:
sd 6:0:0:0: Attached scsi generic sg2 type 0
sd 6:0:0:1: Attached scsi generic sg3 type 0
sd 6:0:0:0: [sdc] Spinning up disk...
sd 6:0:0:1: [sdd] Spinning up disk........ready
sd 6:0:0:0: [sdc] Very big device. Trying to use READ CAPACITY(16).
(scsi6:A:0:0): No or incomplete CDB sent to device.
(scsi6:A:0:0): Protocol violation in Message-in phase.  Attempting to abort.
(scsi6:A:0:0): Abort Tag Message Sent
(scsi6:A:0:0): SCB 2 - Abort Tag Completed.
(scsi6:A:0:0): No or incomplete CDB sent to device.
(scsi6:A:0:0): Protocol violation in Message-in phase.  Attempting to abort.
(scsi6:A:0:0): Abort Tag Message Sent
(scsi6:A:0:0): SCB 3 - Abort Tag Completed.


 ... and so on.
 Maybe this HBA is too old? Newest BIOS for it is from year 2000. I've checked
another controller, sym53c8xx driven, it shows 3,5TB briefly before reverting to 2TB,
so array itself is reporting OK.

-- 
Tomasz Torcz              ,,If you try to upissue this patchset I shall be seeking
xmpp: zdzichubg@chrome.pl   an IP-routable hand grenade.'' -- Andrew Morton (LKML)

Re: can't access LUN >2TB with aic7xxx, Infortrend EonStor and 2.6.31-rc1

[James Bottomley]

On Wed, 2009-07-01 at 08:10 +0200, Tomasz Torcz wrote:
>   Still not good:
> sd 6:0:0:0: Attached scsi generic sg2 type 0
> sd 6:0:0:1: Attached scsi generic sg3 type 0
> sd 6:0:0:0: [sdc] Spinning up disk...
> sd 6:0:0:1: [sdd] Spinning up disk........ready
> sd 6:0:0:0: [sdc] Very big device. Trying to use READ CAPACITY(16).
> (scsi6:A:0:0): No or incomplete CDB sent to device.
> (scsi6:A:0:0): Protocol violation in Message-in phase.  Attempting to abort.
> (scsi6:A:0:0): Abort Tag Message Sent
> (scsi6:A:0:0): SCB 2 - Abort Tag Completed.
> (scsi6:A:0:0): No or incomplete CDB sent to device.
> (scsi6:A:0:0): Protocol violation in Message-in phase.  Attempting to abort.
> (scsi6:A:0:0): Abort Tag Message Sent
> (scsi6:A:0:0): SCB 3 - Abort Tag Completed.
> 
> 
>  ... and so on.
>  Maybe this HBA is too old? Newest BIOS for it is from year 2000. I've checked
> another controller, sym53c8xx driven, it shows 3,5TB briefly before reverting to 2TB,
> so array itself is reporting OK.

OK ... I'll look again.  The only doc I have for this is a 1998 HIM
guide for the u160 chip, so it's possible earlier versions didn't
support the >12 CDBs but I'd expect a field warning in that case and I
don't see one.

Chances are there's simply another issue in the driver that I haven't
seen yet.

James

Re: can't access LUN >2TB with aic7xxx, Infortrend EonStor and 2.6.31-rc1

[James Bottomley]

On Mon, 2009-06-29 at 15:08 +0200, Tomasz Torcz wrote:
> Hello,
> 
>   recently I've acquired Infortrend EonStor A16U-G2421 array
> with 16x 500GB SATA drives inside. It is connected via:
> 04:00.0 SCSI storage controller: Adaptec AHA-2940U2/U2W
> to host running Fedora 12 with following kernel:
> 2.6.31-0.33.rc1.git2.fc12.x86_64
> 
>   When I export LUN >2TB from array I got following errors:
> 
> scsi 5:0:0:1: Direct-Access     VW       VRU1610          347G PQ: 0 ANSI: 5
> scsi target5:0:0: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)
> scsi5:A:0:1: Tagged Queuing enabled.  Depth 4
> sd 5:0:0:1: Attached scsi generic sg3 type 0
> sd 5:0:0:1: [sdd] READ CAPACITY(16) failed
> sd 5:0:0:1: [sdd] Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
> sd 5:0:0:1: [sdd] Sense not available.
> sd 5:0:0:1: [sdd] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB)
> sd 5:0:0:1: [sdd] Write Protect is off
> sd 5:0:0:1: [sdd] Mode Sense: 9b 00 00 08
> sd 5:0:0:1: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
> sd 5:0:0:1: [sdd] READ CAPACITY(16) failed
> sd 5:0:0:1: [sdd] Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
> sd 5:0:0:1: [sdd] Sense not available.
>  sdd: unknown partition table
> sd 5:0:0:1: [sdd] READ CAPACITY(16) failed
> sd 5:0:0:1: [sdd] Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
> sd 5:0:0:1: [sdd] Sense not available.
> sd 5:0:0:1: [sdd] Attached SCSI disk
> 
>   According to manufacturer, this array supports LUNs up to 64TB, 
> although >2TB LUNs require enigmatic ,,support from OS side''.
> I was under impression that aic7xxx supports them, but it seems
> not true. What can I tweak on the Linux side to use 6TB volumes?

Both aic7xxx and aic79xx should support up to 32 byte CDBs, but no-one's
actually bothered to inform the kernel of this.  The attached patch will
do this.

There's a problem for you in that you have an aic7xxx and they use a
different SCB for >12 byte cdbs, so they have a separate code path,
which has obviously never been tested ... so don't try this with data
you care about.

James

----

diff --git a/drivers/scsi/aic7xxx/aic79xx_osm.c b/drivers/scsi/aic7xxx/aic79xx_osm.c
index 75b2331..e36d053 100644
--- a/drivers/scsi/aic7xxx/aic79xx_osm.c
+++ b/drivers/scsi/aic7xxx/aic79xx_osm.c
@@ -1241,6 +1241,7 @@ ahd_linux_register_host(struct ahd_softc *ahd, struct scsi_host_template *templa
 	host->max_lun = AHD_NUM_LUNS;
 	host->max_channel = 0;
 	host->sg_tablesize = AHD_NSEG;
+	host->max_cmd_len = 32;
 	ahd_lock(ahd, &s);
 	ahd_set_unit(ahd, ahd_linux_unit++);
 	ahd_unlock(ahd, &s);
diff --git a/drivers/scsi/aic7xxx/aic7xxx_osm.c b/drivers/scsi/aic7xxx/aic7xxx_osm.c
index fd2b978..2d069de 100644
--- a/drivers/scsi/aic7xxx/aic7xxx_osm.c
+++ b/drivers/scsi/aic7xxx/aic7xxx_osm.c
@@ -1115,6 +1115,7 @@ ahc_linux_register_host(struct ahc_softc *ahc, struct scsi_host_template *templa
 	host->max_lun = AHC_NUM_LUNS;
 	host->max_channel = (ahc->features & AHC_TWIN) ? 1 : 0;
 	host->sg_tablesize = AHC_NSEG;
+	host->max_cmd_len = 32;
 	ahc_lock(ahc, &s);
 	ahc_set_unit(ahc, ahc_linux_unit++);
 	ahc_unlock(ahc, &s);

Re: can't access LUN >2TB with aic7xxx, Infortrend EonStor and 2.6.31-rc1

[Matthew Wilcox]

On Mon, Jun 29, 2009 at 09:33:03AM -0500, James Bottomley wrote:
> +++ b/drivers/scsi/aic7xxx/aic79xx_osm.c
> @@ -1241,6 +1241,7 @@ ahd_linux_register_host(struct ahd_softc *ahd, struct scsi_host_template *templa
>  	host->max_lun = AHD_NUM_LUNS;
>  	host->max_channel = 0;
>  	host->sg_tablesize = AHD_NSEG;
> +	host->max_cmd_len = 32;

James and I are discussing this off-list, but 32 is clearly not correct
for aic79xx as the tag number will overwrite the command (or possibly
vice versa).  We're just picking over the various options at the moment.

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."