mpt2sas logged messages

mpt2sas logged messages

[Ric Wheeler]

We have a new toy to test very large & slow storage with built up from 5 
SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives and 
16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA MUX 
adapter daughter card in the disk sled).

The basic idea is to build a cheap & slow test bed for file & storage 
system scalability. Collectively, we have about 120TB (raw) of capacity 
to play with in one server.

As we work through various issues, a couple of oddities popped out.

The first is that udev grumbles during boot about "file name too long" 
like the following:

Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file 
'/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1': 
File name too long

The second is that the mpt2sas driver spews various messages like:

Aug 17 06:55:17 megadeth kernel: mpt2sas0: log_info(0x31120102): 
originator(PL), code(0x12), sub_code(0x0102)

Any insight into whether these are issues worth pursuing is appreciated, 
thanks!

Ric

Re: mpt2sas logged messages

[Greg KH]

On Tue, Aug 18, 2009 at 09:18:30AM -0400, Ric Wheeler wrote:
> 
> We have a new toy to test very large & slow storage with built up from 5 
> SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives and 
> 16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA MUX 
> adapter daughter card in the disk sled).
> 
> The basic idea is to build a cheap & slow test bed for file & storage 
> system scalability. Collectively, we have about 120TB (raw) of capacity 
> to play with in one server.
> 
> As we work through various issues, a couple of oddities popped out.
> 
> The first is that udev grumbles during boot about "file name too long" 
> like the following:
> 
> Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file 
> '/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1': 
> File name too long

Odd, what is the sysfs tree for this device?  You have expanders
attached to ports attached to expanders?  How deep can you go?

thanks,

greg k-h

Re: mpt2sas logged messages

[James Bottomley]

On Tue, 2009-08-18 at 06:25 -0700, Greg KH wrote: 
> On Tue, Aug 18, 2009 at 09:18:30AM -0400, Ric Wheeler wrote:
> > 
> > We have a new toy to test very large & slow storage with built up from 5 
> > SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives and 
> > 16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA MUX 
> > adapter daughter card in the disk sled).
> > 
> > The basic idea is to build a cheap & slow test bed for file & storage 
> > system scalability. Collectively, we have about 120TB (raw) of capacity 
> > to play with in one server.
> > 
> > As we work through various issues, a couple of oddities popped out.
> > 
> > The first is that udev grumbles during boot about "file name too long" 
> > like the following:
> > 
> > Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file 
> > '/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1': 
> > File name too long
> 
> Odd, what is the sysfs tree for this device?  You have expanders
> attached to ports attached to expanders?  How deep can you go?

In terms of tree depth, there's no real limit (although the deeper it
goes, the slower the network).  The examples in the docs usually go
about 4 expanders deep.

James

Re: mpt2sas logged messages

[Ric Wheeler]

On 08/18/2009 09:25 AM, Greg KH wrote:
> On Tue, Aug 18, 2009 at 09:18:30AM -0400, Ric Wheeler wrote:
>>
>> We have a new toy to test very large&  slow storage with built up from 5
>> SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives and
>> 16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA MUX
>> adapter daughter card in the disk sled).
>>
>> The basic idea is to build a cheap&  slow test bed for file&  storage
>> system scalability. Collectively, we have about 120TB (raw) of capacity
>> to play with in one server.
>>
>> As we work through various issues, a couple of oddities popped out.
>>
>> The first is that udev grumbles during boot about "file name too long"
>> like the following:
>>
>> Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file
>> '/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1':
>> File name too long
>
> Odd, what is the sysfs tree for this device?  You have expanders
> attached to ports attached to expanders?  How deep can you go?
>
> thanks,
>
> greg k-h

There are two dual-port SAS HBA's in the server that plug into the first of the 
5 SAS expansion shelves. Each shelf has two internal SAS loops and is daisy 
chained to the next shelf....  This test is running with just the first 4 
shelves active (although that fifth shelf is plugged in, just not used/active).

Each of the 60 S-ATA disks sits behind a MUX card which lets it appear on both 
loops.

We could break up the shelves into two independent sets of devices which would 
limit the tree depth.

How can I get you the sysfs tree information in a useful way?

Ric

Re: mpt2sas logged messages

[Greg KH]

On Tue, Aug 18, 2009 at 09:59:56AM -0400, Ric Wheeler wrote:
> On 08/18/2009 09:25 AM, Greg KH wrote:
> > On Tue, Aug 18, 2009 at 09:18:30AM -0400, Ric Wheeler wrote:
> >>
> >> We have a new toy to test very large&  slow storage with built up from 5
> >> SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives and
> >> 16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA MUX
> >> adapter daughter card in the disk sled).
> >>
> >> The basic idea is to build a cheap&  slow test bed for file&  storage
> >> system scalability. Collectively, we have about 120TB (raw) of capacity
> >> to play with in one server.
> >>
> >> As we work through various issues, a couple of oddities popped out.
> >>
> >> The first is that udev grumbles during boot about "file name too long"
> >> like the following:
> >>
> >> Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file
> >> '/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1':
> >> File name too long
> >
> > Odd, what is the sysfs tree for this device?  You have expanders
> > attached to ports attached to expanders?  How deep can you go?
> >
> > thanks,
> >
> > greg k-h
> 
> There are two dual-port SAS HBA's in the server that plug into the first of the 
> 5 SAS expansion shelves. Each shelf has two internal SAS loops and is daisy 
> chained to the next shelf....  This test is running with just the first 4 
> shelves active (although that fifth shelf is plugged in, just not used/active).
> 
> Each of the 60 S-ATA disks sits behind a MUX card which lets it appear on both 
> loops.
> 
> We could break up the shelves into two independent sets of devices which would 
> limit the tree depth.
> 
> How can I get you the sysfs tree information in a useful way?

'tree /sys/devices/'
or
'find /sys/devices/'
would be good.

thanks,

greg k-h

Re: mpt2sas logged messages

[Ric Wheeler]

[-- Attachment #1: Type: text/plain, Size: 2362 bytes --]

On 08/18/2009 10:57 AM, Greg KH wrote:
> On Tue, Aug 18, 2009 at 09:59:56AM -0400, Ric Wheeler wrote:
>    
>> On 08/18/2009 09:25 AM, Greg KH wrote:
>>      
>>> On Tue, Aug 18, 2009 at 09:18:30AM -0400, Ric Wheeler wrote:
>>>        
>>>> We have a new toy to test very large&   slow storage with built up from 5
>>>> SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives and
>>>> 16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA MUX
>>>> adapter daughter card in the disk sled).
>>>>
>>>> The basic idea is to build a cheap&   slow test bed for file&   storage
>>>> system scalability. Collectively, we have about 120TB (raw) of capacity
>>>> to play with in one server.
>>>>
>>>> As we work through various issues, a couple of oddities popped out.
>>>>
>>>> The first is that udev grumbles during boot about "file name too long"
>>>> like the following:
>>>>
>>>> Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file
>>>> '/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1':
>>>> File name too long
>>>>          
>>> Odd, what is the sysfs tree for this device?  You have expanders
>>> attached to ports attached to expanders?  How deep can you go?
>>>
>>> thanks,
>>>
>>> greg k-h
>>>        
>> There are two dual-port SAS HBA's in the server that plug into the first of the
>> 5 SAS expansion shelves. Each shelf has two internal SAS loops and is daisy
>> chained to the next shelf....  This test is running with just the first 4
>> shelves active (although that fifth shelf is plugged in, just not used/active).
>>
>> Each of the 60 S-ATA disks sits behind a MUX card which lets it appear on both
>> loops.
>>
>> We could break up the shelves into two independent sets of devices which would
>> limit the tree depth.
>>
>> How can I get you the sysfs tree information in a useful way?
>>      
> 'tree /sys/devices/'
> or
> 'find /sys/devices/'
> would be good.
>
> thanks,
>
> greg k-h
>    

Attached is the bzipped output from - the uncompressed output is quite 
large. Note that this same server has CCIS controllers and fibre HBA's 
as well,

ric


[-- Attachment #2: sysfs_tree.txt.bz2 --]
[-- Type: application/x-bzip, Size: 17250 bytes --]

Re: mpt2sas logged messages

[James Bottomley]

On Tue, 2009-08-18 at 11:53 -0400, Ric Wheeler wrote:
> On 08/18/2009 10:57 AM, Greg KH wrote:
> > On Tue, Aug 18, 2009 at 09:59:56AM -0400, Ric Wheeler wrote:
> >    
> >> On 08/18/2009 09:25 AM, Greg KH wrote:
> >>      
> >>> On Tue, Aug 18, 2009 at 09:18:30AM -0400, Ric Wheeler wrote:
> >>>        
> >>>> We have a new toy to test very large&   slow storage with built up from 5
> >>>> SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives and
> >>>> 16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA MUX
> >>>> adapter daughter card in the disk sled).
> >>>>
> >>>> The basic idea is to build a cheap&   slow test bed for file&   storage
> >>>> system scalability. Collectively, we have about 120TB (raw) of capacity
> >>>> to play with in one server.
> >>>>
> >>>> As we work through various issues, a couple of oddities popped out.
> >>>>
> >>>> The first is that udev grumbles during boot about "file name too long"
> >>>> like the following:
> >>>>
> >>>> Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file
> >>>> '/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1':
> >>>> File name too long
> >>>>          
> >>> Odd, what is the sysfs tree for this device?  You have expanders
> >>> attached to ports attached to expanders?  How deep can you go?
> >>>
> >>> thanks,
> >>>
> >>> greg k-h
> >>>        
> >> There are two dual-port SAS HBA's in the server that plug into the first of the
> >> 5 SAS expansion shelves. Each shelf has two internal SAS loops and is daisy
> >> chained to the next shelf....  This test is running with just the first 4
> >> shelves active (although that fifth shelf is plugged in, just not used/active).
> >>
> >> Each of the 60 S-ATA disks sits behind a MUX card which lets it appear on both
> >> loops.
> >>
> >> We could break up the shelves into two independent sets of devices which would
> >> limit the tree depth.
> >>
> >> How can I get you the sysfs tree information in a useful way?
> >>      
> > 'tree /sys/devices/'
> > or
> > 'find /sys/devices/'
> > would be good.
> >
> > thanks,
> >
> > greg k-h
> >    
> 
> Attached is the bzipped output from - the uncompressed output is quite 
> large. Note that this same server has CCIS controllers and fibre HBA's 
> as well,

It's perfectly legal the way you have it, but I will say you have an
inefficient configuration.  The way a configuration like this is
supposed to look is that there should be a fanout expander at the top
going to the expander in each tray, for a routing depth of two for every
device.

The way you've got it:  expander daisy chained off the next expander
gives an unnecessary routing delay to the disks furthest away in the
chain.

James

Re: mpt2sas logged messages

[Ric Wheeler]

On 08/18/2009 12:09 PM, James Bottomley wrote:
> On Tue, 2009-08-18 at 11:53 -0400, Ric Wheeler wrote:
>> On 08/18/2009 10:57 AM, Greg KH wrote:
>>> On Tue, Aug 18, 2009 at 09:59:56AM -0400, Ric Wheeler wrote:
>>>
>>>> On 08/18/2009 09:25 AM, Greg KH wrote:
>>>>
>>>>> On Tue, Aug 18, 2009 at 09:18:30AM -0400, Ric Wheeler wrote:
>>>>>
>>>>>> We have a new toy to test very large&    slow storage with built up from 5
>>>>>> SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives and
>>>>>> 16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA MUX
>>>>>> adapter daughter card in the disk sled).
>>>>>>
>>>>>> The basic idea is to build a cheap&    slow test bed for file&    storage
>>>>>> system scalability. Collectively, we have about 120TB (raw) of capacity
>>>>>> to play with in one server.
>>>>>>
>>>>>> As we work through various issues, a couple of oddities popped out.
>>>>>>
>>>>>> The first is that udev grumbles during boot about "file name too long"
>>>>>> like the following:
>>>>>>
>>>>>> Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file
>>>>>> '/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1':
>>>>>> File name too long
>>>>>>
>>>>> Odd, what is the sysfs tree for this device?  You have expanders
>>>>> attached to ports attached to expanders?  How deep can you go?
>>>>>
>>>>> thanks,
>>>>>
>>>>> greg k-h
>>>>>
>>>> There are two dual-port SAS HBA's in the server that plug into the first of the
>>>> 5 SAS expansion shelves. Each shelf has two internal SAS loops and is daisy
>>>> chained to the next shelf....  This test is running with just the first 4
>>>> shelves active (although that fifth shelf is plugged in, just not used/active).
>>>>
>>>> Each of the 60 S-ATA disks sits behind a MUX card which lets it appear on both
>>>> loops.
>>>>
>>>> We could break up the shelves into two independent sets of devices which would
>>>> limit the tree depth.
>>>>
>>>> How can I get you the sysfs tree information in a useful way?
>>>>
>>> 'tree /sys/devices/'
>>> or
>>> 'find /sys/devices/'
>>> would be good.
>>>
>>> thanks,
>>>
>>> greg k-h
>>>
>>
>> Attached is the bzipped output from - the uncompressed output is quite
>> large. Note that this same server has CCIS controllers and fibre HBA's
>> as well,
>
> It's perfectly legal the way you have it, but I will say you have an
> inefficient configuration.  The way a configuration like this is
> supposed to look is that there should be a fanout expander at the top
> going to the expander in each tray, for a routing depth of two for every
> device.
>
> The way you've got it:  expander daisy chained off the next expander
> gives an unnecessary routing delay to the disks furthest away in the
> chain.
>
> James

I understand that this configuration is not optimal, but from what I saw with 
some commercial arrays, this is not an uncommon config (up to 4 shelves) when 
going for capacity over performance.

Do you have a particular SAS fanout expander in mind?  I suppose that we could 
always add more hba's as well which would have other benefits...

Thanks!

Ric

Re: mpt2sas logged messages

[James Bottomley]

On Tue, 2009-08-18 at 13:44 -0400, Ric Wheeler wrote:
> On 08/18/2009 12:09 PM, James Bottomley wrote:
> > On Tue, 2009-08-18 at 11:53 -0400, Ric Wheeler wrote:
> >> On 08/18/2009 10:57 AM, Greg KH wrote:
> >>> On Tue, Aug 18, 2009 at 09:59:56AM -0400, Ric Wheeler wrote:
> >>>
> >>>> On 08/18/2009 09:25 AM, Greg KH wrote:
> >>>>
> >>>>> On Tue, Aug 18, 2009 at 09:18:30AM -0400, Ric Wheeler wrote:
> >>>>>
> >>>>>> We have a new toy to test very large&    slow storage with built up from 5
> >>>>>> SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives and
> >>>>>> 16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA MUX
> >>>>>> adapter daughter card in the disk sled).
> >>>>>>
> >>>>>> The basic idea is to build a cheap&    slow test bed for file&    storage
> >>>>>> system scalability. Collectively, we have about 120TB (raw) of capacity
> >>>>>> to play with in one server.
> >>>>>>
> >>>>>> As we work through various issues, a couple of oddities popped out.
> >>>>>>
> >>>>>> The first is that udev grumbles during boot about "file name too long"
> >>>>>> like the following:
> >>>>>>
> >>>>>> Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file
> >>>>>> '/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1':
> >>>>>> File name too long
> >>>>>>
> >>>>> Odd, what is the sysfs tree for this device?  You have expanders
> >>>>> attached to ports attached to expanders?  How deep can you go?
> >>>>>
> >>>>> thanks,
> >>>>>
> >>>>> greg k-h
> >>>>>
> >>>> There are two dual-port SAS HBA's in the server that plug into the first of the
> >>>> 5 SAS expansion shelves. Each shelf has two internal SAS loops and is daisy
> >>>> chained to the next shelf....  This test is running with just the first 4
> >>>> shelves active (although that fifth shelf is plugged in, just not used/active).
> >>>>
> >>>> Each of the 60 S-ATA disks sits behind a MUX card which lets it appear on both
> >>>> loops.
> >>>>
> >>>> We could break up the shelves into two independent sets of devices which would
> >>>> limit the tree depth.
> >>>>
> >>>> How can I get you the sysfs tree information in a useful way?
> >>>>
> >>> 'tree /sys/devices/'
> >>> or
> >>> 'find /sys/devices/'
> >>> would be good.
> >>>
> >>> thanks,
> >>>
> >>> greg k-h
> >>>
> >>
> >> Attached is the bzipped output from - the uncompressed output is quite
> >> large. Note that this same server has CCIS controllers and fibre HBA's
> >> as well,
> >
> > It's perfectly legal the way you have it, but I will say you have an
> > inefficient configuration.  The way a configuration like this is
> > supposed to look is that there should be a fanout expander at the top
> > going to the expander in each tray, for a routing depth of two for every
> > device.
> >
> > The way you've got it:  expander daisy chained off the next expander
> > gives an unnecessary routing delay to the disks furthest away in the
> > chain.
> >
> > James
> 
> I understand that this configuration is not optimal, but from what I saw with 
> some commercial arrays, this is not an uncommon config (up to 4 shelves) when 
> going for capacity over performance.

The config is fine ... it's the way the daisy chain routing is done
which isn't.  You can see that each shelf gets further away from the HBA
by an expander as you go up.

> Do you have a particular SAS fanout expander in mind?  I suppose that we could 
> always add more hba's as well which would have other benefits...

Not really ... I've never seen a real expander in the flesh.  I've got a
set of experimental ones LSI gave me (as bare circuit boards).

If you don't have the expanders, you can likely rig the first expander
to act as a fanout since it must have table routed ports otherwise it
wouldn't work in the daisy chain.

Failing that, as you suggest, multiple HBAs subbing for the fanout
expander would be fine as well.

James

Re: mpt2sas logged messages

[Ric Wheeler]

On 08/18/2009 04:14 PM, James Bottomley wrote:
> On Tue, 2009-08-18 at 13:44 -0400, Ric Wheeler wrote:
>> On 08/18/2009 12:09 PM, James Bottomley wrote:
>>> On Tue, 2009-08-18 at 11:53 -0400, Ric Wheeler wrote:
>>>> On 08/18/2009 10:57 AM, Greg KH wrote:
>>>>> On Tue, Aug 18, 2009 at 09:59:56AM -0400, Ric Wheeler wrote:
>>>>>
>>>>>> On 08/18/2009 09:25 AM, Greg KH wrote:
>>>>>>
>>>>>>> On Tue, Aug 18, 2009 at 09:18:30AM -0400, Ric Wheeler wrote:
>>>>>>>
>>>>>>>> We have a new toy to test very large&     slow storage with built up from 5
>>>>>>>> SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives and
>>>>>>>> 16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA MUX
>>>>>>>> adapter daughter card in the disk sled).
>>>>>>>>
>>>>>>>> The basic idea is to build a cheap&     slow test bed for file&     storage
>>>>>>>> system scalability. Collectively, we have about 120TB (raw) of capacity
>>>>>>>> to play with in one server.
>>>>>>>>
>>>>>>>> As we work through various issues, a couple of oddities popped out.
>>>>>>>>
>>>>>>>> The first is that udev grumbles during boot about "file name too long"
>>>>>>>> like the following:
>>>>>>>>
>>>>>>>> Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file
>>>>>>>> '/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1':
>>>>>>>> File name too long
>>>>>>>>
>>>>>>> Odd, what is the sysfs tree for this device?  You have expanders
>>>>>>> attached to ports attached to expanders?  How deep can you go?
>>>>>>>
>>>>>>> thanks,
>>>>>>>
>>>>>>> greg k-h
>>>>>>>
>>>>>> There are two dual-port SAS HBA's in the server that plug into the first of the
>>>>>> 5 SAS expansion shelves. Each shelf has two internal SAS loops and is daisy
>>>>>> chained to the next shelf....  This test is running with just the first 4
>>>>>> shelves active (although that fifth shelf is plugged in, just not used/active).
>>>>>>
>>>>>> Each of the 60 S-ATA disks sits behind a MUX card which lets it appear on both
>>>>>> loops.
>>>>>>
>>>>>> We could break up the shelves into two independent sets of devices which would
>>>>>> limit the tree depth.
>>>>>>
>>>>>> How can I get you the sysfs tree information in a useful way?
>>>>>>
>>>>> 'tree /sys/devices/'
>>>>> or
>>>>> 'find /sys/devices/'
>>>>> would be good.
>>>>>
>>>>> thanks,
>>>>>
>>>>> greg k-h
>>>>>
>>>>
>>>> Attached is the bzipped output from - the uncompressed output is quite
>>>> large. Note that this same server has CCIS controllers and fibre HBA's
>>>> as well,
>>>
>>> It's perfectly legal the way you have it, but I will say you have an
>>> inefficient configuration.  The way a configuration like this is
>>> supposed to look is that there should be a fanout expander at the top
>>> going to the expander in each tray, for a routing depth of two for every
>>> device.
>>>
>>> The way you've got it:  expander daisy chained off the next expander
>>> gives an unnecessary routing delay to the disks furthest away in the
>>> chain.
>>>
>>> James
>>
>> I understand that this configuration is not optimal, but from what I saw with
>> some commercial arrays, this is not an uncommon config (up to 4 shelves) when
>> going for capacity over performance.
>
> The config is fine ... it's the way the daisy chain routing is done
> which isn't.  You can see that each shelf gets further away from the HBA
> by an expander as you go up.
>
>> Do you have a particular SAS fanout expander in mind?  I suppose that we could
>> always add more hba's as well which would have other benefits...
>
> Not really ... I've never seen a real expander in the flesh.  I've got a
> set of experimental ones LSI gave me (as bare circuit boards).
>
> If you don't have the expanders, you can likely rig the first expander
> to act as a fanout since it must have table routed ports otherwise it
> wouldn't work in the daisy chain.
>
> Failing that, as you suggest, multiple HBAs subbing for the fanout
> expander would be fine as well.
>
> James

Adding more HBA's (one per shelf or pair of shelves) would probably be the 
easiest way around this....

Thanks!

Ric

Re: mpt2sas logged messages

[Greg KH]

On Tue, Aug 18, 2009 at 11:53:32AM -0400, Ric Wheeler wrote:
> On 08/18/2009 10:57 AM, Greg KH wrote:
> > On Tue, Aug 18, 2009 at 09:59:56AM -0400, Ric Wheeler wrote:
> >    
> >> On 08/18/2009 09:25 AM, Greg KH wrote:
> >>      
> >>> On Tue, Aug 18, 2009 at 09:18:30AM -0400, Ric Wheeler wrote:
> >>>        
> >>>> We have a new toy to test very large&   slow storage with built up from 5
> >>>> SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives and
> >>>> 16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA MUX
> >>>> adapter daughter card in the disk sled).
> >>>>
> >>>> The basic idea is to build a cheap&   slow test bed for file&   storage
> >>>> system scalability. Collectively, we have about 120TB (raw) of capacity
> >>>> to play with in one server.
> >>>>
> >>>> As we work through various issues, a couple of oddities popped out.
> >>>>
> >>>> The first is that udev grumbles during boot about "file name too long"
> >>>> like the following:
> >>>>
> >>>> Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file
> >>>> '/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1':
> >>>> File name too long

Ok, it looks like the way that udev handles it's internal "database"
can't handle paths that are larger than a single filename size.

Kay, I think this is all yours :)

thanks,

greg k-h

Re: mpt2sas logged messages

[Ric Wheeler]

On 08/18/2009 12:40 PM, Greg KH wrote:
> On Tue, Aug 18, 2009 at 11:53:32AM -0400, Ric Wheeler wrote:
>> On 08/18/2009 10:57 AM, Greg KH wrote:
>>> On Tue, Aug 18, 2009 at 09:59:56AM -0400, Ric Wheeler wrote:
>>>
>>>> On 08/18/2009 09:25 AM, Greg KH wrote:
>>>>
>>>>> On Tue, Aug 18, 2009 at 09:18:30AM -0400, Ric Wheeler wrote:
>>>>>
>>>>>> We have a new toy to test very large&    slow storage with built up from 5
>>>>>> SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives and
>>>>>> 16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA MUX
>>>>>> adapter daughter card in the disk sled).
>>>>>>
>>>>>> The basic idea is to build a cheap&    slow test bed for file&    storage
>>>>>> system scalability. Collectively, we have about 120TB (raw) of capacity
>>>>>> to play with in one server.
>>>>>>
>>>>>> As we work through various issues, a couple of oddities popped out.
>>>>>>
>>>>>> The first is that udev grumbles during boot about "file name too long"
>>>>>> like the following:
>>>>>>
>>>>>> Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file
>>>>>> '/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1':
>>>>>> File name too long
>
> Ok, it looks like the way that udev handles it's internal "database"
> can't handle paths that are larger than a single filename size.
>
> Kay, I think this is all yours :)
>
> thanks,
>
> greg k-h

Happy to try patches, also happy to try and work around this with a different 
configuration :-)

ric

Re: mpt2sas logged messages

[Kay Sievers]

On Tue, Aug 18, 2009 at 18:40, Greg KH<greg@kroah.com> wrote:
> On Tue, Aug 18, 2009 at 11:53:32AM -0400, Ric Wheeler wrote:
>> On 08/18/2009 10:57 AM, Greg KH wrote:
>> > On Tue, Aug 18, 2009 at 09:59:56AM -0400, Ric Wheeler wrote:
>> >
>> >> On 08/18/2009 09:25 AM, Greg KH wrote:
>> >>
>> >>> On Tue, Aug 18, 2009 at 09:18:30AM -0400, Ric Wheeler wrote:
>> >>>
>> >>>> We have a new toy to test very large&   slow storage with built up from 5
>> >>>> SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives and
>> >>>> 16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA MUX
>> >>>> adapter daughter card in the disk sled).
>> >>>>
>> >>>> The basic idea is to build a cheap&   slow test bed for file&   storage
>> >>>> system scalability. Collectively, we have about 120TB (raw) of capacity
>> >>>> to play with in one server.
>> >>>>
>> >>>> As we work through various issues, a couple of oddities popped out.
>> >>>>
>> >>>> The first is that udev grumbles during boot about "file name too long"
>> >>>> like the following:
>> >>>>
>> >>>> Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file
>> >>>> '/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1':
>> >>>> File name too long
>
> Ok, it looks like the way that udev handles it's internal "database"
> can't handle paths that are larger than a single filename size.
>
> Kay, I think this is all yours :)

Yeah, seems so. Linux can not have filenames longer than 256. We'll
need to change the way we store the DEVPATH in a filename. I'll look
into it.

Thanks,
Kay
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mpt2sas logged messages

[Ric Wheeler]

On 08/18/2009 02:37 PM, Kay Sievers wrote:
> On Tue, Aug 18, 2009 at 18:40, Greg KH<greg@kroah.com>  wrote:
>> On Tue, Aug 18, 2009 at 11:53:32AM -0400, Ric Wheeler wrote:
>>> On 08/18/2009 10:57 AM, Greg KH wrote:
>>>> On Tue, Aug 18, 2009 at 09:59:56AM -0400, Ric Wheeler wrote:
>>>>
>>>>> On 08/18/2009 09:25 AM, Greg KH wrote:
>>>>>
>>>>>> On Tue, Aug 18, 2009 at 09:18:30AM -0400, Ric Wheeler wrote:
>>>>>>
>>>>>>> We have a new toy to test very large&     slow storage with built up from 5
>>>>>>> SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives and
>>>>>>> 16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA MUX
>>>>>>> adapter daughter card in the disk sled).
>>>>>>>
>>>>>>> The basic idea is to build a cheap&     slow test bed for file&     storage
>>>>>>> system scalability. Collectively, we have about 120TB (raw) of capacity
>>>>>>> to play with in one server.
>>>>>>>
>>>>>>> As we work through various issues, a couple of oddities popped out.
>>>>>>>
>>>>>>> The first is that udev grumbles during boot about "file name too long"
>>>>>>> like the following:
>>>>>>>
>>>>>>> Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file
>>>>>>> '/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1':
>>>>>>> File name too long
>>
>> Ok, it looks like the way that udev handles it's internal "database"
>> can't handle paths that are larger than a single filename size.
>>
>> Kay, I think this is all yours :)
>
> Yeah, seems so. Linux can not have filenames longer than 256. We'll
> need to change the way we store the DEVPATH in a filename. I'll look
> into it.
>
> Thanks,
> Kay

Hi Kay,

I think that we can reconfigure the storage to avoid this but will be happy to 
restore this configuration if you have something that you want to test later on...

Thanks!

Ric

Re: mpt2sas logged messages

[Kay Sievers]

On Tue, Aug 18, 2009 at 20:41, Ric Wheeler<rwheeler@redhat.com> wrote:

> I think that we can reconfigure the storage to avoid this but will be happy
> to restore this configuration if you have something that you want to test
> later on...

This hopefully addresses the issue:
  http://git.kernel.org/?p=linux/hotplug/udev.git;a=commit;h=2ffc9cc1917b1bb6fe86881a94a47dce9aa15168

Thanks,
Kay

Re: mpt2sas logged messages

[Ric Wheeler]

On 08/19/2009 03:30 PM, Kay Sievers wrote:
> On Tue, Aug 18, 2009 at 20:41, Ric Wheeler<rwheeler@redhat.com>  wrote:
>
>> I think that we can reconfigure the storage to avoid this but will be happy
>> to restore this configuration if you have something that you want to test
>> later on...
>
> This hopefully addresses the issue:
>    http://git.kernel.org/?p=linux/hotplug/udev.git;a=commit;h=2ffc9cc1917b1bb6fe86881a94a47dce9aa15168
>
> Thanks,
> Kay

Great, I will give these a shot - thanks!

Ric

Re: mpt2sas logged messages

[Ric Wheeler]

On 08/19/2009 03:30 PM, Kay Sievers wrote:
> On Tue, Aug 18, 2009 at 20:41, Ric Wheeler<rwheeler@redhat.com>  wrote:
>
>> I think that we can reconfigure the storage to avoid this but will be happy
>> to restore this configuration if you have something that you want to test
>> later on...
>
> This hopefully addresses the issue:
>    http://git.kernel.org/?p=linux/hotplug/udev.git;a=commit;h=2ffc9cc1917b1bb6fe86881a94a47dce9aa15168
>
> Thanks,
> Kay

Hi Kay,

This seems to resolve the issue on my test rig - no more "too long" error 
messages during boot.

Many thanks!

Ric

Re: mpt2sas logged messages

[Kay Sievers]

On Thu, Aug 20, 2009 at 17:18, Ric Wheeler<rwheeler@redhat.com> wrote:
> On 08/19/2009 03:30 PM, Kay Sievers wrote:
>> On Tue, Aug 18, 2009 at 20:41, Ric Wheeler<rwheeler@redhat.com>  wrote:
>>
>>> I think that we can reconfigure the storage to avoid this but will be
>>> happy
>>> to restore this configuration if you have something that you want to test
>>> later on...
>>
>> This hopefully addresses the issue:
>>
>> http://git.kernel.org/?p=linux/hotplug/udev.git;a=commit;h=2ffc9cc1917b1bb6fe86881a94a47dce9aa15168
>>
>> Thanks,
>> Kay
>
> Hi Kay,
>
> This seems to resolve the issue on my test rig - no more "too long" error
> messages during boot.

Great! Thanks a lot for the testing, and letting us know. It's a
pleasure when things work like this.

Kay
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mpt2sas logged messages

[Ric Wheeler]

On 08/20/2009 11:20 AM, Kay Sievers wrote:
> On Thu, Aug 20, 2009 at 17:18, Ric Wheeler<rwheeler@redhat.com>  wrote:
>> On 08/19/2009 03:30 PM, Kay Sievers wrote:
>>> On Tue, Aug 18, 2009 at 20:41, Ric Wheeler<rwheeler@redhat.com>    wrote:
>>>
>>>> I think that we can reconfigure the storage to avoid this but will be
>>>> happy
>>>> to restore this configuration if you have something that you want to test
>>>> later on...
>>>
>>> This hopefully addresses the issue:
>>>
>>> http://git.kernel.org/?p=linux/hotplug/udev.git;a=commit;h=2ffc9cc1917b1bb6fe86881a94a47dce9aa15168
>>>
>>> Thanks,
>>> Kay
>>
>> Hi Kay,
>>
>> This seems to resolve the issue on my test rig - no more "too long" error
>> messages during boot.
>
> Great! Thanks a lot for the testing, and letting us know. It's a
> pleasure when things work like this.
>
> Kay

The pleasure is all mine - great to get such rapid fixes :-)

It has been a long time challenge for us to get developers on big systems, 
especially big and complicated storage boxes which is why we decided to try and 
build this box to help us stress scale in every way so we can track down exactly 
this kind of issue in a timely way.

It was not that expensive to build a 100+ TB box - 4 SAS expansion shelves, 60 
2TB S-ATA disks and a couple of HBA's for a reasonable server. It won't be that 
fast, but is much cheaper than buying a similar array from a real vendor...

ric

Re: mpt2sas logged messages

[Ric Wheeler]

On 08/18/2009 09:18 AM, Ric Wheeler wrote:
>
> We have a new toy to test very large & slow storage with built up from 
> 5 SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives 
> and 16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA 
> MUX adapter daughter card in the disk sled).
>
> The basic idea is to build a cheap & slow test bed for file & storage 
> system scalability. Collectively, we have about 120TB (raw) of 
> capacity to play with in one server.
>
> As we work through various issues, a couple of oddities popped out.
>
> The first is that udev grumbles during boot about "file name too long" 
> like the following:
>
> Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file 
> '/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1': 
> File name too long
>
> The second is that the mpt2sas driver spews various messages like:
>
> Aug 17 06:55:17 megadeth kernel: mpt2sas0: log_info(0x31120102): 
> originator(PL), code(0x12), sub_code(0x0102)
>
> Any insight into whether these are issues worth pursuing is 
> appreciated, thanks!
>
> Ric
>
>

I left out the interesting part - this was an F11 system with a 
2.6.31-rc6 kernel. I am updating the box to F12 rawhide to make sure 
that udev and related components are updated and will retest,

Ric

RE: mpt2sas logged messages

[Moore, Eric]

On Tuesday, August 18, 2009 7:19 AM,  Ric Wheeler wrote: 
> 
> The second is that the mpt2sas driver spews various messages like:
> 
> Aug 17 06:55:17 megadeth kernel: mpt2sas0: log_info(0x31120102): 
> originator(PL), code(0x12), sub_code(0x0102)

Ric - 

The code (0x12) means PL_LOGINFO_CODE_ABORT
The sub_code(0x102) means PL_LOGINFO_SUB_CODE_OPEN_FAILURE_PATHWAY_BLOCKED

Eric Moore

RE: mpt2sas logged messages

[James Bottomley]

On Tue, 2009-08-18 at 09:39 -0600, Moore, Eric wrote:
> On Tuesday, August 18, 2009 7:19 AM,  Ric Wheeler wrote: 
> > 
> > The second is that the mpt2sas driver spews various messages like:
> > 
> > Aug 17 06:55:17 megadeth kernel: mpt2sas0: log_info(0x31120102): 
> > originator(PL), code(0x12), sub_code(0x0102)
> 
> Ric - 
> 
> The code (0x12) means PL_LOGINFO_CODE_ABORT
> The sub_code(0x102) means PL_LOGINFO_SUB_CODE_OPEN_FAILURE_PATHWAY_BLOCKED

To translate this according to the SAS 1.1 specification, a pathway open
failed because a timeout fired before the path could be established.

You can mitigate this by increasing the expander partial pathway timeout
value. (or just move to a more efficient configuration).

James

RE: mpt2sas logged messages

[Moore, Eric]

On Tuesday, August 18, 2009 10:13 AM,  James Bottomley wrote:
> > The code (0x12) means PL_LOGINFO_CODE_ABORT
> > The sub_code(0x102) means 
> PL_LOGINFO_SUB_CODE_OPEN_FAILURE_PATHWAY_BLOCKED
> 
> To translate this according to the SAS 1.1 specification, a 
> pathway open
> failed because a timeout fired before the path could be established.
> 
> You can mitigate this by increasing the expander partial 
> pathway timeout
> value. (or just move to a more efficient configuration).
> 

I was telling Ric it was something else in seperate email.  The pathway was blocked due to two initiors establishing a connection to single port. That woudl be the case if he wasn't using a sata multiplexor.


anyways, if the pathway timeout is the issue, then I believe you can set the pathway timeout value using Dougs smp tools, smp_phy_control , setting the --pptv command line option.

http://sg.danny.cz/sg/smp_utils.html

Re: mpt2sas logged messages

[Ric Wheeler]

On 08/18/2009 01:02 PM, Moore, Eric wrote:
> On Tuesday, August 18, 2009 10:13 AM,  James Bottomley wrote:
>>> The code (0x12) means PL_LOGINFO_CODE_ABORT
>>> The sub_code(0x102) means
>> PL_LOGINFO_SUB_CODE_OPEN_FAILURE_PATHWAY_BLOCKED
>>
>> To translate this according to the SAS 1.1 specification, a
>> pathway open
>> failed because a timeout fired before the path could be established.
>>
>> You can mitigate this by increasing the expander partial
>> pathway timeout
>> value. (or just move to a more efficient configuration).
>>
>
> I was telling Ric it was something else in seperate email.  The pathway was blocked due to two initiors establishing a connection to single port. That woudl be the case if he wasn't using a sata multiplexor.
>
>
> anyways, if the pathway timeout is the issue, then I believe you can set the pathway timeout value using Dougs smp tools, smp_phy_control , setting the --pptv command line option.
>
> http://sg.danny.cz/sg/smp_utils.html

Thanks, I will give that a try,

ric