[Lustre-devel] loadgen improvements

[Lustre-devel] loadgen improvements

[Yuriy Umanets]

hello Lustre hackers,

ClusterStor would like to start doing real contributions to Lustre stability
and popularity. First off, we would like to improve loadgen utility, written
by Nathan Rutman. We think loadgen is a really good tool for testing and
what is not less important, it is implemented right way from architecture
point of view.

However, these days it has number of issues:
1. Wrong stack size for threads, that results in segfault (find the patch in
attachment);
2. Little locking issues (push_kid() function);
3. Absence of striping functionality, it only may create load on one
OST/ECHO server.

We plan to add striping functionality to loadgen. General idea is to use LOV
module, which already has all the needed code. Loadgen itself and
echo_client also have related code, so it seems this functionality was
planned anyway, though it's not working currently. Currently an attempt to
attach to a LOV results in assertion failures.

Let's discuss these matters. The way we're going to implement this may be
roughly expressed as follows:

1. Attach to LOV device in loadgen, using "device" command. To do so we need
to construct new LOV instance, used by loadgen only, as we cannot use LOV
instance used by LLITE. This requires changes to handling function for
command "device". It should accept more than one OST target;

2. Stripe size and stripe count of new LOV instance should also be specified
while constructing it using "device" command;

3. Setup of virtual clients should take into account that LOV may be used as
a target. In this case attaching dedicated OSCs to created echo_client's is
not needed.

4. In order to be able to run thousands of threads, we need the following:
 - reasonable stack size for threads;
 - change 8192 OBD number limit to something more reasonable.

5. Load itself is created in the way being currently used - via obd_brw().

Thanks.

-- 
umka
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20091205/3f9cc0b6/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: loadgen-stacksize.patch
Type: application/octet-stream
Size: 1193 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20091205/3f9cc0b6/attachment.obj>

[Lustre-devel] loadgen improvements

[Robert Read]

Hi Yury,

Sounds interesting; I'm happy to see loadgen will be improved.  Please create a bug in bugzilla.lustre.org and attach your patch so we can add it to the queue. 

thanks,
robert

On Dec 5, 2009, at 05:21 , Yuriy Umanets wrote:

> hello Lustre hackers,
> 
> ClusterStor would like to start doing real contributions to Lustre stability and popularity. First off, we would like to improve loadgen utility, written by Nathan Rutman. We think loadgen is a really good tool for testing and what is not less important, it is implemented right way from architecture point of view.
> 
> However, these days it has number of issues:
> 1. Wrong stack size for threads, that results in segfault (find the patch in attachment);
> 2. Little locking issues (push_kid() function);
> 3. Absence of striping functionality, it only may create load on one OST/ECHO server.
> 
> We plan to add striping functionality to loadgen. General idea is to use LOV module, which already has all the needed code. Loadgen itself and echo_client also have related code, so it seems this functionality was planned anyway, though it's not working currently. Currently an attempt to attach to a LOV results in assertion failures.
> 
> Let's discuss these matters. The way we're going to implement this may be roughly expressed as follows:
> 
> 1. Attach to LOV device in loadgen, using "device" command. To do so we need to construct new LOV instance, used by loadgen only, as we cannot use LOV instance used by LLITE. This requires changes to handling function for command "device". It should accept more than one OST target;
> 
> 2. Stripe size and stripe count of new LOV instance should also be specified while constructing it using "device" command;
> 
> 3. Setup of virtual clients should take into account that LOV may be used as a target. In this case attaching dedicated OSCs to created echo_client's is not needed.
> 
> 4. In order to be able to run thousands of threads, we need the following:
>  - reasonable stack size for threads;
>  - change 8192 OBD number limit to something more reasonable.
> 
> 5. Load itself is created in the way being currently used - via obd_brw().
> 
> Thanks.
> 
> -- 
> umka
> <loadgen-stacksize.patch>_______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel

[Lustre-devel] loadgen improvements

[Andreas Dilger]

Hello Yuriy,

On 2009-12-05, at 06:21, Yuriy Umanets wrote:
> ClusterStor would like to start doing real contributions to Lustre  
> stability and popularity. First off, we would like to improve  
> loadgen utility, written by Nathan Rutman. We think loadgen is a  
> really good tool for testing and what is not less important, it is  
> implemented right way from architecture point of view.

Great.  We look forward to working with you again.

> However, these days it has number of issues:
> 1. Wrong stack size for threads, that results in segfault (find the  
> patch in attachment);
> 2. Little locking issues (push_kid() function);
> 3. Absence of striping functionality, it only may create load on one  
> OST/ECHO server.

Can you please file a bug for this, and attach this patch and later  
ones there.  That will ensure that it follows the proper inspection  
and testing process.

> We plan to add striping functionality to loadgen. General idea is to  
> use LOV module, which already has all the needed code. Loadgen  
> itself and echo_client also have related code, so it seems this  
> functionality was planned anyway, though it's not working currently.  
> Currently an attempt to attach to a LOV results in assertion failures.

Right - Eric and I worked on that some years ago, but it was never  
used fully.  Can you please ensure that you include a test in  
sanity.sh that exercises the functionality you are adding to loadgen,  
as with echo_client in test 180.  More complex load tests should go  
into their own script.

> Let's discuss these matters. The way we're going to implement this  
> may be roughly expressed as follows:
>
> 1. Attach to LOV device in loadgen, using "device" command. To do so  
> we need to construct new LOV instance, used by loadgen only, as we  
> cannot use LOV instance used by LLITE. This requires changes to  
> handling function for command "device". It should accept more than  
> one OST target;

This sounds reasonable.  It might be useful to support the wildcard  
specification of OSTs like "lustre-OST00[0-30]" or "lustre- 
OST00[0,3,6,9].  Some of that functionality already exists in lustre/ 
utils/nidlist.c.

> 2. Stripe size and stripe count of new LOV instance should also be  
> specified while constructing it using "device" command;

It probably makes sense to have this specified with a separate  
command, so that these parameters can be changed without having to  
tear down the devices and recreate them just to change the striping,  
and it avoids overloading the "device" command (which will soon become  
much more complex by allowing many OSTs to be specified).

> 3. Setup of virtual clients should take into account that LOV may be  
> used as a target. In this case attaching dedicated OSCs to created  
> echo_client's is not needed.
>
> 4. In order to be able to run thousands of threads, we need the  
> following:
>  - reasonable stack size for threads;
>  - change 8192 OBD number limit to something more reasonable.

If you are going beyond 8192 OBD devices just for the loadgen usage  
(presumably you need a new LOV device for each thread?) then it would  
make sense to have the obd_devs array be dynamically allocated and  
remove any upper limit on the number of devices.

Most of this work has already been done - all of the access to  
obd_devs[] is localized in lustre/obdclass/genops.c, with the  
exception of the array initialization, and an old  
EXPORT_SYMBOL(obd_devs), which isn't used anymore. The obd_devs[]  
declaration can be made static to genops.c and the initialization  
moved to obd_init_caches(), and then the actual representation of  
obd_devs[] can be changed without affecting the interface.

Note that the index number of the device is still exported to  
userspace, so it isn't possible/practical to simply go with a linked- 
list implementation.  The actual obd_device structures are already  
allocated and freed dynamically, just the master obd_devs[] array is  
static.  Probably a master array of pointers to pages that hold  
segments of the the obd_devs[] array in order is sufficient.  The  
master array will be quite small (1 pointer per 512 obd_devices) and  
can be reallocated under a rwlock, since nothing references it  
directly.  There is no need to be able to shrink the master array.

It would be trivial to also make the obd_types and obd_types_lock  
declarations static to genops.c, and the initialization in  
obd_init_caches() to properly isolate that code as well.

> 5. Load itself is created in the way being currently used - via  
> obd_brw().


Actually, the real code does not use obd_brw() directly anymore, but  
rather obd_brw_async().  I think Nic Henke submitted a patch to change  
that, I'm not sure of the current status of that patch.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

[Lustre-devel] loadgen improvements

[Nic Henke]

Andreas Dilger wrote:
> Hello Yuriy,
>
>   
>> 5. Load itself is created in the way being currently used - via  
>> obd_brw().
>>     
>
>
> Actually, the real code does not use obd_brw() directly anymore, but  
> rather obd_brw_async().  I think Nic Henke submitted a patch to change  
> that, I'm not sure of the current status of that patch.
>
>   
This is Bug 18225 - looks like it was landed to 1.6.8+ and 1.8.1+, but 
was dropped from HEAD as there is more work that'd need to be done there.

Nic

[Lustre-devel] loadgen improvements

[Alexey Lyashkov]

Hi Andreas,

> 
> > However, these days it has number of issues:
> > 1. Wrong stack size for threads, that results in segfault (find
the  
> > patch in attachment);
> > 2. Little locking issues (push_kid() function);
> > 3. Absence of striping functionality, it only may create load on
one  
> > OST/ECHO server.
> 
> Can you please file a bug for this, and attach this patch and later  
> ones there.  That will ensure that it follows the proper inspection  
> and testing process.
Sure, but looks this will be complete rewrite loadgen.
Reasons for that - loadgen should be use jt functions to set,get lprocfs
data, should be add LOV targets into LOV instance.


> > Let's discuss these matters. The way we're going to implement this  
> > may be roughly expressed as follows:
> >
> > 1. Attach to LOV device in loadgen, using "device" command. To do
so  
> > we need to construct new LOV instance, used by loadgen only, as we  
> > cannot use LOV instance used by LLITE. This requires changes to  
> > handling function for command "device". It should accept more than  
> > one OST target;
> 
> This sounds reasonable.  It might be useful to support the wildcard  
> specification of OSTs like "lustre-OST00[0-30]" or "lustre- 
> OST00[0,3,6,9].  Some of that functionality already exists in lustre/ 
> utils/nidlist.c.
As we need fully setup lustre stack to testing, we can't use willcard in
'device' command, because we don't have access to MGS at this case and
don't have info about all OST tagets in cluster.
But we can use this in 'pool' command.

> > 2. Stripe size and stripe count of new LOV instance should also be  
> > specified while constructing it using "device" command;
> 
> It probably makes sense to have this specified with a separate  
> command, so that these parameters can be changed without having to  
> tear down the devices and recreate them just to change the striping,  
> and it avoids overloading the "device" command (which will soon
become  
> much more complex by allowing many OSTs to be specified).


in first we need call many "device" commands - which add linkage between
UUID and NID.

second step we define OST pool for each workload pattern, this can be
done via new added command 
>>>> pool $name $coma_sepated_OST_uuid
this command used to make LOV_ADD_TARGET commands, but later can be
translated into real OST pool, if need. Main goals of this, add
possibility to use different OST's in different LOV instance.


next step is define workload pattern. currently LoadGen is support only
write, or read command - this translated to obd_brw() or obd_brw_async.
But echo client have support for 3 types of IO.
1) simulate obd_brw/obd_brw_async.
2) simulate obd_queue_async_page()
3) looks direct connect to obdfilter to use obd_prep() & obd_commit()
calls.

Also we need extend this pattern to use OST pool and striping
parameters.
This can be something similar as
>>> pattern $name $pool_name $operation [$stripe_size [$stripe_count]]
where 
name - is pattern name
pool_name - is name of OST pools assigned to this pattern
operation - is one of READ or WRITE.
stripe_size - if set, workload need to setup LOV and add linkage between
LOV and OSC targets, and separate single operation to stripes.
stripe_count - if set, workload should be use for only part of OST pool
to each echo object.

next stage is prepare clients.
In this stage we should be send obd_connect to echo client and wait
until all osc targets is connected (via send obd_statfs() or something
similar).
To start clients we can uses:
>>>>  clients $name $count $workload_name [shared]
where 
name - is client group name
count - is client's count in that group
workload_name - is name of workload parameters.
shared - if set, clients in this pattern is share single OSC or LOV
target. if don't set - each worker have own lustre stack.  
At this stage we also create echo objects with requested LSM for each
client.

final step is spawn one thread for one client and client read own
pattern to call echo client obd IOCTL to start load.


-- 
Alexey Lyashkov <alexey.lyashkov@clusterstor.com>
ClusterStor

[Lustre-devel] loadgen improvements

[Andreas Dilger]

On 2009-12-08, at 22:07, Alexey Lyashkov wrote:
> This can be something similar as
>>
>     pattern $name $pool_name $operation [$stripe_size [$stripe_count]]
>
> where
> name - is pattern name
> pool_name - is name of OST pools assigned to this pattern
> operation - is one of READ or WRITE.
> stripe_size - if set, workload need to setup LOV and add linkage  
> between
> LOV and OSC targets, and separate single operation to stripes.
> stripe_count - if set, workload should be use for only part of OST  
> pool
> to each echo object.

Please do not use positional parameters.  That makes it nearly  
impossible to change the input format in the future, or to skip some  
fields and leave them as defaults.  This needs to use getopt() and  
named parameters:

  --pool=pool_name, --stripe_size=N,  --stripe_count=N

> next stage is prepare clients.
> In this stage we should be send obd_connect to echo client and wait
> until all osc targets is connected (via send obd_statfs() or something
> similar).
> To start clients we can uses:
>>
>     clients $name $count $workload_name [shared]
> where
> name - is client group name
> count - is client's count in that group
> workload_name - is name of workload parameters.
> shared - if set, clients in this pattern is share single OSC or LOV
> target. if don't set - each worker have own lustre stack.
> At this stage we also create echo objects with requested LSM for each
> client.

Same as above - please use named parameters instead of positional ones.

It probably also makes sense to look at the functions provided by LST  
(LNET self test) so that we might have these two tools operate in a  
similar manner. LST testing at the LNET level, loadgen testing at the  
OBD/CLIO level.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

[Lustre-devel] loadgen improvements

[Alexey Lyashkov]

On Tue, 2009-12-08 at 23:35 -0700, Andreas Dilger wrote:
> On 2009-12-08, at 22:07, Alexey Lyashkov wrote:
> > This can be something similar as
> >>
> >     pattern $name $pool_name $operation [$stripe_size [$stripe_count]]
> >
> > where
> > name - is pattern name
> > pool_name - is name of OST pools assigned to this pattern
> > operation - is one of READ or WRITE.
> > stripe_size - if set, workload need to setup LOV and add linkage  
> > between
> > LOV and OSC targets, and separate single operation to stripes.
> > stripe_count - if set, workload should be use for only part of OST  
> > pool
> > to each echo object.
> 
> Please do not use positional parameters.  That makes it nearly  
> impossible to change the input format in the future, or to skip some  
> fields and leave them as defaults.  This needs to use getopt() and  
> named parameters:
> 
>   --pool=pool_name, --stripe_size=N,  --stripe_count=N
This possible, but unlikely. because need will be save current pool
name, instead have parsed as single command.

And I don't have plan to rewrite loadgen to use getopt, instead of
current Parser() function which is used in lctl, lfs, and other.
But i agree about avoid position depended parameters.

How about this style
>>> pool $name { targets:"OST1, OST2" }
and
>>> pattern $pat { stripe_count:X stripe_size:Y block_size:Z };
and 
>>> clients $pattern { count:XX pattern: pat shared }
?


-- 
Alexey Lyashkov <alexey.lyashkov@clusterstor.com>
ClusterStor

[Lustre-devel] loadgen improvements

[Andreas Dilger]

On 2009-12-08, at 23:57, Alexey Lyashkov wrote:
> On Tue, 2009-12-08 at 23:35 -0700, Andreas Dilger wrote:
>> On 2009-12-08, at 22:07, Alexey Lyashkov wrote:
>>> This can be something similar as
>>
>>>
>>>    pattern $name $pool_name $operation [$stripe_size  
>>> [$stripe_count]]
>>
>> Please do not use positional parameters.  That makes it nearly
>> impossible to change the input format in the future, or to skip some
>> fields and leave them as defaults.  This needs to use getopt() and
>> named parameters:
>>
>>  --pool=pool_name, --stripe_size=N,  --stripe_count=N
>
> This possible, but unlikely. because need will be save current pool
> name, instead have parsed as single command.
>
> And I don't have plan to rewrite loadgen to use getopt, instead of
> current Parser() function which is used in lctl, lfs, and other.

Note that the existing parser function passes argc, argv to the helper  
function, which is free to use getopt() or getopt_long() internally as  
desired.  See, for example, lustre/utils/lfs.c::lfs_setstripe(), which  
handles the parsing of "lfs setstripe --count N --pool {pool} ...".

> How about this style
>>
> pool $name { targets:"OST1, OST2" }
> pattern $pat { stripe_count:X stripe_size:Y block_size:Z };
> clients $pattern { count:XX pattern: pat shared }


I have no objection to splitting up the functionality into multiple  
sub-commands.  Allowing declarations of pools and patterns that are re- 
used for each client definition makes a lot of sense.

What I dislike is the introduction of a completely new input format  
and parsing engine for no apparent benefit.  Why not change the above  
to:

loadgen> pool --targets="OST1,OST2,..." NAME
loadgen> pattern --stripe_count=X --stripe_size=Y --block_size=Z PATTERN
loadgen> clients --count=XX --pattern=PATTERN ...

and use well-known input conventions?

It also makes sense to look at how LNET Self Test handles the setup of  
testing loads, and try and make the loadgen usage similar to that, as  
much as possible?

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

[Lustre-devel] loadgen improvements

[Peter Braam]

My message to lustre-devel will be held by the moderator for ever it seems
(13 days ago I sent the first one).

The "standard" conventions have led to an extremely limited set of
benchmarks being executed, of questionable value.

We need expressiveness that is sufficient to also describe realistic
scenarios, like out of order or biased delivery of requests (such was shown
to be critical at ORNL), mixing in smaller and bigger I/O.  I believe what
Umka and Alexey are showing is the beginning of the language we are
developing for this.

peter

On Thu, Dec 10, 2009 at 12:33 AM, Andreas Dilger <adilger@sun.com> wrote:

> On 2009-12-08, at 23:57, Alexey Lyashkov wrote:
> > On Tue, 2009-12-08 at 23:35 -0700, Andreas Dilger wrote:
> >> On 2009-12-08, at 22:07, Alexey Lyashkov wrote:
> >>> This can be something similar as
> >>
> >>>
> >>>    pattern $name $pool_name $operation [$stripe_size
> >>> [$stripe_count]]
> >>
> >> Please do not use positional parameters.  That makes it nearly
> >> impossible to change the input format in the future, or to skip some
> >> fields and leave them as defaults.  This needs to use getopt() and
> >> named parameters:
> >>
> >>  --pool=pool_name, --stripe_size=N,  --stripe_count=N
> >
> > This possible, but unlikely. because need will be save current pool
> > name, instead have parsed as single command.
> >
> > And I don't have plan to rewrite loadgen to use getopt, instead of
> > current Parser() function which is used in lctl, lfs, and other.
>
> Note that the existing parser function passes argc, argv to the helper
> function, which is free to use getopt() or getopt_long() internally as
> desired.  See, for example, lustre/utils/lfs.c::lfs_setstripe(), which
> handles the parsing of "lfs setstripe --count N --pool {pool} ...".
>
> > How about this style
> >>
> > pool $name { targets:"OST1, OST2" }
> > pattern $pat { stripe_count:X stripe_size:Y block_size:Z };
> > clients $pattern { count:XX pattern: pat shared }
>
>
> I have no objection to splitting up the functionality into multiple
> sub-commands.  Allowing declarations of pools and patterns that are re-
> used for each client definition makes a lot of sense.
>
> What I dislike is the introduction of a completely new input format
> and parsing engine for no apparent benefit.  Why not change the above
> to:
>
> loadgen> pool --targets="OST1,OST2,..." NAME
> loadgen> pattern --stripe_count=X --stripe_size=Y --block_size=Z PATTERN
> loadgen> clients --count=XX --pattern=PATTERN ...
>
> and use well-known input conventions?
>
> It also makes sense to look at how LNET Self Test handles the setup of
> testing loads, and try and make the loadgen usage similar to that, as
> much as possible?
>
> Cheers, Andreas
> --
> Andreas Dilger
> Sr. Staff Engineer, Lustre Group
> Sun Microsystems of Canada, Inc.
>
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20091210/b9b18e1b/attachment.htm>

[Lustre-devel] loadgen improvements

[Yuriy Umanets]

On Tue, Dec 8, 2009 at 9:10 PM, Andreas Dilger <adilger@sun.com> wrote:

> Hello Yuriy,
>
>
> On 2009-12-05, at 06:21, Yuriy Umanets wrote:
>
>> ClusterStor would like to start doing real contributions to Lustre
>> stability and popularity. First off, we would like to improve loadgen
>> utility, written by Nathan Rutman. We think loadgen is a really good tool
>> for testing and what is not less important, it is implemented right way from
>> architecture point of view.
>>
>
> Great.  We look forward to working with you again.
>
> Thank you all guys.

>
>  However, these days it has number of issues:
>> 1. Wrong stack size for threads, that results in segfault (find the patch
>> in attachment);
>> 2. Little locking issues (push_kid() function);
>> 3. Absence of striping functionality, it only may create load on one
>> OST/ECHO server.
>>
>
> Can you please file a bug for this, and attach this patch and later ones
> there.  That will ensure that it follows the proper inspection and testing
> process.


Sure, will do.

>
>
>  We plan to add striping functionality to loadgen. General idea is to use
>> LOV module, which already has all the needed code. Loadgen itself and
>> echo_client also have related code, so it seems this functionality was
>> planned anyway, though it's not working currently. Currently an attempt to
>> attach to a LOV results in assertion failures.
>>
>
> Right - Eric and I worked on that some years ago, but it was never used
> fully.  Can you please ensure that you include a test in sanity.sh that
> exercises the functionality you are adding to loadgen, as with echo_client
> in test 180.  More complex load tests should go into their own script.
>
> Ok

>
>  Let's discuss these matters. The way we're going to implement this may be
>> roughly expressed as follows:
>>
>> 1. Attach to LOV device in loadgen, using "device" command. To do so we
>> need to construct new LOV instance, used by loadgen only, as we cannot use
>> LOV instance used by LLITE. This requires changes to handling function for
>> command "device". It should accept more than one OST target;
>>
>
> This sounds reasonable.  It might be useful to support the wildcard
> specification of OSTs like "lustre-OST00[0-30]" or "lustre-OST00[0,3,6,9].
>  Some of that functionality already exists in lustre/utils/nidlist.c.


Looks reasonable. We can even use "dl" command output to validate the names
and prevent using not existing local obd names.

>
>
>  2. Stripe size and stripe count of new LOV instance should also be
>> specified while constructing it using "device" command;
>>
>
> It probably makes sense to have this specified with a separate command, so
> that these parameters can be changed without having to tear down the devices
> and recreate them just to change the striping, and it avoids overloading the
> "device" command (which will soon become much more complex by allowing many
> OSTs to be specified).


Tend to agree with Shadow's idea to use something like a "pool" command
instead of --stripe_count to minimize changes.

>
>
>  3. Setup of virtual clients should take into account that LOV may be used
>> as a target. In this case attaching dedicated OSCs to created echo_client's
>> is not needed.
>>
>> 4. In order to be able to run thousands of threads, we need the following:
>>  - reasonable stack size for threads;
>>  - change 8192 OBD number limit to something more reasonable.
>>
>
> If you are going beyond 8192 OBD devices just for the loadgen usage
> (presumably you need a new LOV device for each thread?) then it would make
> sense to have the obd_devs array be dynamically allocated and remove any
> upper limit on the number of devices.
>
> Event for current functionality, which we want to preserve, when we have 2
OBDs per thread (echo_client and its OSC) this limit will not allow us to
start more than ?4K threads. I think our goal should be at least 100K of
them.


> Most of this work has already been done - all of the access to obd_devs[]
> is localized in lustre/obdclass/genops.c, with the exception of the array
> initialization, and an old EXPORT_SYMBOL(obd_devs), which isn't used
> anymore. The obd_devs[] declaration can be made static to genops.c and the
> initialization moved to obd_init_caches(), and then the actual
> representation of obd_devs[] can be changed without affecting the interface.
>
Agreed.

>
> Note that the index number of the device is still exported to userspace, so
> it isn't possible/practical to simply go with a linked-list implementation.
>  The actual obd_device structures are already allocated and freed
> dynamically, just the master obd_devs[] array is static.  Probably a master
> array of pointers to pages that hold segments of the the obd_devs[] array in
> order is sufficient.  The master array will be quite small (1 pointer per
> 512 obd_devices) and can be reallocated under a rwlock, since nothing
> references it directly.  There is no need to be able to shrink the master
> array.
>
> It would be trivial to also make the obd_types and obd_types_lock
> declarations static to genops.c, and the initialization in obd_init_caches()
> to properly isolate that code as well.
>
> Ok

>
>  5. Load itself is created in the way being currently used - via obd_brw().
>>
>
>
> Actually, the real code does not use obd_brw() directly anymore, but rather
> obd_brw_async().  I think Nic Henke submitted a patch to change that, I'm
> not sure of the current status of that patch.
>
> Ok.

Thanks!

> Cheers, Andreas
> --
> Andreas Dilger
> Sr. Staff Engineer, Lustre Group
> Sun Microsystems of Canada, Inc.
>
>


-- 
umka
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20091209/844d67ad/attachment.htm>

[Lustre-devel] loadgen improvements

[Yuriy Umanets]

On Tue, Dec 8, 2009 at 9:10 PM, Andreas Dilger <adilger@sun.com> wrote:

> Hello Yuriy,
>


> Can you please file a bug for this, and attach this patch and later ones
> there.  That will ensure that it follows the proper inspection and testing
> process.
>
>
> Bug https://bugzilla.lustre.org/show_bug.cgi?id=21571 is created. Please
attach it to right tracker. I'm not aware of any valid these days.

Nathan, Andreas, I requested your inspections there. Hope you don't mind.

Thanks.


>
>> --
umka
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20091209/a5305147/attachment.htm>

[Lustre-devel] loadgen improvements

[Yuriy Umanets]

hello Lustre hackers,

ClusterStor would like to start doing real contributions to Lustre stability
and popularity. First off, we would like to improve loadgen utility, written
by Nathan Rutman. We think loadgen is a really good tool for testing and
what is not less important, it is implemented right way from architecture
point of view.

However, these days it has number of issues:
1. Wrong stack size for threads, that results in segfault (find the patch in
attachment);
2. Little locking issues (push_kid() function);
3. Absence of striping functionality, it only may create load on one
OST/ECHO server.

We plan to add striping functionality to loadgen. General idea is to use LOV
module, which already has all the needed code. Loadgen itself and
echo_client also have related code, so it seems this functionality was
planned anyway, though it's not working currently. Currently an attempt to
attach to a LOV results in assertion failures.

Let's discuss these matters. The way we're going to implement this may be
roughly expressed as follows:

1. Attach to LOV device in loadgen, using "device" command. To do so we need
to construct new LOV instance, used by loadgen only, as we cannot use LOV
instance used by LLITE. This requires changes to handling function for
command "device". It should accept more than one OST target;

2. Stripe size and stripe count of new LOV instance should also be specified
while constructing it using "device" command;

3. Setup of virtual clients should take into account that LOV may be used as
a target. In this case attaching dedicated OSCs to created echo_client's is
not needed.

4. In order to be able to run thousands of threads, we need the following:
 - reasonable stack size for threads;
 - change 8192 OBD number limit to something more reasonable.

5. Load itself is created in the way being currently used - via obd_brw().

Thanks.


-- 
umka
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20091208/a730c324/attachment.htm>