* [Lustre-devel] Control file striping @ 2012-11-03 17:41 Kalyana C 2012-11-05 17:47 ` Atchley, Scott 0 siblings, 1 reply; 10+ messages in thread From: Kalyana C @ 2012-11-03 17:41 UTC (permalink / raw) To: lustre-devel Hello, I am trying to figure out a way to: set striping across a set of OSTs of my choice. lfs setstripe command and API call does not allow for this currently. The user can only set the starting OST through the index parameter. I appreciate why there is no such feature currently; you do not want to let the users have control on how OSTs are used since it can result in OST usage imbalance. However, I do have a valid usage case for such a feature and I am trying to figure out the best way to do it. One way to achieve the same result is through the use of OST pools. However, the userland tools do not work from a client node, it has to be run on the MGS. There are also other restrictions as described here: http://wiki.lustre.org/index.php/Creating_and_Managing_OST_Pools I am also digging through the Lustre source and header files to see if there are other (undocumented) ways to do this. So, my questions are as follows: 1) Is there currently a way for a user to control the striping of a file / directory on a selected set of OSTs? If yes, is it possible to do it through an API call or a command line utility? 2) If answer to (1) is No, Is there a way to do this with admin rights from the MDS or client nodes? 3) If pools are the only way to achieve this, is there an API call to create, manage and destroy pools from a user application? Is there any impact on performance of MDS components if pools are dynamically created and deleted on demand? Can the API be called from a client node (root or user) or does it have to be on the MGS? Thank you. Regards, Kalyana Chadalavada -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20121103/1884bff0/attachment.htm> ^ permalink raw reply [flat|nested] 10+ messages in thread
* [Lustre-devel] Control file striping 2012-11-03 17:41 [Lustre-devel] Control file striping Kalyana C @ 2012-11-05 17:47 ` Atchley, Scott 2012-11-05 17:57 ` Ben Evans 0 siblings, 1 reply; 10+ messages in thread From: Atchley, Scott @ 2012-11-05 17:47 UTC (permalink / raw) To: lustre-devel On Nov 3, 2012, at 1:41 PM, Kalyana C <ckkccf@gmail.com> wrote: > Hello, > > I am trying to figure out a way to: > > set striping across a set of OSTs of my choice. > > lfs setstripe command and API call does not allow for this currently. The user can only set the starting OST through the index parameter. I appreciate why there is no such feature currently; you do not want to let the users have control on how OSTs are used since it can result in OST usage imbalance. > > However, I do have a valid usage case for such a feature and I am trying to figure out the best way to do it. I have a valid use for it as well. It would be nice to have this feature back. Scott > > One way to achieve the same result is through the use of OST pools. However, the userland tools do not work from a client node, it has to be run on the MGS. There are also other restrictions as described here: http://wiki.lustre.org/index.php/Creating_and_Managing_OST_Pools > > I am also digging through the Lustre source and header files to see if there are other (undocumented) ways to do this. > > So, my questions are as follows: > > 1) Is there currently a way for a user to control the striping of a file / directory on a selected set of OSTs? If yes, is it possible to do it through an API call or a command line utility? > > 2) If answer to (1) is No, Is there a way to do this with admin rights from the MDS or client nodes? > > 3) If pools are the only way to achieve this, is there an API call to create, manage and destroy pools from a user application? Is there any impact on performance of MDS components if pools are dynamically created and deleted on demand? Can the API be called from a client node (root or user) or does it have to be on the MGS? > > Thank you. > > Regards, > Kalyana Chadalavada > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel ^ permalink raw reply [flat|nested] 10+ messages in thread
* [Lustre-devel] Control file striping 2012-11-05 17:47 ` Atchley, Scott @ 2012-11-05 17:57 ` Ben Evans 2012-11-05 18:01 ` Kalyana C 2012-11-05 18:42 ` Christopher J. Morrone 0 siblings, 2 replies; 10+ messages in thread From: Ben Evans @ 2012-11-05 17:57 UTC (permalink / raw) To: lustre-devel Have you considered using a combination of OST Pools and striping fit your needs? -----Original Message----- From: lustre-devel-bounces@lists.lustre.org [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of Atchley, Scott Sent: Monday, November 05, 2012 12:47 PM To: Kalyana C Cc: lustre-devel at lists.lustre.org Subject: Re: [Lustre-devel] Control file striping On Nov 3, 2012, at 1:41 PM, Kalyana C <ckkccf@gmail.com> wrote: > Hello, > > I am trying to figure out a way to: > > set striping across a set of OSTs of my choice. > > lfs setstripe command and API call does not allow for this currently. The user can only set the starting OST through the index parameter. I appreciate why there is no such feature currently; you do not want to let the users have control on how OSTs are used since it can result in OST usage imbalance. > > However, I do have a valid usage case for such a feature and I am trying to figure out the best way to do it. I have a valid use for it as well. It would be nice to have this feature back. Scott > > One way to achieve the same result is through the use of OST pools. However, the userland tools do not work from a client node, it has to be run on the MGS. There are also other restrictions as described here: http://wiki.lustre.org/index.php/Creating_and_Managing_OST_Pools > > I am also digging through the Lustre source and header files to see if there are other (undocumented) ways to do this. > > So, my questions are as follows: > > 1) Is there currently a way for a user to control the striping of a file / directory on a selected set of OSTs? If yes, is it possible to do it through an API call or a command line utility? > > 2) If answer to (1) is No, Is there a way to do this with admin rights from the MDS or client nodes? > > 3) If pools are the only way to achieve this, is there an API call to create, manage and destroy pools from a user application? Is there any impact on performance of MDS components if pools are dynamically created and deleted on demand? Can the API be called from a client node (root or user) or does it have to be on the MGS? > > Thank you. > > Regards, > Kalyana Chadalavada > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel _______________________________________________ Lustre-devel mailing list Lustre-devel at lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-devel ^ permalink raw reply [flat|nested] 10+ messages in thread
* [Lustre-devel] Control file striping 2012-11-05 17:57 ` Ben Evans @ 2012-11-05 18:01 ` Kalyana C 2012-11-05 18:16 ` White, Cliff 2012-11-05 18:42 ` Christopher J. Morrone 1 sibling, 1 reply; 10+ messages in thread From: Kalyana C @ 2012-11-05 18:01 UTC (permalink / raw) To: lustre-devel > Have you considered using a combination of OST Pools and striping fit your needs? This is the only way possible that I currently know of. But is it possible to manage OST pools via an API call? Can it be done from a client under a normal user application? Or does it have the same restrictions as described at http://wiki.lustre.org/index.php/Creating_and_Managing_OST_Pools Regards, Kalyan On Mon, Nov 5, 2012 at 11:57 AM, Ben Evans <Ben.Evans@terascala.com> wrote: > Have you considered using a combination of OST Pools and striping fit > your needs? > > -----Original Message----- > From: lustre-devel-bounces at lists.lustre.org > [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of Atchley, > Scott > Sent: Monday, November 05, 2012 12:47 PM > To: Kalyana C > Cc: lustre-devel at lists.lustre.org > Subject: Re: [Lustre-devel] Control file striping > > On Nov 3, 2012, at 1:41 PM, Kalyana C <ckkccf@gmail.com> wrote: > > > Hello, > > > > I am trying to figure out a way to: > > > > set striping across a set of OSTs of my choice. > > > > lfs setstripe command and API call does not allow for this currently. > The user can only set the starting OST through the index parameter. I > appreciate why there is no such feature currently; you do not want to > let the users have control on how OSTs are used since it can result in > OST usage imbalance. > > > > However, I do have a valid usage case for such a feature and I am > trying to figure out the best way to do it. > > I have a valid use for it as well. It would be nice to have this feature > back. > > Scott > > > > > One way to achieve the same result is through the use of OST pools. > However, the userland tools do not work from a client node, it has to be > run on the MGS. There are also other restrictions as described here: > http://wiki.lustre.org/index.php/Creating_and_Managing_OST_Pools > > > > I am also digging through the Lustre source and header files to see if > there are other (undocumented) ways to do this. > > > > So, my questions are as follows: > > > > 1) Is there currently a way for a user to control the striping of a > file / directory on a selected set of OSTs? If yes, is it possible to do > it through an API call or a command line utility? > > > > 2) If answer to (1) is No, Is there a way to do this with admin rights > from the MDS or client nodes? > > > > 3) If pools are the only way to achieve this, is there an API call to > create, manage and destroy pools from a user application? Is there any > impact on performance of MDS components if pools are dynamically created > and deleted on demand? Can the API be called from a client node (root or > user) or does it have to be on the MGS? > > > > Thank you. > > > > Regards, > > Kalyana Chadalavada > > > > _______________________________________________ > > Lustre-devel mailing list > > Lustre-devel at lists.lustre.org > > http://lists.lustre.org/mailman/listinfo/lustre-devel > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20121105/3f279f1b/attachment.htm> ^ permalink raw reply [flat|nested] 10+ messages in thread
* [Lustre-devel] Control file striping 2012-11-05 18:01 ` Kalyana C @ 2012-11-05 18:16 ` White, Cliff 2012-11-05 18:28 ` Atchley, Scott 2012-11-05 18:28 ` Kalyana C 0 siblings, 2 replies; 10+ messages in thread From: White, Cliff @ 2012-11-05 18:16 UTC (permalink / raw) To: lustre-devel Creating/destroying pools must be done as per the manual, from MDS. Do you need to frequently create/destroy pools, or would you have a fixed setup? The normal method is to map various pools (and other striping policies) to directories, then user space applications can access specific pools by pointing at the requisite directory, thus requiring no special API. Once pools are created, the directory creation/striping is all done on a client via lfs setstripe, the liblustreapi can also be used by clients to set striping via C calls. Best cliffw On Nov 5, 2012, at 10:01 AM, Kalyana C <ckkccf at gmail.com<mailto:ckkccf@gmail.com>> wrote: > Have you considered using a combination of OST Pools and striping fit your needs? This is the only way possible that I currently know of. But is it possible to manage OST pools via an API call? Can it be done from a client under a normal user application? Or does it have the same restrictions as described at http://wiki.lustre.org/index.php/Creating_and_Managing_OST_Pools Regards, Kalyan On Mon, Nov 5, 2012 at 11:57 AM, Ben Evans <Ben.Evans at terascala.com<mailto:Ben.Evans@terascala.com>> wrote: Have you considered using a combination of OST Pools and striping fit your needs? -----Original Message----- From: lustre-devel-bounces@lists.lustre.org<mailto:lustre-devel-bounces@lists.lustre.org> [mailto:lustre-devel-bounces at lists.lustre.org<mailto:lustre-devel-bounces@lists.lustre.org>] On Behalf Of Atchley, Scott Sent: Monday, November 05, 2012 12:47 PM To: Kalyana C Cc: lustre-devel at lists.lustre.org<mailto:lustre-devel@lists.lustre.org> Subject: Re: [Lustre-devel] Control file striping On Nov 3, 2012, at 1:41 PM, Kalyana C <ckkccf at gmail.com<mailto:ckkccf@gmail.com>> wrote: > Hello, > > I am trying to figure out a way to: > > set striping across a set of OSTs of my choice. > > lfs setstripe command and API call does not allow for this currently. The user can only set the starting OST through the index parameter. I appreciate why there is no such feature currently; you do not want to let the users have control on how OSTs are used since it can result in OST usage imbalance. > > However, I do have a valid usage case for such a feature and I am trying to figure out the best way to do it. I have a valid use for it as well. It would be nice to have this feature back. Scott > > One way to achieve the same result is through the use of OST pools. However, the userland tools do not work from a client node, it has to be run on the MGS. There are also other restrictions as described here: http://wiki.lustre.org/index.php/Creating_and_Managing_OST_Pools > > I am also digging through the Lustre source and header files to see if there are other (undocumented) ways to do this. > > So, my questions are as follows: > > 1) Is there currently a way for a user to control the striping of a file / directory on a selected set of OSTs? If yes, is it possible to do it through an API call or a command line utility? > > 2) If answer to (1) is No, Is there a way to do this with admin rights from the MDS or client nodes? > > 3) If pools are the only way to achieve this, is there an API call to create, manage and destroy pools from a user application? Is there any impact on performance of MDS components if pools are dynamically created and deleted on demand? Can the API be called from a client node (root or user) or does it have to be on the MGS? > > Thank you. > > Regards, > Kalyana Chadalavada > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org<mailto:Lustre-devel@lists.lustre.org> > http://lists.lustre.org/mailman/listinfo/lustre-devel _______________________________________________ Lustre-devel mailing list Lustre-devel at lists.lustre.org<mailto:Lustre-devel@lists.lustre.org> http://lists.lustre.org/mailman/listinfo/lustre-devel _______________________________________________ Lustre-devel mailing list Lustre-devel at lists.lustre.org<mailto:Lustre-devel@lists.lustre.org> http://lists.lustre.org/mailman/listinfo/lustre-devel ^ permalink raw reply [flat|nested] 10+ messages in thread
* [Lustre-devel] Control file striping 2012-11-05 18:16 ` White, Cliff @ 2012-11-05 18:28 ` Atchley, Scott 2012-11-05 18:28 ` Kalyana C 1 sibling, 0 replies; 10+ messages in thread From: Atchley, Scott @ 2012-11-05 18:28 UTC (permalink / raw) To: lustre-devel On Nov 5, 2012, at 1:16 PM, "White, Cliff" <cliff.white@intel.com> wrote: > Creating/destroying pools must be done as per the manual, from MDS. > Do you need to frequently create/destroy pools, or would you have a fixed setup? In our case, we need to frequently change the OSTs. Possibly per file. Scott > > The normal method is to map various pools (and other striping policies) to directories, then user space applications > can access specific pools by pointing at the requisite directory, thus requiring no special API. > Once pools are created, the directory creation/striping is all done on a client via lfs setstripe, the liblustreapi can > also be used by clients to set striping via C calls. > > Best > cliffw > > > On Nov 5, 2012, at 10:01 AM, Kalyana C <ckkccf at gmail.com<mailto:ckkccf@gmail.com>> > wrote: > >> Have you considered using a combination of OST Pools and striping fit your needs? > > This is the only way possible that I currently know of. > > But is it possible to manage OST pools via an API call? Can it be done from a client under a normal user application? Or does it have the same restrictions as described at http://wiki.lustre.org/index.php/Creating_and_Managing_OST_Pools > > Regards, > Kalyan > > On Mon, Nov 5, 2012 at 11:57 AM, Ben Evans <Ben.Evans at terascala.com<mailto:Ben.Evans@terascala.com>> wrote: > Have you considered using a combination of OST Pools and striping fit > your needs? > > -----Original Message----- > From: lustre-devel-bounces at lists.lustre.org<mailto:lustre-devel-bounces@lists.lustre.org> > [mailto:lustre-devel-bounces at lists.lustre.org<mailto:lustre-devel-bounces@lists.lustre.org>] On Behalf Of Atchley, > Scott > Sent: Monday, November 05, 2012 12:47 PM > To: Kalyana C > Cc: lustre-devel at lists.lustre.org<mailto:lustre-devel@lists.lustre.org> > Subject: Re: [Lustre-devel] Control file striping > > On Nov 3, 2012, at 1:41 PM, Kalyana C <ckkccf at gmail.com<mailto:ckkccf@gmail.com>> wrote: > >> Hello, >> >> I am trying to figure out a way to: >> >> set striping across a set of OSTs of my choice. >> >> lfs setstripe command and API call does not allow for this currently. > The user can only set the starting OST through the index parameter. I > appreciate why there is no such feature currently; you do not want to > let the users have control on how OSTs are used since it can result in > OST usage imbalance. >> >> However, I do have a valid usage case for such a feature and I am > trying to figure out the best way to do it. > > I have a valid use for it as well. It would be nice to have this feature > back. > > Scott > >> >> One way to achieve the same result is through the use of OST pools. > However, the userland tools do not work from a client node, it has to be > run on the MGS. There are also other restrictions as described here: > http://wiki.lustre.org/index.php/Creating_and_Managing_OST_Pools >> >> I am also digging through the Lustre source and header files to see if > there are other (undocumented) ways to do this. >> >> So, my questions are as follows: >> >> 1) Is there currently a way for a user to control the striping of a > file / directory on a selected set of OSTs? If yes, is it possible to do > it through an API call or a command line utility? >> >> 2) If answer to (1) is No, Is there a way to do this with admin rights > from the MDS or client nodes? >> >> 3) If pools are the only way to achieve this, is there an API call to > create, manage and destroy pools from a user application? Is there any > impact on performance of MDS components if pools are dynamically created > and deleted on demand? Can the API be called from a client node (root or > user) or does it have to be on the MGS? >> >> Thank you. >> >> Regards, >> Kalyana Chadalavada >> >> _______________________________________________ >> Lustre-devel mailing list >> Lustre-devel at lists.lustre.org<mailto:Lustre-devel@lists.lustre.org> >> http://lists.lustre.org/mailman/listinfo/lustre-devel > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org<mailto:Lustre-devel@lists.lustre.org> > http://lists.lustre.org/mailman/listinfo/lustre-devel > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org<mailto:Lustre-devel@lists.lustre.org> > http://lists.lustre.org/mailman/listinfo/lustre-devel > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel ^ permalink raw reply [flat|nested] 10+ messages in thread
* [Lustre-devel] Control file striping 2012-11-05 18:16 ` White, Cliff 2012-11-05 18:28 ` Atchley, Scott @ 2012-11-05 18:28 ` Kalyana C 1 sibling, 0 replies; 10+ messages in thread From: Kalyana C @ 2012-11-05 18:28 UTC (permalink / raw) To: lustre-devel I will need to setup striping on specific OSTs on demand. Its not a fixed setup. Here is what I am thinking, with my current understanding: *) user app sends the details (OST ids, name of the pool) to a listener on MGS *) a process on MGS executes the command and sends back an ack *) user app sets up striping on the newly created pool *) pool is deleted once striping is set I was hoping to find out if there was a better (and easier) way to do this. A simple usage case is as follows: There are a mix of disks with varying performance in the file system. Depending on the application, I would like to change the disks used for that instance. Say, 10% spindle, 90% SSD, 50-50 for another etc. The same can be applied to a mix of storage devices in the same file system - disks with varying levels of redundancy or even varying devices. Regards, Kalyan On Mon, Nov 5, 2012 at 12:16 PM, White, Cliff <cliff.white@intel.com> wrote: > Creating/destroying pools must be done as per the manual, from MDS. > Do you need to frequently create/destroy pools, or would you have a fixed > setup? > > The normal method is to map various pools (and other striping policies) > to directories, then user space applications > can access specific pools by pointing at the requisite directory, thus > requiring no special API. > Once pools are created, the directory creation/striping is all done on a > client via lfs setstripe, the liblustreapi can > also be used by clients to set striping via C calls. > > Best > cliffw > > > On Nov 5, 2012, at 10:01 AM, Kalyana C <ckkccf@gmail.com> > wrote: > > > Have you considered using a combination of OST Pools and striping > fit your needs? > > This is the only way possible that I currently know of. > > But is it possible to manage OST pools via an API call? Can it be done > from a client under a normal user application? Or does it have the same > restrictions as described at > http://wiki.lustre.org/index.php/Creating_and_Managing_OST_Pools > > Regards, > Kalyan > > On Mon, Nov 5, 2012 at 11:57 AM, Ben Evans <Ben.Evans@terascala.com>wrote: > >> Have you considered using a combination of OST Pools and striping fit >> your needs? >> >> -----Original Message----- >> From: lustre-devel-bounces at lists.lustre.org >> [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of Atchley, >> Scott >> Sent: Monday, November 05, 2012 12:47 PM >> To: Kalyana C >> Cc: lustre-devel at lists.lustre.org >> Subject: Re: [Lustre-devel] Control file striping >> >> On Nov 3, 2012, at 1:41 PM, Kalyana C <ckkccf@gmail.com> wrote: >> >> > Hello, >> > >> > I am trying to figure out a way to: >> > >> > set striping across a set of OSTs of my choice. >> > >> > lfs setstripe command and API call does not allow for this currently. >> The user can only set the starting OST through the index parameter. I >> appreciate why there is no such feature currently; you do not want to >> let the users have control on how OSTs are used since it can result in >> OST usage imbalance. >> > >> > However, I do have a valid usage case for such a feature and I am >> trying to figure out the best way to do it. >> >> I have a valid use for it as well. It would be nice to have this feature >> back. >> >> Scott >> >> > >> > One way to achieve the same result is through the use of OST pools. >> However, the userland tools do not work from a client node, it has to be >> run on the MGS. There are also other restrictions as described here: >> http://wiki.lustre.org/index.php/Creating_and_Managing_OST_Pools >> > >> > I am also digging through the Lustre source and header files to see if >> there are other (undocumented) ways to do this. >> > >> > So, my questions are as follows: >> > >> > 1) Is there currently a way for a user to control the striping of a >> file / directory on a selected set of OSTs? If yes, is it possible to do >> it through an API call or a command line utility? >> > >> > 2) If answer to (1) is No, Is there a way to do this with admin rights >> from the MDS or client nodes? >> > >> > 3) If pools are the only way to achieve this, is there an API call to >> create, manage and destroy pools from a user application? Is there any >> impact on performance of MDS components if pools are dynamically created >> and deleted on demand? Can the API be called from a client node (root or >> user) or does it have to be on the MGS? >> > >> > Thank you. >> > >> > Regards, >> > Kalyana Chadalavada >> > >> > _______________________________________________ >> > Lustre-devel mailing list >> > Lustre-devel at lists.lustre.org >> > http://lists.lustre.org/mailman/listinfo/lustre-devel >> >> _______________________________________________ >> Lustre-devel mailing list >> Lustre-devel at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-devel >> > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20121105/86822b0b/attachment.htm> ^ permalink raw reply [flat|nested] 10+ messages in thread
* [Lustre-devel] Control file striping 2012-11-05 17:57 ` Ben Evans 2012-11-05 18:01 ` Kalyana C @ 2012-11-05 18:42 ` Christopher J. Morrone 2012-11-05 19:00 ` Atchley, Scott 1 sibling, 1 reply; 10+ messages in thread From: Christopher J. Morrone @ 2012-11-05 18:42 UTC (permalink / raw) To: lustre-devel Pools only address one kind of problem: constraining object allocation to a set of OSTs. But what if the user wants to guarantee that their files' objects are evenly distributed across the OSTs, whether in a pool or not? Even a slight unevenness in object allocation (one OST getting more objects than the others) can drastically change the apparent aggregate throughput that the user sees. Most of the applications that I am aware of essentially do: compute barrier write files barrier repeat So any unevenness of object allocation means that the entire application sits and waits for that one OST that is more heavily loaded than the rest. Which generally means for us that Lustre's object allocation algorithms that attempt to provide space-balancing have noticeably negative impacts on application performance. So I agree that I would like to see the set-stripe API and wire protocols expanded to allow specifying all of the desired object indices, not just the first one. It would probably be good to make the use of the ability to manually set object locations permission-based, and allow the sysadmin to decide if users can use that functionality. Best would be if they could set it on a per-user basis, but I'd settle for a single large on/off switch. Chris On 11/05/2012 09:57 AM, Ben Evans wrote: > Have you considered using a combination of OST Pools and striping fit > your needs? > > -----Original Message----- > From: lustre-devel-bounces at lists.lustre.org > [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of Atchley, > Scott > Sent: Monday, November 05, 2012 12:47 PM > To: Kalyana C > Cc: lustre-devel at lists.lustre.org > Subject: Re: [Lustre-devel] Control file striping > > On Nov 3, 2012, at 1:41 PM, Kalyana C <ckkccf@gmail.com> wrote: > >> Hello, >> >> I am trying to figure out a way to: >> >> set striping across a set of OSTs of my choice. >> >> lfs setstripe command and API call does not allow for this currently. > The user can only set the starting OST through the index parameter. I > appreciate why there is no such feature currently; you do not want to > let the users have control on how OSTs are used since it can result in > OST usage imbalance. >> >> However, I do have a valid usage case for such a feature and I am > trying to figure out the best way to do it. > > I have a valid use for it as well. It would be nice to have this feature > back. > > Scott > >> >> One way to achieve the same result is through the use of OST pools. > However, the userland tools do not work from a client node, it has to be > run on the MGS. There are also other restrictions as described here: > http://wiki.lustre.org/index.php/Creating_and_Managing_OST_Pools >> >> I am also digging through the Lustre source and header files to see if > there are other (undocumented) ways to do this. >> >> So, my questions are as follows: >> >> 1) Is there currently a way for a user to control the striping of a > file / directory on a selected set of OSTs? If yes, is it possible to do > it through an API call or a command line utility? >> >> 2) If answer to (1) is No, Is there a way to do this with admin rights > from the MDS or client nodes? >> >> 3) If pools are the only way to achieve this, is there an API call to > create, manage and destroy pools from a user application? Is there any > impact on performance of MDS components if pools are dynamically created > and deleted on demand? Can the API be called from a client node (root or > user) or does it have to be on the MGS? >> >> Thank you. >> >> Regards, >> Kalyana Chadalavada >> >> _______________________________________________ >> Lustre-devel mailing list >> Lustre-devel at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-devel > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > . > ^ permalink raw reply [flat|nested] 10+ messages in thread
* [Lustre-devel] Control file striping 2012-11-05 18:42 ` Christopher J. Morrone @ 2012-11-05 19:00 ` Atchley, Scott 0 siblings, 0 replies; 10+ messages in thread From: Atchley, Scott @ 2012-11-05 19:00 UTC (permalink / raw) To: lustre-devel On Nov 5, 2012, at 1:42 PM, Christopher J. Morrone <morrone2@llnl.gov> wrote: > Pools only address one kind of problem: constraining object allocation > to a set of OSTs. > > But what if the user wants to guarantee that their files' objects are > evenly distributed across the OSTs, whether in a pool or not? Even a > slight unevenness in object allocation (one OST getting more objects > than the others) can drastically change the apparent aggregate > throughput that the user sees. > > Most of the applications that I am aware of essentially do: > > compute > barrier > write files > barrier > repeat > > So any unevenness of object allocation means that the entire application > sits and waits for that one OST that is more heavily loaded than the rest. > > Which generally means for us that Lustre's object allocation algorithms > that attempt to provide space-balancing have noticeably negative impacts > on application performance. > > So I agree that I would like to see the set-stripe API and wire > protocols expanded to allow specifying all of the desired object > indices, not just the first one. > > It would probably be good to make the use of the ability to manually set > object locations permission-based, and allow the sysadmin to decide if > users can use that functionality. Best would be if they could set it on > a per-user basis, but I'd settle for a single large on/off switch. > > Chris Ditto. Scott > > On 11/05/2012 09:57 AM, Ben Evans wrote: >> Have you considered using a combination of OST Pools and striping fit >> your needs? >> >> -----Original Message----- >> From: lustre-devel-bounces at lists.lustre.org >> [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of Atchley, >> Scott >> Sent: Monday, November 05, 2012 12:47 PM >> To: Kalyana C >> Cc: lustre-devel at lists.lustre.org >> Subject: Re: [Lustre-devel] Control file striping >> >> On Nov 3, 2012, at 1:41 PM, Kalyana C <ckkccf@gmail.com> wrote: >> >>> Hello, >>> >>> I am trying to figure out a way to: >>> >>> set striping across a set of OSTs of my choice. >>> >>> lfs setstripe command and API call does not allow for this currently. >> The user can only set the starting OST through the index parameter. I >> appreciate why there is no such feature currently; you do not want to >> let the users have control on how OSTs are used since it can result in >> OST usage imbalance. >>> >>> However, I do have a valid usage case for such a feature and I am >> trying to figure out the best way to do it. >> >> I have a valid use for it as well. It would be nice to have this feature >> back. >> >> Scott >> >>> >>> One way to achieve the same result is through the use of OST pools. >> However, the userland tools do not work from a client node, it has to be >> run on the MGS. There are also other restrictions as described here: >> http://wiki.lustre.org/index.php/Creating_and_Managing_OST_Pools >>> >>> I am also digging through the Lustre source and header files to see if >> there are other (undocumented) ways to do this. >>> >>> So, my questions are as follows: >>> >>> 1) Is there currently a way for a user to control the striping of a >> file / directory on a selected set of OSTs? If yes, is it possible to do >> it through an API call or a command line utility? >>> >>> 2) If answer to (1) is No, Is there a way to do this with admin rights >> from the MDS or client nodes? >>> >>> 3) If pools are the only way to achieve this, is there an API call to >> create, manage and destroy pools from a user application? Is there any >> impact on performance of MDS components if pools are dynamically created >> and deleted on demand? Can the API be called from a client node (root or >> user) or does it have to be on the MGS? >>> >>> Thank you. >>> >>> Regards, >>> Kalyana Chadalavada >>> >>> _______________________________________________ >>> Lustre-devel mailing list >>> Lustre-devel at lists.lustre.org >>> http://lists.lustre.org/mailman/listinfo/lustre-devel >> >> _______________________________________________ >> Lustre-devel mailing list >> Lustre-devel at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-devel >> _______________________________________________ >> Lustre-devel mailing list >> Lustre-devel at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-devel >> . >> > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel ^ permalink raw reply [flat|nested] 10+ messages in thread
* [Lustre-devel] Control file striping
@ 2012-11-05 19:41 Kalyana C
0 siblings, 0 replies; 10+ messages in thread
From: Kalyana C @ 2012-11-05 19:41 UTC (permalink / raw)
To: lustre-devel
> So I agree that I would like to see the set-stripe API and wire
> protocols expanded to allow specifying all of the desired object
> indices, not just the first one.
> It would probably be good to make the use of the ability to manually set
> object locations permission-based, and allow the sysadmin to decide if
> users can use that functionality. Best would be if they could set it on
> a per-user basis, but I'd settle for a single large on/off switch.
This would be nice to have.
Regards,
Kalyan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20121105/58f8aed9/attachment.htm>
^ permalink raw reply [flat|nested] 10+ messages in threadend of thread, other threads:[~2012-11-05 19:41 UTC | newest] Thread overview: 10+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-11-03 17:41 [Lustre-devel] Control file striping Kalyana C 2012-11-05 17:47 ` Atchley, Scott 2012-11-05 17:57 ` Ben Evans 2012-11-05 18:01 ` Kalyana C 2012-11-05 18:16 ` White, Cliff 2012-11-05 18:28 ` Atchley, Scott 2012-11-05 18:28 ` Kalyana C 2012-11-05 18:42 ` Christopher J. Morrone 2012-11-05 19:00 ` Atchley, Scott -- strict thread matches above, loose matches on Subject: below -- 2012-11-05 19:41 Kalyana C
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.