* [lustre-devel] [lustre-discuss] more on lustre striping [not found] <1ef5a267-334c-be0d-13f4-c0fab917d1bf@iodoctors.com> @ 2016-05-21 5:51 ` Dilger, Andreas 2016-05-21 20:01 ` John Bauer 0 siblings, 1 reply; 13+ messages in thread From: Dilger, Andreas @ 2016-05-21 5:51 UTC (permalink / raw) To: lustre-devel This is probably getting to be more of a topic for lustre-devel. There currently isn't any way to do what you ask, since (IIRC) it will cause an error for apps that try to write to the files before the layout is set. What you could do is to create an LD_PRELOAD library to intercept the open() calls and set O_LOV_DELAY_CREATE and set the layout explicitly for each file. This might be a win if each file needs a different layout, but since it uses two RPCs per file it would be slower than using the default layout. Cheers, Andreas On May 18, 2016, at 16:46, John Bauer <bauerj at iodoctors.com<mailto:bauerj@iodoctors.com>> wrote: Since today's topic seems to be Lustre striping, I will revisit a previous line of questions I had. Andreas had put me on to O_LOV_DELAY_CREATE which I have been experimenting with. My question is : Is there a way to flag a directory with O_LOV_DELAY_CREATE so that a file created in that directory will be created with O_LOV_DELAY_CREATE also. Much like a file can inherit a directory's stripe count and stripe size, it would be convenient if a file could also inherit O_LOV_DELAY_CREATE? That way, for open()s that I can not intercept ( and thus can not set O_LOV_DELAY_CREATE in oflags) , such as those issued by fopen(), I can then get the fd with fileno() and set the striping with ioctl(fd, LL_IOC_LOV_SETSTRIPE, lum). Thanks John -- I/O Doctors, LLC 507-766-0378 bauerj at iodoctors.com<mailto:bauerj@iodoctors.com> _______________________________________________ lustre-discuss mailing list lustre-discuss at lists.lustre.org<mailto:lustre-discuss@lists.lustre.org> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20160521/de0cd052/attachment.htm> ^ permalink raw reply [flat|nested] 13+ messages in thread
* [lustre-devel] [lustre-discuss] more on lustre striping 2016-05-21 5:51 ` [lustre-devel] [lustre-discuss] more on lustre striping Dilger, Andreas @ 2016-05-21 20:01 ` John Bauer 2016-05-21 20:08 ` Drokin, Oleg 0 siblings, 1 reply; 13+ messages in thread From: John Bauer @ 2016-05-21 20:01 UTC (permalink / raw) To: lustre-devel Andreas, Thanks for the reply. For what it's worth, extending a file that does not have layout set does work. % *rm -f file.dat* % *./no_stripe.exe file.dat* fd=3 % *lfs getstripe file.dat* file.dat has no stripe info % *date >> file.dat* % *lfs getstripe file.dat* file.dat lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 21 obdidx objid objid group 21 6143298 0x5dbd42 0 % The LD_PRELOAD is exactly what I am doing in my I/O library. Unfortunately, one can not intercept the open() that results from a call to fopen(). That open is hard linked to the open in libc and not satisfied by the runtime linker. This is what is driving this topic for me. I can not conveniently set the striping for a file opened with fopen() and other functions where the open is called from inside libc. I used to believe that not too many application use stdio for heavy I/O, but I have been come across several recently. John On 5/21/2016 12:51 AM, Dilger, Andreas wrote: > This is probably getting to be more of a topic for lustre-devel. > > There currently isn't any way to do what you ask, since (IIRC) it will > cause an error for apps that try to write to the files before the > layout is set. > > What you could do is to create an LD_PRELOAD library to intercept the > open() calls and set O_LOV_DELAY_CREATE and set the layout explicitly > for each file. This might be a win if each file needs a different > layout, but since it uses two RPCs per file it would be slower than > using the default layout. > > Cheers, Andreas > > On May 18, 2016, at 16:46, John Bauer <bauerj@iodoctors.com > <mailto:bauerj@iodoctors.com>> wrote: > >> Since today's topic seems to be Lustre striping, I will revisit a >> previous line of questions I had. >> >> Andreas had put me on to O_LOV_DELAY_CREATE which I have been >> experimenting with. My question is : Is there a way to flag a >> directory with O_LOV_DELAY_CREATE so that a file created in that >> directory will be created with O_LOV_DELAY_CREATE also. Much like a >> file can inherit a directory's stripe count and stripe size, it would >> be convenient if a file could also inherit O_LOV_DELAY_CREATE? That >> way, for open()s that I can not intercept ( and thus can not set >> O_LOV_DELAY_CREATE in oflags) , such as those issued by fopen(), I >> can then get the fd with fileno() and set the striping withioctl(fd, >> LL_IOC_LOV_SETSTRIPE, lum). >> >> Thanks >> >> John >> >> -- >> I/O Doctors, LLC >> 507-766-0378 >> bauerj at iodoctors.com >> _______________________________________________ >> lustre-discuss mailing list >> lustre-discuss at lists.lustre.org <mailto:lustre-discuss@lists.lustre.org> >> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org -- I/O Doctors, LLC 507-766-0378 bauerj at iodoctors.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20160521/35b3264d/attachment-0001.htm> ^ permalink raw reply [flat|nested] 13+ messages in thread
* [lustre-devel] [lustre-discuss] more on lustre striping 2016-05-21 20:01 ` John Bauer @ 2016-05-21 20:08 ` Drokin, Oleg 2016-05-21 20:25 ` John Bauer 0 siblings, 1 reply; 13+ messages in thread From: Drokin, Oleg @ 2016-05-21 20:08 UTC (permalink / raw) To: lustre-devel The thing is, when you open a file with no layout (the one you cteate with P_LOB_DELAY_CREATE) for write the next time - the default layout is created just the same as it would have been created on the first open. So if you want custom layouts - you do need to insert setstripe call between the creation and actual open for write. On the other hand if you open with O_LOV_DELAY_CREATE and then try to write into that fd - you will get a failure. On May 21, 2016, at 4:01 PM, John Bauer wrote: > Andreas, > > Thanks for the reply. For what it's worth, extending a file that does not have layout set does work. > > % rm -f file.dat > % ./no_stripe.exe file.dat > fd=3 > % lfs getstripe file.dat > file.dat has no stripe info > % date >> file.dat > % lfs getstripe file.dat > file.dat > lmm_stripe_count: 1 > lmm_stripe_size: 1048576 > lmm_pattern: 1 > lmm_layout_gen: 0 > lmm_stripe_offset: 21 > obdidx objid objid group > 21 6143298 0x5dbd42 0 > > % > The LD_PRELOAD is exactly what I am doing in my I/O library. Unfortunately, one can not intercept the open() that results from a call to fopen(). That open is hard linked to the open in libc and not satisfied by the runtime linker. This is what is driving this topic for me. I can not conveniently set the striping for a file opened with fopen() and other functions where the open is called from inside libc. I used to believe that not too many application use stdio for heavy I/O, but I have been come across several recently. > > John > > On 5/21/2016 12:51 AM, Dilger, Andreas wrote: >> This is probably getting to be more of a topic for lustre-devel. >> >> There currently isn't any way to do what you ask, since (IIRC) it will cause an error for apps that try to write to the files before the layout is set. >> >> What you could do is to create an LD_PRELOAD library to intercept the open() calls and set O_LOV_DELAY_CREATE and set the layout explicitly for each file. This might be a win if each file needs a different layout, but since it uses two RPCs per file it would be slower than using the default layout. >> >> Cheers, Andreas >> >> On May 18, 2016, at 16:46, John Bauer <bauerj@iodoctors.com> wrote: >> >>> Since today's topic seems to be Lustre striping, I will revisit a previous line of questions I had. >>> >>> Andreas had put me on to O_LOV_DELAY_CREATE which I have been experimenting with. My question is : Is there a way to flag a directory with O_LOV_DELAY_CREATE so that a file created in that directory will be created with O_LOV_DELAY_CREATE also. Much like a file can inherit a directory's stripe count and stripe size, it would be convenient if a file could also inherit O_LOV_DELAY_CREATE? That way, for open()s that I can not intercept ( and thus can not set O_LOV_DELAY_CREATE in oflags) , such as those issued by fopen(), I can then get the fd with fileno() and set the striping with ioctl(fd, LL_IOC_LOV_SETSTRIPE, lum). >>> >>> Thanks >>> >>> John >>> -- >>> I/O Doctors, LLC >>> 507-766-0378 >>> >>> bauerj at iodoctors.com >>> _______________________________________________ >>> lustre-discuss mailing list >>> lustre-discuss at lists.lustre.org >>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > > -- > I/O Doctors, LLC > 507-766-0378 > > bauerj at iodoctors.com > _______________________________________________ > lustre-discuss mailing list > lustre-discuss at lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org ^ permalink raw reply [flat|nested] 13+ messages in thread
* [lustre-devel] [lustre-discuss] more on lustre striping 2016-05-21 20:08 ` Drokin, Oleg @ 2016-05-21 20:25 ` John Bauer 2016-05-22 0:33 ` Drokin, Oleg 0 siblings, 1 reply; 13+ messages in thread From: John Bauer @ 2016-05-21 20:25 UTC (permalink / raw) To: lustre-devel Oleg So in my simple test, the second open of the file caused the layout to be created. Indeed, a write to the original fd did fail. That complicates things considerably. Disregard the entire topic. Thanks John On 5/21/2016 3:08 PM, Drokin, Oleg wrote: > The thing is, when you open a file with no layout (the one you cteate with P_LOB_DELAY_CREATE) for write the next time - > the default layout is created just the same as it would have been created on the first open. > So if you want custom layouts - you do need to insert setstripe call between the creation and actual open for write. > > On the other hand if you open with O_LOV_DELAY_CREATE and then try to write into that fd - you will get a failure. > > > On May 21, 2016, at 4:01 PM, John Bauer wrote: > >> Andreas, >> >> Thanks for the reply. For what it's worth, extending a file that does not have layout set does work. >> >> % rm -f file.dat >> % ./no_stripe.exe file.dat >> fd=3 >> % lfs getstripe file.dat >> file.dat has no stripe info >> % date >> file.dat >> % lfs getstripe file.dat >> file.dat >> lmm_stripe_count: 1 >> lmm_stripe_size: 1048576 >> lmm_pattern: 1 >> lmm_layout_gen: 0 >> lmm_stripe_offset: 21 >> obdidx objid objid group >> 21 6143298 0x5dbd42 0 >> >> % >> The LD_PRELOAD is exactly what I am doing in my I/O library. Unfortunately, one can not intercept the open() that results from a call to fopen(). That open is hard linked to the open in libc and not satisfied by the runtime linker. This is what is driving this topic for me. I can not conveniently set the striping for a file opened with fopen() and other functions where the open is called from inside libc. I used to believe that not too many application use stdio for heavy I/O, but I have been come across several recently. >> >> John >> >> On 5/21/2016 12:51 AM, Dilger, Andreas wrote: >>> This is probably getting to be more of a topic for lustre-devel. >>> >>> There currently isn't any way to do what you ask, since (IIRC) it will cause an error for apps that try to write to the files before the layout is set. >>> >>> What you could do is to create an LD_PRELOAD library to intercept the open() calls and set O_LOV_DELAY_CREATE and set the layout explicitly for each file. This might be a win if each file needs a different layout, but since it uses two RPCs per file it would be slower than using the default layout. >>> >>> Cheers, Andreas >>> >>> On May 18, 2016, at 16:46, John Bauer <bauerj@iodoctors.com> wrote: >>> >>>> Since today's topic seems to be Lustre striping, I will revisit a previous line of questions I had. >>>> >>>> Andreas had put me on to O_LOV_DELAY_CREATE which I have been experimenting with. My question is : Is there a way to flag a directory with O_LOV_DELAY_CREATE so that a file created in that directory will be created with O_LOV_DELAY_CREATE also. Much like a file can inherit a directory's stripe count and stripe size, it would be convenient if a file could also inherit O_LOV_DELAY_CREATE? That way, for open()s that I can not intercept ( and thus can not set O_LOV_DELAY_CREATE in oflags) , such as those issued by fopen(), I can then get the fd with fileno() and set the striping with ioctl(fd, LL_IOC_LOV_SETSTRIPE, lum). >>>> >>>> Thanks >>>> >>>> John >>>> -- >>>> I/O Doctors, LLC >>>> 507-766-0378 >>>> >>>> bauerj at iodoctors.com >>>> _______________________________________________ >>>> lustre-discuss mailing list >>>> lustre-discuss at lists.lustre.org >>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org >> -- >> I/O Doctors, LLC >> 507-766-0378 >> >> bauerj at iodoctors.com >> _______________________________________________ >> lustre-discuss mailing list >> lustre-discuss at lists.lustre.org >> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org -- I/O Doctors, LLC 507-766-0378 bauerj at iodoctors.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20160521/6ca261f0/attachment-0001.htm> ^ permalink raw reply [flat|nested] 13+ messages in thread
* [lustre-devel] [lustre-discuss] more on lustre striping 2016-05-21 20:25 ` John Bauer @ 2016-05-22 0:33 ` Drokin, Oleg 2016-05-22 1:56 ` John Bauer 0 siblings, 1 reply; 13+ messages in thread From: Drokin, Oleg @ 2016-05-22 0:33 UTC (permalink / raw) To: lustre-devel btw I find it strange that you cannot intercept fopen (and in fact intercepting every library call like that is counterproductive). We used to have this "liblustre" library, that you an LD_PRELOAD into your application and it would work with Lustre even if you are not root and if Lustre is not mounted on that node (and in fact even if the node is not Linux at all). That had no problems at all to intercept all sorts of opens by intercepting syscalls. I wonder if you can intercept something deeper like sys_open or something like that? Perhaps checkout lustre 1.8 sources (or even 2.1) and see how we did it back there? On May 21, 2016, at 4:25 PM, John Bauer wrote: > Oleg > > So in my simple test, the second open of the file caused the layout to be created. Indeed, a write to the original fd did fail. > That complicates things considerably. > > Disregard the entire topic. > > Thanks > > John > > > On 5/21/2016 3:08 PM, Drokin, Oleg wrote: >> The thing is, when you open a file with no layout (the one you cteate with P_LOB_DELAY_CREATE) for write the next time - >> the default layout is created just the same as it would have been created on the first open. >> So if you want custom layouts - you do need to insert setstripe call between the creation and actual open for write. >> >> On the other hand if you open with O_LOV_DELAY_CREATE and then try to write into that fd - you will get a failure. >> >> >> On May 21, 2016, at 4:01 PM, John Bauer wrote: >> >> >>> Andreas, >>> >>> Thanks for the reply. For what it's worth, extending a file that does not have layout set does work. >>> >>> % rm -f file.dat >>> % ./no_stripe.exe file.dat >>> fd=3 >>> % lfs getstripe file.dat >>> file.dat has no stripe info >>> % date >> file.dat >>> % lfs getstripe file.dat >>> file.dat >>> lmm_stripe_count: 1 >>> lmm_stripe_size: 1048576 >>> lmm_pattern: 1 >>> lmm_layout_gen: 0 >>> lmm_stripe_offset: 21 >>> obdidx objid objid group >>> 21 6143298 0x5dbd42 0 >>> >>> % >>> The LD_PRELOAD is exactly what I am doing in my I/O library. Unfortunately, one can not intercept the open() that results from a call to fopen(). That open is hard linked to the open in libc and not satisfied by the runtime linker. This is what is driving this topic for me. I can not conveniently set the striping for a file opened with fopen() and other functions where the open is called from inside libc. I used to believe that not too many application use stdio for heavy I/O, but I have been come across several recently. >>> >>> John >>> >>> On 5/21/2016 12:51 AM, Dilger, Andreas wrote: >>> >>>> This is probably getting to be more of a topic for lustre-devel. >>>> >>>> There currently isn't any way to do what you ask, since (IIRC) it will cause an error for apps that try to write to the files before the layout is set. >>>> >>>> What you could do is to create an LD_PRELOAD library to intercept the open() calls and set O_LOV_DELAY_CREATE and set the layout explicitly for each file. This might be a win if each file needs a different layout, but since it uses two RPCs per file it would be slower than using the default layout. >>>> >>>> Cheers, Andreas >>>> >>>> On May 18, 2016, at 16:46, John Bauer >>>> <bauerj@iodoctors.com> >>>> wrote: >>>> >>>> >>>>> Since today's topic seems to be Lustre striping, I will revisit a previous line of questions I had. >>>>> >>>>> Andreas had put me on to O_LOV_DELAY_CREATE which I have been experimenting with. My question is : Is there a way to flag a directory with O_LOV_DELAY_CREATE so that a file created in that directory will be created with O_LOV_DELAY_CREATE also. Much like a file can inherit a directory's stripe count and stripe size, it would be convenient if a file could also inherit O_LOV_DELAY_CREATE? That way, for open()s that I can not intercept ( and thus can not set O_LOV_DELAY_CREATE in oflags) , such as those issued by fopen(), I can then get the fd with fileno() and set the striping with ioctl(fd, LL_IOC_LOV_SETSTRIPE, lum). >>>>> >>>>> Thanks >>>>> >>>>> John >>>>> -- >>>>> I/O Doctors, LLC >>>>> 507-766-0378 >>>>> >>>>> >>>>> bauerj at iodoctors.com >>>>> >>>>> _______________________________________________ >>>>> lustre-discuss mailing list >>>>> >>>>> lustre-discuss at lists.lustre.org >>>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org >>> -- >>> I/O Doctors, LLC >>> 507-766-0378 >>> >>> >>> bauerj at iodoctors.com >>> >>> _______________________________________________ >>> lustre-discuss mailing list >>> >>> lustre-discuss at lists.lustre.org >>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > > -- > I/O Doctors, LLC > 507-766-0378 > > bauerj at iodoctors.com ^ permalink raw reply [flat|nested] 13+ messages in thread
* [lustre-devel] [lustre-discuss] more on lustre striping 2016-05-22 0:33 ` Drokin, Oleg @ 2016-05-22 1:56 ` John Bauer 2016-05-22 2:49 ` Drokin, Oleg 2016-06-10 12:29 ` Ashley Pittman 0 siblings, 2 replies; 13+ messages in thread From: John Bauer @ 2016-05-22 1:56 UTC (permalink / raw) To: lustre-devel Oleg I can intercept the fopen(), but that does me no good as I can't set the O_LOV_DELAY_CREATE bit. What I can not intercept is the open() downstream of fopen(). If one examines the symbols in libc you will see there are no unsatisfied externals relating to open, which means there is nothing for the runtime linker to find concerning open's. I will have a look at the Lustre 1.8 source, but I seriously doubt that the open beneath fopen() was intercepted with LD_PRELOAD. I would love to find a way to do that. I could throw away a lot of code. Thanks, John % nm -g /lib64/libc.so.6 | grep open 0000000000033d70 T catopen 00000000003bfb80 B _dl_open_hook 00000000000b9a60 W fdopendir 000000000006b140 T fdopen@@GLIBC_2.2.5 00000000000755c0 T fmemopen 000000000006ba00 W fopen64 000000000006bb60 T fopencookie@@GLIBC_2.2.5 000000000006ba00 T fopen@@GLIBC_2.2.5 00000000000736f0 T freopen 0000000000074b50 T freopen64 00000000000ead40 T fts_open 0000000000022220 T iconv_open 000000000006b140 T _IO_fdopen@@GLIBC_2.2.5 0000000000077220 T _IO_file_fopen@@GLIBC_2.2.5 0000000000077170 T _IO_file_open 000000000006ba00 T _IO_fopen@@GLIBC_2.2.5 000000000006d1d0 T _IO_popen@@GLIBC_2.2.5 000000000006cee0 T _IO_proc_open@@GLIBC_2.2.5 0000000000130b20 T __libc_dlopen_mode 00000000000e7840 W open 00000000000e7840 W __open 00000000000ec690 T __open_2 00000000000e7840 W open64 00000000000e7840 W __open64 00000000000ec6b0 T __open64_2 00000000000e78d0 W openat 00000000000e79b0 T __openat_2 00000000000e78d0 W openat64 00000000000e79b0 W __openat64_2 00000000000f6e00 T open_by_handle_at 00000000000340b0 T __open_catalog 00000000000b9510 W opendir 00000000000f0850 T openlog 0000000000073e90 T open_memstream 00000000000731b0 T open_wmemstream 000000000006d1d0 T popen@@GLIBC_2.2.5 000000000012fbd0 W posix_openpt 00000000000e6460 T posix_spawn_file_actions_addopen % John On 5/21/2016 7:33 PM, Drokin, Oleg wrote: > btw I find it strange that you cannot intercept fopen (and in fact intercepting every library call like that is counterproductive). > > We used to have this "liblustre" library, that you an LD_PRELOAD into your application and it would work with Lustre even if you are not root and if Lustre is not mounted on that node > (and in fact even if the node is not Linux at all). That had no problems at all to intercept all sorts of opens by intercepting syscalls. > I wonder if you can intercept something deeper like sys_open or something like that? > Perhaps checkout lustre 1.8 sources (or even 2.1) and see how we did it back there? > > On May 21, 2016, at 4:25 PM, John Bauer wrote: > >> Oleg >> >> So in my simple test, the second open of the file caused the layout to be created. Indeed, a write to the original fd did fail. >> That complicates things considerably. >> >> Disregard the entire topic. >> >> Thanks >> >> John >> >> >> On 5/21/2016 3:08 PM, Drokin, Oleg wrote: >>> The thing is, when you open a file with no layout (the one you cteate with P_LOB_DELAY_CREATE) for write the next time - >>> the default layout is created just the same as it would have been created on the first open. >>> So if you want custom layouts - you do need to insert setstripe call between the creation and actual open for write. >>> >>> On the other hand if you open with O_LOV_DELAY_CREATE and then try to write into that fd - you will get a failure. >>> >>> >>> On May 21, 2016, at 4:01 PM, John Bauer wrote: >>> >>> >>>> Andreas, >>>> >>>> Thanks for the reply. For what it's worth, extending a file that does not have layout set does work. >>>> >>>> % rm -f file.dat >>>> % ./no_stripe.exe file.dat >>>> fd=3 >>>> % lfs getstripe file.dat >>>> file.dat has no stripe info >>>> % date >> file.dat >>>> % lfs getstripe file.dat >>>> file.dat >>>> lmm_stripe_count: 1 >>>> lmm_stripe_size: 1048576 >>>> lmm_pattern: 1 >>>> lmm_layout_gen: 0 >>>> lmm_stripe_offset: 21 >>>> obdidx objid objid group >>>> 21 6143298 0x5dbd42 0 >>>> >>>> % >>>> The LD_PRELOAD is exactly what I am doing in my I/O library. Unfortunately, one can not intercept the open() that results from a call to fopen(). That open is hard linked to the open in libc and not satisfied by the runtime linker. This is what is driving this topic for me. I can not conveniently set the striping for a file opened with fopen() and other functions where the open is called from inside libc. I used to believe that not too many application use stdio for heavy I/O, but I have been come across several recently. >>>> >>>> John >>>> >>>> On 5/21/2016 12:51 AM, Dilger, Andreas wrote: >>>> >>>>> This is probably getting to be more of a topic for lustre-devel. >>>>> >>>>> There currently isn't any way to do what you ask, since (IIRC) it will cause an error for apps that try to write to the files before the layout is set. >>>>> >>>>> What you could do is to create an LD_PRELOAD library to intercept the open() calls and set O_LOV_DELAY_CREATE and set the layout explicitly for each file. This might be a win if each file needs a different layout, but since it uses two RPCs per file it would be slower than using the default layout. >>>>> >>>>> Cheers, Andreas >>>>> >>>>> On May 18, 2016, at 16:46, John Bauer >>>>> <bauerj@iodoctors.com> >>>>> wrote: >>>>> >>>>> >>>>>> Since today's topic seems to be Lustre striping, I will revisit a previous line of questions I had. >>>>>> >>>>>> Andreas had put me on to O_LOV_DELAY_CREATE which I have been experimenting with. My question is : Is there a way to flag a directory with O_LOV_DELAY_CREATE so that a file created in that directory will be created with O_LOV_DELAY_CREATE also. Much like a file can inherit a directory's stripe count and stripe size, it would be convenient if a file could also inherit O_LOV_DELAY_CREATE? That way, for open()s that I can not intercept ( and thus can not set O_LOV_DELAY_CREATE in oflags) , such as those issued by fopen(), I can then get the fd with fileno() and set the striping with ioctl(fd, LL_IOC_LOV_SETSTRIPE, lum). >>>>>> >>>>>> Thanks >>>>>> >>>>>> John >>>>>> -- >>>>>> I/O Doctors, LLC >>>>>> 507-766-0378 >>>>>> >>>>>> >>>>>> bauerj at iodoctors.com >>>>>> >>>>>> _______________________________________________ >>>>>> lustre-discuss mailing list >>>>>> >>>>>> lustre-discuss at lists.lustre.org >>>>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org >>>> -- >>>> I/O Doctors, LLC >>>> 507-766-0378 >>>> >>>> >>>> bauerj at iodoctors.com >>>> >>>> _______________________________________________ >>>> lustre-discuss mailing list >>>> >>>> lustre-discuss at lists.lustre.org >>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org >> -- >> I/O Doctors, LLC >> 507-766-0378 >> >> bauerj at iodoctors.com -- I/O Doctors, LLC 507-766-0378 bauerj at iodoctors.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20160521/bdbc3cca/attachment-0001.htm> ^ permalink raw reply [flat|nested] 13+ messages in thread
* [lustre-devel] [lustre-discuss] more on lustre striping 2016-05-22 1:56 ` John Bauer @ 2016-05-22 2:49 ` Drokin, Oleg 2016-06-10 12:29 ` Ashley Pittman 1 sibling, 0 replies; 13+ messages in thread From: Drokin, Oleg @ 2016-05-22 2:49 UTC (permalink / raw) To: lustre-devel $ nm -g lustre/liblustre/liblustre.so | grep open 0000000000237820 W __open 0000000000237820 W __open64 000000000023ba30 W __opendir 00000000002376c0 T _sysio_open U fopen@@GLIBC_2.2.5 0000000000237820 T open 0000000000237820 W open64 000000000023ba30 T opendir These are the open symbols we have in the .so it most certainly intercepts the open syscall no matter if it comes via open or fopen. so I suspect you just need to catch __open* stuff and this will catch both open and fopen for you too. At least quick googling around seems to confirm this. All intercepting was done via libsysio (since it reimplemented VFS in userspace), so if you need more info, perhaps you can consult with Lee Ward who is the main author of it. I know he used to read this list too so he might decide to chime in. On May 21, 2016, at 9:56 PM, John Bauer wrote: > Oleg > > I can intercept the fopen(), but that does me no good as I can't set the O_LOV_DELAY_CREATE bit. What I can not intercept is the open() downstream of fopen(). If one examines the symbols in libc you will see there are no unsatisfied externals relating to open, which means there is nothing for the runtime linker to find concerning open's. I will have a look at the Lustre 1.8 source, but I seriously doubt that the open beneath fopen() was intercepted with LD_PRELOAD. I would love to find a way to do that. I could throw away a lot of code. Thanks, John > % nm -g /lib64/libc.so.6 | grep open > 0000000000033d70 T catopen > 00000000003bfb80 B _dl_open_hook > 00000000000b9a60 W fdopendir > 000000000006b140 T fdopen@@GLIBC_2.2.5 > 00000000000755c0 T fmemopen > 000000000006ba00 W fopen64 > 000000000006bb60 T fopencookie@@GLIBC_2.2.5 > 000000000006ba00 T fopen@@GLIBC_2.2.5 > 00000000000736f0 T freopen > 0000000000074b50 T freopen64 > 00000000000ead40 T fts_open > 0000000000022220 T iconv_open > 000000000006b140 T _IO_fdopen@@GLIBC_2.2.5 > 0000000000077220 T _IO_file_fopen@@GLIBC_2.2.5 > 0000000000077170 T _IO_file_open > 000000000006ba00 T _IO_fopen@@GLIBC_2.2.5 > 000000000006d1d0 T _IO_popen@@GLIBC_2.2.5 > 000000000006cee0 T _IO_proc_open@@GLIBC_2.2.5 > 0000000000130b20 T __libc_dlopen_mode > 00000000000e7840 W open > 00000000000e7840 W __open > 00000000000ec690 T __open_2 > 00000000000e7840 W open64 > 00000000000e7840 W __open64 > 00000000000ec6b0 T __open64_2 > 00000000000e78d0 W openat > 00000000000e79b0 T __openat_2 > 00000000000e78d0 W openat64 > 00000000000e79b0 W __openat64_2 > 00000000000f6e00 T open_by_handle_at > 00000000000340b0 T __open_catalog > 00000000000b9510 W opendir > 00000000000f0850 T openlog > 0000000000073e90 T open_memstream > 00000000000731b0 T open_wmemstream > 000000000006d1d0 T popen@@GLIBC_2.2.5 > 000000000012fbd0 W posix_openpt > 00000000000e6460 T posix_spawn_file_actions_addopen > % > John > > On 5/21/2016 7:33 PM, Drokin, Oleg wrote: >> btw I find it strange that you cannot intercept fopen (and in fact intercepting every library call like that is counterproductive). >> >> We used to have this "liblustre" library, that you an LD_PRELOAD into your application and it would work with Lustre even if you are not root and if Lustre is not mounted on that node >> (and in fact even if the node is not Linux at all). That had no problems at all to intercept all sorts of opens by intercepting syscalls. >> I wonder if you can intercept something deeper like sys_open or something like that? >> Perhaps checkout lustre 1.8 sources (or even 2.1) and see how we did it back there? >> >> On May 21, 2016, at 4:25 PM, John Bauer wrote: >> >> >>> Oleg >>> >>> So in my simple test, the second open of the file caused the layout to be created. Indeed, a write to the original fd did fail. >>> That complicates things considerably. >>> >>> Disregard the entire topic. >>> >>> Thanks >>> >>> John >>> >>> >>> On 5/21/2016 3:08 PM, Drokin, Oleg wrote: >>> >>>> The thing is, when you open a file with no layout (the one you cteate with P_LOB_DELAY_CREATE) for write the next time - >>>> the default layout is created just the same as it would have been created on the first open. >>>> So if you want custom layouts - you do need to insert setstripe call between the creation and actual open for write. >>>> >>>> On the other hand if you open with O_LOV_DELAY_CREATE and then try to write into that fd - you will get a failure. >>>> >>>> >>>> On May 21, 2016, at 4:01 PM, John Bauer wrote: >>>> >>>> >>>> >>>>> Andreas, >>>>> >>>>> Thanks for the reply. For what it's worth, extending a file that does not have layout set does work. >>>>> >>>>> % rm -f file.dat >>>>> % ./no_stripe.exe file.dat >>>>> fd=3 >>>>> % lfs getstripe file.dat >>>>> file.dat has no stripe info >>>>> % date >> file.dat >>>>> % lfs getstripe file.dat >>>>> file.dat >>>>> lmm_stripe_count: 1 >>>>> lmm_stripe_size: 1048576 >>>>> lmm_pattern: 1 >>>>> lmm_layout_gen: 0 >>>>> lmm_stripe_offset: 21 >>>>> obdidx objid objid group >>>>> 21 6143298 0x5dbd42 0 >>>>> >>>>> % >>>>> The LD_PRELOAD is exactly what I am doing in my I/O library. Unfortunately, one can not intercept the open() that results from a call to fopen(). That open is hard linked to the open in libc and not satisfied by the runtime linker. This is what is driving this topic for me. I can not conveniently set the striping for a file opened with fopen() and other functions where the open is called from inside libc. I used to believe that not too many application use stdio for heavy I/O, but I have been come across several recently. >>>>> >>>>> John >>>>> >>>>> On 5/21/2016 12:51 AM, Dilger, Andreas wrote: >>>>> >>>>> >>>>>> This is probably getting to be more of a topic for lustre-devel. >>>>>> >>>>>> There currently isn't any way to do what you ask, since (IIRC) it will cause an error for apps that try to write to the files before the layout is set. >>>>>> >>>>>> What you could do is to create an LD_PRELOAD library to intercept the open() calls and set O_LOV_DELAY_CREATE and set the layout explicitly for each file. This might be a win if each file needs a different layout, but since it uses two RPCs per file it would be slower than using the default layout. >>>>>> >>>>>> Cheers, Andreas >>>>>> >>>>>> On May 18, 2016, at 16:46, John Bauer >>>>>> >>>>>> <bauerj@iodoctors.com> >>>>>> >>>>>> wrote: >>>>>> >>>>>> >>>>>> >>>>>>> Since today's topic seems to be Lustre striping, I will revisit a previous line of questions I had. >>>>>>> >>>>>>> Andreas had put me on to O_LOV_DELAY_CREATE which I have been experimenting with. My question is : Is there a way to flag a directory with O_LOV_DELAY_CREATE so that a file created in that directory will be created with O_LOV_DELAY_CREATE also. Much like a file can inherit a directory's stripe count and stripe size, it would be convenient if a file could also inherit O_LOV_DELAY_CREATE? That way, for open()s that I can not intercept ( and thus can not set O_LOV_DELAY_CREATE in oflags) , such as those issued by fopen(), I can then get the fd with fileno() and set the striping with ioctl(fd, LL_IOC_LOV_SETSTRIPE, lum). >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> John >>>>>>> -- >>>>>>> I/O Doctors, LLC >>>>>>> 507-766-0378 >>>>>>> >>>>>>> >>>>>>> >>>>>>> bauerj at iodoctors.com >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> lustre-discuss mailing list >>>>>>> >>>>>>> >>>>>>> lustre-discuss at lists.lustre.org >>>>>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org >>>>> -- >>>>> I/O Doctors, LLC >>>>> 507-766-0378 >>>>> >>>>> >>>>> >>>>> bauerj at iodoctors.com >>>>> >>>>> >>>>> _______________________________________________ >>>>> lustre-discuss mailing list >>>>> >>>>> >>>>> lustre-discuss at lists.lustre.org >>>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org >>> -- >>> I/O Doctors, LLC >>> 507-766-0378 >>> >>> >>> bauerj at iodoctors.com > > -- > I/O Doctors, LLC > 507-766-0378 > > bauerj at iodoctors.com ^ permalink raw reply [flat|nested] 13+ messages in thread
* [lustre-devel] [lustre-discuss] more on lustre striping 2016-05-22 1:56 ` John Bauer 2016-05-22 2:49 ` Drokin, Oleg @ 2016-06-10 12:29 ` Ashley Pittman 2016-06-10 15:28 ` John Bauer 2016-06-10 16:04 ` John Bauer 1 sibling, 2 replies; 13+ messages in thread From: Ashley Pittman @ 2016-06-10 12:29 UTC (permalink / raw) To: lustre-devel On 22/05/16 02:56, John Bauer wrote: > > Oleg > > I can intercept the fopen(), but that does me no good as I can't set > the O_LOV_DELAY_CREATE bit. What I can not intercept is the open() > downstream of fopen(). If one examines the symbols in libc you will > see there are no unsatisfied externals relating to open, which means > there is nothing for the runtime linker to find concerning open's. I > will have a look at the Lustre 1.8 source, but I seriously doubt that > the open beneath fopen() was intercepted with LD_PRELOAD. I would > love to find a way to do that. I could throw away a lot of code. > Thanks, John > Could you not intercept fopen() and implement it with calls to open() and fdopen() yourself which would give you full control over what you're looking for here? Ashley. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20160610/ac16a068/attachment.htm> ^ permalink raw reply [flat|nested] 13+ messages in thread
* [lustre-devel] [lustre-discuss] more on lustre striping 2016-06-10 12:29 ` Ashley Pittman @ 2016-06-10 15:28 ` John Bauer 2016-06-10 16:04 ` John Bauer 1 sibling, 0 replies; 13+ messages in thread From: John Bauer @ 2016-06-10 15:28 UTC (permalink / raw) To: lustre-devel Ashley That is what I have done for this case. But it does not solve the entire problem. One stdio call that is problematic is the freopen(). There is no way to pass the fd from open() to an existing FILE *. Thanks, John On 6/10/2016 7:29 AM, Ashley Pittman wrote: > On 22/05/16 02:56, John Bauer wrote: >> >> Oleg >> >> I can intercept the fopen(), but that does me no good as I can't set >> the O_LOV_DELAY_CREATE bit. What I can not intercept is the open() >> downstream of fopen(). If one examines the symbols in libc you will >> see there are no unsatisfied externals relating to open, which means >> there is nothing for the runtime linker to find concerning open's. I >> will have a look at the Lustre 1.8 source, but I seriously doubt that >> the open beneath fopen() was intercepted with LD_PRELOAD. I would >> love to find a way to do that. I could throw away a lot of code. >> Thanks, John >> > > Could you not intercept fopen() and implement it with calls to open() > and fdopen() yourself which would give you full control over what > you're looking for here? > > Ashley. -- I/O Doctors, LLC 507-766-0378 bauerj at iodoctors.com ^ permalink raw reply [flat|nested] 13+ messages in thread
* [lustre-devel] [lustre-discuss] more on lustre striping 2016-06-10 12:29 ` Ashley Pittman 2016-06-10 15:28 ` John Bauer @ 2016-06-10 16:04 ` John Bauer 2016-06-17 18:28 ` Dilger, Andreas 2016-06-23 21:50 ` Oleg Drokin 1 sibling, 2 replies; 13+ messages in thread From: John Bauer @ 2016-06-10 16:04 UTC (permalink / raw) To: lustre-devel To confirm the point that you can not intercept the open called by fopen by using LD_PRELOAD, I have written a simple test case. Note that the runtime linker never looks for open(). Only fopen() *$ cat a.c* #include <unistd.h> #include <stdlib.h> #include <fcntl.h> #include <stdio.h> int main(int argc, char ** argv ){ FILE *f = fopen("a", "r" ) ; fprintf(stderr,"f=%p\n",f); fclose(f); } *$ file a* a: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=dfe043b4ec8cf19d5fd3fab524d7c72ed1453574, not stripped *$ cat a.csh* #!/bin/csh setenv LD_DEBUG all ./a >&! a.cpr *$ ./a.csh* *$ grep -i open a.cpr* 120584: symbol=fopen; lookup in file=./a [0] 120584: symbol=fopen; lookup in file=/lib64/libc.so.6 [0] 120584: binding file ./a [0] to /lib64/libc.so.6 [0]: normal symbol `fopen' [GLIBC_2.2.5] *$* On 6/10/2016 7:29 AM, Ashley Pittman wrote: > On 22/05/16 02:56, John Bauer wrote: >> >> Oleg >> >> I can intercept the fopen(), but that does me no good as I can't set >> the O_LOV_DELAY_CREATE bit. What I can not intercept is the open() >> downstream of fopen(). If one examines the symbols in libc you will >> see there are no unsatisfied externals relating to open, which means >> there is nothing for the runtime linker to find concerning open's. I >> will have a look at the Lustre 1.8 source, but I seriously doubt that >> the open beneath fopen() was intercepted with LD_PRELOAD. I would >> love to find a way to do that. I could throw away a lot of code. >> Thanks, John >> > > Could you not intercept fopen() and implement it with calls to open() > and fdopen() yourself which would give you full control over what > you're looking for here? > > Ashley. -- I/O Doctors, LLC 507-766-0378 bauerj at iodoctors.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20160610/0da6b324/attachment.htm> ^ permalink raw reply [flat|nested] 13+ messages in thread
* [lustre-devel] [lustre-discuss] more on lustre striping 2016-06-10 16:04 ` John Bauer @ 2016-06-17 18:28 ` Dilger, Andreas 2016-06-23 21:50 ` Oleg Drokin 1 sibling, 0 replies; 13+ messages in thread From: Dilger, Andreas @ 2016-06-17 18:28 UTC (permalink / raw) To: lustre-devel If I recall correctly from when we implemented liblustre, which hooked into userspace using LD_PRELOAD, we had to capture the __open() call instead of open() (or something similar) to ensure that glibc didn't bypass our LD_PRELOAD from inside of fopen(). You might consider downloading the source for glibc to see what it is doing in fopen(). Cheers, Andreas -- Andreas Dilger Lustre Principal Architect Intel High Performance Data Division On 2016/06/10, 10:04, "lustre-discuss on behalf of John Bauer" <lustre-discuss-bounces at lists.lustre.org<mailto:lustre-discuss-bounces@lists.lustre.org> on behalf of bauerj at iodoctors.com<mailto:bauerj@iodoctors.com>> wrote: To confirm the point that you can not intercept the open called by fopen by using LD_PRELOAD, I have written a simple test case. Note that the runtime linker never looks for open(). Only fopen() $ cat a.c #include <unistd.h> #include <stdlib.h> #include <fcntl.h> #include <stdio.h> int main(int argc, char ** argv ){ FILE *f = fopen("a", "r" ) ; fprintf(stderr,"f=%p\n",f); fclose(f); } $ file a a: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=dfe043b4ec8cf19d5fd3fab524d7c72ed1453574, not stripped $ cat a.csh #!/bin/csh setenv LD_DEBUG all ./a >&! a.cpr $ ./a.csh $ grep -i open a.cpr 120584: symbol=fopen; lookup in file=./a [0] 120584: symbol=fopen; lookup in file=/lib64/libc.so.6 [0] 120584: binding file ./a [0] to /lib64/libc.so.6 [0]: normal symbol `fopen' [GLIBC_2.2.5] $ On 6/10/2016 7:29 AM, Ashley Pittman wrote: On 22/05/16 02:56, John Bauer wrote: Oleg I can intercept the fopen(), but that does me no good as I can't set the O_LOV_DELAY_CREATE bit. What I can not intercept is the open() downstream of fopen(). If one examines the symbols in libc you will see there are no unsatisfied externals relating to open, which means there is nothing for the runtime linker to find concerning open's. I will have a look at the Lustre 1.8 source, but I seriously doubt that the open beneath fopen() was intercepted with LD_PRELOAD. I would love to find a way to do that. I could throw away a lot of code. Thanks, John Could you not intercept fopen() and implement it with calls to open() and fdopen() yourself which would give you full control over what you're looking for here? Ashley. -- I/O Doctors, LLC 507-766-0378 bauerj at iodoctors.com<mailto:bauerj@iodoctors.com> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20160617/fe11079b/attachment.htm> ^ permalink raw reply [flat|nested] 13+ messages in thread
* [lustre-devel] [lustre-discuss] more on lustre striping 2016-06-10 16:04 ` John Bauer 2016-06-17 18:28 ` Dilger, Andreas @ 2016-06-23 21:50 ` Oleg Drokin 2016-06-26 19:17 ` John Bauer 1 sibling, 1 reply; 13+ messages in thread From: Oleg Drokin @ 2016-06-23 21:50 UTC (permalink / raw) To: lustre-devel I think what you are missing is that __open that you need to intercept comes from libc, so it does not need to be bound from your file because your binary does not call it, glibc calls it AND if your LD_PRELOAD'ed library defines it you'll catch it. The proper test would have been to write a LD_PRELOADable library that defines __open and then run your test application with it. On Jun 10, 2016, at 12:04 PM, John Bauer wrote: > To confirm the point that you can not intercept the open called by fopen by using LD_PRELOAD, I have written a simple test case. Note that the runtime linker never looks for open(). Only fopen() > $ cat a.c > #include <unistd.h> > #include <stdlib.h> > #include <fcntl.h> > #include <stdio.h> > > int > main(int argc, char ** argv ){ > FILE *f = fopen("a", "r" ) ; > fprintf(stderr,"f=%p\n",f); > fclose(f); > } > $ file a > a: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=dfe043b4ec8cf19d5fd3fab524d7c72ed1453574, not stripped > $ cat a.csh > #!/bin/csh > setenv LD_DEBUG all > ./a >&! a.cpr > $ ./a.csh > $ grep -i open a.cpr > 120584: symbol=fopen; lookup in file=./a [0] > 120584: symbol=fopen; lookup in file=/lib64/libc.so.6 [0] > 120584: binding file ./a [0] to /lib64/libc.so.6 [0]: normal symbol `fopen' [GLIBC_2.2.5] > $ > > On 6/10/2016 7:29 AM, Ashley Pittman wrote: >> On 22/05/16 02:56, John Bauer wrote: >>> Oleg >>> >>> I can intercept the fopen(), but that does me no good as I can't set the O_LOV_DELAY_CREATE bit. What I can not intercept is the open() downstream of fopen(). If one examines the symbols in libc you will see there are no unsatisfied externals relating to open, which means there is nothing for the runtime linker to find concerning open's. I will have a look at the Lustre 1.8 source, but I seriously doubt that the open beneath fopen() was intercepted with LD_PRELOAD. I would love to find a way to do that. I could throw away a lot of code. Thanks, John >> >> Could you not intercept fopen() and implement it with calls to open() and fdopen() yourself which would give you full control over what you're looking for here? >> >> Ashley. > > -- > I/O Doctors, LLC > 507-766-0378 > > bauerj at iodoctors.com > _______________________________________________ > lustre-devel mailing list > lustre-devel at lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org ^ permalink raw reply [flat|nested] 13+ messages in thread
* [lustre-devel] [lustre-discuss] more on lustre striping 2016-06-23 21:50 ` Oleg Drokin @ 2016-06-26 19:17 ` John Bauer 0 siblings, 0 replies; 13+ messages in thread From: John Bauer @ 2016-06-26 19:17 UTC (permalink / raw) To: lustre-devel Oleg I have written a simple test case that addresses your suggestions. My simple test program now calls __open() and the LD_PRELOAD'd shared object, libopen.so, exports __open(). The attached tar file creates a directory stdio. cd into it and *make clean all*. Then run *./stdio.csh*. You will see that the program's call to __open() is intercepted, as evidenced by the fprintf() from the preloaded __open(). But the open() call made by fopen() is still not reported by the preloaded function. I am thoroughly convinced fopen()'s call to open(), by whatever name it is using for open(), can not be intercepted. The most compelling evidence is the output from the run-time loader when LD_DEBUG=all is set. If I remove the call to __open() from the test program, the only function with open in its name that the run-time loader reports searching for is fopen(). The call that fopen() makes to open() is probably satisfied when libc is built. Per Andreas's suggestion, I have tried to build a debug version of glibc, but I think this is a bit outside my wheelhouse. It always turns into some type of struggle to get glibc to build correctly for the system I am on. I am seeking assistance on that front from admins on the system I am using. This URL is an interesting read, http://blog.hostilefork.com/where-printf-rubber-meets-road/ . It covers the write() call from fprintf(), but I assume open() would have the same issues. If I understand this correctly, the write() call from fprintf() boils down to a syscall at compile time. It doesn't even bother will calling write(). glibc skips the step of calling write() and goes straight to syscall() when calling write() from inside of glibc. Thanks again. John On 6/23/2016 4:50 PM, Oleg Drokin wrote: > #include <unistd.h> > >#include <stdlib.h> > >#include <fcntl.h> > >#include <stdio.h> > > > >int > >main(int argc, char ** argv ){ > > FILE *f = fopen("a", "r" ) ; > > fprintf(stderr,"f=%p\n",f); > > fclose(f); > >} -- I/O Doctors, LLC 507-766-0378 bauerj at iodoctors.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20160626/41199577/attachment.htm> -------------- next part -------------- A non-text attachment was scrubbed... Name: stdio.tgz Type: application/octet-stream Size: 806 bytes Desc: not available URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20160626/41199577/attachment.obj> ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2016-06-26 19:17 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <1ef5a267-334c-be0d-13f4-c0fab917d1bf@iodoctors.com>
2016-05-21 5:51 ` [lustre-devel] [lustre-discuss] more on lustre striping Dilger, Andreas
2016-05-21 20:01 ` John Bauer
2016-05-21 20:08 ` Drokin, Oleg
2016-05-21 20:25 ` John Bauer
2016-05-22 0:33 ` Drokin, Oleg
2016-05-22 1:56 ` John Bauer
2016-05-22 2:49 ` Drokin, Oleg
2016-06-10 12:29 ` Ashley Pittman
2016-06-10 15:28 ` John Bauer
2016-06-10 16:04 ` John Bauer
2016-06-17 18:28 ` Dilger, Andreas
2016-06-23 21:50 ` Oleg Drokin
2016-06-26 19:17 ` John Bauer
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.