From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alphe Salas Michels Subject: Re: writing a ceph cliente for MS windows Date: Thu, 07 Nov 2013 17:47:09 -0300 Message-ID: <527BFC4D.6040607@kepler.cl> References: <988607914.48.1383835814711.JavaMail.root@thunderbeast.private.linuxbox.com> <527BD5CC.6050002@kepler.cl> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE Return-path: Received: from mail-qc0-f169.google.com ([209.85.216.169]:50463 "EHLO mail-qc0-f169.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751010Ab3KGUrP (ORCPT ); Thu, 7 Nov 2013 15:47:15 -0500 Received: by mail-qc0-f169.google.com with SMTP id x12so939642qcv.28 for ; Thu, 07 Nov 2013 12:47:14 -0800 (PST) In-Reply-To: <527BD5CC.6050002@kepler.cl> Sender: ceph-devel-owner@vger.kernel.org List-ID: To: "Matt W. Benjamin" , Ketor D Cc: ceph-devel Hello all I finally finished my first source code extraction that start= s=20 from ceph/src/client/fuse_ll.c The result is accurate unlike previous provided results. basically the=20 script start from a file extract all the private includes definitions =20 #include "something.h" and recursively extract private includes too. th= e=20 best way to know who is related with who. starting from fuse_ll.cc I optain 390 files retreived and 120 000 lines= =20 of code ! involved dirs are : in ceph/src objclass/, common/, msg/, common/, osdc/, include/, client/, mds/,=20 global/, json_spirit/, log/, os/, crush/, mon/, osd/, auth/ probably not a good way to analyse what amount of work it means since=20 most of those directories are the implementation of servers (osd, mon,=20 mds) and even if only a tiny bit of them is needed at client level. you= =20 need two structures from ./osd/OSD.h and my script by relation will=20 take into acount the whole directory... I ran the script with libcephfs.cc as start point and got almost the=20 same results. 131 000 lines of code and 386 files most of the same dirs= =20 involved. I think I will spend alot of time doing the manual source code isolatio= n=20 and understand way each #include is set in the files I read (what=20 purpose they have do they allow to integrate a crucial data type or not= =2E The other way around will be to read src/libcephfs.cc. It seems shorter= =20 but without understanding what part is used for each included header I=20 can t say anything... I will keep reading the source code and take notes. I think in the case= =20 of libcephfs I will gain alot of time. signature *Alph=C3=A9 Salas* Ingeniero T.I asalas@kepler.cl *www.kepler.cl * On 11/07/13 15:02, Alphe Salas Michels wrote: > Hello D.Ketor and Matt Benjamin, > You give me alot to think about and this is great! > I merged your previous post to make a single reply that anyone can=20 > report to easyly > > Windows NFS 4.1 is available here: > http://www.citi.umich.edu/projects/nfsv4/windows/readme.html > > pnfs is another name for NFS4.X. It is presented as alternative to=20 > ceph and we get known terminology as MDS and OSD but without the self= =20 > healing part if I understand well my rapid look on the topic. (when I= =20 > say rapid look I mean ... 5 minutes spent in that... which is really=20 > small amount of time to get an accurate view on something) > > > starting from mount.ceph ... I know that mount.ceph does little but i= t=20 > is a great hint to know what ceph needs and do things. > Basically mount.ceph modprobe the ceph driver in the linux kernel the= n=20 > call mount with the line command passed args and the cephfs type as=20 > argument. Then the kernel does the work I don t understand yet what i= s=20 > the start calls that are made to the ceph driver but it seemed to me=20 > that is was relatively light. (a first impression compared to ceph-fu= se.) > > I think I will do both isolate source code from ceph-client kernel=20 > (cephfs module for linux kernel) and the one pointed by Sage starting= =20 > from client/fuse_ll.cc in ceph master branch. The common files betwin= =20 > those 2 extractions will be our core set of mandatory features. > > Then we try to compile with cygwin a cephfs client library . Then we=20 > will try to interface with a modified windows nfs 4.1 client or pnfs=20 > or any other that will accept to be compiled with gcc for win32... > > the fact that windows 8.1 is and windows 2012 are out of reach at the= =20 > moment is not a problem to me. > > Our first concern is to understand what is ceph protocol. Then adapt=20 > it to something that can be used on windows prior windows 8.1. Dokan=20 > fs if I remember well use too the WDK (windows driver dev-kit ) for i= t=20 > s compilation so possibly we will see the same limitations. > > We need to multiply our source of information by example regarding=20 > ceph-client (kernel or fuse, radosgw is on a different layer so I wil= l=20 > not try anything around it at first.) And we need to multiply our=20 > source of information by example regarding virtual file system=20 > technologies on windoes OS. > Alot of work but all of those available source code everyone point at= =20 > me will make our best solution. And in the end we will choose=20 > technologies knowing what we do and what concequencies they have. > > regards, > > > > > Regards > > signature > > *Alph=C3=A9 Salas* > Ingeniero T.I > > asalas@kepler.cl > > > On 11/07/13 11:29, Ketor D wrote: >> Hi Alphe: >> Yes Callback Filesystem is very expensive and can't open sour= ce. >> It's not a good choice for ceph4win. >> Another way for ceph4win maybe develop a kernel-mode fs like >> pnfs. pnfs has a kernel-mode windows client. I think you can read it= s >> src code and maybe migrating from ceph kernel client to windows kern= el >> fs is easier than from userspace ceph fuse client.And a kernel-mode = fs >> client has greater performance than userspace fs like ceph-fuse clie= nt >> and ceph kernel client. >> >> Regards. >> > On 11/07/13 11:50, Matt W. Benjamin wrote: >> Hi, >> >> The Window NFS v4.1 client is what we work on, so this may be good f= or >> code sharing. The license is lgplv2, like Ceph's. >> >> Something important to be aware of is that the client uses rdbss, wh= ich >> is a (partial) fsd abstraction that simplified implementation >> quite a bit, kind of like a mini driver. However, Microsoft's suppo= rt >> for rdbss has been in limbo for a bit. For example, to link with >> the rdbss symbols you can't use the Windows 8 driver kit--you'll nee= d >> to use the one for Windows 7. (There's a private rdbss2 used intern= ally >> by Microsoft's SMB implemenation. A the moment, 3rd party drivers >> can't use that.) >> >> We've been in communication with Microsoft about this issue, and kno= w of >> a few other fsds using it, but it could be a good thing for that=20 >> lobbying >> effort to have another user--or it could be a dead end :(. >> >> There are a couple of other choices if you're looking to go this rou= te, >> that I'm aware of (and we may need to take them too, if RDBSS has no >> way forward), but the required work could be a lot larger. >> >> Matt >> >> ----- "Ketor D" wrote: >> >>> Hi Alphe: >>> Yes Callback Filesystem is very expensive and can't open >>> source. >>> It's not a good choice for ceph4win. >>> Another way for ceph4win maybe develop a kernel-mode fs like >>> pnfs. pnfs has a kernel-mode windows client. I think you can read i= ts >>> src code and maybe migrating from ceph kernel client to windows >>> kernel >>> fs is easier than from userspace ceph fuse client.And a kernel-mode >>> fs >>> client has greater performance than userspace fs like ceph-fuse >>> client >>> and ceph kernel client. >>> >>> Regards. >>> >>> >>> >>> On Thu, Nov 7, 2013 at 8:13 PM, Alphe Salas Michels >>> wrote: >>>> Commercial libraries are a pain ... >>>> >>>> If we want the more permossive licence offered by callback file >>> system we >>>> have to buy it for 20.000 usd. Then we will have to provide a >>> backbox that >>>> we have no control upon and that will kill our product anytime the= y >>> want anf >>>> if they decide to stop their commercial activity we will be in the >>> same >>>> situation that with dokanfs but without having the source code of >>> the black >>>> box. If i have to spend 20 000 dollars i would prefere paying >>> someone to >>>> retake dokanfs or to write from scratch a dokanfs fuselike softwar= e >>> make it >>>> all shiny and pumpy fantastic and ready to plug to ceph client. >>>> >>>> I would prefere if people have to pay something to get access to >>> ceph4win >>>> that this money goes in ceph main branch pockets... Or as a gift y= ou >>> donante >>>> to ceph 10 dollars you get 2 free registration codes for >>> ceph4win... or >>>> something like that. >>>> >>>> If ceph4win as to be comercial then I would prefer delegate the ta= sk >>> to a >>>> company like south river technologies and their great product >>> webdrive. I >>>> would mininaly get involved in that project and simply buy the fin= al >>> product >>>> to sell it together with my ceph based product (which could be a >>> calxeda >>>> ceph box or something like that). >>>> >>>> I m open anyway to any proposition. But I doubt that callback >>> filesystem >>>> offers us a suitable solution in the way I see ceph4win to be spre= ad >>> and >>>> used... I m maybe wrong. And anything that will be done around >>> ceph4win will >>>> be public documented etc... And licensed the way that if someone >>> want to >>>> build a commercial solution on top of it, that would be a >>> possibility. >>>> My idea is to giveback somehow to ceph project and at same time >>> forge a >>>> better knowledge in ceph technologies. Because like many in libre >>> world I >>>> think the business is in the services around the software more tha= n >>> on the >>>> software. That the ones writing code should be financed and benefi= ts >>> from >>>> the one selling and giving support of the software at all levels. = I >>> m >>>> probably too idealistic. And too optimistic after all I m the one >>> saying I >>>> will do this stuff I have no idea how but well it is interesting a= nd >>> fun so >>>> lets do it. >>>> >>>> Regards, >>>> >>>> P.S: using commercial backend libraries appart including their own >>> cost will >>>> force you to use commercial IDE like MS VisualStudio because their >>> library >>>> has some kind of drm that only that IDE compiler can use. So alot = of >>> cost >>>> and yet there is nothing done. If I had to open a kickstarter >>> project saying >>>> we need 60 000 USD to do ceph4win with that monney we will buy the >>> right to >>>> use and share a commercial copyrighted library but abandonned >>> punctually to >>>> us in public domaine and that we will eventually produce somethin= g >>> out of >>>> it. I doubt I will get a dollar. >>>> >>>> We still can suggest the idea to Edlos the commercial company that >>> has the >>>> copyright of Callback FS, Or to buy them their product in a blende= r >>> way >>>> (blender was bought with donation before being put opensource and >>> public >>>> domaine), Or to open source their library. But in commercial minds >>>> opensourcing =3D death of their technical advantage and death of >>> their >>>> marketing strategy. They will have to invent something more to >>> retrieve >>>> monney from it. >>>> >>>> El nov 6, 2013 11:22 p.m., "Ketor D" >>> > escribi=C3=B3: >>>> >>>> >>>> Hi Alphe, >>>> I think you could try Callback Filesystem dev >>> framework. It >>>> is a commerical dev framework and is maintained by Edlos today= =2E >>>> I have communicated with Edlos to get a try code for >>>> development. To dokan, Callback Filesystem has vary document a= nd >>> maybe >>>> more stabilize. >>>> >>>> Regards. >>>> >>>> >>>> >>>> On Thu, Nov 7, 2013 at 10:00 AM, Alphe Salas >>> > wrote: >>>> > Hello ketor thank you for your interest un ceph4win. Since >>> muy >>>> first mail I >>>> > exposed the lacks of dokanfs and that I m far from being a >>>> specialist un >>>> > filesystems. >>>> > I exposed what i like un dokanfs bit I not a fan=C3=A1tic o= f it. >>> Muy >>>> goal is to >>>> > have something working quickly. >>>> > >>>> > So I am up to any proposici=C3=B3n sure the one with the mo= re docs >>> and >>>> support >>>> > will be the best choice. As for right now what I need is >>>> understand what are >>>> > the files involved what are the interfaces functions and wh= at >>> are >>>> the needed >>>> > library dependencies and if they exist ported to windows wi= th >>>> cygwin. And >>>> > all that is retrieved from source code. >>>> > >>>> > Regards. >>>> > >>>> > El nov 6, 2013 10:34 p.m., "Ketor D" >>> > escribi=C3=B3: >>>> >>>> > >>>> >> Hi Alphe, >>>> >> We are taking an interest in your work on Ceph Client >>> for >>>> Windows >>>> >> with Dokan.As we know, the performance of Dokan is not ver= y >>>> good, and it's >>>> >> abandoned 3 years ago. >>>> >> I have learned and used OpenDedup(SDFS) for a long >>> time. >>>> OpenDedup >>>> >> has a Dokan version. And the author of OpenDedup said >>>> >> >>>> >> The Dokan library is quite flakey and testing should be >>>> performed before >>>> >> putting into production >>>> >> >>>> >> So what do you think about this? And if there is >>> another >>>> solution of >>>> >> fuse-like filesystem dev framwork on Windows? >>>> >> >>>> >> Best Wish! >>>> >> >>>> >> >>>> >> >>>> >> On Thu, Nov 7, 2013 at 5:47 AM, Alphe Salas Michels >>>> > >>>> >>>> >> wrote: >>>> >>> >>>> >>> Hello I created the github repository for this project >>>> >>>https://github.com/alphe/Ceph4Win >>>> >>> >>>> >>> Regards, >>>> >>> >>>> >>> signature >>>> >>> >>>> >>> *Alph=C3=A9 Salas* >>>> >>> Ingeniero T.I >>>> >>> >>>> >>>asalas@kepler.cl >>>> >>>> >>> ** >>>> >>> >>>> >>> On 11/05/13 21:00, Sage Weil wrote: >>>> >>>> >>>> >>>> Hi Alphe, >>>> >>>> >>>> >>>> On Tue, 5 Nov 2013, Alphe Salas Michels wrote: >>>> >>>>> >>>> >>>>> signature *Hi, Sage ! >>>> >>>>> thank you for you enthousiast reply. >>>> >>>>> I sure want to make the best use of everything or >>> anything >>>> previously >>>> >>>>> done to >>>> >>>>> tend to >>>> >>>>> write ceph cliente for windows. >>>> >>>>> >>>> >>>>> Apart using libre tools for building the future ceph >>> cliente >>>> I am open >>>> >>>>> to >>>> >>>>> anything. >>>> >>>>> I would recommand eclipse CDT or Code::BLocks they are >>> based >>>> on mingwin >>>> >>>>> open >>>> >>>>> and easyly enhanceable.** >>>> >>>>> >>>> >>>>> more free tools can be found here: >>>> >>>>>http://www.freebyte.com/programming/cpp/#cppcompilers >>>> >>>>> >>>> >>>>> >>>> >>>>> I will read libcephfs source code and take some notes >>> about the >>>> >>>>> protocol. >>>> >>>> >>>> >>>> I think you don't need to worry about hte protocol at al= l, >>> since >>>> >>>> libcephs >>>> >>>> implements it for you (and will capture any future >>> changes). >>>> >>>> >>>> >>>>> I was more going from what I know and trying to track d= own >>> how >>>> >>>>> mount.ceph work >>>> >>>>> with the parameters passed to it. >>>> >>>>> since it point finally to Kernel/fs/ceph and that I don= t >>> really >>>> >>>>> understand >>>> >>>>> how that module work and that it probably points to som= e >>> other >>>> >>>>> dependencies >>>> >>>>> Reading libcephfs source code could be a big gain of >>> time. >>>> >>>> >>>> >>>> (I would also ignore mount.ceph as everything it does it >>>> specific to >>>> >>>> how Linux mounts work.) >>>> >>>> >>>> >>>>> basically on the protocol what is need are: >>>> >>>>> >>>> >>>>> 1) open and maintain a connection (socket open, auth, e= tc >>> ) >>>> >>>>> 2) retreive a map of directories and disk Quota (disk >>> sizing >>>> Y TB free, >>>> >>>>> Z TB >>>> >>>>> total) >>>> >>>>> 3) procedure to send files / directories in a maner tha= t >>> it >>>> will allow >>>> >>>>> our >>>> >>>>> client to fit ceph transmission protocols >>>> >>>>> (limit bandwith for stability?, limit connection amount= ?, >>>> limit cpu >>>> >>>>> use?, >>>> >>>>> Cache for preparing data transfer (a FIFO cache)?) >>>> >>>>> 4)Procedure to retreive files / directory from ceph >>> cluster >>>> >>>>> 5) Management copy/move files /Directories, FS stats, >>>> Connection Stats. >>>> >>>>> logging. >>>> >>>>> >>>> >>>>> My idea to progress is to take those main bulletpoint i= n >>> ceph >>>> protocol >>>> >>>>> based >>>> >>>>> on general ideas of what ceph file system does and star= t >>>> identifying >>>> >>>>> parts >>>> >>>>> from libcephfs to match those "needs". >>>> >>>> >>>> >>>> Instead, I would look at include/cephfs/libcephfs.h, the >>>> interface that >>>> >>>> libcephfs provides, and try to map that to what the fuse >>> layer >>>> expects. >>>> >>>> There is both a path-based that I suspsect lends itself >>> well >>>> to the >>>> >>>> Windows interface and (very soon now) a handle based API >>> that is >>>> >>>> targetted >>>> >>>> at the Unix-style VFS layers. I'm mostly guessing, >>> though, >>>> since I've >>>> >>>> never seen any low-level fs code in windows before. >>>> >>>> >>>> >>>> In this case, the analogous code for Linux should be >>>> client/fuse_ll.cc >>>> >>>> itself (and not much else), although there will probably= be >>> a >>>> few tricks >>>> >>>> necessary to map cleanly onto how the windows interfaces >>> work. >>>> >>>> >>>> >>>> Does that make sense? >>>> >>>> >>>> >>>> Cheers! >>>> >>>> sage >>>> >>>> >>>> >>>> >>>> >>>>> Any suggestion and contributions are welcome. >>>> >>>>> >>>> >>>>> >>>> >>>>> * >>>> >>>>> On 11/05/13 11:23, Sage Weil wrote: >>>> >>>>>> >>>> >>>>>> Hi Alphe, >>>> >>>>>> >>>> >>>>>> On Mon, 4 Nov 2013, Alphe Salas Michels wrote: >>>> >>>>>>> >>>> >>>>>>> Good day developers! >>>> >>>>>>> >>>> >>>>>>> I would like to propose to the one interested work w= ith >>> me to >>>> >>>>>>> develop a >>>> >>>>>>> ceph >>>> >>>>>>> cliente for MS windows world, Basing us on dokanFS. >>>> >>>>>>> >>>> >>>>>>> My company is a ceph enthousiast that use on a dayly >>> basis >>>> ceph and >>>> >>>>>>> that >>>> >>>>>>> need >>>> >>>>>>> both transfer speed and big expendable and cheap >>> storage. >>>> >>>>>>> My company is specialised in data recovery and we wan= t >>> to >>>> participate >>>> >>>>>>> to >>>> >>>>>>> ceph >>>> >>>>>>> effort by bringing a ceph cliente for windows. >>>> >>>>>> >>>> >>>>>> Awesome! >>>> >>>>>> >>>> >>>>>>> Our experience shows us that the best gateway is each >>>> clientes being >>>> >>>>>>> its >>>> >>>>>>> own >>>> >>>>>>> gateway, instead of having a bottle neck server or a >>> cluster of >>>> >>>>>>> bottle >>>> >>>>>>> neck >>>> >>>>>>> servers as gateway (FTP, samba, SFTP,webdav, s3, >>> etc..). >>>> >>>>>>> >>>> >>>>>>> We already did some research in that domain. >>>> >>>>>>> >>>> >>>>>>> Dokan FS is an intent to write an opensource fuse li= ke >>>> cliente for >>>> >>>>>>> MS >>>> >>>>>>> windows. >>>> >>>>>>> >>>> >>>>>>> More information on DOKANFS can be triggered here >>>> >>>>>>>http://dokan-dev.net/en/download/ >>>> >>>>>>> >>>> >>>>>>> Positive points of using DOKANFS. >>>> >>>>>>> >>>> >>>>>>> - its opensourced and well licenced mit licence, gpl >>>> licence and lgpl >>>> >>>>>>> licence. >>>> >>>>>>> >>>> >>>>>>> Negative point of using DOKAN FS. >>>> >>>>>>> - unreachable author >>>> >>>>>>> - Poor documentation . Dev comments in japanese. >>>> >>>>>>> - Work in progress so it is unstable and needs to be >>> updated, >>>> >>>>>>> debugged and >>>> >>>>>>> maintained by a MS Windows file system expert >>> developper. >>>> >>>>>> >>>> >>>>>> I am not very familiar with windows storage APIs, but >>>> somebody told me >>>> >>>>>> at once point there were several interfaces against wh= ich >>> a >>>> new file >>>> >>>>>> system could be implemented, everything from a full >>>> in-kernel driver >>>> >>>>>> to >>>> >>>>>> something that is explorer-based. Are any of those >>>> suitable? Using a >>>> >>>>>> potentially abandoned fuse-like layer makes me a bit >>> nervous. >>>> >>>>>> >>>> >>>>>> That said, >>>> >>>>>> >>>> >>>>>>> >>>> >>>>>>> I try past year to do a merge from ceph-fuse to dokan= fs >>>> >>>>>>> here are what I learnt. >>>> >>>>>>> - Ceph-fuse and related source code is around 60 000 >>> lines >>>> of code. >>>> >>>>>>> - Ceph protocol isn t documented so it is like trying >>> to >>>> draw a map >>>> >>>>>>> of >>>> >>>>>>> america >>>> >>>>>>> using only a sextan and a compass. >>>> >>>>>>> >>>> >>>>>>> Those led me to those conclusions: >>>> >>>>>>> - I can t do it alone. >>>> >>>>>>> - It is easier to draw down the ceph protocol way to >>> work from >>>> >>>>>>> kernel/fs/ceph >>>> >>>>>>> sources and mount.ceph >>>> >>>>>>> - Ceph depending libraries may be unexistant or not u= p >>> to >>>> date in >>>> >>>>>>> their >>>> >>>>>>> port >>>> >>>>>>> on MS Windows (cygwin) >>>> >>>>>> >>>> >>>>>> I think the most sane path should be to make libcephfs >>>> sufficiently >>>> >>>>>> portable to build on windows (or cygwin). For the bit= s >>> used >>>> by the >>>> >>>>>> client-side coe, I don't think there should be much in >>> the >>>> way of >>>> >>>>>> dependencies, and the main challenge would be untangli= ng >>> the >>>> build for >>>> >>>>>> the necessary pieces out from the rest of Ceph. >>>> >>>>>> >>>> >>>>>> Have you seen the wip-port portability work that is >>>> currently underway >>>> >>>>>> by >>>> >>>>>> Noah and Alan? That may solve many of the cygwin >>> problems >>>> you are >>>> >>>>>> seeing >>>> >>>>>> today. >>>> >>>>>> >>>> >>>>>>> - MS file system specialist are hard do find in the >>> "open >>>> source >>>> >>>>>>> libre >>>> >>>>>>> world" >>>> >>>>>>> so I will try in the commercial world. >>>> >>>>>>> >>>> >>>>>>> The commercial world has some problems too. They need >>> ceph >>>> protocol >>>> >>>>>>> draft >>>> >>>>>>> to >>>> >>>>>>> implemente it to their own product They will have >>> licencing >>>> >>>>>>> /commercial >>>> >>>>>>> politics that infringe lgpl, and hide that most of th= e >>> work >>>> is done >>>> >>>>>>> by >>>> >>>>>>> people >>>> >>>>>>> other than them. They will not participate in a >>> financial >>>> way to ceph >>>> >>>>>>> enhancement and growth. >>>> >>>>>> >>>> >>>>>> I don't think reimplementing the client code is an >>> efficient way >>>> >>>>>> forward. >>>> >>>>>> Unless the goal is a pure kernel implementation...but = a >>>> significant >>>> >>>>>> ongoing investment in development resources would be >>> needed >>>> for that >>>> >>>>>> going >>>> >>>>>> forward. I suspect that is a challenge for a platform >>> that >>>> does not >>>> >>>>>> typically rally that sort of community effort. >>>> >>>>>> >>>> >>>>>> The easiest thing is of course just to use CIFS and >>> Samba >>>> (which works >>>> >>>>>> today). A fuse-like approach is probably a reasonably >>>> middle ground >>>> >>>>>> (both >>>> >>>>>> in initial effort and maintainability going forward)..= =2E >>>> >>>>>> >>>> >>>>>> sage >>>> >>>>>> >>>> >>>>>> >>>> >>>>> >>>> >>> >>>> >>> -- >>>> >>> To unsubscribe from this list: send the line "unsubscribe >>>> ceph-devel" in >>>> >>> the body of a message tomajordomo@vger.kernel.org >>>> >>>> >>>> >>> More majordomo info at >>> http://vger.kernel.org/majordomo-info.html >>>> >> >>>> >> >>>> > >>>> >>>> >>>> >>> --=20 >>> To unsubscribe from this list: send the line "unsubscribe ceph-deve= l" >>> in >>> the body of a message tomajordomo@vger.kernel.org >>> More majordomo info athttp://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" i= n the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html