* Crush and Monitor questions @ 2012-12-11 20:23 Bryant Ng 2012-12-12 1:15 ` Bryant Ng 2012-12-12 17:37 ` Joao Eduardo Luis 0 siblings, 2 replies; 6+ messages in thread From: Bryant Ng @ 2012-12-11 20:23 UTC (permalink / raw) To: ceph-devel Hi, I'm pretty new to Ceph and am just learning about it. Where are the CRUSH maps stored in Ceph? In the documentation I see you use the 'crushtool' to compile and decompile the crush map. I understand that if a single monitor comes online, it can talk to the other existing monitors to get the cluster map but how does it work on initial startup? Or if the entire Ceph clusters goes down b/c of power failure or something. What is the recommended hardware configuration for monitors? In the Hardware Recommendation page it says "A monitor requires approximately 10GB of storage space per daemon instance." per daemon instance is talking about the monitor daemon, not the osd daemons? Also further down on that page, it list some hardware examples where it mentions a ligher configuration for monitors. I am assuming that is the Dell PE R510 which contains 8 x 2 TB drives. Why does the monitor need so much space if it's "10GB of storage space per daemon instance". -Bryant ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Crush and Monitor questions 2012-12-11 20:23 Crush and Monitor questions Bryant Ng @ 2012-12-12 1:15 ` Bryant Ng 2012-12-12 17:37 ` Joao Eduardo Luis 1 sibling, 0 replies; 6+ messages in thread From: Bryant Ng @ 2012-12-12 1:15 UTC (permalink / raw) To: ceph-devel Sorry, I misread the Hardware Configuration section of the Ceph documentation. I thought one of the Dell's was a configuration for the monitors but both of the Dell R510 and R515 are OSD configuration. I had another question on the monitors though. What kind of load (requests/second) can a monitor node handle? My understanding is that it just returns the cluster map to the client requesting it? The documentation mentions 3 to 5 monitors in a ceph cluster but what is the request rate expected on each of these monitors? thanks. Bryant Bryant Ng wrote: > Hi, > > I'm pretty new to Ceph and am just learning about it. > > Where are the CRUSH maps stored in Ceph? In the documentation I see you > use the 'crushtool' to compile and decompile the crush map. I > understand that if a single monitor comes online, it can talk to the > other existing monitors to get the cluster map but how does it work on > initial startup? Or if the entire Ceph clusters goes down b/c of power > failure or something. > > What is the recommended hardware configuration for monitors? In the > Hardware Recommendation page it says "A monitor requires approximately > 10GB of storage space per daemon instance." per daemon instance is > talking about the monitor daemon, not the osd daemons? > > Also further down on that page, it list some hardware examples where it > mentions a ligher configuration for monitors. I am assuming that is the > Dell PE R510 which contains 8 x 2 TB drives. Why does the monitor need > so much space if it's "10GB of storage space per daemon instance". > > -Bryant > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Crush and Monitor questions 2012-12-11 20:23 Crush and Monitor questions Bryant Ng 2012-12-12 1:15 ` Bryant Ng @ 2012-12-12 17:37 ` Joao Eduardo Luis 2012-12-12 19:02 ` Bryant Ng 1 sibling, 1 reply; 6+ messages in thread From: Joao Eduardo Luis @ 2012-12-12 17:37 UTC (permalink / raw) To: Bryant Ng; +Cc: ceph-devel Hello Bryant, On 12/11/2012 08:23 PM, Bryant Ng wrote: > Hi, > > I'm pretty new to Ceph and am just learning about it. > > Where are the CRUSH maps stored in Ceph? In the documentation I see you > use the 'crushtool' to compile and decompile the crush map. The crushmap is kept alongside with the osdmap, and shared by the monitors. > I understand that if a single monitor comes online, it can talk to the > other existing monitors to get the cluster map but how does it work on > initial startup? Or if the entire Ceph clusters goes down b/c of power > failure or something. If you add a new monitor to an existing cluster, it will synchronize with the existing monitors and will obtain all their infos, including the crushmap. Updates to the crushmap will also be shared among the monitors in the quorum. If you are starting up fresh, you will have to either add your custom crushmap to the monitors (using the ceph tool), or stick with the default crushmap (which only defines something along the lines of a 'default' root, a 'defaultrack' rack and a 'localhost' host). If the entire cluster goes down... well, if the monitors are not up they won't be able to share the crushmap. When they are brought back up, then they will pick up where they left. But I'm not sure if I understand what your question is. -Joao ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Crush and Monitor questions 2012-12-12 17:37 ` Joao Eduardo Luis @ 2012-12-12 19:02 ` Bryant Ng 2012-12-12 19:32 ` Joao Eduardo Luis 0 siblings, 1 reply; 6+ messages in thread From: Bryant Ng @ 2012-12-12 19:02 UTC (permalink / raw) To: ceph-devel I guess my question was where is the crushmap (and osdmap) persisted on the monitor node? If the entire cluster goes down, I assume the monitor is reading the crushmap from some persistent file stored on disk or a db? Is that why the minimum recommended storage for monitors is 10GB? Is the crushmap and osdmap stored in those 10GB? -Bryant Joao Eduardo Luis wrote: > Hello Bryant, > > On 12/11/2012 08:23 PM, Bryant Ng wrote: >> Hi, >> >> I'm pretty new to Ceph and am just learning about it. >> >> Where are the CRUSH maps stored in Ceph? In the documentation I see you >> use the 'crushtool' to compile and decompile the crush map. > > The crushmap is kept alongside with the osdmap, and shared by the monitors. > >> I understand that if a single monitor comes online, it can talk to the >> other existing monitors to get the cluster map but how does it work on >> initial startup? Or if the entire Ceph clusters goes down b/c of power >> failure or something. > > If you add a new monitor to an existing cluster, it will synchronize > with the existing monitors and will obtain all their infos, including > the crushmap. Updates to the crushmap will also be shared among the > monitors in the quorum. > > If you are starting up fresh, you will have to either add your custom > crushmap to the monitors (using the ceph tool), or stick with the > default crushmap (which only defines something along the lines of a > 'default' root, a 'defaultrack' rack and a 'localhost' host). > > If the entire cluster goes down... well, if the monitors are not up they > won't be able to share the crushmap. When they are brought back up, then > they will pick up where they left. But I'm not sure if I understand what > your question is. > > -Joao > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Crush and Monitor questions 2012-12-12 19:02 ` Bryant Ng @ 2012-12-12 19:32 ` Joao Eduardo Luis 2012-12-13 19:43 ` Bryant Ng 0 siblings, 1 reply; 6+ messages in thread From: Joao Eduardo Luis @ 2012-12-12 19:32 UTC (permalink / raw) To: Bryant Ng; +Cc: ceph-devel On 12/12/2012 07:02 PM, Bryant Ng wrote: > I guess my question was where is the crushmap (and osdmap) persisted on > the monitor node? > > If the entire cluster goes down, I assume the monitor is reading the > crushmap from some persistent file stored on disk or a db? Is that why > the minimum recommended storage for monitors is 10GB? Is the crushmap > and osdmap stored in those 10GB? > > -Bryant > The monitor maintains a 'store'. This is why you have to 'ceph-mon --mkfs' before you can run the monitor. The monitor store needs to room to grow, given that the monitor will store pretty much every update to the osdmap, monmap, crushmap, keyring,... Some of this info will also be pruned regularly though, but it's advised to keep enough space around. Hope this clarifies things. -Joao > Joao Eduardo Luis wrote: >> Hello Bryant, >> >> On 12/11/2012 08:23 PM, Bryant Ng wrote: >>> Hi, >>> >>> I'm pretty new to Ceph and am just learning about it. >>> >>> Where are the CRUSH maps stored in Ceph? In the documentation I see you >>> use the 'crushtool' to compile and decompile the crush map. >> >> The crushmap is kept alongside with the osdmap, and shared by the >> monitors. >> >>> I understand that if a single monitor comes online, it can talk to the >>> other existing monitors to get the cluster map but how does it work on >>> initial startup? Or if the entire Ceph clusters goes down b/c of power >>> failure or something. >> >> If you add a new monitor to an existing cluster, it will synchronize >> with the existing monitors and will obtain all their infos, including >> the crushmap. Updates to the crushmap will also be shared among the >> monitors in the quorum. >> >> If you are starting up fresh, you will have to either add your custom >> crushmap to the monitors (using the ceph tool), or stick with the >> default crushmap (which only defines something along the lines of a >> 'default' root, a 'defaultrack' rack and a 'localhost' host). >> >> If the entire cluster goes down... well, if the monitors are not up they >> won't be able to share the crushmap. When they are brought back up, then >> they will pick up where they left. But I'm not sure if I understand what >> your question is. >> >> -Joao >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Crush and Monitor questions 2012-12-12 19:32 ` Joao Eduardo Luis @ 2012-12-13 19:43 ` Bryant Ng 0 siblings, 0 replies; 6+ messages in thread From: Bryant Ng @ 2012-12-13 19:43 UTC (permalink / raw) To: ceph-devel Thanks Joao. Makes more sense now. What are your thoughts on my other question about expected load a monitor can handle? My understanding is that it just returns the cluster map to the client requesting it? The documentation mentions 3 to 5 monitors in a ceph cluster but what is the request rate expected on each of these monitors? If we are expecting a request rate of about 40 requests/second to the ceph cluster from clients, how many monitors would be needed to handle that? -Bryant Joao Eduardo Luis wrote: > On 12/12/2012 07:02 PM, Bryant Ng wrote: >> I guess my question was where is the crushmap (and osdmap) persisted on >> the monitor node? >> >> If the entire cluster goes down, I assume the monitor is reading the >> crushmap from some persistent file stored on disk or a db? Is that why >> the minimum recommended storage for monitors is 10GB? Is the crushmap >> and osdmap stored in those 10GB? >> >> -Bryant >> > > The monitor maintains a 'store'. This is why you have to 'ceph-mon > --mkfs' before you can run the monitor. > > The monitor store needs to room to grow, given that the monitor will > store pretty much every update to the osdmap, monmap, crushmap, keyring,... > > Some of this info will also be pruned regularly though, but it's advised > to keep enough space around. > > Hope this clarifies things. > > -Joao > >> Joao Eduardo Luis wrote: >>> Hello Bryant, >>> >>> On 12/11/2012 08:23 PM, Bryant Ng wrote: >>>> Hi, >>>> >>>> I'm pretty new to Ceph and am just learning about it. >>>> >>>> Where are the CRUSH maps stored in Ceph? In the documentation I see you >>>> use the 'crushtool' to compile and decompile the crush map. >>> >>> The crushmap is kept alongside with the osdmap, and shared by the >>> monitors. >>> >>>> I understand that if a single monitor comes online, it can talk to the >>>> other existing monitors to get the cluster map but how does it work on >>>> initial startup? Or if the entire Ceph clusters goes down b/c of power >>>> failure or something. >>> >>> If you add a new monitor to an existing cluster, it will synchronize >>> with the existing monitors and will obtain all their infos, including >>> the crushmap. Updates to the crushmap will also be shared among the >>> monitors in the quorum. >>> >>> If you are starting up fresh, you will have to either add your custom >>> crushmap to the monitors (using the ceph tool), or stick with the >>> default crushmap (which only defines something along the lines of a >>> 'default' root, a 'defaultrack' rack and a 'localhost' host). >>> >>> If the entire cluster goes down... well, if the monitors are not up they >>> won't be able to share the crushmap. When they are brought back up, then >>> they will pick up where they left. But I'm not sure if I understand what >>> your question is. >>> >>> -Joao >>> -- >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >> >> >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2012-12-13 19:44 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-12-11 20:23 Crush and Monitor questions Bryant Ng 2012-12-12 1:15 ` Bryant Ng 2012-12-12 17:37 ` Joao Eduardo Luis 2012-12-12 19:02 ` Bryant Ng 2012-12-12 19:32 ` Joao Eduardo Luis 2012-12-13 19:43 ` Bryant Ng
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.