* BlueStore deep-dive over bluejeans @ 2016-02-24 19:51 Sage Weil 2016-02-25 13:59 ` Erwan Velu 2016-02-29 21:01 ` Sage Weil 0 siblings, 2 replies; 14+ messages in thread From: Sage Weil @ 2016-02-24 19:51 UTC (permalink / raw) To: ceph-devel Hey all, I'm planning on doing an hour-ish deep-dive into the inner workings of BlueStore next Monday, 2/29, 5pm-6pm ET / 2pm-3pm PT. It'll be a mostly-informal brain dump and walk-through of the code over bluejeans, and we'll record it. Anyone who is or is planning to be involved in bluestore development, please join and ask questions etc! I'll send out the bluejeans link shortly before we start. sage ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: BlueStore deep-dive over bluejeans 2016-02-24 19:51 BlueStore deep-dive over bluejeans Sage Weil @ 2016-02-25 13:59 ` Erwan Velu 2016-02-29 21:01 ` Sage Weil 1 sibling, 0 replies; 14+ messages in thread From: Erwan Velu @ 2016-02-25 13:59 UTC (permalink / raw) To: Sage Weil; +Cc: ceph-devel I cannot attend but will watch the recording. If you can paste it on this thread, would be nice. Thx, ----- Original Message ----- From: "Sage Weil" <sweil@redhat.com> To: ceph-devel@vger.kernel.org Sent: Wednesday, February 24, 2016 8:51:39 PM Subject: BlueStore deep-dive over bluejeans Hey all, I'm planning on doing an hour-ish deep-dive into the inner workings of BlueStore next Monday, 2/29, 5pm-6pm ET / 2pm-3pm PT. It'll be a mostly-informal brain dump and walk-through of the code over bluejeans, and we'll record it. Anyone who is or is planning to be involved in bluestore development, please join and ask questions etc! I'll send out the bluejeans link shortly before we start. sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: BlueStore deep-dive over bluejeans 2016-02-24 19:51 BlueStore deep-dive over bluejeans Sage Weil 2016-02-25 13:59 ` Erwan Velu @ 2016-02-29 21:01 ` Sage Weil 2016-03-01 15:19 ` Sage Weil 1 sibling, 1 reply; 14+ messages in thread From: Sage Weil @ 2016-02-29 21:01 UTC (permalink / raw) To: ceph-devel On Wed, 24 Feb 2016, Sage Weil wrote: > Hey all, > > I'm planning on doing an hour-ish deep-dive into the inner workings of > BlueStore next Monday, 2/29, 5pm-6pm ET / 2pm-3pm PT. It'll be a > mostly-informal brain dump and walk-through of the code over bluejeans, > and we'll record it. Anyone who is or is planning to be involved in > bluestore development, please join and ask questions etc! I'll send out > the bluejeans link shortly before we start. I'm going to defer this until later--I'm not feeling well. I'll send out another email soon with a new time slot. Sorry! sage ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: BlueStore deep-dive over bluejeans 2016-02-29 21:01 ` Sage Weil @ 2016-03-01 15:19 ` Sage Weil 2016-03-02 13:36 ` Sage Weil 0 siblings, 1 reply; 14+ messages in thread From: Sage Weil @ 2016-03-01 15:19 UTC (permalink / raw) To: ceph-devel On Mon, 29 Feb 2016, Sage Weil wrote: > On Wed, 24 Feb 2016, Sage Weil wrote: > > Hey all, > > > > I'm planning on doing an hour-ish deep-dive into the inner workings of > > BlueStore next Monday, 2/29, 5pm-6pm ET / 2pm-3pm PT. It'll be a > > mostly-informal brain dump and walk-through of the code over bluejeans, > > and we'll record it. Anyone who is or is planning to be involved in > > bluestore development, please join and ask questions etc! I'll send out > > the bluejeans link shortly before we start. > > I'm going to defer this until later--I'm not feeling well. I'll send out > another email soon with a new time slot. We try again with the same time slot today (5pm ET, 2pm PT). The bluejeans URL will be https://bluejeans.com/SWeil09 sage ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: BlueStore deep-dive over bluejeans 2016-03-01 15:19 ` Sage Weil @ 2016-03-02 13:36 ` Sage Weil 2016-03-02 13:55 ` Sage Weil 0 siblings, 1 reply; 14+ messages in thread From: Sage Weil @ 2016-03-02 13:36 UTC (permalink / raw) To: ceph-devel On Tue, 1 Mar 2016, Sage Weil wrote: > On Mon, 29 Feb 2016, Sage Weil wrote: > > On Wed, 24 Feb 2016, Sage Weil wrote: > > > Hey all, > > > > > > I'm planning on doing an hour-ish deep-dive into the inner workings of > > > BlueStore next Monday, 2/29, 5pm-6pm ET / 2pm-3pm PT. It'll be a > > > mostly-informal brain dump and walk-through of the code over bluejeans, > > > and we'll record it. Anyone who is or is planning to be involved in > > > bluestore development, please join and ask questions etc! I'll send out > > > the bluejeans link shortly before we start. > > > > I'm going to defer this until later--I'm not feeling well. I'll send out > > another email soon with a new time slot. > > We try again with the same time slot today (5pm ET, 2pm PT). The > bluejeans URL will be > > https://bluejeans.com/SWeil09 Here's the recording: https://trello.com/b/Zc0QlFzs/rhcs-2-0 sage ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: BlueStore deep-dive over bluejeans 2016-03-02 13:36 ` Sage Weil @ 2016-03-02 13:55 ` Sage Weil 2016-03-03 4:08 ` Dan Mick 0 siblings, 1 reply; 14+ messages in thread From: Sage Weil @ 2016-03-02 13:55 UTC (permalink / raw) To: ceph-devel On Wed, 2 Mar 2016, Sage Weil wrote: > On Tue, 1 Mar 2016, Sage Weil wrote: > > On Mon, 29 Feb 2016, Sage Weil wrote: > > > On Wed, 24 Feb 2016, Sage Weil wrote: > > > > Hey all, > > > > > > > > I'm planning on doing an hour-ish deep-dive into the inner workings of > > > > BlueStore next Monday, 2/29, 5pm-6pm ET / 2pm-3pm PT. It'll be a > > > > mostly-informal brain dump and walk-through of the code over bluejeans, > > > > and we'll record it. Anyone who is or is planning to be involved in > > > > bluestore development, please join and ask questions etc! I'll send out > > > > the bluejeans link shortly before we start. > > > > > > I'm going to defer this until later--I'm not feeling well. I'll send out > > > another email soon with a new time slot. > > > > We try again with the same time slot today (5pm ET, 2pm PT). The > > bluejeans URL will be > > > > https://bluejeans.com/SWeil09 > > Here's the recording: Oops, here's the correct URL: https://bluejeans.com/s/9dck/ sage ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: BlueStore deep-dive over bluejeans 2016-03-02 13:55 ` Sage Weil @ 2016-03-03 4:08 ` Dan Mick 2016-03-03 4:14 ` Haomai Wang 2016-03-03 13:19 ` Sage Weil 0 siblings, 2 replies; 14+ messages in thread From: Dan Mick @ 2016-03-03 4:08 UTC (permalink / raw) To: Sage Weil, ceph-devel > https://bluejeans.com/s/9dck/ Can you set this to allow download? I can't get it to play in the bluejeans interface, but downloadable files usually work for me -- Dan Mick Red Hat, Inc. Ceph docs: http://ceph.com/docs ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: BlueStore deep-dive over bluejeans 2016-03-03 4:08 ` Dan Mick @ 2016-03-03 4:14 ` Haomai Wang 2016-03-03 13:19 ` Sage Weil 1 sibling, 0 replies; 14+ messages in thread From: Haomai Wang @ 2016-03-03 4:14 UTC (permalink / raw) To: Dan Mick; +Cc: Sage Weil, ceph-devel@vger.kernel.org On Thu, Mar 3, 2016 at 12:08 PM, Dan Mick <dmick@redhat.com> wrote: > >> https://bluejeans.com/s/9dck/ > > Can you set this to allow download? I can't get it to play in the > bluejeans interface, but downloadable files usually work for me yes, I can't play it online too. > > > -- > Dan Mick > Red Hat, Inc. > Ceph docs: http://ceph.com/docs > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: BlueStore deep-dive over bluejeans 2016-03-03 4:08 ` Dan Mick 2016-03-03 4:14 ` Haomai Wang @ 2016-03-03 13:19 ` Sage Weil 2016-03-05 10:41 ` huang jun 1 sibling, 1 reply; 14+ messages in thread From: Sage Weil @ 2016-03-03 13:19 UTC (permalink / raw) To: Dan Mick; +Cc: ceph-devel On Wed, 2 Mar 2016, Dan Mick wrote: > > > https://bluejeans.com/s/9dck/ > > Can you set this to allow download? I can't get it to play in the > bluejeans interface, but downloadable files usually work for me Fixed! sage ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: BlueStore deep-dive over bluejeans 2016-03-03 13:19 ` Sage Weil @ 2016-03-05 10:41 ` huang jun 2016-03-05 18:54 ` Sage Weil [not found] ` <20160306184642014525100@chinanetcenter.com> 0 siblings, 2 replies; 14+ messages in thread From: huang jun @ 2016-03-05 10:41 UTC (permalink / raw) To: Sage Weil; +Cc: Dan Mick, ceph-devel where to get the ppt in the video? 2016-03-03 21:19 GMT+08:00 Sage Weil <sage@newdream.net>: > On Wed, 2 Mar 2016, Dan Mick wrote: >> >> > https://bluejeans.com/s/9dck/ >> >> Can you set this to allow download? I can't get it to play in the >> bluejeans interface, but downloadable files usually work for me > > Fixed! > > sage > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- thanks huangjun ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: BlueStore deep-dive over bluejeans 2016-03-05 10:41 ` huang jun @ 2016-03-05 18:54 ` Sage Weil [not found] ` <20160306184642014525100@chinanetcenter.com> 1 sibling, 0 replies; 14+ messages in thread From: Sage Weil @ 2016-03-05 18:54 UTC (permalink / raw) To: huang jun; +Cc: Dan Mick, ceph-devel On Sat, 5 Mar 2016, huang jun wrote: > where to get the ppt in the video? There's no PPT.. I was just sharing my screen and walking through some of the code. sage > > 2016-03-03 21:19 GMT+08:00 Sage Weil <sage@newdream.net>: > > On Wed, 2 Mar 2016, Dan Mick wrote: > >> > >> > https://bluejeans.com/s/9dck/ > >> > >> Can you set this to allow download? I can't get it to play in the > >> bluejeans interface, but downloadable files usually work for me > > > > Fixed! > > > > sage > > -- > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > -- > thanks > huangjun > > ^ permalink raw reply [flat|nested] 14+ messages in thread
[parent not found: <20160306184642014525100@chinanetcenter.com>]
* Re: Re: BlueStore deep-dive over bluejeans [not found] ` <20160306184642014525100@chinanetcenter.com> @ 2016-03-06 16:34 ` Sage Weil 2016-04-12 8:33 ` chen kael 0 siblings, 1 reply; 14+ messages in thread From: Sage Weil @ 2016-03-06 16:34 UTC (permalink / raw) To: 陈静; +Cc: huang jun, Dan Mick, ceph-devel [-- Attachment #1: Type: TEXT/PLAIN, Size: 2516 bytes --] Hi, On Sun, 6 Mar 2016, 陈静 wrote: > Hi Sage, > > So BlueStore will rely on RocksDB a lot, which just remind me of some > experience regarding to LevelDB recovery and compaction some time ago. > > We tried Ceph RGW 0.80, which had no Bucket Index Sharding feature. SSD > managed by FileStore is used for Bucket Index Pool. > After we put tens of millions of files into a bucket, LevelDB of the OSD > holding the corresponding bucket index object grew very big. > The OSD once crashed unexpectly. When we try to start the OSD again, it took > hours to fully get up. > We used pstack to observed what it was doing and noticed the OSD is busy in > recovering and compacting its LevelDB. > The recovering and compaction consumed a lot CPU and memory as well. > > BlueStore seems to rely more on RocksDB, which is a variant of LevelDB. > Then when the number of objects in BlueStore is large, will it take even > more time for the OSD to start up (for recoverying and compaction) if it was > not shut down cleanly previously? In general, rocksdb/leveldb shouldn't need to compact on startup. I'm not sure what caused that in your situation. We've seen a few leveldb bugs in the past that prevented compaction from happening when it should; perhaps it was one of those. We did do some testing with rocksdb where we inflated the size to he db to be very large and it performed pretty well. We didn't try clearing large swaths of the keyspace to test the compaction side of things, though. sage > > > Thanks, > Jeegn > > From: huang jun > Date: 2016-03-05 18:41 > To: Sage Weil > CC: Dan Mick; ceph-devel > Subject: Re: BlueStore deep-dive over bluejeans > where to get the ppt in the video? > > 2016-03-03 21:19 GMT+08:00 Sage Weil <sage@newdream.net>: > > On Wed, 2 Mar 2016, Dan Mick wrote: > >> > >> > https://bluejeans.com/s/9dck/ > >> > >> Can you set this to allow download? I can't get it to play in the > >> bluejeans interface, but downloadable files usually work for me > > > > Fixed! > > > > sage > > -- > > To unsubscribe from this list: send the line "unsubscribe > ceph-devel" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > -- > thanks > huangjun > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Re: BlueStore deep-dive over bluejeans 2016-03-06 16:34 ` Sage Weil @ 2016-04-12 8:33 ` chen kael 2016-04-12 17:43 ` Sage Weil 0 siblings, 1 reply; 14+ messages in thread From: chen kael @ 2016-04-12 8:33 UTC (permalink / raw) To: Sage Weil; +Cc: 陈静, huang jun, Dan Mick, ceph-devel Hi,sage I have a question, does small files smaller than min_alloc_size will be store in RocksDB for ever? And I am not quite clear about the differences between overlay write which is set off by default and WAL write. 2016-03-07 0:34 GMT+08:00 Sage Weil <sage@newdream.net>: > Hi, > > On Sun, 6 Mar 2016, 陈静 wrote: >> Hi Sage, >> >> So BlueStore will rely on RocksDB a lot, which just remind me of some >> experience regarding to LevelDB recovery and compaction some time ago. >> >> We tried Ceph RGW 0.80, which had no Bucket Index Sharding feature. SSD >> managed by FileStore is used for Bucket Index Pool. >> After we put tens of millions of files into a bucket, LevelDB of the OSD >> holding the corresponding bucket index object grew very big. >> The OSD once crashed unexpectly. When we try to start the OSD again, it took >> hours to fully get up. >> We used pstack to observed what it was doing and noticed the OSD is busy in >> recovering and compacting its LevelDB. >> The recovering and compaction consumed a lot CPU and memory as well. >> >> BlueStore seems to rely more on RocksDB, which is a variant of LevelDB. >> Then when the number of objects in BlueStore is large, will it take even >> more time for the OSD to start up (for recoverying and compaction) if it was >> not shut down cleanly previously? > > In general, rocksdb/leveldb shouldn't need to compact on startup. I'm not > sure what caused that in your situation. We've seen a few leveldb bugs in > the past that prevented compaction from happening when it should; perhaps > it was one of those. > > We did do some testing with rocksdb where we inflated the size to he db to > be very large and it performed pretty well. We didn't try clearing large > swaths of the keyspace to test the compaction side of things, though. > > sage > >> >> >> Thanks, >> Jeegn >> >> From: huang jun >> Date: 2016-03-05 18:41 >> To: Sage Weil >> CC: Dan Mick; ceph-devel >> Subject: Re: BlueStore deep-dive over bluejeans >> where to get the ppt in the video? >> >> 2016-03-03 21:19 GMT+08:00 Sage Weil <sage@newdream.net>: >> > On Wed, 2 Mar 2016, Dan Mick wrote: >> >> >> >> > https://bluejeans.com/s/9dck/ >> >> >> >> Can you set this to allow download? I can't get it to play in the >> >> bluejeans interface, but downloadable files usually work for me >> > >> > Fixed! >> > >> > sage >> > -- >> > To unsubscribe from this list: send the line "unsubscribe >> ceph-devel" in >> > the body of a message to majordomo@vger.kernel.org >> > More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> >> >> -- >> thanks >> huangjun >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" >> in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> >> -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Re: BlueStore deep-dive over bluejeans 2016-04-12 8:33 ` chen kael @ 2016-04-12 17:43 ` Sage Weil 0 siblings, 0 replies; 14+ messages in thread From: Sage Weil @ 2016-04-12 17:43 UTC (permalink / raw) To: chen kael; +Cc: 陈静, huang jun, Dan Mick, ceph-devel [-- Attachment #1: Type: TEXT/PLAIN, Size: 3895 bytes --] On Tue, 12 Apr 2016, chen kael wrote: > Hi,sage > I have a question, does small files smaller than min_alloc_size will > be store in RocksDB for ever? And I am not quite clear about the > differences between overlay write which is set off by default and WAL > write. A WAL write is included in the transaction written to rocksdb. Once that commits, the IO (usually an overwrite) is immediately queued, and the WAL record is removed from rocksdb during the next commit cycle. An overlay write is intended to stick around in rocksdb until some threshold is reached (N overlay records), at which point all of the IOs are done at once. The overlay path wasn't fully updated after the most recent rewrite, and prior to that didn't seem to make things any faster. I'm not sure if it's worth keeping around or not. In theory it should help with spinning disks, but in practice it doesn't seem to. sage > > 2016-03-07 0:34 GMT+08:00 Sage Weil <sage@newdream.net>: > > Hi, > > > > On Sun, 6 Mar 2016, 陈静 wrote: > >> Hi Sage, > >> > >> So BlueStore will rely on RocksDB a lot, which just remind me of some > >> experience regarding to LevelDB recovery and compaction some time ago. > >> > >> We tried Ceph RGW 0.80, which had no Bucket Index Sharding feature. SSD > >> managed by FileStore is used for Bucket Index Pool. > >> After we put tens of millions of files into a bucket, LevelDB of the OSD > >> holding the corresponding bucket index object grew very big. > >> The OSD once crashed unexpectly. When we try to start the OSD again, it took > >> hours to fully get up. > >> We used pstack to observed what it was doing and noticed the OSD is busy in > >> recovering and compacting its LevelDB. > >> The recovering and compaction consumed a lot CPU and memory as well. > >> > >> BlueStore seems to rely more on RocksDB, which is a variant of LevelDB. > >> Then when the number of objects in BlueStore is large, will it take even > >> more time for the OSD to start up (for recoverying and compaction) if it was > >> not shut down cleanly previously? > > > > In general, rocksdb/leveldb shouldn't need to compact on startup. I'm not > > sure what caused that in your situation. We've seen a few leveldb bugs in > > the past that prevented compaction from happening when it should; perhaps > > it was one of those. > > > > We did do some testing with rocksdb where we inflated the size to he db to > > be very large and it performed pretty well. We didn't try clearing large > > swaths of the keyspace to test the compaction side of things, though. > > > > sage > > > >> > >> > >> Thanks, > >> Jeegn > >> > >> From: huang jun > >> Date: 2016-03-05 18:41 > >> To: Sage Weil > >> CC: Dan Mick; ceph-devel > >> Subject: Re: BlueStore deep-dive over bluejeans > >> where to get the ppt in the video? > >> > >> 2016-03-03 21:19 GMT+08:00 Sage Weil <sage@newdream.net>: > >> > On Wed, 2 Mar 2016, Dan Mick wrote: > >> >> > >> >> > https://bluejeans.com/s/9dck/ > >> >> > >> >> Can you set this to allow download? I can't get it to play in the > >> >> bluejeans interface, but downloadable files usually work for me > >> > > >> > Fixed! > >> > > >> > sage > >> > -- > >> > To unsubscribe from this list: send the line "unsubscribe > >> ceph-devel" in > >> > the body of a message to majordomo@vger.kernel.org > >> > More majordomo info at http://vger.kernel.org/majordomo-info.html > >> > >> > >> > >> -- > >> thanks > >> huangjun > >> -- > >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" > >> in > >> the body of a message to majordomo@vger.kernel.org > >> More majordomo info at http://vger.kernel.org/majordomo-info.html > >> > >> > >> > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2016-04-12 17:43 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-02-24 19:51 BlueStore deep-dive over bluejeans Sage Weil
2016-02-25 13:59 ` Erwan Velu
2016-02-29 21:01 ` Sage Weil
2016-03-01 15:19 ` Sage Weil
2016-03-02 13:36 ` Sage Weil
2016-03-02 13:55 ` Sage Weil
2016-03-03 4:08 ` Dan Mick
2016-03-03 4:14 ` Haomai Wang
2016-03-03 13:19 ` Sage Weil
2016-03-05 10:41 ` huang jun
2016-03-05 18:54 ` Sage Weil
[not found] ` <20160306184642014525100@chinanetcenter.com>
2016-03-06 16:34 ` Sage Weil
2016-04-12 8:33 ` chen kael
2016-04-12 17:43 ` Sage Weil
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox