* OSDs don't actually delete files when two CephFS (or more) are in use @ 2012-12-10 18:29 Geoffrey Hartz 2012-12-10 20:23 ` Sam Lang 0 siblings, 1 reply; 5+ messages in thread From: Geoffrey Hartz @ 2012-12-10 18:29 UTC (permalink / raw) To: ceph-devel Hi! I'm new to Ceph and I have a strange behavior with CephFS Config is : Ubuntu 12.04 Kernel 3.6.9 Ceph V0.55 2 OSD, 1 mon, 1 MDS, all on same host 2 clients, separate Hosts Ceph.conf: http://paste.ubuntu.com/1423712/ To mount the share I use : sudo ceph-fuse -m 192.168.80.139:6789 /mnt When I create a file on one client, the other see the file, can be downloaded etc. But when I delete the file, both clients don't see the file anymore BUT the file is still there on OSD (using space disk). When I umount from BOTH clients, OSD are update and file is actually delete (same behavior with mount -t ceph) I'm missing something? Thanks! -- Geoffrey HARTZ ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: OSDs don't actually delete files when two CephFS (or more) are in use 2012-12-10 18:29 OSDs don't actually delete files when two CephFS (or more) are in use Geoffrey Hartz @ 2012-12-10 20:23 ` Sam Lang 2012-12-10 20:34 ` Geoffrey Hartz 0 siblings, 1 reply; 5+ messages in thread From: Sam Lang @ 2012-12-10 20:23 UTC (permalink / raw) To: Geoffrey Hartz; +Cc: ceph-devel On 12/10/2012 08:29 AM, Geoffrey Hartz wrote: > Hi! > > I'm new to Ceph and I have a strange behavior with CephFS > > Config is : > > Ubuntu 12.04 > Kernel 3.6.9 > Ceph V0.55 > > 2 OSD, 1 mon, 1 MDS, all on same host > 2 clients, separate Hosts > > Ceph.conf: > > http://paste.ubuntu.com/1423712/ > > To mount the share I use : sudo ceph-fuse -m 192.168.80.139:6789 /mnt > > When I create a file on one client, the other see the file, can be > downloaded etc. > > But when I delete the file, both clients don't see the file anymore > BUT the file is still there on OSD (using space disk). Removing a file removes the directory entry (as you've seen), but the inode itself doesn't get removed until all references to it are dropped. The clients may cache the capability for those inodes for a period of time, so you're not seeing the references drop until they get evicted from the cache. Unmounting ensures that they get evicted from the client caches, so all references go to zero. Also, removal of the underlying objects is done lazily, so you may not see the space get freed up right away. -sam > > When I umount from BOTH clients, OSD are update and file is actually > delete (same behavior with mount -t ceph) > > I'm missing something? > > Thanks! > > -- > Geoffrey HARTZ > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: OSDs don't actually delete files when two CephFS (or more) are in use 2012-12-10 20:23 ` Sam Lang @ 2012-12-10 20:34 ` Geoffrey Hartz 2012-12-10 21:25 ` Sam Lang 0 siblings, 1 reply; 5+ messages in thread From: Geoffrey Hartz @ 2012-12-10 20:34 UTC (permalink / raw) To: Sam Lang; +Cc: ceph-devel Hi. I wait about more than 15 minutes. I was doing some benchmark when I noticed that the space never go back to normal.. How can I disable this behavior? It's on Ceph side? With one client, OSD are cleaned after few seconds, this sounds normal 2012/12/10 Sam Lang <sam.lang@inktank.com>: > On 12/10/2012 08:29 AM, Geoffrey Hartz wrote: >> >> Hi! >> >> I'm new to Ceph and I have a strange behavior with CephFS >> >> Config is : >> >> Ubuntu 12.04 >> Kernel 3.6.9 >> Ceph V0.55 >> >> 2 OSD, 1 mon, 1 MDS, all on same host >> 2 clients, separate Hosts >> >> Ceph.conf: >> >> http://paste.ubuntu.com/1423712/ >> >> To mount the share I use : sudo ceph-fuse -m 192.168.80.139:6789 /mnt >> >> When I create a file on one client, the other see the file, can be >> downloaded etc. >> >> But when I delete the file, both clients don't see the file anymore >> BUT the file is still there on OSD (using space disk). > > > Removing a file removes the directory entry (as you've seen), but the inode > itself doesn't get removed until all references to it are dropped. The > clients may cache the capability for those inodes for a period of time, so > you're not seeing the references drop until they get evicted from the cache. > Unmounting ensures that they get evicted from the client caches, so all > references go to zero. > > Also, removal of the underlying objects is done lazily, so you may not see > the space get freed up right away. > > -sam > >> >> When I umount from BOTH clients, OSD are update and file is actually >> delete (same behavior with mount -t ceph) >> >> I'm missing something? >> >> Thanks! >> >> -- >> Geoffrey HARTZ >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > -- Geoffrey HARTZ ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: OSDs don't actually delete files when two CephFS (or more) are in use 2012-12-10 20:34 ` Geoffrey Hartz @ 2012-12-10 21:25 ` Sam Lang 2012-12-10 22:08 ` Geoffrey Hartz 0 siblings, 1 reply; 5+ messages in thread From: Sam Lang @ 2012-12-10 21:25 UTC (permalink / raw) To: Geoffrey Hartz; +Cc: ceph-devel On 12/10/2012 10:34 AM, Geoffrey Hartz wrote: > Hi. > > I wait about more than 15 minutes. I was doing some benchmark when I > noticed that the space never go back to normal.. > > How can I disable this behavior? It's on Ceph side? > > With one client, OSD are cleaned after few seconds, this sounds normal Actually after discussing with Greg and Sage, it sounds like this is a bug. The issue is that the client that didn't remove the file is caching the dentry indefinitely. I've created a ticket to track the issue here: http://tracker.newdream.net/issues/3601. Unfortunately, I don't think there's a good workaround for the time being. You can try to evict those cached dentries by creating/accessing a bunch of other files, but the default cache size on the client is 16384, which is a lot of files to touch just to free up the space for those removed files. :-) You can decrease the cache size with the config option client_cache_size: [client] client cache size = 128 Then you only have to create/touch 128 files to evict the other files from the cache. That's not ideal, because reducing the cache size will affect your overall performance, but if you know that you won't be accessing a lot of files anyway, its probably your best bet. -sam > > 2012/12/10 Sam Lang <sam.lang@inktank.com>: >> On 12/10/2012 08:29 AM, Geoffrey Hartz wrote: >>> >>> Hi! >>> >>> I'm new to Ceph and I have a strange behavior with CephFS >>> >>> Config is : >>> >>> Ubuntu 12.04 >>> Kernel 3.6.9 >>> Ceph V0.55 >>> >>> 2 OSD, 1 mon, 1 MDS, all on same host >>> 2 clients, separate Hosts >>> >>> Ceph.conf: >>> >>> http://paste.ubuntu.com/1423712/ >>> >>> To mount the share I use : sudo ceph-fuse -m 192.168.80.139:6789 /mnt >>> >>> When I create a file on one client, the other see the file, can be >>> downloaded etc. >>> >>> But when I delete the file, both clients don't see the file anymore >>> BUT the file is still there on OSD (using space disk). >> >> >> Removing a file removes the directory entry (as you've seen), but the inode >> itself doesn't get removed until all references to it are dropped. The >> clients may cache the capability for those inodes for a period of time, so >> you're not seeing the references drop until they get evicted from the cache. >> Unmounting ensures that they get evicted from the client caches, so all >> references go to zero. >> >> Also, removal of the underlying objects is done lazily, so you may not see >> the space get freed up right away. >> >> -sam >> >>> >>> When I umount from BOTH clients, OSD are update and file is actually >>> delete (same behavior with mount -t ceph) >>> >>> I'm missing something? >>> >>> Thanks! >>> >>> -- >>> Geoffrey HARTZ >>> -- >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >> > > > ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: OSDs don't actually delete files when two CephFS (or more) are in use 2012-12-10 21:25 ` Sam Lang @ 2012-12-10 22:08 ` Geoffrey Hartz 0 siblings, 0 replies; 5+ messages in thread From: Geoffrey Hartz @ 2012-12-10 22:08 UTC (permalink / raw) To: Sam Lang; +Cc: ceph-devel Ok! I'm not in production yet. Only testing. In about 1/2months I will use in real (waiting for v0.56 bobtail) Thanks for your consideration! really appreciate. I have an other strange behavior between CephFS and kernel mount With ceph-fuse command : Client 1 create 2000 files and do "ls" => 1/2sec Client 2 do "ls" => 1/2sec With mount -t ceph ... Client 1 create 2000 files and do "ls" => few ms Client 2 do "ls" => 4-5sec 2012/12/10 Sam Lang <sam.lang@inktank.com>: > On 12/10/2012 10:34 AM, Geoffrey Hartz wrote: >> >> Hi. >> >> I wait about more than 15 minutes. I was doing some benchmark when I >> noticed that the space never go back to normal.. >> >> How can I disable this behavior? It's on Ceph side? >> >> With one client, OSD are cleaned after few seconds, this sounds normal > > > Actually after discussing with Greg and Sage, it sounds like this is a bug. > The issue is that the client that didn't remove the file is caching the > dentry indefinitely. I've created a ticket to track the issue here: > http://tracker.newdream.net/issues/3601. > > Unfortunately, I don't think there's a good workaround for the time being. > You can try to evict those cached dentries by creating/accessing a bunch of > other files, but the default cache size on the client is 16384, which is a > lot of files to touch just to free up the space for those removed files. :-) > > You can decrease the cache size with the config option client_cache_size: > > [client] > client cache size = 128 > > Then you only have to create/touch 128 files to evict the other files from > the cache. That's not ideal, because reducing the cache size will affect > your overall performance, but if you know that you won't be accessing a lot > of files anyway, its probably your best bet. > > -sam > > >> >> 2012/12/10 Sam Lang <sam.lang@inktank.com>: >>> >>> On 12/10/2012 08:29 AM, Geoffrey Hartz wrote: >>>> >>>> >>>> Hi! >>>> >>>> I'm new to Ceph and I have a strange behavior with CephFS >>>> >>>> Config is : >>>> >>>> Ubuntu 12.04 >>>> Kernel 3.6.9 >>>> Ceph V0.55 >>>> >>>> 2 OSD, 1 mon, 1 MDS, all on same host >>>> 2 clients, separate Hosts >>>> >>>> Ceph.conf: >>>> >>>> http://paste.ubuntu.com/1423712/ >>>> >>>> To mount the share I use : sudo ceph-fuse -m 192.168.80.139:6789 /mnt >>>> >>>> When I create a file on one client, the other see the file, can be >>>> downloaded etc. >>>> >>>> But when I delete the file, both clients don't see the file anymore >>>> BUT the file is still there on OSD (using space disk). >>> >>> >>> >>> Removing a file removes the directory entry (as you've seen), but the >>> inode >>> itself doesn't get removed until all references to it are dropped. The >>> clients may cache the capability for those inodes for a period of time, >>> so >>> you're not seeing the references drop until they get evicted from the >>> cache. >>> Unmounting ensures that they get evicted from the client caches, so all >>> references go to zero. >>> >>> Also, removal of the underlying objects is done lazily, so you may not >>> see >>> the space get freed up right away. >>> >>> -sam >>> >>>> >>>> When I umount from BOTH clients, OSD are update and file is actually >>>> delete (same behavior with mount -t ceph) >>>> >>>> I'm missing something? >>>> >>>> Thanks! >>>> >>>> -- >>>> Geoffrey HARTZ >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>>> the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> >>> >> >> >> > -- Geoffrey HARTZ ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2012-12-10 22:08 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-12-10 18:29 OSDs don't actually delete files when two CephFS (or more) are in use Geoffrey Hartz 2012-12-10 20:23 ` Sam Lang 2012-12-10 20:34 ` Geoffrey Hartz 2012-12-10 21:25 ` Sam Lang 2012-12-10 22:08 ` Geoffrey Hartz
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.