* [Ocfs2-devel] Read IOPS storm in case of reflinking running VM disk @ 2015-05-08 5:56 Eugene Istomin 2015-05-11 8:48 ` Eugene Istomin 0 siblings, 1 reply; 6+ messages in thread From: Eugene Istomin @ 2015-05-08 5:56 UTC (permalink / raw) To: ocfs2-devel Hello, after deploying reflink-based VM snapshots to production servers we discovered a performace degradation: OS: Opensuse 13.1, 13.2 Hypervisors: Xen 4.4, 4.5 Dom0 kernels: 3.12, 3.16, 3.18 DomU kernels: 3.12, 3.16, 3.18 Tested DomU disk backends: tapdisk2, qdisk 1) on DomU (VM) #dd if=/dev/zero of=test2 bs=1M count=6000 2) atop on Dom0: sdb - busy:92% - read:375 - write:130902 Reads are from others VMs, seems OK 3) DomU dd finished: 6291456000 bytes (6.3 GB) copied, 16.6265 s, 378 MB/s 4) Lets start dd again & do a snapshot: #dd if=/dev/zero of=test2 bs=1M count=6000 #reflink test.raw ref/ 5) atop on Dom0: sdb - busy:97% - read:112740 - write:28037 So, Read IOPS = 112740, why? 6) DomU dd finished: 6291456000 bytes (6.3 GB) copied, 175.45 s, 35.9 MB/s 7) Second & further reflinks do not change the atop stat & dd time #dd if=/dev/zero of=test2 bs=1M count=6000 #reflink --backup=t test.raw ref/ \\ * n times ~ 6291456000 bytes (6.3 GB) copied, 162.959 s, 38.6 MB/s The question is why reflinking a running VM disk leads to read IOPS storm? Thanks! -- Best regards, Eugene Istomin ^ permalink raw reply [flat|nested] 6+ messages in thread
* [Ocfs2-devel] Read IOPS storm in case of reflinking running VM disk 2015-05-08 5:56 [Ocfs2-devel] Read IOPS storm in case of reflinking running VM disk Eugene Istomin @ 2015-05-11 8:48 ` Eugene Istomin 2015-05-18 10:05 ` Eugene Istomin 0 siblings, 1 reply; 6+ messages in thread From: Eugene Istomin @ 2015-05-11 8:48 UTC (permalink / raw) To: ocfs2-devel Hello Goldwyn, Do you know something about such behavior? The question is why a reflink operation on VM disk leads to plenty of read ops? Is this related to CoW specific structures? We can provide others details & ssh to testbed. -- Best regards, Eugene Istomin On Friday, May 08, 2015 08:56:57 AM Eugene Istomin wrote: > Hello, > > after deploying reflink-based VM snapshots to production servers we > discovered a performace degradation: > > OS: Opensuse 13.1, 13.2 > Hypervisors: Xen 4.4, 4.5 > Dom0 kernels: 3.12, 3.16, 3.18 > DomU kernels: 3.12, 3.16, 3.18 > Tested DomU disk backends: tapdisk2, qdisk > > > 1) on DomU (VM) > #dd if=/dev/zero of=test2 bs=1M count=6000 > > 2) atop on Dom0: > sdb - busy:92% - read:375 - write:130902 > Reads are from others VMs, seems OK > > 3) DomU dd finished: > 6291456000 bytes (6.3 GB) copied, 16.6265 s, 378 MB/s > > 4) Lets start dd again & do a snapshot: > #dd if=/dev/zero of=test2 bs=1M count=6000 > #reflink test.raw ref/ > > 5) atop on Dom0: > sdb - busy:97% - read:112740 - write:28037 > So, Read IOPS = 112740, why? > > 6) DomU dd finished: > 6291456000 bytes (6.3 GB) copied, 175.45 s, 35.9 MB/s > > 7) Second & further reflinks do not change the atop stat & dd time > #dd if=/dev/zero of=test2 bs=1M count=6000 > #reflink --backup=t test.raw ref/ \\ * n times > ~ 6291456000 bytes (6.3 GB) copied, 162.959 s, 38.6 MB/s > > The question is why reflinking a running VM disk leads to read IOPS storm? > > > Thanks! ^ permalink raw reply [flat|nested] 6+ messages in thread
* [Ocfs2-devel] Read IOPS storm in case of reflinking running VM disk 2015-05-11 8:48 ` Eugene Istomin @ 2015-05-18 10:05 ` Eugene Istomin 2015-05-18 17:45 ` Goldwyn Rodrigues 0 siblings, 1 reply; 6+ messages in thread From: Eugene Istomin @ 2015-05-18 10:05 UTC (permalink / raw) To: ocfs2-devel Hello, ping -- Best regards, Eugene Istomin On Monday, May 11, 2015 11:48:11 AM Eugene Istomin wrote: > Hello Goldwyn, > > Do you know something about such behavior? > The question is why a reflink operation on VM disk leads to plenty of read > ops? Is this related to CoW specific structures? > > We can provide others details & ssh to testbed. > > > Hello, > > > > after deploying reflink-based VM snapshots to production servers we > > discovered a performace degradation: > > > > OS: Opensuse 13.1, 13.2 > > Hypervisors: Xen 4.4, 4.5 > > Dom0 kernels: 3.12, 3.16, 3.18 > > DomU kernels: 3.12, 3.16, 3.18 > > Tested DomU disk backends: tapdisk2, qdisk > > > > > > 1) on DomU (VM) > > #dd if=/dev/zero of=test2 bs=1M count=6000 > > > > 2) atop on Dom0: > > sdb - busy:92% - read:375 - write:130902 > > Reads are from others VMs, seems OK > > > > 3) DomU dd finished: > > 6291456000 bytes (6.3 GB) copied, 16.6265 s, 378 MB/s > > > > 4) Lets start dd again & do a snapshot: > > #dd if=/dev/zero of=test2 bs=1M count=6000 > > #reflink test.raw ref/ > > > > 5) atop on Dom0: > > sdb - busy:97% - read:112740 - write:28037 > > So, Read IOPS = 112740, why? > > > > 6) DomU dd finished: > > 6291456000 bytes (6.3 GB) copied, 175.45 s, 35.9 MB/s > > > > 7) Second & further reflinks do not change the atop stat & dd time > > #dd if=/dev/zero of=test2 bs=1M count=6000 > > #reflink --backup=t test.raw ref/ \\ * n times > > ~ 6291456000 bytes (6.3 GB) copied, 162.959 s, 38.6 MB/s > > > > The question is why reflinking a running VM disk leads to read IOPS storm? > > > > > > Thanks! > > _______________________________________________ > Ocfs2-devel mailing list > Ocfs2-devel at oss.oracle.com > https://oss.oracle.com/mailman/listinfo/ocfs2-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20150518/fd9107e5/attachment.html ^ permalink raw reply [flat|nested] 6+ messages in thread
* [Ocfs2-devel] Read IOPS storm in case of reflinking running VM disk 2015-05-18 10:05 ` Eugene Istomin @ 2015-05-18 17:45 ` Goldwyn Rodrigues 2015-05-20 22:33 ` Eugene Istomin 0 siblings, 1 reply; 6+ messages in thread From: Goldwyn Rodrigues @ 2015-05-18 17:45 UTC (permalink / raw) To: ocfs2-devel Hi Eugene, Sorry, had been busy with other work and this slipped on the list. > > > Do you know something about such behavior? > > > The question is why a reflink operation on VM disk leads to plenty of > read > > > ops? Is this related to CoW specific structures? > This is in fact related to the CoW. An ocfs2 file is an extent tree, which the extent headers marking if the extent is a reflinked or not with the number of reflinks. If you perform a reflink on a file which is being changed constantly, not only recreate the extent tree, but also decrease the refcount of the ones already present. Add to it, the extents which need to be read for replication. HTH, > > > > > We can provide others details & ssh to testbed. > > > > > > > Hello, > > > > > > > > after deploying reflink-based VM snapshots to production servers we > > > > discovered a performace degradation: > > > > > > > > OS: Opensuse 13.1, 13.2 > > > > Hypervisors: Xen 4.4, 4.5 > > > > Dom0 kernels: 3.12, 3.16, 3.18 > > > > DomU kernels: 3.12, 3.16, 3.18 > > > > Tested DomU disk backends: tapdisk2, qdisk > > > > > > > > > > > > 1) on DomU (VM) > > > > #dd if=/dev/zero of=test2 bs=1M count=6000 > > > > > > > > 2) atop on Dom0: > > > > sdb - busy:92% - read:375 - write:130902 > > > > Reads are from others VMs, seems OK > > > > > > > > 3) DomU dd finished: > > > > 6291456000 bytes (6.3 GB) copied, 16.6265 s, 378 MB/s > > > > > > > > 4) Lets start dd again & do a snapshot: > > > > #dd if=/dev/zero of=test2 bs=1M count=6000 > > > > #reflink test.raw ref/ > > > > > > > > 5) atop on Dom0: > > > > sdb - busy:97% - read:112740 - write:28037 > > > > So, Read IOPS = 112740, why? > > > > > > > > 6) DomU dd finished: > > > > 6291456000 bytes (6.3 GB) copied, 175.45 s, 35.9 MB/s > > > > > > > > 7) Second & further reflinks do not change the atop stat & dd time > > > > #dd if=/dev/zero of=test2 bs=1M count=6000 > > > > #reflink --backup=t test.raw ref/ \\ * n times > > > > ~ 6291456000 bytes (6.3 GB) copied, 162.959 s, 38.6 MB/s > > > > > > > > The question is why reflinking a running VM disk leads to read IOPS > storm? > > > > > > > > > > > > Thanks! > > > > > > _______________________________________________ > > > Ocfs2-devel mailing list > > > Ocfs2-devel at oss.oracle.com > > > https://oss.oracle.com/mailman/listinfo/ocfs2-devel > -- Goldwyn ^ permalink raw reply [flat|nested] 6+ messages in thread
* [Ocfs2-devel] Read IOPS storm in case of reflinking running VM disk 2015-05-18 17:45 ` Goldwyn Rodrigues @ 2015-05-20 22:33 ` Eugene Istomin 2015-05-21 11:57 ` Goldwyn Rodrigues 0 siblings, 1 reply; 6+ messages in thread From: Eugene Istomin @ 2015-05-20 22:33 UTC (permalink / raw) To: ocfs2-devel Goldwyn, thanks for the answer! I read https://oss.oracle.com/osswiki/OCFS2(2f)DesignDocs(2f)RefcountTrees.html carefully to understand the problem. As i understand: There are B-Tree structures for reflink: ocfs2_refcount_tree; ocfs2_refcount_block -> ocfs2_refcount_list -> ocfs2_refcount_rec "The refcount tree root is a refcount block pointed to by i_refcount_loc" Some operations needs extra uncached lookups Also i dumped frag/stat/refcount from production hypervisor node using debugfs.ocfs2, files are in attach (url as alt way - http://public.edss.ee/tmp/debugfs.tar.gz ). Hypervisor OCFS2 mount options: rw,nosuid,noexec,noatime,heartbeat=none,nointr,data=ordered,errors=remount- ro,localalloc=2048,coherency=full,user_xattr,acl Mkfs string: mkfs.ocfs2 -b 4KB -C 1MB -N 2 -T vmstore -L "storage" --fs- features=local,backup-super,sparse,unwritten,inline- data,metaecc,refcount,xattr,indexed-dirs,discontig-bg Can you please explain why there are so many extent blocks (204)? Is it really impossible to store plenty of clusters in single extent (like #25, block 3874095 -> 20847 clusters)? -- Best regards, Eugene Istomin IT Architect On Monday, May 18, 2015 12:45:40 PM Goldwyn Rodrigues wrote: > Hi Eugene, > > Sorry, had been busy with other work and this slipped on the list. > > > > Do you know something about such behavior? > > > > > > The question is why a reflink operation on VM disk leads to plenty of > > > > read > > > > > ops? Is this related to CoW specific structures? > > This is in fact related to the CoW. An ocfs2 file is an extent tree, > which the extent headers marking if the extent is a reflinked or not > with the number of reflinks. > > If you perform a reflink on a file which is being changed constantly, > not only recreate the extent tree, but also decrease the refcount of the > ones already present. Add to it, the extents which need to be read for > replication. > > > HTH, > > > > We can provide others details & ssh to testbed. > > > > > > > Hello, > > > > > > > > > > > > > > > > after deploying reflink-based VM snapshots to production servers we > > > > > > > > discovered a performace degradation: > > > > > > > > > > > > > > > > OS: Opensuse 13.1, 13.2 > > > > > > > > Hypervisors: Xen 4.4, 4.5 > > > > > > > > Dom0 kernels: 3.12, 3.16, 3.18 > > > > > > > > DomU kernels: 3.12, 3.16, 3.18 > > > > > > > > Tested DomU disk backends: tapdisk2, qdisk > > > > > > > > > > > > > > > > > > > > > > > > 1) on DomU (VM) > > > > > > > > #dd if=/dev/zero of=test2 bs=1M count=6000 > > > > > > > > > > > > > > > > 2) atop on Dom0: > > > > > > > > sdb - busy:92% - read:375 - write:130902 > > > > > > > > Reads are from others VMs, seems OK > > > > > > > > > > > > > > > > 3) DomU dd finished: > > > > > > > > 6291456000 bytes (6.3 GB) copied, 16.6265 s, 378 MB/s > > > > > > > > > > > > > > > > 4) Lets start dd again & do a snapshot: > > > > > > > > #dd if=/dev/zero of=test2 bs=1M count=6000 > > > > > > > > #reflink test.raw ref/ > > > > > > > > > > > > > > > > 5) atop on Dom0: > > > > > > > > sdb - busy:97% - read:112740 - write:28037 > > > > > > > > So, Read IOPS = 112740, why? > > > > > > > > > > > > > > > > 6) DomU dd finished: > > > > > > > > 6291456000 bytes (6.3 GB) copied, 175.45 s, 35.9 MB/s > > > > > > > > > > > > > > > > 7) Second & further reflinks do not change the atop stat & dd time > > > > > > > > #dd if=/dev/zero of=test2 bs=1M count=6000 > > > > > > > > #reflink --backup=t test.raw ref/ \\ * n times > > > > > > > > ~ 6291456000 bytes (6.3 GB) copied, 162.959 s, 38.6 MB/s > > > > > > > > > > > > > > > > The question is why reflinking a running VM disk leads to read IOPS > > > > storm? > > > > > > Thanks! > > > > > > _______________________________________________ > > > > > > Ocfs2-devel mailing list > > > > > > Ocfs2-devel at oss.oracle.com > > > > > > https://oss.oracle.com/mailman/listinfo/ocfs2-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20150521/23cd43e2/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: debugfs.tar.gz Type: application/x-compressed-tar Size: 729820 bytes Desc: not available Url : http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20150521/23cd43e2/attachment-0001.bin ^ permalink raw reply [flat|nested] 6+ messages in thread
* [Ocfs2-devel] Read IOPS storm in case of reflinking running VM disk 2015-05-20 22:33 ` Eugene Istomin @ 2015-05-21 11:57 ` Goldwyn Rodrigues 0 siblings, 0 replies; 6+ messages in thread From: Goldwyn Rodrigues @ 2015-05-21 11:57 UTC (permalink / raw) To: ocfs2-devel On 05/20/2015 05:33 PM, Eugene Istomin wrote: > Goldwyn, > > thanks for the answer! > > I read > https://oss.oracle.com/osswiki/OCFS2(2f)DesignDocs(2f)RefcountTrees.html > carefully to understand the problem. > > As i understand: > > 1. There are B-Tree structures for reflink: ocfs2_refcount_tree; > ocfs2_refcount_block -> ocfs2_refcount_list -> ocfs2_refcount_rec > 2. "The refcount tree root is a refcount block pointed to by > i_refcount_loc" > 3. Some operations needs extra uncached lookups > > Also i dumped frag/stat/refcount from production hypervisor node using > debugfs.ocfs2, files are in attach (url as alt way - > http://public.edss.ee/tmp/debugfs.tar.gz ). > > Hypervisor OCFS2 mount options: > rw,nosuid,noexec,noatime,heartbeat=none,nointr,data=ordered,errors=remount-ro,localalloc=2048,coherency=full,user_xattr,acl > > Mkfs string: > > mkfs.ocfs2 -b 4KB -C 1MB -N 2 -T vmstore -L "storage" > --fs-features=local,backup-super,sparse,unwritten,inline-data,metaecc,refcount,xattr,indexed-dirs,discontig-bg > > Can you please explain why there are so many extent blocks (204)? Is it > really impossible to store plenty of clusters in single extent (like > #25, block 3874095 -> 20847 clusters)? > A file's extent tree is based on your usage pattern and what is already present on disk. Creating a new file, with large block writes, on a new filesystem with no other nodes may create a file with small number of extents. Modifying refcounted files can increase number of extents. The answer lies in the document you mentioned: <quote> Refcount records do not map 1:1 with extent records. A large extent may be split by a CoW operation. To unchanged inodes, they have one extent record covering the entire extent. The changed inode will have an extent record for the unchanged portion and a new extent record for the changed portion. The refcount tree will have similarly split the single refcount record into two. The changed portion will have decremented the reference count by one, as the changed inode is no longer using that physical extent. </quote> HTH, -- Goldwyn ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2015-05-21 11:57 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2015-05-08 5:56 [Ocfs2-devel] Read IOPS storm in case of reflinking running VM disk Eugene Istomin 2015-05-11 8:48 ` Eugene Istomin 2015-05-18 10:05 ` Eugene Istomin 2015-05-18 17:45 ` Goldwyn Rodrigues 2015-05-20 22:33 ` Eugene Istomin 2015-05-21 11:57 ` Goldwyn Rodrigues
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.