diff for duplicates of <20120301143859.GX5054@shiny> diff --git a/a/1.txt b/N1/1.txt index ff515f0..6ba698a 100644 --- a/a/1.txt +++ b/N1/1.txt @@ -1,34 +1,16 @@ On Wed, Feb 29, 2012 at 11:44:31PM -0500, Theodore Tso wrote: -> You might try sorting the entries returned by readdir by inode number= - before you stat them. This is a long-standing weakness in ext3/ext4= -, and it has to do with how we added hashed tree indexes to directories= - in (a) a backwards compatible way, that (b) was POSIX compliant with r= -espect to adding and removing directory entries concurrently with readi= -ng all of the directory entries using readdir. ->=20 -> You might try compiling spd_readdir from the e2fsprogs source tree (i= -n the contrib directory): ->=20 -> http://git.kernel.org/?p=3Dfs/ext2/e2fsprogs.git;a=3Dblob;f=3Dcontrib= -/spd_readdir.c;h=3Df89832cd7146a6f5313162255f057c5a754a4b84;hb=3Dd9a5d3= -7535794842358e1cfe4faa4a89804ed209 ->=20 -> =E2=80=A6 and then using that as a LD_PRELOAD, and see how that chang= -es things. ->=20 -> The short version is that we can't easily do this in the kernel since= - it's a problem that primarily shows up with very big directories, and = -using non-swappable kernel memory to store all of the directory entries= - and then sort them so they can be returned in inode number just isn't = -practical. It is something which can be easily done in userspace, tho= -ugh, and a number of programs (including mutt for its Maildir support) = -does do, and it helps greatly for workloads where you are calling readd= -ir() followed by something that needs to access the inode (i.e., stat, = -unlink, etc.) ->=20 +> You might try sorting the entries returned by readdir by inode number before you stat them. This is a long-standing weakness in ext3/ext4, and it has to do with how we added hashed tree indexes to directories in (a) a backwards compatible way, that (b) was POSIX compliant with respect to adding and removing directory entries concurrently with reading all of the directory entries using readdir. +> +> You might try compiling spd_readdir from the e2fsprogs source tree (in the contrib directory): +> +> http://git.kernel.org/?p=fs/ext2/e2fsprogs.git;a=blob;f=contrib/spd_readdir.c;h=f89832cd7146a6f5313162255f057c5a754a4b84;hb=d9a5d37535794842358e1cfe4faa4a89804ed209 +> +> … and then using that as a LD_PRELOAD, and see how that changes things. +> +> The short version is that we can't easily do this in the kernel since it's a problem that primarily shows up with very big directories, and using non-swappable kernel memory to store all of the directory entries and then sort them so they can be returned in inode number just isn't practical. It is something which can be easily done in userspace, though, and a number of programs (including mutt for its Maildir support) does do, and it helps greatly for workloads where you are calling readdir() followed by something that needs to access the inode (i.e., stat, unlink, etc.) +> -=46or reading the files, the acp program I sent him tries to do somethi= -ng +For reading the files, the acp program I sent him tries to do something similar. I had forgotten about spd_readdir though, we should consider hacking that into cp and tar. @@ -44,8 +26,7 @@ unmount / flush caches B) time tar cf /dev/zero /new_dir_in_new_fs On ext, The time for B used to be much faster than the time for A -because the files would get written back to disk in roughly htree order= -=2E +because the files would get written back to disk in roughly htree order. Based on Jacek's data, that isn't true anymore. -chris diff --git a/a/content_digest b/N1/content_digest index 00ac4e5..ef7f20c 100644 --- a/a/content_digest +++ b/N1/content_digest @@ -12,36 +12,18 @@ "\00:1\0" "b\0" "On Wed, Feb 29, 2012 at 11:44:31PM -0500, Theodore Tso wrote:\n" - "> You might try sorting the entries returned by readdir by inode number=\n" - " before you stat them. This is a long-standing weakness in ext3/ext4=\n" - ", and it has to do with how we added hashed tree indexes to directories=\n" - " in (a) a backwards compatible way, that (b) was POSIX compliant with r=\n" - "espect to adding and removing directory entries concurrently with readi=\n" - "ng all of the directory entries using readdir.\n" - ">=20\n" - "> You might try compiling spd_readdir from the e2fsprogs source tree (i=\n" - "n the contrib directory):\n" - ">=20\n" - "> http://git.kernel.org/?p=3Dfs/ext2/e2fsprogs.git;a=3Dblob;f=3Dcontrib=\n" - "/spd_readdir.c;h=3Df89832cd7146a6f5313162255f057c5a754a4b84;hb=3Dd9a5d3=\n" - "7535794842358e1cfe4faa4a89804ed209\n" - ">=20\n" - "> =E2=80=A6 and then using that as a LD_PRELOAD, and see how that chang=\n" - "es things.\n" - ">=20\n" - "> The short version is that we can't easily do this in the kernel since=\n" - " it's a problem that primarily shows up with very big directories, and =\n" - "using non-swappable kernel memory to store all of the directory entries=\n" - " and then sort them so they can be returned in inode number just isn't =\n" - "practical. It is something which can be easily done in userspace, tho=\n" - "ugh, and a number of programs (including mutt for its Maildir support) =\n" - "does do, and it helps greatly for workloads where you are calling readd=\n" - "ir() followed by something that needs to access the inode (i.e., stat, =\n" - "unlink, etc.)\n" - ">=20\n" + "> You might try sorting the entries returned by readdir by inode number before you stat them. This is a long-standing weakness in ext3/ext4, and it has to do with how we added hashed tree indexes to directories in (a) a backwards compatible way, that (b) was POSIX compliant with respect to adding and removing directory entries concurrently with reading all of the directory entries using readdir.\n" + "> \n" + "> You might try compiling spd_readdir from the e2fsprogs source tree (in the contrib directory):\n" + "> \n" + "> http://git.kernel.org/?p=fs/ext2/e2fsprogs.git;a=blob;f=contrib/spd_readdir.c;h=f89832cd7146a6f5313162255f057c5a754a4b84;hb=d9a5d37535794842358e1cfe4faa4a89804ed209\n" + "> \n" + "> \342\200\246 and then using that as a LD_PRELOAD, and see how that changes things.\n" + "> \n" + "> The short version is that we can't easily do this in the kernel since it's a problem that primarily shows up with very big directories, and using non-swappable kernel memory to store all of the directory entries and then sort them so they can be returned in inode number just isn't practical. It is something which can be easily done in userspace, though, and a number of programs (including mutt for its Maildir support) does do, and it helps greatly for workloads where you are calling readdir() followed by something that needs to access the inode (i.e., stat, unlink, etc.)\n" + "> \n" "\n" - "=46or reading the files, the acp program I sent him tries to do somethi=\n" - "ng\n" + "For reading the files, the acp program I sent him tries to do something\n" "similar. I had forgotten about spd_readdir though, we should consider\n" "hacking that into cp and tar.\n" "\n" @@ -57,10 +39,9 @@ "B) time tar cf /dev/zero /new_dir_in_new_fs\n" "\n" "On ext, The time for B used to be much faster than the time for A\n" - "because the files would get written back to disk in roughly htree order=\n" - "=2E\n" + "because the files would get written back to disk in roughly htree order.\n" "Based on Jacek's data, that isn't true anymore.\n" "\n" -chris -1a21dd0e1d54a2ac5c29ca3273997de9ccb676d40926cee301ab80fe5281301a +a72287fe7ec09b1cb671c81e4a55d92d658895e1b7cbaafc517ee2031ca5e0f7
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.