* Re: [PATCH 22/32] vfs: inode cache conversion to hash-bl
@ 2023-05-10 19:55 kernel test robot
0 siblings, 0 replies; 7+ messages in thread
From: kernel test robot @ 2023-05-10 19:55 UTC (permalink / raw)
To: oe-kbuild; +Cc: lkp, Dan Carpenter
BCC: lkp@intel.com
CC: oe-kbuild-all@lists.linux.dev
In-Reply-To: <20230509165657.1735798-23-kent.overstreet@linux.dev>
References: <20230509165657.1735798-23-kent.overstreet@linux.dev>
TO: Kent Overstreet <kmo@daterainc.com>
TO: linux-kernel@vger.kernel.org
TO: linux-fsdevel@vger.kernel.org
TO: linux-bcachefs@vger.kernel.org
CC: Dave Chinner <dchinner@redhat.com>
CC: Alexander Viro <viro@zeniv.linux.org.uk>
CC: Christian Brauner <brauner@kernel.org>
CC: Kent Overstreet <kmo@daterainc.com>
Hi Kent,
kernel test robot noticed the following build warnings:
[auto build test WARNING on tip/locking/core]
[cannot apply to axboe-block/for-next akpm-mm/mm-everything kdave/for-next linus/master v6.4-rc1 next-20230510]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Kent-Overstreet/Compiler-Attributes-add-__flatten/20230510-010302
base: tip/locking/core
patch link: https://lore.kernel.org/r/20230509165657.1735798-23-kent.overstreet%40linux.dev
patch subject: [PATCH 22/32] vfs: inode cache conversion to hash-bl
:::::: branch date: 27 hours ago
:::::: commit date: 27 hours ago
config: x86_64-randconfig-m001 (https://download.01.org/0day-ci/archive/20230511/202305110304.n2TEQd6l-lkp@intel.com/config)
compiler: gcc-11 (Debian 11.3.0-12) 11.3.0
If you fix the issue, kindly add following tag where applicable
| Reported-by: kernel test robot <lkp@intel.com>
| Reported-by: Dan Carpenter <error27@gmail.com>
| Link: https://lore.kernel.org/r/202305110304.n2TEQd6l-lkp@intel.com/
New smatch warnings:
fs/inode.c:1232 inode_insert5() warn: sleeping in atomic context
fs/inode.c:1331 iget_locked() warn: sleeping in atomic context
fs/inode.c:1543 ilookup() warn: sleeping in atomic context
fs/inode.c:1724 insert_inode_locked() warn: sleeping in atomic context
Old smatch warnings:
fs/inode.c:1371 iget_locked() warn: sleeping in atomic context
vim +1232 fs/inode.c
375e289ea85166 J. Bruce Fields 2012-04-18 1193
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1194 /**
80ea09a002bf43 Miklos Szeredi 2018-05-17 1195 * inode_insert5 - obtain an inode from a mounted file system
80ea09a002bf43 Miklos Szeredi 2018-05-17 1196 * @inode: pre-allocated inode to use for insert to cache
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1197 * @hashval: hash value (usually inode number) to get
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1198 * @test: callback used for comparisons between inodes
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1199 * @set: callback used to initialize a new struct inode
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1200 * @data: opaque data pointer to pass to @test and @set
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1201 *
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1202 * Search for the inode specified by @hashval and @data in the inode cache,
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1203 * and if present it is return it with an increased reference count. This is
80ea09a002bf43 Miklos Szeredi 2018-05-17 1204 * a variant of iget5_locked() for callers that don't want to fail on memory
80ea09a002bf43 Miklos Szeredi 2018-05-17 1205 * allocation of inode.
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1206 *
80ea09a002bf43 Miklos Szeredi 2018-05-17 1207 * If the inode is not in cache, insert the pre-allocated inode to cache and
80ea09a002bf43 Miklos Szeredi 2018-05-17 1208 * return it locked, hashed, and with the I_NEW flag set. The file system gets
80ea09a002bf43 Miklos Szeredi 2018-05-17 1209 * to fill it in before unlocking it via unlock_new_inode().
^1da177e4c3f41 Linus Torvalds 2005-04-16 1210 *
c41f58169b510c Dave Chinner 2023-05-09 1211 * Note both @test and @set are called with the inode hash chain lock held,
c41f58169b510c Dave Chinner 2023-05-09 1212 * so can't sleep.
^1da177e4c3f41 Linus Torvalds 2005-04-16 1213 */
80ea09a002bf43 Miklos Szeredi 2018-05-17 1214 struct inode *inode_insert5(struct inode *inode, unsigned long hashval,
6b3304b5317047 Manish Katiyar 2009-03-31 1215 int (*test)(struct inode *, void *),
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1216 int (*set)(struct inode *, void *), void *data)
^1da177e4c3f41 Linus Torvalds 2005-04-16 1217 {
c41f58169b510c Dave Chinner 2023-05-09 1218 struct hlist_bl_head *b = i_hash_head(inode->i_sb, hashval);
80ea09a002bf43 Miklos Szeredi 2018-05-17 1219 struct inode *old;
80ea09a002bf43 Miklos Szeredi 2018-05-17 1220
2864f301424227 Al Viro 2016-07-03 1221 again:
c41f58169b510c Dave Chinner 2023-05-09 1222 hlist_bl_lock(b);
c41f58169b510c Dave Chinner 2023-05-09 1223 old = find_inode(inode->i_sb, b, test, data);
80ea09a002bf43 Miklos Szeredi 2018-05-17 1224 if (unlikely(old)) {
80ea09a002bf43 Miklos Szeredi 2018-05-17 1225 /*
80ea09a002bf43 Miklos Szeredi 2018-05-17 1226 * Uhhuh, somebody else created the same inode under us.
80ea09a002bf43 Miklos Szeredi 2018-05-17 1227 * Use the old inode instead of the preallocated one.
80ea09a002bf43 Miklos Szeredi 2018-05-17 1228 */
c41f58169b510c Dave Chinner 2023-05-09 1229 hlist_bl_unlock(b);
c2b6d621c4ffe9 Al Viro 2018-06-28 1230 if (IS_ERR(old))
c2b6d621c4ffe9 Al Viro 2018-06-28 1231 return NULL;
80ea09a002bf43 Miklos Szeredi 2018-05-17 @1232 wait_on_inode(old);
80ea09a002bf43 Miklos Szeredi 2018-05-17 1233 if (unlikely(inode_unhashed(old))) {
80ea09a002bf43 Miklos Szeredi 2018-05-17 1234 iput(old);
2864f301424227 Al Viro 2016-07-03 1235 goto again;
2864f301424227 Al Viro 2016-07-03 1236 }
80ea09a002bf43 Miklos Szeredi 2018-05-17 1237 return old;
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1238 }
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1239
80ea09a002bf43 Miklos Szeredi 2018-05-17 1240 if (set && unlikely(set(inode, data))) {
80ea09a002bf43 Miklos Szeredi 2018-05-17 1241 inode = NULL;
80ea09a002bf43 Miklos Szeredi 2018-05-17 1242 goto unlock;
80ea09a002bf43 Miklos Szeredi 2018-05-17 1243 }
^1da177e4c3f41 Linus Torvalds 2005-04-16 1244
80ea09a002bf43 Miklos Szeredi 2018-05-17 1245 /*
80ea09a002bf43 Miklos Szeredi 2018-05-17 1246 * Return the locked inode with I_NEW set, the
80ea09a002bf43 Miklos Szeredi 2018-05-17 1247 * caller is responsible for filling in the contents
80ea09a002bf43 Miklos Szeredi 2018-05-17 1248 */
250df6ed274d76 Dave Chinner 2011-03-22 1249 spin_lock(&inode->i_lock);
80ea09a002bf43 Miklos Szeredi 2018-05-17 1250 inode->i_state |= I_NEW;
c41f58169b510c Dave Chinner 2023-05-09 1251 __insert_inode_hash_head(inode, b);
250df6ed274d76 Dave Chinner 2011-03-22 1252 spin_unlock(&inode->i_lock);
18cc912b8a2aca Jeff Layton 2022-03-31 1253
18cc912b8a2aca Jeff Layton 2022-03-31 1254 /*
18cc912b8a2aca Jeff Layton 2022-03-31 1255 * Add inode to the sb list if it's not already. It has I_NEW at this
18cc912b8a2aca Jeff Layton 2022-03-31 1256 * point, so it should be safe to test i_sb_list locklessly.
18cc912b8a2aca Jeff Layton 2022-03-31 1257 */
18cc912b8a2aca Jeff Layton 2022-03-31 1258 if (list_empty(&inode->i_sb_list))
e950564b97fd0f Miklos Szeredi 2018-07-24 1259 inode_sb_list_add(inode);
80ea09a002bf43 Miklos Szeredi 2018-05-17 1260 unlock:
c41f58169b510c Dave Chinner 2023-05-09 1261 hlist_bl_unlock(b);
^1da177e4c3f41 Linus Torvalds 2005-04-16 1262
^1da177e4c3f41 Linus Torvalds 2005-04-16 1263 return inode;
^1da177e4c3f41 Linus Torvalds 2005-04-16 1264 }
80ea09a002bf43 Miklos Szeredi 2018-05-17 1265 EXPORT_SYMBOL(inode_insert5);
^1da177e4c3f41 Linus Torvalds 2005-04-16 1266
80ea09a002bf43 Miklos Szeredi 2018-05-17 1267 /**
80ea09a002bf43 Miklos Szeredi 2018-05-17 1268 * iget5_locked - obtain an inode from a mounted file system
80ea09a002bf43 Miklos Szeredi 2018-05-17 1269 * @sb: super block of file system
80ea09a002bf43 Miklos Szeredi 2018-05-17 1270 * @hashval: hash value (usually inode number) to get
80ea09a002bf43 Miklos Szeredi 2018-05-17 1271 * @test: callback used for comparisons between inodes
80ea09a002bf43 Miklos Szeredi 2018-05-17 1272 * @set: callback used to initialize a new struct inode
80ea09a002bf43 Miklos Szeredi 2018-05-17 1273 * @data: opaque data pointer to pass to @test and @set
80ea09a002bf43 Miklos Szeredi 2018-05-17 1274 *
80ea09a002bf43 Miklos Szeredi 2018-05-17 1275 * Search for the inode specified by @hashval and @data in the inode cache,
80ea09a002bf43 Miklos Szeredi 2018-05-17 1276 * and if present it is return it with an increased reference count. This is
80ea09a002bf43 Miklos Szeredi 2018-05-17 1277 * a generalized version of iget_locked() for file systems where the inode
80ea09a002bf43 Miklos Szeredi 2018-05-17 1278 * number is not sufficient for unique identification of an inode.
80ea09a002bf43 Miklos Szeredi 2018-05-17 1279 *
80ea09a002bf43 Miklos Szeredi 2018-05-17 1280 * If the inode is not in cache, allocate a new inode and return it locked,
80ea09a002bf43 Miklos Szeredi 2018-05-17 1281 * hashed, and with the I_NEW flag set. The file system gets to fill it in
80ea09a002bf43 Miklos Szeredi 2018-05-17 1282 * before unlocking it via unlock_new_inode().
80ea09a002bf43 Miklos Szeredi 2018-05-17 1283 *
80ea09a002bf43 Miklos Szeredi 2018-05-17 1284 * Note both @test and @set are called with the inode_hash_lock held, so can't
80ea09a002bf43 Miklos Szeredi 2018-05-17 1285 * sleep.
^1da177e4c3f41 Linus Torvalds 2005-04-16 1286 */
80ea09a002bf43 Miklos Szeredi 2018-05-17 1287 struct inode *iget5_locked(struct super_block *sb, unsigned long hashval,
80ea09a002bf43 Miklos Szeredi 2018-05-17 1288 int (*test)(struct inode *, void *),
80ea09a002bf43 Miklos Szeredi 2018-05-17 1289 int (*set)(struct inode *, void *), void *data)
80ea09a002bf43 Miklos Szeredi 2018-05-17 1290 {
80ea09a002bf43 Miklos Szeredi 2018-05-17 1291 struct inode *inode = ilookup5(sb, hashval, test, data);
80ea09a002bf43 Miklos Szeredi 2018-05-17 1292
80ea09a002bf43 Miklos Szeredi 2018-05-17 1293 if (!inode) {
e950564b97fd0f Miklos Szeredi 2018-07-24 1294 struct inode *new = alloc_inode(sb);
80ea09a002bf43 Miklos Szeredi 2018-05-17 1295
80ea09a002bf43 Miklos Szeredi 2018-05-17 1296 if (new) {
e950564b97fd0f Miklos Szeredi 2018-07-24 1297 new->i_state = 0;
80ea09a002bf43 Miklos Szeredi 2018-05-17 1298 inode = inode_insert5(new, hashval, test, set, data);
80ea09a002bf43 Miklos Szeredi 2018-05-17 1299 if (unlikely(inode != new))
e950564b97fd0f Miklos Szeredi 2018-07-24 1300 destroy_inode(new);
2864f301424227 Al Viro 2016-07-03 1301 }
^1da177e4c3f41 Linus Torvalds 2005-04-16 1302 }
^1da177e4c3f41 Linus Torvalds 2005-04-16 1303 return inode;
^1da177e4c3f41 Linus Torvalds 2005-04-16 1304 }
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1305 EXPORT_SYMBOL(iget5_locked);
^1da177e4c3f41 Linus Torvalds 2005-04-16 1306
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1307 /**
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1308 * iget_locked - obtain an inode from a mounted file system
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1309 * @sb: super block of file system
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1310 * @ino: inode number to get
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1311 *
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1312 * Search for the inode specified by @ino in the inode cache and if present
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1313 * return it with an increased reference count. This is for file systems
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1314 * where the inode number is sufficient for unique identification of an inode.
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1315 *
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1316 * If the inode is not in cache, allocate a new inode and return it locked,
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1317 * hashed, and with the I_NEW flag set. The file system gets to fill it in
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1318 * before unlocking it via unlock_new_inode().
^1da177e4c3f41 Linus Torvalds 2005-04-16 1319 */
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1320 struct inode *iget_locked(struct super_block *sb, unsigned long ino)
^1da177e4c3f41 Linus Torvalds 2005-04-16 1321 {
c41f58169b510c Dave Chinner 2023-05-09 1322 struct hlist_bl_head *b = i_hash_head(sb, ino);
^1da177e4c3f41 Linus Torvalds 2005-04-16 1323 struct inode *inode;
2864f301424227 Al Viro 2016-07-03 1324 again:
c41f58169b510c Dave Chinner 2023-05-09 1325 hlist_bl_lock(b);
c41f58169b510c Dave Chinner 2023-05-09 1326 inode = find_inode_fast(sb, b, ino);
c41f58169b510c Dave Chinner 2023-05-09 1327 hlist_bl_unlock(b);
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1328 if (inode) {
c2b6d621c4ffe9 Al Viro 2018-06-28 1329 if (IS_ERR(inode))
c2b6d621c4ffe9 Al Viro 2018-06-28 1330 return NULL;
0b2d0724e26a33 Christoph Hellwig 2011-03-23 @1331 wait_on_inode(inode);
2864f301424227 Al Viro 2016-07-03 1332 if (unlikely(inode_unhashed(inode))) {
2864f301424227 Al Viro 2016-07-03 1333 iput(inode);
2864f301424227 Al Viro 2016-07-03 1334 goto again;
2864f301424227 Al Viro 2016-07-03 1335 }
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1336 return inode;
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1337 }
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1338
^1da177e4c3f41 Linus Torvalds 2005-04-16 1339 inode = alloc_inode(sb);
^1da177e4c3f41 Linus Torvalds 2005-04-16 1340 if (inode) {
^1da177e4c3f41 Linus Torvalds 2005-04-16 1341 struct inode *old;
^1da177e4c3f41 Linus Torvalds 2005-04-16 1342
c41f58169b510c Dave Chinner 2023-05-09 1343 hlist_bl_lock(b);
^1da177e4c3f41 Linus Torvalds 2005-04-16 1344 /* We released the lock, so.. */
c41f58169b510c Dave Chinner 2023-05-09 1345 old = find_inode_fast(sb, b, ino);
^1da177e4c3f41 Linus Torvalds 2005-04-16 1346 if (!old) {
^1da177e4c3f41 Linus Torvalds 2005-04-16 1347 inode->i_ino = ino;
250df6ed274d76 Dave Chinner 2011-03-22 1348 spin_lock(&inode->i_lock);
250df6ed274d76 Dave Chinner 2011-03-22 1349 inode->i_state = I_NEW;
c41f58169b510c Dave Chinner 2023-05-09 1350 __insert_inode_hash_head(inode, b);
250df6ed274d76 Dave Chinner 2011-03-22 1351 spin_unlock(&inode->i_lock);
55fa6091d83160 Dave Chinner 2011-03-22 1352 inode_sb_list_add(inode);
c41f58169b510c Dave Chinner 2023-05-09 1353 hlist_bl_unlock(b);
^1da177e4c3f41 Linus Torvalds 2005-04-16 1354
^1da177e4c3f41 Linus Torvalds 2005-04-16 1355 /* Return the locked inode with I_NEW set, the
^1da177e4c3f41 Linus Torvalds 2005-04-16 1356 * caller is responsible for filling in the contents
^1da177e4c3f41 Linus Torvalds 2005-04-16 1357 */
^1da177e4c3f41 Linus Torvalds 2005-04-16 1358 return inode;
^1da177e4c3f41 Linus Torvalds 2005-04-16 1359 }
^1da177e4c3f41 Linus Torvalds 2005-04-16 1360
^1da177e4c3f41 Linus Torvalds 2005-04-16 1361 /*
^1da177e4c3f41 Linus Torvalds 2005-04-16 1362 * Uhhuh, somebody else created the same inode under
^1da177e4c3f41 Linus Torvalds 2005-04-16 1363 * us. Use the old inode instead of the one we just
^1da177e4c3f41 Linus Torvalds 2005-04-16 1364 * allocated.
^1da177e4c3f41 Linus Torvalds 2005-04-16 1365 */
c41f58169b510c Dave Chinner 2023-05-09 1366 hlist_bl_unlock(b);
^1da177e4c3f41 Linus Torvalds 2005-04-16 1367 destroy_inode(inode);
c2b6d621c4ffe9 Al Viro 2018-06-28 1368 if (IS_ERR(old))
c2b6d621c4ffe9 Al Viro 2018-06-28 1369 return NULL;
^1da177e4c3f41 Linus Torvalds 2005-04-16 1370 inode = old;
^1da177e4c3f41 Linus Torvalds 2005-04-16 1371 wait_on_inode(inode);
2864f301424227 Al Viro 2016-07-03 1372 if (unlikely(inode_unhashed(inode))) {
2864f301424227 Al Viro 2016-07-03 1373 iput(inode);
2864f301424227 Al Viro 2016-07-03 1374 goto again;
2864f301424227 Al Viro 2016-07-03 1375 }
^1da177e4c3f41 Linus Torvalds 2005-04-16 1376 }
^1da177e4c3f41 Linus Torvalds 2005-04-16 1377 return inode;
^1da177e4c3f41 Linus Torvalds 2005-04-16 1378 }
0b2d0724e26a33 Christoph Hellwig 2011-03-23 1379 EXPORT_SYMBOL(iget_locked);
^1da177e4c3f41 Linus Torvalds 2005-04-16 1380
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests
^ permalink raw reply [flat|nested] 7+ messages in thread* [PATCH 00/32] bcachefs - a new COW filesystem
@ 2023-05-09 16:56 Kent Overstreet
2023-05-09 16:56 ` [PATCH 22/32] vfs: inode cache conversion to hash-bl Kent Overstreet
0 siblings, 1 reply; 7+ messages in thread
From: Kent Overstreet @ 2023-05-09 16:56 UTC (permalink / raw)
To: linux-kernel, linux-fsdevel, linux-block, linux-mm,
linux-bcachefs
Cc: Kent Overstreet, viro, akpm, boqun.feng, brauner, hch, colyli,
djwong, mingo, jack, axboe, willy, ojeda, ming.lei, ndesaulniers,
peterz, phillip, urezki, longman, will
I'm submitting the bcachefs filesystem for review and inclusion.
Included in this patch series are all the non fs/bcachefs/ patches. The
entire tree, based on v6.3, may be found at:
http://evilpiepirate.org/git/bcachefs.git bcachefs-for-upstream
----------------------------------------------------------------
bcachefs overview, status:
Features:
- too many to list
Known bugs:
- too many to list
Status:
- Snapshots have been declared stable; one serious bug report
outstanding to look into, most users report it working well.
These are RW btrfs-style snapshots, but with far better scalability
and no scalability issues with sparse snapshots due to key level
versioning.
- Erasure coding is getting really close; hope to have it ready for
users to beat on it by this summer. This is a novel RAID/erasure
coding design with no write hole, and no fragmentation of writes
(e.g. RAIDZ).
- Tons of scalabality work finished over the past year, users are
running it on 100 TB filesystems without complaint, waiting for first
1 PB user; next thing to address re: scalability is fsck/recovery
memory usage.
- Test infrastructure! Major project milestone, check out our test
dashboard at
https://evilpiepirate.org/~testdashboard/ci?branch=bcachefs
Other project notes:
irc::/irc.oftc.net/bcache is where most activity happens; I'm always
there, and most code review happens there - I find the conversational
format more productive.
------------------------------------------------
patches in this series:
Christopher James Halse Rogers (1):
stacktrace: Export stack_trace_save_tsk
Daniel Hill (1):
lib: add mean and variance module.
Dave Chinner (3):
vfs: factor out inode hash head calculation
hlist-bl: add hlist_bl_fake()
vfs: inode cache conversion to hash-bl
Kent Overstreet (27):
Compiler Attributes: add __flatten
locking/lockdep: lock_class_is_held()
locking/lockdep: lockdep_set_no_check_recursion()
locking: SIX locks (shared/intent/exclusive)
MAINTAINERS: Add entry for six locks
sched: Add task_struct->faults_disabled_mapping
mm: Bring back vmalloc_exec
fs: factor out d_mark_tmpfile()
block: Add some exports for bcachefs
block: Allow bio_iov_iter_get_pages() with bio->bi_bdev unset
block: Bring back zero_fill_bio_iter
block: Rework bio_for_each_segment_all()
block: Rework bio_for_each_folio_all()
block: Don't block on s_umount from __invalidate_super()
bcache: move closures to lib/
MAINTAINERS: Add entry for closures
closures: closure_wait_event()
closures: closure_nr_remaining()
closures: Add a missing include
iov_iter: copy_folio_from_iter_atomic()
MAINTAINERS: Add entry for generic-radix-tree
lib/generic-radix-tree.c: Don't overflow in peek()
lib/generic-radix-tree.c: Add a missing include
lib/generic-radix-tree.c: Add peek_prev()
lib/string_helpers: string_get_size() now returns characters wrote
lib: Export errname
MAINTAINERS: Add entry for bcachefs
MAINTAINERS | 39 +
block/bdev.c | 2 +-
block/bio.c | 57 +-
block/blk-core.c | 1 +
block/blk-map.c | 38 +-
block/blk.h | 1 -
block/bounce.c | 12 +-
drivers/md/bcache/Kconfig | 10 +-
drivers/md/bcache/Makefile | 4 +-
drivers/md/bcache/bcache.h | 2 +-
drivers/md/bcache/btree.c | 8 +-
drivers/md/bcache/super.c | 1 -
drivers/md/bcache/util.h | 3 +-
drivers/md/dm-crypt.c | 10 +-
drivers/md/raid1.c | 4 +-
fs/btrfs/disk-io.c | 4 +-
fs/btrfs/extent_io.c | 50 +-
fs/btrfs/raid56.c | 14 +-
fs/crypto/bio.c | 9 +-
fs/dcache.c | 12 +-
fs/erofs/zdata.c | 4 +-
fs/ext4/page-io.c | 8 +-
fs/ext4/readpage.c | 4 +-
fs/f2fs/data.c | 20 +-
fs/gfs2/lops.c | 10 +-
fs/gfs2/meta_io.c | 8 +-
fs/inode.c | 218 +++--
fs/iomap/buffered-io.c | 14 +-
fs/mpage.c | 4 +-
fs/squashfs/block.c | 48 +-
fs/squashfs/lz4_wrapper.c | 17 +-
fs/squashfs/lzo_wrapper.c | 17 +-
fs/squashfs/xz_wrapper.c | 19 +-
fs/squashfs/zlib_wrapper.c | 18 +-
fs/squashfs/zstd_wrapper.c | 19 +-
fs/super.c | 40 +-
fs/verity/verify.c | 9 +-
include/linux/bio.h | 132 +--
include/linux/blkdev.h | 1 +
include/linux/bvec.h | 70 +-
.../md/bcache => include/linux}/closure.h | 46 +-
include/linux/compiler_attributes.h | 5 +
include/linux/dcache.h | 1 +
include/linux/fs.h | 10 +-
include/linux/generic-radix-tree.h | 68 +-
include/linux/list_bl.h | 22 +
include/linux/lockdep.h | 10 +
include/linux/lockdep_types.h | 2 +-
include/linux/mean_and_variance.h | 219 +++++
include/linux/sched.h | 1 +
include/linux/six.h | 210 +++++
include/linux/string_helpers.h | 4 +-
include/linux/uio.h | 2 +
include/linux/vmalloc.h | 1 +
init/init_task.c | 1 +
kernel/Kconfig.locks | 3 +
kernel/locking/Makefile | 1 +
kernel/locking/lockdep.c | 46 ++
kernel/locking/six.c | 779 ++++++++++++++++++
kernel/module/main.c | 4 +-
kernel/stacktrace.c | 2 +
lib/Kconfig | 3 +
lib/Kconfig.debug | 18 +
lib/Makefile | 2 +
{drivers/md/bcache => lib}/closure.c | 36 +-
lib/errname.c | 1 +
lib/generic-radix-tree.c | 76 +-
lib/iov_iter.c | 53 +-
lib/math/Kconfig | 3 +
lib/math/Makefile | 2 +
lib/math/mean_and_variance.c | 136 +++
lib/math/mean_and_variance_test.c | 155 ++++
lib/string_helpers.c | 8 +-
mm/nommu.c | 18 +
mm/vmalloc.c | 21 +
75 files changed, 2485 insertions(+), 445 deletions(-)
rename {drivers/md/bcache => include/linux}/closure.h (93%)
create mode 100644 include/linux/mean_and_variance.h
create mode 100644 include/linux/six.h
create mode 100644 kernel/locking/six.c
rename {drivers/md/bcache => lib}/closure.c (88%)
create mode 100644 lib/math/mean_and_variance.c
create mode 100644 lib/math/mean_and_variance_test.c
--
2.40.1
^ permalink raw reply [flat|nested] 7+ messages in thread* [PATCH 22/32] vfs: inode cache conversion to hash-bl 2023-05-09 16:56 [PATCH 00/32] bcachefs - a new COW filesystem Kent Overstreet @ 2023-05-09 16:56 ` Kent Overstreet 2023-05-10 4:45 ` Dave Chinner 0 siblings, 1 reply; 7+ messages in thread From: Kent Overstreet @ 2023-05-09 16:56 UTC (permalink / raw) To: linux-kernel, linux-fsdevel, linux-bcachefs Cc: Dave Chinner, Alexander Viro, Christian Brauner, Kent Overstreet From: Dave Chinner <dchinner@redhat.com> Because scalability of the global inode_hash_lock really, really sucks. 32-way concurrent create on a couple of different filesystems before: - 52.13% 0.04% [kernel] [k] ext4_create - 52.09% ext4_create - 41.03% __ext4_new_inode - 29.92% insert_inode_locked - 25.35% _raw_spin_lock - do_raw_spin_lock - 24.97% __pv_queued_spin_lock_slowpath - 72.33% 0.02% [kernel] [k] do_filp_open - 72.31% do_filp_open - 72.28% path_openat - 57.03% bch2_create - 56.46% __bch2_create - 40.43% inode_insert5 - 36.07% _raw_spin_lock - do_raw_spin_lock 35.86% __pv_queued_spin_lock_slowpath 4.02% find_inode Convert the inode hash table to a RCU-aware hash-bl table just like the dentry cache. Note that we need to store a pointer to the hlist_bl_head the inode has been added to in the inode so that when it comes to unhash the inode we know what list to lock. We need to do this because the hash value that is used to hash the inode is generated from the inode itself - filesystems can provide this themselves so we have to either store the hash or the head pointer in the inode to be able to find the right list head for removal... Same workload after: Signed-off-by: Dave Chinner <dchinner@redhat.com> Cc: Alexander Viro <viro@zeniv.linux.org.uk> Cc: Christian Brauner <brauner@kernel.org> Cc: linux-fsdevel@vger.kernel.org Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev> --- fs/inode.c | 200 ++++++++++++++++++++++++++++----------------- include/linux/fs.h | 9 +- 2 files changed, 132 insertions(+), 77 deletions(-) diff --git a/fs/inode.c b/fs/inode.c index 41a10bcda1..d446b054ec 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -57,8 +57,7 @@ static unsigned int i_hash_mask __read_mostly; static unsigned int i_hash_shift __read_mostly; -static struct hlist_head *inode_hashtable __read_mostly; -static __cacheline_aligned_in_smp DEFINE_SPINLOCK(inode_hash_lock); +static struct hlist_bl_head *inode_hashtable __read_mostly; static unsigned long hash(struct super_block *sb, unsigned long hashval) { @@ -70,7 +69,7 @@ static unsigned long hash(struct super_block *sb, unsigned long hashval) return tmp & i_hash_mask; } -static inline struct hlist_head *i_hash_head(struct super_block *sb, +static inline struct hlist_bl_head *i_hash_head(struct super_block *sb, unsigned int hashval) { return inode_hashtable + hash(sb, hashval); @@ -433,7 +432,7 @@ EXPORT_SYMBOL(address_space_init_once); void inode_init_once(struct inode *inode) { memset(inode, 0, sizeof(*inode)); - INIT_HLIST_NODE(&inode->i_hash); + INIT_HLIST_BL_NODE(&inode->i_hash); INIT_LIST_HEAD(&inode->i_devices); INIT_LIST_HEAD(&inode->i_io_list); INIT_LIST_HEAD(&inode->i_wb_list); @@ -522,6 +521,17 @@ static inline void inode_sb_list_del(struct inode *inode) } } +/* + * Ensure that we store the hash head in the inode when we insert the inode into + * the hlist_bl_head... + */ +static inline void +__insert_inode_hash_head(struct inode *inode, struct hlist_bl_head *b) +{ + hlist_bl_add_head_rcu(&inode->i_hash, b); + inode->i_hash_head = b; +} + /** * __insert_inode_hash - hash an inode * @inode: unhashed inode @@ -532,13 +542,13 @@ static inline void inode_sb_list_del(struct inode *inode) */ void __insert_inode_hash(struct inode *inode, unsigned long hashval) { - struct hlist_head *b = inode_hashtable + hash(inode->i_sb, hashval); + struct hlist_bl_head *b = i_hash_head(inode->i_sb, hashval); - spin_lock(&inode_hash_lock); + hlist_bl_lock(b); spin_lock(&inode->i_lock); - hlist_add_head_rcu(&inode->i_hash, b); + __insert_inode_hash_head(inode, b); spin_unlock(&inode->i_lock); - spin_unlock(&inode_hash_lock); + hlist_bl_unlock(b); } EXPORT_SYMBOL(__insert_inode_hash); @@ -550,11 +560,44 @@ EXPORT_SYMBOL(__insert_inode_hash); */ void __remove_inode_hash(struct inode *inode) { - spin_lock(&inode_hash_lock); - spin_lock(&inode->i_lock); - hlist_del_init_rcu(&inode->i_hash); - spin_unlock(&inode->i_lock); - spin_unlock(&inode_hash_lock); + struct hlist_bl_head *b = inode->i_hash_head; + + /* + * There are some callers that come through here without synchronisation + * and potentially with multiple references to the inode. Hence we have + * to handle the case that we might race with a remove and insert to a + * different list. Coda, in particular, seems to have a userspace API + * that can directly trigger "unhash/rehash to different list" behaviour + * without any serialisation at all. + * + * Hence we have to handle the situation where the inode->i_hash_head + * might point to a different list than what we expect, indicating that + * we raced with another unhash and potentially a new insertion. This + * means we have to retest the head once we have everything locked up + * and loop again if it doesn't match. + */ + while (b) { + hlist_bl_lock(b); + spin_lock(&inode->i_lock); + if (b != inode->i_hash_head) { + hlist_bl_unlock(b); + b = inode->i_hash_head; + spin_unlock(&inode->i_lock); + continue; + } + /* + * Need to set the pprev pointer to NULL after list removal so + * that both RCU traversals and hlist_bl_unhashed() work + * correctly at this point. + */ + hlist_bl_del_rcu(&inode->i_hash); + inode->i_hash.pprev = NULL; + inode->i_hash_head = NULL; + spin_unlock(&inode->i_lock); + hlist_bl_unlock(b); + break; + } + } EXPORT_SYMBOL(__remove_inode_hash); @@ -904,26 +947,28 @@ long prune_icache_sb(struct super_block *sb, struct shrink_control *sc) return freed; } -static void __wait_on_freeing_inode(struct inode *inode); +static void __wait_on_freeing_inode(struct hlist_bl_head *b, + struct inode *inode); /* * Called with the inode lock held. */ static struct inode *find_inode(struct super_block *sb, - struct hlist_head *head, + struct hlist_bl_head *b, int (*test)(struct inode *, void *), void *data) { + struct hlist_bl_node *node; struct inode *inode = NULL; repeat: - hlist_for_each_entry(inode, head, i_hash) { + hlist_bl_for_each_entry(inode, node, b, i_hash) { if (inode->i_sb != sb) continue; if (!test(inode, data)) continue; spin_lock(&inode->i_lock); if (inode->i_state & (I_FREEING|I_WILL_FREE)) { - __wait_on_freeing_inode(inode); + __wait_on_freeing_inode(b, inode); goto repeat; } if (unlikely(inode->i_state & I_CREATING)) { @@ -942,19 +987,20 @@ static struct inode *find_inode(struct super_block *sb, * iget_locked for details. */ static struct inode *find_inode_fast(struct super_block *sb, - struct hlist_head *head, unsigned long ino) + struct hlist_bl_head *b, unsigned long ino) { + struct hlist_bl_node *node; struct inode *inode = NULL; repeat: - hlist_for_each_entry(inode, head, i_hash) { + hlist_bl_for_each_entry(inode, node, b, i_hash) { if (inode->i_ino != ino) continue; if (inode->i_sb != sb) continue; spin_lock(&inode->i_lock); if (inode->i_state & (I_FREEING|I_WILL_FREE)) { - __wait_on_freeing_inode(inode); + __wait_on_freeing_inode(b, inode); goto repeat; } if (unlikely(inode->i_state & I_CREATING)) { @@ -1162,25 +1208,25 @@ EXPORT_SYMBOL(unlock_two_nondirectories); * return it locked, hashed, and with the I_NEW flag set. The file system gets * to fill it in before unlocking it via unlock_new_inode(). * - * Note both @test and @set are called with the inode_hash_lock held, so can't - * sleep. + * Note both @test and @set are called with the inode hash chain lock held, + * so can't sleep. */ struct inode *inode_insert5(struct inode *inode, unsigned long hashval, int (*test)(struct inode *, void *), int (*set)(struct inode *, void *), void *data) { - struct hlist_head *head = i_hash_head(inode->i_sb, hashval); + struct hlist_bl_head *b = i_hash_head(inode->i_sb, hashval); struct inode *old; again: - spin_lock(&inode_hash_lock); - old = find_inode(inode->i_sb, head, test, data); + hlist_bl_lock(b); + old = find_inode(inode->i_sb, b, test, data); if (unlikely(old)) { /* * Uhhuh, somebody else created the same inode under us. * Use the old inode instead of the preallocated one. */ - spin_unlock(&inode_hash_lock); + hlist_bl_unlock(b); if (IS_ERR(old)) return NULL; wait_on_inode(old); @@ -1202,7 +1248,7 @@ struct inode *inode_insert5(struct inode *inode, unsigned long hashval, */ spin_lock(&inode->i_lock); inode->i_state |= I_NEW; - hlist_add_head_rcu(&inode->i_hash, head); + __insert_inode_hash_head(inode, b); spin_unlock(&inode->i_lock); /* @@ -1212,7 +1258,7 @@ struct inode *inode_insert5(struct inode *inode, unsigned long hashval, if (list_empty(&inode->i_sb_list)) inode_sb_list_add(inode); unlock: - spin_unlock(&inode_hash_lock); + hlist_bl_unlock(b); return inode; } @@ -1273,12 +1319,12 @@ EXPORT_SYMBOL(iget5_locked); */ struct inode *iget_locked(struct super_block *sb, unsigned long ino) { - struct hlist_head *head = i_hash_head(sb, ino); + struct hlist_bl_head *b = i_hash_head(sb, ino); struct inode *inode; again: - spin_lock(&inode_hash_lock); - inode = find_inode_fast(sb, head, ino); - spin_unlock(&inode_hash_lock); + hlist_bl_lock(b); + inode = find_inode_fast(sb, b, ino); + hlist_bl_unlock(b); if (inode) { if (IS_ERR(inode)) return NULL; @@ -1294,17 +1340,17 @@ struct inode *iget_locked(struct super_block *sb, unsigned long ino) if (inode) { struct inode *old; - spin_lock(&inode_hash_lock); + hlist_bl_lock(b); /* We released the lock, so.. */ - old = find_inode_fast(sb, head, ino); + old = find_inode_fast(sb, b, ino); if (!old) { inode->i_ino = ino; spin_lock(&inode->i_lock); inode->i_state = I_NEW; - hlist_add_head_rcu(&inode->i_hash, head); + __insert_inode_hash_head(inode, b); spin_unlock(&inode->i_lock); inode_sb_list_add(inode); - spin_unlock(&inode_hash_lock); + hlist_bl_unlock(b); /* Return the locked inode with I_NEW set, the * caller is responsible for filling in the contents @@ -1317,7 +1363,7 @@ struct inode *iget_locked(struct super_block *sb, unsigned long ino) * us. Use the old inode instead of the one we just * allocated. */ - spin_unlock(&inode_hash_lock); + hlist_bl_unlock(b); destroy_inode(inode); if (IS_ERR(old)) return NULL; @@ -1341,10 +1387,11 @@ EXPORT_SYMBOL(iget_locked); */ static int test_inode_iunique(struct super_block *sb, unsigned long ino) { - struct hlist_head *b = i_hash_head(sb, ino); + struct hlist_bl_head *b = i_hash_head(sb, ino); + struct hlist_bl_node *node; struct inode *inode; - hlist_for_each_entry_rcu(inode, b, i_hash) { + hlist_bl_for_each_entry_rcu(inode, node, b, i_hash) { if (inode->i_ino == ino && inode->i_sb == sb) return 0; } @@ -1428,12 +1475,12 @@ EXPORT_SYMBOL(igrab); struct inode *ilookup5_nowait(struct super_block *sb, unsigned long hashval, int (*test)(struct inode *, void *), void *data) { - struct hlist_head *head = i_hash_head(sb, hashval); + struct hlist_bl_head *b = i_hash_head(sb, hashval); struct inode *inode; - spin_lock(&inode_hash_lock); - inode = find_inode(sb, head, test, data); - spin_unlock(&inode_hash_lock); + hlist_bl_lock(b); + inode = find_inode(sb, b, test, data); + hlist_bl_unlock(b); return IS_ERR(inode) ? NULL : inode; } @@ -1483,12 +1530,12 @@ EXPORT_SYMBOL(ilookup5); */ struct inode *ilookup(struct super_block *sb, unsigned long ino) { - struct hlist_head *head = i_hash_head(sb, ino); + struct hlist_bl_head *b = i_hash_head(sb, ino); struct inode *inode; again: - spin_lock(&inode_hash_lock); - inode = find_inode_fast(sb, head, ino); - spin_unlock(&inode_hash_lock); + hlist_bl_lock(b); + inode = find_inode_fast(sb, b, ino); + hlist_bl_unlock(b); if (inode) { if (IS_ERR(inode)) @@ -1532,12 +1579,13 @@ struct inode *find_inode_nowait(struct super_block *sb, void *), void *data) { - struct hlist_head *head = i_hash_head(sb, hashval); + struct hlist_bl_head *b = i_hash_head(sb, hashval); + struct hlist_bl_node *node; struct inode *inode, *ret_inode = NULL; int mval; - spin_lock(&inode_hash_lock); - hlist_for_each_entry(inode, head, i_hash) { + hlist_bl_lock(b); + hlist_bl_for_each_entry(inode, node, b, i_hash) { if (inode->i_sb != sb) continue; mval = match(inode, hashval, data); @@ -1548,7 +1596,7 @@ struct inode *find_inode_nowait(struct super_block *sb, goto out; } out: - spin_unlock(&inode_hash_lock); + hlist_bl_unlock(b); return ret_inode; } EXPORT_SYMBOL(find_inode_nowait); @@ -1577,13 +1625,14 @@ EXPORT_SYMBOL(find_inode_nowait); struct inode *find_inode_rcu(struct super_block *sb, unsigned long hashval, int (*test)(struct inode *, void *), void *data) { - struct hlist_head *head = i_hash_head(sb, hashval); + struct hlist_bl_head *b = i_hash_head(sb, hashval); + struct hlist_bl_node *node; struct inode *inode; RCU_LOCKDEP_WARN(!rcu_read_lock_held(), "suspicious find_inode_rcu() usage"); - hlist_for_each_entry_rcu(inode, head, i_hash) { + hlist_bl_for_each_entry_rcu(inode, node, b, i_hash) { if (inode->i_sb == sb && !(READ_ONCE(inode->i_state) & (I_FREEING | I_WILL_FREE)) && test(inode, data)) @@ -1615,13 +1664,14 @@ EXPORT_SYMBOL(find_inode_rcu); struct inode *find_inode_by_ino_rcu(struct super_block *sb, unsigned long ino) { - struct hlist_head *head = i_hash_head(sb, ino); + struct hlist_bl_head *b = i_hash_head(sb, ino); + struct hlist_bl_node *node; struct inode *inode; RCU_LOCKDEP_WARN(!rcu_read_lock_held(), "suspicious find_inode_by_ino_rcu() usage"); - hlist_for_each_entry_rcu(inode, head, i_hash) { + hlist_bl_for_each_entry_rcu(inode, node, b, i_hash) { if (inode->i_ino == ino && inode->i_sb == sb && !(READ_ONCE(inode->i_state) & (I_FREEING | I_WILL_FREE))) @@ -1635,39 +1685,42 @@ int insert_inode_locked(struct inode *inode) { struct super_block *sb = inode->i_sb; ino_t ino = inode->i_ino; - struct hlist_head *head = i_hash_head(sb, ino); + struct hlist_bl_head *b = i_hash_head(sb, ino); while (1) { - struct inode *old = NULL; - spin_lock(&inode_hash_lock); - hlist_for_each_entry(old, head, i_hash) { - if (old->i_ino != ino) + struct hlist_bl_node *node; + struct inode *old = NULL, *t; + + hlist_bl_lock(b); + hlist_bl_for_each_entry(t, node, b, i_hash) { + if (t->i_ino != ino) continue; - if (old->i_sb != sb) + if (t->i_sb != sb) continue; - spin_lock(&old->i_lock); - if (old->i_state & (I_FREEING|I_WILL_FREE)) { - spin_unlock(&old->i_lock); + spin_lock(&t->i_lock); + if (t->i_state & (I_FREEING|I_WILL_FREE)) { + spin_unlock(&t->i_lock); continue; } + old = t; break; } if (likely(!old)) { spin_lock(&inode->i_lock); inode->i_state |= I_NEW | I_CREATING; - hlist_add_head_rcu(&inode->i_hash, head); + __insert_inode_hash_head(inode, b); spin_unlock(&inode->i_lock); - spin_unlock(&inode_hash_lock); + hlist_bl_unlock(b); return 0; } if (unlikely(old->i_state & I_CREATING)) { spin_unlock(&old->i_lock); - spin_unlock(&inode_hash_lock); + hlist_bl_unlock(b); return -EBUSY; } __iget(old); spin_unlock(&old->i_lock); - spin_unlock(&inode_hash_lock); + hlist_bl_unlock(b); wait_on_inode(old); if (unlikely(!inode_unhashed(old))) { iput(old); @@ -2192,17 +2245,18 @@ EXPORT_SYMBOL(inode_needs_sync); * wake_up_bit(&inode->i_state, __I_NEW) after removing from the hash list * will DTRT. */ -static void __wait_on_freeing_inode(struct inode *inode) +static void __wait_on_freeing_inode(struct hlist_bl_head *b, + struct inode *inode) { wait_queue_head_t *wq; DEFINE_WAIT_BIT(wait, &inode->i_state, __I_NEW); wq = bit_waitqueue(&inode->i_state, __I_NEW); prepare_to_wait(wq, &wait.wq_entry, TASK_UNINTERRUPTIBLE); spin_unlock(&inode->i_lock); - spin_unlock(&inode_hash_lock); + hlist_bl_unlock(b); schedule(); finish_wait(wq, &wait.wq_entry); - spin_lock(&inode_hash_lock); + hlist_bl_lock(b); } static __initdata unsigned long ihash_entries; @@ -2228,7 +2282,7 @@ void __init inode_init_early(void) inode_hashtable = alloc_large_system_hash("Inode-cache", - sizeof(struct hlist_head), + sizeof(struct hlist_bl_head), ihash_entries, 14, HASH_EARLY | HASH_ZERO, @@ -2254,7 +2308,7 @@ void __init inode_init(void) inode_hashtable = alloc_large_system_hash("Inode-cache", - sizeof(struct hlist_head), + sizeof(struct hlist_bl_head), ihash_entries, 14, HASH_ZERO, diff --git a/include/linux/fs.h b/include/linux/fs.h index 1a6f951942..db8d49cbf7 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -647,7 +647,8 @@ struct inode { unsigned long dirtied_when; /* jiffies of first dirtying */ unsigned long dirtied_time_when; - struct hlist_node i_hash; + struct hlist_bl_node i_hash; + struct hlist_bl_head *i_hash_head; struct list_head i_io_list; /* backing dev IO list */ #ifdef CONFIG_CGROUP_WRITEBACK struct bdi_writeback *i_wb; /* the associated cgroup wb */ @@ -713,7 +714,7 @@ static inline unsigned int i_blocksize(const struct inode *node) static inline int inode_unhashed(struct inode *inode) { - return hlist_unhashed(&inode->i_hash); + return hlist_bl_unhashed(&inode->i_hash); } /* @@ -724,7 +725,7 @@ static inline int inode_unhashed(struct inode *inode) */ static inline void inode_fake_hash(struct inode *inode) { - hlist_add_fake(&inode->i_hash); + hlist_bl_add_fake(&inode->i_hash); } /* @@ -2695,7 +2696,7 @@ static inline void insert_inode_hash(struct inode *inode) extern void __remove_inode_hash(struct inode *); static inline void remove_inode_hash(struct inode *inode) { - if (!inode_unhashed(inode) && !hlist_fake(&inode->i_hash)) + if (!inode_unhashed(inode) && !hlist_bl_fake(&inode->i_hash)) __remove_inode_hash(inode); } -- 2.40.1 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH 22/32] vfs: inode cache conversion to hash-bl 2023-05-09 16:56 ` [PATCH 22/32] vfs: inode cache conversion to hash-bl Kent Overstreet @ 2023-05-10 4:45 ` Dave Chinner 2023-05-16 15:45 ` Christian Brauner 0 siblings, 1 reply; 7+ messages in thread From: Dave Chinner @ 2023-05-10 4:45 UTC (permalink / raw) To: Kent Overstreet Cc: linux-kernel, linux-fsdevel, linux-bcachefs, Dave Chinner, Alexander Viro, Christian Brauner On Tue, May 09, 2023 at 12:56:47PM -0400, Kent Overstreet wrote: > From: Dave Chinner <dchinner@redhat.com> > > Because scalability of the global inode_hash_lock really, really > sucks. > > 32-way concurrent create on a couple of different filesystems > before: > > - 52.13% 0.04% [kernel] [k] ext4_create > - 52.09% ext4_create > - 41.03% __ext4_new_inode > - 29.92% insert_inode_locked > - 25.35% _raw_spin_lock > - do_raw_spin_lock > - 24.97% __pv_queued_spin_lock_slowpath > > - 72.33% 0.02% [kernel] [k] do_filp_open > - 72.31% do_filp_open > - 72.28% path_openat > - 57.03% bch2_create > - 56.46% __bch2_create > - 40.43% inode_insert5 > - 36.07% _raw_spin_lock > - do_raw_spin_lock > 35.86% __pv_queued_spin_lock_slowpath > 4.02% find_inode > > Convert the inode hash table to a RCU-aware hash-bl table just like > the dentry cache. Note that we need to store a pointer to the > hlist_bl_head the inode has been added to in the inode so that when > it comes to unhash the inode we know what list to lock. We need to > do this because the hash value that is used to hash the inode is > generated from the inode itself - filesystems can provide this > themselves so we have to either store the hash or the head pointer > in the inode to be able to find the right list head for removal... > > Same workload after: > > Signed-off-by: Dave Chinner <dchinner@redhat.com> > Cc: Alexander Viro <viro@zeniv.linux.org.uk> > Cc: Christian Brauner <brauner@kernel.org> > Cc: linux-fsdevel@vger.kernel.org > Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev> I have been maintaining this patchset uptodate in my own local trees and the code in this patch looks the same. The commit message above, however, has been mangled. The full commit message should be: vfs: inode cache conversion to hash-bl Because scalability of the global inode_hash_lock really, really sucks and prevents me from doing scalability characterisation and analysis of bcachefs algorithms. Profiles of a 32-way concurrent create of 51.2m inodes with fsmark on a couple of different filesystems on a 5.10 kernel: - 52.13% 0.04% [kernel] [k] ext4_create - 52.09% ext4_create - 41.03% __ext4_new_inode - 29.92% insert_inode_locked - 25.35% _raw_spin_lock - do_raw_spin_lock - 24.97% __pv_queued_spin_lock_slowpath - 72.33% 0.02% [kernel] [k] do_filp_open - 72.31% do_filp_open - 72.28% path_openat - 57.03% bch2_create - 56.46% __bch2_create - 40.43% inode_insert5 - 36.07% _raw_spin_lock - do_raw_spin_lock 35.86% __pv_queued_spin_lock_slowpath 4.02% find_inode btrfs was tested but it is limited by internal lock contention at >=2 threads on this workload, so never hammers the inode cache lock hard enough for this change to matter to it's performance. However, both bcachefs and ext4 demonstrate poor scaling at >=8 threads on concurrent lookup or create workloads. Hence convert the inode hash table to a RCU-aware hash-bl table just like the dentry cache. Note that we need to store a pointer to the hlist_bl_head the inode has been added to in the inode so that when it comes to unhash the inode we know what list to lock. We need to do this because, unlike the dentry cache, the hash value that is used to hash the inode is not generated from the inode itself. i.e. filesystems can provide this themselves so we have to either store the hashval or the hlist head pointer in the inode to be able to find the right list head for removal... Concurrent create with variying thread count (files/s): ext4 bcachefs threads vanilla patched vanilla patched 2 117k 112k 80k 85k 4 185k 190k 133k 145k 8 303k 346k 185k 255k 16 389k 465k 190k 420k 32 360k 437k 142k 481k CPU usage for both bcachefs and ext4 at 16 and 32 threads has been halved on the patched kernel, while performance has increased marginally on ext4 and massively on bcachefs. Internal filesystem algorithms now limit performance on these workloads, not the global inode_hash_lock. Profile of the workloads on the patched kernels: - 35.94% 0.07% [kernel] [k] ext4_create - 35.87% ext4_create - 20.45% __ext4_new_inode ... 3.36% insert_inode_locked - 78.43% do_filp_open - 78.36% path_openat - 53.95% bch2_create - 47.99% __bch2_create .... - 7.57% inode_insert5 6.94% find_inode Spinlock contention is largely gone from the inode hash operations and the filesystems are limited by contention in their internal algorithms. Signed-off-by: Dave Chinner <dchinner@redhat.com> --- Other than that, the diffstat is the same and I don't see any obvious differences in the code comapred to what I've been running locally. -Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 22/32] vfs: inode cache conversion to hash-bl 2023-05-10 4:45 ` Dave Chinner @ 2023-05-16 15:45 ` Christian Brauner 2023-05-16 16:17 ` Kent Overstreet 0 siblings, 1 reply; 7+ messages in thread From: Christian Brauner @ 2023-05-16 15:45 UTC (permalink / raw) To: Dave Chinner Cc: Kent Overstreet, linux-kernel, linux-fsdevel, linux-bcachefs, Dave Chinner, Alexander Viro On Wed, May 10, 2023 at 02:45:57PM +1000, Dave Chinner wrote: > On Tue, May 09, 2023 at 12:56:47PM -0400, Kent Overstreet wrote: > > From: Dave Chinner <dchinner@redhat.com> > > > > Because scalability of the global inode_hash_lock really, really > > sucks. > > > > 32-way concurrent create on a couple of different filesystems > > before: > > > > - 52.13% 0.04% [kernel] [k] ext4_create > > - 52.09% ext4_create > > - 41.03% __ext4_new_inode > > - 29.92% insert_inode_locked > > - 25.35% _raw_spin_lock > > - do_raw_spin_lock > > - 24.97% __pv_queued_spin_lock_slowpath > > > > - 72.33% 0.02% [kernel] [k] do_filp_open > > - 72.31% do_filp_open > > - 72.28% path_openat > > - 57.03% bch2_create > > - 56.46% __bch2_create > > - 40.43% inode_insert5 > > - 36.07% _raw_spin_lock > > - do_raw_spin_lock > > 35.86% __pv_queued_spin_lock_slowpath > > 4.02% find_inode > > > > Convert the inode hash table to a RCU-aware hash-bl table just like > > the dentry cache. Note that we need to store a pointer to the > > hlist_bl_head the inode has been added to in the inode so that when > > it comes to unhash the inode we know what list to lock. We need to > > do this because the hash value that is used to hash the inode is > > generated from the inode itself - filesystems can provide this > > themselves so we have to either store the hash or the head pointer > > in the inode to be able to find the right list head for removal... > > > > Same workload after: > > > > Signed-off-by: Dave Chinner <dchinner@redhat.com> > > Cc: Alexander Viro <viro@zeniv.linux.org.uk> > > Cc: Christian Brauner <brauner@kernel.org> > > Cc: linux-fsdevel@vger.kernel.org > > Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev> > > I have been maintaining this patchset uptodate in my own local trees > and the code in this patch looks the same. The commit message above, > however, has been mangled. The full commit message should be: > > vfs: inode cache conversion to hash-bl > > Because scalability of the global inode_hash_lock really, really > sucks and prevents me from doing scalability characterisation and > analysis of bcachefs algorithms. > > Profiles of a 32-way concurrent create of 51.2m inodes with fsmark > on a couple of different filesystems on a 5.10 kernel: > > - 52.13% 0.04% [kernel] [k] ext4_create > - 52.09% ext4_create > - 41.03% __ext4_new_inode > - 29.92% insert_inode_locked > - 25.35% _raw_spin_lock > - do_raw_spin_lock > - 24.97% __pv_queued_spin_lock_slowpath > > > - 72.33% 0.02% [kernel] [k] do_filp_open > - 72.31% do_filp_open > - 72.28% path_openat > - 57.03% bch2_create > - 56.46% __bch2_create > - 40.43% inode_insert5 > - 36.07% _raw_spin_lock > - do_raw_spin_lock > 35.86% __pv_queued_spin_lock_slowpath > 4.02% find_inode > > btrfs was tested but it is limited by internal lock contention at > >=2 threads on this workload, so never hammers the inode cache lock > hard enough for this change to matter to it's performance. > > However, both bcachefs and ext4 demonstrate poor scaling at >=8 > threads on concurrent lookup or create workloads. > > Hence convert the inode hash table to a RCU-aware hash-bl table just > like the dentry cache. Note that we need to store a pointer to the > hlist_bl_head the inode has been added to in the inode so that when > it comes to unhash the inode we know what list to lock. We need to > do this because, unlike the dentry cache, the hash value that is > used to hash the inode is not generated from the inode itself. i.e. > filesystems can provide this themselves so we have to either store > the hashval or the hlist head pointer in the inode to be able to > find the right list head for removal... > > Concurrent create with variying thread count (files/s): > > ext4 bcachefs > threads vanilla patched vanilla patched > 2 117k 112k 80k 85k > 4 185k 190k 133k 145k > 8 303k 346k 185k 255k > 16 389k 465k 190k 420k > 32 360k 437k 142k 481k > > CPU usage for both bcachefs and ext4 at 16 and 32 threads has been > halved on the patched kernel, while performance has increased > marginally on ext4 and massively on bcachefs. Internal filesystem > algorithms now limit performance on these workloads, not the global > inode_hash_lock. > > Profile of the workloads on the patched kernels: > > - 35.94% 0.07% [kernel] [k] ext4_create > - 35.87% ext4_create > - 20.45% __ext4_new_inode > ... > 3.36% insert_inode_locked > > - 78.43% do_filp_open > - 78.36% path_openat > - 53.95% bch2_create > - 47.99% __bch2_create > .... > - 7.57% inode_insert5 > 6.94% find_inode > > Spinlock contention is largely gone from the inode hash operations > and the filesystems are limited by contention in their internal > algorithms. > > Signed-off-by: Dave Chinner <dchinner@redhat.com> > --- > > Other than that, the diffstat is the same and I don't see any obvious > differences in the code comapred to what I've been running locally. There's a bit of a backlog before I get around to looking at this but it'd be great if we'd have a few reviewers for this change. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 22/32] vfs: inode cache conversion to hash-bl 2023-05-16 15:45 ` Christian Brauner @ 2023-05-16 16:17 ` Kent Overstreet 2023-05-16 23:15 ` Dave Chinner 0 siblings, 1 reply; 7+ messages in thread From: Kent Overstreet @ 2023-05-16 16:17 UTC (permalink / raw) To: Christian Brauner Cc: Dave Chinner, linux-kernel, linux-fsdevel, linux-bcachefs, Dave Chinner, Alexander Viro On Tue, May 16, 2023 at 05:45:19PM +0200, Christian Brauner wrote: > On Wed, May 10, 2023 at 02:45:57PM +1000, Dave Chinner wrote: > There's a bit of a backlog before I get around to looking at this but > it'd be great if we'd have a few reviewers for this change. It is well tested - it's been in the bcachefs tree for ages with zero issues. I'm pulling it out of the bcachefs-prerequisites series though since Dave's still got it in his tree, he's got a newer version with better commit messages. It's a significant performance boost on metadata heavy workloads for any non-XFS filesystem, we should definitely get it in. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 22/32] vfs: inode cache conversion to hash-bl 2023-05-16 16:17 ` Kent Overstreet @ 2023-05-16 23:15 ` Dave Chinner 2023-05-22 13:04 ` Christian Brauner 0 siblings, 1 reply; 7+ messages in thread From: Dave Chinner @ 2023-05-16 23:15 UTC (permalink / raw) To: Kent Overstreet Cc: Christian Brauner, linux-kernel, linux-fsdevel, linux-bcachefs, Dave Chinner, Alexander Viro On Tue, May 16, 2023 at 12:17:04PM -0400, Kent Overstreet wrote: > On Tue, May 16, 2023 at 05:45:19PM +0200, Christian Brauner wrote: > > On Wed, May 10, 2023 at 02:45:57PM +1000, Dave Chinner wrote: > > There's a bit of a backlog before I get around to looking at this but > > it'd be great if we'd have a few reviewers for this change. > > It is well tested - it's been in the bcachefs tree for ages with zero > issues. I'm pulling it out of the bcachefs-prerequisites series though > since Dave's still got it in his tree, he's got a newer version with > better commit messages. > > It's a significant performance boost on metadata heavy workloads for any > non-XFS filesystem, we should definitely get it in. I've got an up to date vfs-scale tree here (6.4-rc1) but I have not been able to test it effectively right now because my local performance test server is broken. I'll do what I can on the old small machine that I have to validate it when I get time, but that might be a few weeks away.... git://git.kernel.org/pub/scm/linux/kernel/git/dgc/linux-xfs.git vfs-scale As it is, the inode hash-bl changes have zero impact on XFS because it has it's own highly scalable lockless, sharded inode cache. So unless I'm explicitly testing ext4 or btrfs scalability (rare) it's not getting a lot of scalability exercise. It is being used by the root filesytsems on all those test VMs, but that's about it... That said, my vfs-scale tree also has Waiman Long's old dlist code (per cpu linked list) which converts the sb inode list and removes the global lock there. This does make a huge impact for XFS - the current code limits inode cache cycling to about 600,000 inodes/sec on >=16p machines. With dlists, however: | 5.17.0 on a XFS filesystem with 50 million inodes in it on a 32p | machine with a 1.6MIOPS/6.5GB/s block device. | | Fully concurrent full filesystem bulkstat: | | wall time sys time IOPS BW rate | unpatched: 1m56.035s 56m12.234s 8k 200MB/s 0.4M/s | patched: 0m15.710s 3m45.164s 70k 1.9GB/s 3.4M/s | | Unpatched flat kernel profile: | | 81.97% [kernel] [k] __pv_queued_spin_lock_slowpath | 1.84% [kernel] [k] do_raw_spin_lock | 1.33% [kernel] [k] __raw_callee_save___pv_queued_spin_unlock | 0.50% [kernel] [k] memset_erms | 0.42% [kernel] [k] do_raw_spin_unlock | 0.42% [kernel] [k] xfs_perag_get | 0.40% [kernel] [k] xfs_buf_find | 0.39% [kernel] [k] __raw_spin_lock_init | | Patched flat kernel profile: | | 10.90% [kernel] [k] do_raw_spin_lock | 7.21% [kernel] [k] __raw_callee_save___pv_queued_spin_unlock | 3.16% [kernel] [k] xfs_buf_find | 3.06% [kernel] [k] rcu_segcblist_enqueue | 2.73% [kernel] [k] memset_erms | 2.31% [kernel] [k] __pv_queued_spin_lock_slowpath | 2.15% [kernel] [k] __raw_spin_lock_init | 2.15% [kernel] [k] do_raw_spin_unlock | 2.12% [kernel] [k] xfs_perag_get | 1.93% [kernel] [k] xfs_btree_lookup Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 22/32] vfs: inode cache conversion to hash-bl 2023-05-16 23:15 ` Dave Chinner @ 2023-05-22 13:04 ` Christian Brauner 0 siblings, 0 replies; 7+ messages in thread From: Christian Brauner @ 2023-05-22 13:04 UTC (permalink / raw) To: Dave Chinner Cc: Kent Overstreet, linux-kernel, linux-fsdevel, linux-bcachefs, Dave Chinner, Alexander Viro On Wed, May 17, 2023 at 09:15:34AM +1000, Dave Chinner wrote: > On Tue, May 16, 2023 at 12:17:04PM -0400, Kent Overstreet wrote: > > On Tue, May 16, 2023 at 05:45:19PM +0200, Christian Brauner wrote: > > > On Wed, May 10, 2023 at 02:45:57PM +1000, Dave Chinner wrote: > > > There's a bit of a backlog before I get around to looking at this but > > > it'd be great if we'd have a few reviewers for this change. > > > > It is well tested - it's been in the bcachefs tree for ages with zero > > issues. I'm pulling it out of the bcachefs-prerequisites series though > > since Dave's still got it in his tree, he's got a newer version with > > better commit messages. > > > > It's a significant performance boost on metadata heavy workloads for any > > non-XFS filesystem, we should definitely get it in. > > I've got an up to date vfs-scale tree here (6.4-rc1) but I have not > been able to test it effectively right now because my local > performance test server is broken. I'll do what I can on the old > small machine that I have to validate it when I get time, but that > might be a few weeks away.... > > git://git.kernel.org/pub/scm/linux/kernel/git/dgc/linux-xfs.git vfs-scale > > As it is, the inode hash-bl changes have zero impact on XFS because > it has it's own highly scalable lockless, sharded inode cache. So > unless I'm explicitly testing ext4 or btrfs scalability (rare) it's > not getting a lot of scalability exercise. It is being used by the > root filesytsems on all those test VMs, but that's about it... I think there's a bunch of perf tests being run on -next. So we can stuff it into a vfs.unstable.* branch and see what -next thinks of this performance wise. ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2023-05-22 13:05 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2023-05-10 19:55 [PATCH 22/32] vfs: inode cache conversion to hash-bl kernel test robot -- strict thread matches above, loose matches on Subject: below -- 2023-05-09 16:56 [PATCH 00/32] bcachefs - a new COW filesystem Kent Overstreet 2023-05-09 16:56 ` [PATCH 22/32] vfs: inode cache conversion to hash-bl Kent Overstreet 2023-05-10 4:45 ` Dave Chinner 2023-05-16 15:45 ` Christian Brauner 2023-05-16 16:17 ` Kent Overstreet 2023-05-16 23:15 ` Dave Chinner 2023-05-22 13:04 ` Christian Brauner
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.