From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pj2-f12.google.com (mail-pj2-f12.google.com [74.125.227.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 87C1D43E06C for ; Mon, 28 Sep 2026 06:14:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.140 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790576079; cv=none; b=DZTblH2ynP41pbW9F0v6CLzAk/KqVnHcT5rFooePJMeyFyTWHaTZQ1Qjiv64Eyld6RqwqHBmKOfrNyiH18yOOhSoiMXjR7s3h+KiZh/hczcr0gxNF1Nw6ILYUUurhjQ8mXD/b1nWcps2uAF3NUbmTqgyVDj0e/1UQ0nYSUQN4sc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790576079; c=relaxed/simple; bh=QiJca58XzZkqmWwyVBYyOkEVUTnI4Kx5mky2VB2MbSk=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=PxI1rLBqf7azP2zwJF73IOdz6VzMxdw3ggSbsgcUCQSQU+5A4webnzSrj5w+yYiLnTBnAGt2SGAS8JkAfaOB4geQ+6lVoTGkVEsgIfr70/vo0KkbomSqiVfpxqOxRsEV9ZIL1V2OR/G16ol1e8hssbRWvCgF9v/tqcLmpQJD2Sw= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=OM3jYLyw; arc=none smtp.client-ip=74.125.227.140 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="OM3jYLyw" Received: by mail-pj2-f12.google.com with SMTP id 98e67ed59e1d1-396ccafb751so1208196a91.2 for ; Sun, 27 Sep 2026 23:14:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1790576061; x=1791180861; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to:content-type; bh=oOOwgR3Na06hDrMsbkRMKNZF35be9FXivq+8tVzj7v0=; b=OM3jYLywMoM/Fayxd11fCnRm6BFXdU9kTL2Rcb3l/KJJx/idTK49l+ikJtXT5xLRo3 mtRZyq8L/wGBjH/1lyAHd8Nj5lzBUQ2sGbnoVPYSweP6d14oIXhS3AngaB5/JGW4Md93 lS2LMOh5wub3V7/K9ogBpjOM30IdXXa/K2zYP4pr+7sdN5Q9VJX+2HC1lA/l4m11f28h EDYuzIdbNt4FvmI5aghp2kdwTooHmqX2pWI9ym8qKBIfcMkoHQhkIVX0XfN9s69das6/ XBsE8hSe8TXva3VSIUL4G9fLJ811XI4kSwONWyQhG7tN5I3EgeflMqNYptfiCUD/6ouP ZfPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790576061; x=1791180861; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=oOOwgR3Na06hDrMsbkRMKNZF35be9FXivq+8tVzj7v0=; b=QdfqoKu6Cq8FSEY2CJjX9ElUl+jIjcFM/g4k9CFGJOa2PQ1Q/+wvV0yeuDKplST+UG h4MhRZ4EBoXak95xlE1RHP5pd997RlToZdNzg5oWQctFR9oXer4fbNi+utpMOx/DXKvE EE9PdCzMEd6SxdZHSLvODGrkVn/H9xZiK7y5igp18eqCeeIoQOhRT04l+fF/rhd2QVfK DuGU/7Y1EVHOHvdUsSU/Mbu43i6bX1eWC+iIZeR8mDwb/V6CCxxjCHSYCsqu4gbQDCH0 lPkhteiqdeGOIg3ke/avb/8b+qLSvVhsKwThciggQqoHyqEwngv+jYf1zE5aHkGi8zde 4bkQ== X-Forwarded-Encrypted: i=1; AKwUvBxuRfL8EMt5pXtFoMPQ5Lv0oy8oM9SMNRaLxCA+tTc+SKEn8t0VSrBfHKVbGbEgd9ZiiqVX8sLXNji7Eg==@vger.kernel.org X-Gm-Message-State: AFq9FYIXDtX0bt9ztwnNiN3FUBR2rWCJN7GZpXPA/we27zX59kxmi1rr YthG+1MEjpxZheXq1ga/Rfj5uVOB8LrhnjvQzbt45OEh2ML4mUALX7oj X-Gm-Gg: AYBFou2dG34qrEOoYokaEUVk5TKjGSloQqcB7QuxP8JIebCQQmJNR6ip1r16OUmYGH3 vOGiKCUp/WWu1K5dko2//XouobykASDAgU4k9/jFNLFeu45svfePpVV42cdSpvI59Mn9dFJ8u3j GnMMdAwEaFNXzEjrdR1FnS0WIvDtbQuAIZAqojepS8HvkkhvmW1KZtU/HPbgVVi6ZRfKk6LKubR ZJ11anzCAKcIdKTXjPN23WdOEC7685EKMRitx5tq0vi1/3i4VWg5bk85MFCJioQ011tczWuCzVc An1cF3JWSBs3+bUpqQL8uacoQQgjZ+PTLsuXQ/j8Jbzq9NDu71dMqZNBytjvfBny4v0h1spEtKG edem5Kx/HRtUZDBLm3RP1pa99yH+qw1og28Y3tugy+PvBkkQwflvF6rmLdZZWOFQSkqRue8WW/p oHsKa1ZVnvyyOdehCO+nLBYPzGDzWm8Sc1eXOv1qfM8mSLGfnHje7JZl2X9Xd9XColhQSRMnVhe 3rCXAiAGlvnGyyLAU9k/X4EoHQjJFCvvOlzpA1X4UYdRo9Bpyp6IVHRnnQESUuEda8mDTbypRWR /HkVQ3Up99YVsA== X-Received: by 2002:a17:90b:2c86:b0:3a2:bb45:d37b with SMTP id 98e67ed59e1d1-3a2bb45da47mr1186529a91.25.1790576060904; Sun, 27 Sep 2026 23:14:20 -0700 (PDT) Received: from yupeng-XPS-15-9520.. (c-73-169-192-12.hsd1.wa.comcast.net. [73.169.192.12]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-3a0b9892dbfsm18864088a91.11.2026.09.27.23.14.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 27 Sep 2026 23:14:20 -0700 (PDT) From: Peng Yu To: Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni Cc: Tejun Heo , Johannes Weiner , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Josef Bacik , Jens Axboe , Maurizio Lombardi , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, Peng Yu Subject: [PATCH v6 0/3] nvmet: add cgroup_id to charge namespace I/O to a cgroup Date: Sun, 27 Sep 2026 23:14:14 -0700 Message-ID: <20260928061417.1574676-1-yupeng0921@gmail.com> X-Mailer: git-send-email 2.43.0 Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Scenario: * Create multiple nvmet subsystems/namespaces. * The namespaces are backed by different LVM logical volumes. * Some of the logical volumes share the same physical volumes. * The subsystems are exported to different users. * We should provide each user a specific iops/bps quota, thus a noisy neighbor won't impact the performance of other logical volumes. Test: 1 Load kernel modules. sudo modprobe brd rd_nr=2 rd_size=1048576 sudo modprobe nvmet sudo modprobe nvmet_tcp sudo modprobe nvme_tcp 2 The two brd devices are used for a bdev backed namespace and a file backed namespace, remember their major/minor. lsblk --noheadings --nodeps --output NAME,MAJ:MIN,SIZE /dev/ram0 /dev/ram1 ram0 1:0 1G ram1 1:1 1G 3 Enable the io controller and create two cgroups, one for the bdev backed namespace, another for the file backed namespace. echo "+io" | sudo tee /sys/fs/cgroup/cgroup.subtree_control sudo mkdir -p /sys/fs/cgroup/nvmet-bdev sudo mkdir -p /sys/fs/cgroup/nvmet-file 4 Create a file system and a file on ram0. sudo mkfs.ext4 /dev/ram0 sudo mkdir -p /mnt/nvmet-file sudo mount -o noatime /dev/ram0 /mnt/nvmet-file sudo dd if=/dev/zero of=/mnt/nvmet-file/ns.img bs=1M count=512 oflag=direct 5 Create the bdev backed subsystem and namespace. sudo mkdir -p /sys/kernel/config/nvmet/subsystems/nqn.2026-09.io.test01:bdev/namespaces/1 echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/nqn.2026-09.io.test01:bdev/attr_allow_any_host echo /dev/ram1 | sudo tee /sys/kernel/config/nvmet/subsystems/nqn.2026-09.io.test01:bdev/namespaces/1/device_path stat -c %i /sys/fs/cgroup/nvmet-bdev | sudo tee /sys/kernel/config/nvmet/subsystems/nqn.2026-09.io.test01:bdev/namespaces/1/cgroup_id echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/nqn.2026-09.io.test01:bdev/namespaces/1/enable 6 Create the file backed subsystem and namespace. sudo mkdir -p /sys/kernel/config/nvmet/subsystems/nqn.2026-09.io.test01:file/namespaces/1 echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/nqn.2026-09.io.test01:file/attr_allow_any_host echo /mnt/nvmet-file/ns.img | sudo tee /sys/kernel/config/nvmet/subsystems/nqn.2026-09.io.test01:file/namespaces/1/device_path stat -c %i /sys/fs/cgroup/nvmet-file | sudo tee /sys/kernel/config/nvmet/subsystems/nqn.2026-09.io.test01:file/namespaces/1/cgroup_id echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/nqn.2026-09.io.test01:file/namespaces/1/enable 7 Export the two subsystems to 127.0.0.1. sudo mkdir -p /sys/kernel/config/nvmet/ports/1 echo ipv4 | sudo tee /sys/kernel/config/nvmet/ports/1/addr_adrfam echo tcp | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype echo 127.0.0.1 | sudo tee /sys/kernel/config/nvmet/ports/1/addr_traddr echo 4420 | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trsvcid sudo ln -s /sys/kernel/config/nvmet/subsystems/nqn.2026-09.io.test01:bdev /sys/kernel/config/nvmet/ports/1/subsystems/nqn.2026-09.io.test01:bdev sudo ln -s /sys/kernel/config/nvmet/subsystems/nqn.2026-09.io.test01:file /sys/kernel/config/nvmet/ports/1/subsystems/nqn.2026-09.io.test01:file 8 Connect the two subsystems from 127.0.0.1. sudo nvme connect -t tcp -a 127.0.0.1 -s 4420 -n nqn.2026-09.io.test01:bdev sudo nvme connect -t tcp -a 127.0.0.1 -s 4420 -n nqn.2026-09.io.test01:file sudo udevadm settle 9 Set riops to 20k and run fio. echo "1:1 riops=20000 wiops=max rbps=max wbps=max" | sudo tee /sys/fs/cgroup/nvmet-bdev/io.max echo "1:0 riops=20000 wiops=max rbps=max wbps=max" | sudo tee /sys/fs/cgroup/nvmet-file/io.max sudo fio --name=t --filename=/dev/nvme0n1 --rw=randread --bs=4k --iodepth=32 --numjobs=1 \ --direct=1 --ioengine=libaio --time_based --runtime=10 --ramp_time=3 \ --group_reporting 2>/dev/null | grep -E '^ +(read|write): IOPS' read: IOPS=20.0k, BW=78.1MiB/s (81.9MB/s)(781MiB/10001msec) sudo fio --name=t --filename=/dev/nvme1n1 --rw=randread --bs=4k --iodepth=32 --numjobs=1 \ --direct=1 --ioengine=libaio --time_based --runtime=10 --ramp_time=3 \ --group_reporting 2>/dev/null | grep -E '^ +(read|write): IOPS' read: IOPS=20.0k, BW=78.2MiB/s (82.0MB/s)(782MiB/10001msec) 10 Set rbps to 100M and run fio. echo "1:1 riops=max wiops=max rbps=104857600 wbps=max" | sudo tee /sys/fs/cgroup/nvmet-bdev/io.max echo "1:0 riops=max wiops=max rbps=104857600 wbps=max" | sudo tee /sys/fs/cgroup/nvmet-file/io.max sudo fio --name=t --filename=/dev/nvme0n1 --rw=read --bs=1M --iodepth=32 --numjobs=1 \ --direct=1 --ioengine=libaio --time_based --runtime=10 --ramp_time=3 \ --group_reporting 2>/dev/null | grep -E '^ +(read|write): IOPS' read: IOPS=97, BW=100MiB/s (105MB/s)(1041MiB/10399msec) sudo fio --name=t --filename=/dev/nvme1n1 --rw=read --bs=1M --iodepth=32 --numjobs=1 \ --direct=1 --ioengine=libaio --time_based --runtime=10 --ramp_time=3 \ --group_reporting 2>/dev/null | grep -E '^ +(read|write): IOPS' read: IOPS=97, BW=100MiB/s (105MB/s)(1031MiB/10299msec) 11 Set wiops to 20k and run fio. echo "1:1 riops=max wiops=20000 rbps=max wbps=max" | sudo tee /sys/fs/cgroup/nvmet-bdev/io.max echo "1:0 riops=max wiops=20000 rbps=max wbps=max" | sudo tee /sys/fs/cgroup/nvmet-file/io.max sudo fio --name=t --filename=/dev/nvme0n1 --rw=randwrite --bs=4k --iodepth=32 --numjobs=1 \ --direct=1 --ioengine=libaio --time_based --runtime=10 --ramp_time=3 \ --group_reporting 2>/dev/null | grep -E '^ +(read|write): IOPS' write: IOPS=20.0k, BW=78.1MiB/s (81.9MB/s)(781MiB/10001msec); 0 zone resets sudo fio --name=t --filename=/dev/nvme1n1 --rw=randwrite --bs=4k --iodepth=32 --numjobs=1 \ --direct=1 --ioengine=libaio --time_based --runtime=10 --ramp_time=3 \ --group_reporting 2>/dev/null | grep -E '^ +(read|write): IOPS' write: IOPS=20.0k, BW=78.1MiB/s (81.9MB/s)(781MiB/10002msec); 0 zone resets 12 Set wbps to 100M and run fio. echo "1:1 riops=max wiops=max rbps=max wbps=104857600" | sudo tee /sys/fs/cgroup/nvmet-bdev/io.max echo "1:0 riops=max wiops=max rbps=max wbps=104857600" | sudo tee /sys/fs/cgroup/nvmet-file/io.max sudo fio --name=t --filename=/dev/nvme0n1 --rw=write --bs=1M --iodepth=32 --numjobs=1 \ --direct=1 --ioengine=libaio --time_based --runtime=10 --ramp_time=3 \ --group_reporting 2>/dev/null | grep -E '^ +(read|write): IOPS' write: IOPS=97, BW=100MiB/s (105MB/s)(1032MiB/10307msec); 0 zone resets sudo fio --name=t --filename=/dev/nvme1n1 --rw=write --bs=1M --iodepth=32 --numjobs=1 \ --direct=1 --ioengine=libaio --time_based --runtime=10 --ramp_time=3 \ --group_reporting 2>/dev/null | grep -E '^ +(read|write): IOPS' write: IOPS=97, BW=100MiB/s (105MB/s)(1033MiB/10296msec); 0 zone resets 13 Cleanup the environment. sudo nvme disconnect -n nqn.2026-09.io.test01:bdev sudo nvme disconnect -n nqn.2026-09.io.test01:file sudo udevadm settle sudo rm -f /sys/kernel/config/nvmet/ports/1/subsystems/nqn.2026-09.io.test01:bdev sudo rm -f /sys/kernel/config/nvmet/ports/1/subsystems/nqn.2026-09.io.test01:file sudo rmdir /sys/kernel/config/nvmet/ports/1 echo 0 | sudo tee /sys/kernel/config/nvmet/subsystems/nqn.2026-09.io.test01:bdev/namespaces/1/enable sudo rmdir /sys/kernel/config/nvmet/subsystems/nqn.2026-09.io.test01:bdev/namespaces/1 sudo rmdir /sys/kernel/config/nvmet/subsystems/nqn.2026-09.io.test01:bdev echo 0 | sudo tee /sys/kernel/config/nvmet/subsystems/nqn.2026-09.io.test01:file/namespaces/1/enable sudo rmdir /sys/kernel/config/nvmet/subsystems/nqn.2026-09.io.test01:file/namespaces/1 sudo rmdir /sys/kernel/config/nvmet/subsystems/nqn.2026-09.io.test01:file sudo umount /mnt/nvmet-file sudo rmdir /mnt/nvmet-file sudo modprobe -r brd sudo rmdir /sys/fs/cgroup/nvmet-bdev sudo rmdir /sys/fs/cgroup/nvmet-file sudo rmmod nvme_tcp sudo rmmod nvmet_tcp sudo rmmod nvmet Peng Yu (3): cgroup: track the effective css in each cgroup cgroup: export cgroup_e_css() nvmet: add cgroup_id to charge namespace I/O to a cgroup drivers/nvme/target/configfs.c | 47 ++++++++++++++++++++++++ drivers/nvme/target/core.c | 54 ++++++++++++++++++++++++++++ drivers/nvme/target/io-cmd-bdev.c | 18 +++++++++- drivers/nvme/target/io-cmd-file.c | 21 +++++++++-- drivers/nvme/target/nvmet.h | 59 +++++++++++++++++++++++++++++++ drivers/nvme/target/zns.c | 5 +++ include/linux/cgroup-defs.h | 2 ++ kernel/cgroup/cgroup.c | 54 +++++++++++++++++----------- 8 files changed, 236 insertions(+), 24 deletions(-) base-commit: d9cc476535d29a44df3f2aa3b14af9a833fddf90 -- 2.53.0