* [PATCHSET v2 block/for-linus] IO cost model based work-conserving porportional controller
From: Tejun Heo @ 2019-07-10 20:51 UTC (permalink / raw)
To: axboe, newella, clm, josef, dennisz, lizefan, hannes
Cc: linux-kernel, kernel-team, linux-block, cgroups
Hello,
Changes from v1[1]:
* Prerequisite patchsets had cosmetic changes and merged. Refreshed
on top.
* Renamed from ioweight to iocost. All source code and tools are
updated accordingly. Control knobs io.weight.qos and
io.weight.cost_model are renamed to io.cost.qos and io.cost.model
respectively. This is a more fitting name which won't become a
misnomer when, for example, cost based io.max is added.
* Various bug fixes and improvements. A few bugs were discovered
while testing against high-iops nvme device. Auto parameter
selection improved and verified across different classes of SSDs.
* Dropped bpf iocost support for now.
* Added coef generation script.
* Verified on high-iops nvme device. Result is included below.
One challenge of controlling IO resources is the lack of trivially
observable cost metric. This is distinguished from CPU and memory
where wallclock time and the number of bytes can serve as accurate
enough approximations.
Bandwidth and iops are the most commonly used metrics for IO devices
but depending on the type and specifics of the device, different IO
patterns easily lead to multiple orders of magnitude variations
rendering them useless for the purpose of IO capacity distribution.
While on-device time, with a lot of clutches, could serve as a useful
approximation for non-queued rotational devices, this is no longer
viable with modern devices, even the rotational ones.
While there is no cost metric we can trivially observe, it isn't a
complete mystery. For example, on a rotational device, seek cost
dominates while a contiguous transfer contributes a smaller amount
proportional to the size. If we can characterize at least the
relative costs of these different types of IOs, it should be possible
to implement a reasonable work-conserving proportional IO resource
distribution.
This patchset implements IO cost model based work-conserving
proportional controller. It currently has a simple linear cost model
builtin where each IO is classified as sequential or random and given
a base cost accordingly and additional size-proportional cost is added
on top. Each IO is given a cost based on the model and the controller
issues IOs for each cgroup according to their hierarchical weight.
By default, the controller adapts its overall IO rate so that it
doesn't build up buffer bloat in the request_queue layer, which
guarantees that the controller doesn't lose significant amount of
total work. However, this may not provide sufficient differentiation
as the underlying device may have a deep queue and not be fair in how
the queued IOs are serviced. The controller provides extra QoS
control knobs which allow tightening control feedback loop as
necessary.
For more details on the control mechanism, implementation and
interface, please refer to the comment at the top of
block/blk-iocost.c and Documentation/admin-guide/cgroup-v2.rst changes
in the "blkcg: implement blk-iocost" patch.
Here are some test results. Each test run goes through the following
combinations with each combination running for a minute. All tests
are performed against regular files on btrfs w/ deadline as the IO
scheduler. Random IOs are direct w/ queue depth of 64. Sequential
are normal buffered IOs.
high priority (weight=500) low priority (weight=100)
Rand read None
ditto Rand read
ditto Seq read
ditto Rand write
ditto Seq write
Seq read None
ditto Rand read
ditto Seq read
ditto Rand write
ditto Seq write
Rand write None
ditto Rand read
ditto Seq read
ditto Rand write
ditto Seq write
Seq write None
ditto Rand read
ditto Seq read
ditto Rand write
ditto Seq write
* 7200RPM SATA hard disk
* No IO control
https://photos.app.goo.gl/1KBHn7ykpC1LXRkB8
* iocost, QoS: None
https://photos.app.goo.gl/MLNQGxCtBQ8wAmjm7
* iocost, QoS: rpct=95.00 rlat=40000 wpct=95.00 wlat=40000 min=25.00 max=200.00
https://photos.app.goo.gl/XqXHm3Mkbm9w6Db46
* NCQ-blacklisted SATA SSD (QD==1)
* No IO control
https://photos.app.goo.gl/wCTXeu2uJ6LYL4pk8
* iocost, QoS: None
https://photos.app.goo.gl/T2HedKD2sywQgj7R9
* iocost, QoS: rpct=95.00 rlat=20000 wpct=95.00 wlat=20000 min=50.00 max=200.00
https://photos.app.goo.gl/urBTV8XQc1UqPJJw7
* SATA SSD (QD==32)
* No IO control
https://photos.app.goo.gl/TjEVykuVudSQcryh6
* iocost, QoS: None
https://photos.app.goo.gl/iyQBsky7bmM54Xiq7
* iocost, QoS: rpct=95.00 rlat=10000 wpct=95.00 wlat=20000 min=50.00 max=400.00
https://photos.app.goo.gl/q1a6URLDxPLMrnHy5
* NVME SSD (ran with 8 concurrent fio jobs to achieve saturation)
* No IO control
https://photos.app.goo.gl/S6xjEVTJzcfb3w1j7
* iocost, QoS: None
https://photos.app.goo.gl/SjQUUotJBAGr7vqz7
* iocost, QoS: rpct=95.00 rlat=5000 wpct=95.00 wlat=5000 min=1.00 max=10000.00
https://photos.app.goo.gl/RsaYBd2muX7CegoN7
Even without explicit QoS configuration, read-heavy scenarios can
obtain acceptable differentiation. However, when write-heavy, the
deep buffering on the device side makes it difficult to maintain
control. With QoS parameters set, the differentiation is acceptable
across all combinations.
The implementation comes with default cost model parameters which are
selected automatically which should provide acceptable behavior across
most common devices. The parameters for hdd and consumer-grade SSDs
seem pretty robust. The default parameter set and selection criteria
for highend SSDs might need further adjustments.
It is fairly easy to configure the QoS parameters and, if needed, cost
model coefficients. We'll follow up with tooling and further
documentation. Also, the last RFC patch in the series implements
support for bpf-based custom cost function. Originally we thought
that we'd need per-device-type cost functions but the simple linear
model now seem good enough to cover all common device classes. In
case custom cost functions become necessary, we can fully develop the
bpf based extension and also easily add different builtin cost models.
Andy Newell did the heavy lifting of analyzing IO workloads and device
characteristics, exploring various cost models, determining the
default model and parameters to use.
Josef Bacik implemented a prototype which explored the use of
different types of cost metrics including on-device time and Andy's
linear model.
This patchset is on top of block/for-linus 3a10f999ffd4
("blk-throttle: fix zero wait time for iops throttled group")
and contains the following ten patches.
0001-blkcg-pass-q-and-blkcg-into-blkcg_pol_alloc_pd_fn.patch
0002-blkcg-make-cpd_init_fn-optional.patch
0003-blkcg-separate-blkcg_conf_get_disk-out-of-blkg_conf_.patch
0004-block-rq_qos-add-rq_qos_merge.patch
0005-block-rq_qos-implement-rq_qos_ops-queue_depth_change.patch
0006-blkcg-s-RQ_QOS_CGROUP-RQ_QOS_LATENCY.patch
0007-blk-mq-add-optional-request-pre_start_time_ns.patch
0008-blkcg-implement-blk-iocost.patch
0009-blkcg-add-tools-cgroup-iocost_monitor.py.patch
0010-blkcg-add-tools-cgroup-iocost_coef_gen.py.patch
0001-0007 are prep patches. 0008 implements blk-iocost. 0009 adds
monitoring script. 0010 adds linear model coef generation script.
The patchset is also available in the following git branch.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git review-iocost
diffstat follows, Thanks.
Documentation/admin-guide/cgroup-v2.rst | 97 +
block/Kconfig | 9
block/Makefile | 1
block/bfq-cgroup.c | 5
block/blk-cgroup.c | 71
block/blk-core.c | 4
block/blk-iocost.c | 2394 ++++++++++++++++++++++++++++++++
block/blk-iolatency.c | 8
block/blk-mq.c | 11
block/blk-rq-qos.c | 18
block/blk-rq-qos.h | 28
block/blk-settings.c | 2
block/blk-throttle.c | 6
block/blk-wbt.c | 18
block/blk-wbt.h | 4
include/linux/blk-cgroup.h | 4
include/linux/blk_types.h | 3
include/linux/blkdev.h | 7
include/trace/events/iocost.h | 174 ++
tools/cgroup/iocost_coef_gen.py | 178 ++
tools/cgroup/iocost_monitor.py | 270 +++
21 files changed, 3259 insertions(+), 53 deletions(-)
--
tejun
[1] http://lkml.kernel.org/r/20190614015620.1587672-1-tj@kernel.org
^ permalink raw reply
* [PATCH 02/10] blkcg: make ->cpd_init_fn() optional
From: Tejun Heo @ 2019-07-10 20:51 UTC (permalink / raw)
To: axboe, newella, clm, josef, dennisz, lizefan, hannes
Cc: linux-kernel, kernel-team, linux-block, cgroups, Tejun Heo
In-Reply-To: <20190710205128.1316483-1-tj@kernel.org>
For policies which can do enough initialization from ->cpd_alloc_fn(),
make ->cpd_init_fn() optional.
Signed-off-by: Tejun Heo <tj@kernel.org>
---
block/blk-cgroup.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index 75f7f78b87b2..818e50b4cc7a 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -1478,7 +1478,8 @@ int blkcg_policy_register(struct blkcg_policy *pol)
blkcg->cpd[pol->plid] = cpd;
cpd->blkcg = blkcg;
cpd->plid = pol->plid;
- pol->cpd_init_fn(cpd);
+ if (pol->cpd_init_fn)
+ pol->cpd_init_fn(cpd);
}
}
--
2.17.1
^ permalink raw reply related
* [PATCH 06/10] blkcg: s/RQ_QOS_CGROUP/RQ_QOS_LATENCY/
From: Tejun Heo @ 2019-07-10 20:51 UTC (permalink / raw)
To: axboe, newella, clm, josef, dennisz, lizefan, hannes
Cc: linux-kernel, kernel-team, linux-block, cgroups, Tejun Heo
In-Reply-To: <20190710205128.1316483-1-tj@kernel.org>
io.weight is gonna be another rq_qos cgroup mechanism. Let's rename
RQ_QOS_CGROUP which is being used by io.latency to RQ_QOS_LATENCY in
preparation.
Signed-off-by: Tejun Heo <tj@kernel.org>
---
block/blk-iolatency.c | 2 +-
block/blk-rq-qos.h | 8 ++++----
2 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/block/blk-iolatency.c b/block/blk-iolatency.c
index e613f89b37d3..5c53adbea3fa 100644
--- a/block/blk-iolatency.c
+++ b/block/blk-iolatency.c
@@ -725,7 +725,7 @@ int blk_iolatency_init(struct request_queue *q)
return -ENOMEM;
rqos = &blkiolat->rqos;
- rqos->id = RQ_QOS_CGROUP;
+ rqos->id = RQ_QOS_LATENCY;
rqos->ops = &blkcg_iolatency_ops;
rqos->q = q;
diff --git a/block/blk-rq-qos.h b/block/blk-rq-qos.h
index e15b6907b76d..5f8b75826a98 100644
--- a/block/blk-rq-qos.h
+++ b/block/blk-rq-qos.h
@@ -14,7 +14,7 @@ struct blk_mq_debugfs_attr;
enum rq_qos_id {
RQ_QOS_WBT,
- RQ_QOS_CGROUP,
+ RQ_QOS_LATENCY,
};
struct rq_wait {
@@ -74,7 +74,7 @@ static inline struct rq_qos *wbt_rq_qos(struct request_queue *q)
static inline struct rq_qos *blkcg_rq_qos(struct request_queue *q)
{
- return rq_qos_id(q, RQ_QOS_CGROUP);
+ return rq_qos_id(q, RQ_QOS_LATENCY);
}
static inline const char *rq_qos_id_to_name(enum rq_qos_id id)
@@ -82,8 +82,8 @@ static inline const char *rq_qos_id_to_name(enum rq_qos_id id)
switch (id) {
case RQ_QOS_WBT:
return "wbt";
- case RQ_QOS_CGROUP:
- return "cgroup";
+ case RQ_QOS_LATENCY:
+ return "latency";
}
return "unknown";
}
--
2.17.1
^ permalink raw reply related
* [PATCH 07/10] blk-mq: add optional request->pre_start_time_ns
From: Tejun Heo @ 2019-07-10 20:51 UTC (permalink / raw)
To: axboe, newella, clm, josef, dennisz, lizefan, hannes
Cc: linux-kernel, kernel-team, linux-block, cgroups, Tejun Heo
In-Reply-To: <20190710205128.1316483-1-tj@kernel.org>
There are currently two start time timestamps - start_time_ns and
io_start_time_ns. The former marks the request allocation and and the
second issue-to-device time. The planned io.weight controller needs
to measure the total time bios take to execute after it leaves rq_qos
including the time spent waiting for request to become available,
which can easily dominate on saturated devices.
This patch adds request->pre_start_time_ns which records when the
request allocation attempt started. As it isn't used for the usual
stats, make it optional behind QUEUE_FLAG_REC_PRESTART.
Signed-off-by: Tejun Heo <tj@kernel.org>
---
block/blk-mq.c | 11 +++++++++--
include/linux/blkdev.h | 7 ++++++-
2 files changed, 15 insertions(+), 3 deletions(-)
diff --git a/block/blk-mq.c b/block/blk-mq.c
index e5ef40c603ca..90b46988cc02 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -291,7 +291,7 @@ static inline bool blk_mq_need_time_stamp(struct request *rq)
}
static struct request *blk_mq_rq_ctx_init(struct blk_mq_alloc_data *data,
- unsigned int tag, unsigned int op)
+ unsigned int tag, unsigned int op, u64 pre_start_time_ns)
{
struct blk_mq_tags *tags = blk_mq_tags_from_data(data);
struct request *rq = tags->static_rqs[tag];
@@ -325,6 +325,7 @@ static struct request *blk_mq_rq_ctx_init(struct blk_mq_alloc_data *data,
RB_CLEAR_NODE(&rq->rb_node);
rq->rq_disk = NULL;
rq->part = NULL;
+ rq->pre_start_time_ns = pre_start_time_ns;
if (blk_mq_need_time_stamp(rq))
rq->start_time_ns = ktime_get_ns();
else
@@ -356,8 +357,14 @@ static struct request *blk_mq_get_request(struct request_queue *q,
struct request *rq;
unsigned int tag;
bool clear_ctx_on_error = false;
+ u64 pre_start_time_ns = 0;
blk_queue_enter_live(q);
+
+ /* pre_start_time includes depth and tag waits */
+ if (blk_queue_rec_prestart(q))
+ pre_start_time_ns = ktime_get_ns();
+
data->q = q;
if (likely(!data->ctx)) {
data->ctx = blk_mq_get_ctx(q);
@@ -393,7 +400,7 @@ static struct request *blk_mq_get_request(struct request_queue *q,
return NULL;
}
- rq = blk_mq_rq_ctx_init(data, tag, data->cmd_flags);
+ rq = blk_mq_rq_ctx_init(data, tag, data->cmd_flags, pre_start_time_ns);
if (!op_is_flush(data->cmd_flags)) {
rq->elv.icq = NULL;
if (e && e->type->ops.prepare_request) {
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 259bd7ad8312..4247a9bc44b7 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -194,7 +194,9 @@ struct request {
struct gendisk *rq_disk;
struct hd_struct *part;
- /* Time that I/O was submitted to the kernel. */
+ /* Time that the first bio started allocating this request. */
+ u64 pre_start_time_ns;
+ /* Time that this request was allocated for this IO. */
u64 start_time_ns;
/* Time that I/O was submitted to the device. */
u64 io_start_time_ns;
@@ -606,6 +608,7 @@ struct request_queue {
#define QUEUE_FLAG_SCSI_PASSTHROUGH 23 /* queue supports SCSI commands */
#define QUEUE_FLAG_QUIESCED 24 /* queue has been quiesced */
#define QUEUE_FLAG_PCI_P2PDMA 25 /* device supports PCI p2p requests */
+#define QUEUE_FLAG_REC_PRESTART 26 /* record pre_start_time_ns */
#define QUEUE_FLAG_MQ_DEFAULT ((1 << QUEUE_FLAG_IO_STAT) | \
(1 << QUEUE_FLAG_SAME_COMP))
@@ -632,6 +635,8 @@ bool blk_queue_flag_test_and_set(unsigned int flag, struct request_queue *q);
test_bit(QUEUE_FLAG_SCSI_PASSTHROUGH, &(q)->queue_flags)
#define blk_queue_pci_p2pdma(q) \
test_bit(QUEUE_FLAG_PCI_P2PDMA, &(q)->queue_flags)
+#define blk_queue_rec_prestart(q) \
+ test_bit(QUEUE_FLAG_REC_PRESTART, &(q)->queue_flags)
#define blk_noretry_request(rq) \
((rq)->cmd_flags & (REQ_FAILFAST_DEV|REQ_FAILFAST_TRANSPORT| \
--
2.17.1
^ permalink raw reply related
* [PATCH 08/10] blkcg: implement blk-iocost
From: Tejun Heo @ 2019-07-10 20:51 UTC (permalink / raw)
To: axboe, newella, clm, josef, dennisz, lizefan, hannes
Cc: linux-kernel, kernel-team, linux-block, cgroups, Tejun Heo,
Josef Bacik
In-Reply-To: <20190710205128.1316483-1-tj@kernel.org>
This patchset implements IO cost model based work-conserving
proportional controller.
While io.latency provides the capability to comprehensively prioritize
and protect IOs depending on the cgroups, its protection is binary -
the lowest latency target cgroup which is suffering is protected at
the cost of all others. In many use cases including stacking multiple
workload containers in a single system, it's necessary to distribute
IO capacity with better granularity.
One challenge of controlling IO resources is the lack of trivially
observable cost metric. The most common metrics - bandwidth and iops
- can be off by orders of magnitude depending on the device type and
IO pattern. However, the cost isn't a complete mystery. Given
several key attributes, we can make fairly reliable predictions on how
expensive a given stream of IOs would be, at least compared to other
IO patterns.
The function which determines the cost of a given IO is the IO cost
model for the device. This controller distributes IO capacity based
on the costs estimated by such model. The more accurate the cost
model the better but the controller adapts based on IO completion
latency and as long as the relative costs across differents IO
patterns are consistent and sensible, it'll adapt to the actual
performance of the device.
Currently, the only implemented cost model is a simple linear one with
a few sets of default parameters for different classes of device.
This covers most common devices reasonably well. All the
infrastructure to tune and add different cost models is already in
place and a later patch will also allow using bpf progs for cost
models.
Please see the top comment in blk-iocost.c and documentation for
more details.
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Andy Newell <newella@fb.com>
Cc: Josef Bacik <jbacik@fb.com>
---
Documentation/admin-guide/cgroup-v2.rst | 94 +
block/Kconfig | 9 +
block/Makefile | 1 +
block/blk-iocost.c | 2370 +++++++++++++++++++++++
block/blk-rq-qos.h | 3 +
include/linux/blk_types.h | 3 +
include/trace/events/iocost.h | 174 ++
7 files changed, 2654 insertions(+)
create mode 100644 block/blk-iocost.c
create mode 100644 include/trace/events/iocost.h
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 6223f485f7e1..aff812631c40 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1425,6 +1425,100 @@ IO Interface Files
8:16 rbytes=1459200 wbytes=314773504 rios=192 wios=353 dbytes=0 dios=0
8:0 rbytes=90430464 wbytes=299008000 rios=8950 wios=1252 dbytes=50331648 dios=3021
+ io.cost.qos
+ A read-write nested-keyed file with exists only on the root
+ cgroup.
+
+ This file configures the Quality of Service of the IO cost
+ model based controller (CONFIG_BLK_CGROUP_IOCOST) which
+ currently implements "io.weight" proportional control. Lines
+ are keyed by $MAJ:$MIN device numbers and not ordered. The
+ line for a given device is populated on the first write for
+ the device on "io.cost.qos" or "io.cost.model". The following
+ nested keys are defined.
+
+ ====== =====================================
+ enable Weight-based control enable
+ ctrl "auto" or "user"
+ rpct Read latency percentile [0, 100]
+ rlat Read latency threshold
+ wpct Write latency percentile [0, 100]
+ wlat Write latency threshold
+ min Minimum scaling percentage [1, 10000]
+ max Maximum scaling percentage [1, 10000]
+ ====== =====================================
+
+ The controller is disabled by default and can be enabled by
+ setting "enable" to 1. "rpct" and "wpct" parameters default
+ to zero and the controller uses internal device saturation
+ state to adjust the overall IO rate between "min" and "max".
+
+ When a better control quality is needed, latency QoS
+ parameters can be configured. For example::
+
+ 8:16 enable=1 ctrl=auto rpct=95.00 rlat=75000 wpct=95.00 wlat=150000 min=50.00 max=150.0
+
+ shows that on sdb, the controller is enabled, will consider
+ the device saturated if the 95th percentile of read completion
+ latencies is above 75ms or write 150ms, and adjust the overall
+ IO issue rate between 50% and 150% accordingly.
+
+ The lower the saturation point, the better the latency QoS at
+ the cost of aggregate bandwidth. The narrower the allowed
+ adjustment range between "min" and "max", the more conformant
+ to the cost model the IO behavior. Note that the IO issue
+ base rate may be far off from 100% and setting "min" and "max"
+ blindly can lead to a significant loss of device capacity or
+ control quality. "min" and "max" are useful for regulating
+ devices which show wide temporary behavior changes - e.g. a
+ ssd which accepts writes at the line speed for a while and
+ then completely stalls for multiple seconds.
+
+ When "ctrl" is "auto", the parameters are controlled by the
+ kernel and may change automatically. Setting "ctrl" to "user"
+ or setting any of the percentile and latency parameters puts
+ it into "user" mode and disables the automatic changes. The
+ automatic mode can be restored by setting "ctrl" to "auto".
+
+ io.cost.model
+ A read-write nested-keyed file with exists only on the root
+ cgroup.
+
+ This file configures the cost model of the IO cost model based
+ controller (CONFIG_BLK_CGROUP_IOCOST) which currently
+ implements "io.weight" proportional control. Lines are keyed
+ by $MAJ:$MIN device numbers and not ordered. The line for a
+ given device is populated on the first write for the device on
+ "io.cost.qos" or "io.cost.model". The following nested keys
+ are defined.
+
+ ===== ================================
+ ctrl "auto" or "user"
+ model The cost model in use - "linear"
+ ===== ================================
+
+ When "ctrl" is "auto", the kernel may change all parameters
+ dynamically. When "ctrl" is set to "user" or any other
+ parameters are written to, "ctrl" become "user" and the
+ automatic changes are disabled.
+
+ When "model" is "linear", the following model parameters are
+ defined.
+
+ ============= ========================================
+ [r|w]bps The maximum sequential IO throughput
+ [r|w]seqiops The maximum 4k sequential IOs per second
+ [r|w]randiops The maximum 4k random IOs per second
+ ============= ========================================
+
+ From the above, the builtin linear model determines the base
+ costs of a sequential and random IO and the cost coefficient
+ for the IO size. While simple, this model can cover most
+ common device classes acceptably.
+
+ The IO cost model isn't expected to be accurate in absolute
+ sense and is scaled to the device behavior dynamically.
+
io.weight
A read-write flat-keyed file which exists on non-root cgroups.
The default is "default 100".
diff --git a/block/Kconfig b/block/Kconfig
index 56cb1695cd87..be1b82d94425 100644
--- a/block/Kconfig
+++ b/block/Kconfig
@@ -132,6 +132,15 @@ config BLK_CGROUP_IOLATENCY
Note, this is an experimental interface and could be changed someday.
+config BLK_CGROUP_IOCOST
+ bool "Enable support for cost model based cgroup IO controller"
+ depends on BLK_CGROUP=y
+ ---help---
+ Enabling this option enables the .weight interface for cost
+ model based proportional IO control. The IO controller
+ distributes IO capacity between different groups based on
+ their share of the overall weight distribution.
+
config BLK_WBT_MQ
bool "Multiqueue writeback throttling"
default y
diff --git a/block/Makefile b/block/Makefile
index eee1b4ceecf9..9ef57ace90d4 100644
--- a/block/Makefile
+++ b/block/Makefile
@@ -18,6 +18,7 @@ obj-$(CONFIG_BLK_DEV_BSGLIB) += bsg-lib.o
obj-$(CONFIG_BLK_CGROUP) += blk-cgroup.o
obj-$(CONFIG_BLK_DEV_THROTTLING) += blk-throttle.o
obj-$(CONFIG_BLK_CGROUP_IOLATENCY) += blk-iolatency.o
+obj-$(CONFIG_BLK_CGROUP_IOCOST) += blk-iocost.o
obj-$(CONFIG_MQ_IOSCHED_DEADLINE) += mq-deadline.o
obj-$(CONFIG_MQ_IOSCHED_KYBER) += kyber-iosched.o
bfq-y := bfq-iosched.o bfq-wf2q.o bfq-cgroup.o
diff --git a/block/blk-iocost.c b/block/blk-iocost.c
new file mode 100644
index 000000000000..1f697e5729f6
--- /dev/null
+++ b/block/blk-iocost.c
@@ -0,0 +1,2370 @@
+/* SPDX-License-Identifier: GPL-2.0
+ *
+ * IO cost model based controller.
+ *
+ * Copyright (C) 2019 Tejun Heo <tj@kernel.org>
+ * Copyright (C) 2019 Andy Newell <newella@fb.com>
+ * Copyright (C) 2019 Facebook
+ *
+ * One challenge of controlling IO resources is the lack of trivially
+ * observable cost metric. This is distinguished from CPU and memory where
+ * wallclock time and the number of bytes can serve as accurate enough
+ * approximations.
+ *
+ * Bandwidth and iops are the most commonly used metrics for IO devices but
+ * depending on the type and specifics of the device, different IO patterns
+ * easily lead to multiple orders of magnitude variations rendering them
+ * useless for the purpose of IO capacity distribution. While on-device
+ * time, with a lot of clutches, could serve as a useful approximation for
+ * non-queued rotational devices, this is no longer viable with modern
+ * devices, even the rotational ones.
+ *
+ * While there is no cost metric we can trivially observe, it isn't a
+ * complete mystery. For example, on a rotational device, seek cost
+ * dominates while a contiguous transfer contributes a smaller amount
+ * proportional to the size. If we can characterize at least the relative
+ * costs of these different types of IOs, it should be possible to
+ * implement a reasonable work-conserving proportional IO resource
+ * distribution.
+ *
+ * 1. IO Cost Model
+ *
+ * IO cost model estimates the cost of an IO given its basic parameters and
+ * history (e.g. the end sector of the last IO). The cost is measured in
+ * device time. If a given IO is estimated to cost 10ms, the device should
+ * be able to process ~100 of those IOs in a second.
+ *
+ * Currently, there's only one builtin cost model - linear. Each IO is
+ * classified as sequential or random and given a base cost accordingly.
+ * On top of that, a size cost proportional to the length of the IO is
+ * added. While simple, this model captures the operational
+ * characteristics of a wide varienty of devices well enough. Default
+ * paramters for several different classes of devices are provided and the
+ * parameters can be configured from userspace via
+ * /sys/fs/cgroup/io.cost.model.
+ *
+ * If needed, tools/cgroup/iocost_coef_gen.py can be used to generate
+ * device-specific coefficients.
+ *
+ * 2. Control Strategy
+ *
+ * The device virtual time (vtime) is used as the primary control metric.
+ * The control strategy is composed of the following three parts.
+ *
+ * 2-1. Vtime Distribution
+ *
+ * When a cgroup becomes active in terms of IOs, its hierarchical share is
+ * calculated. Please consider the following hierarchy where the numbers
+ * inside parentheses denote the configured weights.
+ *
+ * root
+ * / \
+ * A (w:100) B (w:300)
+ * / \
+ * A0 (w:100) A1 (w:100)
+ *
+ * If B is idle and only A0 and A1 are actively issuing IOs, as the two are
+ * of equal weight, each gets 50% share. If then B starts issuing IOs, B
+ * gets 300/(100+300) or 75% share, and A0 and A1 equally splits the rest,
+ * 12.5% each. The distribution mechanism only cares about these flattened
+ * shares. They're called hweights (hierarchical weights) and always add
+ * upto 1 (HWEIGHT_WHOLE).
+ *
+ * A given cgroup's vtime runs slower in inverse proportion to its hweight.
+ * For example, with 12.5% weight, A0's time runs 8 times slower (100/12.5)
+ * against the device vtime - an IO which takes 10ms on the underlying
+ * device is considered to take 80ms on A0.
+ *
+ * This constitutes the basis of IO capacity distribution. Each cgroup's
+ * vtime is running at a rate determined by its hweight. A cgroup tracks
+ * the vtime consumed by past IOs and can issue a new IO iff doing so
+ * wouldn't outrun the current device vtime. Otherwise, the IO is
+ * suspended until the vtime has progressed enough to cover it.
+ *
+ * 2-2. Vrate Adjustment
+ *
+ * It's unrealistic to expect the cost model to be perfect. There are too
+ * many devices and even on the same device the overall performance
+ * fluctuates depending on numerous factors such as IO mixture and device
+ * internal garbage collection. The controller needs to adapt dynamically.
+ *
+ * This is achieved by adjusting the overall IO rate according to how busy
+ * the device is. If the device becomes overloaded, we're sending down too
+ * many IOs and should generally slow down. If there are waiting issuers
+ * but the device isn't saturated, we're issuing too few and should
+ * generally speed up.
+ *
+ * To slow down, we lower the vrate - the rate at which the device vtime
+ * passes compared to the wall clock. For example, if the vtime is running
+ * at the vrate of 75%, all cgroups added up would only be able to issue
+ * 750ms worth of IOs per second, and vice-versa for speeding up.
+ *
+ * Device business is determined using two criteria - rq wait and
+ * completion latencies.
+ *
+ * When a device gets saturated, the on-device and then the request queues
+ * fill up and a bio which is ready to be issued has to wait for a request
+ * to become available. When this delay becomes noticeable, it's a clear
+ * indication that the device is saturated and we lower the vrate. This
+ * saturation signal is fairly conservative as it only triggers when both
+ * hardware and software queues are filled up, and is used as the default
+ * busy signal.
+ *
+ * As devices can have deep queues and be unfair in how the queued commands
+ * are executed, soley depending on rq wait may not result in satisfactory
+ * control quality. For a better control quality, completion latency QoS
+ * parameters can be configured so that the device is considered saturated
+ * if N'th percentile completion latency rises above the set point.
+ *
+ * The completion latency requirements are a function of both the
+ * underlying device characteristics and the desired IO latency quality of
+ * service. There is an inherent trade-off - the tighter the latency QoS,
+ * the higher the bandwidth lossage. Latency QoS is disabled by default
+ * and can be set through /sys/fs/cgroup/io.cost.qos.
+ *
+ * 2-3. Work Conservation
+ *
+ * Imagine two cgroups A and B with equal weights. A is issuing a small IO
+ * periodically while B is sending out enough parallel IOs to saturate the
+ * device on its own. Let's say A's usage amounts to 100ms worth of IO
+ * cost per second, i.e., 10% of the device capacity. The naive
+ * distribution of half and half would lead to 60% utilization of the
+ * device, a significant reduction in the total amount of work done
+ * compared to free-for-all competition. This is too high a cost to pay
+ * for IO control.
+ *
+ * To conserve the total amount of work done, we keep track of how much
+ * each active cgroup is actually using and yield part of its weight if
+ * there are other cgroups which can make use of it. In the above case,
+ * A's weight will be lowered so that it hovers above the actual usage and
+ * B would be able to use the rest.
+ *
+ * As we don't want to penalize a cgroup for donating its weight, the
+ * surplus weight adjustment factors in a margin and has an immediate
+ * snapback mechanism in case the cgroup needs more IO vtime for itself.
+ *
+ * Note that adjusting down surplus weights has the same effects as
+ * accelerating vtime for other cgroups and work conservation can also be
+ * implemented by adjusting vrate dynamically. However, squaring who can
+ * donate and should take back how much requires hweight propagations
+ * anyway making it easier to implement and understand as a separate
+ * mechanism.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/timer.h>
+#include <linux/time64.h>
+#include <linux/parser.h>
+#include <linux/sched/signal.h>
+#include <linux/blk-cgroup.h>
+#include "blk-rq-qos.h"
+#include "blk-stat.h"
+#include "blk-wbt.h"
+
+#ifdef CONFIG_TRACEPOINTS
+
+/* copied from TRACE_CGROUP_PATH, see cgroup-internal.h */
+#define TRACE_IOCG_PATH_LEN 1024
+static DEFINE_SPINLOCK(trace_iocg_path_lock);
+static char trace_iocg_path[TRACE_IOCG_PATH_LEN];
+
+#define TRACE_IOCG_PATH(type, iocg, ...) \
+ do { \
+ unsigned long flags; \
+ if (trace_iocost_##type##_enabled()) { \
+ spin_lock_irqsave(&trace_iocg_path_lock, flags); \
+ cgroup_path(iocg_to_blkg(iocg)->blkcg->css.cgroup, \
+ trace_iocg_path, TRACE_IOCG_PATH_LEN); \
+ trace_iocost_##type(iocg, trace_iocg_path, \
+ ##__VA_ARGS__); \
+ spin_unlock_irqrestore(&trace_iocg_path_lock, flags); \
+ } \
+ } while (0)
+
+#else /* CONFIG_TRACE_POINTS */
+#define TRACE_IOCG_PATH(type, iocg, ...) do { } while (0)
+#endif /* CONFIG_TRACE_POINTS */
+
+enum {
+ MILLION = 1000000,
+
+ /* timer period is calculated from latency requirements, bound it */
+ MIN_PERIOD = USEC_PER_MSEC,
+ MAX_PERIOD = USEC_PER_SEC,
+
+ /*
+ * A cgroup's vtime can run 50% behind the device vtime, which
+ * serves as its IO credit buffer. Surplus weight adjustment is
+ * immediately canceled if the vtime margin runs below 10%.
+ */
+ MARGIN_PCT = 50,
+ INUSE_MARGIN_PCT = 10,
+
+ /* Have some play in waitq timer operations */
+ WAITQ_TIMER_MARGIN_PCT = 5,
+
+ /*
+ * vtime can wrap well within a reasonable uptime when vrate is
+ * consistently raised. Don't trust recorded cgroup vtime if the
+ * period counter indicates that it's older than 5mins.
+ */
+ VTIME_VALID_DUR = 300 * USEC_PER_SEC,
+
+ /*
+ * Remember the past three non-zero usages and use the max for
+ * surplus calculation. Three slots guarantee that we remember one
+ * full period usage from the last active stretch even after
+ * partial deactivation and re-activation periods. Don't start
+ * giving away weight before collecting two data points to prevent
+ * hweight adjustments based on one partial activation period.
+ */
+ NR_USAGE_SLOTS = 3,
+ MIN_VALID_USAGES = 2,
+
+ /* 1/64k is granular enough and can easily be handled w/ u32 */
+ HWEIGHT_WHOLE = 1 << 16,
+
+ /*
+ * As vtime is used to calculate the cost of each IO, it needs to
+ * be fairly high precision. For example, it should be able to
+ * represent the cost of a single page worth of discard with
+ * suffificient accuracy. At the same time, it should be able to
+ * represent reasonably long enough durations to be useful and
+ * convenient during operation.
+ *
+ * 1s worth of vtime is 2^37. This gives us both sub-nanosecond
+ * granularity and days of wrap-around time even at extreme vrates.
+ */
+ VTIME_PER_SEC_SHIFT = 37,
+ VTIME_PER_SEC = 1LLU << VTIME_PER_SEC_SHIFT,
+ VTIME_PER_USEC = VTIME_PER_SEC / USEC_PER_SEC,
+
+ /* bound vrate adjustments within two orders of magnitude */
+ VRATE_MIN_PPM = 10000, /* 1% */
+ VRATE_MAX_PPM = 100000000, /* 10000% */
+
+ VRATE_MIN = VTIME_PER_USEC * VRATE_MIN_PPM / MILLION,
+ VRATE_CLAMP_ADJ_PCT = 4,
+
+ /* if IOs end up waiting for requests, issue less */
+ RQ_WAIT_BUSY_PCT = 5,
+
+ /* unbusy hysterisis */
+ UNBUSY_THR_PCT = 75,
+
+ /* don't let cmds which take a very long time pin lagging for too long */
+ MAX_LAGGING_PERIODS = 10,
+
+ /*
+ * If usage% * 1.25 + 2% is lower than hweight% by more than 3%,
+ * donate the surplus.
+ */
+ SURPLUS_SCALE_PCT = 125, /* * 125% */
+ SURPLUS_SCALE_ABS = HWEIGHT_WHOLE / 50, /* + 2% */
+ SURPLUS_MIN_ADJ_DELTA = HWEIGHT_WHOLE / 33, /* 3% */
+
+ /* switch iff the conditions are met for longer than this */
+ AUTOP_CYCLE_NSEC = 10LLU * NSEC_PER_SEC,
+
+ /*
+ * Count IO size in 4k pages. The 12bit shift helps keeping
+ * size-proportional components of cost calculation in closer
+ * numbers of digits to per-IO cost components.
+ */
+ IOC_PAGE_SHIFT = 12,
+ IOC_PAGE_SIZE = 1 << IOC_PAGE_SHIFT,
+ IOC_SECT_TO_PAGE_SHIFT = IOC_PAGE_SHIFT - SECTOR_SHIFT,
+
+ /* if apart further than 16M, consider randio for linear model */
+ LCOEF_RANDIO_PAGES = 4096,
+};
+
+enum ioc_running {
+ IOC_IDLE,
+ IOC_RUNNING,
+ IOC_STOP,
+};
+
+/* io.cost.qos controls including per-dev enable of the whole controller */
+enum {
+ QOS_ENABLE,
+ QOS_CTRL,
+ NR_QOS_CTRL_PARAMS,
+};
+
+/* io.cost.qos params */
+enum {
+ QOS_RPPM,
+ QOS_RLAT,
+ QOS_WPPM,
+ QOS_WLAT,
+ QOS_MIN,
+ QOS_MAX,
+ NR_QOS_PARAMS,
+};
+
+/* io.cost.model controls */
+enum {
+ COST_CTRL,
+ COST_MODEL,
+ NR_COST_CTRL_PARAMS,
+};
+
+/* builtin linear cost model coefficients */
+enum {
+ I_LCOEF_RBPS,
+ I_LCOEF_RSEQIOPS,
+ I_LCOEF_RRANDIOPS,
+ I_LCOEF_WBPS,
+ I_LCOEF_WSEQIOPS,
+ I_LCOEF_WRANDIOPS,
+ NR_I_LCOEFS,
+};
+
+enum {
+ LCOEF_RPAGE,
+ LCOEF_RSEQIO,
+ LCOEF_RRANDIO,
+ LCOEF_WPAGE,
+ LCOEF_WSEQIO,
+ LCOEF_WRANDIO,
+ NR_LCOEFS,
+};
+
+enum {
+ AUTOP_INVALID,
+ AUTOP_HDD,
+ AUTOP_SSD_QD1,
+ AUTOP_SSD_DFL,
+ AUTOP_SSD_FAST,
+};
+
+struct ioc_gq;
+
+struct ioc_params {
+ u32 qos[NR_QOS_PARAMS];
+ u64 i_lcoefs[NR_I_LCOEFS];
+ u64 lcoefs[NR_LCOEFS];
+ u32 too_fast_vrate_pct;
+ u32 too_slow_vrate_pct;
+};
+
+struct ioc_missed {
+ u32 nr_met;
+ u32 nr_missed;
+ u32 last_met;
+ u32 last_missed;
+};
+
+struct ioc_pcpu_stat {
+ struct ioc_missed missed[2];
+
+ u64 rq_wait_ns;
+ u64 last_rq_wait_ns;
+};
+
+/* per device */
+struct ioc {
+ struct rq_qos rqos;
+
+ bool enabled;
+
+ struct ioc_params params;
+ u32 period_us;
+ u32 margin_us;
+ u64 vrate_min;
+ u64 vrate_max;
+
+ spinlock_t lock;
+ struct timer_list timer;
+ struct list_head active_iocgs; /* active cgroups */
+ struct ioc_pcpu_stat __percpu *pcpu_stat;
+
+ enum ioc_running running;
+ atomic64_t vtime_rate;
+
+ seqcount_t period_seqcount;
+ u32 period_at; /* wallclock starttime */
+ u64 period_at_vtime; /* vtime starttime */
+
+ atomic64_t cur_period; /* inc'd each period */
+ int busy_level; /* saturation history */
+
+ u64 inuse_margin_vtime;
+ bool weights_updated;
+ atomic_t hweight_gen; /* for lazy hweights */
+
+ u64 autop_too_fast_at;
+ u64 autop_too_slow_at;
+ int autop_idx;
+ bool user_qos_params:1;
+ bool user_cost_model:1;
+};
+
+/* per device-cgroup pair */
+struct ioc_gq {
+ struct blkg_policy_data pd;
+ struct ioc *ioc;
+
+ /*
+ * A iocg can get its weight from two sources - an explicit
+ * per-device-cgroup configuration or the default weight of the
+ * cgroup. `cfg_weight` is the explicit per-device-cgroup
+ * configuration. `weight` is the effective considering both
+ * sources.
+ *
+ * When an idle cgroup becomes active its `active` goes from 0 to
+ * `weight`. `inuse` is the surplus adjusted active weight.
+ * `active` and `inuse` are used to calculate `hweight_active` and
+ * `hweight_inuse`.
+ *
+ * `last_inuse` remembers `inuse` while an iocg is idle to persist
+ * surplus adjustments.
+ */
+ u32 cfg_weight;
+ u32 weight;
+ u32 active;
+ u32 inuse;
+ u32 last_inuse;
+
+ sector_t cursor; /* to detect randio */
+
+ /*
+ * `vtime` is this iocg's vtime cursor which progresses as IOs are
+ * issued. If lagging behind device vtime, the delta represents
+ * the currently available IO budget. If runnning ahead, the
+ * overage.
+ *
+ * `vtime_done` is the same but progressed on completion rather
+ * than issue. The delta behind `vtime` represents the cost of
+ * currently in-flight IOs.
+ *
+ * `last_vtime` is used to remember `vtime` at the end of the last
+ * period to calculate utilization.
+ */
+ atomic64_t vtime;
+ atomic64_t done_vtime;
+ u64 last_vtime;
+
+ /*
+ * The period this iocg was last active in. Used for deactivation
+ * and invalidating `vtime`.
+ */
+ atomic64_t active_period;
+ struct list_head active_list;
+
+ /* see __propagate_active_weight() and current_hweight() for details */
+ u64 child_active_sum;
+ u64 child_inuse_sum;
+ int hweight_gen;
+ u32 hweight_active;
+ u32 hweight_inuse;
+ bool has_surplus;
+
+ struct wait_queue_head waitq;
+ struct hrtimer waitq_timer;
+ struct hrtimer delay_timer;
+
+ /* usage is recorded as fractions of HWEIGHT_WHOLE */
+ int usage_idx;
+ u32 usages[NR_USAGE_SLOTS];
+
+ /* this iocg's depth in the hierarchy and ancestors including self */
+ int level;
+ struct ioc_gq *ancestors[];
+};
+
+/* per cgroup */
+struct ioc_cgrp {
+ struct blkcg_policy_data cpd;
+ unsigned int dfl_weight;
+};
+
+struct ioc_now {
+ u64 now_ns;
+ u32 now;
+ u64 vnow;
+ u64 vrate;
+};
+
+struct iocg_wait {
+ struct wait_queue_entry wait;
+ struct bio *bio;
+ u64 abs_cost;
+ bool committed;
+};
+
+struct iocg_wake_ctx {
+ struct ioc_gq *iocg;
+ u32 hw_inuse;
+ s64 vbudget;
+};
+
+static const struct ioc_params autop[] = {
+ [AUTOP_HDD] = {
+ .qos = {
+ [QOS_RLAT] = 50000, /* 50ms */
+ [QOS_WLAT] = 50000,
+ [QOS_MIN] = VRATE_MIN_PPM,
+ [QOS_MAX] = VRATE_MAX_PPM,
+ },
+ .i_lcoefs = {
+ [I_LCOEF_RBPS] = 174019176,
+ [I_LCOEF_RSEQIOPS] = 41708,
+ [I_LCOEF_RRANDIOPS] = 370,
+ [I_LCOEF_WBPS] = 178075866,
+ [I_LCOEF_WSEQIOPS] = 42705,
+ [I_LCOEF_WRANDIOPS] = 378,
+ },
+ },
+ [AUTOP_SSD_QD1] = {
+ .qos = {
+ [QOS_RLAT] = 25000, /* 25ms */
+ [QOS_WLAT] = 25000,
+ [QOS_MIN] = VRATE_MIN_PPM,
+ [QOS_MAX] = VRATE_MAX_PPM,
+ },
+ .i_lcoefs = {
+ [I_LCOEF_RBPS] = 245855193,
+ [I_LCOEF_RSEQIOPS] = 61575,
+ [I_LCOEF_RRANDIOPS] = 6946,
+ [I_LCOEF_WBPS] = 141365009,
+ [I_LCOEF_WSEQIOPS] = 33716,
+ [I_LCOEF_WRANDIOPS] = 26796,
+ },
+ },
+ [AUTOP_SSD_DFL] = {
+ .qos = {
+ [QOS_RLAT] = 25000, /* 25ms */
+ [QOS_WLAT] = 25000,
+ [QOS_MIN] = VRATE_MIN_PPM,
+ [QOS_MAX] = VRATE_MAX_PPM,
+ },
+ .i_lcoefs = {
+ [I_LCOEF_RBPS] = 488636629,
+ [I_LCOEF_RSEQIOPS] = 8932,
+ [I_LCOEF_RRANDIOPS] = 8518,
+ [I_LCOEF_WBPS] = 427891549,
+ [I_LCOEF_WSEQIOPS] = 28755,
+ [I_LCOEF_WRANDIOPS] = 21940,
+ },
+ .too_fast_vrate_pct = 500,
+ },
+ [AUTOP_SSD_FAST] = {
+ .qos = {
+ [QOS_RLAT] = 5000, /* 5ms */
+ [QOS_WLAT] = 5000,
+ [QOS_MIN] = VRATE_MIN_PPM,
+ [QOS_MAX] = VRATE_MAX_PPM,
+ },
+ .i_lcoefs = {
+ [I_LCOEF_RBPS] = 3102524156LLU,
+ [I_LCOEF_RSEQIOPS] = 724816,
+ [I_LCOEF_RRANDIOPS] = 778122,
+ [I_LCOEF_WBPS] = 1742780862LLU,
+ [I_LCOEF_WSEQIOPS] = 425702,
+ [I_LCOEF_WRANDIOPS] = 443193,
+ },
+ .too_slow_vrate_pct = 10,
+ },
+};
+
+/*
+ * vrate adjust percentages indexed by ioc->busy_level. We adjust up on
+ * vtime credit shortage and down on device saturation.
+ */
+static u32 vrate_adj_pct[] =
+ { 0, 0, 0, 0,
+ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
+ 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
+ 4, 4, 4, 4, 4, 4, 4, 4, 8, 8, 8, 8, 8, 8, 8, 8, 16 };
+
+static struct blkcg_policy blkcg_policy_iocost;
+
+/* accessors and helpers */
+static struct ioc *rqos_to_ioc(struct rq_qos *rqos)
+{
+ return container_of(rqos, struct ioc, rqos);
+}
+
+static struct ioc *q_to_ioc(struct request_queue *q)
+{
+ return rqos_to_ioc(rq_qos_id(q, RQ_QOS_COST));
+}
+
+static const char *q_name(struct request_queue *q)
+{
+ if (test_bit(QUEUE_FLAG_REGISTERED, &q->queue_flags))
+ return kobject_name(q->kobj.parent);
+ else
+ return "<unknown>";
+}
+
+static const char __maybe_unused *ioc_name(struct ioc *ioc)
+{
+ return q_name(ioc->rqos.q);
+}
+
+static struct ioc_gq *pd_to_iocg(struct blkg_policy_data *pd)
+{
+ return pd ? container_of(pd, struct ioc_gq, pd) : NULL;
+}
+
+static struct ioc_gq *blkg_to_iocg(struct blkcg_gq *blkg)
+{
+ return pd_to_iocg(blkg_to_pd(blkg, &blkcg_policy_iocost));
+}
+
+static struct blkcg_gq *iocg_to_blkg(struct ioc_gq *iocg)
+{
+ return pd_to_blkg(&iocg->pd);
+}
+
+static struct ioc_cgrp *blkcg_to_iocc(struct blkcg *blkcg)
+{
+ return container_of(blkcg_to_cpd(blkcg, &blkcg_policy_iocost),
+ struct ioc_cgrp, cpd);
+}
+
+/*
+ * Scale @abs_cost to the inverse of @hw_inuse. The lower the hierarchical
+ * weight, the more expensive each IO.
+ */
+static u64 abs_cost_to_cost(u64 abs_cost, u32 hw_inuse)
+{
+ return DIV64_U64_ROUND_UP(abs_cost * HWEIGHT_WHOLE, hw_inuse);
+}
+
+static void iocg_commit_bio(struct ioc_gq *iocg, struct bio *bio, u64 cost)
+{
+ bio->bi_iocost_cost = cost;
+ atomic64_add(cost, &iocg->vtime);
+}
+
+#define CREATE_TRACE_POINTS
+#include <trace/events/iocost.h>
+
+/* latency Qos params changed, update period_us and all the dependent params */
+static void ioc_refresh_period_us(struct ioc *ioc)
+{
+ u32 ppm, lat, multi, period_us;
+
+ lockdep_assert_held(&ioc->lock);
+
+ /* pick the higher latency target */
+ if (ioc->params.qos[QOS_RLAT] >= ioc->params.qos[QOS_WLAT]) {
+ ppm = ioc->params.qos[QOS_RPPM];
+ lat = ioc->params.qos[QOS_RLAT];
+ } else {
+ ppm = ioc->params.qos[QOS_WPPM];
+ lat = ioc->params.qos[QOS_WLAT];
+ }
+
+ /*
+ * We want the period to be long enough to contain a healthy number
+ * of IOs while short enough for granular control. Define it as a
+ * multiple of the latency target. Ideally, the multiplier should
+ * be scaled according to the percentile so that it would nominally
+ * contain a certain number of requests. Let's be simpler and
+ * scale it linearly so that it's 2x >= pct(90) and 10x at pct(50).
+ */
+ if (ppm)
+ multi = max_t(u32, (MILLION - ppm) / 50000, 2);
+ else
+ multi = 2;
+ period_us = multi * lat;
+ period_us = clamp_t(u32, period_us, MIN_PERIOD, MAX_PERIOD);
+
+ /* calculate dependent params */
+ ioc->period_us = period_us;
+ ioc->margin_us = period_us * MARGIN_PCT / 100;
+ ioc->inuse_margin_vtime = DIV64_U64_ROUND_UP(
+ period_us * VTIME_PER_USEC * INUSE_MARGIN_PCT, 100);
+}
+
+static int ioc_autop_idx(struct ioc *ioc)
+{
+ int idx = ioc->autop_idx;
+ const struct ioc_params *p = &autop[idx];
+ u32 vrate_pct;
+ u64 now_ns;
+
+ /* rotational? */
+ if (!blk_queue_nonrot(ioc->rqos.q))
+ return AUTOP_HDD;
+
+ /* handle SATA SSDs w/ broken NCQ */
+ if (blk_queue_depth(ioc->rqos.q) == 1)
+ return AUTOP_SSD_QD1;
+
+ /* use one of the normal ssd sets */
+ if (idx < AUTOP_SSD_DFL)
+ return AUTOP_SSD_DFL;
+
+ /* if user is overriding anything, maintain what was there */
+ if (ioc->user_qos_params || ioc->user_cost_model)
+ return idx;
+
+ /* step up/down based on the vrate */
+ vrate_pct = div64_u64(atomic64_read(&ioc->vtime_rate) * 100,
+ VTIME_PER_USEC);
+ now_ns = ktime_get_ns();
+
+ if (p->too_fast_vrate_pct && p->too_fast_vrate_pct <= vrate_pct) {
+ if (!ioc->autop_too_fast_at)
+ ioc->autop_too_fast_at = now_ns;
+ if (now_ns - ioc->autop_too_fast_at >= AUTOP_CYCLE_NSEC)
+ return idx + 1;
+ } else {
+ ioc->autop_too_fast_at = 0;
+ }
+
+ if (p->too_slow_vrate_pct && p->too_slow_vrate_pct >= vrate_pct) {
+ if (!ioc->autop_too_slow_at)
+ ioc->autop_too_slow_at = now_ns;
+ if (now_ns - ioc->autop_too_slow_at >= AUTOP_CYCLE_NSEC)
+ return idx - 1;
+ } else {
+ ioc->autop_too_slow_at = 0;
+ }
+
+ return idx;
+}
+
+/*
+ * Take the followings as input
+ *
+ * @bps maximum sequential throughput
+ * @seqiops maximum sequential 4k iops
+ * @randiops maximum random 4k iops
+ *
+ * and calculate the linear model cost coefficients.
+ *
+ * *@page per-page cost 1s / (@bps / 4096)
+ * *@seqio base cost of a seq IO max((1s / @seqiops) - *@page, 0)
+ * @randiops base cost of a rand IO max((1s / @randiops) - *@page, 0)
+ */
+static void calc_lcoefs(u64 bps, u64 seqiops, u64 randiops,
+ u64 *page, u64 *seqio, u64 *randio)
+{
+ u64 v;
+
+ *page = *seqio = *randio = 0;
+
+ if (bps)
+ *page = DIV64_U64_ROUND_UP(VTIME_PER_SEC,
+ DIV_ROUND_UP_ULL(bps, IOC_PAGE_SIZE));
+
+ if (seqiops) {
+ v = DIV64_U64_ROUND_UP(VTIME_PER_SEC, seqiops);
+ if (v > *page)
+ *seqio = v - *page;
+ }
+
+ if (randiops) {
+ v = DIV64_U64_ROUND_UP(VTIME_PER_SEC, randiops);
+ if (v > *page)
+ *randio = v - *page;
+ }
+}
+
+static void ioc_refresh_lcoefs(struct ioc *ioc)
+{
+ u64 *u = ioc->params.i_lcoefs;
+ u64 *c = ioc->params.lcoefs;
+
+ calc_lcoefs(u[I_LCOEF_RBPS], u[I_LCOEF_RSEQIOPS], u[I_LCOEF_RRANDIOPS],
+ &c[LCOEF_RPAGE], &c[LCOEF_RSEQIO], &c[LCOEF_RRANDIO]);
+ calc_lcoefs(u[I_LCOEF_WBPS], u[I_LCOEF_WSEQIOPS], u[I_LCOEF_WRANDIOPS],
+ &c[LCOEF_WPAGE], &c[LCOEF_WSEQIO], &c[LCOEF_WRANDIO]);
+}
+
+static bool ioc_refresh_params(struct ioc *ioc, bool force)
+{
+ const struct ioc_params *p;
+ int idx;
+
+ lockdep_assert_held(&ioc->lock);
+
+ idx = ioc_autop_idx(ioc);
+ p = &autop[idx];
+
+ if (idx == ioc->autop_idx && !force)
+ return false;
+
+ if (idx != ioc->autop_idx)
+ atomic64_set(&ioc->vtime_rate, VTIME_PER_USEC);
+
+ ioc->autop_idx = idx;
+ ioc->autop_too_fast_at = 0;
+ ioc->autop_too_slow_at = 0;
+
+ if (!ioc->user_qos_params)
+ memcpy(ioc->params.qos, p->qos, sizeof(p->qos));
+ if (!ioc->user_cost_model)
+ memcpy(ioc->params.i_lcoefs, p->i_lcoefs, sizeof(p->i_lcoefs));
+
+ ioc_refresh_period_us(ioc);
+ ioc_refresh_lcoefs(ioc);
+
+ ioc->vrate_min = DIV64_U64_ROUND_UP((u64)ioc->params.qos[QOS_MIN] *
+ VTIME_PER_USEC, MILLION);
+ ioc->vrate_max = div64_u64((u64)ioc->params.qos[QOS_MAX] *
+ VTIME_PER_USEC, MILLION);
+
+ return true;
+}
+
+/* take a snapshot of the current [v]time and vrate */
+static void ioc_now(struct ioc *ioc, struct ioc_now *now)
+{
+ unsigned seq;
+
+ now->now_ns = ktime_get();
+ now->now = ktime_to_us(now->now_ns);
+ now->vrate = atomic64_read(&ioc->vtime_rate);
+
+ /*
+ * The current vtime is
+ *
+ * vtime at period start + (wallclock time since the start) * vrate
+ *
+ * As a consistent snapshot of `period_at_vtime` and `period_at` is
+ * needed, they're seqcount protected.
+ */
+ do {
+ seq = read_seqcount_begin(&ioc->period_seqcount);
+ now->vnow = ioc->period_at_vtime +
+ (now->now - ioc->period_at) * now->vrate;
+ } while (read_seqcount_retry(&ioc->period_seqcount, seq));
+}
+
+static void ioc_start_period(struct ioc *ioc, struct ioc_now *now)
+{
+ lockdep_assert_held(&ioc->lock);
+ WARN_ON_ONCE(ioc->running != IOC_RUNNING);
+
+ write_seqcount_begin(&ioc->period_seqcount);
+ ioc->period_at = now->now;
+ ioc->period_at_vtime = now->vnow;
+ write_seqcount_end(&ioc->period_seqcount);
+
+ ioc->timer.expires = jiffies + usecs_to_jiffies(ioc->period_us);
+ add_timer(&ioc->timer);
+}
+
+/*
+ * Update @iocg's `active` and `inuse` to @active and @inuse, update level
+ * weight sums and propagate upwards accordingly.
+ */
+static void __propagate_active_weight(struct ioc_gq *iocg, u32 active, u32 inuse)
+{
+ struct ioc *ioc = iocg->ioc;
+ int lvl;
+
+ lockdep_assert_held(&ioc->lock);
+
+ inuse = min(active, inuse);
+
+ for (lvl = iocg->level - 1; lvl >= 0; lvl--) {
+ struct ioc_gq *parent = iocg->ancestors[lvl];
+ struct ioc_gq *child = iocg->ancestors[lvl + 1];
+ u32 parent_active = 0, parent_inuse = 0;
+
+ /* update the level sums */
+ parent->child_active_sum += (s32)(active - child->active);
+ parent->child_inuse_sum += (s32)(inuse - child->inuse);
+ /* apply the udpates */
+ child->active = active;
+ child->inuse = inuse;
+
+ /*
+ * The delta between inuse and active sums indicates that
+ * that much of weight is being given away. Parent's inuse
+ * and active should reflect the ratio.
+ */
+ if (parent->child_active_sum) {
+ parent_active = parent->weight;
+ parent_inuse = DIV64_U64_ROUND_UP(
+ parent_active * parent->child_inuse_sum,
+ parent->child_active_sum);
+ }
+
+ /* do we need to keep walking up? */
+ if (parent_active == parent->active &&
+ parent_inuse == parent->inuse)
+ break;
+
+ active = parent_active;
+ inuse = parent_inuse;
+ }
+
+ ioc->weights_updated = true;
+}
+
+static void commit_active_weights(struct ioc *ioc)
+{
+ lockdep_assert_held(&ioc->lock);
+
+ if (ioc->weights_updated) {
+ /* paired with rmb in current_hweight(), see there */
+ smp_wmb();
+ atomic_inc(&ioc->hweight_gen);
+ ioc->weights_updated = false;
+ }
+}
+
+static void propagate_active_weight(struct ioc_gq *iocg, u32 active, u32 inuse)
+{
+ __propagate_active_weight(iocg, active, inuse);
+ commit_active_weights(iocg->ioc);
+}
+
+static void current_hweight(struct ioc_gq *iocg, u32 *hw_activep, u32 *hw_inusep)
+{
+ struct ioc *ioc = iocg->ioc;
+ int lvl;
+ u32 hwa, hwi;
+ int ioc_gen;
+
+ /* hot path - if uptodate, use cached */
+ ioc_gen = atomic_read(&ioc->hweight_gen);
+ if (ioc_gen == iocg->hweight_gen)
+ goto out;
+
+ /*
+ * Paired with wmb in commit_active_weights(). If we saw the
+ * updated hweight_gen, all the weight updates from
+ * __propagate_active_weight() are visible too.
+ *
+ * We can race with weight updates during calculation and get it
+ * wrong. However, hweight_gen would have changed and a future
+ * reader will recalculate and we're guaranteed to discard the
+ * wrong result soon.
+ */
+ smp_rmb();
+
+ hwa = hwi = HWEIGHT_WHOLE;
+ for (lvl = 0; lvl <= iocg->level - 1; lvl++) {
+ struct ioc_gq *parent = iocg->ancestors[lvl];
+ struct ioc_gq *child = iocg->ancestors[lvl + 1];
+ u32 active_sum = READ_ONCE(parent->child_active_sum);
+ u32 inuse_sum = READ_ONCE(parent->child_inuse_sum);
+ u32 active = READ_ONCE(child->active);
+ u32 inuse = READ_ONCE(child->inuse);
+
+ if (!active_sum)
+ continue;
+
+ active_sum = max(active, active_sum);
+ hwa = hwa * active / active_sum; /* max 16bits * 10000 */
+
+ inuse_sum = max(inuse, inuse_sum);
+ hwi = hwi * inuse / inuse_sum; /* max 16bits * 10000 */
+ }
+
+ iocg->hweight_active = max_t(u32, hwa, 1);
+ iocg->hweight_inuse = max_t(u32, hwi, 1);
+ iocg->hweight_gen = ioc_gen;
+out:
+ if (hw_activep)
+ *hw_activep = iocg->hweight_active;
+ if (hw_inusep)
+ *hw_inusep = iocg->hweight_inuse;
+}
+
+static void weight_updated(struct ioc_gq *iocg)
+{
+ struct ioc *ioc = iocg->ioc;
+ struct blkcg_gq *blkg = iocg_to_blkg(iocg);
+ struct ioc_cgrp *iocc = blkcg_to_iocc(blkg->blkcg);
+ u32 weight;
+
+ lockdep_assert_held(&ioc->lock);
+
+ weight = iocg->cfg_weight ?: iocc->dfl_weight;
+ if (weight != iocg->weight && iocg->active)
+ propagate_active_weight(iocg, weight,
+ DIV64_U64_ROUND_UP(iocg->inuse * weight, iocg->weight));
+ iocg->weight = weight;
+}
+
+static bool iocg_activate(struct ioc_gq *iocg, struct ioc_now *now)
+{
+ struct ioc *ioc = iocg->ioc;
+ u64 last_period, cur_period, max_period_delta;
+ u64 vtime, vmargin, vmin;
+ int i;
+
+ /*
+ * If seem to be already active, just update the stamp to tell the
+ * timer that we're still active. We don't mind occassional races.
+ */
+ if (!list_empty(&iocg->active_list)) {
+ ioc_now(ioc, now);
+ cur_period = atomic64_read(&ioc->cur_period);
+ if (atomic64_read(&iocg->active_period) != cur_period)
+ atomic64_set(&iocg->active_period, cur_period);
+ return true;
+ }
+
+ /* racy check on internal node IOs, treat as root level IOs */
+ if (iocg->child_active_sum)
+ return false;
+
+ spin_lock_irq(&ioc->lock);
+
+ ioc_now(ioc, now);
+
+ /* update period */
+ cur_period = atomic64_read(&ioc->cur_period);
+ last_period = atomic64_read(&iocg->active_period);
+ atomic64_set(&iocg->active_period, cur_period);
+
+ /* already activated or breaking leaf-only constraint? */
+ for (i = iocg->level; i > 0; i--)
+ if (!list_empty(&iocg->active_list))
+ goto fail_unlock;
+ if (iocg->child_active_sum)
+ goto fail_unlock;
+
+ /*
+ * vtime may wrap when vrate is raised substantially due to
+ * underestimated IO costs. Look at the period and ignore its
+ * vtime if the iocg has been idle for too long. Also, cap the
+ * budget it can start with to the margin.
+ */
+ max_period_delta = DIV64_U64_ROUND_UP(VTIME_VALID_DUR, ioc->period_us);
+ vtime = atomic64_read(&iocg->vtime);
+ vmargin = ioc->margin_us * now->vrate;
+ vmin = now->vnow - vmargin;
+
+ if (last_period + max_period_delta < cur_period ||
+ time_before64(vtime, vmin)) {
+ atomic64_add(vmin - vtime, &iocg->vtime);
+ atomic64_add(vmin - vtime, &iocg->done_vtime);
+ vtime = vmin;
+ }
+
+ /*
+ * Activate, propagate weight and start period timer if not
+ * running. Reset hweight_gen to avoid accidental match from
+ * wrapping.
+ */
+ iocg->hweight_gen = atomic_read(&ioc->hweight_gen) - 1;
+ list_add(&iocg->active_list, &ioc->active_iocgs);
+ propagate_active_weight(iocg, iocg->weight,
+ iocg->last_inuse ?: iocg->weight);
+
+ TRACE_IOCG_PATH(iocg_activate, iocg, now,
+ last_period, cur_period, vtime);
+
+ iocg->last_vtime = vtime;
+
+ if (ioc->running == IOC_IDLE) {
+ ioc->running = IOC_RUNNING;
+ ioc_start_period(ioc, now);
+ }
+
+ spin_unlock_irq(&ioc->lock);
+ return true;
+
+fail_unlock:
+ spin_unlock_irq(&ioc->lock);
+ return false;
+}
+
+static int iocg_wake_fn(struct wait_queue_entry *wq_entry, unsigned mode,
+ int flags, void *key)
+{
+ struct iocg_wait *wait = container_of(wq_entry, struct iocg_wait, wait);
+ struct iocg_wake_ctx *ctx = (struct iocg_wake_ctx *)key;
+ u64 cost = abs_cost_to_cost(wait->abs_cost, ctx->hw_inuse);
+
+ ctx->vbudget -= cost;
+
+ if (ctx->vbudget < 0)
+ return -1;
+
+ iocg_commit_bio(ctx->iocg, wait->bio, cost);
+
+ /*
+ * autoremove_wake_function() removes the wait entry only when it
+ * actually changed the task state. We want the wait always
+ * removed. Remove explicitly and use default_wake_function().
+ */
+ list_del_init(&wq_entry->entry);
+ wait->committed = true;
+
+ default_wake_function(wq_entry, mode, flags, key);
+ return 0;
+}
+
+static void iocg_kick_waitq(struct ioc_gq *iocg, struct ioc_now *now)
+{
+ struct ioc *ioc = iocg->ioc;
+ struct iocg_wake_ctx ctx = { .iocg = iocg };
+ u64 margin_ns = (u64)(ioc->period_us *
+ WAITQ_TIMER_MARGIN_PCT / 100) * NSEC_PER_USEC;
+ u64 vshortage, expires, oexpires;
+
+ lockdep_assert_held(&iocg->waitq.lock);
+
+ /*
+ * Wake up the ones which are due and see how much vtime we'll need
+ * for the next one.
+ */
+ current_hweight(iocg, NULL, &ctx.hw_inuse);
+ ctx.vbudget = now->vnow - atomic64_read(&iocg->vtime);
+ __wake_up_locked_key(&iocg->waitq, TASK_NORMAL, &ctx);
+ if (!waitqueue_active(&iocg->waitq))
+ return;
+ if (WARN_ON_ONCE(ctx.vbudget >= 0))
+ return;
+
+ /* determine next wakeup, add a quarter margin to guarantee chunking */
+ vshortage = -ctx.vbudget;
+ expires = now->now_ns +
+ DIV64_U64_ROUND_UP(vshortage, now->vrate) * NSEC_PER_USEC;
+ expires += margin_ns / 4;
+
+ /* if already active and close enough, don't bother */
+ oexpires = ktime_to_ns(hrtimer_get_softexpires(&iocg->waitq_timer));
+ if (hrtimer_is_queued(&iocg->waitq_timer) &&
+ abs(oexpires - expires) <= margin_ns / 4)
+ return;
+
+ hrtimer_start_range_ns(&iocg->waitq_timer, ns_to_ktime(expires),
+ margin_ns / 4, HRTIMER_MODE_ABS);
+}
+
+static enum hrtimer_restart iocg_waitq_timer_fn(struct hrtimer *timer)
+{
+ struct ioc_gq *iocg = container_of(timer, struct ioc_gq, waitq_timer);
+ struct ioc_now now;
+ unsigned long flags;
+
+ ioc_now(iocg->ioc, &now);
+
+ spin_lock_irqsave(&iocg->waitq.lock, flags);
+ iocg_kick_waitq(iocg, &now);
+ spin_unlock_irqrestore(&iocg->waitq.lock, flags);
+
+ return HRTIMER_NORESTART;
+}
+
+static void iocg_kick_delay(struct ioc_gq *iocg, struct ioc_now *now, u64 cost)
+{
+ struct ioc *ioc = iocg->ioc;
+ struct blkcg_gq *blkg = iocg_to_blkg(iocg);
+ u64 vtime = atomic64_read(&iocg->vtime);
+ u64 vmargin = ioc->margin_us * now->vrate;
+ u64 margin_ns = ioc->margin_us * NSEC_PER_USEC;
+ u64 expires, oexpires;
+
+ /* clear or maintain depending on the overage */
+ if (time_before_eq64(vtime, now->vnow)) {
+ blkcg_clear_delay(blkg);
+ return;
+ }
+ if (!atomic_read(&blkg->use_delay) &&
+ time_before_eq64(vtime, now->vnow + vmargin))
+ return;
+
+ /* use delay */
+ if (cost) {
+ u64 cost_ns = DIV64_U64_ROUND_UP(cost * NSEC_PER_USEC,
+ now->vrate);
+ blkcg_add_delay(blkg, now->now_ns, cost_ns);
+ }
+ blkcg_use_delay(blkg);
+
+ expires = now->now_ns + DIV64_U64_ROUND_UP(vtime - now->vnow,
+ now->vrate) * NSEC_PER_USEC;
+
+ /* if already active and close enough, don't bother */
+ oexpires = ktime_to_ns(hrtimer_get_softexpires(&iocg->delay_timer));
+ if (hrtimer_is_queued(&iocg->delay_timer) &&
+ abs(oexpires - expires) <= margin_ns / 4)
+ return;
+
+ hrtimer_start_range_ns(&iocg->delay_timer, ns_to_ktime(expires),
+ margin_ns / 4, HRTIMER_MODE_ABS);
+}
+
+static enum hrtimer_restart iocg_delay_timer_fn(struct hrtimer *timer)
+{
+ struct ioc_gq *iocg = container_of(timer, struct ioc_gq, delay_timer);
+ struct ioc_now now;
+
+ ioc_now(iocg->ioc, &now);
+ iocg_kick_delay(iocg, &now, 0);
+
+ return HRTIMER_NORESTART;
+}
+
+static void ioc_lat_stat(struct ioc *ioc, u32 *missed_ppm_ar, u32 *rq_wait_pct_p)
+{
+ u32 nr_met[2] = { };
+ u32 nr_missed[2] = { };
+ u64 rq_wait_ns = 0;
+ int cpu, rw;
+
+ for_each_online_cpu(cpu) {
+ struct ioc_pcpu_stat *stat = per_cpu_ptr(ioc->pcpu_stat, cpu);
+ u64 this_rq_wait_ns;
+
+ for (rw = READ; rw <= WRITE; rw++) {
+ u32 this_met = READ_ONCE(stat->missed[rw].nr_met);
+ u32 this_missed = READ_ONCE(stat->missed[rw].nr_missed);
+
+ nr_met[rw] += this_met - stat->missed[rw].last_met;
+ nr_missed[rw] += this_missed - stat->missed[rw].last_missed;
+ stat->missed[rw].last_met = this_met;
+ stat->missed[rw].last_missed = this_missed;
+ }
+
+ this_rq_wait_ns = READ_ONCE(stat->rq_wait_ns);
+ rq_wait_ns += this_rq_wait_ns - stat->last_rq_wait_ns;
+ stat->last_rq_wait_ns = this_rq_wait_ns;
+ }
+
+ for (rw = READ; rw <= WRITE; rw++) {
+ if (nr_met[rw] + nr_missed[rw])
+ missed_ppm_ar[rw] =
+ DIV64_U64_ROUND_UP((u64)nr_missed[rw] * MILLION,
+ nr_met[rw] + nr_missed[rw]);
+ else
+ missed_ppm_ar[rw] = 0;
+ }
+
+ *rq_wait_pct_p = div64_u64(rq_wait_ns * 100,
+ ioc->period_us * NSEC_PER_USEC);
+}
+
+/* was iocg idle this period? */
+static bool iocg_is_idle(struct ioc_gq *iocg)
+{
+ struct ioc *ioc = iocg->ioc;
+
+ /* did something get issued this period? */
+ if (atomic64_read(&iocg->active_period) ==
+ atomic64_read(&ioc->cur_period))
+ return false;
+
+ /* is something in flight? */
+ if (atomic64_read(&iocg->done_vtime) < atomic64_read(&iocg->vtime))
+ return false;
+
+ return true;
+}
+
+/* returns usage with margin added if surplus is large enough */
+static u32 surplus_adjusted_hweight_inuse(u32 usage, u32 hw_inuse)
+{
+ /* add margin */
+ usage = DIV_ROUND_UP(usage * SURPLUS_SCALE_PCT, 100);
+ usage += SURPLUS_SCALE_ABS;
+
+ /* don't bother if the surplus is too small */
+ if (usage + SURPLUS_MIN_ADJ_DELTA > hw_inuse)
+ return 0;
+
+ return usage;
+}
+
+static void ioc_timer_fn(struct timer_list *timer)
+{
+ struct ioc *ioc = container_of(timer, struct ioc, timer);
+ struct ioc_gq *iocg, *tiocg;
+ struct ioc_now now;
+ int nr_surpluses = 0, nr_shortages = 0, nr_lagging = 0;
+ u32 ppm_rthr = MILLION - ioc->params.qos[QOS_RPPM];
+ u32 ppm_wthr = MILLION - ioc->params.qos[QOS_WPPM];
+ u32 missed_ppm[2], rq_wait_pct;
+ u64 period_vtime;
+ int i;
+
+ /* how were the latencies during the period? */
+ ioc_lat_stat(ioc, missed_ppm, &rq_wait_pct);
+
+ /* take care of active iocgs */
+ spin_lock_irq(&ioc->lock);
+
+ ioc_now(ioc, &now);
+
+ period_vtime = now.vnow - ioc->period_at_vtime;
+ if (WARN_ON_ONCE(!period_vtime)) {
+ spin_unlock_irq(&ioc->lock);
+ return;
+ }
+
+ /*
+ * Waiters determine the sleep durations based on the vrate they
+ * saw at the time of sleep. If vrate has increased, some waiters
+ * could be sleeping for too long. Wake up tardy waiters which
+ * should have woken up in the last period and expire idle iocgs.
+ */
+ list_for_each_entry_safe(iocg, tiocg, &ioc->active_iocgs, active_list) {
+ if (!waitqueue_active(&iocg->waitq) && !iocg_is_idle(iocg))
+ continue;
+
+ spin_lock(&iocg->waitq.lock);
+
+ if (waitqueue_active(&iocg->waitq)) {
+ /* might be oversleeping vtime / hweight changes, kick */
+ iocg_kick_waitq(iocg, &now);
+ iocg_kick_delay(iocg, &now, 0);
+ } else if (iocg_is_idle(iocg)) {
+ /* no waiter and idle, deactivate */
+ iocg->last_inuse = iocg->inuse;
+ __propagate_active_weight(iocg, 0, 0);
+ list_del_init(&iocg->active_list);
+ }
+
+ spin_unlock(&iocg->waitq.lock);
+ }
+ commit_active_weights(ioc);
+
+ /* calc usages and see whether some weights need to be moved around */
+ list_for_each_entry(iocg, &ioc->active_iocgs, active_list) {
+ u64 vdone, vtime, vusage, vmargin, vmin;
+ u32 hw_active, hw_inuse, usage;
+
+ /*
+ * Collect unused and wind vtime closer to vnow to prevent
+ * iocgs from accumulating a large amount of budget.
+ */
+ vdone = atomic64_read(&iocg->done_vtime);
+ vtime = atomic64_read(&iocg->vtime);
+ current_hweight(iocg, &hw_active, &hw_inuse);
+
+ /*
+ * Latency QoS detection doesn't account for IOs which are
+ * in-flight for longer than a period. Detect them by
+ * comparing vdone against period start. If lagging behind
+ * IOs from past periods, don't increase vrate.
+ */
+ if (!atomic_read(&iocg_to_blkg(iocg)->use_delay) &&
+ time_after64(vtime, vdone) &&
+ time_after64(vtime, now.vnow -
+ MAX_LAGGING_PERIODS * period_vtime) &&
+ time_before64(vdone, now.vnow - period_vtime))
+ nr_lagging++;
+
+ if (waitqueue_active(&iocg->waitq))
+ vusage = now.vnow - iocg->last_vtime;
+ else if (time_before64(iocg->last_vtime, vtime))
+ vusage = vtime - iocg->last_vtime;
+ else
+ vusage = 0;
+
+ iocg->last_vtime += vusage;
+ /*
+ * Factor in in-flight vtime into vusage to avoid
+ * high-latency completions appearing as idle. This should
+ * be done after the above ->last_time adjustment.
+ */
+ vusage = max(vusage, vtime - vdone);
+
+ /* calculate hweight based usage ratio and record */
+ if (vusage) {
+ usage = DIV64_U64_ROUND_UP(vusage * hw_inuse,
+ period_vtime);
+ iocg->usage_idx = (iocg->usage_idx + 1) % NR_USAGE_SLOTS;
+ iocg->usages[iocg->usage_idx] = usage;
+ } else {
+ usage = 0;
+ }
+
+ /* see whether there's surplus vtime */
+ vmargin = ioc->margin_us * now.vrate;
+ vmin = now.vnow - vmargin;
+
+ iocg->has_surplus = false;
+
+ if (!waitqueue_active(&iocg->waitq) &&
+ time_before64(vtime, vmin)) {
+ u64 delta = vmin - vtime;
+
+ /* throw away surplus vtime */
+ atomic64_add(delta, &iocg->vtime);
+ atomic64_add(delta, &iocg->done_vtime);
+ iocg->last_vtime += delta;
+ /* if usage is sufficiently low, maybe it can donate */
+ if (surplus_adjusted_hweight_inuse(usage, hw_inuse)) {
+ iocg->has_surplus = true;
+ nr_surpluses++;
+ }
+ } else if (hw_inuse < hw_active) {
+ u32 new_hwi, new_inuse;
+
+ /* was donating but might need to take back some */
+ if (waitqueue_active(&iocg->waitq)) {
+ new_hwi = hw_active;
+ } else {
+ new_hwi = max(hw_inuse,
+ usage * SURPLUS_SCALE_PCT / 100 +
+ SURPLUS_SCALE_ABS);
+ }
+
+ new_inuse = div64_u64((u64)iocg->inuse * new_hwi,
+ hw_inuse);
+ new_inuse = clamp_t(u32, new_inuse, 1, iocg->active);
+
+ if (new_inuse > iocg->inuse) {
+ TRACE_IOCG_PATH(inuse_takeback, iocg, &now,
+ iocg->inuse, new_inuse,
+ hw_inuse, new_hwi);
+ __propagate_active_weight(iocg, iocg->weight,
+ new_inuse);
+ }
+ } else {
+ /* genuninely out of vtime */
+ nr_shortages++;
+ }
+ }
+
+ if (!nr_shortages || !nr_surpluses)
+ goto skip_surplus_transfers;
+
+ /* there are both shortages and surpluses, transfer surpluses */
+ list_for_each_entry(iocg, &ioc->active_iocgs, active_list) {
+ u32 usage, hw_active, hw_inuse, new_hwi, new_inuse;
+ int nr_valid = 0;
+
+ if (!iocg->has_surplus)
+ continue;
+
+ /* base the decision on max historical usage */
+ for (i = 0, usage = 0; i < NR_USAGE_SLOTS; i++) {
+ if (iocg->usages[i]) {
+ usage = max(usage, iocg->usages[i]);
+ nr_valid++;
+ }
+ }
+ if (nr_valid < MIN_VALID_USAGES)
+ continue;
+
+ current_hweight(iocg, &hw_active, &hw_inuse);
+ new_hwi = surplus_adjusted_hweight_inuse(usage, hw_inuse);
+ if (!new_hwi)
+ continue;
+
+ new_inuse = DIV64_U64_ROUND_UP((u64)iocg->inuse * new_hwi,
+ hw_inuse);
+ if (new_inuse < iocg->inuse) {
+ TRACE_IOCG_PATH(inuse_giveaway, iocg, &now,
+ iocg->inuse, new_inuse,
+ hw_inuse, new_hwi);
+ __propagate_active_weight(iocg, iocg->weight, new_inuse);
+ }
+ }
+skip_surplus_transfers:
+ commit_active_weights(ioc);
+
+ /*
+ * If q is getting clogged or we're missing too much, we're issuing
+ * too much IO and should lower vtime rate. If we're not missing
+ * and experiencing shortages but not surpluses, we're too stingy
+ * and should increase vtime rate.
+ */
+ if (rq_wait_pct > RQ_WAIT_BUSY_PCT ||
+ missed_ppm[READ] > ppm_rthr ||
+ missed_ppm[WRITE] > ppm_wthr) {
+ ioc->busy_level = max(ioc->busy_level, 0);
+ ioc->busy_level++;
+ } else if (nr_lagging) {
+ ioc->busy_level = max(ioc->busy_level, 0);
+ } else if (nr_shortages && !nr_surpluses &&
+ rq_wait_pct <= RQ_WAIT_BUSY_PCT * UNBUSY_THR_PCT / 100 &&
+ missed_ppm[READ] <= ppm_rthr * UNBUSY_THR_PCT / 100 &&
+ missed_ppm[WRITE] <= ppm_wthr * UNBUSY_THR_PCT / 100) {
+ ioc->busy_level = min(ioc->busy_level, 0);
+ ioc->busy_level--;
+ } else {
+ ioc->busy_level = 0;
+ }
+
+ ioc->busy_level = clamp(ioc->busy_level, -1000, 1000);
+
+ if (ioc->busy_level) {
+ u64 vrate = atomic64_read(&ioc->vtime_rate);
+ u64 vrate_min = ioc->vrate_min, vrate_max = ioc->vrate_max;
+
+ /* rq_wait signal is always reliable, ignore user vrate_min */
+ if (rq_wait_pct > RQ_WAIT_BUSY_PCT)
+ vrate_min = VRATE_MIN;
+
+ /*
+ * If vrate is out of bounds, apply clamp gradually as the
+ * bounds can change abruptly. Otherwise, apply busy_level
+ * based adjustment.
+ */
+ if (vrate < vrate_min) {
+ vrate = div64_u64(vrate * (100 + VRATE_CLAMP_ADJ_PCT),
+ 100);
+ vrate = min(vrate, vrate_min);
+ } else if (vrate > vrate_max) {
+ vrate = div64_u64(vrate * (100 - VRATE_CLAMP_ADJ_PCT),
+ 100);
+ vrate = max(vrate, vrate_max);
+ } else {
+ int idx = min_t(int, abs(ioc->busy_level),
+ ARRAY_SIZE(vrate_adj_pct) - 1);
+ u32 adj_pct = vrate_adj_pct[idx];
+
+ if (ioc->busy_level > 0)
+ adj_pct = 100 - adj_pct;
+ else
+ adj_pct = 100 + adj_pct;
+
+ vrate = clamp(DIV64_U64_ROUND_UP(vrate * adj_pct, 100),
+ vrate_min, vrate_max);
+ }
+
+ trace_iocost_ioc_vrate_adj(ioc, vrate, &missed_ppm, rq_wait_pct,
+ nr_lagging, nr_shortages,
+ nr_surpluses);
+
+ atomic64_set(&ioc->vtime_rate, vrate);
+ ioc->inuse_margin_vtime = DIV64_U64_ROUND_UP(
+ ioc->period_us * vrate * INUSE_MARGIN_PCT, 100);
+ }
+
+ ioc_refresh_params(ioc, false);
+
+ /*
+ * This period is done. Move onto the next one. If nothing's
+ * going on with the device, stop the timer.
+ */
+ atomic64_inc(&ioc->cur_period);
+
+ if (ioc->running != IOC_STOP) {
+ if (!list_empty(&ioc->active_iocgs)) {
+ ioc_start_period(ioc, &now);
+ } else {
+ ioc->busy_level = 0;
+ ioc->running = IOC_IDLE;
+ }
+ }
+
+ spin_unlock_irq(&ioc->lock);
+}
+
+static void calc_vtime_cost_builtin(struct bio *bio, struct ioc_gq *iocg,
+ bool is_merge, u64 *costp)
+{
+ struct ioc *ioc = iocg->ioc;
+ u64 coef_seqio, coef_randio, coef_page;
+ u64 pages = max_t(u64, bio_sectors(bio) >> IOC_SECT_TO_PAGE_SHIFT, 1);
+ u64 seek_pages = 0;
+ u64 cost = 0;
+
+ switch (bio_op(bio)) {
+ case REQ_OP_READ:
+ coef_seqio = ioc->params.lcoefs[LCOEF_RSEQIO];
+ coef_randio = ioc->params.lcoefs[LCOEF_RRANDIO];
+ coef_page = ioc->params.lcoefs[LCOEF_RPAGE];
+ break;
+ case REQ_OP_WRITE:
+ coef_seqio = ioc->params.lcoefs[LCOEF_WSEQIO];
+ coef_randio = ioc->params.lcoefs[LCOEF_WRANDIO];
+ coef_page = ioc->params.lcoefs[LCOEF_WPAGE];
+ break;
+ default:
+ goto out;
+ }
+
+ if (iocg->cursor) {
+ seek_pages = abs(bio->bi_iter.bi_sector - iocg->cursor);
+ seek_pages >>= IOC_SECT_TO_PAGE_SHIFT;
+ }
+
+ if (!is_merge) {
+ if (seek_pages > LCOEF_RANDIO_PAGES) {
+ cost += coef_randio;
+ } else {
+ cost += coef_seqio;
+ }
+ }
+ cost += pages * coef_page;
+out:
+ *costp = cost;
+}
+
+static u64 calc_vtime_cost(struct bio *bio, struct ioc_gq *iocg, bool is_merge)
+{
+ u64 cost;
+
+ calc_vtime_cost_builtin(bio, iocg, is_merge, &cost);
+ return cost;
+}
+
+static void ioc_rqos_throttle(struct rq_qos *rqos, struct bio *bio)
+{
+ struct blkcg_gq *blkg = bio->bi_blkg;
+ struct ioc *ioc = rqos_to_ioc(rqos);
+ struct ioc_gq *iocg = blkg_to_iocg(blkg);
+ struct ioc_now now;
+ struct iocg_wait wait;
+ u32 hw_active, hw_inuse;
+ u64 abs_cost, cost, vtime;
+
+ /* bypass IOs if disabled or for root cgroup */
+ if (!ioc->enabled || !iocg->level)
+ return;
+
+ /* always activate so that even 0 cost IOs get protected to some level */
+ if (!iocg_activate(iocg, &now))
+ return;
+
+ /* calculate the absolute vtime cost */
+ abs_cost = calc_vtime_cost(bio, iocg, false);
+ if (!abs_cost)
+ return;
+
+ iocg->cursor = bio_end_sector(bio);
+
+ vtime = atomic64_read(&iocg->vtime);
+ current_hweight(iocg, &hw_active, &hw_inuse);
+
+ if (hw_inuse < hw_active &&
+ time_after_eq64(vtime + ioc->inuse_margin_vtime, now.vnow)) {
+ TRACE_IOCG_PATH(inuse_reset, iocg, &now,
+ iocg->inuse, iocg->weight, hw_inuse, hw_active);
+ spin_lock_irq(&ioc->lock);
+ propagate_active_weight(iocg, iocg->weight, iocg->weight);
+ spin_unlock_irq(&ioc->lock);
+ current_hweight(iocg, &hw_active, &hw_inuse);
+ }
+
+ cost = abs_cost_to_cost(abs_cost, hw_inuse);
+
+ /*
+ * If no one's waiting and within budget, issue right away. The
+ * tests are racy but the races aren't systemic - we only miss once
+ * in a while which is fine.
+ */
+ if (!waitqueue_active(&iocg->waitq) &&
+ time_before_eq64(vtime + cost, now.vnow)) {
+ iocg_commit_bio(iocg, bio, cost);
+ return;
+ }
+
+ if (bio_issue_as_root_blkg(bio) || fatal_signal_pending(current)) {
+ iocg_commit_bio(iocg, bio, cost);
+ iocg_kick_delay(iocg, &now, cost);
+ return;
+ }
+
+ /*
+ * Append self to the waitq and schedule the wakeup timer if we're
+ * the first waiter. The timer duration is calculated based on the
+ * current vrate. vtime and hweight changes can make it too short
+ * or too long. Each wait entry records the absolute cost it's
+ * waiting for to allow re-evaluation using a custom wait entry.
+ *
+ * If too short, the timer simply reschedules itself. If too long,
+ * the period timer will notice and trigger wakeups.
+ *
+ * All waiters are on iocg->waitq and the wait states are
+ * synchronized using waitq.lock.
+ */
+ spin_lock_irq(&iocg->waitq.lock);
+
+ /*
+ * We activated above but w/o any synchronization. Deactivation is
+ * synchronized with waitq.lock and we won't get deactivated as
+ * long as we're waiting, so we're good if we're activated here.
+ * In the unlikely case that we are deactivated, just issue the IO.
+ */
+ if (unlikely(list_empty(&iocg->active_list))) {
+ spin_unlock_irq(&iocg->waitq.lock);
+ iocg_commit_bio(iocg, bio, cost);
+ return;
+ }
+
+ init_waitqueue_func_entry(&wait.wait, iocg_wake_fn);
+ wait.wait.private = current;
+ wait.bio = bio;
+ wait.abs_cost = abs_cost;
+ wait.committed = false; /* will be set true by waker */
+
+ __add_wait_queue_entry_tail(&iocg->waitq, &wait.wait);
+ iocg_kick_waitq(iocg, &now);
+
+ spin_unlock_irq(&iocg->waitq.lock);
+
+ while (true) {
+ set_current_state(TASK_UNINTERRUPTIBLE);
+ if (wait.committed)
+ break;
+ io_schedule();
+ }
+
+ /* waker already committed us, proceed */
+ finish_wait(&iocg->waitq, &wait.wait);
+}
+
+static void ioc_rqos_merge(struct rq_qos *rqos, struct request *rq,
+ struct bio *bio)
+{
+ struct ioc_gq *iocg = blkg_to_iocg(bio->bi_blkg);
+ sector_t bio_end = bio_end_sector(bio);
+ u32 hw_inuse;
+ u64 abs_cost, cost;
+
+ /* add iff the existing request has cost assigned */
+ if (!rq->bio || !rq->bio->bi_iocost_cost)
+ return;
+
+ abs_cost = calc_vtime_cost(bio, iocg, true);
+ if (!abs_cost)
+ return;
+
+ /* update cursor if backmerging into the request at the cursor */
+ if (blk_rq_pos(rq) < bio_end &&
+ blk_rq_pos(rq) + blk_rq_sectors(rq) == iocg->cursor)
+ iocg->cursor = bio_end;
+
+ current_hweight(iocg, NULL, &hw_inuse);
+ cost = div64_u64(abs_cost * HWEIGHT_WHOLE, hw_inuse);
+ bio->bi_iocost_cost = cost;
+
+ atomic64_add(cost, &iocg->vtime);
+}
+
+static void ioc_rqos_done_bio(struct rq_qos *rqos, struct bio *bio)
+{
+ struct ioc_gq *iocg = blkg_to_iocg(bio->bi_blkg);
+
+ if (iocg && bio->bi_iocost_cost)
+ atomic64_add(bio->bi_iocost_cost, &iocg->done_vtime);
+}
+
+static void ioc_rqos_done(struct rq_qos *rqos, struct request *rq)
+{
+ struct ioc *ioc = rqos_to_ioc(rqos);
+ u64 on_q_ns, rq_wait_ns;
+ int pidx, rw;
+
+ if (!ioc->enabled || !rq->pre_start_time_ns || !rq->start_time_ns)
+ return;
+
+ switch (req_op(rq) & REQ_OP_MASK) {
+ case REQ_OP_READ:
+ pidx = QOS_RLAT;
+ rw = READ;
+ break;
+ case REQ_OP_WRITE:
+ pidx = QOS_WLAT;
+ rw = WRITE;
+ break;
+ default:
+ return;
+ }
+
+ on_q_ns = ktime_get_ns() - rq->pre_start_time_ns;
+ rq_wait_ns = rq->start_time_ns - rq->pre_start_time_ns;
+
+ if (on_q_ns <= ioc->params.qos[pidx] * NSEC_PER_USEC)
+ this_cpu_inc(ioc->pcpu_stat->missed[rw].nr_met);
+ else
+ this_cpu_inc(ioc->pcpu_stat->missed[rw].nr_missed);
+
+ this_cpu_add(ioc->pcpu_stat->rq_wait_ns, rq_wait_ns);
+}
+
+static void ioc_rqos_queue_depth_changed(struct rq_qos *rqos)
+{
+ struct ioc *ioc = rqos_to_ioc(rqos);
+
+ spin_lock_irq(&ioc->lock);
+ ioc_refresh_params(ioc, false);
+ spin_unlock_irq(&ioc->lock);
+}
+
+static void ioc_rqos_exit(struct rq_qos *rqos)
+{
+ struct ioc *ioc = rqos_to_ioc(rqos);
+
+ blkcg_deactivate_policy(rqos->q, &blkcg_policy_iocost);
+
+ spin_lock_irq(&ioc->lock);
+ ioc->running = IOC_STOP;
+ spin_unlock_irq(&ioc->lock);
+
+ del_timer_sync(&ioc->timer);
+ free_percpu(ioc->pcpu_stat);
+ kfree(ioc);
+}
+
+static struct rq_qos_ops ioc_rqos_ops = {
+ .throttle = ioc_rqos_throttle,
+ .merge = ioc_rqos_merge,
+ .done_bio = ioc_rqos_done_bio,
+ .done = ioc_rqos_done,
+ .queue_depth_changed = ioc_rqos_queue_depth_changed,
+ .exit = ioc_rqos_exit,
+};
+
+static int blk_iocost_init(struct request_queue *q)
+{
+ struct ioc *ioc;
+ struct rq_qos *rqos;
+ int ret;
+
+ ioc = kzalloc(sizeof(*ioc), GFP_KERNEL);
+ if (!ioc)
+ return -ENOMEM;
+
+ ioc->pcpu_stat = alloc_percpu(struct ioc_pcpu_stat);
+ if (!ioc->pcpu_stat) {
+ kfree(ioc);
+ return -ENOMEM;
+ }
+
+ rqos = &ioc->rqos;
+ rqos->id = RQ_QOS_COST;
+ rqos->ops = &ioc_rqos_ops;
+ rqos->q = q;
+
+ spin_lock_init(&ioc->lock);
+ timer_setup(&ioc->timer, ioc_timer_fn, 0);
+ INIT_LIST_HEAD(&ioc->active_iocgs);
+
+ ioc->running = IOC_IDLE;
+ atomic64_set(&ioc->vtime_rate, VTIME_PER_USEC);
+ seqcount_init(&ioc->period_seqcount);
+ ioc->period_at = ktime_to_us(ktime_get());
+ atomic64_set(&ioc->cur_period, 0);
+ atomic_set(&ioc->hweight_gen, 0);
+
+ spin_lock_irq(&ioc->lock);
+ ioc->autop_idx = AUTOP_INVALID;
+ ioc_refresh_params(ioc, true);
+ spin_unlock_irq(&ioc->lock);
+
+ rq_qos_add(q, rqos);
+ ret = blkcg_activate_policy(q, &blkcg_policy_iocost);
+ if (ret) {
+ rq_qos_del(q, rqos);
+ kfree(ioc);
+ return ret;
+ }
+ return 0;
+}
+
+static struct blkcg_policy_data *ioc_cpd_alloc(gfp_t gfp)
+{
+ struct ioc_cgrp *iocc;
+
+ iocc = kzalloc(sizeof(struct ioc_cgrp), gfp);
+ iocc->dfl_weight = CGROUP_WEIGHT_DFL;
+
+ return &iocc->cpd;
+}
+
+static void ioc_cpd_free(struct blkcg_policy_data *cpd)
+{
+ kfree(container_of(cpd, struct ioc_cgrp, cpd));
+}
+
+static struct blkg_policy_data *ioc_pd_alloc(gfp_t gfp, struct request_queue *q,
+ struct blkcg *blkcg)
+{
+ int levels = blkcg->css.cgroup->level + 1;
+ struct ioc_gq *iocg;
+
+ iocg = kzalloc_node(sizeof(*iocg) + levels * sizeof(iocg->ancestors[0]),
+ gfp, q->node);
+ if (!iocg)
+ return NULL;
+
+ return &iocg->pd;
+}
+
+static void ioc_pd_init(struct blkg_policy_data *pd)
+{
+ struct ioc_gq *iocg = pd_to_iocg(pd);
+ struct blkcg_gq *blkg = pd_to_blkg(&iocg->pd);
+ struct ioc *ioc = q_to_ioc(blkg->q);
+ struct ioc_now now;
+ struct blkcg_gq *tblkg;
+ unsigned long flags;
+
+ ioc_now(ioc, &now);
+
+ iocg->ioc = ioc;
+ atomic64_set(&iocg->vtime, now.vnow);
+ atomic64_set(&iocg->done_vtime, now.vnow);
+ atomic64_set(&iocg->active_period, atomic64_read(&ioc->cur_period));
+ INIT_LIST_HEAD(&iocg->active_list);
+ iocg->hweight_active = HWEIGHT_WHOLE;
+ iocg->hweight_inuse = HWEIGHT_WHOLE;
+
+ init_waitqueue_head(&iocg->waitq);
+ hrtimer_init(&iocg->waitq_timer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS);
+ iocg->waitq_timer.function = iocg_waitq_timer_fn;
+ hrtimer_init(&iocg->delay_timer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS);
+ iocg->delay_timer.function = iocg_delay_timer_fn;
+
+ iocg->level = blkg->blkcg->css.cgroup->level;
+
+ for (tblkg = blkg; tblkg; tblkg = tblkg->parent) {
+ struct ioc_gq *tiocg = blkg_to_iocg(tblkg);
+ iocg->ancestors[tiocg->level] = tiocg;
+ }
+
+ spin_lock_irqsave(&ioc->lock, flags);
+ weight_updated(iocg);
+ spin_unlock_irqrestore(&ioc->lock, flags);
+}
+
+static void ioc_pd_free(struct blkg_policy_data *pd)
+{
+ struct ioc_gq *iocg = pd_to_iocg(pd);
+ struct ioc *ioc = iocg->ioc;
+
+ if (ioc) {
+ hrtimer_cancel(&iocg->waitq_timer);
+ hrtimer_cancel(&iocg->delay_timer);
+
+ spin_lock(&ioc->lock);
+ if (!list_empty(&iocg->active_list)) {
+ propagate_active_weight(iocg, 0, 0);
+ list_del_init(&iocg->active_list);
+ }
+ spin_unlock(&ioc->lock);
+ }
+ kfree(iocg);
+}
+
+static u64 ioc_weight_prfill(struct seq_file *sf, struct blkg_policy_data *pd,
+ int off)
+{
+ const char *dname = blkg_dev_name(pd->blkg);
+ struct ioc_gq *iocg = pd_to_iocg(pd);
+
+ if (dname && iocg->cfg_weight)
+ seq_printf(sf, "%s %u\n", dname, iocg->cfg_weight);
+ return 0;
+}
+
+
+static int ioc_weight_show(struct seq_file *sf, void *v)
+{
+ struct blkcg *blkcg = css_to_blkcg(seq_css(sf));
+ struct ioc_cgrp *iocc = blkcg_to_iocc(blkcg);
+
+ seq_printf(sf, "default %u\n", iocc->dfl_weight);
+ blkcg_print_blkgs(sf, blkcg, ioc_weight_prfill,
+ &blkcg_policy_iocost, seq_cft(sf)->private, false);
+ return 0;
+}
+
+static ssize_t ioc_weight_write(struct kernfs_open_file *of, char *buf,
+ size_t nbytes, loff_t off)
+{
+ struct blkcg *blkcg = css_to_blkcg(of_css(of));
+ struct ioc_cgrp *iocc = blkcg_to_iocc(blkcg);
+ struct blkg_conf_ctx ctx;
+ struct ioc_gq *iocg;
+ u32 v;
+ int ret;
+
+ if (!strchr(buf, ':')) {
+ struct blkcg_gq *blkg;
+
+ if (!sscanf(buf, "default %u", &v) && !sscanf(buf, "%u", &v))
+ return -EINVAL;
+
+ if (v < CGROUP_WEIGHT_MIN || v > CGROUP_WEIGHT_MAX)
+ return -EINVAL;
+
+ spin_lock(&blkcg->lock);
+ iocc->dfl_weight = v;
+ hlist_for_each_entry(blkg, &blkcg->blkg_list, blkcg_node) {
+ struct ioc_gq *iocg = blkg_to_iocg(blkg);
+
+ if (iocg) {
+ spin_lock_irq(&iocg->ioc->lock);
+ weight_updated(iocg);
+ spin_unlock_irq(&iocg->ioc->lock);
+ }
+ }
+ spin_unlock(&blkcg->lock);
+
+ return nbytes;
+ }
+
+ ret = blkg_conf_prep(blkcg, &blkcg_policy_iocost, buf, &ctx);
+ if (ret)
+ return ret;
+
+ iocg = blkg_to_iocg(ctx.blkg);
+
+ if (!strncmp(ctx.body, "default", 7)) {
+ v = 0;
+ } else {
+ if (!sscanf(ctx.body, "%u", &v))
+ goto einval;
+ if (v < CGROUP_WEIGHT_MIN || v > CGROUP_WEIGHT_MAX)
+ goto einval;
+ }
+
+ spin_lock_irq(&iocg->ioc->lock);
+ iocg->cfg_weight = v;
+ weight_updated(iocg);
+ spin_unlock_irq(&iocg->ioc->lock);
+
+ blkg_conf_finish(&ctx);
+ return nbytes;
+
+einval:
+ blkg_conf_finish(&ctx);
+ return -EINVAL;
+}
+
+static u64 ioc_qos_prfill(struct seq_file *sf, struct blkg_policy_data *pd,
+ int off)
+{
+ const char *dname = blkg_dev_name(pd->blkg);
+ struct ioc *ioc = pd_to_iocg(pd)->ioc;
+
+ if (!dname)
+ return 0;
+
+ seq_printf(sf, "%s enable=%d ctrl=%s rpct=%u.%02u rlat=%u wpct=%u.%02u wlat=%u min=%u.%02u max=%u.%02u\n",
+ dname, ioc->enabled, ioc->user_qos_params ? "user" : "auto",
+ ioc->params.qos[QOS_RPPM] / 10000,
+ ioc->params.qos[QOS_RPPM] % 10000 / 100,
+ ioc->params.qos[QOS_RLAT],
+ ioc->params.qos[QOS_WPPM] / 10000,
+ ioc->params.qos[QOS_WPPM] % 10000 / 100,
+ ioc->params.qos[QOS_WLAT],
+ ioc->params.qos[QOS_MIN] / 10000,
+ ioc->params.qos[QOS_MIN] % 10000 / 100,
+ ioc->params.qos[QOS_MAX] / 10000,
+ ioc->params.qos[QOS_MAX] % 10000 / 100);
+ return 0;
+}
+
+static int ioc_qos_show(struct seq_file *sf, void *v)
+{
+ struct blkcg *blkcg = css_to_blkcg(seq_css(sf));
+
+ blkcg_print_blkgs(sf, blkcg, ioc_qos_prfill,
+ &blkcg_policy_iocost, seq_cft(sf)->private, false);
+ return 0;
+}
+
+static const match_table_t qos_ctrl_tokens = {
+ { QOS_ENABLE, "enable=%u" },
+ { QOS_CTRL, "ctrl=%s" },
+ { NR_QOS_CTRL_PARAMS, NULL },
+};
+
+static const match_table_t qos_tokens = {
+ { QOS_RPPM, "rpct=%s" },
+ { QOS_RLAT, "rlat=%u" },
+ { QOS_WPPM, "wpct=%s" },
+ { QOS_WLAT, "wlat=%u" },
+ { QOS_MIN, "min=%s" },
+ { QOS_MAX, "max=%s" },
+ { NR_QOS_PARAMS, NULL },
+};
+
+static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input,
+ size_t nbytes, loff_t off)
+{
+ struct gendisk *disk;
+ struct ioc *ioc;
+ u32 qos[NR_QOS_PARAMS];
+ bool enable, user;
+ char *p;
+ int ret;
+
+ disk = blkcg_conf_get_disk(&input);
+ if (IS_ERR(disk))
+ return PTR_ERR(disk);
+
+ ioc = q_to_ioc(disk->queue);
+ if (!ioc) {
+ ret = blk_iocost_init(disk->queue);
+ if (ret)
+ goto err;
+ ioc = q_to_ioc(disk->queue);
+ }
+
+ spin_lock_irq(&ioc->lock);
+ memcpy(qos, ioc->params.qos, sizeof(qos));
+ enable = ioc->enabled;
+ user = ioc->user_qos_params;
+ spin_unlock_irq(&ioc->lock);
+
+ while ((p = strsep(&input, " \t\n"))) {
+ substring_t args[MAX_OPT_ARGS];
+ char buf[32];
+ int tok;
+ s64 v;
+
+ if (!*p)
+ continue;
+
+ switch (match_token(p, qos_ctrl_tokens, args)) {
+ case QOS_ENABLE:
+ match_u64(&args[0], &v);
+ enable = v;
+ continue;
+ case QOS_CTRL:
+ match_strlcpy(buf, &args[0], sizeof(buf));
+ if (!strcmp(buf, "auto"))
+ user = false;
+ else if (!strcmp(buf, "user"))
+ user = true;
+ else
+ goto einval;
+ continue;
+ }
+
+ tok = match_token(p, qos_tokens, args);
+ switch (tok) {
+ case QOS_RPPM:
+ case QOS_WPPM:
+ if (match_strlcpy(buf, &args[0], sizeof(buf)) >=
+ sizeof(buf))
+ goto einval;
+ if (cgroup_parse_float(buf, 2, &v))
+ goto einval;
+ if (v < 0 || v > 10000)
+ goto einval;
+ qos[tok] = v * 100;
+ break;
+ case QOS_RLAT:
+ case QOS_WLAT:
+ if (match_u64(&args[0], &v))
+ goto einval;
+ qos[tok] = v;
+ break;
+ case QOS_MIN:
+ case QOS_MAX:
+ if (match_strlcpy(buf, &args[0], sizeof(buf)) >=
+ sizeof(buf))
+ goto einval;
+ if (cgroup_parse_float(buf, 2, &v))
+ goto einval;
+ if (v < 0)
+ goto einval;
+ qos[tok] = clamp_t(s64, v * 100,
+ VRATE_MIN_PPM, VRATE_MAX_PPM);
+ break;
+ default:
+ goto einval;
+ }
+ user = true;
+ }
+
+ if (qos[QOS_MIN] > qos[QOS_MAX])
+ goto einval;
+
+ spin_lock_irq(&ioc->lock);
+
+ if (enable) {
+ blk_queue_flag_set(QUEUE_FLAG_REC_PRESTART, ioc->rqos.q);
+ ioc->enabled = true;
+ } else {
+ blk_queue_flag_clear(QUEUE_FLAG_REC_PRESTART, ioc->rqos.q);
+ ioc->enabled = false;
+ }
+
+ if (user) {
+ memcpy(ioc->params.qos, qos, sizeof(qos));
+ ioc->user_qos_params = true;
+ } else {
+ ioc->user_qos_params = false;
+ }
+
+ ioc_refresh_params(ioc, true);
+ spin_unlock_irq(&ioc->lock);
+
+ put_disk_and_module(disk);
+ return nbytes;
+einval:
+ ret = -EINVAL;
+err:
+ put_disk_and_module(disk);
+ return ret;
+}
+
+static u64 ioc_cost_model_prfill(struct seq_file *sf,
+ struct blkg_policy_data *pd, int off)
+{
+ const char *dname = blkg_dev_name(pd->blkg);
+ struct ioc *ioc = pd_to_iocg(pd)->ioc;
+ u64 *u = ioc->params.i_lcoefs;
+
+ if (!dname)
+ return 0;
+
+ seq_printf(sf, "%s ctrl=%s model=linear "
+ "rbps=%llu rseqiops=%llu rrandiops=%llu "
+ "wbps=%llu wseqiops=%llu wrandiops=%llu\n",
+ dname, ioc->user_cost_model ? "user" : "auto",
+ u[I_LCOEF_RBPS], u[I_LCOEF_RSEQIOPS], u[I_LCOEF_RRANDIOPS],
+ u[I_LCOEF_WBPS], u[I_LCOEF_WSEQIOPS], u[I_LCOEF_WRANDIOPS]);
+ return 0;
+}
+
+static int ioc_cost_model_show(struct seq_file *sf, void *v)
+{
+ struct blkcg *blkcg = css_to_blkcg(seq_css(sf));
+
+ blkcg_print_blkgs(sf, blkcg, ioc_cost_model_prfill,
+ &blkcg_policy_iocost, seq_cft(sf)->private, false);
+ return 0;
+}
+
+static const match_table_t cost_ctrl_tokens = {
+ { COST_CTRL, "ctrl=%s" },
+ { COST_MODEL, "model=%s" },
+ { NR_COST_CTRL_PARAMS, NULL },
+};
+
+static const match_table_t i_lcoef_tokens = {
+ { I_LCOEF_RBPS, "rbps=%u" },
+ { I_LCOEF_RSEQIOPS, "rseqiops=%u" },
+ { I_LCOEF_RRANDIOPS, "rrandiops=%u" },
+ { I_LCOEF_WBPS, "wbps=%u" },
+ { I_LCOEF_WSEQIOPS, "wseqiops=%u" },
+ { I_LCOEF_WRANDIOPS, "wrandiops=%u" },
+ { NR_I_LCOEFS, NULL },
+};
+
+static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input,
+ size_t nbytes, loff_t off)
+{
+ struct gendisk *disk;
+ struct ioc *ioc;
+ u64 u[NR_I_LCOEFS];
+ bool user;
+ char *p;
+ int ret;
+
+ disk = blkcg_conf_get_disk(&input);
+ if (IS_ERR(disk))
+ return PTR_ERR(disk);
+
+ ioc = q_to_ioc(disk->queue);
+ if (!ioc) {
+ ret = blk_iocost_init(disk->queue);
+ if (ret)
+ goto err;
+ ioc = q_to_ioc(disk->queue);
+ }
+
+ spin_lock_irq(&ioc->lock);
+ memcpy(u, ioc->params.i_lcoefs, sizeof(u));
+ user = ioc->user_cost_model;
+ spin_unlock_irq(&ioc->lock);
+
+ while ((p = strsep(&input, " \t\n"))) {
+ substring_t args[MAX_OPT_ARGS];
+ char buf[32];
+ int tok;
+ u64 v;
+
+ if (!*p)
+ continue;
+
+ switch (match_token(p, cost_ctrl_tokens, args)) {
+ case COST_CTRL:
+ match_strlcpy(buf, &args[0], sizeof(buf));
+ if (!strcmp(buf, "auto"))
+ user = false;
+ else if (!strcmp(buf, "user"))
+ user = true;
+ else
+ goto einval;
+ continue;
+ case COST_MODEL:
+ match_strlcpy(buf, &args[0], sizeof(buf));
+ if (strcmp(buf, "linear"))
+ goto einval;
+ continue;
+ }
+
+ tok = match_token(p, i_lcoef_tokens, args);
+ if (tok == NR_I_LCOEFS)
+ goto einval;
+ if (match_u64(&args[0], &v))
+ goto einval;
+ u[tok] = v;
+ user = true;
+ }
+
+ spin_lock_irq(&ioc->lock);
+ if (user) {
+ memcpy(ioc->params.i_lcoefs, u, sizeof(u));
+ ioc->user_cost_model = true;
+ } else {
+ ioc->user_cost_model = false;
+ }
+ ioc_refresh_params(ioc, true);
+ spin_unlock_irq(&ioc->lock);
+
+ put_disk_and_module(disk);
+ return nbytes;
+
+einval:
+ ret = -EINVAL;
+err:
+ put_disk_and_module(disk);
+ return ret;
+}
+
+static struct cftype ioc_files[] = {
+ {
+ .name = "weight",
+ .flags = CFTYPE_NOT_ON_ROOT,
+ .seq_show = ioc_weight_show,
+ .write = ioc_weight_write,
+ },
+ {
+ .name = "cost.qos",
+ .flags = CFTYPE_ONLY_ON_ROOT,
+ .seq_show = ioc_qos_show,
+ .write = ioc_qos_write,
+ },
+ {
+ .name = "cost.model",
+ .flags = CFTYPE_ONLY_ON_ROOT,
+ .seq_show = ioc_cost_model_show,
+ .write = ioc_cost_model_write,
+ },
+ {}
+};
+
+static struct blkcg_policy blkcg_policy_iocost = {
+ .dfl_cftypes = ioc_files,
+ .cpd_alloc_fn = ioc_cpd_alloc,
+ .cpd_free_fn = ioc_cpd_free,
+ .pd_alloc_fn = ioc_pd_alloc,
+ .pd_init_fn = ioc_pd_init,
+ .pd_free_fn = ioc_pd_free,
+};
+
+static int __init ioc_init(void)
+{
+ return blkcg_policy_register(&blkcg_policy_iocost);
+}
+
+static void __exit ioc_exit(void)
+{
+ return blkcg_policy_unregister(&blkcg_policy_iocost);
+}
+
+module_init(ioc_init);
+module_exit(ioc_exit);
diff --git a/block/blk-rq-qos.h b/block/blk-rq-qos.h
index 5f8b75826a98..08a09dbe0f4b 100644
--- a/block/blk-rq-qos.h
+++ b/block/blk-rq-qos.h
@@ -15,6 +15,7 @@ struct blk_mq_debugfs_attr;
enum rq_qos_id {
RQ_QOS_WBT,
RQ_QOS_LATENCY,
+ RQ_QOS_COST,
};
struct rq_wait {
@@ -84,6 +85,8 @@ static inline const char *rq_qos_id_to_name(enum rq_qos_id id)
return "wbt";
case RQ_QOS_LATENCY:
return "latency";
+ case RQ_QOS_COST:
+ return "cost";
}
return "unknown";
}
diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h
index feff3fe4467e..e53e84bf83a6 100644
--- a/include/linux/blk_types.h
+++ b/include/linux/blk_types.h
@@ -169,6 +169,9 @@ struct bio {
*/
struct blkcg_gq *bi_blkg;
struct bio_issue bi_issue;
+#ifdef CONFIG_BLK_CGROUP_IOCOST
+ u64 bi_iocost_cost;
+#endif
#endif
union {
#if defined(CONFIG_BLK_DEV_INTEGRITY)
diff --git a/include/trace/events/iocost.h b/include/trace/events/iocost.h
new file mode 100644
index 000000000000..ec2217dd57ac
--- /dev/null
+++ b/include/trace/events/iocost.h
@@ -0,0 +1,174 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM iocost
+
+#if !defined(_TRACE_BLK_IOCOST_H) || defined(TRACE_HEADER_MULTI_READ)
+#define _TRACE_BLK_IOCOST_H
+
+#include <linux/tracepoint.h>
+
+TRACE_EVENT(iocost_iocg_activate,
+
+ TP_PROTO(struct ioc_gq *iocg, const char *path, struct ioc_now *now,
+ u64 last_period, u64 cur_period, u64 vtime),
+
+ TP_ARGS(iocg, path, now, last_period, cur_period, vtime),
+
+ TP_STRUCT__entry (
+ __string(devname, ioc_name(iocg->ioc))
+ __string(cgroup, path)
+ __field(u64, now)
+ __field(u64, vnow)
+ __field(u64, vrate)
+ __field(u64, last_period)
+ __field(u64, cur_period)
+ __field(u64, last_vtime)
+ __field(u64, vtime)
+ __field(u32, weight)
+ __field(u32, inuse)
+ __field(u64, hweight_active)
+ __field(u64, hweight_inuse)
+ ),
+
+ TP_fast_assign(
+ __assign_str(devname, ioc_name(iocg->ioc));
+ __assign_str(cgroup, path);
+ __entry->now = now->now;
+ __entry->vnow = now->vnow;
+ __entry->vrate = now->vrate;
+ __entry->last_period = last_period;
+ __entry->cur_period = cur_period;
+ __entry->last_vtime = iocg->last_vtime;
+ __entry->vtime = vtime;
+ __entry->weight = iocg->weight;
+ __entry->inuse = iocg->inuse;
+ __entry->hweight_active = iocg->hweight_active;
+ __entry->hweight_inuse = iocg->hweight_inuse;
+ ),
+
+ TP_printk("[%s:%s] now=%llu:%llu vrate=%llu "
+ "period=%llu->%llu vtime=%llu->%llu "
+ "weight=%u/%u hweight=%llu/%llu",
+ __get_str(devname), __get_str(cgroup),
+ __entry->now, __entry->vnow, __entry->vrate,
+ __entry->last_period, __entry->cur_period,
+ __entry->last_vtime, __entry->vtime,
+ __entry->inuse, __entry->weight,
+ __entry->hweight_inuse, __entry->hweight_active
+ )
+);
+
+DECLARE_EVENT_CLASS(iocg_inuse_update,
+
+ TP_PROTO(struct ioc_gq *iocg, const char *path, struct ioc_now *now,
+ u32 old_inuse, u32 new_inuse,
+ u64 old_hw_inuse, u64 new_hw_inuse),
+
+ TP_ARGS(iocg, path, now, old_inuse, new_inuse,
+ old_hw_inuse, new_hw_inuse),
+
+ TP_STRUCT__entry (
+ __string(devname, ioc_name(iocg->ioc))
+ __string(cgroup, path)
+ __field(u64, now)
+ __field(u32, old_inuse)
+ __field(u32, new_inuse)
+ __field(u64, old_hweight_inuse)
+ __field(u64, new_hweight_inuse)
+ ),
+
+ TP_fast_assign(
+ __assign_str(devname, ioc_name(iocg->ioc));
+ __assign_str(cgroup, path);
+ __entry->now = now->now;
+ __entry->old_inuse = old_inuse;
+ __entry->new_inuse = new_inuse;
+ __entry->old_hweight_inuse = old_hw_inuse;
+ __entry->new_hweight_inuse = new_hw_inuse;
+ ),
+
+ TP_printk("[%s:%s] now=%llu inuse=%u->%u hw_inuse=%llu->%llu",
+ __get_str(devname), __get_str(cgroup), __entry->now,
+ __entry->old_inuse, __entry->new_inuse,
+ __entry->old_hweight_inuse, __entry->new_hweight_inuse
+ )
+);
+
+DEFINE_EVENT(iocg_inuse_update, iocost_inuse_takeback,
+
+ TP_PROTO(struct ioc_gq *iocg, const char *path, struct ioc_now *now,
+ u32 old_inuse, u32 new_inuse,
+ u64 old_hw_inuse, u64 new_hw_inuse),
+
+ TP_ARGS(iocg, path, now, old_inuse, new_inuse,
+ old_hw_inuse, new_hw_inuse)
+);
+
+DEFINE_EVENT(iocg_inuse_update, iocost_inuse_giveaway,
+
+ TP_PROTO(struct ioc_gq *iocg, const char *path, struct ioc_now *now,
+ u32 old_inuse, u32 new_inuse,
+ u64 old_hw_inuse, u64 new_hw_inuse),
+
+ TP_ARGS(iocg, path, now, old_inuse, new_inuse,
+ old_hw_inuse, new_hw_inuse)
+);
+
+DEFINE_EVENT(iocg_inuse_update, iocost_inuse_reset,
+
+ TP_PROTO(struct ioc_gq *iocg, const char *path, struct ioc_now *now,
+ u32 old_inuse, u32 new_inuse,
+ u64 old_hw_inuse, u64 new_hw_inuse),
+
+ TP_ARGS(iocg, path, now, old_inuse, new_inuse,
+ old_hw_inuse, new_hw_inuse)
+);
+
+TRACE_EVENT(iocost_ioc_vrate_adj,
+
+ TP_PROTO(struct ioc *ioc, u64 new_vrate, u32 (*missed_ppm)[2],
+ u32 rq_wait_pct, int nr_lagging, int nr_shortages,
+ int nr_surpluses),
+
+ TP_ARGS(ioc, new_vrate, missed_ppm, rq_wait_pct, nr_lagging, nr_shortages,
+ nr_surpluses),
+
+ TP_STRUCT__entry (
+ __string(devname, ioc_name(ioc))
+ __field(u64, old_vrate)
+ __field(u64, new_vrate)
+ __field(int, busy_level)
+ __field(u32, read_missed_ppm)
+ __field(u32, write_missed_ppm)
+ __field(u32, rq_wait_pct)
+ __field(int, nr_lagging)
+ __field(int, nr_shortages)
+ __field(int, nr_surpluses)
+ ),
+
+ TP_fast_assign(
+ __assign_str(devname, ioc_name(ioc));
+ __entry->old_vrate = atomic64_read(&ioc->vtime_rate);;
+ __entry->new_vrate = new_vrate;
+ __entry->busy_level = ioc->busy_level;
+ __entry->read_missed_ppm = (*missed_ppm)[READ];
+ __entry->write_missed_ppm = (*missed_ppm)[WRITE];
+ __entry->rq_wait_pct = rq_wait_pct;
+ __entry->nr_lagging = nr_lagging;
+ __entry->nr_shortages = nr_shortages;
+ __entry->nr_surpluses = nr_surpluses;
+ ),
+
+ TP_printk("[%s] vrate=%llu->%llu busy=%d missed_ppm=%u:%u rq_wait_pct=%u lagging=%d shortages=%d surpluses=%d",
+ __get_str(devname), __entry->old_vrate, __entry->new_vrate,
+ __entry->busy_level,
+ __entry->read_missed_ppm, __entry->write_missed_ppm,
+ __entry->rq_wait_pct, __entry->nr_lagging, __entry->nr_shortages,
+ __entry->nr_surpluses
+ )
+);
+
+#endif /* _TRACE_BLK_IOCOST_H */
+
+/* This part must be outside protection */
+#include <trace/define_trace.h>
--
2.17.1
^ permalink raw reply related
* [PATCH 10/10] blkcg: add tools/cgroup/iocost_coef_gen.py
From: Tejun Heo @ 2019-07-10 20:51 UTC (permalink / raw)
To: axboe, newella, clm, josef, dennisz, lizefan, hannes
Cc: linux-kernel, kernel-team, linux-block, cgroups, Tejun Heo
In-Reply-To: <20190710205128.1316483-1-tj@kernel.org>
Add a script which can be used to generate device-specific iocost
linear model coefficients.
Signed-off-by: Tejun Heo <tj@kernel.org>
---
Documentation/admin-guide/cgroup-v2.rst | 3 +
block/blk-iocost.c | 3 +
tools/cgroup/iocost_coef_gen.py | 178 ++++++++++++++++++++++++
3 files changed, 184 insertions(+)
create mode 100644 tools/cgroup/iocost_coef_gen.py
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index aff812631c40..e683c51f7ba3 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1519,6 +1519,9 @@ IO Interface Files
The IO cost model isn't expected to be accurate in absolute
sense and is scaled to the device behavior dynamically.
+ If needed, tools/cgroup/iocost_coef_gen.py can be used to
+ generate device-specific coefficients.
+
io.weight
A read-write flat-keyed file which exists on non-root cgroups.
The default is "default 100".
diff --git a/block/blk-iocost.c b/block/blk-iocost.c
index d608c5aa84ed..a8d5c90de0c2 100644
--- a/block/blk-iocost.c
+++ b/block/blk-iocost.c
@@ -46,6 +46,9 @@
* If needed, tools/cgroup/iocost_coef_gen.py can be used to generate
* device-specific coefficients.
*
+ * If needed, tools/cgroup/iocost_coef_gen.py can be used to generate
+ * device-specific coefficients.
+ *
* 2. Control Strategy
*
* The device virtual time (vtime) is used as the primary control metric.
diff --git a/tools/cgroup/iocost_coef_gen.py b/tools/cgroup/iocost_coef_gen.py
new file mode 100644
index 000000000000..df17a2ae80e5
--- /dev/null
+++ b/tools/cgroup/iocost_coef_gen.py
@@ -0,0 +1,178 @@
+#!/usr/bin/env python3
+#
+# Copyright (C) 2019 Tejun Heo <tj@kernel.org>
+# Copyright (C) 2019 Andy Newell <newella@fb.com>
+# Copyright (C) 2019 Facebook
+
+desc = """
+Generate linear IO cost model coefficients used by the blk-iocost
+controller. If the target raw testdev is specified, destructive tests
+are performed against the whole device; otherwise, on
+./iocost-coef-fio.testfile. The result can be written directly to
+/sys/fs/cgroup/io.cost.model.
+
+On high performance devices, --numjobs > 1 is needed to achieve
+saturation.
+
+See Documentation/admin-guide/cgroup-v2.rst and block/blk-iocost.c
+for more details.
+"""
+
+import argparse
+import re
+import json
+import glob
+import os
+import sys
+import atexit
+import shutil
+import tempfile
+import subprocess
+
+parser = argparse.ArgumentParser(description=desc,
+ formatter_class=argparse.RawTextHelpFormatter)
+parser.add_argument('--testdev', metavar='DEV',
+ help='Raw block device to use for testing, ignores --testfile-size')
+parser.add_argument('--testfile-size-gb', type=float, metavar='GIGABYTES', default=16,
+ help='Testfile size in gigabytes (default: %(default)s)')
+parser.add_argument('--duration', type=int, metavar='SECONDS', default=120,
+ help='Individual test run duration in seconds (default: %(default)s)')
+parser.add_argument('--seqio-block-mb', metavar='MEGABYTES', type=int, default=128,
+ help='Sequential test block size in megabytes (default: %(default)s)')
+parser.add_argument('--seq-depth', type=int, metavar='DEPTH', default=64,
+ help='Sequential test queue depth (default: %(default)s)')
+parser.add_argument('--rand-depth', type=int, metavar='DEPTH', default=64,
+ help='Random test queue depth (default: %(default)s)')
+parser.add_argument('--numjobs', type=int, metavar='JOBS', default=1,
+ help='Number of parallel fio jobs to run (default: %(default)s)')
+parser.add_argument('--quiet', action='store_true')
+parser.add_argument('--verbose', action='store_true')
+
+def info(msg):
+ if not args.quiet:
+ print(msg)
+
+def dbg(msg):
+ if args.verbose and not args.quiet:
+ print(msg)
+
+# determine ('DEVNAME', 'MAJ:MIN') for @path
+def dir_to_dev(path):
+ # find the block device the current directory is on
+ devname = subprocess.run(f'findmnt -nvo SOURCE -T{path}',
+ stdout=subprocess.PIPE, shell=True).stdout
+ devname = os.path.basename(devname).decode('utf-8').strip()
+
+ # partition -> whole device
+ parents = glob.glob('/sys/block/*/' + devname)
+ if len(parents):
+ devname = os.path.basename(os.path.dirname(parents[0]))
+ rdev = os.stat(f'/dev/{devname}').st_rdev
+ return (devname, f'{os.major(rdev)}:{os.minor(rdev)}')
+
+def create_testfile(path, size):
+ global args
+
+ if os.path.isfile(path) and os.stat(path).st_size == size:
+ return
+
+ info(f'Creating testfile {path}')
+ subprocess.check_call(f'rm -f {path}', shell=True)
+ subprocess.check_call(f'touch {path}', shell=True)
+ subprocess.call(f'chattr +C {path}', shell=True)
+ subprocess.check_call(
+ f'pv -s {size} -pr /dev/urandom {"-q" if args.quiet else ""} | '
+ f'dd of={path} count={size} '
+ f'iflag=count_bytes,fullblock oflag=direct bs=16M status=none',
+ shell=True)
+
+def run_fio(testfile, duration, iotype, iodepth, blocksize, jobs):
+ global args
+
+ eta = 'never' if args.quiet else 'always'
+ outfile = tempfile.NamedTemporaryFile()
+ cmd = (f'fio --direct=1 --ioengine=libaio --name=coef '
+ f'--filename={testfile} --runtime={round(duration)} '
+ f'--readwrite={iotype} --iodepth={iodepth} --blocksize={blocksize} '
+ f'--eta={eta} --output-format json --output={outfile.name} '
+ f'--time_based --numjobs={jobs}')
+ if args.verbose:
+ dbg(f'Running {cmd}')
+ subprocess.check_call(cmd, shell=True)
+ with open(outfile.name, 'r') as f:
+ d = json.loads(f.read())
+ return sum(j['read']['bw_bytes'] + j['write']['bw_bytes'] for j in d['jobs'])
+
+def restore_elevator_nomerges():
+ global elevator_path, nomerges_path, elevator, nomerges
+
+ info(f'Restoring elevator to {elevator} and nomerges to {nomerges}')
+ with open(elevator_path, 'w') as f:
+ f.write(elevator)
+ with open(nomerges_path, 'w') as f:
+ f.write(nomerges)
+
+
+args = parser.parse_args()
+
+missing = False
+for cmd in [ 'findmnt', 'pv', 'dd', 'fio' ]:
+ if not shutil.which(cmd):
+ print(f'Required command "{cmd}" is missing', file=sys.stderr)
+ missing = True
+if missing:
+ sys.exit(1)
+
+if args.testdev:
+ devname = os.path.basename(args.testdev)
+ rdev = os.stat(f'/dev/{devname}').st_rdev
+ devno = f'{os.major(rdev)}:{os.minor(rdev)}'
+ testfile = f'/dev/{devname}'
+ info(f'Test target: {devname}({devno})')
+else:
+ devname, devno = dir_to_dev('.')
+ testfile = 'iocost-coef-fio.testfile'
+ testfile_size = int(args.testfile_size_gb * 2 ** 30)
+ create_testfile(testfile, testfile_size)
+ info(f'Test target: {testfile} on {devname}({devno})')
+
+elevator_path = f'/sys/block/{devname}/queue/scheduler'
+nomerges_path = f'/sys/block/{devname}/queue/nomerges'
+
+with open(elevator_path, 'r') as f:
+ elevator = re.sub(r'.*\[(.*)\].*', r'\1', f.read().strip())
+with open(nomerges_path, 'r') as f:
+ nomerges = f.read().strip()
+
+info(f'Temporarily disabling elevator and merges')
+atexit.register(restore_elevator_nomerges)
+with open(elevator_path, 'w') as f:
+ f.write('none')
+with open(nomerges_path, 'w') as f:
+ f.write('1')
+
+info('Determining rbps...')
+rbps = run_fio(testfile, args.duration, 'read',
+ 1, args.seqio_block_mb * (2 ** 20), args.numjobs)
+info(f'\nrbps={rbps}, determining rseqiops...')
+rseqiops = round(run_fio(testfile, args.duration, 'read',
+ args.seq_depth, 4096, args.numjobs) / 4096)
+info(f'\nrseqiops={rseqiops}, determining rrandiops...')
+rrandiops = round(run_fio(testfile, args.duration, 'randread',
+ args.rand_depth, 4096, args.numjobs) / 4096)
+info(f'\nrrandiops={rrandiops}, determining wbps...')
+wbps = run_fio(testfile, args.duration, 'write',
+ 1, args.seqio_block_mb * (2 ** 20), args.numjobs)
+info(f'\nwbps={wbps}, determining wseqiops...')
+wseqiops = round(run_fio(testfile, args.duration, 'write',
+ args.seq_depth, 4096, args.numjobs) / 4096)
+info(f'\nwseqiops={wseqiops}, determining wrandiops...')
+wrandiops = round(run_fio(testfile, args.duration, 'randwrite',
+ args.rand_depth, 4096, args.numjobs) / 4096)
+info(f'\nwrandiops={wrandiops}')
+restore_elevator_nomerges()
+atexit.unregister(restore_elevator_nomerges)
+info('')
+
+print(f'{devno} rbps={rbps} rseqiops={rseqiops} rrandiops={rrandiops} '
+ f'wbps={wbps} wseqiops={wseqiops} wrandiops={wrandiops}')
--
2.17.1
^ permalink raw reply related
* [PATCH 09/10] blkcg: add tools/cgroup/iocost_monitor.py
From: Tejun Heo @ 2019-07-10 20:51 UTC (permalink / raw)
To: axboe, newella, clm, josef, dennisz, lizefan, hannes
Cc: linux-kernel, kernel-team, linux-block, cgroups, Tejun Heo,
Omar Sandoval
In-Reply-To: <20190710205128.1316483-1-tj@kernel.org>
Instead of mucking with debugfs and ->pd_stat(), add drgn based
monitoring script.
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Omar Sandoval <osandov@fb.com>
---
block/blk-iocost.c | 21 +++
tools/cgroup/iocost_monitor.py | 270 +++++++++++++++++++++++++++++++++
2 files changed, 291 insertions(+)
create mode 100644 tools/cgroup/iocost_monitor.py
diff --git a/block/blk-iocost.c b/block/blk-iocost.c
index 1f697e5729f6..d608c5aa84ed 100644
--- a/block/blk-iocost.c
+++ b/block/blk-iocost.c
@@ -149,6 +149,27 @@
* donate and should take back how much requires hweight propagations
* anyway making it easier to implement and understand as a separate
* mechanism.
+ *
+ * 3. Monitoring
+ *
+ * Instead of debugfs or other clumsy monitoring mechanisms, this
+ * controller uses a drgn based monitoring script -
+ * tools/cgroup/iocost_monitor.py. For details on drgn, please see
+ * https://github.com/osandov/drgn. The ouput looks like the following.
+ *
+ * sdb RUN per=300ms cur_per=234.218:v203.695 busy= +1 vrate= 62.12%
+ * active weight hweight% inflt% del_ms usages%
+ * test/a * 50/ 50 33.33/ 33.33 27.65 0*041 033:033:033
+ * test/b * 100/ 100 66.67/ 66.67 17.56 0*000 066:079:077
+ *
+ * - per : Timer period
+ * - cur_per : Internal wall and device vtime clock
+ * - vrate : Device virtual time rate against wall clock
+ * - weight : Surplus-adjusted and configured weights
+ * - hweight : Surplus-adjusted and configured hierarchical weights
+ * - inflt : The percentage of in-flight IO cost at the end of last period
+ * - del_ms : Deferred issuer delay induction level and duration
+ * - usages : Usage history
*/
#include <linux/kernel.h>
diff --git a/tools/cgroup/iocost_monitor.py b/tools/cgroup/iocost_monitor.py
new file mode 100644
index 000000000000..2c9445e966d8
--- /dev/null
+++ b/tools/cgroup/iocost_monitor.py
@@ -0,0 +1,270 @@
+#!/usr/bin/env drgn
+#
+# Copyright (C) 2019 Tejun Heo <tj@kernel.org>
+# Copyright (C) 2019 Facebook
+
+desc = """
+This is a drgn script to monitor the blk-iocost cgroup controller.
+See the comment at the top of block/blk-iocost.c for more details.
+For drgn, visit https://github.com/osandov/drgn.
+"""
+
+import sys
+import re
+import time
+import json
+
+import drgn
+from drgn import container_of
+from drgn.helpers.linux.list import list_for_each_entry,list_empty
+from drgn.helpers.linux.radixtree import radix_tree_for_each,radix_tree_lookup
+
+import argparse
+parser = argparse.ArgumentParser(description=desc,
+ formatter_class=argparse.RawTextHelpFormatter)
+parser.add_argument('devname', metavar='DEV',
+ help='Target block device name (e.g. sda)')
+parser.add_argument('--cgroup', action='append', metavar='REGEX',
+ help='Regex for target cgroups, ')
+parser.add_argument('--interval', '-i', metavar='SECONDS', type=float, default=1,
+ help='Monitoring interval in seconds')
+parser.add_argument('--json', action='store_true',
+ help='Output in json')
+args = parser.parse_args()
+
+def err(s):
+ print(s, file=sys.stderr, flush=True)
+ sys.exit(1)
+
+try:
+ blkcg_root = prog['blkcg_root']
+ plid = prog['blkcg_policy_iocost'].plid.value_()
+except:
+ err('The kernel does not have iocost enabled')
+
+IOC_RUNNING = prog['IOC_RUNNING'].value_()
+NR_USAGE_SLOTS = prog['NR_USAGE_SLOTS'].value_()
+HWEIGHT_WHOLE = prog['HWEIGHT_WHOLE'].value_()
+VTIME_PER_SEC = prog['VTIME_PER_SEC'].value_()
+VTIME_PER_USEC = prog['VTIME_PER_USEC'].value_()
+AUTOP_SSD_FAST = prog['AUTOP_SSD_FAST'].value_()
+AUTOP_SSD_DFL = prog['AUTOP_SSD_DFL'].value_()
+AUTOP_SSD_QD1 = prog['AUTOP_SSD_QD1'].value_()
+AUTOP_HDD = prog['AUTOP_HDD'].value_()
+
+autop_names = {
+ AUTOP_SSD_FAST: 'ssd_fast',
+ AUTOP_SSD_DFL: 'ssd_dfl',
+ AUTOP_SSD_QD1: 'ssd_qd1',
+ AUTOP_HDD: 'hdd',
+}
+
+class BlkgIterator:
+ def blkcg_name(blkcg):
+ return blkcg.css.cgroup.kn.name.string_().decode('utf-8')
+
+ def walk(self, blkcg, q_id, parent_path):
+ if not self.include_dying and \
+ not (blkcg.css.flags.value_() & prog['CSS_ONLINE'].value_()):
+ return
+
+ name = BlkgIterator.blkcg_name(blkcg)
+ path = parent_path + '/' + name if parent_path else name
+ blkg = drgn.Object(prog, 'struct blkcg_gq',
+ address=radix_tree_lookup(blkcg.blkg_tree, q_id))
+ if not blkg.address_:
+ return
+
+ self.blkgs.append((path if path else '/', blkg))
+
+ for c in list_for_each_entry('struct blkcg',
+ blkcg.css.children.address_of_(), 'css.sibling'):
+ self.walk(c, q_id, path)
+
+ def __init__(self, root_blkcg, q_id, include_dying=False):
+ self.include_dying = include_dying
+ self.blkgs = []
+ self.walk(root_blkcg, q_id, '')
+
+ def __iter__(self):
+ return iter(self.blkgs)
+
+class IocStat:
+ def __init__(self, ioc):
+ global autop_names
+
+ self.enabled = ioc.enabled.value_()
+ self.running = ioc.running.value_() == IOC_RUNNING
+ self.period_ms = round(ioc.period_us.value_() / 1_000)
+ self.period_at = ioc.period_at.value_() / 1_000_000
+ self.vperiod_at = ioc.period_at_vtime.value_() / VTIME_PER_SEC
+ self.vrate_pct = ioc.vtime_rate.counter.value_() * 100 / VTIME_PER_USEC
+ self.busy_level = ioc.busy_level.value_()
+ self.autop_idx = ioc.autop_idx.value_()
+ self.user_cost_model = ioc.user_cost_model.value_()
+ self.user_qos_params = ioc.user_qos_params.value_()
+
+ if self.autop_idx in autop_names:
+ self.autop_name = autop_names[self.autop_idx]
+ else:
+ self.autop_name = '?'
+
+ def dict(self, now):
+ return { 'device' : devname,
+ 'timestamp' : now,
+ 'enabled' : self.enabled,
+ 'running' : self.running,
+ 'period_ms' : self.period_ms,
+ 'period_at' : self.period_at,
+ 'period_vtime_at' : self.vperiod_at,
+ 'busy_level' : self.busy_level,
+ 'vrate_pct' : self.vrate_pct, }
+
+ def table_preamble_str(self):
+ state = ('RUN' if self.running else 'IDLE') if self.enabled else 'OFF'
+ output = f'{devname} {state:4} ' \
+ f'per={self.period_ms}ms ' \
+ f'cur_per={self.period_at:.3f}:v{self.vperiod_at:.3f} ' \
+ f'busy={self.busy_level:+3} ' \
+ f'vrate={self.vrate_pct:6.2f}% ' \
+ f'params={self.autop_name}'
+ if self.user_cost_model or self.user_qos_params:
+ output += f'({"C" if self.user_cost_model else ""}{"Q" if self.user_qos_params else ""})'
+ return output
+
+ def table_header_str(self):
+ return f'{"":25} active {"weight":>9} {"hweight%":>13} {"inflt%":>6} ' \
+ f'{"del_ms":>6} {"usages%"}'
+
+class IocgStat:
+ def __init__(self, iocg):
+ ioc = iocg.ioc
+ blkg = iocg.pd.blkg
+
+ self.is_active = not list_empty(iocg.active_list.address_of_())
+ self.weight = iocg.weight.value_()
+ self.active = iocg.active.value_()
+ self.inuse = iocg.inuse.value_()
+ self.hwa_pct = iocg.hweight_active.value_() * 100 / HWEIGHT_WHOLE
+ self.hwi_pct = iocg.hweight_inuse.value_() * 100 / HWEIGHT_WHOLE
+
+ vdone = iocg.done_vtime.counter.value_()
+ vtime = iocg.vtime.counter.value_()
+ vrate = ioc.vtime_rate.counter.value_()
+ period_vtime = ioc.period_us.value_() * vrate
+ if period_vtime:
+ self.inflight_pct = (vtime - vdone) * 100 / period_vtime
+ else:
+ self.inflight_pct = 0
+
+ self.use_delay = min(blkg.use_delay.counter.value_(), 99)
+ self.delay_ms = min(round(blkg.delay_nsec.counter.value_() / 1_000_000), 999)
+
+ usage_idx = iocg.usage_idx.value_()
+ self.usages = []
+ self.usage = 0
+ for i in range(NR_USAGE_SLOTS):
+ usage = iocg.usages[(usage_idx + i) % NR_USAGE_SLOTS].value_()
+ upct = min(usage * 100 / HWEIGHT_WHOLE, 999)
+ self.usages.append(upct)
+ self.usage = max(self.usage, upct)
+
+ def dict(self, now, path):
+ out = { 'cgroup' : path,
+ 'timestamp' : now,
+ 'is_active' : self.is_active,
+ 'weight' : self.weight,
+ 'weight_active' : self.active,
+ 'weight_inuse' : self.inuse,
+ 'hweight_active_pct' : self.hwa_pct,
+ 'hweight_inuse_pct' : self.hwi_pct,
+ 'inflight_pct' : self.inflight_pct,
+ 'use_delay' : self.use_delay,
+ 'delay_ms' : self.delay_ms,
+ 'usage_pct' : self.usage }
+ for i in range(len(self.usages)):
+ out[f'usage_pct_{i}'] = f'{self.usages[i]}'
+ return out
+
+ def table_row_str(self, path):
+ out = f'{path[-28:]:28} ' \
+ f'{"*" if self.is_active else " "} ' \
+ f'{self.inuse:5}/{self.active:5} ' \
+ f'{self.hwi_pct:6.2f}/{self.hwa_pct:6.2f} ' \
+ f'{self.inflight_pct:6.2f} ' \
+ f'{self.use_delay:2}*{self.delay_ms:03} '
+ for u in self.usages:
+ out += f'{round(u):03d}:'
+ out = out.rstrip(':')
+ return out
+
+# handle args
+table_fmt = not args.json
+interval = args.interval
+devname = args.devname
+
+if args.json:
+ table_fmt = False
+
+re_str = None
+if args.cgroup:
+ for r in args.cgroup:
+ if re_str is None:
+ re_str = r
+ else:
+ re_str += '|' + r
+
+filter_re = re.compile(re_str) if re_str else None
+
+# Locate the roots
+q_id = None
+root_iocg = None
+ioc = None
+
+for i, ptr in radix_tree_for_each(blkcg_root.blkg_tree):
+ blkg = drgn.Object(prog, 'struct blkcg_gq', address=ptr)
+ try:
+ if devname == blkg.q.kobj.parent.name.string_().decode('utf-8'):
+ q_id = blkg.q.id.value_()
+ if blkg.pd[plid]:
+ root_iocg = container_of(blkg.pd[plid], 'struct ioc_gq', 'pd')
+ ioc = root_iocg.ioc
+ break
+ except:
+ pass
+
+if ioc is None:
+ err(f'Could not find ioc for {devname}');
+
+# Keep printing
+while True:
+ now = time.time()
+ iocstat = IocStat(ioc)
+ output = ''
+
+ if table_fmt:
+ output += '\n' + iocstat.table_preamble_str()
+ output += '\n' + iocstat.table_header_str()
+ else:
+ output += json.dumps(iocstat.dict(now))
+
+ for path, blkg in BlkgIterator(blkcg_root, q_id):
+ if filter_re and not filter_re.match(path):
+ continue
+ if not blkg.pd[plid]:
+ continue
+
+ iocg = container_of(blkg.pd[plid], 'struct ioc_gq', 'pd')
+ iocg_stat = IocgStat(iocg)
+
+ if not filter_re and not iocg_stat.is_active:
+ continue
+
+ if table_fmt:
+ output += '\n' + iocg_stat.table_row_str(path)
+ else:
+ output += '\n' + json.dumps(iocg_stat.dict(now, path))
+
+ print(output)
+ sys.stdout.flush()
+ time.sleep(interval)
--
2.17.1
^ permalink raw reply related
* [PATCH 03/10] blkcg: separate blkcg_conf_get_disk() out of blkg_conf_prep()
From: Tejun Heo @ 2019-07-10 20:51 UTC (permalink / raw)
To: axboe, newella, clm, josef, dennisz, lizefan, hannes
Cc: linux-kernel, kernel-team, linux-block, cgroups, Tejun Heo
In-Reply-To: <20190710205128.1316483-1-tj@kernel.org>
Separate out blkcg_conf_get_disk() so that it can be used by blkcg
policy interface file input parsers before the policy is actually
enabled. This doesn't introduce any functional changes.
Signed-off-by: Tejun Heo <tj@kernel.org>
---
block/blk-cgroup.c | 62 ++++++++++++++++++++++++++------------
include/linux/blk-cgroup.h | 1 +
2 files changed, 44 insertions(+), 19 deletions(-)
diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index 818e50b4cc7a..aed46a2c553e 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -753,6 +753,44 @@ static struct blkcg_gq *blkg_lookup_check(struct blkcg *blkcg,
return __blkg_lookup(blkcg, q, true /* update_hint */);
}
+/**
+ * blkg_conf_prep - parse and prepare for per-blkg config update
+ * @inputp: input string pointer
+ *
+ * Parse the device node prefix part, MAJ:MIN, of per-blkg config update
+ * from @input and get and return the matching gendisk. *@inputp is
+ * updated to point past the device node prefix. Returns an ERR_PTR()
+ * value on error.
+ *
+ * Use this function iff blkg_conf_prep() can't be used for some reason.
+ */
+struct gendisk *blkcg_conf_get_disk(char **inputp)
+{
+ char *input = *inputp;
+ unsigned int major, minor;
+ struct gendisk *disk;
+ int key_len, part;
+
+ if (sscanf(input, "%u:%u%n", &major, &minor, &key_len) != 2)
+ return ERR_PTR(-EINVAL);
+
+ input += key_len;
+ if (!isspace(*input))
+ return ERR_PTR(-EINVAL);
+ input = skip_spaces(input);
+
+ disk = get_gendisk(MKDEV(major, minor), &part);
+ if (!disk)
+ return ERR_PTR(-ENODEV);
+ if (part) {
+ put_disk_and_module(disk);
+ return ERR_PTR(-ENODEV);
+ }
+
+ *inputp = input;
+ return disk;
+}
+
/**
* blkg_conf_prep - parse and prepare for per-blkg config update
* @blkcg: target block cgroup
@@ -772,25 +810,11 @@ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol,
struct gendisk *disk;
struct request_queue *q;
struct blkcg_gq *blkg;
- unsigned int major, minor;
- int key_len, part, ret;
- char *body;
-
- if (sscanf(input, "%u:%u%n", &major, &minor, &key_len) != 2)
- return -EINVAL;
-
- body = input + key_len;
- if (!isspace(*body))
- return -EINVAL;
- body = skip_spaces(body);
+ int ret;
- disk = get_gendisk(MKDEV(major, minor), &part);
- if (!disk)
- return -ENODEV;
- if (part) {
- ret = -ENODEV;
- goto fail;
- }
+ disk = blkcg_conf_get_disk(&input);
+ if (IS_ERR(disk))
+ return PTR_ERR(disk);
q = disk->queue;
@@ -856,7 +880,7 @@ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol,
success:
ctx->disk = disk;
ctx->blkg = blkg;
- ctx->body = body;
+ ctx->body = input;
return 0;
fail_unlock:
diff --git a/include/linux/blk-cgroup.h b/include/linux/blk-cgroup.h
index 930631cad5cf..b0eacec11037 100644
--- a/include/linux/blk-cgroup.h
+++ b/include/linux/blk-cgroup.h
@@ -233,6 +233,7 @@ struct blkg_conf_ctx {
char *body;
};
+struct gendisk *blkcg_conf_get_disk(char **inputp);
int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol,
char *input, struct blkg_conf_ctx *ctx);
void blkg_conf_finish(struct blkg_conf_ctx *ctx);
--
2.17.1
^ permalink raw reply related
* [PATCH 05/10] block/rq_qos: implement rq_qos_ops->queue_depth_changed()
From: Tejun Heo @ 2019-07-10 20:51 UTC (permalink / raw)
To: axboe, newella, clm, josef, dennisz, lizefan, hannes
Cc: linux-kernel, kernel-team, linux-block, cgroups, Tejun Heo
In-Reply-To: <20190710205128.1316483-1-tj@kernel.org>
wbt already gets queue depth changed notification through
wbt_set_queue_depth(). Generalize it into
rq_qos_ops->queue_depth_changed() so that other rq_qos policies can
easily hook into the events too.
Signed-off-by: Tejun Heo <tj@kernel.org>
---
block/blk-rq-qos.c | 9 +++++++++
block/blk-rq-qos.h | 8 ++++++++
block/blk-settings.c | 2 +-
block/blk-wbt.c | 18 ++++++++----------
block/blk-wbt.h | 4 ----
5 files changed, 26 insertions(+), 15 deletions(-)
diff --git a/block/blk-rq-qos.c b/block/blk-rq-qos.c
index 7debcaf1ee53..fb11652348fc 100644
--- a/block/blk-rq-qos.c
+++ b/block/blk-rq-qos.c
@@ -101,6 +101,15 @@ void __rq_qos_done_bio(struct rq_qos *rqos, struct bio *bio)
} while (rqos);
}
+void __rq_qos_queue_depth_changed(struct rq_qos *rqos)
+{
+ do {
+ if (rqos->ops->queue_depth_changed)
+ rqos->ops->queue_depth_changed(rqos);
+ rqos = rqos->next;
+ } while (rqos);
+}
+
/*
* Return true, if we can't increase the depth further by scaling
*/
diff --git a/block/blk-rq-qos.h b/block/blk-rq-qos.h
index 8e426a8505b6..e15b6907b76d 100644
--- a/block/blk-rq-qos.h
+++ b/block/blk-rq-qos.h
@@ -41,6 +41,7 @@ struct rq_qos_ops {
void (*done)(struct rq_qos *, struct request *);
void (*done_bio)(struct rq_qos *, struct bio *);
void (*cleanup)(struct rq_qos *, struct bio *);
+ void (*queue_depth_changed)(struct rq_qos *);
void (*exit)(struct rq_qos *);
const struct blk_mq_debugfs_attr *debugfs_attrs;
};
@@ -138,6 +139,7 @@ void __rq_qos_throttle(struct rq_qos *rqos, struct bio *bio);
void __rq_qos_track(struct rq_qos *rqos, struct request *rq, struct bio *bio);
void __rq_qos_merge(struct rq_qos *rqos, struct request *rq, struct bio *bio);
void __rq_qos_done_bio(struct rq_qos *rqos, struct bio *bio);
+void __rq_qos_queue_depth_changed(struct rq_qos *rqos);
static inline void rq_qos_cleanup(struct request_queue *q, struct bio *bio)
{
@@ -194,6 +196,12 @@ static inline void rq_qos_merge(struct request_queue *q, struct request *rq,
__rq_qos_merge(q->rq_qos, rq, bio);
}
+static inline void rq_qos_queue_depth_changed(struct request_queue *q)
+{
+ if (q->rq_qos)
+ __rq_qos_queue_depth_changed(q->rq_qos);
+}
+
void rq_qos_exit(struct request_queue *);
#endif
diff --git a/block/blk-settings.c b/block/blk-settings.c
index 2ae348c101a0..df323ea448de 100644
--- a/block/blk-settings.c
+++ b/block/blk-settings.c
@@ -804,7 +804,7 @@ EXPORT_SYMBOL(blk_queue_update_dma_alignment);
void blk_set_queue_depth(struct request_queue *q, unsigned int depth)
{
q->queue_depth = depth;
- wbt_set_queue_depth(q, depth);
+ rq_qos_queue_depth_changed(q);
}
EXPORT_SYMBOL(blk_set_queue_depth);
diff --git a/block/blk-wbt.c b/block/blk-wbt.c
index 313f45a37e9d..8118f95a194b 100644
--- a/block/blk-wbt.c
+++ b/block/blk-wbt.c
@@ -629,15 +629,6 @@ static void wbt_requeue(struct rq_qos *rqos, struct request *rq)
}
}
-void wbt_set_queue_depth(struct request_queue *q, unsigned int depth)
-{
- struct rq_qos *rqos = wbt_rq_qos(q);
- if (rqos) {
- RQWB(rqos)->rq_depth.queue_depth = depth;
- __wbt_update_limits(RQWB(rqos));
- }
-}
-
void wbt_set_write_cache(struct request_queue *q, bool write_cache_on)
{
struct rq_qos *rqos = wbt_rq_qos(q);
@@ -689,6 +680,12 @@ static int wbt_data_dir(const struct request *rq)
return -1;
}
+static void wbt_queue_depth_changed(struct rq_qos *rqos)
+{
+ RQWB(rqos)->rq_depth.queue_depth = blk_queue_depth(rqos->q);
+ __wbt_update_limits(RQWB(rqos));
+}
+
static void wbt_exit(struct rq_qos *rqos)
{
struct rq_wb *rwb = RQWB(rqos);
@@ -811,6 +808,7 @@ static struct rq_qos_ops wbt_rqos_ops = {
.requeue = wbt_requeue,
.done = wbt_done,
.cleanup = wbt_cleanup,
+ .queue_depth_changed = wbt_queue_depth_changed,
.exit = wbt_exit,
#ifdef CONFIG_BLK_DEBUG_FS
.debugfs_attrs = wbt_debugfs_attrs,
@@ -853,7 +851,7 @@ int wbt_init(struct request_queue *q)
rwb->min_lat_nsec = wbt_default_latency_nsec(q);
- wbt_set_queue_depth(q, blk_queue_depth(q));
+ wbt_queue_depth_changed(&rwb->rqos);
wbt_set_write_cache(q, test_bit(QUEUE_FLAG_WC, &q->queue_flags));
return 0;
diff --git a/block/blk-wbt.h b/block/blk-wbt.h
index f47218d5b3b2..8e4e37660971 100644
--- a/block/blk-wbt.h
+++ b/block/blk-wbt.h
@@ -95,7 +95,6 @@ void wbt_enable_default(struct request_queue *);
u64 wbt_get_min_lat(struct request_queue *q);
void wbt_set_min_lat(struct request_queue *q, u64 val);
-void wbt_set_queue_depth(struct request_queue *, unsigned int);
void wbt_set_write_cache(struct request_queue *, bool);
u64 wbt_default_latency_nsec(struct request_queue *);
@@ -118,9 +117,6 @@ static inline void wbt_disable_default(struct request_queue *q)
static inline void wbt_enable_default(struct request_queue *q)
{
}
-static inline void wbt_set_queue_depth(struct request_queue *q, unsigned int depth)
-{
-}
static inline void wbt_set_write_cache(struct request_queue *q, bool wc)
{
}
--
2.17.1
^ permalink raw reply related
* [PATCH 04/10] block/rq_qos: add rq_qos_merge()
From: Tejun Heo @ 2019-07-10 20:51 UTC (permalink / raw)
To: axboe, newella, clm, josef, dennisz, lizefan, hannes
Cc: linux-kernel, kernel-team, linux-block, cgroups, Tejun Heo
In-Reply-To: <20190710205128.1316483-1-tj@kernel.org>
Add a merge hook for rq_qos. This will be used by io.weight.
Signed-off-by: Tejun Heo <tj@kernel.org>
---
block/blk-core.c | 4 ++++
block/blk-rq-qos.c | 9 +++++++++
block/blk-rq-qos.h | 9 +++++++++
3 files changed, 22 insertions(+)
diff --git a/block/blk-core.c b/block/blk-core.c
index 260e36a2c343..4696a89ca039 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -601,6 +601,7 @@ bool bio_attempt_back_merge(struct request *req, struct bio *bio,
return false;
trace_block_bio_backmerge(req->q, req, bio);
+ rq_qos_merge(req->q, req, bio);
if ((req->cmd_flags & REQ_FAILFAST_MASK) != ff)
blk_rq_set_mixed_merge(req);
@@ -622,6 +623,7 @@ bool bio_attempt_front_merge(struct request *req, struct bio *bio,
return false;
trace_block_bio_frontmerge(req->q, req, bio);
+ rq_qos_merge(req->q, req, bio);
if ((req->cmd_flags & REQ_FAILFAST_MASK) != ff)
blk_rq_set_mixed_merge(req);
@@ -647,6 +649,8 @@ bool bio_attempt_discard_merge(struct request_queue *q, struct request *req,
blk_rq_get_max_sectors(req, blk_rq_pos(req)))
goto no_merge;
+ rq_qos_merge(q, req, bio);
+
req->biotail->bi_next = bio;
req->biotail = bio;
req->__data_len += bio->bi_iter.bi_size;
diff --git a/block/blk-rq-qos.c b/block/blk-rq-qos.c
index 659ccb8b693f..7debcaf1ee53 100644
--- a/block/blk-rq-qos.c
+++ b/block/blk-rq-qos.c
@@ -83,6 +83,15 @@ void __rq_qos_track(struct rq_qos *rqos, struct request *rq, struct bio *bio)
} while (rqos);
}
+void __rq_qos_merge(struct rq_qos *rqos, struct request *rq, struct bio *bio)
+{
+ do {
+ if (rqos->ops->merge)
+ rqos->ops->merge(rqos, rq, bio);
+ rqos = rqos->next;
+ } while (rqos);
+}
+
void __rq_qos_done_bio(struct rq_qos *rqos, struct bio *bio)
{
do {
diff --git a/block/blk-rq-qos.h b/block/blk-rq-qos.h
index 2300e038b9fa..8e426a8505b6 100644
--- a/block/blk-rq-qos.h
+++ b/block/blk-rq-qos.h
@@ -35,6 +35,7 @@ struct rq_qos {
struct rq_qos_ops {
void (*throttle)(struct rq_qos *, struct bio *);
void (*track)(struct rq_qos *, struct request *, struct bio *);
+ void (*merge)(struct rq_qos *, struct request *, struct bio *);
void (*issue)(struct rq_qos *, struct request *);
void (*requeue)(struct rq_qos *, struct request *);
void (*done)(struct rq_qos *, struct request *);
@@ -135,6 +136,7 @@ void __rq_qos_issue(struct rq_qos *rqos, struct request *rq);
void __rq_qos_requeue(struct rq_qos *rqos, struct request *rq);
void __rq_qos_throttle(struct rq_qos *rqos, struct bio *bio);
void __rq_qos_track(struct rq_qos *rqos, struct request *rq, struct bio *bio);
+void __rq_qos_merge(struct rq_qos *rqos, struct request *rq, struct bio *bio);
void __rq_qos_done_bio(struct rq_qos *rqos, struct bio *bio);
static inline void rq_qos_cleanup(struct request_queue *q, struct bio *bio)
@@ -185,6 +187,13 @@ static inline void rq_qos_track(struct request_queue *q, struct request *rq,
__rq_qos_track(q->rq_qos, rq, bio);
}
+static inline void rq_qos_merge(struct request_queue *q, struct request *rq,
+ struct bio *bio)
+{
+ if (q->rq_qos)
+ __rq_qos_merge(q->rq_qos, rq, bio);
+}
+
void rq_qos_exit(struct request_queue *);
#endif
--
2.17.1
^ permalink raw reply related
* [PATCH 01/10] blkcg: pass @q and @blkcg into blkcg_pol_alloc_pd_fn()
From: Tejun Heo @ 2019-07-10 20:51 UTC (permalink / raw)
To: axboe, newella, clm, josef, dennisz, lizefan, hannes
Cc: linux-kernel, kernel-team, linux-block, cgroups, Tejun Heo
In-Reply-To: <20190710205128.1316483-1-tj@kernel.org>
Instead of @node, pass in @q and @blkcg so that the alloc function has
more context. This doesn't cause any behavior change and will be used
by io.weight implementation.
Signed-off-by: Tejun Heo <tj@kernel.org>
---
block/bfq-cgroup.c | 5 +++--
block/blk-cgroup.c | 6 +++---
block/blk-iolatency.c | 6 ++++--
block/blk-throttle.c | 6 ++++--
include/linux/blk-cgroup.h | 3 ++-
5 files changed, 16 insertions(+), 10 deletions(-)
diff --git a/block/bfq-cgroup.c b/block/bfq-cgroup.c
index 0f6cd688924f..e6fb537b4bfc 100644
--- a/block/bfq-cgroup.c
+++ b/block/bfq-cgroup.c
@@ -501,11 +501,12 @@ static void bfq_cpd_free(struct blkcg_policy_data *cpd)
kfree(cpd_to_bfqgd(cpd));
}
-static struct blkg_policy_data *bfq_pd_alloc(gfp_t gfp, int node)
+static struct blkg_policy_data *bfq_pd_alloc(gfp_t gfp, struct request_queue *q,
+ struct blkcg *blkcg)
{
struct bfq_group *bfqg;
- bfqg = kzalloc_node(sizeof(*bfqg), gfp, node);
+ bfqg = kzalloc_node(sizeof(*bfqg), gfp, q->node);
if (!bfqg)
return NULL;
diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index 24ed26957367..75f7f78b87b2 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -175,7 +175,7 @@ static struct blkcg_gq *blkg_alloc(struct blkcg *blkcg, struct request_queue *q,
continue;
/* alloc per-policy data and attach it to blkg */
- pd = pol->pd_alloc_fn(gfp_mask, q->node);
+ pd = pol->pd_alloc_fn(gfp_mask, q, blkcg);
if (!pd)
goto err_free;
@@ -1349,7 +1349,7 @@ int blkcg_activate_policy(struct request_queue *q,
blk_mq_freeze_queue(q);
pd_prealloc:
if (!pd_prealloc) {
- pd_prealloc = pol->pd_alloc_fn(GFP_KERNEL, q->node);
+ pd_prealloc = pol->pd_alloc_fn(GFP_KERNEL, q, &blkcg_root);
if (!pd_prealloc) {
ret = -ENOMEM;
goto out_bypass_end;
@@ -1365,7 +1365,7 @@ int blkcg_activate_policy(struct request_queue *q,
if (blkg->pd[pol->plid])
continue;
- pd = pol->pd_alloc_fn(GFP_NOWAIT | __GFP_NOWARN, q->node);
+ pd = pol->pd_alloc_fn(GFP_NOWAIT | __GFP_NOWARN, q, &blkcg_root);
if (!pd)
swap(pd, pd_prealloc);
if (!pd) {
diff --git a/block/blk-iolatency.c b/block/blk-iolatency.c
index d973c38ee4fd..e613f89b37d3 100644
--- a/block/blk-iolatency.c
+++ b/block/blk-iolatency.c
@@ -931,11 +931,13 @@ static size_t iolatency_pd_stat(struct blkg_policy_data *pd, char *buf,
}
-static struct blkg_policy_data *iolatency_pd_alloc(gfp_t gfp, int node)
+static struct blkg_policy_data *iolatency_pd_alloc(gfp_t gfp,
+ struct request_queue *q,
+ struct blkcg *blkcg)
{
struct iolatency_grp *iolat;
- iolat = kzalloc_node(sizeof(*iolat), gfp, node);
+ iolat = kzalloc_node(sizeof(*iolat), gfp, q->node);
if (!iolat)
return NULL;
iolat->stats = __alloc_percpu_gfp(sizeof(struct latency_stat),
diff --git a/block/blk-throttle.c b/block/blk-throttle.c
index 8ab6c8153223..0445c998c377 100644
--- a/block/blk-throttle.c
+++ b/block/blk-throttle.c
@@ -478,12 +478,14 @@ static void throtl_service_queue_init(struct throtl_service_queue *sq)
timer_setup(&sq->pending_timer, throtl_pending_timer_fn, 0);
}
-static struct blkg_policy_data *throtl_pd_alloc(gfp_t gfp, int node)
+static struct blkg_policy_data *throtl_pd_alloc(gfp_t gfp,
+ struct request_queue *q,
+ struct blkcg *blkcg)
{
struct throtl_grp *tg;
int rw;
- tg = kzalloc_node(sizeof(*tg), gfp, node);
+ tg = kzalloc_node(sizeof(*tg), gfp, q->node);
if (!tg)
return NULL;
diff --git a/include/linux/blk-cgroup.h b/include/linux/blk-cgroup.h
index 689a58231288..930631cad5cf 100644
--- a/include/linux/blk-cgroup.h
+++ b/include/linux/blk-cgroup.h
@@ -149,7 +149,8 @@ typedef struct blkcg_policy_data *(blkcg_pol_alloc_cpd_fn)(gfp_t gfp);
typedef void (blkcg_pol_init_cpd_fn)(struct blkcg_policy_data *cpd);
typedef void (blkcg_pol_free_cpd_fn)(struct blkcg_policy_data *cpd);
typedef void (blkcg_pol_bind_cpd_fn)(struct blkcg_policy_data *cpd);
-typedef struct blkg_policy_data *(blkcg_pol_alloc_pd_fn)(gfp_t gfp, int node);
+typedef struct blkg_policy_data *(blkcg_pol_alloc_pd_fn)(gfp_t gfp,
+ struct request_queue *q, struct blkcg *blkcg);
typedef void (blkcg_pol_init_pd_fn)(struct blkg_policy_data *pd);
typedef void (blkcg_pol_online_pd_fn)(struct blkg_policy_data *pd);
typedef void (blkcg_pol_offline_pd_fn)(struct blkg_policy_data *pd);
--
2.17.1
^ permalink raw reply related
* [PATCH 11/10] block: omit request->pre_start_time_ns if !CONFIG_BLK_CGROUP_IOCOST work-conserving porportional controller
From: Tejun Heo @ 2019-07-10 21:21 UTC (permalink / raw)
To: axboe, newella, clm, josef, dennisz, lizefan, hannes
Cc: linux-kernel, kernel-team, linux-block, cgroups
In-Reply-To: <20190710205128.1316483-1-tj@kernel.org>
From e2693136fa64d5c9dde73d2d663bde84f8326877 Mon Sep 17 00:00:00 2001
From: Tejun Heo <tj@kernel.org>
Date: Wed, 10 Jul 2019 14:18:12 -0700
request->pre_start_time is currently only used by the iocost
controller. Let's omit the field if disabled and avoid wasting space
in struct request.
Signed-off-by: Tejun Heo <tj@kernel.org>
Suggested-by: Jens Axboe <axboe@kernel.dk>
---
The git branch is updated accordingly.
Thanks.
block/blk-mq.c | 2 ++
include/linux/blkdev.h | 7 +++++++
2 files changed, 9 insertions(+)
diff --git a/block/blk-mq.c b/block/blk-mq.c
index 90b46988cc02..ce96bcd7e260 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -325,7 +325,9 @@ static struct request *blk_mq_rq_ctx_init(struct blk_mq_alloc_data *data,
RB_CLEAR_NODE(&rq->rb_node);
rq->rq_disk = NULL;
rq->part = NULL;
+#ifdef CONFIG_BLK_CGROUP_IOCOST
rq->pre_start_time_ns = pre_start_time_ns;
+#endif
if (blk_mq_need_time_stamp(rq))
rq->start_time_ns = ktime_get_ns();
else
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 4247a9bc44b7..2425af6d3f5e 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -194,8 +194,10 @@ struct request {
struct gendisk *rq_disk;
struct hd_struct *part;
+#ifdef CONFIG_BLK_CGROUP_IOCOST
/* Time that the first bio started allocating this request. */
u64 pre_start_time_ns;
+#endif
/* Time that this request was allocated for this IO. */
u64 start_time_ns;
/* Time that I/O was submitted to the device. */
@@ -635,8 +637,13 @@ bool blk_queue_flag_test_and_set(unsigned int flag, struct request_queue *q);
test_bit(QUEUE_FLAG_SCSI_PASSTHROUGH, &(q)->queue_flags)
#define blk_queue_pci_p2pdma(q) \
test_bit(QUEUE_FLAG_PCI_P2PDMA, &(q)->queue_flags)
+
+#ifdef CONFIG_BLK_CGROUP_IOCOST
#define blk_queue_rec_prestart(q) \
test_bit(QUEUE_FLAG_REC_PRESTART, &(q)->queue_flags)
+#else
+#define blk_queue_rec_prestart(q) false
+#endif
#define blk_noretry_request(rq) \
((rq)->cmd_flags & (REQ_FAILFAST_DEV|REQ_FAILFAST_TRANSPORT| \
--
2.17.1
^ permalink raw reply related
* [PATCH v3 0/8] Inline Encryption Support
From: Satya Tangirala @ 2019-07-10 22:56 UTC (permalink / raw)
To: linux-block, linux-scsi, linux-fscrypt, linux-fsdevel,
linux-f2fs-devel
Cc: Parshuram Raju Thombare, Ladvine D Almeida, Barani Muthukumaran,
Kuohong Wang, Greg Kroah-Hartman, Satya Tangirala
This patch series adds support for Inline Encryption to the block layer,
UFS, fscrypt and f2fs.
Inline Encryption hardware allows software to specify an encryption context
(an encryption key, crypto algorithm, data unit num, data unit size, etc.)
along with a data transfer request to a storage device, and the inline
encryption hardware will use that context to en/decrypt the data. The
inline encryption hardware is part of the storage device, and it
conceptually sits on the data path between system memory and the storage
device. Inline Encryption hardware has become increasingly common, and we
want to support it in the kernel.
Inline Encryption hardware implementations often function around the
concept of a limited number of "keyslots", which can hold an encryption
context each. The storage device can be directed to en/decrypt any
particular request with the encryption context stored in any particular
keyslot.
Patch 1 introduces a Keyslot Manager to efficiently manage keyslots.
The keyslot manager also functions as the interface that blk-crypto
(introduced in Path 3), will use to program keys into inline encryption
hardware. For more information on the Keyslot Manager, refer to
documentation found in block/keyslot-manager.c and linux/keyslot-manager.h.
Patch 2 introduces struct bio_crypt_ctx, and a ptr to one in struct bio,
which allows struct bio to represent an encryption context that can be
passed down the storage stack from the filesystem layer to the storage
driver.
Patch 3 introduces blk-crypto. Blk-crypto delegates crypto operations to
inline encryption hardware when available, and also contains a software
fallback to the kernel crypto API. Blk-crypto also makes it possible for
layered devices like device mapper to make use of inline encryption
hardware. Given that blk-crypto works as a software fallback, we are
considering removing file content en/decryption from fscrypt and simply
using blk-crypto in a future patch. For more details on blk-crypto, refer
to Documentation/block/blk-crypto.txt.
Patches 4-6 add support for inline encryption into the UFS driver according
to the JEDEC UFS HCI v2.1 specification. Inline encryption support for
other drivers (like eMMC) may be added in the same way - the device driver
should set up a Keyslot Manager in the device's request_queue (refer to
the UFS crypto additions in ufshcd-crypto.c and ufshcd.c for an example).
Patches 7 and 8 add support to fscrypt and f2fs, so that we have
a complete stack that can make use of inline encryption.
There have been a few patch sets addressing Inline Encryption Support in
the past. Briefly, this patch set differs from those as follows:
1) "crypto: qce: ice: Add support for Inline Crypto Engine"
is specific to certain hardware, while our patch set's Inline
Encryption support for UFS is implemented according to the JEDEC UFS
specification.
2) "scsi: ufs: UFS Host Controller crypto changes" registers inline
encryption support as a kernel crypto algorithm. Our patch views inline
encryption as being fundamentally different from a generic crypto
provider (in that inline encryption is tied to a device), and so does
not use the kernel crypto API to represent inline encryption hardware.
3) "scsi: ufs: add real time/inline crypto support to UFS HCD" requires
the device mapper to work - our patch does not.
Changes v2 => v3:
- Overhauled keyslot manager's get keyslot logic and optimized LRU.
- Block crypto en/decryption fallback now supports data unit sizes
that divide each of the bio's segment's lengths (instead of requiring
each segment's length to be the same as the data unit size).
- fscrypt master key is now keyed additionally by super_block and
ci_ctfm != NULL.
- all references of "hw encryption" are replaced by inline encryption.
- address various other review comments from Eric.
Changes v1 => v2:
- Block layer and UFS changes are split into 3 patches each.
- We now only have a ptr to a struct bio_crypt_ctx in struct bio, instead
of the struct itself.
- struct bio_crypt_ctx no longer has flags.
- blk-crypto now correctly handles the case when it fails to init
(because of insufficient memory), but kernel continues to boot.
- ufshcd-crypto now works on big endian cpus.
- Many cleanups.
Satya Tangirala (8):
block: Keyslot Manager for Inline Encryption
block: Add encryption context to struct bio
block: blk-crypto for Inline Encryption
scsi: ufs: UFS driver v2.1 spec crypto additions
scsi: ufs: UFS crypto API
scsi: ufs: Add inline encryption support to UFS
fscrypt: wire up fscrypt to use blk-crypto
f2fs: Wire up f2fs to use inline encryption via fscrypt
Documentation/block/inline-encryption.txt | 185 +++++++
block/Kconfig | 8 +
block/Makefile | 2 +
block/bio-crypt-ctx.c | 122 +++++
block/bio.c | 16 +-
block/blk-core.c | 11 +-
block/blk-crypto.c | 585 ++++++++++++++++++++++
block/blk-merge.c | 34 +-
block/bounce.c | 9 +-
block/keyslot-manager.c | 314 ++++++++++++
drivers/md/dm.c | 15 +-
drivers/scsi/ufs/Kconfig | 10 +
drivers/scsi/ufs/Makefile | 1 +
drivers/scsi/ufs/ufshcd-crypto.c | 435 ++++++++++++++++
drivers/scsi/ufs/ufshcd-crypto.h | 86 ++++
drivers/scsi/ufs/ufshcd.c | 84 +++-
drivers/scsi/ufs/ufshcd.h | 29 ++
drivers/scsi/ufs/ufshci.h | 67 ++-
fs/crypto/Kconfig | 6 +
fs/crypto/bio.c | 138 ++++-
fs/crypto/crypto.c | 4 +
fs/crypto/fscrypt_private.h | 11 +
fs/crypto/keyinfo.c | 94 +++-
fs/crypto/policy.c | 10 +
fs/f2fs/data.c | 83 ++-
fs/f2fs/super.c | 13 +-
include/linux/bio.h | 208 ++++++++
include/linux/blk-crypto.h | 40 ++
include/linux/blk_types.h | 7 +
include/linux/blkdev.h | 6 +
include/linux/fscrypt.h | 62 +++
include/linux/keyslot-manager.h | 75 +++
include/uapi/linux/fs.h | 3 +-
33 files changed, 2697 insertions(+), 76 deletions(-)
create mode 100644 Documentation/block/inline-encryption.txt
create mode 100644 block/bio-crypt-ctx.c
create mode 100644 block/blk-crypto.c
create mode 100644 block/keyslot-manager.c
create mode 100644 drivers/scsi/ufs/ufshcd-crypto.c
create mode 100644 drivers/scsi/ufs/ufshcd-crypto.h
create mode 100644 include/linux/blk-crypto.h
create mode 100644 include/linux/keyslot-manager.h
--
2.22.0.410.gd8fdbe21b5-goog
^ permalink raw reply
* [PATCH 1/8] block: Keyslot Manager for Inline Encryption
From: Satya Tangirala @ 2019-07-10 22:56 UTC (permalink / raw)
To: linux-block, linux-scsi, linux-fscrypt, linux-fsdevel,
linux-f2fs-devel
Cc: Parshuram Raju Thombare, Ladvine D Almeida, Barani Muthukumaran,
Kuohong Wang, Greg Kroah-Hartman, Satya Tangirala
In-Reply-To: <20190710225609.192252-1-satyat@google.com>
Inline Encryption hardware allows software to specify an encryption context
(an encryption key, crypto algorithm, data unit num, data unit size, etc.)
along with a data transfer request to a storage device, and the inline
encryption hardware will use that context to en/decrypt the data. The
inline encryption hardware is part of the storage device, and it
conceptually sits on the data path between system memory and the storage
device.
Inline Encryption hardware implementations often function around the
concept of "keyslots". These implementations often have a limited number
of "keyslots", each of which can hold an encryption context (we say that
an encryption context can be "programmed" into a keyslot). Requests made
to the storage device may have a keyslot associated with them, and the
inline encryption hardware will en/decrypt the data in the requests using
the encryption context programmed into that associated keyslot. As
keyslots are limited, and programming keys may be expensive in many
implementations, and multiple requests may use exactly the same encryption
contexts, we introduce a Keyslot Manager to efficiently manage keyslots.
The keyslot manager also functions as the interface that upper layers will
use to program keys into inline encryption hardware. For more information
on the Keyslot Manager, refer to documentation found in
block/keyslot-manager.c and linux/keyslot-manager.h.
Signed-off-by: Satya Tangirala <satyat@google.com>
---
block/Kconfig | 8 +
block/Makefile | 1 +
block/keyslot-manager.c | 314 ++++++++++++++++++++++++++++++++
include/linux/bio.h | 11 ++
include/linux/blkdev.h | 6 +
include/linux/keyslot-manager.h | 75 ++++++++
6 files changed, 415 insertions(+)
create mode 100644 block/keyslot-manager.c
create mode 100644 include/linux/keyslot-manager.h
diff --git a/block/Kconfig b/block/Kconfig
index 2466dcc3ef1d..fef7c60f6d8e 100644
--- a/block/Kconfig
+++ b/block/Kconfig
@@ -164,6 +164,14 @@ config BLK_SED_OPAL
Enabling this option enables users to setup/unlock/lock
Locking ranges for SED devices using the Opal protocol.
+config BLK_INLINE_ENCRYPTION
+ bool "Enable inline encryption support in block layer"
+ help
+ Build the blk-crypto subsystem.
+ Enabling this lets the block layer handle encryption,
+ so users can take advantage of inline encryption
+ hardware if present.
+
menu "Partition Types"
source "block/partitions/Kconfig"
diff --git a/block/Makefile b/block/Makefile
index eee1b4ceecf9..a72abd61b220 100644
--- a/block/Makefile
+++ b/block/Makefile
@@ -35,3 +35,4 @@ obj-$(CONFIG_BLK_DEBUG_FS) += blk-mq-debugfs.o
obj-$(CONFIG_BLK_DEBUG_FS_ZONED)+= blk-mq-debugfs-zoned.o
obj-$(CONFIG_BLK_SED_OPAL) += sed-opal.o
obj-$(CONFIG_BLK_PM) += blk-pm.o
+obj-$(CONFIG_BLK_INLINE_ENCRYPTION) += keyslot-manager.o
diff --git a/block/keyslot-manager.c b/block/keyslot-manager.c
new file mode 100644
index 000000000000..22afd00c9235
--- /dev/null
+++ b/block/keyslot-manager.c
@@ -0,0 +1,314 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * keyslot-manager.c
+ *
+ * Copyright 2019 Google LLC
+ */
+
+/**
+ * DOC: The Keyslot Manager
+ *
+ * Many devices with inline encryption support have a limited number of "slots"
+ * into which encryption contexts may be programmed, and requests can be tagged
+ * with a slot number to specify the key to use for en/decryption.
+ *
+ * As the number of slots are limited, and programming keys is expensive on
+ * many inline encryption hardware, we don't want to program the same key into
+ * multiple slots - if multiple requests are using the same key, we want to
+ * program just one slot with that key and use that slot for all requests.
+ *
+ * The keyslot manager manages these keyslots appropriately, and also acts as
+ * an abstraction between the inline encryption hardware and the upper layers.
+ *
+ * Lower layer devices will set up a keyslot manager in their request queue
+ * and tell it how to perform device specific operations like programming/
+ * evicting keys from keyslots.
+ *
+ * Upper layers will call keyslot_manager_get_slot_for_key() to program a
+ * key into some slot in the inline encryption hardware.
+ */
+#include <linux/keyslot-manager.h>
+#include <linux/atomic.h>
+#include <linux/mutex.h>
+#include <linux/wait.h>
+
+struct keyslot {
+ atomic_t slot_refs;
+ struct list_head idle_slot_node;
+};
+
+struct keyslot_manager {
+ unsigned int num_slots;
+ atomic_t num_idle_slots;
+ struct keyslot_mgmt_ll_ops ksm_ll_ops;
+ void *ll_priv_data;
+
+ /* Protects programming and evicting keys from the device */
+ struct rw_semaphore lock;
+
+ /* List of idle slots, with least recently used slot at front */
+ wait_queue_head_t idle_slots_wait_queue;
+ struct list_head idle_slots;
+ spinlock_t idle_slots_lock;
+
+ /* Per-keyslot data */
+ struct keyslot slots[];
+};
+
+/**
+ * keyslot_manager_create() - Create a keyslot manager
+ * @num_slots: The number of key slots to manage.
+ * @ksm_ll_ops: The struct keyslot_mgmt_ll_ops for the device that this keyslot
+ * manager will use to perform operations like programming and
+ * evicting keys.
+ * @ll_priv_data: Private data passed as is to the functions in ksm_ll_ops.
+ *
+ * Allocate memory for and initialize a keyslot manager. Called by for e.g.
+ * storage drivers to set up a keyslot manager in their request_queue.
+ *
+ * Context: May sleep
+ * Return: Pointer to constructed keyslot manager or NULL on error.
+ */
+struct keyslot_manager *keyslot_manager_create(unsigned int num_slots,
+ const struct keyslot_mgmt_ll_ops *ksm_ll_ops,
+ void *ll_priv_data)
+{
+ struct keyslot_manager *ksm;
+ int slot;
+
+ if (num_slots == 0)
+ return NULL;
+
+ /* Check that all ops are specified */
+ if (ksm_ll_ops->keyslot_program == NULL ||
+ ksm_ll_ops->keyslot_evict == NULL ||
+ ksm_ll_ops->crypt_mode_supported == NULL ||
+ ksm_ll_ops->keyslot_find == NULL)
+ return NULL;
+
+ ksm = kvzalloc(struct_size(ksm, slots, num_slots), GFP_KERNEL);
+ if (!ksm)
+ return NULL;
+
+ ksm->num_slots = num_slots;
+ atomic_set(&ksm->num_idle_slots, num_slots);
+ ksm->ksm_ll_ops = *ksm_ll_ops;
+ ksm->ll_priv_data = ll_priv_data;
+
+ init_rwsem(&ksm->lock);
+
+ init_waitqueue_head(&ksm->idle_slots_wait_queue);
+ INIT_LIST_HEAD(&ksm->idle_slots);
+
+ for (slot = 0; slot < num_slots; slot++)
+ list_add(&ksm->slots[slot].idle_slot_node, &ksm->idle_slots);
+
+ spin_lock_init(&ksm->idle_slots_lock);
+
+ return ksm;
+}
+EXPORT_SYMBOL(keyslot_manager_create);
+
+static int find_and_grab_keyslot(struct keyslot_manager *ksm, const u8 *key,
+ enum blk_crypt_mode_num crypt_mode,
+ unsigned int data_unit_size)
+{
+ int slot;
+ unsigned long flags;
+
+ slot = ksm->ksm_ll_ops.keyslot_find(ksm->ll_priv_data, key,
+ crypt_mode, data_unit_size);
+ if (slot < 0)
+ return slot;
+ if (WARN_ON(slot >= ksm->num_slots))
+ return -EINVAL;
+ if (atomic_inc_return(&ksm->slots[slot].slot_refs) == 1) {
+ /* Took first reference to this slot; remove it from LRU list */
+ spin_lock_irqsave(&ksm->idle_slots_lock, flags);
+ list_del(&ksm->slots[slot].idle_slot_node);
+ spin_unlock_irqrestore(&ksm->idle_slots_lock, flags);
+ atomic_dec(&ksm->num_idle_slots);
+ }
+ return slot;
+}
+
+/**
+ * keyslot_manager_get_slot_for_key() - Program a key into a keyslot.
+ * @ksm: The keyslot manager to program the key into.
+ * @key: Pointer to the bytes of the key to program. Must be the correct length
+ * for the chosen @crypt_mode; see blk_crypt_modes in blk-crypto.c.
+ * @crypt_mode: Identifier for the encryption algorithm to use.
+ * @data_unit_size: The data unit size to use for en/decryption.
+ *
+ * Get a keyslot that's been programmed with the specified key, crypt_mode, and
+ * data_unit_size. If one already exists, return it with incremented refcount.
+ * Otherwise, wait for a keyslot to become idle and program it.
+ *
+ * Context: Process context. Takes and releases ksm->lock.
+ * Return: The keyslot on success, else a -errno value.
+ */
+int keyslot_manager_get_slot_for_key(struct keyslot_manager *ksm,
+ const u8 *key,
+ enum blk_crypt_mode_num crypt_mode,
+ unsigned int data_unit_size)
+{
+ int slot;
+ int err;
+ struct keyslot *idle_slot;
+ unsigned long flags;
+
+ down_read(&ksm->lock);
+ slot = find_and_grab_keyslot(ksm, key, crypt_mode, data_unit_size);
+ up_read(&ksm->lock);
+ if (slot != -ENOKEY)
+ return slot;
+
+ while (true) {
+ down_write(&ksm->lock);
+ slot = find_and_grab_keyslot(ksm, key, crypt_mode,
+ data_unit_size);
+ if (slot != -ENOKEY) {
+ up_write(&ksm->lock);
+ return slot;
+ }
+
+ /*
+ * If we're here, that means there wasn't a slot that was
+ * already programmed with the key. So try to program it.
+ */
+ if (atomic_read(&ksm->num_idle_slots) > 0)
+ break;
+
+ up_write(&ksm->lock);
+ wait_event(ksm->idle_slots_wait_queue,
+ (atomic_read(&ksm->num_idle_slots) > 0));
+ }
+
+ idle_slot = list_first_entry(&ksm->idle_slots, struct keyslot,
+ idle_slot_node);
+ slot = idle_slot - ksm->slots;
+
+ err = ksm->ksm_ll_ops.keyslot_program(ksm->ll_priv_data, key,
+ crypt_mode,
+ data_unit_size,
+ slot);
+
+ if (err) {
+ wake_up(&ksm->idle_slots_wait_queue);
+ up_write(&ksm->lock);
+ return err;
+ }
+
+ atomic_inc(&ksm->slots[slot].slot_refs);
+ spin_lock_irqsave(&ksm->idle_slots_lock, flags);
+ list_del(&idle_slot->idle_slot_node);
+ spin_unlock_irqrestore(&ksm->idle_slots_lock, flags);
+ atomic_dec(&ksm->num_idle_slots);
+
+ up_write(&ksm->lock);
+ return slot;
+
+}
+EXPORT_SYMBOL(keyslot_manager_get_slot_for_key);
+
+/**
+ * keyslot_manager_get_slot() - Increment the refcount on the specified slot.
+ * @ksm - The keyslot manager that we want to modify.
+ * @slot - The slot to increment the refcount of.
+ *
+ * This function assumes that there is already an active reference to that slot
+ * and simply increments the refcount. This is useful when cloning a bio that
+ * already has a reference to a keyslot, and we want the cloned bio to also have
+ * its own reference.
+ *
+ * Context: Any context.
+ */
+void keyslot_manager_get_slot(struct keyslot_manager *ksm, unsigned int slot)
+{
+ if (WARN_ON(slot >= ksm->num_slots))
+ return;
+
+ WARN_ON(atomic_inc_return(&ksm->slots[slot].slot_refs) < 2);
+}
+EXPORT_SYMBOL(keyslot_manager_get_slot);
+
+/**
+ * keyslot_manager_put_slot() - Release a reference to a slot
+ * @ksm: The keyslot manager to release the reference from.
+ * @slot: The slot to release the reference from.
+ *
+ * Context: Any context.
+ */
+void keyslot_manager_put_slot(struct keyslot_manager *ksm, unsigned int slot)
+{
+ unsigned long flags;
+
+ if (WARN_ON(slot >= ksm->num_slots))
+ return;
+
+ spin_lock_irqsave(&ksm->idle_slots_lock, flags);
+ if (atomic_dec_and_test(&ksm->slots[slot].slot_refs)) {
+ list_add_tail(&ksm->slots[slot].idle_slot_node,
+ &ksm->idle_slots);
+ spin_unlock_irqrestore(&ksm->idle_slots_lock, flags);
+ atomic_inc(&ksm->num_idle_slots);
+ wake_up(&ksm->idle_slots_wait_queue);
+ } else {
+ spin_unlock_irqrestore(&ksm->idle_slots_lock, flags);
+ }
+}
+EXPORT_SYMBOL(keyslot_manager_put_slot);
+
+/**
+ * keyslot_manager_evict_key() - Evict a key from the lower layer device.
+ * @ksm - The keyslot manager to evict from
+ * @key - The key to evict
+ * @crypt_mode - The crypto algorithm the key was programmed with.
+ * @data_unit_size - The data_unit_size the key was programmed with.
+ *
+ * Finds the slot that the specified key, crypt_mode, data_unit_size combo
+ * was programmed into, and evicts that slot from the lower layer device if
+ * the refcount on the slot is 0. Returns -EBUSY if the refcount is not 0, and
+ * -errno on error.
+ *
+ * Context: Process context. Takes and releases ksm->lock.
+ */
+int keyslot_manager_evict_key(struct keyslot_manager *ksm,
+ const u8 *key,
+ enum blk_crypt_mode_num crypt_mode,
+ unsigned int data_unit_size)
+{
+ int slot;
+ int err = 0;
+
+ down_write(&ksm->lock);
+ slot = ksm->ksm_ll_ops.keyslot_find(ksm->ll_priv_data, key,
+ crypt_mode,
+ data_unit_size);
+
+ if (slot < 0) {
+ up_write(&ksm->lock);
+ return slot;
+ }
+
+ if (atomic_read(&ksm->slots[slot].slot_refs) == 0) {
+ err = ksm->ksm_ll_ops.keyslot_evict(ksm->ll_priv_data, key,
+ crypt_mode,
+ data_unit_size,
+ slot);
+ } else {
+ err = -EBUSY;
+ }
+
+ up_write(&ksm->lock);
+ return err;
+}
+EXPORT_SYMBOL(keyslot_manager_evict_key);
+
+void keyslot_manager_destroy(struct keyslot_manager *ksm)
+{
+ if (!ksm)
+ return;
+ kvfree(ksm);
+}
+EXPORT_SYMBOL(keyslot_manager_destroy);
diff --git a/include/linux/bio.h b/include/linux/bio.h
index f87abaa898f0..ef9c6e2e92bc 100644
--- a/include/linux/bio.h
+++ b/include/linux/bio.h
@@ -561,6 +561,17 @@ static inline void bvec_kunmap_irq(char *buffer, unsigned long *flags)
}
#endif
+enum blk_crypt_mode_num {
+ BLK_ENCRYPTION_MODE_AES_256_XTS = 0,
+ /*
+ * TODO: Support these too
+ * BLK_ENCRYPTION_MODE_AES_256_CTS = 1,
+ * BLK_ENCRYPTION_MODE_AES_128_CBC = 2,
+ * BLK_ENCRYPTION_MODE_AES_128_CTS = 3,
+ * BLK_ENCRYPTION_MODE_ADIANTUM = 4,
+ */
+};
+
/*
* BIO list management for use by remapping drivers (e.g. DM or MD) and loop.
*
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 592669bcc536..3bb4e7e1bd78 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -43,6 +43,7 @@ struct pr_ops;
struct rq_qos;
struct blk_queue_stats;
struct blk_stat_callback;
+struct keyslot_manager;
#define BLKDEV_MIN_RQ 4
#define BLKDEV_MAX_RQ 128 /* Default maximum */
@@ -473,6 +474,11 @@ struct request_queue {
unsigned int dma_pad_mask;
unsigned int dma_alignment;
+#ifdef CONFIG_BLK_INLINE_ENCRYPTION
+ /* Inline crypto capabilities */
+ struct keyslot_manager *ksm;
+#endif
+
unsigned int rq_timeout;
int poll_nsec;
diff --git a/include/linux/keyslot-manager.h b/include/linux/keyslot-manager.h
new file mode 100644
index 000000000000..ee97692997c8
--- /dev/null
+++ b/include/linux/keyslot-manager.h
@@ -0,0 +1,75 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright 2019 Google LLC
+ */
+
+#include <linux/bio.h>
+
+#ifndef __LINUX_KEYSLOT_MANAGER_H
+#define __LINUX_KEYSLOT_MANAGER_H
+
+/**
+ * struct keyslot_mgmt_ll_ops - functions to manage keyslots in hardware
+ * @keyslot_program: Program the specified key and algorithm into the
+ * specified slot in the inline encryption hardware.
+ * @keyslot_evict: Evict key from the specified keyslot in the hardware.
+ * The key, crypt_mode and data_unit_size are also passed
+ * down so that for e.g. dm layers can evict keys from
+ * the devices that they map over.
+ * Returns 0 on success, -errno otherwise.
+ * @crypt_mode_supported: Check whether a crypt_mode and data_unit_size
+ * combo is supported.
+ * @keyslot_find: Returns the slot number that matches the key,
+ * or -ENOKEY if no match found, or -errno on
+ * error.
+ *
+ * This structure should be provided by storage device drivers when they set up
+ * a keyslot manager - this structure holds the function ptrs that the keyslot
+ * manager will use to manipulate keyslots in the hardware.
+ */
+struct keyslot_mgmt_ll_ops {
+ int (*keyslot_program)(void *ll_priv_data, const u8 *key,
+ enum blk_crypt_mode_num crypt_mode,
+ unsigned int data_unit_size,
+ unsigned int slot);
+ int (*keyslot_evict)(void *ll_priv_data, const u8 *key,
+ enum blk_crypt_mode_num crypt_mode,
+ unsigned int data_unit_size,
+ unsigned int slot);
+ bool (*crypt_mode_supported)(void *ll_priv_data,
+ enum blk_crypt_mode_num crypt_mode,
+ unsigned int data_unit_size);
+ int (*keyslot_find)(void *ll_priv_data, const u8 *key,
+ enum blk_crypt_mode_num crypt_mode,
+ unsigned int data_unit_size);
+};
+
+#ifdef CONFIG_BLK_INLINE_ENCRYPTION
+struct keyslot_manager;
+
+extern struct keyslot_manager *keyslot_manager_create(unsigned int num_slots,
+ const struct keyslot_mgmt_ll_ops *ksm_ops,
+ void *ll_priv_data);
+
+extern int
+keyslot_manager_get_slot_for_key(struct keyslot_manager *ksm,
+ const u8 *key,
+ enum blk_crypt_mode_num crypt_mode,
+ unsigned int data_unit_size);
+
+extern void keyslot_manager_get_slot(struct keyslot_manager *ksm,
+ unsigned int slot);
+
+extern void keyslot_manager_put_slot(struct keyslot_manager *ksm,
+ unsigned int slot);
+
+extern int keyslot_manager_evict_key(struct keyslot_manager *ksm,
+ const u8 *key,
+ enum blk_crypt_mode_num crypt_mode,
+ unsigned int data_unit_size);
+
+extern void keyslot_manager_destroy(struct keyslot_manager *ksm);
+
+#endif /* CONFIG_BLK_INLINE_ENCRYPTION */
+
+#endif /* __LINUX_KEYSLOT_MANAGER_H */
--
2.22.0.410.gd8fdbe21b5-goog
^ permalink raw reply related
* [PATCH 2/8] block: Add encryption context to struct bio
From: Satya Tangirala @ 2019-07-10 22:56 UTC (permalink / raw)
To: linux-block, linux-scsi, linux-fscrypt, linux-fsdevel,
linux-f2fs-devel
Cc: Parshuram Raju Thombare, Ladvine D Almeida, Barani Muthukumaran,
Kuohong Wang, Greg Kroah-Hartman, Satya Tangirala
In-Reply-To: <20190710225609.192252-1-satyat@google.com>
We must have some way of letting a storage device driver know what
encryption context it should use for en/decrypting a request. However,
it's the filesystem/fscrypt that knows about and manages encryption
contexts. As such, when the filesystem layer submits a bio to the block
layer, and this bio eventually reaches a device driver with support for
inline encryption, the device driver will need to have been told the
encryption context for that bio.
We want to communicate the encryption context from the filesystem layer
to the storage device along with the bio, when the bio is submitted to the
block layer. To do this, we add a struct bio_crypt_ctx to struct bio, which
can represent an encryption context (note that we can't use the bi_private
field in struct bio to do this because that field does not function to pass
information across layers in the storage stack). We also introduce various
functions to manipulate the bio_crypt_ctx and make the bio/request merging
logic aware of the bio_crypt_ctx.
Signed-off-by: Satya Tangirala <satyat@google.com>
---
block/Makefile | 2 +-
block/bio-crypt-ctx.c | 117 +++++++++++++++++++++++
block/bio.c | 11 ++-
block/blk-merge.c | 34 ++++++-
block/bounce.c | 9 +-
drivers/md/dm.c | 15 ++-
include/linux/bio.h | 190 ++++++++++++++++++++++++++++++++++++++
include/linux/blk_types.h | 7 ++
8 files changed, 371 insertions(+), 14 deletions(-)
create mode 100644 block/bio-crypt-ctx.c
diff --git a/block/Makefile b/block/Makefile
index a72abd61b220..4147ffa63631 100644
--- a/block/Makefile
+++ b/block/Makefile
@@ -35,4 +35,4 @@ obj-$(CONFIG_BLK_DEBUG_FS) += blk-mq-debugfs.o
obj-$(CONFIG_BLK_DEBUG_FS_ZONED)+= blk-mq-debugfs-zoned.o
obj-$(CONFIG_BLK_SED_OPAL) += sed-opal.o
obj-$(CONFIG_BLK_PM) += blk-pm.o
-obj-$(CONFIG_BLK_INLINE_ENCRYPTION) += keyslot-manager.o
+obj-$(CONFIG_BLK_INLINE_ENCRYPTION) += keyslot-manager.o bio-crypt-ctx.o
diff --git a/block/bio-crypt-ctx.c b/block/bio-crypt-ctx.c
new file mode 100644
index 000000000000..8b884ef32007
--- /dev/null
+++ b/block/bio-crypt-ctx.c
@@ -0,0 +1,117 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright 2019 Google LLC
+ */
+
+#include <linux/bio.h>
+#include <linux/blkdev.h>
+#include <linux/slab.h>
+#include <linux/keyslot-manager.h>
+
+struct bio_crypt_ctx *bio_crypt_alloc_ctx(gfp_t gfp_mask)
+{
+ return kzalloc(sizeof(struct bio_crypt_ctx), gfp_mask);
+}
+EXPORT_SYMBOL(bio_crypt_alloc_ctx);
+
+void bio_crypt_free_ctx(struct bio *bio)
+{
+ kzfree(bio->bi_crypt_context);
+ bio->bi_crypt_context = NULL;
+}
+EXPORT_SYMBOL(bio_crypt_free_ctx);
+
+int bio_crypt_clone(struct bio *dst, struct bio *src, gfp_t gfp_mask)
+{
+ if (!bio_is_encrypted(src))
+ return 0;
+
+ dst->bi_crypt_context = bio_crypt_alloc_ctx(gfp_mask);
+ if (!dst->bi_crypt_context)
+ return -ENOMEM;
+
+ *dst->bi_crypt_context = *src->bi_crypt_context;
+
+ if (bio_crypt_has_keyslot(src))
+ keyslot_manager_get_slot(src->bi_crypt_context->processing_ksm,
+ src->bi_crypt_context->keyslot);
+
+ return 0;
+}
+EXPORT_SYMBOL(bio_crypt_clone);
+
+bool bio_crypt_should_process(struct bio *bio, struct request_queue *q)
+{
+ if (!bio_is_encrypted(bio))
+ return false;
+
+ WARN_ON(!bio_crypt_has_keyslot(bio));
+ return q->ksm == bio->bi_crypt_context->processing_ksm;
+}
+EXPORT_SYMBOL(bio_crypt_should_process);
+
+/*
+ * Checks that two bio crypt contexts are compatible - i.e. that
+ * they are mergeable except for data_unit_num continuity.
+ */
+bool bio_crypt_ctx_compatible(struct bio *b_1, struct bio *b_2)
+{
+ struct bio_crypt_ctx *bc1 = b_1->bi_crypt_context;
+ struct bio_crypt_ctx *bc2 = b_2->bi_crypt_context;
+
+ if (bio_is_encrypted(b_1) != bio_is_encrypted(b_2))
+ return false;
+
+ if (!bio_is_encrypted(b_1))
+ return true;
+
+ return bc1->keyslot == bc2->keyslot &&
+ bc1->data_unit_size_bits == bc2->data_unit_size_bits;
+}
+
+/*
+ * Checks that two bio crypt contexts are compatible, and also
+ * that their data_unit_nums are continuous (and can hence be merged)
+ */
+bool bio_crypt_ctx_back_mergeable(struct bio *b_1,
+ unsigned int b1_sectors,
+ struct bio *b_2)
+{
+ struct bio_crypt_ctx *bc1 = b_1->bi_crypt_context;
+ struct bio_crypt_ctx *bc2 = b_2->bi_crypt_context;
+
+ if (!bio_crypt_ctx_compatible(b_1, b_2))
+ return false;
+
+ return !bio_is_encrypted(b_1) ||
+ (bc1->data_unit_num +
+ (b1_sectors >> (bc1->data_unit_size_bits - 9)) ==
+ bc2->data_unit_num);
+}
+
+void bio_crypt_ctx_release_keyslot(struct bio *bio)
+{
+ struct bio_crypt_ctx *crypt_ctx = bio->bi_crypt_context;
+
+ keyslot_manager_put_slot(crypt_ctx->processing_ksm, crypt_ctx->keyslot);
+ bio->bi_crypt_context->processing_ksm = NULL;
+ bio->bi_crypt_context->keyslot = -1;
+}
+
+int bio_crypt_ctx_acquire_keyslot(struct bio *bio, struct keyslot_manager *ksm)
+{
+ int slot;
+ enum blk_crypt_mode_num crypt_mode = bio_crypt_mode(bio);
+
+ if (!ksm)
+ return -ENOMEM;
+
+ slot = keyslot_manager_get_slot_for_key(ksm,
+ bio_crypt_raw_key(bio), crypt_mode,
+ 1 << bio->bi_crypt_context->data_unit_size_bits);
+ if (slot < 0)
+ return slot;
+
+ bio_crypt_set_keyslot(bio, slot, ksm);
+ return 0;
+}
diff --git a/block/bio.c b/block/bio.c
index ce797d73bb43..2a272cda6dfa 100644
--- a/block/bio.c
+++ b/block/bio.c
@@ -240,6 +240,7 @@ static void bio_free(struct bio *bio)
struct bio_set *bs = bio->bi_pool;
void *p;
+ bio_crypt_free_ctx(bio);
bio_uninit(bio);
if (bs) {
@@ -612,6 +613,7 @@ EXPORT_SYMBOL(__bio_clone_fast);
struct bio *bio_clone_fast(struct bio *bio, gfp_t gfp_mask, struct bio_set *bs)
{
struct bio *b;
+ int ret;
b = bio_alloc_bioset(gfp_mask, 0, bs);
if (!b)
@@ -619,9 +621,13 @@ struct bio *bio_clone_fast(struct bio *bio, gfp_t gfp_mask, struct bio_set *bs)
__bio_clone_fast(b, bio);
- if (bio_integrity(bio)) {
- int ret;
+ ret = bio_crypt_clone(b, bio, gfp_mask);
+ if (ret < 0) {
+ bio_put(b);
+ return NULL;
+ }
+ if (bio_integrity(bio)) {
ret = bio_integrity_clone(b, bio, gfp_mask);
if (ret < 0) {
@@ -1022,6 +1028,7 @@ void bio_advance(struct bio *bio, unsigned bytes)
if (bio_integrity(bio))
bio_integrity_advance(bio, bytes);
+ bio_crypt_advance(bio, bytes);
bio_advance_iter(bio, &bio->bi_iter, bytes);
}
EXPORT_SYMBOL(bio_advance);
diff --git a/block/blk-merge.c b/block/blk-merge.c
index 17713d7d98d5..f416e7f38270 100644
--- a/block/blk-merge.c
+++ b/block/blk-merge.c
@@ -531,6 +531,9 @@ static inline int ll_new_hw_segment(struct request_queue *q,
if (blk_integrity_merge_bio(q, req, bio) == false)
goto no_merge;
+ if (WARN_ON(!bio_crypt_ctx_compatible(bio, req->bio)))
+ goto no_merge;
+
/*
* This will form the start of a new hw segment. Bump both
* counters.
@@ -696,8 +699,13 @@ static enum elv_merge blk_try_req_merge(struct request *req,
{
if (blk_discard_mergable(req))
return ELEVATOR_DISCARD_MERGE;
- else if (blk_rq_pos(req) + blk_rq_sectors(req) == blk_rq_pos(next))
+ else if (blk_rq_pos(req) + blk_rq_sectors(req) == blk_rq_pos(next)) {
+ if (!bio_crypt_ctx_back_mergeable(
+ req->bio, blk_rq_sectors(req), next->bio)) {
+ return ELEVATOR_NO_MERGE;
+ }
return ELEVATOR_BACK_MERGE;
+ }
return ELEVATOR_NO_MERGE;
}
@@ -733,6 +741,9 @@ static struct request *attempt_merge(struct request_queue *q,
if (req->ioprio != next->ioprio)
return NULL;
+ if (!bio_crypt_ctx_compatible(req->bio, next->bio))
+ return NULL;
+
/*
* If we are allowed to merge, then append bio list
* from next to rq and release next. merge_requests_fn
@@ -865,16 +876,31 @@ bool blk_rq_merge_ok(struct request *rq, struct bio *bio)
if (rq->ioprio != bio_prio(bio))
return false;
+ /* Only merge if the crypt contexts are compatible */
+ if (!bio_crypt_ctx_compatible(bio, rq->bio))
+ return false;
+
return true;
}
enum elv_merge blk_try_merge(struct request *rq, struct bio *bio)
{
- if (blk_discard_mergable(rq))
+ if (blk_discard_mergable(rq)) {
return ELEVATOR_DISCARD_MERGE;
- else if (blk_rq_pos(rq) + blk_rq_sectors(rq) == bio->bi_iter.bi_sector)
+ } else if (blk_rq_pos(rq) + blk_rq_sectors(rq) ==
+ bio->bi_iter.bi_sector) {
+ if (!bio_crypt_ctx_back_mergeable(rq->bio,
+ blk_rq_sectors(rq), bio)) {
+ return ELEVATOR_NO_MERGE;
+ }
return ELEVATOR_BACK_MERGE;
- else if (blk_rq_pos(rq) - bio_sectors(bio) == bio->bi_iter.bi_sector)
+ } else if (blk_rq_pos(rq) - bio_sectors(bio) ==
+ bio->bi_iter.bi_sector) {
+ if (!bio_crypt_ctx_back_mergeable(bio,
+ bio_sectors(bio), rq->bio)) {
+ return ELEVATOR_NO_MERGE;
+ }
return ELEVATOR_FRONT_MERGE;
+ }
return ELEVATOR_NO_MERGE;
}
diff --git a/block/bounce.c b/block/bounce.c
index f8ed677a1bf7..1307238d3951 100644
--- a/block/bounce.c
+++ b/block/bounce.c
@@ -220,6 +220,7 @@ static struct bio *bounce_clone_bio(struct bio *bio_src, gfp_t gfp_mask,
struct bvec_iter iter;
struct bio_vec bv;
struct bio *bio;
+ int ret;
/*
* Pre immutable biovecs, __bio_clone() used to just do a memcpy from
@@ -267,9 +268,13 @@ static struct bio *bounce_clone_bio(struct bio *bio_src, gfp_t gfp_mask,
break;
}
- if (bio_integrity(bio_src)) {
- int ret;
+ ret = bio_crypt_clone(bio, bio_src, gfp_mask);
+ if (ret < 0) {
+ bio_put(bio);
+ return NULL;
+ }
+ if (bio_integrity(bio_src)) {
ret = bio_integrity_clone(bio, bio_src, gfp_mask);
if (ret < 0) {
bio_put(bio);
diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index 5475081dcbd6..22517d03aa39 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -1324,12 +1324,15 @@ static int clone_bio(struct dm_target_io *tio, struct bio *bio,
sector_t sector, unsigned len)
{
struct bio *clone = &tio->clone;
+ int ret;
__bio_clone_fast(clone, bio);
- if (bio_integrity(bio)) {
- int r;
+ ret = bio_crypt_clone(clone, bio, GFP_NOIO);
+ if (ret < 0)
+ return ret;
+ if (bio_integrity(bio)) {
if (unlikely(!dm_target_has_integrity(tio->ti->type) &&
!dm_target_passes_integrity(tio->ti->type))) {
DMWARN("%s: the target %s doesn't support integrity data.",
@@ -1338,9 +1341,11 @@ static int clone_bio(struct dm_target_io *tio, struct bio *bio,
return -EIO;
}
- r = bio_integrity_clone(clone, bio, GFP_NOIO);
- if (r < 0)
- return r;
+ ret = bio_integrity_clone(clone, bio, GFP_NOIO);
+ if (ret < 0) {
+ bio_crypt_free_ctx(clone);
+ return ret;
+ }
}
bio_advance(clone, to_bytes(sector - clone->bi_iter.bi_sector));
diff --git a/include/linux/bio.h b/include/linux/bio.h
index ef9c6e2e92bc..4e664d6441d5 100644
--- a/include/linux/bio.h
+++ b/include/linux/bio.h
@@ -572,6 +572,196 @@ enum blk_crypt_mode_num {
*/
};
+#ifdef CONFIG_BLK_INLINE_ENCRYPTION
+struct bio_crypt_ctx {
+ int keyslot;
+ u8 *raw_key;
+ enum blk_crypt_mode_num crypt_mode;
+ u64 data_unit_num;
+ unsigned int data_unit_size_bits;
+
+ /*
+ * The keyslot manager where the key has been programmed
+ * with keyslot.
+ */
+ struct keyslot_manager *processing_ksm;
+
+ /*
+ * Copy of the bvec_iter when this bio was submitted.
+ * We only want to en/decrypt the part of the bio
+ * as described by the bvec_iter upon submission because
+ * bio might be split before being resubmitted
+ */
+ struct bvec_iter crypt_iter;
+ u64 sw_data_unit_num;
+};
+
+extern int bio_crypt_clone(struct bio *dst, struct bio *src,
+ gfp_t gfp_mask);
+
+static inline bool bio_is_encrypted(struct bio *bio)
+{
+ return bio->bi_crypt_context;
+}
+
+static inline void bio_crypt_advance(struct bio *bio, unsigned int bytes)
+{
+ if (bio_is_encrypted(bio)) {
+ bio->bi_crypt_context->data_unit_num +=
+ bytes >> bio->bi_crypt_context->data_unit_size_bits;
+ }
+}
+
+static inline bool bio_crypt_has_keyslot(struct bio *bio)
+{
+ return bio->bi_crypt_context->keyslot >= 0;
+}
+
+extern struct bio_crypt_ctx *bio_crypt_alloc_ctx(gfp_t gfp_mask);
+
+extern void bio_crypt_free_ctx(struct bio *bio);
+
+static inline int bio_crypt_set_ctx(struct bio *bio,
+ u8 *raw_key,
+ enum blk_crypt_mode_num crypt_mode,
+ u64 dun,
+ unsigned int dun_bits)
+{
+ struct bio_crypt_ctx *crypt_ctx;
+
+ crypt_ctx = bio_crypt_alloc_ctx(GFP_NOIO);
+ if (!crypt_ctx)
+ return -ENOMEM;
+
+ crypt_ctx->raw_key = raw_key;
+ crypt_ctx->data_unit_num = dun;
+ crypt_ctx->data_unit_size_bits = dun_bits;
+ crypt_ctx->crypt_mode = crypt_mode;
+ crypt_ctx->processing_ksm = NULL;
+ crypt_ctx->keyslot = -1;
+ bio->bi_crypt_context = crypt_ctx;
+
+ return 0;
+}
+
+static inline int bio_crypt_get_keyslot(struct bio *bio)
+{
+ return bio->bi_crypt_context->keyslot;
+}
+
+static inline void bio_crypt_set_keyslot(struct bio *bio,
+ unsigned int keyslot,
+ struct keyslot_manager *ksm)
+{
+ bio->bi_crypt_context->keyslot = keyslot;
+ bio->bi_crypt_context->processing_ksm = ksm;
+}
+
+extern void bio_crypt_ctx_release_keyslot(struct bio *bio);
+
+extern int bio_crypt_ctx_acquire_keyslot(struct bio *bio,
+ struct keyslot_manager *ksm);
+
+static inline u8 *bio_crypt_raw_key(struct bio *bio)
+{
+ return bio->bi_crypt_context->raw_key;
+}
+
+static inline enum blk_crypt_mode_num bio_crypt_mode(struct bio *bio)
+{
+ return bio->bi_crypt_context->crypt_mode;
+}
+
+static inline u64 bio_crypt_data_unit_num(struct bio *bio)
+{
+ WARN_ON(!bio_is_encrypted(bio));
+ return bio->bi_crypt_context->data_unit_num;
+}
+
+static inline u64 bio_crypt_sw_data_unit_num(struct bio *bio)
+{
+ WARN_ON(!bio_is_encrypted(bio));
+ return bio->bi_crypt_context->sw_data_unit_num;
+}
+
+extern bool bio_crypt_should_process(struct bio *bio, struct request_queue *q);
+
+extern bool bio_crypt_ctx_compatible(struct bio *b_1, struct bio *b_2);
+
+extern bool bio_crypt_ctx_back_mergeable(struct bio *b_1,
+ unsigned int b1_sectors,
+ struct bio *b_2);
+
+#else /* CONFIG_BLK_INLINE_ENCRYPTION */
+struct keyslot_manager;
+
+static inline int bio_crypt_clone(struct bio *dst, struct bio *src,
+ gfp_t gfp_mask)
+{
+ return 0;
+}
+
+static inline void bio_crypt_advance(struct bio *bio,
+ unsigned int bytes) { }
+
+static inline bool bio_is_encrypted(struct bio *bio)
+{
+ return false;
+}
+
+static inline void bio_crypt_free_ctx(struct bio *bio) { }
+
+static inline void bio_crypt_set_ctx(struct bio *bio,
+ u8 *raw_key,
+ enum blk_crypt_mode_num crypt_mode,
+ u64 dun,
+ unsigned int dun_bits) { }
+
+static inline bool bio_crypt_has_keyslot(struct bio *bio)
+{
+ return false;
+}
+
+static inline void bio_crypt_set_keyslot(struct bio *bio,
+ unsigned int keyslot,
+ struct keyslot_manager *ksm) { }
+
+static inline int bio_crypt_get_keyslot(struct bio *bio)
+{
+ return -1;
+}
+
+static inline u8 *bio_crypt_raw_key(struct bio *bio)
+{
+ return NULL;
+}
+
+static inline u64 bio_crypt_data_unit_num(struct bio *bio)
+{
+ WARN_ON(1);
+ return 0;
+}
+
+static inline bool bio_crypt_should_process(struct bio *bio,
+ struct request_queue *q)
+{
+ return false;
+}
+
+static inline bool bio_crypt_ctx_compatible(struct bio *b_1, struct bio *b_2)
+{
+ return true;
+}
+
+static inline bool bio_crypt_ctx_back_mergeable(struct bio *b_1,
+ unsigned int b1_sectors,
+ struct bio *b_2)
+{
+ return true;
+}
+
+#endif /* CONFIG_BLK_INLINE_ENCRYPTION */
+
/*
* BIO list management for use by remapping drivers (e.g. DM or MD) and loop.
*
diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h
index 95202f80676c..0b794fe3530a 100644
--- a/include/linux/blk_types.h
+++ b/include/linux/blk_types.h
@@ -137,6 +137,8 @@ static inline void bio_issue_init(struct bio_issue *issue,
((u64)size << BIO_ISSUE_SIZE_SHIFT));
}
+struct bio_crypt_ctx;
+
/*
* main unit of I/O for the block layer and lower layers (ie drivers and
* stacking drivers)
@@ -175,6 +177,11 @@ struct bio {
struct blkcg_gq *bi_blkg;
struct bio_issue bi_issue;
#endif
+
+#ifdef CONFIG_BLK_INLINE_ENCRYPTION
+ struct bio_crypt_ctx *bi_crypt_context;
+#endif
+
union {
#if defined(CONFIG_BLK_DEV_INTEGRITY)
struct bio_integrity_payload *bi_integrity; /* data integrity */
--
2.22.0.410.gd8fdbe21b5-goog
^ permalink raw reply related
* [PATCH 3/8] block: blk-crypto for Inline Encryption
From: Satya Tangirala @ 2019-07-10 22:56 UTC (permalink / raw)
To: linux-block, linux-scsi, linux-fscrypt, linux-fsdevel,
linux-f2fs-devel
Cc: Parshuram Raju Thombare, Ladvine D Almeida, Barani Muthukumaran,
Kuohong Wang, Greg Kroah-Hartman, Satya Tangirala
In-Reply-To: <20190710225609.192252-1-satyat@google.com>
We introduce blk-crypto, which manages programming keyslots for struct
bios. With blk-crypto, filesystems only need to call bio_crypt_set_ctx with
the encryption key, algorithm and data_unit_num; they don't have to worry
about getting a keyslot for each encryption context, as blk-crypto handles
that. Blk-crypto also makes it possible for layered devices like device
mapper to make use of inline encryption hardware.
Blk-crypto delegates crypto operations to inline encryption hardware when
available, and also contains a software fallback to the kernel crypto API.
For more details, refer to Documentation/block/blk-crypto.txt.
Known issues:
1) We're allocating crypto_skcipher in blk_crypto_keyslot_program, which
uses GFP_KERNEL to allocate memory, but this function is on the write
path for IO - we need to add support for specifying a different flags
to the crypto API.
Signed-off-by: Satya Tangirala <satyat@google.com>
---
Documentation/block/inline-encryption.txt | 185 +++++++
block/Makefile | 3 +-
block/bio-crypt-ctx.c | 7 +-
block/bio.c | 5 +
block/blk-core.c | 11 +-
block/blk-crypto.c | 585 ++++++++++++++++++++++
include/linux/bio.h | 7 +
include/linux/blk-crypto.h | 40 ++
8 files changed, 840 insertions(+), 3 deletions(-)
create mode 100644 Documentation/block/inline-encryption.txt
create mode 100644 block/blk-crypto.c
create mode 100644 include/linux/blk-crypto.h
diff --git a/Documentation/block/inline-encryption.txt b/Documentation/block/inline-encryption.txt
new file mode 100644
index 000000000000..96a7983a117d
--- /dev/null
+++ b/Documentation/block/inline-encryption.txt
@@ -0,0 +1,185 @@
+BLK-CRYPTO and KEYSLOT MANAGER
+===========================
+
+CONTENTS
+1. Objective
+2. Constraints and notes
+3. Design
+4. Blk-crypto
+ 4-1 What does blk-crypto do on bio submission
+5. Layered Devices
+6. Future optimizations for layered devices
+
+1. Objective
+============
+
+We want to support inline encryption (IE) in the kernel.
+To allow for testing, we also want a software fallback when actual
+IE hardware is absent. We also want IE to work with layered devices
+like dm and loopback (i.e. we want to be able to use the IE hardware
+of the underlying devices if present, or else fall back to software
+en/decryption).
+
+
+2. Constraints and notes
+========================
+
+1) IE hardware have a limited number of “keyslots” that can be programmed
+with an encryption context (key, algorithm, data unit size, etc.) at any time.
+One can specify a keyslot in a data request made to the device, and the
+device will en/decrypt the data using the encryption context programmed into
+that specified keyslot. When possible, we want to make multiple requests with
+the same encryption context share the same keyslot.
+
+2) We need a way for filesystems to specify an encryption context to use for
+en/decrypting a struct bio, and a device driver (like UFS) needs to be able
+to use that encryption context when it processes the bio.
+
+3) We need a way for device drivers to expose their capabilities in a unified
+way to the upper layers.
+
+
+3. Design
+=========
+
+We add a struct bio_crypt_context to struct bio that can represent an
+encryption context, because we need to able to pass this encryption context
+from the FS layer to the device driver to act upon.
+
+While IE hardware works on the notion of keyslots, the FS layer has no
+knowledge of keyslots - it simply wants to specify an encryption context to
+use while en/decrypting a bio.
+
+We introduce a keyslot manager (KSM) that handles the translation from
+encryption contexts specified by the FS to keyslots on the IE hardware.
+This KSM also serves as the way IE hardware can expose their capabilities to
+upper layers. The generic mode of operation is: each device driver that wants
+to support IE will construct a KSM and set it up in its struct request_queue.
+Upper layers that want to use IE on this device can then use this KSM in
+the device’s struct request_queue to translate an encryption context into
+a keyslot. The presence of the KSM in the request queue shall be used to mean
+that the device supports IE.
+
+On the device driver end of the interface, the device driver needs to tell the
+KSM how to actually manipulate the IE hardware in the device to do things like
+programming the crypto key into the IE hardware into a particular keyslot. All
+this is achieved through the struct keyslot_mgmt_ll_ops that the device driver
+passes to the KSM when creating it.
+
+It uses refcounts to track which keyslots are idle (either they have no
+encryption context programmed, or there are no in flight struct bios
+referencing that keyslot). When a new encryption context needs a keyslot, it
+tries to find a keyslot that has already been programmed with the same
+encryption context, and if there is no such keyslot, it evicts the least
+recently used idle keyslot and programs the new encryption context into that
+one. If no idle keyslots are available, then the caller will sleep until there
+is at least one.
+
+
+4. Blk-crypto
+=============
+
+The above is sufficient for simple cases, but does not work if there is a
+need for a software fallback, or if we are want to use IE with layered devices.
+To these ends, we introduce blk-crypto. Blk-crypto allows us to present a
+unified view of encryption to the FS (so FS only needs to specify an
+encryption context and not worry about keyslots at all), and blk-crypto can
+decide whether to delegate the en/decryption to IE hardware or to software
+(i.e. to the kernel crypto API). Blk-crypto maintains an internal KSM that
+serves as the software fallback to the kernel crypto API.
+
+Blk-crypto needs to ensure that the encryption context is programmed into the
+"correct" keyslot manager for IE. If a bio is submitted to a layered device
+that eventually passes the bio down to a device that really does support IE, we
+want the encryption context to be programmed into a keyslot for the KSM of the
+device with IE support. However, blk-crypto does not know a priori whether a
+particular device is the final device in the layering structure for a bio or
+not. So in the case that a particular device does not support IE, since it is
+possibly the final destination device for the bio, if the bio requires
+encryption (i.e. the bio is doing a write operation), blk-crypto must fallback
+to software *before* sending the bio to the device.
+
+Blk-crypto ensures that
+1) The bio’s encryption context is programmed into a keyslot in the KSM of the
+request queue that the bio is being submitted to (or the software fallback KSM
+if the request queue doesn’t have a KSM), and that the processing_ksm in the
+bi_crypt_context is set to this KSM
+
+2) That the bio has its own individual reference to the keyslot in this KSM.
+Once the bio passes through blk-crypto, its encryption context is programmed
+in some KSM. The “its own individual reference to the keyslot” ensures that
+keyslots can be released by each bio independently of other bios while ensuring
+that the bio has a valid reference to the keyslot when, for e.g., the software
+fallback KSM in blk-crypto performs crypto for on the device’s behalf. The
+individual references are ensured by increasing the refcount for the keyslot in
+the processing_ksm when a bio with a programmed encryption context is cloned.
+
+
+4-1. What blk-crypto does on bio submission
+-------------------------------------------
+
+Case 1: blk-crypto is given a bio with only an encryption context that hasn’t
+been programmed into any keyslot in any KSM (for e.g. a bio from the FS). In
+this case, blk-crypto will program the encryption context into the KSM of the
+request queue the bio is being submitted to (and if this KSM does not exist,
+then it will program it into blk-crypto’s internal KSM for software fallback).
+The KSM that this encryption context was programmed into is stored as the
+processing_ksm in the bio’s bi_crypt_context.
+
+Case 2: blk-crypto is given a bio whose encryption context has already been
+programmed into a keyslot in the *software fallback KSM*. In this case,
+blk-crypto does nothing; it treats the bio as not having specified an
+encryption context. Note that we cannot do what we will do in Case 3 here
+because we would have already encrypted the bio in software by this point.
+
+Case 3: blk-crypto is given a bio whose encryption context has already been
+programmed into a keyslot in some KSM (that is *not* the software fallback
+KSM). In this case, blk-crypto first releases that keyslot from that KSM and
+then treats the bio as in Case 1.
+
+This way, when a device driver is processing a bio, it can be sure that
+the bio’s encryption context has been programmed into some KSM (either the
+device driver’s request queue’s KSM, or blk-crypto’s software fallback KSM).
+It then simply needs to check if the bio’s processing_ksm is the device’s
+request queue’s KSM. If so, then it should proceed with IE. If not, it should
+simply do nothing with respect to crypto, because some other KSM (perhaps the
+blk-crypto software fallback KSM) is handling the en/decryption.
+
+Blk-crypto will release the keyslot that is being held by the bio (and also
+decrypt it if the bio is using the software fallback KSM) once
+bio_remaining_done returns true for the bio.
+
+
+5. Layered Devices
+==================
+
+Layered devices that wish to support IE need to create their own keyslot
+manager for their request queue, and expose whatever functionality they choose.
+When a layered device wants to pass a bio to another layer (either by
+resubmitting the same bio, or by submitting a clone), it doesn’t need to do
+anything special because the bio (or the clone) will once again pass through
+blk-crypto, which will work as described in Case 3. If a layered device wants
+for some reason to do the IO by itself instead of passing it on to a child
+device, but it also chose to expose IE capabilities by setting up a KSM in its
+request queue, it is then responsible for en/decrypting the data itself. In
+such cases, the device can choose to call the blk-crypto function
+blk_crypto_fallback_to_software (TODO: Not yet implemented), which will
+cause the en/decryption to be done via software fallback.
+
+
+6. Future Optimizations for layered devices
+===========================================
+
+Creating a keyslot manager for the layered device uses up memory for each
+keyslot, and in general, a layered device (like dm-linear) merely passes the
+request on to a “child” device, so the keyslots in the layered device itself
+might be completely unused. We can instead define a new type of KSM; the
+“passthrough KSM”, that layered devices can use to let blk-crypto know that
+this layered device *will* pass the bio to some child device (and hence
+through blk-crypto again, at which point blk-crypto can program the encryption
+context, instead of programming it into the layered device’s KSM). Again, if
+the device “lies” and decides to do the IO itself instead of passing it on to
+a child device, it is responsible for doing the en/decryption (and can choose
+to call blk_crypto_fallback_to_software). Another use case for the
+"passthrough KSM" is for IE devices that want to manage their own keyslots/do
+not have a limited number of keyslots.
diff --git a/block/Makefile b/block/Makefile
index 4147ffa63631..1ba7de84dbaf 100644
--- a/block/Makefile
+++ b/block/Makefile
@@ -35,4 +35,5 @@ obj-$(CONFIG_BLK_DEBUG_FS) += blk-mq-debugfs.o
obj-$(CONFIG_BLK_DEBUG_FS_ZONED)+= blk-mq-debugfs-zoned.o
obj-$(CONFIG_BLK_SED_OPAL) += sed-opal.o
obj-$(CONFIG_BLK_PM) += blk-pm.o
-obj-$(CONFIG_BLK_INLINE_ENCRYPTION) += keyslot-manager.o bio-crypt-ctx.o
+obj-$(CONFIG_BLK_INLINE_ENCRYPTION) += keyslot-manager.o bio-crypt-ctx.o \
+ blk-crypto.o
diff --git a/block/bio-crypt-ctx.c b/block/bio-crypt-ctx.c
index 8b884ef32007..91015645c4ea 100644
--- a/block/bio-crypt-ctx.c
+++ b/block/bio-crypt-ctx.c
@@ -23,7 +23,12 @@ EXPORT_SYMBOL(bio_crypt_free_ctx);
int bio_crypt_clone(struct bio *dst, struct bio *src, gfp_t gfp_mask)
{
- if (!bio_is_encrypted(src))
+ /*
+ * If a bio is swhandled, then it will be decrypted when bio_endio
+ * is called. As we only want the data to be decrypted once, copies
+ * of the bio must not have have a crypt context.
+ */
+ if (!bio_is_encrypted(src) || bio_crypt_swhandled(src))
return 0;
dst->bi_crypt_context = bio_crypt_alloc_ctx(gfp_mask);
diff --git a/block/bio.c b/block/bio.c
index 2a272cda6dfa..390777a76468 100644
--- a/block/bio.c
+++ b/block/bio.c
@@ -16,6 +16,7 @@
#include <linux/workqueue.h>
#include <linux/cgroup.h>
#include <linux/blk-cgroup.h>
+#include <linux/blk-crypto.h>
#include <trace/events/block.h>
#include "blk.h"
@@ -1832,6 +1833,10 @@ void bio_endio(struct bio *bio)
again:
if (!bio_remaining_done(bio))
return;
+
+ if (!blk_crypto_endio(bio))
+ return;
+
if (!bio_integrity_endio(bio))
return;
diff --git a/block/blk-core.c b/block/blk-core.c
index 8340f69670d8..aec4da8c27fa 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -36,6 +36,7 @@
#include <linux/blk-cgroup.h>
#include <linux/debugfs.h>
#include <linux/bpf.h>
+#include <linux/blk-crypto.h>
#define CREATE_TRACE_POINTS
#include <trace/events/block.h>
@@ -1018,7 +1019,9 @@ blk_qc_t generic_make_request(struct bio *bio)
/* Create a fresh bio_list for all subordinate requests */
bio_list_on_stack[1] = bio_list_on_stack[0];
bio_list_init(&bio_list_on_stack[0]);
- ret = q->make_request_fn(q, bio);
+
+ if (!blk_crypto_submit_bio(&bio))
+ ret = q->make_request_fn(q, bio);
blk_queue_exit(q);
@@ -1071,6 +1074,9 @@ blk_qc_t direct_make_request(struct bio *bio)
if (!generic_make_request_checks(bio))
return BLK_QC_T_NONE;
+ if (blk_crypto_submit_bio(&bio))
+ return BLK_QC_T_NONE;
+
if (unlikely(blk_queue_enter(q, nowait ? BLK_MQ_REQ_NOWAIT : 0))) {
if (nowait && !blk_queue_dying(q))
bio->bi_status = BLK_STS_AGAIN;
@@ -1750,5 +1756,8 @@ int __init blk_dev_init(void)
blk_debugfs_root = debugfs_create_dir("block", NULL);
#endif
+ if (blk_crypto_init() < 0)
+ panic("Failed to init blk-crypto\n");
+
return 0;
}
diff --git a/block/blk-crypto.c b/block/blk-crypto.c
new file mode 100644
index 000000000000..f41fb7819ae9
--- /dev/null
+++ b/block/blk-crypto.c
@@ -0,0 +1,585 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright 2019 Google LLC
+ */
+#include <linux/blk-crypto.h>
+#include <linux/keyslot-manager.h>
+#include <linux/mempool.h>
+#include <linux/blk-cgroup.h>
+#include <crypto/skcipher.h>
+#include <crypto/algapi.h>
+#include <linux/module.h>
+
+struct blk_crypt_mode {
+ const char *cipher_str;
+ size_t keysize;
+};
+
+static const struct blk_crypt_mode blk_crypt_modes[] = {
+ [BLK_ENCRYPTION_MODE_AES_256_XTS] = {
+ .cipher_str = "xts(aes)",
+ .keysize = 64,
+ },
+};
+
+#define BLK_CRYPTO_MAX_KEY_SIZE 64
+/* TODO: Do we want to make this user configurable somehow? */
+static int blk_crypto_num_keyslots = 100;
+
+static struct blk_crypto_keyslot {
+ struct crypto_skcipher *tfm;
+ enum blk_crypt_mode_num crypt_mode;
+ u8 key[BLK_CRYPTO_MAX_KEY_SIZE];
+} *blk_crypto_keyslots;
+
+struct work_mem {
+ struct work_struct crypto_work;
+ struct bio *bio;
+};
+
+static struct keyslot_manager *blk_crypto_ksm;
+static struct workqueue_struct *blk_crypto_wq;
+static mempool_t *blk_crypto_page_pool;
+static struct kmem_cache *blk_crypto_work_mem_cache;
+
+static unsigned int num_prealloc_bounce_pg = 32;
+
+bool bio_crypt_swhandled(struct bio *bio)
+{
+ return bio_is_encrypted(bio) &&
+ bio->bi_crypt_context->processing_ksm == blk_crypto_ksm;
+}
+
+static void evict_keyslot(unsigned int slot)
+{
+ struct blk_crypto_keyslot *slotp = &blk_crypto_keyslots[slot];
+
+ crypto_free_skcipher(slotp->tfm);
+ slotp->tfm = NULL;
+ memzero_explicit(slotp->key, BLK_CRYPTO_MAX_KEY_SIZE);
+}
+
+/* TODO: handle modes that need essiv */
+static int blk_crypto_keyslot_program(void *priv, const u8 *key,
+ enum blk_crypt_mode_num crypt_mode,
+ unsigned int data_unit_size,
+ unsigned int slot)
+{
+ struct blk_crypto_keyslot *slotp = &blk_crypto_keyslots[slot];
+ struct crypto_skcipher *tfm = slotp->tfm;
+ const struct blk_crypt_mode *mode = &blk_crypt_modes[crypt_mode];
+ size_t keysize = mode->keysize;
+ int err;
+
+ if (crypt_mode != slotp->crypt_mode || !tfm) {
+ evict_keyslot(slot);
+ tfm = crypto_alloc_skcipher(
+ mode->cipher_str, 0, 0);
+ if (IS_ERR(tfm))
+ return PTR_ERR(tfm);
+
+ crypto_skcipher_set_flags(tfm,
+ CRYPTO_TFM_REQ_FORBID_WEAK_KEYS);
+ slotp->crypt_mode = crypt_mode;
+ slotp->tfm = tfm;
+ }
+
+ err = crypto_skcipher_setkey(tfm, key, keysize);
+
+ if (err) {
+ evict_keyslot(slot);
+ return err;
+ }
+
+ memcpy(slotp->key, key, keysize);
+
+ return 0;
+}
+
+static int blk_crypto_keyslot_evict(void *priv, const u8 *key,
+ enum blk_crypt_mode_num crypt_mode,
+ unsigned int data_unit_size,
+ unsigned int slot)
+{
+ evict_keyslot(slot);
+ return 0;
+}
+
+static int blk_crypto_keyslot_find(void *priv,
+ const u8 *key,
+ enum blk_crypt_mode_num crypt_mode,
+ unsigned int data_unit_size_bytes)
+{
+ int slot;
+ const size_t keysize = blk_crypt_modes[crypt_mode].keysize;
+
+ /* TODO: hashmap? */
+ for (slot = 0; slot < blk_crypto_num_keyslots; slot++) {
+ if (blk_crypto_keyslots[slot].crypt_mode == crypt_mode &&
+ !crypto_memneq(blk_crypto_keyslots[slot].key, key, keysize))
+ return slot;
+ }
+
+ return -ENOKEY;
+}
+
+static bool blk_crypt_mode_supported(void *priv,
+ enum blk_crypt_mode_num crypt_mode,
+ unsigned int data_unit_size)
+{
+ /* All blk_crypt_modes are required to have a software fallback. */
+ return true;
+}
+
+static const struct keyslot_mgmt_ll_ops blk_crypto_ksm_ll_ops = {
+ .keyslot_program = blk_crypto_keyslot_program,
+ .keyslot_evict = blk_crypto_keyslot_evict,
+ .keyslot_find = blk_crypto_keyslot_find,
+ .crypt_mode_supported = blk_crypt_mode_supported,
+};
+
+static void blk_crypto_encrypt_endio(struct bio *enc_bio)
+{
+ struct bio *src_bio = enc_bio->bi_private;
+ struct bio_vec *enc_bvec, *enc_bvec_end;
+
+ enc_bvec = enc_bio->bi_io_vec;
+ enc_bvec_end = enc_bvec + enc_bio->bi_vcnt;
+ for (; enc_bvec != enc_bvec_end; enc_bvec++)
+ mempool_free(enc_bvec->bv_page, blk_crypto_page_pool);
+
+ src_bio->bi_status = enc_bio->bi_status;
+
+ bio_put(enc_bio);
+ bio_endio(src_bio);
+}
+
+static struct bio *blk_crypto_clone_bio(struct bio *bio_src)
+{
+ struct bvec_iter iter;
+ struct bio_vec bv;
+ struct bio *bio;
+
+ bio = bio_alloc_bioset(GFP_NOIO, bio_segments(bio_src), NULL);
+ if (!bio)
+ return NULL;
+ bio->bi_disk = bio_src->bi_disk;
+ bio->bi_opf = bio_src->bi_opf;
+ bio->bi_ioprio = bio_src->bi_ioprio;
+ bio->bi_write_hint = bio_src->bi_write_hint;
+ bio->bi_iter.bi_sector = bio_src->bi_iter.bi_sector;
+ bio->bi_iter.bi_size = bio_src->bi_iter.bi_size;
+
+ bio_for_each_segment(bv, bio_src, iter)
+ bio->bi_io_vec[bio->bi_vcnt++] = bv;
+
+ if (bio_integrity(bio_src)) {
+ int ret;
+
+ ret = bio_integrity_clone(bio, bio_src, GFP_NOIO);
+ if (ret < 0) {
+ bio_put(bio);
+ return NULL;
+ }
+ }
+
+ bio_clone_blkg_association(bio, bio_src);
+ blkcg_bio_issue_init(bio);
+
+ return bio;
+}
+
+static int blk_crypto_encrypt_bio(struct bio **bio_ptr)
+{
+ struct bio *src_bio = *bio_ptr;
+ int slot;
+ struct skcipher_request *ciph_req = NULL;
+ DECLARE_CRYPTO_WAIT(wait);
+ struct bio_vec bv;
+ struct bvec_iter iter;
+ int err = 0;
+ u64 curr_dun;
+ union {
+ __le64 dun;
+ u8 bytes[16];
+ } iv;
+ struct scatterlist src, dst;
+ struct bio *enc_bio;
+ struct bio_vec *enc_bvec;
+ int i, j;
+ unsigned int num_sectors;
+ int data_unit_size;
+
+ /* Split the bio if it's too big for single page bvec */
+ i = 0;
+ num_sectors = 0;
+ data_unit_size = 1 << src_bio->bi_crypt_context->data_unit_size_bits;
+ bio_for_each_segment(bv, src_bio, iter) {
+ num_sectors += bv.bv_len >> 9;
+ if (bv.bv_len % data_unit_size != 0) {
+ src_bio->bi_status = BLK_STS_IOERR;
+ return -EIO;
+ }
+ if (++i == BIO_MAX_PAGES)
+ break;
+ }
+ if (num_sectors < bio_sectors(src_bio)) {
+ struct bio *split_bio;
+
+ split_bio = bio_split(src_bio, num_sectors, GFP_NOIO, NULL);
+ if (!split_bio) {
+ src_bio->bi_status = BLK_STS_RESOURCE;
+ return -ENOMEM;
+ }
+ bio_chain(split_bio, src_bio);
+ generic_make_request(src_bio);
+ *bio_ptr = split_bio;
+ src_bio = *bio_ptr;
+ }
+
+ enc_bio = blk_crypto_clone_bio(src_bio);
+ if (!enc_bio) {
+ src_bio->bi_status = BLK_STS_RESOURCE;
+ return -ENOMEM;
+ }
+
+ err = bio_crypt_ctx_acquire_keyslot(src_bio, blk_crypto_ksm);
+ if (err) {
+ src_bio->bi_status = BLK_STS_IOERR;
+ bio_put(enc_bio);
+ return err;
+ }
+ slot = bio_crypt_get_keyslot(src_bio);
+
+ ciph_req = skcipher_request_alloc(blk_crypto_keyslots[slot].tfm,
+ GFP_NOIO);
+ if (!ciph_req) {
+ src_bio->bi_status = BLK_STS_RESOURCE;
+ err = -ENOMEM;
+ bio_put(enc_bio);
+ goto out_release_keyslot;
+ }
+
+ skcipher_request_set_callback(ciph_req,
+ CRYPTO_TFM_REQ_MAY_BACKLOG |
+ CRYPTO_TFM_REQ_MAY_SLEEP,
+ crypto_req_done, &wait);
+
+ curr_dun = bio_crypt_data_unit_num(src_bio);
+ sg_init_table(&src, 1);
+ sg_init_table(&dst, 1);
+ for (i = 0, enc_bvec = enc_bio->bi_io_vec; i < enc_bio->bi_vcnt;
+ enc_bvec++, i++) {
+ struct page *page = enc_bvec->bv_page;
+ struct page *ciphertext_page =
+ mempool_alloc(blk_crypto_page_pool, GFP_NOIO);
+
+ enc_bvec->bv_page = ciphertext_page;
+
+ if (!ciphertext_page)
+ goto no_mem_for_ciph_page;
+
+ for (j = 0; j < enc_bvec->bv_len / data_unit_size; j++) {
+ memset(&iv, 0, sizeof(iv));
+ iv.dun = cpu_to_le64(curr_dun);
+
+ sg_set_page(&src, page, data_unit_size,
+ enc_bvec->bv_offset + j * data_unit_size);
+ sg_set_page(&dst, ciphertext_page, data_unit_size,
+ enc_bvec->bv_offset + j * data_unit_size);
+
+ skcipher_request_set_crypt(ciph_req, &src, &dst,
+ data_unit_size, iv.bytes);
+ err = crypto_wait_req(crypto_skcipher_encrypt(ciph_req),
+ &wait);
+ if (err)
+ goto no_mem_for_ciph_page;
+ curr_dun++;
+ }
+ continue;
+no_mem_for_ciph_page:
+ err = -ENOMEM;
+ for (j = i - 1; j >= 0; j--) {
+ mempool_free(enc_bio->bi_io_vec[j].bv_page,
+ blk_crypto_page_pool);
+ }
+ src_bio->bi_status = BLK_STS_RESOURCE;
+ bio_put(enc_bio);
+ goto out_release_cipher;
+ }
+
+ enc_bio->bi_private = src_bio;
+ enc_bio->bi_end_io = blk_crypto_encrypt_endio;
+
+ *bio_ptr = enc_bio;
+out_release_cipher:
+ skcipher_request_free(ciph_req);
+out_release_keyslot:
+ bio_crypt_ctx_release_keyslot(src_bio);
+ return err;
+}
+
+/*
+ * TODO: assumption right now is:
+ * each segment in bio has length divisible by the data_unit_size
+ */
+static void blk_crypto_decrypt_bio(struct work_struct *w)
+{
+ struct work_mem *work_mem =
+ container_of(w, struct work_mem, crypto_work);
+ struct bio *bio = work_mem->bio;
+ int slot;
+ struct skcipher_request *ciph_req;
+ DECLARE_CRYPTO_WAIT(wait);
+ struct bio_vec bv;
+ struct bvec_iter iter;
+ u64 curr_dun;
+ union {
+ __le64 dun;
+ u8 bytes[16];
+ } iv;
+ struct scatterlist sg;
+ int err;
+ int data_unit_size = 1 << bio->bi_crypt_context->data_unit_size_bits;
+ int i;
+
+ kmem_cache_free(blk_crypto_work_mem_cache, work_mem);
+
+ err = bio_crypt_ctx_acquire_keyslot(bio, blk_crypto_ksm);
+ if (err) {
+ bio->bi_status = BLK_STS_RESOURCE;
+ goto out_no_keyslot;
+ }
+
+ slot = bio_crypt_get_keyslot(bio);
+ ciph_req = skcipher_request_alloc(blk_crypto_keyslots[slot].tfm,
+ GFP_NOIO);
+ if (!ciph_req) {
+ bio->bi_status = BLK_STS_RESOURCE;
+ goto out;
+ }
+
+ skcipher_request_set_callback(ciph_req,
+ CRYPTO_TFM_REQ_MAY_BACKLOG |
+ CRYPTO_TFM_REQ_MAY_SLEEP,
+ crypto_req_done, &wait);
+
+ curr_dun = bio_crypt_sw_data_unit_num(bio);
+ sg_init_table(&sg, 1);
+ __bio_for_each_segment(bv, bio, iter,
+ bio->bi_crypt_context->crypt_iter) {
+ if (bv.bv_len % data_unit_size != 0) {
+ bio->bi_status = BLK_STS_IOERR;
+ err = -EIO;
+ goto out;
+ }
+ for (i = 0; i < bv.bv_len / data_unit_size; i++) {
+ struct page *page = bv.bv_page;
+
+ memset(&iv, 0, sizeof(iv));
+ iv.dun = cpu_to_le64(curr_dun);
+
+ sg_set_page(&sg, page, data_unit_size,
+ bv.bv_offset + i * data_unit_size);
+ skcipher_request_set_crypt(ciph_req, &sg, &sg,
+ data_unit_size, iv.bytes);
+ err = crypto_wait_req(crypto_skcipher_decrypt(ciph_req),
+ &wait);
+ if (err) {
+ bio->bi_status = BLK_STS_IOERR;
+ goto out;
+ }
+ curr_dun++;
+ }
+ }
+
+out:
+ skcipher_request_free(ciph_req);
+ bio_crypt_ctx_release_keyslot(bio);
+out_no_keyslot:
+ bio_endio(bio);
+}
+
+static void blk_crypto_queue_decrypt_bio(struct bio *bio)
+{
+ struct work_mem *work_mem =
+ kmem_cache_zalloc(blk_crypto_work_mem_cache, GFP_ATOMIC);
+
+ if (!work_mem) {
+ bio->bi_status = BLK_STS_RESOURCE;
+ return bio_endio(bio);
+ }
+
+ INIT_WORK(&work_mem->crypto_work, blk_crypto_decrypt_bio);
+ work_mem->bio = bio;
+ queue_work(blk_crypto_wq, &work_mem->crypto_work);
+}
+
+/**
+ * blk_crypto_submit_bio - handle submitting bio for inline encryption
+ *
+ * @bio_ptr: pointer to original bio pointer
+ *
+ * If the bio doesn't have inline encryption enabled or the submitter already
+ * specified a keyslot for the target device, do nothing. Else, a raw key must
+ * have been provided, so acquire a device keyslot for it if supported. Else,
+ * use the software crypto fallback.
+ *
+ * When the software fallback is used for encryption, blk-crypto may choose to
+ * split the bio into 2 - one that will continue to be processed and the other
+ * that will be resubmitted via generic_make_request. *bio_ptr will be updated
+ * to the first bio (the one that will continue to be processed).
+ *
+ * Return: 0 if bio submission should continue; nonzero if bio_endio() was
+ * already called so bio submission should abort.
+ */
+int blk_crypto_submit_bio(struct bio **bio_ptr)
+{
+ struct bio *bio = *bio_ptr;
+ struct request_queue *q;
+ int err;
+ struct bio_crypt_ctx *crypt_ctx;
+
+ if (!bio_is_encrypted(bio) || !bio_has_data(bio))
+ return 0;
+
+ /*
+ * When a read bio is marked for sw decryption, its bi_iter is saved
+ * so that when we decrypt the bio later, we know what part of it was
+ * marked for sw decryption (when the bio is passed down after
+ * blk_crypto_submit bio, it may be split or advanced so we cannot rely
+ * on the bi_iter while decrypting in blk_crypto_endio)
+ */
+ if (bio_crypt_swhandled(bio))
+ return 0;
+
+ crypt_ctx = bio->bi_crypt_context;
+ q = bio->bi_disk->queue;
+
+ if (bio_crypt_has_keyslot(bio)) {
+ /* Key already programmed into device? */
+ if (q->ksm == crypt_ctx->processing_ksm)
+ return 0;
+
+ /* Nope, release the existing keyslot. */
+ bio_crypt_ctx_release_keyslot(bio);
+ }
+
+ /* Get device keyslot if supported */
+ if (q->ksm) {
+ err = bio_crypt_ctx_acquire_keyslot(bio, q->ksm);
+ if (!err) {
+ pr_warn_once("blk-crypto: failed to acquire keyslot for %s (err=%d). Falling back to software crypto.\n",
+ bio->bi_disk->disk_name, err);
+ return 0;
+ }
+ }
+
+ /* Fallback to software crypto */
+ if (bio_data_dir(bio) == WRITE) {
+ /* Encrypt the data now */
+ err = blk_crypto_encrypt_bio(bio_ptr);
+ if (err)
+ goto out_encrypt_err;
+ } else {
+ /* Mark bio as swhandled */
+ bio->bi_crypt_context->processing_ksm = blk_crypto_ksm;
+ bio->bi_crypt_context->crypt_iter = bio->bi_iter;
+ bio->bi_crypt_context->sw_data_unit_num =
+ bio->bi_crypt_context->data_unit_num;
+ }
+ return 0;
+out_encrypt_err:
+ bio_endio(bio);
+ return err;
+}
+
+/**
+ * blk_crypto_endio - clean up bio w.r.t inline encryption during bio_endio
+ *
+ * @bio - the bio to clean up
+ *
+ * If blk_crypto_submit_bio decided to fallback to software crypto for this
+ * bio, we queue the bio for decryption into a workqueue and return false,
+ * and call bio_endio(bio) at a later time (after the bio has been decrypted).
+ *
+ * If the bio is not to be decrypted in software, this function releases the
+ * reference to the keyslot that blk_crypto_submit_bio got.
+ *
+ * Return: true if bio_endio should continue; false otherwise (bio_endio will
+ * be called again when bio has been decrypted).
+ */
+bool blk_crypto_endio(struct bio *bio)
+{
+ if (bio_crypt_swhandled(bio)) {
+ /*
+ * The only bios that are swhandled when they reach here
+ * are those with bio_data_dir(bio) == READ, since WRITE
+ * bios that are encrypted by software fallback are handled
+ * by blk_crypto_encrypt_endio.
+ */
+ blk_crypto_queue_decrypt_bio(bio);
+ return false;
+ }
+
+ if (bio_is_encrypted(bio) && bio_crypt_has_keyslot(bio))
+ bio_crypt_ctx_release_keyslot(bio);
+
+ return true;
+}
+
+int __init blk_crypto_init(void)
+{
+ blk_crypto_ksm = keyslot_manager_create(blk_crypto_num_keyslots,
+ &blk_crypto_ksm_ll_ops,
+ NULL);
+ if (!blk_crypto_ksm)
+ goto out_ksm;
+
+ blk_crypto_wq = alloc_workqueue("blk_crypto_wq",
+ WQ_UNBOUND | WQ_HIGHPRI |
+ WQ_MEM_RECLAIM,
+ num_online_cpus());
+ if (!blk_crypto_wq)
+ goto out_wq;
+
+ blk_crypto_keyslots = kcalloc(blk_crypto_num_keyslots,
+ sizeof(*blk_crypto_keyslots),
+ GFP_KERNEL);
+ if (!blk_crypto_keyslots)
+ goto out_blk_crypto_keyslots;
+
+ blk_crypto_page_pool =
+ mempool_create_page_pool(num_prealloc_bounce_pg, 0);
+ if (!blk_crypto_page_pool)
+ goto out_bounce_pool;
+
+ blk_crypto_work_mem_cache = KMEM_CACHE(work_mem, SLAB_RECLAIM_ACCOUNT);
+ if (!blk_crypto_work_mem_cache)
+ goto out_work_mem_cache;
+
+ return 0;
+
+out_work_mem_cache:
+ mempool_destroy(blk_crypto_page_pool);
+ blk_crypto_page_pool = NULL;
+out_bounce_pool:
+ kzfree(blk_crypto_keyslots);
+ blk_crypto_keyslots = NULL;
+out_blk_crypto_keyslots:
+ destroy_workqueue(blk_crypto_wq);
+ blk_crypto_wq = NULL;
+out_wq:
+ keyslot_manager_destroy(blk_crypto_ksm);
+ blk_crypto_ksm = NULL;
+out_ksm:
+ pr_warn("No memory for blk-crypto software fallback.");
+ return -ENOMEM;
+}
+
+module_param(blk_crypto_num_keyslots, int, 0);
+MODULE_PARM_DESC(blk_crypto_num_keyslots,
+ "Number of keyslots for software fallback in blk-crypto.");
+module_param(num_prealloc_bounce_pg, uint, 0);
+MODULE_PARM_DESC(num_prealloc_bounce_pg,
+ "Number of preallocated bounce pages for blk-crypto to use during software fallback encryption");
diff --git a/include/linux/bio.h b/include/linux/bio.h
index 4e664d6441d5..66ad1508911f 100644
--- a/include/linux/bio.h
+++ b/include/linux/bio.h
@@ -612,6 +612,8 @@ static inline void bio_crypt_advance(struct bio *bio, unsigned int bytes)
}
}
+extern bool bio_crypt_swhandled(struct bio *bio);
+
static inline bool bio_crypt_has_keyslot(struct bio *bio)
{
return bio->bi_crypt_context->keyslot >= 0;
@@ -717,6 +719,11 @@ static inline void bio_crypt_set_ctx(struct bio *bio,
u64 dun,
unsigned int dun_bits) { }
+static inline bool bio_crypt_swhandled(struct bio *bio)
+{
+ return false;
+}
+
static inline bool bio_crypt_has_keyslot(struct bio *bio)
{
return false;
diff --git a/include/linux/blk-crypto.h b/include/linux/blk-crypto.h
new file mode 100644
index 000000000000..cbb5bea6dcdb
--- /dev/null
+++ b/include/linux/blk-crypto.h
@@ -0,0 +1,40 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright 2019 Google LLC
+ */
+
+#ifndef __LINUX_BLK_CRYPTO_H
+#define __LINUX_BLK_CRYPTO_H
+
+#include <linux/types.h>
+
+#ifdef CONFIG_BLK_INLINE_ENCRYPTION
+
+struct bio;
+
+int blk_crypto_init(void);
+
+int blk_crypto_submit_bio(struct bio **bio_ptr);
+
+bool blk_crypto_endio(struct bio *bio);
+
+#else /* CONFIG_BLK_INLINE_ENCRYPTION */
+
+static inline int blk_crypto_init(void)
+{
+ return 0;
+}
+
+static inline int blk_crypto_submit_bio(struct bio **bio)
+{
+ return 0;
+}
+
+static inline bool blk_crypto_endio(struct bio *bio)
+{
+ return true;
+}
+
+#endif /* CONFIG_BLK_INLINE_ENCRYPTION */
+
+#endif /* __LINUX_BLK_CRYPTO_H */
--
2.22.0.410.gd8fdbe21b5-goog
^ permalink raw reply related
* [PATCH 4/8] scsi: ufs: UFS driver v2.1 spec crypto additions
From: Satya Tangirala @ 2019-07-10 22:56 UTC (permalink / raw)
To: linux-block, linux-scsi, linux-fscrypt, linux-fsdevel,
linux-f2fs-devel
Cc: Parshuram Raju Thombare, Ladvine D Almeida, Barani Muthukumaran,
Kuohong Wang, Greg Kroah-Hartman, Satya Tangirala
In-Reply-To: <20190710225609.192252-1-satyat@google.com>
Add the crypto registers and structs defined in v2.1 of the JEDEC UFSHCI
specification in preparation to add support for inline encryption to
UFS.
Signed-off-by: Satya Tangirala <satyat@google.com>
---
drivers/scsi/ufs/ufshcd.c | 2 ++
drivers/scsi/ufs/ufshcd.h | 5 +++
drivers/scsi/ufs/ufshci.h | 67 +++++++++++++++++++++++++++++++++++++--
3 files changed, 72 insertions(+), 2 deletions(-)
diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
index 3fe3029617a8..36413f97074d 100644
--- a/drivers/scsi/ufs/ufshcd.c
+++ b/drivers/scsi/ufs/ufshcd.c
@@ -4724,6 +4724,8 @@ ufshcd_transfer_rsp_status(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)
case OCS_MISMATCH_RESP_UPIU_SIZE:
case OCS_PEER_COMM_FAILURE:
case OCS_FATAL_ERROR:
+ case OCS_INVALID_CRYPTO_CONFIG:
+ case OCS_GENERAL_CRYPTO_ERROR:
default:
result |= DID_ERROR << 16;
dev_err(hba->dev,
diff --git a/drivers/scsi/ufs/ufshcd.h b/drivers/scsi/ufs/ufshcd.h
index ecfa898b9ccc..d3b6a6b57a37 100644
--- a/drivers/scsi/ufs/ufshcd.h
+++ b/drivers/scsi/ufs/ufshcd.h
@@ -692,6 +692,11 @@ struct ufs_hba {
* the performance of ongoing read/write operations.
*/
#define UFSHCD_CAP_KEEP_AUTO_BKOPS_ENABLED_EXCEPT_SUSPEND (1 << 5)
+ /*
+ * This capability allows the host controller driver to use the
+ * inline crypto engine, if it is present
+ */
+#define UFSHCD_CAP_CRYPTO (1 << 6)
struct devfreq *devfreq;
struct ufs_clk_scaling clk_scaling;
diff --git a/drivers/scsi/ufs/ufshci.h b/drivers/scsi/ufs/ufshci.h
index 6fa889de5ee5..a757eaf99a19 100644
--- a/drivers/scsi/ufs/ufshci.h
+++ b/drivers/scsi/ufs/ufshci.h
@@ -90,6 +90,7 @@ enum {
MASK_64_ADDRESSING_SUPPORT = 0x01000000,
MASK_OUT_OF_ORDER_DATA_DELIVERY_SUPPORT = 0x02000000,
MASK_UIC_DME_TEST_MODE_SUPPORT = 0x04000000,
+ MASK_CRYPTO_SUPPORT = 0x10000000,
};
#define UFS_MASK(mask, offset) ((mask) << (offset))
@@ -143,6 +144,7 @@ enum {
#define DEVICE_FATAL_ERROR 0x800
#define CONTROLLER_FATAL_ERROR 0x10000
#define SYSTEM_BUS_FATAL_ERROR 0x20000
+#define CRYPTO_ENGINE_FATAL_ERROR 0x40000
#define UFSHCD_UIC_PWR_MASK (UIC_HIBERNATE_ENTER |\
UIC_HIBERNATE_EXIT |\
@@ -153,11 +155,13 @@ enum {
#define UFSHCD_ERROR_MASK (UIC_ERROR |\
DEVICE_FATAL_ERROR |\
CONTROLLER_FATAL_ERROR |\
- SYSTEM_BUS_FATAL_ERROR)
+ SYSTEM_BUS_FATAL_ERROR |\
+ CRYPTO_ENGINE_FATAL_ERROR)
#define INT_FATAL_ERRORS (DEVICE_FATAL_ERROR |\
CONTROLLER_FATAL_ERROR |\
- SYSTEM_BUS_FATAL_ERROR)
+ SYSTEM_BUS_FATAL_ERROR |\
+ CRYPTO_ENGINE_FATAL_ERROR)
/* HCS - Host Controller Status 30h */
#define DEVICE_PRESENT 0x1
@@ -316,6 +320,61 @@ enum {
INTERRUPT_MASK_ALL_VER_21 = 0x71FFF,
};
+/* CCAP - Crypto Capability 100h */
+union ufs_crypto_capabilities {
+ __le32 reg_val;
+ struct {
+ u8 num_crypto_cap;
+ u8 config_count;
+ u8 reserved;
+ u8 config_array_ptr;
+ };
+};
+
+enum ufs_crypto_key_size {
+ UFS_CRYPTO_KEY_SIZE_INVALID = 0x0,
+ UFS_CRYPTO_KEY_SIZE_128 = 0x1,
+ UFS_CRYPTO_KEY_SIZE_192 = 0x2,
+ UFS_CRYPTO_KEY_SIZE_256 = 0x3,
+ UFS_CRYPTO_KEY_SIZE_512 = 0x4,
+};
+
+enum ufs_crypto_alg {
+ UFS_CRYPTO_ALG_AES_XTS = 0x0,
+ UFS_CRYPTO_ALG_BITLOCKER_AES_CBC = 0x1,
+ UFS_CRYPTO_ALG_AES_ECB = 0x2,
+ UFS_CRYPTO_ALG_ESSIV_AES_CBC = 0x3,
+};
+
+/* x-CRYPTOCAP - Crypto Capability X */
+union ufs_crypto_cap_entry {
+ __le32 reg_val;
+ struct {
+ u8 algorithm_id;
+ u8 sdus_mask; /* Supported data unit size mask */
+ u8 key_size;
+ u8 reserved;
+ };
+};
+
+#define UFS_CRYPTO_CONFIGURATION_ENABLE (1 << 7)
+#define UFS_CRYPTO_KEY_MAX_SIZE 64
+/* x-CRYPTOCFG - Crypto Configuration X */
+union ufs_crypto_cfg_entry {
+ __le32 reg_val[32];
+ struct {
+ u8 crypto_key[UFS_CRYPTO_KEY_MAX_SIZE];
+ u8 data_unit_size;
+ u8 crypto_cap_idx;
+ u8 reserved_1;
+ u8 config_enable;
+ u8 reserved_multi_host;
+ u8 reserved_2;
+ u8 vsb[2];
+ u8 reserved_3[56];
+ };
+};
+
/*
* Request Descriptor Definitions
*/
@@ -337,6 +396,7 @@ enum {
UTP_NATIVE_UFS_COMMAND = 0x10000000,
UTP_DEVICE_MANAGEMENT_FUNCTION = 0x20000000,
UTP_REQ_DESC_INT_CMD = 0x01000000,
+ UTP_REQ_DESC_CRYPTO_ENABLE_CMD = 0x00800000,
};
/* UTP Transfer Request Data Direction (DD) */
@@ -356,6 +416,9 @@ enum {
OCS_PEER_COMM_FAILURE = 0x5,
OCS_ABORTED = 0x6,
OCS_FATAL_ERROR = 0x7,
+ OCS_DEVICE_FATAL_ERROR = 0x8,
+ OCS_INVALID_CRYPTO_CONFIG = 0x9,
+ OCS_GENERAL_CRYPTO_ERROR = 0xA,
OCS_INVALID_COMMAND_STATUS = 0x0F,
MASK_OCS = 0x0F,
};
--
2.22.0.410.gd8fdbe21b5-goog
^ permalink raw reply related
* [PATCH 5/8] scsi: ufs: UFS crypto API
From: Satya Tangirala @ 2019-07-10 22:56 UTC (permalink / raw)
To: linux-block, linux-scsi, linux-fscrypt, linux-fsdevel,
linux-f2fs-devel
Cc: Parshuram Raju Thombare, Ladvine D Almeida, Barani Muthukumaran,
Kuohong Wang, Greg Kroah-Hartman, Satya Tangirala
In-Reply-To: <20190710225609.192252-1-satyat@google.com>
Introduce functions to manipulate UFS inline encryption hardware
in line with the JEDEC UFSHCI v2.1 specification and to work with the
block keyslot manager.
Signed-off-by: Satya Tangirala <satyat@google.com>
---
drivers/scsi/ufs/Kconfig | 10 +
drivers/scsi/ufs/Makefile | 1 +
drivers/scsi/ufs/ufshcd-crypto.c | 435 +++++++++++++++++++++++++++++++
drivers/scsi/ufs/ufshcd-crypto.h | 86 ++++++
drivers/scsi/ufs/ufshcd.h | 18 ++
5 files changed, 550 insertions(+)
create mode 100644 drivers/scsi/ufs/ufshcd-crypto.c
create mode 100644 drivers/scsi/ufs/ufshcd-crypto.h
diff --git a/drivers/scsi/ufs/Kconfig b/drivers/scsi/ufs/Kconfig
index 0b845ab7c3bf..861aabfe791b 100644
--- a/drivers/scsi/ufs/Kconfig
+++ b/drivers/scsi/ufs/Kconfig
@@ -150,3 +150,13 @@ config SCSI_UFS_BSG
Select this if you need a bsg device node for your UFS controller.
If unsure, say N.
+
+config SCSI_UFS_CRYPTO
+ bool "UFS Crypto Engine Support"
+ depends on SCSI_UFSHCD && BLK_INLINE_ENCRYPTION
+ help
+ Enable Crypto Engine Support in UFS.
+ Enabling this makes it possible for the kernel to use the crypto
+ capabilities of the UFS device (if present) to perform crypto
+ operations on data being transferred to/from the device.
+
diff --git a/drivers/scsi/ufs/Makefile b/drivers/scsi/ufs/Makefile
index 2a9097939bcb..094c39989a37 100644
--- a/drivers/scsi/ufs/Makefile
+++ b/drivers/scsi/ufs/Makefile
@@ -11,3 +11,4 @@ obj-$(CONFIG_SCSI_UFSHCD_PCI) += ufshcd-pci.o
obj-$(CONFIG_SCSI_UFSHCD_PLATFORM) += ufshcd-pltfrm.o
obj-$(CONFIG_SCSI_UFS_HISI) += ufs-hisi.o
obj-$(CONFIG_SCSI_UFS_MEDIATEK) += ufs-mediatek.o
+ufshcd-core-$(CONFIG_SCSI_UFS_CRYPTO) += ufshcd-crypto.o
diff --git a/drivers/scsi/ufs/ufshcd-crypto.c b/drivers/scsi/ufs/ufshcd-crypto.c
new file mode 100644
index 000000000000..834c57ca912a
--- /dev/null
+++ b/drivers/scsi/ufs/ufshcd-crypto.c
@@ -0,0 +1,435 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright 2019 Google LLC
+ */
+
+#include <crypto/algapi.h>
+
+#include "ufshcd.h"
+#include "ufshcd-crypto.h"
+
+static bool ufshcd_cap_idx_valid(struct ufs_hba *hba, unsigned int cap_idx)
+{
+ return cap_idx < hba->crypto_capabilities.num_crypto_cap;
+}
+
+static u8 get_data_unit_size_mask(unsigned int data_unit_size)
+{
+ if (data_unit_size < 512 || data_unit_size > 65536 ||
+ !is_power_of_2(data_unit_size))
+ return 0;
+
+ return data_unit_size / 512;
+}
+
+static size_t get_keysize_bytes(enum ufs_crypto_key_size size)
+{
+ switch (size) {
+ case UFS_CRYPTO_KEY_SIZE_128: return 16;
+ case UFS_CRYPTO_KEY_SIZE_192: return 24;
+ case UFS_CRYPTO_KEY_SIZE_256: return 32;
+ case UFS_CRYPTO_KEY_SIZE_512: return 64;
+ default: return 0;
+ }
+}
+
+static int ufshcd_crypto_cap_find(void *hba_p,
+ enum blk_crypt_mode_num crypt_mode,
+ unsigned int data_unit_size)
+{
+ struct ufs_hba *hba = hba_p;
+ enum ufs_crypto_alg ufs_alg;
+ u8 data_unit_mask;
+ int cap_idx;
+ enum ufs_crypto_key_size ufs_key_size;
+ union ufs_crypto_cap_entry *ccap_array = hba->crypto_cap_array;
+
+ if (!ufshcd_hba_is_crypto_supported(hba))
+ return -EINVAL;
+
+ switch (crypt_mode) {
+ case BLK_ENCRYPTION_MODE_AES_256_XTS:
+ ufs_alg = UFS_CRYPTO_ALG_AES_XTS;
+ ufs_key_size = UFS_CRYPTO_KEY_SIZE_256;
+ break;
+ /*
+ * case BLK_CRYPTO_ALG_BITLOCKER_AES_CBC:
+ * ufs_alg = UFS_CRYPTO_ALG_BITLOCKER_AES_CBC;
+ * break;
+ * case INLINECRYPT_ALG_AES_ECB:
+ * ufs_alg = UFS_CRYPTO_ALG_AES_ECB;
+ * break;
+ * case INLINECRYPT_ALG_ESSIV_AES_CBC:
+ * ufs_alg = UFS_CRYPTO_ALG_ESSIV_AES_CBC;
+ * break;
+ */
+ default: return -EINVAL;
+ }
+
+ data_unit_mask = get_data_unit_size_mask(data_unit_size);
+
+ /*
+ * TODO: We can replace this for loop entirely by constructing
+ * a table on init that translates blk_crypt_mode to
+ * ufs crypt alg numbers. (By assuming that each alg/keysize combo
+ * appears only once in the ufs crypto caps array.)
+ */
+ for (cap_idx = 0; cap_idx < hba->crypto_capabilities.num_crypto_cap;
+ cap_idx++) {
+ if (ccap_array[cap_idx].algorithm_id == ufs_alg &&
+ (ccap_array[cap_idx].sdus_mask & data_unit_mask) &&
+ ccap_array[cap_idx].key_size == ufs_key_size)
+ return cap_idx;
+ }
+
+ return -EINVAL;
+}
+
+/**
+ * ufshcd_crypto_cfg_entry_write_key - Write a key into a crypto_cfg_entry
+ *
+ * Writes the key with the appropriate format - for AES_XTS,
+ * the first half of the key is copied as is, the second half is
+ * copied with an offset halfway into the cfg->crypto_key array.
+ * For the other supported crypto algs, the key is just copied.
+ *
+ * @cfg: The crypto config to write to
+ * @key: The key to write
+ * @cap: The crypto capability (which specifies the crypto alg and key size)
+ *
+ * Returns 0 on success, or -EINVAL
+ */
+static int ufshcd_crypto_cfg_entry_write_key(union ufs_crypto_cfg_entry *cfg,
+ const u8 *key,
+ union ufs_crypto_cap_entry cap)
+{
+ size_t key_size_bytes = get_keysize_bytes(cap.key_size);
+
+ if (key_size_bytes == 0)
+ return -EINVAL;
+
+ switch (cap.algorithm_id) {
+ case UFS_CRYPTO_ALG_AES_XTS:
+ key_size_bytes *= 2;
+ if (key_size_bytes > UFS_CRYPTO_KEY_MAX_SIZE)
+ return -EINVAL;
+
+ memcpy(cfg->crypto_key, key, key_size_bytes/2);
+ memcpy(cfg->crypto_key + UFS_CRYPTO_KEY_MAX_SIZE/2,
+ key + key_size_bytes/2, key_size_bytes/2);
+ return 0;
+ case UFS_CRYPTO_ALG_BITLOCKER_AES_CBC: // fallthrough
+ case UFS_CRYPTO_ALG_AES_ECB: // fallthrough
+ case UFS_CRYPTO_ALG_ESSIV_AES_CBC:
+ memcpy(cfg->crypto_key, key, key_size_bytes);
+ return 0;
+ }
+
+ return -EINVAL;
+}
+
+static void program_key(struct ufs_hba *hba,
+ const union ufs_crypto_cfg_entry *cfg,
+ int slot)
+{
+ int i;
+ u32 slot_offset = hba->crypto_cfg_register + slot * sizeof(*cfg);
+
+ /* Clear the dword 16 */
+ ufshcd_writel(hba, 0, slot_offset + 16 * sizeof(cfg->reg_val[0]));
+ /* Ensure that CFGE is cleared before programming the key */
+ wmb();
+ for (i = 0; i < 16; i++) {
+ ufshcd_writel(hba, le32_to_cpu(cfg->reg_val[i]),
+ slot_offset + i * sizeof(cfg->reg_val[0]));
+ /* Spec says each dword in key must be written sequentially */
+ wmb();
+ }
+ /* Write dword 17 */
+ ufshcd_writel(hba, le32_to_cpu(cfg->reg_val[17]),
+ slot_offset + 17 * sizeof(cfg->reg_val[0]));
+ /* Dword 16 must be written last */
+ wmb();
+ /* Write dword 16 */
+ ufshcd_writel(hba, le32_to_cpu(cfg->reg_val[16]),
+ slot_offset + 16 * sizeof(cfg->reg_val[0]));
+ wmb();
+}
+
+static int ufshcd_crypto_keyslot_program(void *hba_p, const u8 *key,
+ enum blk_crypt_mode_num crypt_mode,
+ unsigned int data_unit_size,
+ unsigned int slot)
+{
+ struct ufs_hba *hba = hba_p;
+ int err = 0;
+ u8 data_unit_mask;
+ union ufs_crypto_cfg_entry cfg;
+ union ufs_crypto_cfg_entry *cfg_arr = hba->crypto_cfgs;
+ int cap_idx;
+
+ cap_idx = ufshcd_crypto_cap_find(hba_p, crypt_mode,
+ data_unit_size);
+
+ if (!ufshcd_is_crypto_enabled(hba) ||
+ !ufshcd_keyslot_valid(hba, slot) ||
+ !ufshcd_cap_idx_valid(hba, cap_idx))
+ return -EINVAL;
+
+ data_unit_mask = get_data_unit_size_mask(data_unit_size);
+
+ if (!(data_unit_mask & hba->crypto_cap_array[cap_idx].sdus_mask))
+ return -EINVAL;
+
+ memset(&cfg, 0, sizeof(cfg));
+ cfg.data_unit_size = data_unit_mask;
+ cfg.crypto_cap_idx = cap_idx;
+ cfg.config_enable |= UFS_CRYPTO_CONFIGURATION_ENABLE;
+
+ err = ufshcd_crypto_cfg_entry_write_key(&cfg, key,
+ hba->crypto_cap_array[cap_idx]);
+ if (err)
+ return err;
+
+ program_key(hba, &cfg, slot);
+
+ memcpy(&cfg_arr[slot], &cfg, sizeof(cfg));
+ memzero_explicit(&cfg, sizeof(cfg));
+
+ return 0;
+}
+
+static int ufshcd_crypto_keyslot_find(void *hba_p,
+ const u8 *key,
+ enum blk_crypt_mode_num crypt_mode,
+ unsigned int data_unit_size)
+{
+ struct ufs_hba *hba = hba_p;
+ int err = 0;
+ int slot;
+ u8 data_unit_mask;
+ union ufs_crypto_cfg_entry cfg;
+ union ufs_crypto_cfg_entry *cfg_arr = hba->crypto_cfgs;
+ int cap_idx;
+
+ cap_idx = ufshcd_crypto_cap_find(hba_p, crypt_mode,
+ data_unit_size);
+
+ if (!ufshcd_is_crypto_enabled(hba) ||
+ !ufshcd_cap_idx_valid(hba, cap_idx))
+ return -EINVAL;
+
+ data_unit_mask = get_data_unit_size_mask(data_unit_size);
+
+ if (!(data_unit_mask & hba->crypto_cap_array[cap_idx].sdus_mask))
+ return -EINVAL;
+
+ memset(&cfg, 0, sizeof(cfg));
+ err = ufshcd_crypto_cfg_entry_write_key(&cfg, key,
+ hba->crypto_cap_array[cap_idx]);
+
+ if (err)
+ return -EINVAL;
+
+ for (slot = 0; slot < NUM_KEYSLOTS(hba); slot++) {
+ if ((cfg_arr[slot].config_enable &
+ UFS_CRYPTO_CONFIGURATION_ENABLE) &&
+ data_unit_mask == cfg_arr[slot].data_unit_size &&
+ cap_idx == cfg_arr[slot].crypto_cap_idx &&
+ !crypto_memneq(&cfg.crypto_key, cfg_arr[slot].crypto_key,
+ UFS_CRYPTO_KEY_MAX_SIZE)) {
+ memzero_explicit(&cfg, sizeof(cfg));
+ return slot;
+ }
+ }
+
+ memzero_explicit(&cfg, sizeof(cfg));
+ return -ENOKEY;
+}
+
+static int ufshcd_crypto_keyslot_evict(void *hba_p, const u8 *key,
+ enum blk_crypt_mode_num crypt_mode,
+ unsigned int data_unit_size,
+ unsigned int slot)
+{
+ struct ufs_hba *hba = hba_p;
+ int i = 0;
+ u32 reg_base;
+ union ufs_crypto_cfg_entry *cfg_arr = hba->crypto_cfgs;
+
+ if (!ufshcd_is_crypto_enabled(hba) ||
+ !ufshcd_keyslot_valid(hba, slot))
+ return -EINVAL;
+
+ memset(&cfg_arr[slot], 0, sizeof(cfg_arr[slot]));
+ reg_base = hba->crypto_cfg_register + slot * sizeof(cfg_arr[0]);
+
+ /*
+ * Clear the crypto cfg on the device. Clearing CFGE
+ * might not be sufficient, so just clear the entire cfg.
+ */
+ for (i = 0; i < sizeof(cfg_arr[0]); i += sizeof(__le32))
+ ufshcd_writel(hba, 0, reg_base + i);
+ wmb();
+
+ return 0;
+}
+
+static bool ufshcd_crypt_mode_supported(void *hba_p,
+ enum blk_crypt_mode_num crypt_mode,
+ unsigned int data_unit_size)
+{
+ return ufshcd_crypto_cap_find(hba_p, crypt_mode, data_unit_size) >= 0;
+}
+
+void ufshcd_crypto_enable(struct ufs_hba *hba)
+{
+ union ufs_crypto_cfg_entry *cfg_arr = hba->crypto_cfgs;
+ int slot;
+
+ if (!ufshcd_hba_is_crypto_supported(hba))
+ return;
+
+ hba->caps |= UFSHCD_CAP_CRYPTO;
+ /*
+ * Reset might clear all keys, so reprogram all the keys.
+ * Also serves to clear keys on driver init.
+ */
+ for (slot = 0; slot < NUM_KEYSLOTS(hba); slot++)
+ program_key(hba, &cfg_arr[slot], slot);
+}
+
+void ufshcd_crypto_disable(struct ufs_hba *hba)
+{
+ hba->caps &= ~UFSHCD_CAP_CRYPTO;
+}
+
+
+/**
+ * ufshcd_hba_init_crypto - Read crypto capabilities, init crypto fields in hba
+ * @hba: Per adapter instance
+ *
+ * Returns 0 on success. Returns -ENODEV if such capabilties don't exist, and
+ * -ENOMEM upon OOM.
+ */
+int ufshcd_hba_init_crypto(struct ufs_hba *hba)
+{
+ int cap_idx = 0;
+ int err = 0;
+
+ /* Default to disabling crypto */
+ hba->caps &= ~UFSHCD_CAP_CRYPTO;
+
+ if (!(hba->capabilities & MASK_CRYPTO_SUPPORT)) {
+ err = -ENODEV;
+ goto out;
+ }
+
+ /*
+ * Crypto Capabilities should never be 0, because the
+ * config_array_ptr > 04h. So we use a 0 value to indicate that
+ * crypto init failed, and can't be enabled.
+ */
+ hba->crypto_capabilities.reg_val =
+ cpu_to_le32(ufshcd_readl(hba, REG_UFS_CCAP));
+ hba->crypto_cfg_register =
+ (u32)hba->crypto_capabilities.config_array_ptr * 0x100;
+ hba->crypto_cap_array =
+ devm_kcalloc(hba->dev,
+ hba->crypto_capabilities.num_crypto_cap,
+ sizeof(hba->crypto_cap_array[0]),
+ GFP_KERNEL);
+ if (!hba->crypto_cap_array) {
+ err = -ENOMEM;
+ goto out;
+ }
+
+ hba->crypto_cfgs =
+ devm_kcalloc(hba->dev,
+ hba->crypto_capabilities.config_count + 1,
+ sizeof(hba->crypto_cfgs[0]),
+ GFP_KERNEL);
+ if (!hba->crypto_cfgs) {
+ err = -ENOMEM;
+ goto out_cfg_mem;
+ }
+
+ /*
+ * Store all the capabilities now so that we don't need to repeatedly
+ * access the device each time we want to know its capabilities
+ */
+ for (cap_idx = 0; cap_idx < hba->crypto_capabilities.num_crypto_cap;
+ cap_idx++) {
+ hba->crypto_cap_array[cap_idx].reg_val =
+ cpu_to_le32(ufshcd_readl(hba,
+ REG_UFS_CRYPTOCAP +
+ cap_idx * sizeof(__le32)));
+ }
+
+ hba->ksm = NULL;
+ mutex_init(&hba->ksm_lock);
+ hba->ksm_num_refs = 0;
+
+ return 0;
+out_cfg_mem:
+ devm_kfree(hba->dev, hba->crypto_cap_array);
+out:
+ // TODO: print error?
+ /* Indicate that init failed by setting crypto_capabilities to 0 */
+ hba->crypto_capabilities.reg_val = 0;
+ return err;
+}
+
+static const struct keyslot_mgmt_ll_ops ufshcd_ksm_ops = {
+ .keyslot_program = ufshcd_crypto_keyslot_program,
+ .keyslot_evict = ufshcd_crypto_keyslot_evict,
+ .keyslot_find = ufshcd_crypto_keyslot_find,
+ .crypt_mode_supported = ufshcd_crypt_mode_supported,
+};
+
+void ufshcd_crypto_setup_rq_keyslot_manager(struct ufs_hba *hba,
+ struct request_queue *q)
+{
+ if (!ufshcd_hba_is_crypto_supported(hba))
+ return;
+
+ if (q) {
+ mutex_lock(&hba->ksm_lock);
+ if (!hba->ksm) {
+ hba->ksm = keyslot_manager_create(
+ hba->crypto_capabilities.config_count + 1,
+ &ufshcd_ksm_ops, hba);
+ hba->ksm_num_refs = 0;
+ }
+ hba->ksm_num_refs++;
+ mutex_unlock(&hba->ksm_lock);
+ q->ksm = hba->ksm;
+ }
+ /*
+ * If we fail we make it look like
+ * crypto is not supported, which will avoid issues
+ * with reset
+ */
+ if (!q || !q->ksm) {
+ ufshcd_crypto_disable(hba);
+ hba->crypto_capabilities.reg_val = 0;
+ devm_kfree(hba->dev, hba->crypto_cap_array);
+ devm_kfree(hba->dev, hba->crypto_cfgs);
+ }
+}
+
+void ufshcd_crypto_destroy_rq_keyslot_manager(struct ufs_hba *hba,
+ struct request_queue *q)
+{
+ if (q && q->ksm) {
+ q->ksm = NULL;
+ mutex_lock(&hba->ksm_lock);
+ hba->ksm_num_refs--;
+ if (hba->ksm_num_refs == 0) {
+ keyslot_manager_destroy(hba->ksm);
+ hba->ksm = NULL;
+ }
+ mutex_unlock(&hba->ksm_lock);
+ }
+}
+
diff --git a/drivers/scsi/ufs/ufshcd-crypto.h b/drivers/scsi/ufs/ufshcd-crypto.h
new file mode 100644
index 000000000000..73ddc8e493fb
--- /dev/null
+++ b/drivers/scsi/ufs/ufshcd-crypto.h
@@ -0,0 +1,86 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright 2019 Google LLC
+ */
+
+#ifndef _UFSHCD_CRYPTO_H
+#define _UFSHCD_CRYPTO_H
+
+struct ufs_hba;
+
+#ifdef CONFIG_SCSI_UFS_CRYPTO
+#include <linux/keyslot-manager.h>
+
+#include "ufshci.h"
+
+#define NUM_KEYSLOTS(hba) (hba->crypto_capabilities.config_count + 1)
+
+static inline bool ufshcd_keyslot_valid(struct ufs_hba *hba, unsigned int slot)
+{
+ /*
+ * The actual number of configurations supported is (CFGC+1), so slot
+ * numbers range from 0 to config_count inclusive.
+ */
+ return slot < NUM_KEYSLOTS(hba);
+}
+
+static inline bool ufshcd_hba_is_crypto_supported(struct ufs_hba *hba)
+{
+ return hba->crypto_capabilities.reg_val != 0;
+}
+
+static inline bool ufshcd_is_crypto_enabled(struct ufs_hba *hba)
+{
+ return hba->caps & UFSHCD_CAP_CRYPTO;
+}
+
+void ufshcd_crypto_enable(struct ufs_hba *hba);
+
+void ufshcd_crypto_disable(struct ufs_hba *hba);
+
+int ufshcd_hba_init_crypto(struct ufs_hba *hba);
+
+void ufshcd_crypto_setup_rq_keyslot_manager(struct ufs_hba *hba,
+ struct request_queue *q);
+
+void ufshcd_crypto_destroy_rq_keyslot_manager(struct ufs_hba *hba,
+ struct request_queue *q);
+
+#else /* CONFIG_SCSI_UFS_CRYPTO */
+
+static inline bool ufshcd_keyslot_valid(struct ufs_hba *hba,
+ unsigned int slot)
+{
+ return false;
+}
+
+static inline bool ufshcd_hba_is_crypto_supported(struct ufs_hba *hba)
+{
+ return false;
+}
+
+static inline bool ufshcd_is_crypto_enabled(struct ufs_hba *hba)
+{
+ return false;
+}
+
+static inline void ufshcd_crypto_enable(struct ufs_hba *hba) { }
+
+static inline void ufshcd_crypto_disable(struct ufs_hba *hba) { }
+
+static inline int ufshcd_hba_init_crypto(struct ufs_hba *hba)
+{
+ return 0;
+}
+
+static inline void ufshcd_crypto_setup_rq_keyslot_manager(
+ struct ufs_hba *hba,
+ struct request_queue *q) { }
+
+static inline void ufshcd_crypto_destroy_rq_keyslot_manager(
+ struct ufs_hba *hba,
+ struct request_queue *q) { }
+
+#endif /* CONFIG_SCSI_UFS_CRYPTO */
+
+#endif /* _UFSHCD_CRYPTO_H */
diff --git a/drivers/scsi/ufs/ufshcd.h b/drivers/scsi/ufs/ufshcd.h
index d3b6a6b57a37..ef62b3ba03fd 100644
--- a/drivers/scsi/ufs/ufshcd.h
+++ b/drivers/scsi/ufs/ufshcd.h
@@ -501,6 +501,13 @@ struct ufs_stats {
* @is_urgent_bkops_lvl_checked: keeps track if the urgent bkops level for
* device is known or not.
* @scsi_block_reqs_cnt: reference counting for scsi block requests
+ * @crypto_capabilities: Content of crypto capabilities register (0x100)
+ * @crypto_cap_array: Array of crypto capabilities
+ * @crypto_cfg_register: Start of the crypto cfg array
+ * @crypto_cfgs: Array of crypto configurations (i.e. config for each slot)
+ * @ksm: the keyslot manager tied to this hba
+ * @ksm_lock: lock to protect initialization and refcount of ksm
+ * @ksm_num_refs: refcount for ksm
*/
struct ufs_hba {
void __iomem *mmio_base;
@@ -711,6 +718,17 @@ struct ufs_hba {
struct device bsg_dev;
struct request_queue *bsg_queue;
+
+#ifdef CONFIG_SCSI_UFS_CRYPTO
+ /* crypto */
+ union ufs_crypto_capabilities crypto_capabilities;
+ union ufs_crypto_cap_entry *crypto_cap_array;
+ u32 crypto_cfg_register;
+ union ufs_crypto_cfg_entry *crypto_cfgs;
+ struct keyslot_manager *ksm;
+ struct mutex ksm_lock;
+ unsigned int ksm_num_refs;
+#endif /* CONFIG_SCSI_UFS_CRYPTO */
};
/* Returns true if clocks can be gated. Otherwise false */
--
2.22.0.410.gd8fdbe21b5-goog
^ permalink raw reply related
* [PATCH 6/8] scsi: ufs: Add inline encryption support to UFS
From: Satya Tangirala @ 2019-07-10 22:56 UTC (permalink / raw)
To: linux-block, linux-scsi, linux-fscrypt, linux-fsdevel,
linux-f2fs-devel
Cc: Parshuram Raju Thombare, Ladvine D Almeida, Barani Muthukumaran,
Kuohong Wang, Greg Kroah-Hartman, Satya Tangirala
In-Reply-To: <20190710225609.192252-1-satyat@google.com>
Wire up ufshcd.c with the UFS Crypto API, the block layer inline
encryption additions and the keyslot manager.
Signed-off-by: Satya Tangirala <satyat@google.com>
---
drivers/scsi/ufs/ufshcd.c | 82 ++++++++++++++++++++++++++++++++++++---
drivers/scsi/ufs/ufshcd.h | 6 +++
2 files changed, 83 insertions(+), 5 deletions(-)
diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
index 36413f97074d..492e9db6b50f 100644
--- a/drivers/scsi/ufs/ufshcd.c
+++ b/drivers/scsi/ufs/ufshcd.c
@@ -47,6 +47,7 @@
#include "unipro.h"
#include "ufs-sysfs.h"
#include "ufs_bsg.h"
+#include "ufshcd-crypto.h"
#define CREATE_TRACE_POINTS
#include <trace/events/ufs.h>
@@ -855,7 +856,14 @@ static void ufshcd_enable_run_stop_reg(struct ufs_hba *hba)
*/
static inline void ufshcd_hba_start(struct ufs_hba *hba)
{
- ufshcd_writel(hba, CONTROLLER_ENABLE, REG_CONTROLLER_ENABLE);
+ u32 val = CONTROLLER_ENABLE;
+
+ if (ufshcd_hba_is_crypto_supported(hba)) {
+ ufshcd_crypto_enable(hba);
+ val |= CRYPTO_GENERAL_ENABLE;
+ }
+
+ ufshcd_writel(hba, val, REG_CONTROLLER_ENABLE);
}
/**
@@ -2209,9 +2217,21 @@ static void ufshcd_prepare_req_desc_hdr(struct ufshcd_lrb *lrbp,
dword_0 |= UTP_REQ_DESC_INT_CMD;
/* Transfer request descriptor header fields */
+ if (lrbp->crypto_enable) {
+ dword_0 |= UTP_REQ_DESC_CRYPTO_ENABLE_CMD;
+ dword_0 |= lrbp->crypto_key_slot;
+ req_desc->header.dword_1 =
+ cpu_to_le32((u32)lrbp->data_unit_num);
+ req_desc->header.dword_3 =
+ cpu_to_le32((u32)(lrbp->data_unit_num >> 32));
+ } else {
+ /* dword_1 and dword_3 are reserved, hence they are set to 0 */
+ req_desc->header.dword_1 = 0;
+ req_desc->header.dword_3 = 0;
+ }
+
req_desc->header.dword_0 = cpu_to_le32(dword_0);
- /* dword_1 is reserved, hence it is set to 0 */
- req_desc->header.dword_1 = 0;
+
/*
* assigning invalid value for command status. Controller
* updates OCS on command completion, with the command
@@ -2219,8 +2239,6 @@ static void ufshcd_prepare_req_desc_hdr(struct ufshcd_lrb *lrbp,
*/
req_desc->header.dword_2 =
cpu_to_le32(OCS_INVALID_COMMAND_STATUS);
- /* dword_3 is reserved, hence it is set to 0 */
- req_desc->header.dword_3 = 0;
req_desc->prd_table_length = 0;
}
@@ -2380,6 +2398,37 @@ static inline u16 ufshcd_upiu_wlun_to_scsi_wlun(u8 upiu_wlun_id)
return (upiu_wlun_id & ~UFS_UPIU_WLUN_ID) | SCSI_W_LUN_BASE;
}
+static inline int ufshcd_prepare_lrbp_crypto(struct ufs_hba *hba,
+ struct scsi_cmnd *cmd,
+ struct ufshcd_lrb *lrbp)
+{
+ int key_slot;
+
+ if (!cmd->request->bio ||
+ !bio_crypt_should_process(cmd->request->bio, cmd->request->q)) {
+ lrbp->crypto_enable = false;
+ return 0;
+ }
+
+ if (WARN_ON(!ufshcd_is_crypto_enabled(hba))) {
+ /*
+ * Upper layer asked us to do inline encryption
+ * but that isn't enabled, so we fail this request.
+ */
+ return -EINVAL;
+ }
+ key_slot = bio_crypt_get_keyslot(cmd->request->bio);
+ if (!ufshcd_keyslot_valid(hba, key_slot))
+ return -EINVAL;
+
+ lrbp->crypto_enable = true;
+ lrbp->crypto_key_slot = key_slot;
+ lrbp->data_unit_num = bio_crypt_data_unit_num(cmd->request->bio);
+
+ return 0;
+}
+
+
/**
* ufshcd_queuecommand - main entry point for SCSI requests
* @host: SCSI host pointer
@@ -2467,6 +2516,13 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd)
lrbp->task_tag = tag;
lrbp->lun = ufshcd_scsi_to_upiu_lun(cmd->device->lun);
lrbp->intr_cmd = !ufshcd_is_intr_aggr_allowed(hba) ? true : false;
+
+ err = ufshcd_prepare_lrbp_crypto(hba, cmd, lrbp);
+ if (err) {
+ lrbp->cmd = NULL;
+ clear_bit_unlock(tag, &hba->lrb_in_use);
+ goto out;
+ }
lrbp->req_abort_skip = false;
ufshcd_comp_scsi_upiu(hba, lrbp);
@@ -2500,6 +2556,7 @@ static int ufshcd_compose_dev_cmd(struct ufs_hba *hba,
lrbp->task_tag = tag;
lrbp->lun = 0; /* device management cmd is not specific to any LUN */
lrbp->intr_cmd = true; /* No interrupt aggregation */
+ lrbp->crypto_enable = false; /* No crypto operations */
hba->dev_cmd.type = cmd_type;
return ufshcd_comp_devman_upiu(hba, lrbp);
@@ -4192,6 +4249,8 @@ static inline void ufshcd_hba_stop(struct ufs_hba *hba, bool can_sleep)
{
int err;
+ ufshcd_crypto_disable(hba);
+
ufshcd_writel(hba, CONTROLLER_DISABLE, REG_CONTROLLER_ENABLE);
err = ufshcd_wait_for_register(hba, REG_CONTROLLER_ENABLE,
CONTROLLER_ENABLE, CONTROLLER_DISABLE,
@@ -4585,10 +4644,13 @@ static int ufshcd_change_queue_depth(struct scsi_device *sdev, int depth)
static int ufshcd_slave_configure(struct scsi_device *sdev)
{
struct request_queue *q = sdev->request_queue;
+ struct ufs_hba *hba = shost_priv(sdev->host);
blk_queue_update_dma_pad(q, PRDT_DATA_BYTE_COUNT_PAD - 1);
blk_queue_max_segment_size(q, PRDT_DATA_BYTE_COUNT_MAX);
+ ufshcd_crypto_setup_rq_keyslot_manager(hba, q);
+
return 0;
}
@@ -4599,6 +4661,7 @@ static int ufshcd_slave_configure(struct scsi_device *sdev)
static void ufshcd_slave_destroy(struct scsi_device *sdev)
{
struct ufs_hba *hba;
+ struct request_queue *q = sdev->request_queue;
hba = shost_priv(sdev->host);
/* Drop the reference as it won't be needed anymore */
@@ -4609,6 +4672,8 @@ static void ufshcd_slave_destroy(struct scsi_device *sdev)
hba->sdev_ufs_device = NULL;
spin_unlock_irqrestore(hba->host->host_lock, flags);
}
+
+ ufshcd_crypto_destroy_rq_keyslot_manager(hba, q);
}
/**
@@ -8293,6 +8358,13 @@ int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq)
goto exit_gating;
}
+ /* Init crypto */
+ err = ufshcd_hba_init_crypto(hba);
+ if (err) {
+ dev_err(hba->dev, "crypto setup failed\n");
+ goto out_remove_scsi_host;
+ }
+
/* Host controller enable */
err = ufshcd_hba_enable(hba);
if (err) {
diff --git a/drivers/scsi/ufs/ufshcd.h b/drivers/scsi/ufs/ufshcd.h
index ef62b3ba03fd..ab9c1191c206 100644
--- a/drivers/scsi/ufs/ufshcd.h
+++ b/drivers/scsi/ufs/ufshcd.h
@@ -167,6 +167,9 @@ struct ufs_pm_lvl_states {
* @intr_cmd: Interrupt command (doesn't participate in interrupt aggregation)
* @issue_time_stamp: time stamp for debug purposes
* @compl_time_stamp: time stamp for statistics
+ * @crypto_enable: whether or not the request needs inline crypto operations
+ * @crypto_key_slot: the key slot to use for inline crypto
+ * @data_unit_num: the data unit number for the first block for inline crypto
* @req_abort_skip: skip request abort task flag
*/
struct ufshcd_lrb {
@@ -191,6 +194,9 @@ struct ufshcd_lrb {
bool intr_cmd;
ktime_t issue_time_stamp;
ktime_t compl_time_stamp;
+ bool crypto_enable;
+ u8 crypto_key_slot;
+ u64 data_unit_num;
bool req_abort_skip;
};
--
2.22.0.410.gd8fdbe21b5-goog
^ permalink raw reply related
* [PATCH 7/8] fscrypt: wire up fscrypt to use blk-crypto
From: Satya Tangirala @ 2019-07-10 22:56 UTC (permalink / raw)
To: linux-block, linux-scsi, linux-fscrypt, linux-fsdevel,
linux-f2fs-devel
Cc: Parshuram Raju Thombare, Ladvine D Almeida, Barani Muthukumaran,
Kuohong Wang, Greg Kroah-Hartman, Satya Tangirala
In-Reply-To: <20190710225609.192252-1-satyat@google.com>
Introduce fscrypt_set_bio_crypt_ctx for filesystems to call to set up
encryption contexts in bios, and fscrypt_evict_crypt_key to evict
the encryption context associated with an inode.
Inline encryption is controlled by a policy flag in the fscrypt_info
in the inode, and filesystems may check if an inode should use inline
encryption by calling fscrypt_inode_is_inline_crypted. Files can be marked
as inline encrypted from userspace by appropriately modifying the flags
(OR-ing FS_POLICY_FLAGS_INLINE_ENCRYPTION to it) in the fscrypt_policy
passed to fscrypt_ioctl_set_policy.
To test inline encryption with the fscrypt dummy context, add
ctx.flags |= FS_POLICY_FLAGS_INLINE_ENCRYPTION
when setting up the dummy context in fs/crypto/keyinfo.c.
Note that blk-crypto will fall back to software en/decryption in the
absence of inline crypto hardware, so setting up the ctx.flags in the
dummy context without inline crypto hardware serves as a test for
the software fallback in blk-crypto.
Signed-off-by: Satya Tangirala <satyat@google.com>
---
fs/crypto/Kconfig | 6 ++
fs/crypto/bio.c | 138 +++++++++++++++++++++++++++++++-----
fs/crypto/crypto.c | 4 ++
fs/crypto/fscrypt_private.h | 11 +++
fs/crypto/keyinfo.c | 94 +++++++++++++++++-------
fs/crypto/policy.c | 10 +++
include/linux/fscrypt.h | 62 ++++++++++++++++
include/uapi/linux/fs.h | 3 +-
8 files changed, 285 insertions(+), 43 deletions(-)
diff --git a/fs/crypto/Kconfig b/fs/crypto/Kconfig
index 24ed99e2eca0..738368394a32 100644
--- a/fs/crypto/Kconfig
+++ b/fs/crypto/Kconfig
@@ -15,3 +15,9 @@ config FS_ENCRYPTION
efficient since it avoids caching the encrypted and
decrypted pages in the page cache. Currently Ext4,
F2FS and UBIFS make use of this feature.
+
+config FS_ENCRYPTION_INLINE_CRYPT
+ bool "Enable fscrypt to use inline crypto"
+ depends on FS_ENCRYPTION && BLK_INLINE_ENCRYPTION
+ help
+ Enables fscrypt to use inline crypto hardware if available.
diff --git a/fs/crypto/bio.c b/fs/crypto/bio.c
index b46021ebde85..0e16189e1679 100644
--- a/fs/crypto/bio.c
+++ b/fs/crypto/bio.c
@@ -24,6 +24,9 @@
#include <linux/module.h>
#include <linux/bio.h>
#include <linux/namei.h>
+#include <linux/keyslot-manager.h>
+#include <linux/blkdev.h>
+#include <crypto/algapi.h>
#include "fscrypt_private.h"
static void __fscrypt_decrypt_bio(struct bio *bio, bool done)
@@ -34,7 +37,7 @@ static void __fscrypt_decrypt_bio(struct bio *bio, bool done)
bio_for_each_segment_all(bv, bio, iter_all) {
struct page *page = bv->bv_page;
int ret = fscrypt_decrypt_page(page->mapping->host, page,
- PAGE_SIZE, 0, page->index);
+ PAGE_SIZE, 0, page->index);
if (ret)
SetPageError(page);
@@ -94,29 +97,33 @@ EXPORT_SYMBOL(fscrypt_pullback_bio_page);
int fscrypt_zeroout_range(const struct inode *inode, pgoff_t lblk,
sector_t pblk, unsigned int len)
{
- struct fscrypt_ctx *ctx;
+ struct fscrypt_ctx *ctx = NULL;
struct page *ciphertext_page = NULL;
struct bio *bio;
int ret, err = 0;
BUG_ON(inode->i_sb->s_blocksize != PAGE_SIZE);
- ctx = fscrypt_get_ctx(GFP_NOFS);
- if (IS_ERR(ctx))
- return PTR_ERR(ctx);
+ if (!fscrypt_inode_is_inline_crypted(inode)) {
+ ctx = fscrypt_get_ctx(GFP_NOFS);
+ if (IS_ERR(ctx))
+ return PTR_ERR(ctx);
- ciphertext_page = fscrypt_alloc_bounce_page(ctx, GFP_NOWAIT);
- if (IS_ERR(ciphertext_page)) {
- err = PTR_ERR(ciphertext_page);
- goto errout;
+ ciphertext_page = fscrypt_alloc_bounce_page(ctx, GFP_NOWAIT);
+ if (IS_ERR(ciphertext_page)) {
+ err = PTR_ERR(ciphertext_page);
+ goto errout;
+ }
}
while (len--) {
- err = fscrypt_do_page_crypto(inode, FS_ENCRYPT, lblk,
+ if (!fscrypt_inode_is_inline_crypted(inode)) {
+ err = fscrypt_do_page_crypto(inode, FS_ENCRYPT, lblk,
ZERO_PAGE(0), ciphertext_page,
PAGE_SIZE, 0, GFP_NOFS);
- if (err)
- goto errout;
+ if (err)
+ goto errout;
+ }
bio = bio_alloc(GFP_NOWAIT, 1);
if (!bio) {
@@ -127,8 +134,14 @@ int fscrypt_zeroout_range(const struct inode *inode, pgoff_t lblk,
bio->bi_iter.bi_sector =
pblk << (inode->i_sb->s_blocksize_bits - 9);
bio_set_op_attrs(bio, REQ_OP_WRITE, 0);
- ret = bio_add_page(bio, ciphertext_page,
- inode->i_sb->s_blocksize, 0);
+ if (!fscrypt_inode_is_inline_crypted(inode)) {
+ ret = bio_add_page(bio, ciphertext_page,
+ inode->i_sb->s_blocksize, 0);
+ } else {
+ ret = bio_add_page(bio, ZERO_PAGE(0),
+ inode->i_sb->s_blocksize, 0);
+ }
+
if (ret != inode->i_sb->s_blocksize) {
/* should never happen! */
WARN_ON(1);
@@ -136,9 +149,12 @@ int fscrypt_zeroout_range(const struct inode *inode, pgoff_t lblk,
err = -EIO;
goto errout;
}
- err = submit_bio_wait(bio);
- if (err == 0 && bio->bi_status)
- err = -EIO;
+ err = fscrypt_set_bio_crypt_ctx(inode, bio, pblk);
+ if (!err) {
+ err = submit_bio_wait(bio);
+ if (err == 0 && bio->bi_status)
+ err = -EIO;
+ }
bio_put(bio);
if (err)
goto errout;
@@ -147,7 +163,93 @@ int fscrypt_zeroout_range(const struct inode *inode, pgoff_t lblk,
}
err = 0;
errout:
- fscrypt_release_ctx(ctx);
+ if (!fscrypt_inode_is_inline_crypted(inode))
+ fscrypt_release_ctx(ctx);
return err;
}
EXPORT_SYMBOL(fscrypt_zeroout_range);
+
+#ifdef CONFIG_FS_ENCRYPTION_INLINE_CRYPT
+static enum blk_crypt_mode_num
+get_blk_crypto_alg_for_fscryptalg(u8 fscrypt_alg)
+{
+ switch (fscrypt_alg) {
+ case FS_ENCRYPTION_MODE_AES_256_XTS:
+ return BLK_ENCRYPTION_MODE_AES_256_XTS;
+ default: return -EINVAL;
+ }
+}
+
+int fscrypt_set_bio_crypt_ctx(const struct inode *inode,
+ struct bio *bio, u64 data_unit_num)
+{
+ struct fscrypt_info *ci = inode->i_crypt_info;
+
+ /* If inode is not inline encrypted, nothing to do. */
+ if (!fscrypt_inode_is_inline_crypted(inode))
+ return 0;
+
+ return bio_crypt_set_ctx(bio, ci->ci_master_key->mk_raw,
+ get_blk_crypto_alg_for_fscryptalg(ci->ci_data_mode),
+ data_unit_num,
+ inode->i_blkbits);
+}
+EXPORT_SYMBOL(fscrypt_set_bio_crypt_ctx);
+
+void fscrypt_unset_bio_crypt_ctx(struct bio *bio)
+{
+ bio_crypt_free_ctx(bio);
+}
+EXPORT_SYMBOL(fscrypt_unset_bio_crypt_ctx);
+
+int fscrypt_evict_crypt_key(struct inode *inode)
+{
+ struct request_queue *q;
+ struct fscrypt_info *ci;
+
+ if (!inode)
+ return 0;
+
+ q = inode->i_sb->s_bdev->bd_queue;
+ ci = inode->i_crypt_info;
+
+ if (!q || !q->ksm || !ci ||
+ !fscrypt_inode_is_inline_crypted(inode)) {
+ return 0;
+ }
+
+ return keyslot_manager_evict_key(q->ksm,
+ ci->ci_master_key->mk_raw,
+ get_blk_crypto_alg_for_fscryptalg(
+ ci->ci_data_mode),
+ 1 << inode->i_blkbits);
+}
+EXPORT_SYMBOL(fscrypt_evict_crypt_key);
+
+bool fscrypt_inode_crypt_mergeable(const struct inode *inode_1,
+ const struct inode *inode_2)
+{
+ struct fscrypt_info *ci_1, *ci_2;
+ bool enc_1 = fscrypt_inode_is_inline_crypted(inode_1);
+ bool enc_2 = fscrypt_inode_is_inline_crypted(inode_2);
+
+ if (enc_1 != enc_2)
+ return false;
+
+ if (!enc_1)
+ return true;
+
+ if (inode_1 == inode_2)
+ return true;
+
+ ci_1 = inode_1->i_crypt_info;
+ ci_2 = inode_2->i_crypt_info;
+
+ return ci_1->ci_data_mode == ci_2->ci_data_mode &&
+ crypto_memneq(ci_1->ci_master_key->mk_raw,
+ ci_2->ci_master_key->mk_raw,
+ ci_1->ci_master_key->mk_mode->keysize) == 0;
+}
+EXPORT_SYMBOL(fscrypt_inode_crypt_mergeable);
+
+#endif /* FS_ENCRYPTION_INLINE_CRYPT */
diff --git a/fs/crypto/crypto.c b/fs/crypto/crypto.c
index 335a362ee446..58a01889fac7 100644
--- a/fs/crypto/crypto.c
+++ b/fs/crypto/crypto.c
@@ -302,6 +302,10 @@ int fscrypt_decrypt_page(const struct inode *inode, struct page *page,
if (!(inode->i_sb->s_cop->flags & FS_CFLG_OWN_PAGES))
BUG_ON(!PageLocked(page));
+ /* If we have HW encryption, then this page is already decrypted */
+ if (fscrypt_inode_is_inline_crypted(inode))
+ return 0;
+
return fscrypt_do_page_crypto(inode, FS_DECRYPT, lblk_num, page, page,
len, offs, GFP_NOFS);
}
diff --git a/fs/crypto/fscrypt_private.h b/fs/crypto/fscrypt_private.h
index 7da276159593..67a8ca058996 100644
--- a/fs/crypto/fscrypt_private.h
+++ b/fs/crypto/fscrypt_private.h
@@ -49,6 +49,17 @@ struct fscrypt_symlink_data {
char encrypted_path[1];
} __packed;
+/* Master key referenced by FS_POLICY_FLAG_DIRECT_KEY policy */
+struct fscrypt_master_key {
+ struct hlist_node mk_node;
+ refcount_t mk_refcount;
+ const struct fscrypt_mode *mk_mode;
+ struct crypto_skcipher *mk_ctfm;
+ u8 mk_descriptor[FS_KEY_DESCRIPTOR_SIZE];
+ u8 mk_raw[FS_MAX_KEY_SIZE];
+ struct super_block *mk_sb;
+};
+
/*
* fscrypt_info - the "encryption key" for an inode
*
diff --git a/fs/crypto/keyinfo.c b/fs/crypto/keyinfo.c
index dcd91a3fbe49..3a3b27b99706 100644
--- a/fs/crypto/keyinfo.c
+++ b/fs/crypto/keyinfo.c
@@ -25,6 +25,21 @@ static struct crypto_shash *essiv_hash_tfm;
static DEFINE_HASHTABLE(fscrypt_master_keys, 6); /* 6 bits = 64 buckets */
static DEFINE_SPINLOCK(fscrypt_master_keys_lock);
+#ifdef CONFIG_FS_ENCRYPTION_INLINE_CRYPT
+static inline bool flags_inline_crypted(u8 flags,
+ const struct inode *inode)
+{
+ return inode && (flags & FS_POLICY_FLAGS_INLINE_CRYPT) &&
+ S_ISREG(inode->i_mode);
+}
+#else
+static inline bool flags_inline_crypted(u8 flags,
+ const struct inode *inode)
+{
+ return false;
+}
+#endif /* CONFIG_FS_ENCRYPTION_INLINE_CRYPT */
+
/*
* Key derivation function. This generates the derived key by encrypting the
* master key with AES-128-ECB using the inode's nonce as the AES key.
@@ -220,6 +235,9 @@ static int find_and_derive_key(const struct inode *inode,
memcpy(derived_key, payload->raw, mode->keysize);
err = 0;
}
+ } else if (flags_inline_crypted(ctx->flags, inode)) {
+ memcpy(derived_key, payload->raw, mode->keysize);
+ err = 0;
} else {
err = derive_key_aes(payload->raw, ctx, derived_key,
mode->keysize);
@@ -269,16 +287,6 @@ allocate_skcipher_for_mode(struct fscrypt_mode *mode, const u8 *raw_key,
return ERR_PTR(err);
}
-/* Master key referenced by FS_POLICY_FLAG_DIRECT_KEY policy */
-struct fscrypt_master_key {
- struct hlist_node mk_node;
- refcount_t mk_refcount;
- const struct fscrypt_mode *mk_mode;
- struct crypto_skcipher *mk_ctfm;
- u8 mk_descriptor[FS_KEY_DESCRIPTOR_SIZE];
- u8 mk_raw[FS_MAX_KEY_SIZE];
-};
-
static void free_master_key(struct fscrypt_master_key *mk)
{
if (mk) {
@@ -287,13 +295,15 @@ static void free_master_key(struct fscrypt_master_key *mk)
}
}
-static void put_master_key(struct fscrypt_master_key *mk)
+static void put_master_key(struct fscrypt_master_key *mk,
+ struct inode *inode)
{
if (!refcount_dec_and_lock(&mk->mk_refcount, &fscrypt_master_keys_lock))
return;
hash_del(&mk->mk_node);
spin_unlock(&fscrypt_master_keys_lock);
+ fscrypt_evict_crypt_key(inode);
free_master_key(mk);
}
@@ -306,7 +316,9 @@ static void put_master_key(struct fscrypt_master_key *mk)
static struct fscrypt_master_key *
find_or_insert_master_key(struct fscrypt_master_key *to_insert,
const u8 *raw_key, const struct fscrypt_mode *mode,
- const struct fscrypt_info *ci)
+ const struct fscrypt_info *ci,
+ bool should_have_ctfm,
+ struct super_block *sb)
{
unsigned long hash_key;
struct fscrypt_master_key *mk;
@@ -329,6 +341,10 @@ find_or_insert_master_key(struct fscrypt_master_key *to_insert,
continue;
if (crypto_memneq(raw_key, mk->mk_raw, mode->keysize))
continue;
+ if (should_have_ctfm != (bool)mk->mk_ctfm)
+ continue;
+ if (sb != mk->mk_sb)
+ continue;
/* using existing tfm with same (descriptor, mode, raw_key) */
refcount_inc(&mk->mk_refcount);
spin_unlock(&fscrypt_master_keys_lock);
@@ -348,9 +364,11 @@ fscrypt_get_master_key(const struct fscrypt_info *ci, struct fscrypt_mode *mode,
{
struct fscrypt_master_key *mk;
int err;
+ bool inline_crypted = flags_inline_crypted(ci->ci_flags, inode);
/* Is there already a tfm for this key? */
- mk = find_or_insert_master_key(NULL, raw_key, mode, ci);
+ mk = find_or_insert_master_key(NULL, raw_key, mode, ci, !inline_crypted,
+ inode->i_sb);
if (mk)
return mk;
@@ -360,17 +378,21 @@ fscrypt_get_master_key(const struct fscrypt_info *ci, struct fscrypt_mode *mode,
return ERR_PTR(-ENOMEM);
refcount_set(&mk->mk_refcount, 1);
mk->mk_mode = mode;
- mk->mk_ctfm = allocate_skcipher_for_mode(mode, raw_key, inode);
- if (IS_ERR(mk->mk_ctfm)) {
- err = PTR_ERR(mk->mk_ctfm);
- mk->mk_ctfm = NULL;
- goto err_free_mk;
+ if (!inline_crypted) {
+ mk->mk_ctfm = allocate_skcipher_for_mode(mode, raw_key, inode);
+ if (IS_ERR(mk->mk_ctfm)) {
+ err = PTR_ERR(mk->mk_ctfm);
+ mk->mk_ctfm = NULL;
+ goto err_free_mk;
+ }
}
memcpy(mk->mk_descriptor, ci->ci_master_key_descriptor,
FS_KEY_DESCRIPTOR_SIZE);
memcpy(mk->mk_raw, raw_key, mode->keysize);
+ mk->mk_sb = inode->i_sb;
- return find_or_insert_master_key(mk, raw_key, mode, ci);
+ return find_or_insert_master_key(mk, raw_key, mode, ci, !inline_crypted,
+ inode->i_sb);
err_free_mk:
free_master_key(mk);
@@ -456,7 +478,8 @@ static int setup_crypto_transform(struct fscrypt_info *ci,
struct crypto_skcipher *ctfm;
int err;
- if (ci->ci_flags & FS_POLICY_FLAG_DIRECT_KEY) {
+ if ((ci->ci_flags & FS_POLICY_FLAG_DIRECT_KEY) ||
+ flags_inline_crypted(ci->ci_flags, inode)) {
mk = fscrypt_get_master_key(ci, mode, raw_key, inode);
if (IS_ERR(mk))
return PTR_ERR(mk);
@@ -486,13 +509,13 @@ static int setup_crypto_transform(struct fscrypt_info *ci,
return 0;
}
-static void put_crypt_info(struct fscrypt_info *ci)
+static void put_crypt_info(struct fscrypt_info *ci, struct inode *inode)
{
if (!ci)
return;
if (ci->ci_master_key) {
- put_master_key(ci->ci_master_key);
+ put_master_key(ci->ci_master_key, inode);
} else {
crypto_free_skcipher(ci->ci_ctfm);
crypto_free_cipher(ci->ci_essiv_tfm);
@@ -577,7 +600,7 @@ int fscrypt_get_encryption_info(struct inode *inode)
out:
if (res == -ENOKEY)
res = 0;
- put_crypt_info(crypt_info);
+ put_crypt_info(crypt_info, NULL);
kzfree(raw_key);
return res;
}
@@ -591,7 +614,7 @@ EXPORT_SYMBOL(fscrypt_get_encryption_info);
*/
void fscrypt_put_encryption_info(struct inode *inode)
{
- put_crypt_info(inode->i_crypt_info);
+ put_crypt_info(inode->i_crypt_info, inode);
inode->i_crypt_info = NULL;
}
EXPORT_SYMBOL(fscrypt_put_encryption_info);
@@ -610,3 +633,26 @@ void fscrypt_free_inode(struct inode *inode)
}
}
EXPORT_SYMBOL(fscrypt_free_inode);
+
+#ifdef CONFIG_FS_ENCRYPTION_INLINE_CRYPT
+bool fscrypt_inode_is_inline_crypted(const struct inode *inode)
+{
+ struct fscrypt_info *ci;
+
+ if (!inode)
+ return false;
+ ci = inode->i_crypt_info;
+
+ return ci && flags_inline_crypted(ci->ci_flags, inode);
+}
+EXPORT_SYMBOL(fscrypt_inode_is_inline_crypted);
+
+#endif /* CONFIG_FS_ENCRYPTION_INLINE_CRYPT */
+
+bool fscrypt_needs_fs_layer_crypto(const struct inode *inode)
+{
+ return IS_ENCRYPTED(inode) && S_ISREG(inode->i_mode) &&
+ !fscrypt_inode_is_inline_crypted(inode);
+}
+EXPORT_SYMBOL(fscrypt_needs_fs_layer_crypto);
+
diff --git a/fs/crypto/policy.c b/fs/crypto/policy.c
index d536889ac31b..7bcf5cc0d20d 100644
--- a/fs/crypto/policy.c
+++ b/fs/crypto/policy.c
@@ -36,6 +36,7 @@ static int create_encryption_context_from_policy(struct inode *inode,
struct fscrypt_context ctx;
ctx.format = FS_ENCRYPTION_CONTEXT_FORMAT_V1;
+
memcpy(ctx.master_key_descriptor, policy->master_key_descriptor,
FS_KEY_DESCRIPTOR_SIZE);
@@ -46,8 +47,17 @@ static int create_encryption_context_from_policy(struct inode *inode,
if (policy->flags & ~FS_POLICY_FLAGS_VALID)
return -EINVAL;
+ /*
+ * TODO: expose inline encryption via some toggleable knob
+ * instead of as a policy?
+ */
+ if (!inode->i_sb->s_cop->inline_crypt_supp &&
+ (policy->flags & FS_POLICY_FLAGS_INLINE_CRYPT))
+ return -EINVAL;
+
ctx.contents_encryption_mode = policy->contents_encryption_mode;
ctx.filenames_encryption_mode = policy->filenames_encryption_mode;
+
ctx.flags = policy->flags;
BUILD_BUG_ON(sizeof(ctx.nonce) != FS_KEY_DERIVATION_NONCE_SIZE);
get_random_bytes(ctx.nonce, FS_KEY_DERIVATION_NONCE_SIZE);
diff --git a/include/linux/fscrypt.h b/include/linux/fscrypt.h
index f7680ef1abd2..ea205aa8a24a 100644
--- a/include/linux/fscrypt.h
+++ b/include/linux/fscrypt.h
@@ -61,6 +61,7 @@ struct fscrypt_operations {
bool (*dummy_context)(struct inode *);
bool (*empty_dir)(struct inode *);
unsigned int max_namelen;
+ bool inline_crypt_supp;
};
struct fscrypt_ctx {
@@ -129,6 +130,23 @@ extern int fscrypt_inherit_context(struct inode *, struct inode *,
extern int fscrypt_get_encryption_info(struct inode *);
extern void fscrypt_put_encryption_info(struct inode *);
extern void fscrypt_free_inode(struct inode *);
+extern bool fscrypt_needs_fs_layer_crypto(const struct inode *inode);
+
+#ifdef CONFIG_FS_ENCRYPTION_INLINE_CRYPT
+extern bool fscrypt_inode_is_inline_crypted(const struct inode *inode);
+extern bool fscrypt_inode_crypt_mergeable(const struct inode *inode_1,
+ const struct inode *inode_2);
+#else
+static inline bool fscrypt_inode_is_inline_crypted(const struct inode *inode)
+{
+ return false;
+}
+static inline bool fscrypt_inode_crypt_mergeable(const struct inode *inode_1,
+ const struct inode *inode_2)
+{
+ return true;
+}
+#endif /* CONFIG_FS_ENCRYPTION_INLINE_CRYPT */
/* fname.c */
extern int fscrypt_setup_filename(struct inode *, const struct qstr *,
@@ -226,6 +244,25 @@ extern void fscrypt_enqueue_decrypt_bio(struct fscrypt_ctx *ctx,
extern void fscrypt_pullback_bio_page(struct page **, bool);
extern int fscrypt_zeroout_range(const struct inode *, pgoff_t, sector_t,
unsigned int);
+#ifdef CONFIG_FS_ENCRYPTION_INLINE_CRYPT
+extern int fscrypt_set_bio_crypt_ctx(const struct inode *inode,
+ struct bio *bio, u64 data_unit_num);
+extern void fscrypt_unset_bio_crypt_ctx(struct bio *bio);
+extern int fscrypt_evict_crypt_key(struct inode *inode);
+#else
+static inline int fscrypt_set_bio_crypt_ctx(const struct inode *inode,
+ struct bio *bio, u64 data_unit_num)
+{
+ return 0;
+}
+
+static inline void fscrypt_unset_bio_crypt_ctx(struct bio *bio) { }
+
+static inline int fscrypt_evict_crypt_key(struct inode *inode)
+{
+ return 0;
+}
+#endif
/* hooks.c */
extern int fscrypt_file_open(struct inode *inode, struct file *filp);
@@ -351,6 +388,17 @@ static inline void fscrypt_free_inode(struct inode *inode)
{
}
+static inline bool fscrypt_inode_is_inline_crypted(const struct inode *inode)
+{
+ return false;
+}
+
+static inline bool fscrypt_inode_crypt_mergeable(const struct inode *inode_1,
+ const struct inode *inode_2)
+{
+ return true;
+}
+
/* fname.c */
static inline int fscrypt_setup_filename(struct inode *dir,
const struct qstr *iname,
@@ -421,6 +469,20 @@ static inline int fscrypt_zeroout_range(const struct inode *inode, pgoff_t lblk,
return -EOPNOTSUPP;
}
+static inline int fscrypt_set_bio_crypt_ctx(const struct inode *inode,
+ struct bio *bio,
+ u64 data_unit_num)
+{
+ return -EOPNOTSUPP;
+}
+
+static inline void fscrypt_unset_bio_crypt_ctx(struct bio *bio) { }
+
+static inline int fscrypt_evict_crypt_key(struct inode *inode)
+{
+ return 0;
+}
+
/* hooks.c */
static inline int fscrypt_file_open(struct inode *inode, struct file *filp)
diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h
index 59c71fa8c553..ec5a9913886c 100644
--- a/include/uapi/linux/fs.h
+++ b/include/uapi/linux/fs.h
@@ -224,7 +224,8 @@ struct fsxattr {
#define FS_POLICY_FLAGS_PAD_32 0x03
#define FS_POLICY_FLAGS_PAD_MASK 0x03
#define FS_POLICY_FLAG_DIRECT_KEY 0x04 /* use master key directly */
-#define FS_POLICY_FLAGS_VALID 0x07
+#define FS_POLICY_FLAGS_INLINE_CRYPT 0x08
+#define FS_POLICY_FLAGS_VALID 0x0F
/* Encryption algorithms */
#define FS_ENCRYPTION_MODE_INVALID 0
--
2.22.0.410.gd8fdbe21b5-goog
^ permalink raw reply related
* [PATCH 8/8] f2fs: Wire up f2fs to use inline encryption via fscrypt
From: Satya Tangirala @ 2019-07-10 22:56 UTC (permalink / raw)
To: linux-block, linux-scsi, linux-fscrypt, linux-fsdevel,
linux-f2fs-devel
Cc: Parshuram Raju Thombare, Ladvine D Almeida, Barani Muthukumaran,
Kuohong Wang, Greg Kroah-Hartman, Satya Tangirala
In-Reply-To: <20190710225609.192252-1-satyat@google.com>
Signed-off-by: Satya Tangirala <satyat@google.com>
---
fs/f2fs/data.c | 83 +++++++++++++++++++++++++++++++++++++++++++++----
fs/f2fs/super.c | 13 ++++----
2 files changed, 84 insertions(+), 12 deletions(-)
diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c
index eda4181d2092..6fa947725fc2 100644
--- a/fs/f2fs/data.c
+++ b/fs/f2fs/data.c
@@ -142,6 +142,8 @@ static bool f2fs_bio_post_read_required(struct bio *bio)
static void f2fs_read_end_io(struct bio *bio)
{
+ fscrypt_unset_bio_crypt_ctx(bio);
+
if (time_to_inject(F2FS_P_SB(bio_first_page_all(bio)),
FAULT_READ_IO)) {
f2fs_show_injection_info(FAULT_READ_IO);
@@ -165,6 +167,8 @@ static void f2fs_write_end_io(struct bio *bio)
struct bio_vec *bvec;
struct bvec_iter_all iter_all;
+ fscrypt_unset_bio_crypt_ctx(bio);
+
if (time_to_inject(sbi, FAULT_WRITE_IO)) {
f2fs_show_injection_info(FAULT_WRITE_IO);
bio->bi_status = BLK_STS_IOERR;
@@ -282,9 +286,18 @@ static struct bio *__bio_alloc(struct f2fs_sb_info *sbi, block_t blk_addr,
return bio;
}
+static inline u64 inline_crypt_dun(struct inode *inode, pgoff_t offset)
+{
+ return (((u64)inode->i_ino) << 32) | lower_32_bits(offset);
+}
+
static inline void __submit_bio(struct f2fs_sb_info *sbi,
struct bio *bio, enum page_type type)
{
+ struct page *page;
+ struct inode *inode;
+ int err = 0;
+
if (!is_read_io(bio_op(bio))) {
unsigned int start;
@@ -326,7 +339,22 @@ static inline void __submit_bio(struct f2fs_sb_info *sbi,
trace_f2fs_submit_read_bio(sbi->sb, type, bio);
else
trace_f2fs_submit_write_bio(sbi->sb, type, bio);
- submit_bio(bio);
+
+ if (bio_has_data(bio)) {
+ page = bio_page(bio);
+ if (page && page->mapping && page->mapping->host) {
+ inode = page->mapping->host;
+ err = fscrypt_set_bio_crypt_ctx(inode, bio,
+ inline_crypt_dun(inode,
+ page->index));
+ }
+ }
+ if (err) {
+ bio->bi_status = BLK_STS_IOERR;
+ bio_endio(bio);
+ } else {
+ submit_bio(bio);
+ }
}
static void __submit_merged_bio(struct f2fs_bio_info *io)
@@ -487,6 +515,9 @@ void f2fs_submit_page_write(struct f2fs_io_info *fio)
enum page_type btype = PAGE_TYPE_OF_BIO(fio->type);
struct f2fs_bio_info *io = sbi->write_io[btype] + fio->temp;
struct page *bio_page;
+ struct inode *fio_inode, *bio_inode;
+ struct page *first_page;
+ u64 next_dun = 0;
f2fs_bug_on(sbi, is_read_io(fio->op));
@@ -513,10 +544,28 @@ void f2fs_submit_page_write(struct f2fs_io_info *fio)
inc_page_count(sbi, WB_DATA_TYPE(bio_page));
+ fio_inode = fio->page->mapping->host;
+ bio_inode = NULL;
+ first_page = NULL;
+ next_dun = 0;
+ if (io->bio && bio_page(io->bio)->mapping) {
+ first_page = bio_page(io->bio);
+ bio_inode = first_page->mapping->host;
+ if (fscrypt_inode_is_inline_crypted(bio_inode)) {
+ next_dun = inline_crypt_dun(bio_inode,
+ first_page->index) +
+ (io->bio->bi_iter.bi_size >> PAGE_SHIFT);
+ }
+ }
if (io->bio && (io->last_block_in_bio != fio->new_blkaddr - 1 ||
(io->fio.op != fio->op || io->fio.op_flags != fio->op_flags) ||
- !__same_bdev(sbi, fio->new_blkaddr, io->bio)))
+ !__same_bdev(sbi, fio->new_blkaddr, io->bio) ||
+ !fscrypt_inode_crypt_mergeable(bio_inode, fio_inode) ||
+ (fscrypt_inode_is_inline_crypted(bio_inode) &&
+ next_dun != inline_crypt_dun(fio_inode,
+ fio->page->index))))
__submit_merged_bio(io);
+
alloc_new:
if (io->bio == NULL) {
if ((fio->type == DATA || fio->type == NODE) &&
@@ -568,8 +617,9 @@ static struct bio *f2fs_grab_read_bio(struct inode *inode, block_t blkaddr,
bio->bi_end_io = f2fs_read_end_io;
bio_set_op_attrs(bio, REQ_OP_READ, op_flag);
- if (f2fs_encrypted_file(inode))
+ if (fscrypt_needs_fs_layer_crypto(inode))
post_read_steps |= 1 << STEP_DECRYPT;
+
if (post_read_steps) {
ctx = mempool_alloc(bio_post_read_ctx_pool, GFP_NOFS);
if (!ctx) {
@@ -1519,6 +1569,7 @@ static int f2fs_read_single_page(struct inode *inode, struct page *page,
struct f2fs_map_blocks *map,
struct bio **bio_ret,
sector_t *last_block_in_bio,
+ u64 *next_dun,
bool is_readahead)
{
struct bio *bio = *bio_ret;
@@ -1592,6 +1643,13 @@ static int f2fs_read_single_page(struct inode *inode, struct page *page,
__submit_bio(F2FS_I_SB(inode), bio, DATA);
bio = NULL;
}
+
+ if (bio && fscrypt_inode_is_inline_crypted(inode) &&
+ *next_dun != inline_crypt_dun(inode, page->index)) {
+ __submit_bio(F2FS_I_SB(inode), bio, DATA);
+ bio = NULL;
+ }
+
if (bio == NULL) {
bio = f2fs_grab_read_bio(inode, block_nr, nr_pages,
is_readahead ? REQ_RAHEAD : 0);
@@ -1611,6 +1669,9 @@ static int f2fs_read_single_page(struct inode *inode, struct page *page,
if (bio_add_page(bio, page, blocksize, 0) < blocksize)
goto submit_and_realloc;
+ if (fscrypt_inode_is_inline_crypted(inode))
+ *next_dun = inline_crypt_dun(inode, page->index) + 1;
+
inc_page_count(F2FS_I_SB(inode), F2FS_RD_DATA);
ClearPageError(page);
*last_block_in_bio = block_nr;
@@ -1644,6 +1705,7 @@ static int f2fs_mpage_readpages(struct address_space *mapping,
struct inode *inode = mapping->host;
struct f2fs_map_blocks map;
int ret = 0;
+ u64 next_dun = 0;
map.m_pblk = 0;
map.m_lblk = 0;
@@ -1667,7 +1729,8 @@ static int f2fs_mpage_readpages(struct address_space *mapping,
}
ret = f2fs_read_single_page(inode, page, nr_pages, &map, &bio,
- &last_block_in_bio, is_readahead);
+ &last_block_in_bio, &next_dun,
+ is_readahead);
if (ret) {
SetPageError(page);
zero_user_segment(page, 0, PAGE_SIZE);
@@ -1720,7 +1783,7 @@ static int encrypt_one_page(struct f2fs_io_info *fio)
struct page *mpage;
gfp_t gfp_flags = GFP_NOFS;
- if (!f2fs_encrypted_file(inode))
+ if (!fscrypt_needs_fs_layer_crypto(inode))
return 0;
/* wait for GCed page writeback via META_MAPPING */
@@ -1899,7 +1962,7 @@ int f2fs_do_write_data_page(struct f2fs_io_info *fio)
f2fs_unlock_op(fio->sbi);
err = f2fs_inplace_write_data(fio);
if (err) {
- if (f2fs_encrypted_file(inode))
+ if (fscrypt_needs_fs_layer_crypto(inode))
fscrypt_pullback_bio_page(&fio->encrypted_page,
true);
if (PageWriteback(page))
@@ -2617,6 +2680,8 @@ static void f2fs_dio_end_io(struct bio *bio)
{
struct f2fs_private_dio *dio = bio->bi_private;
+ fscrypt_unset_bio_crypt_ctx(bio);
+
dec_page_count(F2FS_I_SB(dio->inode),
dio->write ? F2FS_DIO_WRITE : F2FS_DIO_READ);
@@ -2633,12 +2698,18 @@ static void f2fs_dio_submit_bio(struct bio *bio, struct inode *inode,
{
struct f2fs_private_dio *dio;
bool write = (bio_op(bio) == REQ_OP_WRITE);
+ u64 data_unit_num = inline_crypt_dun(inode, file_offset >> PAGE_SHIFT);
dio = f2fs_kzalloc(F2FS_I_SB(inode),
sizeof(struct f2fs_private_dio), GFP_NOFS);
if (!dio)
goto out;
+ if (fscrypt_set_bio_crypt_ctx(inode, bio, data_unit_num) != 0) {
+ kvfree(dio);
+ goto out;
+ }
+
dio->inode = inode;
dio->orig_end_io = bio->bi_end_io;
dio->orig_private = bio->bi_private;
diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c
index 6b959bbb336a..2a7b3bae98aa 100644
--- a/fs/f2fs/super.c
+++ b/fs/f2fs/super.c
@@ -2223,12 +2223,13 @@ static bool f2fs_dummy_context(struct inode *inode)
}
static const struct fscrypt_operations f2fs_cryptops = {
- .key_prefix = "f2fs:",
- .get_context = f2fs_get_context,
- .set_context = f2fs_set_context,
- .dummy_context = f2fs_dummy_context,
- .empty_dir = f2fs_empty_dir,
- .max_namelen = F2FS_NAME_LEN,
+ .key_prefix = "f2fs:",
+ .get_context = f2fs_get_context,
+ .set_context = f2fs_set_context,
+ .dummy_context = f2fs_dummy_context,
+ .empty_dir = f2fs_empty_dir,
+ .max_namelen = F2FS_NAME_LEN,
+ .inline_crypt_supp = true,
};
#endif
--
2.22.0.410.gd8fdbe21b5-goog
^ permalink raw reply related
* Re: [PATCH V3] block: Disable write plugging for zoned block devices
From: Damien Le Moal @ 2019-07-10 23:58 UTC (permalink / raw)
To: Jens Axboe, linux-block@vger.kernel.org
In-Reply-To: <26e4505d-7de8-5e7c-d20c-3d82b9cff2d9@kernel.dk>
On 2019/07/11 5:18, Jens Axboe wrote:
> On 7/10/19 2:05 PM, Jens Axboe wrote:
>> Looks fine to me now, but doesn't apply cleanly to master/for-linus.
>
> I fixed it up manually, fwiw.
>
Thanks !
--
Damien Le Moal
Western Digital Research
^ permalink raw reply
* [RFC][PATCH] io_uring: Incapsulate conditional mem accounting
From: Pavel Begunkov (Silence) @ 2019-07-11 1:07 UTC (permalink / raw)
To: Jens Axboe, linux-block, linux-kernel; +Cc: Pavel Begunkov
From: Pavel Begunkov <asml.silence@gmail.com>
It's quite tedious and error-prone to manually check before each call
to io_{,un}account_mem() whether we need memory accounting. Instead,
the functions can work directly with struct io_ring_ctx and handle
checks themselves. In any case, they're perfectly inlined.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
fs/io_uring.c | 59 ++++++++++++++++++++++-----------------------------
1 file changed, 25 insertions(+), 34 deletions(-)
diff --git a/fs/io_uring.c b/fs/io_uring.c
index 3fd884b4e0be..f47f7abe19eb 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -2704,14 +2704,19 @@ static int io_sq_offload_start(struct io_ring_ctx *ctx,
return ret;
}
-static void io_unaccount_mem(struct user_struct *user, unsigned long nr_pages)
+static void io_unaccount_mem(struct io_ring_ctx *ctx, unsigned long nr_pages)
{
- atomic_long_sub(nr_pages, &user->locked_vm);
+ if (ctx->account_mem)
+ atomic_long_sub(nr_pages, &ctx->user->locked_vm);
}
-static int io_account_mem(struct user_struct *user, unsigned long nr_pages)
+static int io_account_mem(struct io_ring_ctx *ctx, unsigned long nr_pages)
{
unsigned long page_limit, cur_pages, new_pages;
+ struct user_struct *user = ctx->user;
+
+ if (!ctx->account_mem)
+ return 0;
/* Don't allow more pages than we can safely lock */
page_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
@@ -2773,8 +2778,7 @@ static int io_sqe_buffer_unregister(struct io_ring_ctx *ctx)
for (j = 0; j < imu->nr_bvecs; j++)
put_page(imu->bvec[j].bv_page);
- if (ctx->account_mem)
- io_unaccount_mem(ctx->user, imu->nr_bvecs);
+ io_unaccount_mem(ctx, imu->nr_bvecs);
kvfree(imu->bvec);
imu->nr_bvecs = 0;
}
@@ -2857,11 +2861,9 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, void __user *arg,
start = ubuf >> PAGE_SHIFT;
nr_pages = end - start;
- if (ctx->account_mem) {
- ret = io_account_mem(ctx->user, nr_pages);
- if (ret)
- goto err;
- }
+ ret = io_account_mem(ctx, nr_pages);
+ if (ret)
+ goto err;
ret = 0;
if (!pages || nr_pages > got_pages) {
@@ -2874,8 +2876,7 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, void __user *arg,
GFP_KERNEL);
if (!pages || !vmas) {
ret = -ENOMEM;
- if (ctx->account_mem)
- io_unaccount_mem(ctx->user, nr_pages);
+ io_unaccount_mem(ctx, nr_pages);
goto err;
}
got_pages = nr_pages;
@@ -2885,8 +2886,7 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, void __user *arg,
GFP_KERNEL);
ret = -ENOMEM;
if (!imu->bvec) {
- if (ctx->account_mem)
- io_unaccount_mem(ctx->user, nr_pages);
+ io_unaccount_mem(ctx, nr_pages);
goto err;
}
@@ -2919,8 +2919,7 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, void __user *arg,
for (j = 0; j < pret; j++)
put_page(pages[j]);
}
- if (ctx->account_mem)
- io_unaccount_mem(ctx->user, nr_pages);
+ io_unaccount_mem(ctx, nr_pages);
kvfree(imu->bvec);
goto err;
}
@@ -3009,9 +3008,7 @@ static void io_ring_ctx_free(struct io_ring_ctx *ctx)
io_mem_free(ctx->cq_ring);
percpu_ref_exit(&ctx->refs);
- if (ctx->account_mem)
- io_unaccount_mem(ctx->user,
- ring_pages(ctx->sq_entries, ctx->cq_entries));
+ io_unaccount_mem(ctx, ring_pages(ctx->sq_entries, ctx->cq_entries));
free_uid(ctx->user);
kfree(ctx);
}
@@ -3253,7 +3250,6 @@ static int io_uring_create(unsigned entries, struct io_uring_params *p)
{
struct user_struct *user = NULL;
struct io_ring_ctx *ctx;
- bool account_mem;
int ret;
if (!entries || entries > IORING_MAX_ENTRIES)
@@ -3269,29 +3265,24 @@ static int io_uring_create(unsigned entries, struct io_uring_params *p)
p->cq_entries = 2 * p->sq_entries;
user = get_uid(current_user());
- account_mem = !capable(CAP_IPC_LOCK);
-
- if (account_mem) {
- ret = io_account_mem(user,
- ring_pages(p->sq_entries, p->cq_entries));
- if (ret) {
- free_uid(user);
- return ret;
- }
- }
ctx = io_ring_ctx_alloc(p);
if (!ctx) {
- if (account_mem)
- io_unaccount_mem(user, ring_pages(p->sq_entries,
- p->cq_entries));
free_uid(user);
return -ENOMEM;
}
+
ctx->compat = in_compat_syscall();
- ctx->account_mem = account_mem;
+ ctx->account_mem = !capable(CAP_IPC_LOCK);
ctx->user = user;
+ ret = io_account_mem(ctx, ring_pages(p->sq_entries, p->cq_entries));
+ if (ret) {
+ free_uid(user);
+ kfree(ctx);
+ return ret;
+ }
+
ret = io_allocate_scq_urings(ctx, p);
if (ret)
goto err;
--
2.22.0
^ permalink raw reply related
* Re: [PATCH 0/2] Fix misuse of blk_rq_stats in blk-iolatency
From: Pavel Begunkov @ 2019-07-11 1:17 UTC (permalink / raw)
To: Josef Bacik; +Cc: Jens Axboe, linux-block, linux-kernel, dennis
In-Reply-To: <226043f8-4dc6-1ad3-7c66-8d85312f4cae@gmail.com>
Hi,
Any thoughts? Is there something wrong with the patchset?
On 29/06/2019 18:37, Pavel Begunkov wrote:
> Ping?
>
> On 20/06/2019 10:18, Pavel Begunkov wrote:
>> Hi,
>>
>> Josef, thanks for taking a look.
>>
>>
>> Although, there is nothing critical yet -- just a not working / disabled
>> optimisation, but changes in stats could sublty break it. E.g. grouping
>> @batch and @mean into a union will increase estimated average by several
>> orders of magnitude.
>>
>> Jens, what do you think?
>>
>>
>>
>> On 14/06/2019 16:40, Josef Bacik wrote:
>>> On Fri, Jun 14, 2019 at 02:44:11PM +0300, Pavel Begunkov (Silence) wrote:
>>>> From: Pavel Begunkov <asml.silence@gmail.com>
>>>>
>>>> There are implicit assumptions about struct blk_rq_stats, which make
>>>> it's very easy to misuse. The first patch fixes consequences, and the
>>>> second employs type-system to prevent recurrences.
>>>>
>>>>
>>>> Pavel Begunkov (2):
>>>> blk-iolatency: Fix zero mean in previous stats
>>>> blk-stats: Introduce explicit stat staging buffers
>>>>
>>>
>>> I don't have a problem with this, but it's up to Jens I suppose
>>>
>>> Acked-by: Josef Bacik <josef@toxicpanda.com>
>>>
>>> Thanks,
>>>
>>> Josef
>>>
>>
>
--
Yours sincerely,
Pavel Begunkov
^ permalink raw reply
* Re: [PATCH 3/8] block: blk-crypto for Inline Encryption
From: Randy Dunlap @ 2019-07-11 5:47 UTC (permalink / raw)
To: Satya Tangirala, linux-block, linux-scsi, linux-fscrypt,
linux-fsdevel, linux-f2fs-devel
Cc: Parshuram Raju Thombare, Ladvine D Almeida, Barani Muthukumaran,
Kuohong Wang, Greg Kroah-Hartman
In-Reply-To: <20190710225609.192252-4-satyat@google.com>
Hi,
Documentation nits, typos. questions...
On 7/10/19 3:56 PM, Satya Tangirala wrote:
> diff --git a/Documentation/block/inline-encryption.txt b/Documentation/block/inline-encryption.txt
> new file mode 100644
> index 000000000000..96a7983a117d
> --- /dev/null
> +++ b/Documentation/block/inline-encryption.txt
> @@ -0,0 +1,185 @@
> +BLK-CRYPTO and KEYSLOT MANAGER
> +===========================
> +
> +CONTENTS
> +1. Objective
> +2. Constraints and notes
> +3. Design
> +4. Blk-crypto
> + 4-1 What does blk-crypto do on bio submission
> +5. Layered Devices
> +6. Future optimizations for layered devices
> +
> +1. Objective
> +============
> +
> +We want to support inline encryption (IE) in the kernel.
> +To allow for testing, we also want a software fallback when actual
> +IE hardware is absent. We also want IE to work with layered devices
> +like dm and loopback (i.e. we want to be able to use the IE hardware
> +of the underlying devices if present, or else fall back to software
> +en/decryption).
> +
> +
> +2. Constraints and notes
> +========================
> +
> +1) IE hardware have a limited number of “keyslots” that can be programmed
> +with an encryption context (key, algorithm, data unit size, etc.) at any time.
> +One can specify a keyslot in a data request made to the device, and the
> +device will en/decrypt the data using the encryption context programmed into
> +that specified keyslot. When possible, we want to make multiple requests with
> +the same encryption context share the same keyslot.
> +
> +2) We need a way for filesystems to specify an encryption context to use for
> +en/decrypting a struct bio, and a device driver (like UFS) needs to be able
> +to use that encryption context when it processes the bio.
> +
> +3) We need a way for device drivers to expose their capabilities in a unified
> +way to the upper layers.
> +
> +
> +3. Design
> +=========
> +
> +We add a struct bio_crypt_context to struct bio that can represent an
is this bi_crypt_context ??
> +encryption context, because we need to able to pass this encryption context
to be able
> +from the FS layer to the device driver to act upon.
> +
> +While IE hardware works on the notion of keyslots, the FS layer has no
> +knowledge of keyslots - it simply wants to specify an encryption context to
> +use while en/decrypting a bio.
> +
> +We introduce a keyslot manager (KSM) that handles the translation from
> +encryption contexts specified by the FS to keyslots on the IE hardware.
> +This KSM also serves as the way IE hardware can expose their capabilities to
> +upper layers. The generic mode of operation is: each device driver that wants
> +to support IE will construct a KSM and set it up in its struct request_queue.
> +Upper layers that want to use IE on this device can then use this KSM in
> +the device’s struct request_queue to translate an encryption context into
> +a keyslot. The presence of the KSM in the request queue shall be used to mean
> +that the device supports IE.
> +
> +On the device driver end of the interface, the device driver needs to tell the
> +KSM how to actually manipulate the IE hardware in the device to do things like
> +programming the crypto key into the IE hardware into a particular keyslot. All
> +this is achieved through the struct keyslot_mgmt_ll_ops that the device driver
> +passes to the KSM when creating it.
> +
> +It uses refcounts to track which keyslots are idle (either they have no
> +encryption context programmed, or there are no in flight struct bios
in-flight
> +referencing that keyslot). When a new encryption context needs a keyslot, it
> +tries to find a keyslot that has already been programmed with the same
> +encryption context, and if there is no such keyslot, it evicts the least
> +recently used idle keyslot and programs the new encryption context into that
> +one. If no idle keyslots are available, then the caller will sleep until there
> +is at least one.
> +
> +
> +4. Blk-crypto
> +=============
> +
> +The above is sufficient for simple cases, but does not work if there is a
> +need for a software fallback, or if we are want to use IE with layered devices.
> +To these ends, we introduce blk-crypto. Blk-crypto allows us to present a
> +unified view of encryption to the FS (so FS only needs to specify an
> +encryption context and not worry about keyslots at all), and blk-crypto can
> +decide whether to delegate the en/decryption to IE hardware or to software
> +(i.e. to the kernel crypto API). Blk-crypto maintains an internal KSM that
> +serves as the software fallback to the kernel crypto API.
> +
> +Blk-crypto needs to ensure that the encryption context is programmed into the
> +"correct" keyslot manager for IE. If a bio is submitted to a layered device
> +that eventually passes the bio down to a device that really does support IE, we
> +want the encryption context to be programmed into a keyslot for the KSM of the
> +device with IE support. However, blk-crypto does not know a priori whether a
> +particular device is the final device in the layering structure for a bio or
> +not. So in the case that a particular device does not support IE, since it is
> +possibly the final destination device for the bio, if the bio requires
> +encryption (i.e. the bio is doing a write operation), blk-crypto must fallback
> +to software *before* sending the bio to the device.
> +
> +Blk-crypto ensures that
> +1) The bio’s encryption context is programmed into a keyslot in the KSM of the
> +request queue that the bio is being submitted to (or the software fallback KSM
> +if the request queue doesn’t have a KSM), and that the processing_ksm in the
> +bi_crypt_context is set to this KSM
> +
> +2) That the bio has its own individual reference to the keyslot in this KSM.
> +Once the bio passes through blk-crypto, its encryption context is programmed
> +in some KSM. The “its own individual reference to the keyslot” ensures that
> +keyslots can be released by each bio independently of other bios while ensuring
> +that the bio has a valid reference to the keyslot when, for e.g., the software
> +fallback KSM in blk-crypto performs crypto for on the device’s behalf. The
> +individual references are ensured by increasing the refcount for the keyslot in
> +the processing_ksm when a bio with a programmed encryption context is cloned.
> +
> +
> +4-1. What blk-crypto does on bio submission
> +-------------------------------------------
> +
> +Case 1: blk-crypto is given a bio with only an encryption context that hasn’t
> +been programmed into any keyslot in any KSM (for e.g. a bio from the FS). In
> +this case, blk-crypto will program the encryption context into the KSM of the
> +request queue the bio is being submitted to (and if this KSM does not exist,
> +then it will program it into blk-crypto’s internal KSM for software fallback).
> +The KSM that this encryption context was programmed into is stored as the
> +processing_ksm in the bio’s bi_crypt_context.
> +
> +Case 2: blk-crypto is given a bio whose encryption context has already been
> +programmed into a keyslot in the *software fallback KSM*. In this case,
> +blk-crypto does nothing; it treats the bio as not having specified an
> +encryption context. Note that we cannot do what we will do in Case 3 here
Note that we cannot do here what we will do in Case 3
> +because we would have already encrypted the bio in software by this point.
> +
> +Case 3: blk-crypto is given a bio whose encryption context has already been
> +programmed into a keyslot in some KSM (that is *not* the software fallback
> +KSM). In this case, blk-crypto first releases that keyslot from that KSM and
> +then treats the bio as in Case 1.
> +
> +This way, when a device driver is processing a bio, it can be sure that
> +the bio’s encryption context has been programmed into some KSM (either the
> +device driver’s request queue’s KSM, or blk-crypto’s software fallback KSM).
> +It then simply needs to check if the bio’s processing_ksm is the device’s
> +request queue’s KSM. If so, then it should proceed with IE. If not, it should
> +simply do nothing with respect to crypto, because some other KSM (perhaps the
> +blk-crypto software fallback KSM) is handling the en/decryption.
> +
> +Blk-crypto will release the keyslot that is being held by the bio (and also
> +decrypt it if the bio is using the software fallback KSM) once
> +bio_remaining_done returns true for the bio.
> +
> +
> +5. Layered Devices
> +==================
> +
> +Layered devices that wish to support IE need to create their own keyslot
> +manager for their request queue, and expose whatever functionality they choose.
> +When a layered device wants to pass a bio to another layer (either by
> +resubmitting the same bio, or by submitting a clone), it doesn’t need to do
> +anything special because the bio (or the clone) will once again pass through
> +blk-crypto, which will work as described in Case 3. If a layered device wants
> +for some reason to do the IO by itself instead of passing it on to a child
> +device, but it also chose to expose IE capabilities by setting up a KSM in its
> +request queue, it is then responsible for en/decrypting the data itself. In
> +such cases, the device can choose to call the blk-crypto function
> +blk_crypto_fallback_to_software (TODO: Not yet implemented), which will
> +cause the en/decryption to be done via software fallback.
> +
> +
> +6. Future Optimizations for layered devices
> +===========================================
> +
> +Creating a keyslot manager for the layered device uses up memory for each
> +keyslot, and in general, a layered device (like dm-linear) merely passes the
> +request on to a “child” device, so the keyslots in the layered device itself
> +might be completely unused. We can instead define a new type of KSM; the
> +“passthrough KSM”, that layered devices can use to let blk-crypto know that
> +this layered device *will* pass the bio to some child device (and hence
> +through blk-crypto again, at which point blk-crypto can program the encryption
> +context, instead of programming it into the layered device’s KSM). Again, if
> +the device “lies” and decides to do the IO itself instead of passing it on to
> +a child device, it is responsible for doing the en/decryption (and can choose
> +to call blk_crypto_fallback_to_software). Another use case for the
> +"passthrough KSM" is for IE devices that want to manage their own keyslots/do
> +not have a limited number of keyslots.
--
~Randy
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox