* [PATCH v7 3/5] dmaengine: xilinx: dpdma: Add the Xilinx DisplayPort DMA engine driver
From: Laurent Pinchart @ 2020-07-17 1:33 UTC (permalink / raw)
To: dmaengine
Cc: Michal Simek, Hyun Kwon, Tejas Upadhyay, Satish Kumar Nagireddy,
Vinod Koul, Peter Ujfalusi
In-Reply-To: <20200717013337.24122-1-laurent.pinchart@ideasonboard.com>
From: Hyun Kwon <hyun.kwon@xilinx.com>
The ZynqMP DisplayPort subsystem includes a DMA engine called DPDMA with
6 DMa channels (4 for display and 2 for audio). This driver exposes the
DPDMA through the dmaengine API, to be used by audio (ALSA) and display
(DRM) drivers for the DisplayPort subsystem.
Signed-off-by: Hyun Kwon <hyun.kwon@xilinx.com>
Signed-off-by: Tejas Upadhyay <tejasu@xilinx.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
---
Changes since v6:
- Reject transactions without DMA_PREP_LOAD_EOT
Changes since v5:
- Update copyright year
- Use GFP_NOWAIT to allocate descriptor
- Drop check of deprecated direction parameter in channel configuration
- Don't reject channel configuration if the channel is busy
Changes since v4:
- Support DMA_PREP_LOAD_EOT, ignoring any transaction that doesn't have
the flag set, as requested in the review of v4
Changes since v3:
- Fix uninitialized variable in xilinx_dpdma_config()
- Free pending and active descriptors upon termination
- Switch to DMA_PREP_REPEAT
Changes since v2:
- Switch to virt-dma
- Support interleaved cyclic transfers and nothing else
- Fix terminate_all behaviour (don't wait)
- Fix bug in extended address handling for hw desc
- Clean up video group handling
- Update driver name
- Use macros for bitfields
- Remove unneeded header
- Coding style and typo fixes
Changes since v1:
- Remove unneeded #include
- Drop enum xilinx_dpdma_chan_id
- Update compatible string
- Drop DT subnodes
- Replace XILINX_DPDMA_NUM_CHAN with ARRAY_SIZE(xdev->chan)
- Disable IRQ at remove() time
- Use devm_platform_ioremap_resource()
- Don't inline functions manually
- Add section headers
- Merge DMA engine implementation in their wrappers
- Rename xilinx_dpdma_sw_desc::phys to dma_addr
- Use GENMASK()
- Use FIELD_PREP/FIELD_GET
- Fix MSB handling in xilinx_dpdma_sw_desc_addr_64()
- Fix logic in xilinx_dpdma_chan_prep_slave_sg()
- Document why xilinx_dpdma_config() doesn't need to check most
parameters
- Remove debugfs support
- Rechedule errored descriptor
- Align the line size with 128bit
- SPDX header formatting
---
MAINTAINERS | 1 +
drivers/dma/Kconfig | 10 +
drivers/dma/xilinx/Makefile | 1 +
drivers/dma/xilinx/xilinx_dpdma.c | 1533 +++++++++++++++++++++++++++++
4 files changed, 1545 insertions(+)
create mode 100644 drivers/dma/xilinx/xilinx_dpdma.c
diff --git a/MAINTAINERS b/MAINTAINERS
index fa52d4f9f8c8..6c20a6d338f0 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -18858,6 +18858,7 @@ M: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
L: dmaengine@vger.kernel.org
S: Supported
F: Documentation/devicetree/bindings/dma/xilinx/xlnx,zynqmp-dpdma.yaml
+F: drivers/dma/xilinx/xilinx_dpdma.c
F: include/dt-bindings/dma/xlnx-zynqmp-dpdma.h
XILLYBUS DRIVER
diff --git a/drivers/dma/Kconfig b/drivers/dma/Kconfig
index de41d7928bff..668e9636c547 100644
--- a/drivers/dma/Kconfig
+++ b/drivers/dma/Kconfig
@@ -707,6 +707,16 @@ config XILINX_ZYNQMP_DMA
help
Enable support for Xilinx ZynqMP DMA controller.
+config XILINX_ZYNQMP_DPDMA
+ tristate "Xilinx DPDMA Engine"
+ select DMA_ENGINE
+ select DMA_VIRTUAL_CHANNELS
+ help
+ Enable support for Xilinx ZynqMP DisplayPort DMA. Choose this option
+ if you have a Xilinx ZynqMP SoC with a DisplayPort subsystem. The
+ driver provides the dmaengine required by the DisplayPort subsystem
+ display driver.
+
config ZX_DMA
tristate "ZTE ZX DMA support"
depends on ARCH_ZX || COMPILE_TEST
diff --git a/drivers/dma/xilinx/Makefile b/drivers/dma/xilinx/Makefile
index e921de575b55..767bb45f641f 100644
--- a/drivers/dma/xilinx/Makefile
+++ b/drivers/dma/xilinx/Makefile
@@ -1,3 +1,4 @@
# SPDX-License-Identifier: GPL-2.0-only
obj-$(CONFIG_XILINX_DMA) += xilinx_dma.o
obj-$(CONFIG_XILINX_ZYNQMP_DMA) += zynqmp_dma.o
+obj-$(CONFIG_XILINX_ZYNQMP_DPDMA) += xilinx_dpdma.o
diff --git a/drivers/dma/xilinx/xilinx_dpdma.c b/drivers/dma/xilinx/xilinx_dpdma.c
new file mode 100644
index 000000000000..af88a6762ef4
--- /dev/null
+++ b/drivers/dma/xilinx/xilinx_dpdma.c
@@ -0,0 +1,1533 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Xilinx ZynqMP DPDMA Engine driver
+ *
+ * Copyright (C) 2015 - 2020 Xilinx, Inc.
+ *
+ * Author: Hyun Woo Kwon <hyun.kwon@xilinx.com>
+ */
+
+#include <linux/bitfield.h>
+#include <linux/bits.h>
+#include <linux/clk.h>
+#include <linux/delay.h>
+#include <linux/dmaengine.h>
+#include <linux/dmapool.h>
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/of_dma.h>
+#include <linux/platform_device.h>
+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/wait.h>
+
+#include <dt-bindings/dma/xlnx-zynqmp-dpdma.h>
+
+#include "../dmaengine.h"
+#include "../virt-dma.h"
+
+/* DPDMA registers */
+#define XILINX_DPDMA_ERR_CTRL 0x000
+#define XILINX_DPDMA_ISR 0x004
+#define XILINX_DPDMA_IMR 0x008
+#define XILINX_DPDMA_IEN 0x00c
+#define XILINX_DPDMA_IDS 0x010
+#define XILINX_DPDMA_INTR_DESC_DONE(n) BIT((n) + 0)
+#define XILINX_DPDMA_INTR_DESC_DONE_MASK GENMASK(5, 0)
+#define XILINX_DPDMA_INTR_NO_OSTAND(n) BIT((n) + 6)
+#define XILINX_DPDMA_INTR_NO_OSTAND_MASK GENMASK(11, 6)
+#define XILINX_DPDMA_INTR_AXI_ERR(n) BIT((n) + 12)
+#define XILINX_DPDMA_INTR_AXI_ERR_MASK GENMASK(17, 12)
+#define XILINX_DPDMA_INTR_DESC_ERR(n) BIT((n) + 16)
+#define XILINX_DPDMA_INTR_DESC_ERR_MASK GENMASK(23, 18)
+#define XILINX_DPDMA_INTR_WR_CMD_FIFO_FULL BIT(24)
+#define XILINX_DPDMA_INTR_WR_DATA_FIFO_FULL BIT(25)
+#define XILINX_DPDMA_INTR_AXI_4K_CROSS BIT(26)
+#define XILINX_DPDMA_INTR_VSYNC BIT(27)
+#define XILINX_DPDMA_INTR_CHAN_ERR_MASK 0x00041000
+#define XILINX_DPDMA_INTR_CHAN_ERR 0x00fff000
+#define XILINX_DPDMA_INTR_GLOBAL_ERR 0x07000000
+#define XILINX_DPDMA_INTR_ERR_ALL 0x07fff000
+#define XILINX_DPDMA_INTR_CHAN_MASK 0x00041041
+#define XILINX_DPDMA_INTR_GLOBAL_MASK 0x0f000000
+#define XILINX_DPDMA_INTR_ALL 0x0fffffff
+#define XILINX_DPDMA_EISR 0x014
+#define XILINX_DPDMA_EIMR 0x018
+#define XILINX_DPDMA_EIEN 0x01c
+#define XILINX_DPDMA_EIDS 0x020
+#define XILINX_DPDMA_EINTR_INV_APB BIT(0)
+#define XILINX_DPDMA_EINTR_RD_AXI_ERR(n) BIT((n) + 1)
+#define XILINX_DPDMA_EINTR_RD_AXI_ERR_MASK GENMASK(6, 1)
+#define XILINX_DPDMA_EINTR_PRE_ERR(n) BIT((n) + 7)
+#define XILINX_DPDMA_EINTR_PRE_ERR_MASK GENMASK(12, 7)
+#define XILINX_DPDMA_EINTR_CRC_ERR(n) BIT((n) + 13)
+#define XILINX_DPDMA_EINTR_CRC_ERR_MASK GENMASK(18, 13)
+#define XILINX_DPDMA_EINTR_WR_AXI_ERR(n) BIT((n) + 19)
+#define XILINX_DPDMA_EINTR_WR_AXI_ERR_MASK GENMASK(24, 19)
+#define XILINX_DPDMA_EINTR_DESC_DONE_ERR(n) BIT((n) + 25)
+#define XILINX_DPDMA_EINTR_DESC_DONE_ERR_MASK GENMASK(30, 25)
+#define XILINX_DPDMA_EINTR_RD_CMD_FIFO_FULL BIT(32)
+#define XILINX_DPDMA_EINTR_CHAN_ERR_MASK 0x02082082
+#define XILINX_DPDMA_EINTR_CHAN_ERR 0x7ffffffe
+#define XILINX_DPDMA_EINTR_GLOBAL_ERR 0x80000001
+#define XILINX_DPDMA_EINTR_ALL 0xffffffff
+#define XILINX_DPDMA_CNTL 0x100
+#define XILINX_DPDMA_GBL 0x104
+#define XILINX_DPDMA_GBL_TRIG_MASK(n) ((n) << 0)
+#define XILINX_DPDMA_GBL_RETRIG_MASK(n) ((n) << 6)
+#define XILINX_DPDMA_ALC0_CNTL 0x108
+#define XILINX_DPDMA_ALC0_STATUS 0x10c
+#define XILINX_DPDMA_ALC0_MAX 0x110
+#define XILINX_DPDMA_ALC0_MIN 0x114
+#define XILINX_DPDMA_ALC0_ACC 0x118
+#define XILINX_DPDMA_ALC0_ACC_TRAN 0x11c
+#define XILINX_DPDMA_ALC1_CNTL 0x120
+#define XILINX_DPDMA_ALC1_STATUS 0x124
+#define XILINX_DPDMA_ALC1_MAX 0x128
+#define XILINX_DPDMA_ALC1_MIN 0x12c
+#define XILINX_DPDMA_ALC1_ACC 0x130
+#define XILINX_DPDMA_ALC1_ACC_TRAN 0x134
+
+/* Channel register */
+#define XILINX_DPDMA_CH_BASE 0x200
+#define XILINX_DPDMA_CH_OFFSET 0x100
+#define XILINX_DPDMA_CH_DESC_START_ADDRE 0x000
+#define XILINX_DPDMA_CH_DESC_START_ADDRE_MASK GENMASK(15, 0)
+#define XILINX_DPDMA_CH_DESC_START_ADDR 0x004
+#define XILINX_DPDMA_CH_DESC_NEXT_ADDRE 0x008
+#define XILINX_DPDMA_CH_DESC_NEXT_ADDR 0x00c
+#define XILINX_DPDMA_CH_PYLD_CUR_ADDRE 0x010
+#define XILINX_DPDMA_CH_PYLD_CUR_ADDR 0x014
+#define XILINX_DPDMA_CH_CNTL 0x018
+#define XILINX_DPDMA_CH_CNTL_ENABLE BIT(0)
+#define XILINX_DPDMA_CH_CNTL_PAUSE BIT(1)
+#define XILINX_DPDMA_CH_CNTL_QOS_DSCR_WR_MASK GENMASK(5, 2)
+#define XILINX_DPDMA_CH_CNTL_QOS_DSCR_RD_MASK GENMASK(9, 6)
+#define XILINX_DPDMA_CH_CNTL_QOS_DATA_RD_MASK GENMASK(13, 10)
+#define XILINX_DPDMA_CH_CNTL_QOS_VID_CLASS 11
+#define XILINX_DPDMA_CH_STATUS 0x01c
+#define XILINX_DPDMA_CH_STATUS_OTRAN_CNT_MASK GENMASK(24, 21)
+#define XILINX_DPDMA_CH_VDO 0x020
+#define XILINX_DPDMA_CH_PYLD_SZ 0x024
+#define XILINX_DPDMA_CH_DESC_ID 0x028
+
+/* DPDMA descriptor fields */
+#define XILINX_DPDMA_DESC_CONTROL_PREEMBLE 0xa5
+#define XILINX_DPDMA_DESC_CONTROL_COMPLETE_INTR BIT(8)
+#define XILINX_DPDMA_DESC_CONTROL_DESC_UPDATE BIT(9)
+#define XILINX_DPDMA_DESC_CONTROL_IGNORE_DONE BIT(10)
+#define XILINX_DPDMA_DESC_CONTROL_FRAG_MODE BIT(18)
+#define XILINX_DPDMA_DESC_CONTROL_LAST BIT(19)
+#define XILINX_DPDMA_DESC_CONTROL_ENABLE_CRC BIT(20)
+#define XILINX_DPDMA_DESC_CONTROL_LAST_OF_FRAME BIT(21)
+#define XILINX_DPDMA_DESC_ID_MASK GENMASK(15, 0)
+#define XILINX_DPDMA_DESC_HSIZE_STRIDE_HSIZE_MASK GENMASK(17, 0)
+#define XILINX_DPDMA_DESC_HSIZE_STRIDE_STRIDE_MASK GENMASK(31, 18)
+#define XILINX_DPDMA_DESC_ADDR_EXT_NEXT_ADDR_MASK GENMASK(15, 0)
+#define XILINX_DPDMA_DESC_ADDR_EXT_SRC_ADDR_MASK GENMASK(31, 16)
+
+#define XILINX_DPDMA_ALIGN_BYTES 256
+#define XILINX_DPDMA_LINESIZE_ALIGN_BITS 128
+
+#define XILINX_DPDMA_NUM_CHAN 6
+
+struct xilinx_dpdma_chan;
+
+/**
+ * struct xilinx_dpdma_hw_desc - DPDMA hardware descriptor
+ * @control: control configuration field
+ * @desc_id: descriptor ID
+ * @xfer_size: transfer size
+ * @hsize_stride: horizontal size and stride
+ * @timestamp_lsb: LSB of time stamp
+ * @timestamp_msb: MSB of time stamp
+ * @addr_ext: upper 16 bit of 48 bit address (next_desc and src_addr)
+ * @next_desc: next descriptor 32 bit address
+ * @src_addr: payload source address (1st page, 32 LSB)
+ * @addr_ext_23: payload source address (3nd and 3rd pages, 16 LSBs)
+ * @addr_ext_45: payload source address (4th and 5th pages, 16 LSBs)
+ * @src_addr2: payload source address (2nd page, 32 LSB)
+ * @src_addr3: payload source address (3rd page, 32 LSB)
+ * @src_addr4: payload source address (4th page, 32 LSB)
+ * @src_addr5: payload source address (5th page, 32 LSB)
+ * @crc: descriptor CRC
+ */
+struct xilinx_dpdma_hw_desc {
+ u32 control;
+ u32 desc_id;
+ u32 xfer_size;
+ u32 hsize_stride;
+ u32 timestamp_lsb;
+ u32 timestamp_msb;
+ u32 addr_ext;
+ u32 next_desc;
+ u32 src_addr;
+ u32 addr_ext_23;
+ u32 addr_ext_45;
+ u32 src_addr2;
+ u32 src_addr3;
+ u32 src_addr4;
+ u32 src_addr5;
+ u32 crc;
+} __aligned(XILINX_DPDMA_ALIGN_BYTES);
+
+/**
+ * struct xilinx_dpdma_sw_desc - DPDMA software descriptor
+ * @hw: DPDMA hardware descriptor
+ * @node: list node for software descriptors
+ * @dma_addr: DMA address of the software descriptor
+ */
+struct xilinx_dpdma_sw_desc {
+ struct xilinx_dpdma_hw_desc hw;
+ struct list_head node;
+ dma_addr_t dma_addr;
+};
+
+/**
+ * struct xilinx_dpdma_tx_desc - DPDMA transaction descriptor
+ * @vdesc: virtual DMA descriptor
+ * @chan: DMA channel
+ * @descriptors: list of software descriptors
+ * @error: an error has been detected with this descriptor
+ */
+struct xilinx_dpdma_tx_desc {
+ struct virt_dma_desc vdesc;
+ struct xilinx_dpdma_chan *chan;
+ struct list_head descriptors;
+ bool error;
+};
+
+#define to_dpdma_tx_desc(_desc) \
+ container_of(_desc, struct xilinx_dpdma_tx_desc, vdesc)
+
+/**
+ * struct xilinx_dpdma_chan - DPDMA channel
+ * @vchan: virtual DMA channel
+ * @reg: register base address
+ * @id: channel ID
+ * @wait_to_stop: queue to wait for outstanding transacitons before stopping
+ * @running: true if the channel is running
+ * @first_frame: flag for the first frame of stream
+ * @video_group: flag if multi-channel operation is needed for video channels
+ * @lock: lock to access struct xilinx_dpdma_chan
+ * @desc_pool: descriptor allocation pool
+ * @err_task: error IRQ bottom half handler
+ * @desc.pending: Descriptor schedule to the hardware, pending execution
+ * @desc.active: Descriptor being executed by the hardware
+ * @xdev: DPDMA device
+ */
+struct xilinx_dpdma_chan {
+ struct virt_dma_chan vchan;
+ void __iomem *reg;
+ unsigned int id;
+
+ wait_queue_head_t wait_to_stop;
+ bool running;
+ bool first_frame;
+ bool video_group;
+
+ spinlock_t lock; /* lock to access struct xilinx_dpdma_chan */
+ struct dma_pool *desc_pool;
+ struct tasklet_struct err_task;
+
+ struct {
+ struct xilinx_dpdma_tx_desc *pending;
+ struct xilinx_dpdma_tx_desc *active;
+ } desc;
+
+ struct xilinx_dpdma_device *xdev;
+};
+
+#define to_xilinx_chan(_chan) \
+ container_of(_chan, struct xilinx_dpdma_chan, vchan.chan)
+
+/**
+ * struct xilinx_dpdma_device - DPDMA device
+ * @common: generic dma device structure
+ * @reg: register base address
+ * @dev: generic device structure
+ * @irq: the interrupt number
+ * @axi_clk: axi clock
+ * @chan: DPDMA channels
+ * @ext_addr: flag for 64 bit system (48 bit addressing)
+ */
+struct xilinx_dpdma_device {
+ struct dma_device common;
+ void __iomem *reg;
+ struct device *dev;
+ int irq;
+
+ struct clk *axi_clk;
+ struct xilinx_dpdma_chan *chan[XILINX_DPDMA_NUM_CHAN];
+
+ bool ext_addr;
+};
+
+/* -----------------------------------------------------------------------------
+ * I/O Accessors
+ */
+
+static inline u32 dpdma_read(void __iomem *base, u32 offset)
+{
+ return ioread32(base + offset);
+}
+
+static inline void dpdma_write(void __iomem *base, u32 offset, u32 val)
+{
+ iowrite32(val, base + offset);
+}
+
+static inline void dpdma_clr(void __iomem *base, u32 offset, u32 clr)
+{
+ dpdma_write(base, offset, dpdma_read(base, offset) & ~clr);
+}
+
+static inline void dpdma_set(void __iomem *base, u32 offset, u32 set)
+{
+ dpdma_write(base, offset, dpdma_read(base, offset) | set);
+}
+
+/* -----------------------------------------------------------------------------
+ * Descriptor Operations
+ */
+
+/**
+ * xilinx_dpdma_sw_desc_set_dma_addrs - Set DMA addresses in the descriptor
+ * @sw_desc: The software descriptor in which to set DMA addresses
+ * @prev: The previous descriptor
+ * @dma_addr: array of dma addresses
+ * @num_src_addr: number of addresses in @dma_addr
+ *
+ * Set all the DMA addresses in the hardware descriptor corresponding to @dev
+ * from @dma_addr. If a previous descriptor is specified in @prev, its next
+ * descriptor DMA address is set to the DMA address of @sw_desc. @prev may be
+ * identical to @sw_desc for cyclic transfers.
+ */
+static void xilinx_dpdma_sw_desc_set_dma_addrs(struct xilinx_dpdma_device *xdev,
+ struct xilinx_dpdma_sw_desc *sw_desc,
+ struct xilinx_dpdma_sw_desc *prev,
+ dma_addr_t dma_addr[],
+ unsigned int num_src_addr)
+{
+ struct xilinx_dpdma_hw_desc *hw_desc = &sw_desc->hw;
+ unsigned int i;
+
+ hw_desc->src_addr = lower_32_bits(dma_addr[0]);
+ if (xdev->ext_addr)
+ hw_desc->addr_ext |=
+ FIELD_PREP(XILINX_DPDMA_DESC_ADDR_EXT_SRC_ADDR_MASK,
+ upper_32_bits(dma_addr[0]));
+
+ for (i = 1; i < num_src_addr; i++) {
+ u32 *addr = &hw_desc->src_addr2;
+
+ addr[i-1] = lower_32_bits(dma_addr[i]);
+
+ if (xdev->ext_addr) {
+ u32 *addr_ext = &hw_desc->addr_ext_23;
+ u32 addr_msb;
+
+ addr_msb = upper_32_bits(dma_addr[i]) & GENMASK(15, 0);
+ addr_msb <<= 16 * ((i - 1) % 2);
+ addr_ext[(i - 1) / 2] |= addr_msb;
+ }
+ }
+
+ if (!prev)
+ return;
+
+ prev->hw.next_desc = lower_32_bits(sw_desc->dma_addr);
+ if (xdev->ext_addr)
+ prev->hw.addr_ext |=
+ FIELD_PREP(XILINX_DPDMA_DESC_ADDR_EXT_NEXT_ADDR_MASK,
+ upper_32_bits(sw_desc->dma_addr));
+}
+
+/**
+ * xilinx_dpdma_chan_alloc_sw_desc - Allocate a software descriptor
+ * @chan: DPDMA channel
+ *
+ * Allocate a software descriptor from the channel's descriptor pool.
+ *
+ * Return: a software descriptor or NULL.
+ */
+static struct xilinx_dpdma_sw_desc *
+xilinx_dpdma_chan_alloc_sw_desc(struct xilinx_dpdma_chan *chan)
+{
+ struct xilinx_dpdma_sw_desc *sw_desc;
+ dma_addr_t dma_addr;
+
+ sw_desc = dma_pool_zalloc(chan->desc_pool, GFP_ATOMIC, &dma_addr);
+ if (!sw_desc)
+ return NULL;
+
+ sw_desc->dma_addr = dma_addr;
+
+ return sw_desc;
+}
+
+/**
+ * xilinx_dpdma_chan_free_sw_desc - Free a software descriptor
+ * @chan: DPDMA channel
+ * @sw_desc: software descriptor to free
+ *
+ * Free a software descriptor from the channel's descriptor pool.
+ */
+static void
+xilinx_dpdma_chan_free_sw_desc(struct xilinx_dpdma_chan *chan,
+ struct xilinx_dpdma_sw_desc *sw_desc)
+{
+ dma_pool_free(chan->desc_pool, sw_desc, sw_desc->dma_addr);
+}
+
+/**
+ * xilinx_dpdma_chan_dump_tx_desc - Dump a tx descriptor
+ * @chan: DPDMA channel
+ * @tx_desc: tx descriptor to dump
+ *
+ * Dump contents of a tx descriptor
+ */
+static void xilinx_dpdma_chan_dump_tx_desc(struct xilinx_dpdma_chan *chan,
+ struct xilinx_dpdma_tx_desc *tx_desc)
+{
+ struct xilinx_dpdma_sw_desc *sw_desc;
+ struct device *dev = chan->xdev->dev;
+ unsigned int i = 0;
+
+ dev_dbg(dev, "------- TX descriptor dump start -------\n");
+ dev_dbg(dev, "------- channel ID = %d -------\n", chan->id);
+
+ list_for_each_entry(sw_desc, &tx_desc->descriptors, node) {
+ struct xilinx_dpdma_hw_desc *hw_desc = &sw_desc->hw;
+
+ dev_dbg(dev, "------- HW descriptor %d -------\n", i++);
+ dev_dbg(dev, "descriptor DMA addr: %pad\n", &sw_desc->dma_addr);
+ dev_dbg(dev, "control: 0x%08x\n", hw_desc->control);
+ dev_dbg(dev, "desc_id: 0x%08x\n", hw_desc->desc_id);
+ dev_dbg(dev, "xfer_size: 0x%08x\n", hw_desc->xfer_size);
+ dev_dbg(dev, "hsize_stride: 0x%08x\n", hw_desc->hsize_stride);
+ dev_dbg(dev, "timestamp_lsb: 0x%08x\n", hw_desc->timestamp_lsb);
+ dev_dbg(dev, "timestamp_msb: 0x%08x\n", hw_desc->timestamp_msb);
+ dev_dbg(dev, "addr_ext: 0x%08x\n", hw_desc->addr_ext);
+ dev_dbg(dev, "next_desc: 0x%08x\n", hw_desc->next_desc);
+ dev_dbg(dev, "src_addr: 0x%08x\n", hw_desc->src_addr);
+ dev_dbg(dev, "addr_ext_23: 0x%08x\n", hw_desc->addr_ext_23);
+ dev_dbg(dev, "addr_ext_45: 0x%08x\n", hw_desc->addr_ext_45);
+ dev_dbg(dev, "src_addr2: 0x%08x\n", hw_desc->src_addr2);
+ dev_dbg(dev, "src_addr3: 0x%08x\n", hw_desc->src_addr3);
+ dev_dbg(dev, "src_addr4: 0x%08x\n", hw_desc->src_addr4);
+ dev_dbg(dev, "src_addr5: 0x%08x\n", hw_desc->src_addr5);
+ dev_dbg(dev, "crc: 0x%08x\n", hw_desc->crc);
+ }
+
+ dev_dbg(dev, "------- TX descriptor dump end -------\n");
+}
+
+/**
+ * xilinx_dpdma_chan_alloc_tx_desc - Allocate a transaction descriptor
+ * @chan: DPDMA channel
+ *
+ * Allocate a tx descriptor.
+ *
+ * Return: a tx descriptor or NULL.
+ */
+static struct xilinx_dpdma_tx_desc *
+xilinx_dpdma_chan_alloc_tx_desc(struct xilinx_dpdma_chan *chan)
+{
+ struct xilinx_dpdma_tx_desc *tx_desc;
+
+ tx_desc = kzalloc(sizeof(*tx_desc), GFP_NOWAIT);
+ if (!tx_desc)
+ return NULL;
+
+ INIT_LIST_HEAD(&tx_desc->descriptors);
+ tx_desc->chan = chan;
+ tx_desc->error = false;
+
+ return tx_desc;
+}
+
+/**
+ * xilinx_dpdma_chan_free_tx_desc - Free a virtual DMA descriptor
+ * @vdesc: virtual DMA descriptor
+ *
+ * Free the virtual DMA descriptor @vdesc including its software descriptors.
+ */
+static void xilinx_dpdma_chan_free_tx_desc(struct virt_dma_desc *vdesc)
+{
+ struct xilinx_dpdma_sw_desc *sw_desc, *next;
+ struct xilinx_dpdma_tx_desc *desc;
+
+ if (!vdesc)
+ return;
+
+ desc = to_dpdma_tx_desc(vdesc);
+
+ list_for_each_entry_safe(sw_desc, next, &desc->descriptors, node) {
+ list_del(&sw_desc->node);
+ xilinx_dpdma_chan_free_sw_desc(desc->chan, sw_desc);
+ }
+
+ kfree(desc);
+}
+
+/**
+ * xilinx_dpdma_chan_prep_interleaved_dma - Prepare an interleaved dma
+ * descriptor
+ * @chan: DPDMA channel
+ * @xt: dma interleaved template
+ *
+ * Prepare a tx descriptor including internal software/hardware descriptors
+ * based on @xt.
+ *
+ * Return: A DPDMA TX descriptor on success, or NULL.
+ */
+static struct xilinx_dpdma_tx_desc *
+xilinx_dpdma_chan_prep_interleaved_dma(struct xilinx_dpdma_chan *chan,
+ struct dma_interleaved_template *xt)
+{
+ struct xilinx_dpdma_tx_desc *tx_desc;
+ struct xilinx_dpdma_sw_desc *sw_desc;
+ struct xilinx_dpdma_hw_desc *hw_desc;
+ size_t hsize = xt->sgl[0].size;
+ size_t stride = hsize + xt->sgl[0].icg;
+
+ if (!IS_ALIGNED(xt->src_start, XILINX_DPDMA_ALIGN_BYTES)) {
+ dev_err(chan->xdev->dev, "buffer should be aligned at %d B\n",
+ XILINX_DPDMA_ALIGN_BYTES);
+ return NULL;
+ }
+
+ tx_desc = xilinx_dpdma_chan_alloc_tx_desc(chan);
+ if (!tx_desc)
+ return NULL;
+
+ sw_desc = xilinx_dpdma_chan_alloc_sw_desc(chan);
+ if (!sw_desc) {
+ xilinx_dpdma_chan_free_tx_desc(&tx_desc->vdesc);
+ return NULL;
+ }
+
+ xilinx_dpdma_sw_desc_set_dma_addrs(chan->xdev, sw_desc, sw_desc,
+ &xt->src_start, 1);
+
+ hw_desc = &sw_desc->hw;
+ hsize = ALIGN(hsize, XILINX_DPDMA_LINESIZE_ALIGN_BITS / 8);
+ hw_desc->xfer_size = hsize * xt->numf;
+ hw_desc->hsize_stride =
+ FIELD_PREP(XILINX_DPDMA_DESC_HSIZE_STRIDE_HSIZE_MASK, hsize) |
+ FIELD_PREP(XILINX_DPDMA_DESC_HSIZE_STRIDE_STRIDE_MASK,
+ stride / 16);
+ hw_desc->control |= XILINX_DPDMA_DESC_CONTROL_PREEMBLE;
+ hw_desc->control |= XILINX_DPDMA_DESC_CONTROL_COMPLETE_INTR;
+ hw_desc->control |= XILINX_DPDMA_DESC_CONTROL_IGNORE_DONE;
+ hw_desc->control |= XILINX_DPDMA_DESC_CONTROL_LAST_OF_FRAME;
+
+ list_add_tail(&sw_desc->node, &tx_desc->descriptors);
+
+ return tx_desc;
+}
+
+/* -----------------------------------------------------------------------------
+ * DPDMA Channel Operations
+ */
+
+/**
+ * xilinx_dpdma_chan_enable - Enable the channel
+ * @chan: DPDMA channel
+ *
+ * Enable the channel and its interrupts. Set the QoS values for video class.
+ */
+static void xilinx_dpdma_chan_enable(struct xilinx_dpdma_chan *chan)
+{
+ u32 reg;
+
+ reg = (XILINX_DPDMA_INTR_CHAN_MASK << chan->id)
+ | XILINX_DPDMA_INTR_GLOBAL_MASK;
+ dpdma_write(chan->xdev->reg, XILINX_DPDMA_IEN, reg);
+ reg = (XILINX_DPDMA_EINTR_CHAN_ERR_MASK << chan->id)
+ | XILINX_DPDMA_INTR_GLOBAL_ERR;
+ dpdma_write(chan->xdev->reg, XILINX_DPDMA_EIEN, reg);
+
+ reg = XILINX_DPDMA_CH_CNTL_ENABLE
+ | FIELD_PREP(XILINX_DPDMA_CH_CNTL_QOS_DSCR_WR_MASK,
+ XILINX_DPDMA_CH_CNTL_QOS_VID_CLASS)
+ | FIELD_PREP(XILINX_DPDMA_CH_CNTL_QOS_DSCR_RD_MASK,
+ XILINX_DPDMA_CH_CNTL_QOS_VID_CLASS)
+ | FIELD_PREP(XILINX_DPDMA_CH_CNTL_QOS_DATA_RD_MASK,
+ XILINX_DPDMA_CH_CNTL_QOS_VID_CLASS);
+ dpdma_set(chan->reg, XILINX_DPDMA_CH_CNTL, reg);
+}
+
+/**
+ * xilinx_dpdma_chan_disable - Disable the channel
+ * @chan: DPDMA channel
+ *
+ * Disable the channel and its interrupts.
+ */
+static void xilinx_dpdma_chan_disable(struct xilinx_dpdma_chan *chan)
+{
+ u32 reg;
+
+ reg = XILINX_DPDMA_INTR_CHAN_MASK << chan->id;
+ dpdma_write(chan->xdev->reg, XILINX_DPDMA_IEN, reg);
+ reg = XILINX_DPDMA_EINTR_CHAN_ERR_MASK << chan->id;
+ dpdma_write(chan->xdev->reg, XILINX_DPDMA_EIEN, reg);
+
+ dpdma_clr(chan->reg, XILINX_DPDMA_CH_CNTL, XILINX_DPDMA_CH_CNTL_ENABLE);
+}
+
+/**
+ * xilinx_dpdma_chan_pause - Pause the channel
+ * @chan: DPDMA channel
+ *
+ * Pause the channel.
+ */
+static void xilinx_dpdma_chan_pause(struct xilinx_dpdma_chan *chan)
+{
+ dpdma_set(chan->reg, XILINX_DPDMA_CH_CNTL, XILINX_DPDMA_CH_CNTL_PAUSE);
+}
+
+/**
+ * xilinx_dpdma_chan_unpause - Unpause the channel
+ * @chan: DPDMA channel
+ *
+ * Unpause the channel.
+ */
+static void xilinx_dpdma_chan_unpause(struct xilinx_dpdma_chan *chan)
+{
+ dpdma_clr(chan->reg, XILINX_DPDMA_CH_CNTL, XILINX_DPDMA_CH_CNTL_PAUSE);
+}
+
+static u32 xilinx_dpdma_chan_video_group_ready(struct xilinx_dpdma_chan *chan)
+{
+ struct xilinx_dpdma_device *xdev = chan->xdev;
+ u32 channels = 0;
+ unsigned int i;
+
+ for (i = ZYNQMP_DPDMA_VIDEO0; i <= ZYNQMP_DPDMA_VIDEO2; i++) {
+ if (xdev->chan[i]->video_group && !xdev->chan[i]->running)
+ return 0;
+
+ if (xdev->chan[i]->video_group)
+ channels |= BIT(i);
+ }
+
+ return channels;
+}
+
+/**
+ * xilinx_dpdma_chan_queue_transfer - Queue the next transfer
+ * @chan: DPDMA channel
+ *
+ * Queue the next descriptor, if any, to the hardware. If the channel is
+ * stopped, start it first. Otherwise retrigger it with the next descriptor.
+ */
+static void xilinx_dpdma_chan_queue_transfer(struct xilinx_dpdma_chan *chan)
+{
+ struct xilinx_dpdma_device *xdev = chan->xdev;
+ struct xilinx_dpdma_sw_desc *sw_desc;
+ struct xilinx_dpdma_tx_desc *desc;
+ struct virt_dma_desc *vdesc;
+ u32 reg, channels;
+
+ lockdep_assert_held(&chan->lock);
+
+ if (chan->desc.pending)
+ return;
+
+ if (!chan->running) {
+ xilinx_dpdma_chan_unpause(chan);
+ xilinx_dpdma_chan_enable(chan);
+ chan->first_frame = true;
+ chan->running = true;
+ }
+
+ if (chan->video_group)
+ channels = xilinx_dpdma_chan_video_group_ready(chan);
+ else
+ channels = BIT(chan->id);
+
+ if (!channels)
+ return;
+
+ vdesc = vchan_next_desc(&chan->vchan);
+ if (!vdesc)
+ return;
+
+ desc = to_dpdma_tx_desc(vdesc);
+ chan->desc.pending = desc;
+ list_del(&desc->vdesc.node);
+
+ /*
+ * Assign the cookie to descriptors in this transaction. Only 16 bit
+ * will be used, but it should be enough.
+ */
+ list_for_each_entry(sw_desc, &desc->descriptors, node)
+ sw_desc->hw.desc_id = desc->vdesc.tx.cookie;
+
+ sw_desc = list_first_entry(&desc->descriptors,
+ struct xilinx_dpdma_sw_desc, node);
+ dpdma_write(chan->reg, XILINX_DPDMA_CH_DESC_START_ADDR,
+ lower_32_bits(sw_desc->dma_addr));
+ if (xdev->ext_addr)
+ dpdma_write(chan->reg, XILINX_DPDMA_CH_DESC_START_ADDRE,
+ FIELD_PREP(XILINX_DPDMA_CH_DESC_START_ADDRE_MASK,
+ upper_32_bits(sw_desc->dma_addr)));
+
+ if (chan->first_frame)
+ reg = XILINX_DPDMA_GBL_TRIG_MASK(channels);
+ else
+ reg = XILINX_DPDMA_GBL_RETRIG_MASK(channels);
+
+ chan->first_frame = false;
+
+ dpdma_write(xdev->reg, XILINX_DPDMA_GBL, reg);
+}
+
+/**
+ * xilinx_dpdma_chan_ostand - Number of outstanding transactions
+ * @chan: DPDMA channel
+ *
+ * Read and return the number of outstanding transactions from register.
+ *
+ * Return: Number of outstanding transactions from the status register.
+ */
+static u32 xilinx_dpdma_chan_ostand(struct xilinx_dpdma_chan *chan)
+{
+ return FIELD_GET(XILINX_DPDMA_CH_STATUS_OTRAN_CNT_MASK,
+ dpdma_read(chan->reg, XILINX_DPDMA_CH_STATUS));
+}
+
+/**
+ * xilinx_dpdma_chan_no_ostand - Notify no outstanding transaction event
+ * @chan: DPDMA channel
+ *
+ * Notify waiters for no outstanding event, so waiters can stop the channel
+ * safely. This function is supposed to be called when 'no outstanding'
+ * interrupt is generated. The 'no outstanding' interrupt is disabled and
+ * should be re-enabled when this event is handled. If the channel status
+ * register still shows some number of outstanding transactions, the interrupt
+ * remains enabled.
+ *
+ * Return: 0 on success. On failure, -EWOULDBLOCK if there's still outstanding
+ * transaction(s).
+ */
+static int xilinx_dpdma_chan_notify_no_ostand(struct xilinx_dpdma_chan *chan)
+{
+ u32 cnt;
+
+ cnt = xilinx_dpdma_chan_ostand(chan);
+ if (cnt) {
+ dev_dbg(chan->xdev->dev, "%d outstanding transactions\n", cnt);
+ return -EWOULDBLOCK;
+ }
+
+ /* Disable 'no outstanding' interrupt */
+ dpdma_write(chan->xdev->reg, XILINX_DPDMA_IDS,
+ XILINX_DPDMA_INTR_NO_OSTAND(chan->id));
+ wake_up(&chan->wait_to_stop);
+
+ return 0;
+}
+
+/**
+ * xilinx_dpdma_chan_wait_no_ostand - Wait for the no outstanding irq
+ * @chan: DPDMA channel
+ *
+ * Wait for the no outstanding transaction interrupt. This functions can sleep
+ * for 50ms.
+ *
+ * Return: 0 on success. On failure, -ETIMEOUT for time out, or the error code
+ * from wait_event_interruptible_timeout().
+ */
+static int xilinx_dpdma_chan_wait_no_ostand(struct xilinx_dpdma_chan *chan)
+{
+ int ret;
+
+ /* Wait for a no outstanding transaction interrupt upto 50msec */
+ ret = wait_event_interruptible_timeout(chan->wait_to_stop,
+ !xilinx_dpdma_chan_ostand(chan),
+ msecs_to_jiffies(50));
+ if (ret > 0) {
+ dpdma_write(chan->xdev->reg, XILINX_DPDMA_IEN,
+ XILINX_DPDMA_INTR_NO_OSTAND(chan->id));
+ return 0;
+ }
+
+ dev_err(chan->xdev->dev, "not ready to stop: %d trans\n",
+ xilinx_dpdma_chan_ostand(chan));
+
+ if (ret == 0)
+ return -ETIMEDOUT;
+
+ return ret;
+}
+
+/**
+ * xilinx_dpdma_chan_poll_no_ostand - Poll the outstanding transaction status
+ * @chan: DPDMA channel
+ *
+ * Poll the outstanding transaction status, and return when there's no
+ * outstanding transaction. This functions can be used in the interrupt context
+ * or where the atomicity is required. Calling thread may wait more than 50ms.
+ *
+ * Return: 0 on success, or -ETIMEDOUT.
+ */
+static int xilinx_dpdma_chan_poll_no_ostand(struct xilinx_dpdma_chan *chan)
+{
+ u32 cnt, loop = 50000;
+
+ /* Poll at least for 50ms (20 fps). */
+ do {
+ cnt = xilinx_dpdma_chan_ostand(chan);
+ udelay(1);
+ } while (loop-- > 0 && cnt);
+
+ if (loop) {
+ dpdma_write(chan->xdev->reg, XILINX_DPDMA_IEN,
+ XILINX_DPDMA_INTR_NO_OSTAND(chan->id));
+ return 0;
+ }
+
+ dev_err(chan->xdev->dev, "not ready to stop: %d trans\n",
+ xilinx_dpdma_chan_ostand(chan));
+
+ return -ETIMEDOUT;
+}
+
+/**
+ * xilinx_dpdma_chan_stop - Stop the channel
+ * @chan: DPDMA channel
+ *
+ * Stop a previously paused channel by first waiting for completion of all
+ * outstanding transaction and then disabling the channel.
+ *
+ * Return: 0 on success, or -ETIMEDOUT if the channel failed to stop.
+ */
+static int xilinx_dpdma_chan_stop(struct xilinx_dpdma_chan *chan)
+{
+ unsigned long flags;
+ int ret;
+
+ ret = xilinx_dpdma_chan_wait_no_ostand(chan);
+ if (ret)
+ return ret;
+
+ spin_lock_irqsave(&chan->lock, flags);
+ xilinx_dpdma_chan_disable(chan);
+ chan->running = false;
+ spin_unlock_irqrestore(&chan->lock, flags);
+
+ return 0;
+}
+
+/**
+ * xilinx_dpdma_chan_done_irq - Handle hardware descriptor completion
+ * @chan: DPDMA channel
+ *
+ * Handle completion of the currently active descriptor (@chan->desc.active). As
+ * we currently support cyclic transfers only, this just invokes the cyclic
+ * callback. The descriptor will be completed at the VSYNC interrupt when a new
+ * descriptor replaces it.
+ */
+static void xilinx_dpdma_chan_done_irq(struct xilinx_dpdma_chan *chan)
+{
+ struct xilinx_dpdma_tx_desc *active = chan->desc.active;
+ unsigned long flags;
+
+ spin_lock_irqsave(&chan->lock, flags);
+
+ if (active)
+ vchan_cyclic_callback(&active->vdesc);
+ else
+ dev_warn(chan->xdev->dev,
+ "DONE IRQ with no active descriptor!\n");
+
+ spin_unlock_irqrestore(&chan->lock, flags);
+}
+
+/**
+ * xilinx_dpdma_chan_vsync_irq - Handle hardware descriptor scheduling
+ * @chan: DPDMA channel
+ *
+ * At VSYNC the active descriptor may have been replaced by the pending
+ * descriptor. Detect this through the DESC_ID and perform appropriate
+ * bookkeeping.
+ */
+static void xilinx_dpdma_chan_vsync_irq(struct xilinx_dpdma_chan *chan)
+{
+ struct xilinx_dpdma_tx_desc *pending;
+ struct xilinx_dpdma_sw_desc *sw_desc;
+ unsigned long flags;
+ u32 desc_id;
+
+ spin_lock_irqsave(&chan->lock, flags);
+
+ pending = chan->desc.pending;
+ if (!chan->running || !pending)
+ goto out;
+
+ desc_id = dpdma_read(chan->reg, XILINX_DPDMA_CH_DESC_ID);
+
+ /* If the retrigger raced with vsync, retry at the next frame. */
+ sw_desc = list_first_entry(&pending->descriptors,
+ struct xilinx_dpdma_sw_desc, node);
+ if (sw_desc->hw.desc_id != desc_id)
+ goto out;
+
+ /*
+ * Complete the active descriptor, if any, promote the pending
+ * descriptor to active, and queue the next transfer, if any.
+ */
+ if (chan->desc.active)
+ vchan_cookie_complete(&chan->desc.active->vdesc);
+ chan->desc.active = pending;
+ chan->desc.pending = NULL;
+
+ xilinx_dpdma_chan_queue_transfer(chan);
+
+out:
+ spin_unlock_irqrestore(&chan->lock, flags);
+}
+
+/**
+ * xilinx_dpdma_chan_err - Detect any channel error
+ * @chan: DPDMA channel
+ * @isr: masked Interrupt Status Register
+ * @eisr: Error Interrupt Status Register
+ *
+ * Return: true if any channel error occurs, or false otherwise.
+ */
+static bool
+xilinx_dpdma_chan_err(struct xilinx_dpdma_chan *chan, u32 isr, u32 eisr)
+{
+ if (!chan)
+ return false;
+
+ if (chan->running &&
+ ((isr & (XILINX_DPDMA_INTR_CHAN_ERR_MASK << chan->id)) ||
+ (eisr & (XILINX_DPDMA_EINTR_CHAN_ERR_MASK << chan->id))))
+ return true;
+
+ return false;
+}
+
+/**
+ * xilinx_dpdma_chan_handle_err - DPDMA channel error handling
+ * @chan: DPDMA channel
+ *
+ * This function is called when any channel error or any global error occurs.
+ * The function disables the paused channel by errors and determines
+ * if the current active descriptor can be rescheduled depending on
+ * the descriptor status.
+ */
+static void xilinx_dpdma_chan_handle_err(struct xilinx_dpdma_chan *chan)
+{
+ struct xilinx_dpdma_device *xdev = chan->xdev;
+ struct xilinx_dpdma_tx_desc *active;
+ unsigned long flags;
+
+ spin_lock_irqsave(&chan->lock, flags);
+
+ dev_dbg(xdev->dev, "cur desc addr = 0x%04x%08x\n",
+ dpdma_read(chan->reg, XILINX_DPDMA_CH_DESC_START_ADDRE),
+ dpdma_read(chan->reg, XILINX_DPDMA_CH_DESC_START_ADDR));
+ dev_dbg(xdev->dev, "cur payload addr = 0x%04x%08x\n",
+ dpdma_read(chan->reg, XILINX_DPDMA_CH_PYLD_CUR_ADDRE),
+ dpdma_read(chan->reg, XILINX_DPDMA_CH_PYLD_CUR_ADDR));
+
+ xilinx_dpdma_chan_disable(chan);
+ chan->running = false;
+
+ if (!chan->desc.active)
+ goto out_unlock;
+
+ active = chan->desc.active;
+ chan->desc.active = NULL;
+
+ xilinx_dpdma_chan_dump_tx_desc(chan, active);
+
+ if (active->error)
+ dev_dbg(xdev->dev, "repeated error on desc\n");
+
+ /* Reschedule if there's no new descriptor */
+ if (!chan->desc.pending &&
+ list_empty(&chan->vchan.desc_issued)) {
+ active->error = true;
+ list_add_tail(&active->vdesc.node,
+ &chan->vchan.desc_issued);
+ } else {
+ xilinx_dpdma_chan_free_tx_desc(&active->vdesc);
+ }
+
+out_unlock:
+ spin_unlock_irqrestore(&chan->lock, flags);
+}
+
+/* -----------------------------------------------------------------------------
+ * DMA Engine Operations
+ */
+
+static struct dma_async_tx_descriptor *
+xilinx_dpdma_prep_interleaved_dma(struct dma_chan *dchan,
+ struct dma_interleaved_template *xt,
+ unsigned long flags)
+{
+ struct xilinx_dpdma_chan *chan = to_xilinx_chan(dchan);
+ struct xilinx_dpdma_tx_desc *desc;
+
+ if (xt->dir != DMA_MEM_TO_DEV)
+ return NULL;
+
+ if (!xt->numf || !xt->sgl[0].size)
+ return NULL;
+
+ if (!(flags & DMA_PREP_REPEAT) || !(flags & DMA_PREP_LOAD_EOT))
+ return NULL;
+
+ desc = xilinx_dpdma_chan_prep_interleaved_dma(chan, xt);
+ if (!desc)
+ return NULL;
+
+ vchan_tx_prep(&chan->vchan, &desc->vdesc, flags | DMA_CTRL_ACK);
+
+ return &desc->vdesc.tx;
+}
+
+/**
+ * xilinx_dpdma_alloc_chan_resources - Allocate resources for the channel
+ * @dchan: DMA channel
+ *
+ * Allocate a descriptor pool for the channel.
+ *
+ * Return: 0 on success, or -ENOMEM if failed to allocate a pool.
+ */
+static int xilinx_dpdma_alloc_chan_resources(struct dma_chan *dchan)
+{
+ struct xilinx_dpdma_chan *chan = to_xilinx_chan(dchan);
+ size_t align = __alignof__(struct xilinx_dpdma_sw_desc);
+
+ chan->desc_pool = dma_pool_create(dev_name(chan->xdev->dev),
+ chan->xdev->dev,
+ sizeof(struct xilinx_dpdma_sw_desc),
+ align, 0);
+ if (!chan->desc_pool) {
+ dev_err(chan->xdev->dev,
+ "failed to allocate a descriptor pool\n");
+ return -ENOMEM;
+ }
+
+ return 0;
+}
+
+/**
+ * xilinx_dpdma_free_chan_resources - Free all resources for the channel
+ * @dchan: DMA channel
+ *
+ * Free resources associated with the virtual DMA channel, and destroy the
+ * descriptor pool.
+ */
+static void xilinx_dpdma_free_chan_resources(struct dma_chan *dchan)
+{
+ struct xilinx_dpdma_chan *chan = to_xilinx_chan(dchan);
+
+ vchan_free_chan_resources(&chan->vchan);
+
+ dma_pool_destroy(chan->desc_pool);
+ chan->desc_pool = NULL;
+}
+
+static void xilinx_dpdma_issue_pending(struct dma_chan *dchan)
+{
+ struct xilinx_dpdma_chan *chan = to_xilinx_chan(dchan);
+ unsigned long flags;
+
+ spin_lock_irqsave(&chan->vchan.lock, flags);
+ if (vchan_issue_pending(&chan->vchan))
+ xilinx_dpdma_chan_queue_transfer(chan);
+ spin_unlock_irqrestore(&chan->vchan.lock, flags);
+}
+
+static int xilinx_dpdma_config(struct dma_chan *dchan,
+ struct dma_slave_config *config)
+{
+ struct xilinx_dpdma_chan *chan = to_xilinx_chan(dchan);
+ unsigned long flags;
+
+ /*
+ * The destination address doesn't need to be specified as the DPDMA is
+ * hardwired to the destination (the DP controller). The transfer
+ * width, burst size and port window size are thus meaningless, they're
+ * fixed both on the DPDMA side and on the DP controller side.
+ */
+
+ spin_lock_irqsave(&chan->lock, flags);
+
+ /*
+ * Abuse the slave_id to indicate that the channel is part of a video
+ * group.
+ */
+ if (chan->id >= ZYNQMP_DPDMA_VIDEO0 && chan->id <= ZYNQMP_DPDMA_VIDEO2)
+ chan->video_group = config->slave_id != 0;
+
+ spin_unlock_irqrestore(&chan->lock, flags);
+
+ return 0;
+}
+
+static int xilinx_dpdma_pause(struct dma_chan *dchan)
+{
+ xilinx_dpdma_chan_pause(to_xilinx_chan(dchan));
+
+ return 0;
+}
+
+static int xilinx_dpdma_resume(struct dma_chan *dchan)
+{
+ xilinx_dpdma_chan_unpause(to_xilinx_chan(dchan));
+
+ return 0;
+}
+
+/**
+ * xilinx_dpdma_terminate_all - Terminate the channel and descriptors
+ * @dchan: DMA channel
+ *
+ * Pause the channel without waiting for ongoing transfers to complete. Waiting
+ * for completion is performed by xilinx_dpdma_synchronize() that will disable
+ * the channel to complete the stop.
+ *
+ * All the descriptors associated with the channel that are guaranteed not to
+ * be touched by the hardware. The pending and active descriptor are not
+ * touched, and will be freed either upon completion, or by
+ * xilinx_dpdma_synchronize().
+ *
+ * Return: 0 on success, or -ETIMEDOUT if the channel failed to stop.
+ */
+static int xilinx_dpdma_terminate_all(struct dma_chan *dchan)
+{
+ struct xilinx_dpdma_chan *chan = to_xilinx_chan(dchan);
+ struct xilinx_dpdma_device *xdev = chan->xdev;
+ LIST_HEAD(descriptors);
+ unsigned long flags;
+ unsigned int i;
+
+ /* Pause the channel (including the whole video group if applicable). */
+ if (chan->video_group) {
+ for (i = ZYNQMP_DPDMA_VIDEO0; i <= ZYNQMP_DPDMA_VIDEO2; i++) {
+ if (xdev->chan[i]->video_group &&
+ xdev->chan[i]->running) {
+ xilinx_dpdma_chan_pause(xdev->chan[i]);
+ xdev->chan[i]->video_group = false;
+ }
+ }
+ } else {
+ xilinx_dpdma_chan_pause(chan);
+ }
+
+ /* Gather all the descriptors we can free and free them. */
+ spin_lock_irqsave(&chan->vchan.lock, flags);
+ vchan_get_all_descriptors(&chan->vchan, &descriptors);
+ spin_unlock_irqrestore(&chan->vchan.lock, flags);
+
+ vchan_dma_desc_free_list(&chan->vchan, &descriptors);
+
+ return 0;
+}
+
+/**
+ * xilinx_dpdma_synchronize - Synchronize callback execution
+ * @dchan: DMA channel
+ *
+ * Synchronizing callback execution ensures that all previously issued
+ * transfers have completed and all associated callbacks have been called and
+ * have returned.
+ *
+ * This function waits for the DMA channel to stop. It assumes it has been
+ * paused by a previous call to dmaengine_terminate_async(), and that no new
+ * pending descriptors have been issued with dma_async_issue_pending(). The
+ * behaviour is undefined otherwise.
+ */
+static void xilinx_dpdma_synchronize(struct dma_chan *dchan)
+{
+ struct xilinx_dpdma_chan *chan = to_xilinx_chan(dchan);
+ unsigned long flags;
+
+ xilinx_dpdma_chan_stop(chan);
+
+ spin_lock_irqsave(&chan->vchan.lock, flags);
+ if (chan->desc.pending) {
+ vchan_terminate_vdesc(&chan->desc.pending->vdesc);
+ chan->desc.pending = NULL;
+ }
+ if (chan->desc.active) {
+ vchan_terminate_vdesc(&chan->desc.active->vdesc);
+ chan->desc.active = NULL;
+ }
+ spin_unlock_irqrestore(&chan->vchan.lock, flags);
+
+ vchan_synchronize(&chan->vchan);
+}
+
+/* -----------------------------------------------------------------------------
+ * Interrupt and Tasklet Handling
+ */
+
+/**
+ * xilinx_dpdma_err - Detect any global error
+ * @isr: Interrupt Status Register
+ * @eisr: Error Interrupt Status Register
+ *
+ * Return: True if any global error occurs, or false otherwise.
+ */
+static bool xilinx_dpdma_err(u32 isr, u32 eisr)
+{
+ if (isr & XILINX_DPDMA_INTR_GLOBAL_ERR ||
+ eisr & XILINX_DPDMA_EINTR_GLOBAL_ERR)
+ return true;
+
+ return false;
+}
+
+/**
+ * xilinx_dpdma_handle_err_irq - Handle DPDMA error interrupt
+ * @xdev: DPDMA device
+ * @isr: masked Interrupt Status Register
+ * @eisr: Error Interrupt Status Register
+ *
+ * Handle if any error occurs based on @isr and @eisr. This function disables
+ * corresponding error interrupts, and those should be re-enabled once handling
+ * is done.
+ */
+static void xilinx_dpdma_handle_err_irq(struct xilinx_dpdma_device *xdev,
+ u32 isr, u32 eisr)
+{
+ bool err = xilinx_dpdma_err(isr, eisr);
+ unsigned int i;
+
+ dev_dbg_ratelimited(xdev->dev,
+ "error irq: isr = 0x%08x, eisr = 0x%08x\n",
+ isr, eisr);
+
+ /* Disable channel error interrupts until errors are handled. */
+ dpdma_write(xdev->reg, XILINX_DPDMA_IDS,
+ isr & ~XILINX_DPDMA_INTR_GLOBAL_ERR);
+ dpdma_write(xdev->reg, XILINX_DPDMA_EIDS,
+ eisr & ~XILINX_DPDMA_EINTR_GLOBAL_ERR);
+
+ for (i = 0; i < ARRAY_SIZE(xdev->chan); i++)
+ if (err || xilinx_dpdma_chan_err(xdev->chan[i], isr, eisr))
+ tasklet_schedule(&xdev->chan[i]->err_task);
+}
+
+/**
+ * xilinx_dpdma_enable_irq - Enable interrupts
+ * @xdev: DPDMA device
+ *
+ * Enable interrupts.
+ */
+static void xilinx_dpdma_enable_irq(struct xilinx_dpdma_device *xdev)
+{
+ dpdma_write(xdev->reg, XILINX_DPDMA_IEN, XILINX_DPDMA_INTR_ALL);
+ dpdma_write(xdev->reg, XILINX_DPDMA_EIEN, XILINX_DPDMA_EINTR_ALL);
+}
+
+/**
+ * xilinx_dpdma_disable_irq - Disable interrupts
+ * @xdev: DPDMA device
+ *
+ * Disable interrupts.
+ */
+static void xilinx_dpdma_disable_irq(struct xilinx_dpdma_device *xdev)
+{
+ dpdma_write(xdev->reg, XILINX_DPDMA_IDS, XILINX_DPDMA_INTR_ERR_ALL);
+ dpdma_write(xdev->reg, XILINX_DPDMA_EIDS, XILINX_DPDMA_EINTR_ALL);
+}
+
+/**
+ * xilinx_dpdma_chan_err_task - Per channel tasklet for error handling
+ * @data: tasklet data to be casted to DPDMA channel structure
+ *
+ * Per channel error handling tasklet. This function waits for the outstanding
+ * transaction to complete and triggers error handling. After error handling,
+ * re-enable channel error interrupts, and restart the channel if needed.
+ */
+static void xilinx_dpdma_chan_err_task(unsigned long data)
+{
+ struct xilinx_dpdma_chan *chan = (struct xilinx_dpdma_chan *)data;
+ struct xilinx_dpdma_device *xdev = chan->xdev;
+ unsigned long flags;
+
+ /* Proceed error handling even when polling fails. */
+ xilinx_dpdma_chan_poll_no_ostand(chan);
+
+ xilinx_dpdma_chan_handle_err(chan);
+
+ dpdma_write(xdev->reg, XILINX_DPDMA_IEN,
+ XILINX_DPDMA_INTR_CHAN_ERR_MASK << chan->id);
+ dpdma_write(xdev->reg, XILINX_DPDMA_EIEN,
+ XILINX_DPDMA_EINTR_CHAN_ERR_MASK << chan->id);
+
+ spin_lock_irqsave(&chan->lock, flags);
+ xilinx_dpdma_chan_queue_transfer(chan);
+ spin_unlock_irqrestore(&chan->lock, flags);
+}
+
+static irqreturn_t xilinx_dpdma_irq_handler(int irq, void *data)
+{
+ struct xilinx_dpdma_device *xdev = data;
+ unsigned long mask;
+ unsigned int i;
+ u32 status;
+ u32 error;
+
+ status = dpdma_read(xdev->reg, XILINX_DPDMA_ISR);
+ error = dpdma_read(xdev->reg, XILINX_DPDMA_EISR);
+ if (!status && !error)
+ return IRQ_NONE;
+
+ dpdma_write(xdev->reg, XILINX_DPDMA_ISR, status);
+ dpdma_write(xdev->reg, XILINX_DPDMA_EISR, error);
+
+ if (status & XILINX_DPDMA_INTR_VSYNC) {
+ /*
+ * There's a single VSYNC interrupt that needs to be processed
+ * by each running channel to update the active descriptor.
+ */
+ for (i = 0; i < ARRAY_SIZE(xdev->chan); i++) {
+ struct xilinx_dpdma_chan *chan = xdev->chan[i];
+
+ if (chan)
+ xilinx_dpdma_chan_vsync_irq(chan);
+ }
+ }
+
+ mask = FIELD_GET(XILINX_DPDMA_INTR_DESC_DONE_MASK, status);
+ if (mask) {
+ for_each_set_bit(i, &mask, ARRAY_SIZE(xdev->chan))
+ xilinx_dpdma_chan_done_irq(xdev->chan[i]);
+ }
+
+ mask = FIELD_GET(XILINX_DPDMA_INTR_NO_OSTAND_MASK, status);
+ if (mask) {
+ for_each_set_bit(i, &mask, ARRAY_SIZE(xdev->chan))
+ xilinx_dpdma_chan_notify_no_ostand(xdev->chan[i]);
+ }
+
+ mask = status & XILINX_DPDMA_INTR_ERR_ALL;
+ if (mask || error)
+ xilinx_dpdma_handle_err_irq(xdev, mask, error);
+
+ return IRQ_HANDLED;
+}
+
+/* -----------------------------------------------------------------------------
+ * Initialization & Cleanup
+ */
+
+static int xilinx_dpdma_chan_init(struct xilinx_dpdma_device *xdev,
+ unsigned int chan_id)
+{
+ struct xilinx_dpdma_chan *chan;
+
+ chan = devm_kzalloc(xdev->dev, sizeof(*chan), GFP_KERNEL);
+ if (!chan)
+ return -ENOMEM;
+
+ chan->id = chan_id;
+ chan->reg = xdev->reg + XILINX_DPDMA_CH_BASE
+ + XILINX_DPDMA_CH_OFFSET * chan->id;
+ chan->running = false;
+ chan->xdev = xdev;
+
+ spin_lock_init(&chan->lock);
+ init_waitqueue_head(&chan->wait_to_stop);
+
+ tasklet_init(&chan->err_task, xilinx_dpdma_chan_err_task,
+ (unsigned long)chan);
+
+ chan->vchan.desc_free = xilinx_dpdma_chan_free_tx_desc;
+ vchan_init(&chan->vchan, &xdev->common);
+
+ xdev->chan[chan->id] = chan;
+
+ return 0;
+}
+
+static void xilinx_dpdma_chan_remove(struct xilinx_dpdma_chan *chan)
+{
+ if (!chan)
+ return;
+
+ tasklet_kill(&chan->err_task);
+ list_del(&chan->vchan.chan.device_node);
+}
+
+static struct dma_chan *of_dma_xilinx_xlate(struct of_phandle_args *dma_spec,
+ struct of_dma *ofdma)
+{
+ struct xilinx_dpdma_device *xdev = ofdma->of_dma_data;
+ uint32_t chan_id = dma_spec->args[0];
+
+ if (chan_id >= ARRAY_SIZE(xdev->chan))
+ return NULL;
+
+ if (!xdev->chan[chan_id])
+ return NULL;
+
+ return dma_get_slave_channel(&xdev->chan[chan_id]->vchan.chan);
+}
+
+static int xilinx_dpdma_probe(struct platform_device *pdev)
+{
+ struct xilinx_dpdma_device *xdev;
+ struct dma_device *ddev;
+ unsigned int i;
+ int ret;
+
+ xdev = devm_kzalloc(&pdev->dev, sizeof(*xdev), GFP_KERNEL);
+ if (!xdev)
+ return -ENOMEM;
+
+ xdev->dev = &pdev->dev;
+ xdev->ext_addr = sizeof(dma_addr_t) > 4;
+
+ INIT_LIST_HEAD(&xdev->common.channels);
+
+ platform_set_drvdata(pdev, xdev);
+
+ xdev->axi_clk = devm_clk_get(xdev->dev, "axi_clk");
+ if (IS_ERR(xdev->axi_clk))
+ return PTR_ERR(xdev->axi_clk);
+
+ xdev->reg = devm_platform_ioremap_resource(pdev, 0);
+ if (IS_ERR(xdev->reg))
+ return PTR_ERR(xdev->reg);
+
+ xdev->irq = platform_get_irq(pdev, 0);
+ if (xdev->irq < 0) {
+ dev_err(xdev->dev, "failed to get platform irq\n");
+ return xdev->irq;
+ }
+
+ ret = request_irq(xdev->irq, xilinx_dpdma_irq_handler, IRQF_SHARED,
+ dev_name(xdev->dev), xdev);
+ if (ret) {
+ dev_err(xdev->dev, "failed to request IRQ\n");
+ return ret;
+ }
+
+ ddev = &xdev->common;
+ ddev->dev = &pdev->dev;
+
+ dma_cap_set(DMA_SLAVE, ddev->cap_mask);
+ dma_cap_set(DMA_PRIVATE, ddev->cap_mask);
+ dma_cap_set(DMA_INTERLEAVE, ddev->cap_mask);
+ dma_cap_set(DMA_REPEAT, ddev->cap_mask);
+ dma_cap_set(DMA_LOAD_EOT, ddev->cap_mask);
+ ddev->copy_align = fls(XILINX_DPDMA_ALIGN_BYTES - 1);
+
+ ddev->device_alloc_chan_resources = xilinx_dpdma_alloc_chan_resources;
+ ddev->device_free_chan_resources = xilinx_dpdma_free_chan_resources;
+ ddev->device_prep_interleaved_dma = xilinx_dpdma_prep_interleaved_dma;
+ /* TODO: Can we achieve better granularity ? */
+ ddev->device_tx_status = dma_cookie_status;
+ ddev->device_issue_pending = xilinx_dpdma_issue_pending;
+ ddev->device_config = xilinx_dpdma_config;
+ ddev->device_pause = xilinx_dpdma_pause;
+ ddev->device_resume = xilinx_dpdma_resume;
+ ddev->device_terminate_all = xilinx_dpdma_terminate_all;
+ ddev->device_synchronize = xilinx_dpdma_synchronize;
+ ddev->src_addr_widths = BIT(DMA_SLAVE_BUSWIDTH_UNDEFINED);
+ ddev->directions = BIT(DMA_MEM_TO_DEV);
+ ddev->residue_granularity = DMA_RESIDUE_GRANULARITY_DESCRIPTOR;
+
+ for (i = 0; i < ARRAY_SIZE(xdev->chan); ++i) {
+ ret = xilinx_dpdma_chan_init(xdev, i);
+ if (ret < 0) {
+ dev_err(xdev->dev, "failed to initialize channel %u\n",
+ i);
+ goto error;
+ }
+ }
+
+ ret = clk_prepare_enable(xdev->axi_clk);
+ if (ret) {
+ dev_err(xdev->dev, "failed to enable the axi clock\n");
+ goto error;
+ }
+
+ ret = dma_async_device_register(ddev);
+ if (ret) {
+ dev_err(xdev->dev, "failed to register the dma device\n");
+ goto error_dma_async;
+ }
+
+ ret = of_dma_controller_register(xdev->dev->of_node,
+ of_dma_xilinx_xlate, ddev);
+ if (ret) {
+ dev_err(xdev->dev, "failed to register DMA to DT DMA helper\n");
+ goto error_of_dma;
+ }
+
+ xilinx_dpdma_enable_irq(xdev);
+
+ dev_info(&pdev->dev, "Xilinx DPDMA engine is probed\n");
+
+ return 0;
+
+error_of_dma:
+ dma_async_device_unregister(ddev);
+error_dma_async:
+ clk_disable_unprepare(xdev->axi_clk);
+error:
+ for (i = 0; i < ARRAY_SIZE(xdev->chan); i++)
+ xilinx_dpdma_chan_remove(xdev->chan[i]);
+
+ free_irq(xdev->irq, xdev);
+
+ return ret;
+}
+
+static int xilinx_dpdma_remove(struct platform_device *pdev)
+{
+ struct xilinx_dpdma_device *xdev = platform_get_drvdata(pdev);
+ unsigned int i;
+
+ /* Start by disabling the IRQ to avoid races during cleanup. */
+ free_irq(xdev->irq, xdev);
+
+ xilinx_dpdma_disable_irq(xdev);
+ of_dma_controller_free(pdev->dev.of_node);
+ dma_async_device_unregister(&xdev->common);
+ clk_disable_unprepare(xdev->axi_clk);
+
+ for (i = 0; i < ARRAY_SIZE(xdev->chan); i++)
+ xilinx_dpdma_chan_remove(xdev->chan[i]);
+
+ return 0;
+}
+
+static const struct of_device_id xilinx_dpdma_of_match[] = {
+ { .compatible = "xlnx,zynqmp-dpdma",},
+ { /* end of table */ },
+};
+MODULE_DEVICE_TABLE(of, xilinx_dpdma_of_match);
+
+static struct platform_driver xilinx_dpdma_driver = {
+ .probe = xilinx_dpdma_probe,
+ .remove = xilinx_dpdma_remove,
+ .driver = {
+ .name = "xilinx-zynqmp-dpdma",
+ .of_match_table = xilinx_dpdma_of_match,
+ },
+};
+
+module_platform_driver(xilinx_dpdma_driver);
+
+MODULE_AUTHOR("Xilinx, Inc.");
+MODULE_DESCRIPTION("Xilinx ZynqMP DPDMA driver");
+MODULE_LICENSE("GPL v2");
--
Regards,
Laurent Pinchart
^ permalink raw reply related
* [PATCH v7 2/5] dmaengine: Add support for repeating transactions
From: Laurent Pinchart @ 2020-07-17 1:33 UTC (permalink / raw)
To: dmaengine
Cc: Michal Simek, Hyun Kwon, Tejas Upadhyay, Satish Kumar Nagireddy,
Vinod Koul, Peter Ujfalusi
In-Reply-To: <20200717013337.24122-1-laurent.pinchart@ideasonboard.com>
DMA engines used with displays perform 2D interleaved transfers to read
framebuffers from memory and feed the data to the display engine. As the
same framebuffer can be displayed for multiple frames, the DMA
transactions need to be repeated until a new framebuffer replaces the
current one. This feature is implemented natively by some DMA engines
that have the ability to repeat transactions and switch to a new
transaction at the end of a transfer without any race condition or frame
loss.
This patch implements support for this feature in the DMA engine API. A
new DMA_PREP_REPEAT transaction flag allows DMA clients to instruct the
DMA channel to repeat the transaction automatically until one or more
new transactions are issued on the channel (or until all active DMA
transfers are explicitly terminated with the dmaengine_terminate_*()
functions). A new DMA_REPEAT transaction type is also added for DMA
engine drivers to report their support of the DMA_PREP_REPEAT flag.
A new DMA_PREP_LOAD_EOT transaction flag is also introduced (with a
corresponding DMA_LOAD_EOT capability bit), as requested during the
review of v4. The flag instructs the DMA channel that the transaction
being queued should replace the active repeated transaction when the
latter terminates (at End Of Transaction). Not setting the flag will
result in the active repeated transaction to continue being repeated,
and the new transaction being silently ignored.
The DMA_PREP_REPEAT flag is currently supported for interleaved
transactions only. Its usage can easily be extended to cover more
transaction types simply by adding an appropriate check in the
corresponding dmaengine_prep_*() function.
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Peter Ujfalusi <peter.ujfalusi@ti.com>
---
Changes since v5:
- Document the API in Documentation/driver-api/dmaengine/
- Small improvements to documentation in header file
Changes since v4:
- Add DMA_LOAD_EOT and DMA_PREP_LOAD_EOT flags
---
Documentation/driver-api/dmaengine/client.rst | 4 +-
.../driver-api/dmaengine/provider.rst | 49 +++++++++++++++++++
include/linux/dmaengine.h | 17 +++++++
3 files changed, 69 insertions(+), 1 deletion(-)
diff --git a/Documentation/driver-api/dmaengine/client.rst b/Documentation/driver-api/dmaengine/client.rst
index 2104830a99ae..41938aa2bdeb 100644
--- a/Documentation/driver-api/dmaengine/client.rst
+++ b/Documentation/driver-api/dmaengine/client.rst
@@ -86,7 +86,9 @@ The details of these operations are:
- interleaved_dma: This is common to Slave as well as M2M clients. For slave
address of devices' fifo could be already known to the driver.
Various types of operations could be expressed by setting
- appropriate values to the 'dma_interleaved_template' members.
+ appropriate values to the 'dma_interleaved_template' members. Cyclic
+ interleaved DMA transfers are also possible if supported by the channel by
+ setting the DMA_PREP_REPEAT transfer flag.
A non-NULL return of this transfer API represents a "descriptor" for
the given transaction.
diff --git a/Documentation/driver-api/dmaengine/provider.rst b/Documentation/driver-api/dmaengine/provider.rst
index 56e5833e8a07..f896acccdfee 100644
--- a/Documentation/driver-api/dmaengine/provider.rst
+++ b/Documentation/driver-api/dmaengine/provider.rst
@@ -239,6 +239,27 @@ Currently, the types available are:
want to transfer a portion of uncompressed data directly to the
display to print it
+- DMA_REPEAT
+
+ - The device supports repeated transfers. A repeated transfer, indicated by
+ the DMA_PREP_REPEAT transfer flag, is similar to a cyclic transfer in that
+ it gets automatically repeated when it ends, but can additionally be
+ replaced by the client.
+
+ - This feature is limited to interleaved transfers, this flag should thus not
+ be set if the DMA_INTERLEAVE flag isn't set. This limitation is based on
+ the current needs of DMA clients, support for additional transfer types
+ should be added in the future if and when the need arises.
+
+- DMA_LOAD_EOT
+
+ - The device supports replacing repeated transfers at end of transfer (EOT)
+ by queuing a new transfer with the DMA_PREP_LOAD_EOT flag set.
+
+ - Support for replacing a currently running transfer at another point (such
+ as end of burst instead of end of transfer) will be added in the future
+ based on DMA clients needs, if and when the need arises.
+
These various types will also affect how the source and destination
addresses change over time.
@@ -531,6 +552,34 @@ DMA_CTRL_REUSE
writes for which the descriptor should be in different format from
normal data descriptors.
+- DMA_PREP_REPEAT
+
+ - If set, the transfer will be automatically repeated when it ends until a
+ new transfer is queued on the same channel with the DMA_PREP_LOAD_EOT flag.
+ If the next transfer to be queued on the channel does not have the
+ DMA_PREP_LOAD_EOT flag set, the current transfer will be repeated until the
+ client terminates all transfers.
+
+ - This flag is only supported if the channel reports the DMA_REPEAT
+ capability.
+
+- DMA_PREP_LOAD_EOT
+
+ - If set, the transfer will replace the transfer currently being executed at
+ the end of the transfer.
+
+ - This is the default behaviour for non-repeated transfers, specifying
+ DMA_PREP_LOAD_EOT for non-repeated transfers will thus make no difference.
+
+ - When using repeated transfers, DMA clients will usually need to set the
+ DMA_PREP_LOAD_EOT flag on all transfers, otherwise the channel will keep
+ repeating the last repeated transfer and ignore the new transfers being
+ queued. Failure to set DMA_PREP_LOAD_EOT will appear as if the channel was
+ stuck on the previous transfer.
+
+ - This flag is only supported if the channel reports the DMA_LOAD_EOT
+ capability.
+
General Design Notes
====================
diff --git a/include/linux/dmaengine.h b/include/linux/dmaengine.h
index e1c03339918f..328e3aca7f51 100644
--- a/include/linux/dmaengine.h
+++ b/include/linux/dmaengine.h
@@ -61,6 +61,8 @@ enum dma_transaction_type {
DMA_SLAVE,
DMA_CYCLIC,
DMA_INTERLEAVE,
+ DMA_REPEAT,
+ DMA_LOAD_EOT,
/* last transaction type for creation of the capabilities mask */
DMA_TX_TYPE_END,
};
@@ -176,6 +178,16 @@ struct dma_interleaved_template {
* @DMA_PREP_CMD: tell the driver that the data passed to DMA API is command
* data and the descriptor should be in different format from normal
* data descriptors.
+ * @DMA_PREP_REPEAT: tell the driver that the transaction shall be automatically
+ * repeated when it ends until a transaction is issued on the same channel
+ * with the DMA_PREP_LOAD_EOT flag set. This flag is only applicable to
+ * interleaved transactions and is ignored for all other transaction types.
+ * @DMA_PREP_LOAD_EOT: tell the driver that the transaction shall replace any
+ * active repeated (as indicated by DMA_PREP_REPEAT) transaction when the
+ * repeated transaction ends. Not setting this flag when the previously queued
+ * transaction is marked with DMA_PREP_REPEAT will cause the new transaction
+ * to never be processed and stay in the issued queue forever. The flag is
+ * ignored if the previous transaction is not a repeated transaction.
*/
enum dma_ctrl_flags {
DMA_PREP_INTERRUPT = (1 << 0),
@@ -186,6 +198,8 @@ enum dma_ctrl_flags {
DMA_PREP_FENCE = (1 << 5),
DMA_CTRL_REUSE = (1 << 6),
DMA_PREP_CMD = (1 << 7),
+ DMA_PREP_REPEAT = (1 << 8),
+ DMA_PREP_LOAD_EOT = (1 << 9),
};
/**
@@ -980,6 +994,9 @@ static inline struct dma_async_tx_descriptor *dmaengine_prep_interleaved_dma(
{
if (!chan || !chan->device || !chan->device->device_prep_interleaved_dma)
return NULL;
+ if (flags & DMA_PREP_REPEAT &&
+ !test_bit(DMA_REPEAT, chan->device->cap_mask.bits))
+ return NULL;
return chan->device->device_prep_interleaved_dma(chan, xt, flags);
}
--
Regards,
Laurent Pinchart
^ permalink raw reply related
* [PATCH v7 4/5] dmaengine: xilinx: dpdma: Add debugfs support
From: Laurent Pinchart @ 2020-07-17 1:33 UTC (permalink / raw)
To: dmaengine
Cc: Michal Simek, Hyun Kwon, Tejas Upadhyay, Satish Kumar Nagireddy,
Vinod Koul, Peter Ujfalusi
In-Reply-To: <20200717013337.24122-1-laurent.pinchart@ideasonboard.com>
Expose statistics to debugfs when available. This helps debugging issues
with the DPDMA driver.
Signed-off-by: Hyun Kwon <hyun.kwon@xilinx.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
---
Changes since v6:
- Move debugfs node to the dmaengine directory
- Destroy debugfs directory upon removal
Changes since v3:
- Return -EFAULT instead of bytes remaining after copy_to_user()
Changes since v2:
- Refactor debugfs code
---
drivers/dma/xilinx/xilinx_dpdma.c | 238 ++++++++++++++++++++++++++++++
1 file changed, 238 insertions(+)
diff --git a/drivers/dma/xilinx/xilinx_dpdma.c b/drivers/dma/xilinx/xilinx_dpdma.c
index af88a6762ef4..753c0c8c4833 100644
--- a/drivers/dma/xilinx/xilinx_dpdma.c
+++ b/drivers/dma/xilinx/xilinx_dpdma.c
@@ -10,6 +10,7 @@
#include <linux/bitfield.h>
#include <linux/bits.h>
#include <linux/clk.h>
+#include <linux/debugfs.h>
#include <linux/delay.h>
#include <linux/dmaengine.h>
#include <linux/dmapool.h>
@@ -252,6 +253,7 @@ struct xilinx_dpdma_chan {
* @axi_clk: axi clock
* @chan: DPDMA channels
* @ext_addr: flag for 64 bit system (48 bit addressing)
+ * @debugfs_dir: root debugfs directory for the device
*/
struct xilinx_dpdma_device {
struct dma_device common;
@@ -263,8 +265,238 @@ struct xilinx_dpdma_device {
struct xilinx_dpdma_chan *chan[XILINX_DPDMA_NUM_CHAN];
bool ext_addr;
+
+ struct dentry *debugfs_dir;
};
+/* -----------------------------------------------------------------------------
+ * DebugFS
+ */
+
+#ifdef CONFIG_DEBUG_FS
+
+#define XILINX_DPDMA_DEBUGFS_READ_MAX_SIZE 32
+#define XILINX_DPDMA_DEBUGFS_UINT16_MAX_STR "65535"
+
+/* Match xilinx_dpdma_testcases vs dpdma_debugfs_reqs[] entry */
+enum xilinx_dpdma_testcases {
+ DPDMA_TC_INTR_DONE,
+ DPDMA_TC_NONE
+};
+
+struct xilinx_dpdma_debugfs {
+ enum xilinx_dpdma_testcases testcase;
+ u16 xilinx_dpdma_irq_done_count;
+ unsigned int chan_id;
+};
+
+static struct xilinx_dpdma_debugfs dpdma_debugfs;
+struct xilinx_dpdma_debugfs_request {
+ const char *name;
+ enum xilinx_dpdma_testcases tc;
+ ssize_t (*read)(char *buf);
+ int (*write)(char *args);
+};
+
+static void xilinx_dpdma_debugfs_desc_done_irq(struct xilinx_dpdma_chan *chan)
+{
+ if (chan->id == dpdma_debugfs.chan_id)
+ dpdma_debugfs.xilinx_dpdma_irq_done_count++;
+}
+
+static ssize_t xilinx_dpdma_debugfs_desc_done_irq_read(char *buf)
+{
+ size_t out_str_len;
+
+ dpdma_debugfs.testcase = DPDMA_TC_NONE;
+
+ out_str_len = strlen(XILINX_DPDMA_DEBUGFS_UINT16_MAX_STR);
+ out_str_len = min_t(size_t, XILINX_DPDMA_DEBUGFS_READ_MAX_SIZE,
+ out_str_len);
+ snprintf(buf, out_str_len, "%d",
+ dpdma_debugfs.xilinx_dpdma_irq_done_count);
+
+ return 0;
+}
+
+static int xilinx_dpdma_debugfs_desc_done_irq_write(char *args)
+{
+ char *arg;
+ int ret;
+ u32 id;
+
+ arg = strsep(&args, " ");
+ if (!arg || strncasecmp(arg, "start", 5))
+ return -EINVAL;
+
+ arg = strsep(&args, " ");
+ if (!arg)
+ return -EINVAL;
+
+ ret = kstrtou32(arg, 0, &id);
+ if (ret < 0)
+ return ret;
+
+ if (id < ZYNQMP_DPDMA_VIDEO0 || id > ZYNQMP_DPDMA_AUDIO1)
+ return -EINVAL;
+
+ dpdma_debugfs.testcase = DPDMA_TC_INTR_DONE;
+ dpdma_debugfs.xilinx_dpdma_irq_done_count = 0;
+ dpdma_debugfs.chan_id = id;
+
+ return 0;
+}
+
+/* Match xilinx_dpdma_testcases vs dpdma_debugfs_reqs[] entry */
+struct xilinx_dpdma_debugfs_request dpdma_debugfs_reqs[] = {
+ {
+ .name = "DESCRIPTOR_DONE_INTR",
+ .tc = DPDMA_TC_INTR_DONE,
+ .read = xilinx_dpdma_debugfs_desc_done_irq_read,
+ .write = xilinx_dpdma_debugfs_desc_done_irq_write,
+ },
+};
+
+static ssize_t xilinx_dpdma_debugfs_read(struct file *f, char __user *buf,
+ size_t size, loff_t *pos)
+{
+ enum xilinx_dpdma_testcases testcase;
+ char *kern_buff;
+ int ret = 0;
+
+ if (*pos != 0 || size <= 0)
+ return -EINVAL;
+
+ kern_buff = kzalloc(XILINX_DPDMA_DEBUGFS_READ_MAX_SIZE, GFP_KERNEL);
+ if (!kern_buff) {
+ dpdma_debugfs.testcase = DPDMA_TC_NONE;
+ return -ENOMEM;
+ }
+
+ testcase = READ_ONCE(dpdma_debugfs.testcase);
+ if (testcase != DPDMA_TC_NONE) {
+ ret = dpdma_debugfs_reqs[testcase].read(kern_buff);
+ if (ret < 0)
+ goto done;
+ } else {
+ strlcpy(kern_buff, "No testcase executed",
+ XILINX_DPDMA_DEBUGFS_READ_MAX_SIZE);
+ }
+
+ size = min(size, strlen(kern_buff));
+ if (copy_to_user(buf, kern_buff, size))
+ ret = -EFAULT;
+
+done:
+ kfree(kern_buff);
+ if (ret)
+ return ret;
+
+ *pos = size + 1;
+ return size;
+}
+
+static ssize_t xilinx_dpdma_debugfs_write(struct file *f,
+ const char __user *buf, size_t size,
+ loff_t *pos)
+{
+ char *kern_buff, *kern_buff_start;
+ char *testcase;
+ unsigned int i;
+ int ret;
+
+ if (*pos != 0 || size <= 0)
+ return -EINVAL;
+
+ /* Supporting single instance of test as of now. */
+ if (dpdma_debugfs.testcase != DPDMA_TC_NONE)
+ return -EBUSY;
+
+ kern_buff = kzalloc(size, GFP_KERNEL);
+ if (!kern_buff)
+ return -ENOMEM;
+ kern_buff_start = kern_buff;
+
+ ret = strncpy_from_user(kern_buff, buf, size);
+ if (ret < 0)
+ goto done;
+
+ /* Read the testcase name from a user request. */
+ testcase = strsep(&kern_buff, " ");
+
+ for (i = 0; i < ARRAY_SIZE(dpdma_debugfs_reqs); i++) {
+ if (!strcasecmp(testcase, dpdma_debugfs_reqs[i].name))
+ break;
+ }
+
+ if (i == ARRAY_SIZE(dpdma_debugfs_reqs)) {
+ ret = -EINVAL;
+ goto done;
+ }
+
+ ret = dpdma_debugfs_reqs[i].write(kern_buff);
+ if (ret < 0)
+ goto done;
+
+ ret = size;
+
+done:
+ kfree(kern_buff_start);
+ return ret;
+}
+
+static const struct file_operations fops_xilinx_dpdma_dbgfs = {
+ .owner = THIS_MODULE,
+ .read = xilinx_dpdma_debugfs_read,
+ .write = xilinx_dpdma_debugfs_write,
+};
+
+static void xilinx_dpdma_debugfs_cleanup(struct xilinx_dpdma_device *xdev)
+{
+ debugfs_remove_recursive(xdev->debugfs_dir);
+ xdev->debugfs_dir = NULL;
+}
+
+static int xilinx_dpdma_debugfs_init(struct xilinx_dpdma_device *xdev)
+{
+ struct dentry *dent;
+
+ dpdma_debugfs.testcase = DPDMA_TC_NONE;
+
+ dent = debugfs_create_dir("dpdma", xdev->common.dbg_dev_root);
+ if (IS_ERR(dent)) {
+ dev_err(xdev->dev, "Failed to create debugfs directory\n");
+ return -ENODEV;
+ }
+
+ xdev->debugfs_dir = dent;
+
+ dent = debugfs_create_file("testcase", 0444, xdev->debugfs_dir, NULL,
+ &fops_xilinx_dpdma_dbgfs);
+ if (IS_ERR(dent)) {
+ dev_err(xdev->dev, "Failed to cerate debugfs testcase file\n");
+ xilinx_dpdma_debugfs_cleanup(xdev);
+ return -ENODEV;
+ }
+
+ return 0;
+}
+
+#else
+static void xilinx_dpdma_debugfs_cleanup(struct xilinx_dpdma_device *xdev)
+{
+}
+
+static int xilinx_dpdma_debugfs_init(struct xilinx_dpdma_device *xdev)
+{
+ return 0;
+}
+
+static void xilinx_dpdma_debugfs_desc_done_irq(struct xilinx_dpdma_chan *chan)
+{
+}
+#endif /* CONFIG_DEBUG_FS */
+
/* -----------------------------------------------------------------------------
* I/O Accessors
*/
@@ -840,6 +1072,8 @@ static void xilinx_dpdma_chan_done_irq(struct xilinx_dpdma_chan *chan)
spin_lock_irqsave(&chan->lock, flags);
+ xilinx_dpdma_debugfs_desc_done_irq(chan);
+
if (active)
vchan_cyclic_callback(&active->vdesc);
else
@@ -1475,6 +1709,8 @@ static int xilinx_dpdma_probe(struct platform_device *pdev)
xilinx_dpdma_enable_irq(xdev);
+ xilinx_dpdma_debugfs_init(xdev);
+
dev_info(&pdev->dev, "Xilinx DPDMA engine is probed\n");
return 0;
@@ -1500,6 +1736,8 @@ static int xilinx_dpdma_remove(struct platform_device *pdev)
/* Start by disabling the IRQ to avoid races during cleanup. */
free_irq(xdev->irq, xdev);
+ xilinx_dpdma_debugfs_cleanup(xdev);
+
xilinx_dpdma_disable_irq(xdev);
of_dma_controller_free(pdev->dev.of_node);
dma_async_device_unregister(&xdev->common);
--
Regards,
Laurent Pinchart
^ permalink raw reply related
* [PATCH v7 5/5] arm64: dts: zynqmp: Add DPDMA node
From: Laurent Pinchart @ 2020-07-17 1:33 UTC (permalink / raw)
To: dmaengine
Cc: Michal Simek, Hyun Kwon, Tejas Upadhyay, Satish Kumar Nagireddy,
Vinod Koul, Peter Ujfalusi
In-Reply-To: <20200717013337.24122-1-laurent.pinchart@ideasonboard.com>
Add a DT node for the DisplayPort DMA engine (DPDMA).
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Acked-by: Michal Simek <michal.simek@xilinx.com>
---
arch/arm64/boot/dts/xilinx/zynqmp-clk-ccf.dtsi | 4 ++++
arch/arm64/boot/dts/xilinx/zynqmp.dtsi | 10 ++++++++++
2 files changed, 14 insertions(+)
diff --git a/arch/arm64/boot/dts/xilinx/zynqmp-clk-ccf.dtsi b/arch/arm64/boot/dts/xilinx/zynqmp-clk-ccf.dtsi
index 9868ca15dfc5..32c4914738d9 100644
--- a/arch/arm64/boot/dts/xilinx/zynqmp-clk-ccf.dtsi
+++ b/arch/arm64/boot/dts/xilinx/zynqmp-clk-ccf.dtsi
@@ -57,6 +57,10 @@ &cpu0 {
clocks = <&zynqmp_clk ACPU>;
};
+&dpdma {
+ clocks = <&zynqmp_clk DPDMA_REF>;
+};
+
&fpd_dma_chan1 {
clocks = <&zynqmp_clk GDMA_REF>, <&zynqmp_clk LPD_LSBUS>;
};
diff --git a/arch/arm64/boot/dts/xilinx/zynqmp.dtsi b/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
index 9174ddc76bdc..d2554640db57 100644
--- a/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
+++ b/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
@@ -250,6 +250,16 @@ pmu@9000 {
};
};
+ dpdma: dma-controller@fd4c0000 {
+ compatible = "xlnx,zynqmp-dpdma";
+ status = "disabled";
+ reg = <0x0 0xfd4c0000 0x0 0x1000>;
+ interrupts = <0 122 4>;
+ interrupt-parent = <&gic>;
+ clock-names = "axi_clk";
+ #dma-cells = <1>;
+ };
+
/* GDMA */
fpd_dma_chan1: dma@fd500000 {
status = "disabled";
--
Regards,
Laurent Pinchart
^ permalink raw reply related
* [PATCH v2 0/7] add support for direct I/O with fscrypt using blk-crypto
From: Satya Tangirala @ 2020-07-17 1:35 UTC (permalink / raw)
To: linux-fscrypt, linux-fsdevel, linux-f2fs-devel, linux-ext4
Cc: linux-xfs, Satya Tangirala
This patch series adds support for direct I/O with fscrypt using
blk-crypto. It has been rebased on fscrypt/master.
Patch 1 adds two functions to fscrypt that need to be called to determine
if direct I/O is supported for a request.
Patches 2 and 3 wire up direct-io and iomap respectively with the functions
introduced in Patch 1 and set bio crypt contexts on bios when appropriate
by calling into fscrypt.
Patches 4 and 5 allow ext4 and f2fs direct I/O to support fscrypt without
falling back to buffered I/O.
Patches 6 and 7 update the fscrypt documentation for inline encryption
support and direct I/O. The documentation now notes the required conditions
for inline encryption and direct I/O on encrypted files.
This patch series was tested by running xfstests with test_dummy_encryption
with and without the 'inlinecrypt' mount option, and there were no
meaningful regressions. One regression was for generic/587 on ext4,
but that test isn't compatible with test_dummy_encryption in the first
place, and the test "incorrectly" passes without the 'inlinecrypt' mount
option - a patch will be sent out to exclude that test when
test_dummy_encryption is turned on with ext4 (like the other quota related
tests that use user visible quota files). The other regression was for
generic/252 on ext4, which does direct I/O with a buffer aligned to the
block device's blocksize, but not necessarily aligned to the filesystem's
block size, which direct I/O with fscrypt requires.
Eric Biggers (5):
fscrypt: Add functions for direct I/O support
direct-io: add support for fscrypt using blk-crypto
iomap: support direct I/O with fscrypt using blk-crypto
ext4: support direct I/O with fscrypt using blk-crypto
f2fs: support direct I/O with fscrypt using blk-crypto
Satya Tangirala (2):
fscrypt: document inline encryption support
fscrypt: update documentation for direct I/O support
Documentation/filesystems/fscrypt.rst | 36 +++++++++++-
fs/crypto/crypto.c | 8 +++
fs/crypto/inline_crypt.c | 80 +++++++++++++++++++++++++++
fs/direct-io.c | 15 ++++-
fs/ext4/file.c | 10 ++--
fs/f2fs/f2fs.h | 6 +-
fs/iomap/direct-io.c | 8 +++
include/linux/fscrypt.h | 19 +++++++
8 files changed, 173 insertions(+), 9 deletions(-)
--
2.28.0.rc0.105.gf9edc3c819-goog
^ permalink raw reply
* [PATCH v2 1/7] fscrypt: Add functions for direct I/O support
From: Satya Tangirala @ 2020-07-17 1:35 UTC (permalink / raw)
To: linux-fscrypt, linux-fsdevel, linux-f2fs-devel, linux-ext4
Cc: linux-xfs, Eric Biggers, Satya Tangirala
In-Reply-To: <20200717013518.59219-1-satyat@google.com>
From: Eric Biggers <ebiggers@google.com>
Introduce fscrypt_dio_supported() to check whether a direct I/O request
is unsupported due to encryption constraints, and
fscrypt_limit_io_pages() to check how many pages may be added to a bio
being prepared for direct I/O.
The IV_INO_LBLK_32 fscrypt policy introduced the possibility that DUNs
in logically continuous file blocks might wrap from 0xffffffff to 0.
Since this was particularly difficult to handle when block_size !=
PAGE_SIZE, fscrypt only supports blk-crypto en/decryption with
the IV_INO_LBLK_32 policy when block_size == PAGE_SIZE, and ensures that
the DUN never wraps around within any submitted bio.
fscrypt_limit_io_pages() can be used to determine the number of logically
contiguous blocks/pages that may be added to the bio without causing the
DUN to wrap around within the bio. This is an alternative to calling
fscrypt_mergeable_bio() on each page in a range of logically contiguous
pages.
Signed-off-by: Eric Biggers <ebiggers@google.com>
Co-developed-by: Satya Tangirala <satyat@google.com>
Signed-off-by: Satya Tangirala <satyat@google.com>
---
fs/crypto/crypto.c | 8 ++++
fs/crypto/inline_crypt.c | 80 ++++++++++++++++++++++++++++++++++++++++
include/linux/fscrypt.h | 19 ++++++++++
3 files changed, 107 insertions(+)
diff --git a/fs/crypto/crypto.c b/fs/crypto/crypto.c
index a52cf32733ab..fb34364360b3 100644
--- a/fs/crypto/crypto.c
+++ b/fs/crypto/crypto.c
@@ -69,6 +69,14 @@ void fscrypt_free_bounce_page(struct page *bounce_page)
}
EXPORT_SYMBOL(fscrypt_free_bounce_page);
+/*
+ * Generate the IV for the given logical block number within the given file.
+ * For filenames encryption, lblk_num == 0.
+ *
+ * Keep this in sync with fscrypt_limit_io_pages(). fscrypt_limit_io_pages()
+ * needs to know about any IV generation methods where the low bits of IV don't
+ * simply contain the lblk_num (e.g., IV_INO_LBLK_32).
+ */
void fscrypt_generate_iv(union fscrypt_iv *iv, u64 lblk_num,
const struct fscrypt_info *ci)
{
diff --git a/fs/crypto/inline_crypt.c b/fs/crypto/inline_crypt.c
index d7aecadf33c1..f5af6a63e04c 100644
--- a/fs/crypto/inline_crypt.c
+++ b/fs/crypto/inline_crypt.c
@@ -16,6 +16,7 @@
#include <linux/blkdev.h>
#include <linux/buffer_head.h>
#include <linux/sched/mm.h>
+#include <linux/uio.h>
#include "fscrypt_private.h"
@@ -362,3 +363,82 @@ bool fscrypt_mergeable_bio_bh(struct bio *bio,
return fscrypt_mergeable_bio(bio, inode, next_lblk);
}
EXPORT_SYMBOL_GPL(fscrypt_mergeable_bio_bh);
+
+/**
+ * fscrypt_dio_supported() - check whether a direct I/O request is unsupported
+ * due to encryption constraints
+ * @iocb: the file and position the I/O is targeting
+ * @iter: the I/O data segment(s)
+ *
+ * Return: true if direct I/O is supported
+ */
+bool fscrypt_dio_supported(struct kiocb *iocb, struct iov_iter *iter)
+{
+ const struct inode *inode = file_inode(iocb->ki_filp);
+ const unsigned int blocksize = i_blocksize(inode);
+
+ /* If the file is unencrypted, no veto from us. */
+ if (!fscrypt_needs_contents_encryption(inode))
+ return true;
+
+ /* We only support direct I/O with inline crypto, not fs-layer crypto */
+ if (!fscrypt_inode_uses_inline_crypto(inode))
+ return false;
+
+ /*
+ * Since the granularity of encryption is filesystem blocks, the I/O
+ * must be block aligned -- not just disk sector aligned.
+ */
+ if (!IS_ALIGNED(iocb->ki_pos | iov_iter_alignment(iter), blocksize))
+ return false;
+
+ return true;
+}
+EXPORT_SYMBOL_GPL(fscrypt_dio_supported);
+
+/**
+ * fscrypt_limit_io_pages() - limit I/O pages to avoid discontiguous DUNs
+ * @inode: the file on which I/O is being done
+ * @pos: the file position (in bytes) at which the I/O is being done
+ * @nr_pages: the number of pages we want to submit starting at @pos
+ *
+ * Determine the limit to the number of pages that can be submitted in the bio
+ * targeting @pos without causing a data unit number (DUN) discontinuity.
+ *
+ * For IV generation methods that can't cause DUN wraparounds
+ * within logically continuous data blocks, the maximum number of pages is
+ * simply @nr_pages. For those IV generation methods that *might* cause DUN
+ * wraparounds, the returned number of pages is the largest possible number of
+ * pages (less than @nr_pages) that can be added to the bio without causing a
+ * DUN wraparound within the bio.
+ *
+ * Return: the actual number of pages that can be submitted
+ */
+int fscrypt_limit_io_pages(const struct inode *inode, loff_t pos, int nr_pages)
+{
+ const struct fscrypt_info *ci = inode->i_crypt_info;
+ u32 dun;
+
+ if (!fscrypt_inode_uses_inline_crypto(inode))
+ return nr_pages;
+
+ if (nr_pages <= 1)
+ return nr_pages;
+
+ if (!(fscrypt_policy_flags(&ci->ci_policy) &
+ FSCRYPT_POLICY_FLAG_IV_INO_LBLK_32))
+ return nr_pages;
+
+ /*
+ * fscrypt_select_encryption_impl() ensures that block_size == PAGE_SIZE
+ * when using FSCRYPT_POLICY_FLAG_IV_INO_LBLK_32.
+ */
+ if (WARN_ON_ONCE(i_blocksize(inode) != PAGE_SIZE))
+ return 1;
+
+ /* With IV_INO_LBLK_32, the DUN can wrap around from U32_MAX to 0. */
+
+ dun = ci->ci_hashed_ino + (pos >> inode->i_blkbits);
+
+ return min_t(u64, nr_pages, (u64)U32_MAX + 1 - dun);
+}
diff --git a/include/linux/fscrypt.h b/include/linux/fscrypt.h
index bb257411365f..c205c214b35e 100644
--- a/include/linux/fscrypt.h
+++ b/include/linux/fscrypt.h
@@ -559,6 +559,11 @@ bool fscrypt_mergeable_bio(struct bio *bio, const struct inode *inode,
bool fscrypt_mergeable_bio_bh(struct bio *bio,
const struct buffer_head *next_bh);
+bool fscrypt_dio_supported(struct kiocb *iocb, struct iov_iter *iter);
+
+int fscrypt_limit_io_pages(const struct inode *inode, loff_t pos,
+ int nr_pages);
+
#else /* CONFIG_FS_ENCRYPTION_INLINE_CRYPT */
static inline bool __fscrypt_inode_uses_inline_crypto(const struct inode *inode)
@@ -587,6 +592,20 @@ static inline bool fscrypt_mergeable_bio_bh(struct bio *bio,
{
return true;
}
+
+static inline bool fscrypt_dio_supported(struct kiocb *iocb,
+ struct iov_iter *iter)
+{
+ const struct inode *inode = file_inode(iocb->ki_filp);
+
+ return !fscrypt_needs_contents_encryption(inode);
+}
+
+static inline int fscrypt_limit_io_pages(const struct inode *inode, loff_t pos,
+ int nr_pages)
+{
+ return nr_pages;
+}
#endif /* !CONFIG_FS_ENCRYPTION_INLINE_CRYPT */
/**
--
2.28.0.rc0.105.gf9edc3c819-goog
^ permalink raw reply related
* [PATCH v2 2/7] direct-io: add support for fscrypt using blk-crypto
From: Satya Tangirala @ 2020-07-17 1:35 UTC (permalink / raw)
To: linux-fscrypt, linux-fsdevel, linux-f2fs-devel, linux-ext4
Cc: linux-xfs, Eric Biggers, Satya Tangirala
In-Reply-To: <20200717013518.59219-1-satyat@google.com>
From: Eric Biggers <ebiggers@google.com>
Set bio crypt contexts on bios by calling into fscrypt when required,
and explicitly check for DUN continuity when adding pages to the bio.
(While DUN continuity is usually implied by logical block contiguity,
this is not the case when using certain fscrypt IV generation methods
like IV_INO_LBLK_32).
Signed-off-by: Eric Biggers <ebiggers@google.com>
Co-developed-by: Satya Tangirala <satyat@google.com>
Signed-off-by: Satya Tangirala <satyat@google.com>
---
fs/direct-io.c | 15 ++++++++++++++-
1 file changed, 14 insertions(+), 1 deletion(-)
diff --git a/fs/direct-io.c b/fs/direct-io.c
index 6d5370eac2a8..f27f7e3780ee 100644
--- a/fs/direct-io.c
+++ b/fs/direct-io.c
@@ -24,6 +24,7 @@
#include <linux/module.h>
#include <linux/types.h>
#include <linux/fs.h>
+#include <linux/fscrypt.h>
#include <linux/mm.h>
#include <linux/slab.h>
#include <linux/highmem.h>
@@ -411,6 +412,7 @@ dio_bio_alloc(struct dio *dio, struct dio_submit *sdio,
sector_t first_sector, int nr_vecs)
{
struct bio *bio;
+ struct inode *inode = dio->inode;
/*
* bio_alloc() is guaranteed to return a bio when allowed to sleep and
@@ -418,6 +420,9 @@ dio_bio_alloc(struct dio *dio, struct dio_submit *sdio,
*/
bio = bio_alloc(GFP_KERNEL, nr_vecs);
+ fscrypt_set_bio_crypt_ctx(bio, inode,
+ sdio->cur_page_fs_offset >> inode->i_blkbits,
+ GFP_KERNEL);
bio_set_dev(bio, bdev);
bio->bi_iter.bi_sector = first_sector;
bio_set_op_attrs(bio, dio->op, dio->op_flags);
@@ -782,9 +787,17 @@ static inline int dio_send_cur_page(struct dio *dio, struct dio_submit *sdio,
* current logical offset in the file does not equal what would
* be the next logical offset in the bio, submit the bio we
* have.
+ *
+ * When fscrypt inline encryption is used, data unit number
+ * (DUN) contiguity is also required. Normally that's implied
+ * by logical contiguity. However, certain IV generation
+ * methods (e.g. IV_INO_LBLK_32) don't guarantee it. So, we
+ * must explicitly check fscrypt_mergeable_bio() too.
*/
if (sdio->final_block_in_bio != sdio->cur_page_block ||
- cur_offset != bio_next_offset)
+ cur_offset != bio_next_offset ||
+ !fscrypt_mergeable_bio(sdio->bio, dio->inode,
+ cur_offset >> dio->inode->i_blkbits))
dio_bio_submit(dio, sdio);
}
--
2.28.0.rc0.105.gf9edc3c819-goog
^ permalink raw reply related
* [PATCH v2 4/7] ext4: support direct I/O with fscrypt using blk-crypto
From: Satya Tangirala @ 2020-07-17 1:35 UTC (permalink / raw)
To: linux-fscrypt, linux-fsdevel, linux-f2fs-devel, linux-ext4
Cc: linux-xfs, Eric Biggers, Satya Tangirala
In-Reply-To: <20200717013518.59219-1-satyat@google.com>
From: Eric Biggers <ebiggers@google.com>
Wire up ext4 with fscrypt direct I/O support. direct I/O with fscrypt is
only supported through blk-crypto (i.e. CONFIG_BLK_INLINE_ENCRYPTION must
have been enabled, the 'inlinecrypt' mount option must have been specified,
and either hardware inline encryption support must be present or
CONFIG_BLK_INLINE_ENCYRPTION_FALLBACK must have been enabled). Further,
direct I/O on encrypted files is only supported when I/O is aligned
to the filesystem block size (which is *not* necessarily the same as the
block device's block size).
Signed-off-by: Eric Biggers <ebiggers@google.com>
Co-developed-by: Satya Tangirala <satyat@google.com>
Signed-off-by: Satya Tangirala <satyat@google.com>
---
fs/ext4/file.c | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/fs/ext4/file.c b/fs/ext4/file.c
index 2a01e31a032c..d534f72675d9 100644
--- a/fs/ext4/file.c
+++ b/fs/ext4/file.c
@@ -36,9 +36,11 @@
#include "acl.h"
#include "truncate.h"
-static bool ext4_dio_supported(struct inode *inode)
+static bool ext4_dio_supported(struct kiocb *iocb, struct iov_iter *iter)
{
- if (IS_ENABLED(CONFIG_FS_ENCRYPTION) && IS_ENCRYPTED(inode))
+ struct inode *inode = file_inode(iocb->ki_filp);
+
+ if (!fscrypt_dio_supported(iocb, iter))
return false;
if (fsverity_active(inode))
return false;
@@ -61,7 +63,7 @@ static ssize_t ext4_dio_read_iter(struct kiocb *iocb, struct iov_iter *to)
inode_lock_shared(inode);
}
- if (!ext4_dio_supported(inode)) {
+ if (!ext4_dio_supported(iocb, to)) {
inode_unlock_shared(inode);
/*
* Fallback to buffered I/O if the operation being performed on
@@ -490,7 +492,7 @@ static ssize_t ext4_dio_write_iter(struct kiocb *iocb, struct iov_iter *from)
}
/* Fallback to buffered I/O if the inode does not support direct I/O. */
- if (!ext4_dio_supported(inode)) {
+ if (!ext4_dio_supported(iocb, from)) {
if (ilock_shared)
inode_unlock_shared(inode);
else
--
2.28.0.rc0.105.gf9edc3c819-goog
^ permalink raw reply related
* [f2fs-dev] [PATCH v2 1/7] fscrypt: Add functions for direct I/O support
From: Satya Tangirala via Linux-f2fs-devel @ 2020-07-17 1:35 UTC (permalink / raw)
To: linux-fscrypt, linux-fsdevel, linux-f2fs-devel, linux-ext4
Cc: linux-xfs, Satya Tangirala, Eric Biggers
In-Reply-To: <20200717013518.59219-1-satyat@google.com>
From: Eric Biggers <ebiggers@google.com>
Introduce fscrypt_dio_supported() to check whether a direct I/O request
is unsupported due to encryption constraints, and
fscrypt_limit_io_pages() to check how many pages may be added to a bio
being prepared for direct I/O.
The IV_INO_LBLK_32 fscrypt policy introduced the possibility that DUNs
in logically continuous file blocks might wrap from 0xffffffff to 0.
Since this was particularly difficult to handle when block_size !=
PAGE_SIZE, fscrypt only supports blk-crypto en/decryption with
the IV_INO_LBLK_32 policy when block_size == PAGE_SIZE, and ensures that
the DUN never wraps around within any submitted bio.
fscrypt_limit_io_pages() can be used to determine the number of logically
contiguous blocks/pages that may be added to the bio without causing the
DUN to wrap around within the bio. This is an alternative to calling
fscrypt_mergeable_bio() on each page in a range of logically contiguous
pages.
Signed-off-by: Eric Biggers <ebiggers@google.com>
Co-developed-by: Satya Tangirala <satyat@google.com>
Signed-off-by: Satya Tangirala <satyat@google.com>
---
fs/crypto/crypto.c | 8 ++++
fs/crypto/inline_crypt.c | 80 ++++++++++++++++++++++++++++++++++++++++
include/linux/fscrypt.h | 19 ++++++++++
3 files changed, 107 insertions(+)
diff --git a/fs/crypto/crypto.c b/fs/crypto/crypto.c
index a52cf32733ab..fb34364360b3 100644
--- a/fs/crypto/crypto.c
+++ b/fs/crypto/crypto.c
@@ -69,6 +69,14 @@ void fscrypt_free_bounce_page(struct page *bounce_page)
}
EXPORT_SYMBOL(fscrypt_free_bounce_page);
+/*
+ * Generate the IV for the given logical block number within the given file.
+ * For filenames encryption, lblk_num == 0.
+ *
+ * Keep this in sync with fscrypt_limit_io_pages(). fscrypt_limit_io_pages()
+ * needs to know about any IV generation methods where the low bits of IV don't
+ * simply contain the lblk_num (e.g., IV_INO_LBLK_32).
+ */
void fscrypt_generate_iv(union fscrypt_iv *iv, u64 lblk_num,
const struct fscrypt_info *ci)
{
diff --git a/fs/crypto/inline_crypt.c b/fs/crypto/inline_crypt.c
index d7aecadf33c1..f5af6a63e04c 100644
--- a/fs/crypto/inline_crypt.c
+++ b/fs/crypto/inline_crypt.c
@@ -16,6 +16,7 @@
#include <linux/blkdev.h>
#include <linux/buffer_head.h>
#include <linux/sched/mm.h>
+#include <linux/uio.h>
#include "fscrypt_private.h"
@@ -362,3 +363,82 @@ bool fscrypt_mergeable_bio_bh(struct bio *bio,
return fscrypt_mergeable_bio(bio, inode, next_lblk);
}
EXPORT_SYMBOL_GPL(fscrypt_mergeable_bio_bh);
+
+/**
+ * fscrypt_dio_supported() - check whether a direct I/O request is unsupported
+ * due to encryption constraints
+ * @iocb: the file and position the I/O is targeting
+ * @iter: the I/O data segment(s)
+ *
+ * Return: true if direct I/O is supported
+ */
+bool fscrypt_dio_supported(struct kiocb *iocb, struct iov_iter *iter)
+{
+ const struct inode *inode = file_inode(iocb->ki_filp);
+ const unsigned int blocksize = i_blocksize(inode);
+
+ /* If the file is unencrypted, no veto from us. */
+ if (!fscrypt_needs_contents_encryption(inode))
+ return true;
+
+ /* We only support direct I/O with inline crypto, not fs-layer crypto */
+ if (!fscrypt_inode_uses_inline_crypto(inode))
+ return false;
+
+ /*
+ * Since the granularity of encryption is filesystem blocks, the I/O
+ * must be block aligned -- not just disk sector aligned.
+ */
+ if (!IS_ALIGNED(iocb->ki_pos | iov_iter_alignment(iter), blocksize))
+ return false;
+
+ return true;
+}
+EXPORT_SYMBOL_GPL(fscrypt_dio_supported);
+
+/**
+ * fscrypt_limit_io_pages() - limit I/O pages to avoid discontiguous DUNs
+ * @inode: the file on which I/O is being done
+ * @pos: the file position (in bytes) at which the I/O is being done
+ * @nr_pages: the number of pages we want to submit starting at @pos
+ *
+ * Determine the limit to the number of pages that can be submitted in the bio
+ * targeting @pos without causing a data unit number (DUN) discontinuity.
+ *
+ * For IV generation methods that can't cause DUN wraparounds
+ * within logically continuous data blocks, the maximum number of pages is
+ * simply @nr_pages. For those IV generation methods that *might* cause DUN
+ * wraparounds, the returned number of pages is the largest possible number of
+ * pages (less than @nr_pages) that can be added to the bio without causing a
+ * DUN wraparound within the bio.
+ *
+ * Return: the actual number of pages that can be submitted
+ */
+int fscrypt_limit_io_pages(const struct inode *inode, loff_t pos, int nr_pages)
+{
+ const struct fscrypt_info *ci = inode->i_crypt_info;
+ u32 dun;
+
+ if (!fscrypt_inode_uses_inline_crypto(inode))
+ return nr_pages;
+
+ if (nr_pages <= 1)
+ return nr_pages;
+
+ if (!(fscrypt_policy_flags(&ci->ci_policy) &
+ FSCRYPT_POLICY_FLAG_IV_INO_LBLK_32))
+ return nr_pages;
+
+ /*
+ * fscrypt_select_encryption_impl() ensures that block_size == PAGE_SIZE
+ * when using FSCRYPT_POLICY_FLAG_IV_INO_LBLK_32.
+ */
+ if (WARN_ON_ONCE(i_blocksize(inode) != PAGE_SIZE))
+ return 1;
+
+ /* With IV_INO_LBLK_32, the DUN can wrap around from U32_MAX to 0. */
+
+ dun = ci->ci_hashed_ino + (pos >> inode->i_blkbits);
+
+ return min_t(u64, nr_pages, (u64)U32_MAX + 1 - dun);
+}
diff --git a/include/linux/fscrypt.h b/include/linux/fscrypt.h
index bb257411365f..c205c214b35e 100644
--- a/include/linux/fscrypt.h
+++ b/include/linux/fscrypt.h
@@ -559,6 +559,11 @@ bool fscrypt_mergeable_bio(struct bio *bio, const struct inode *inode,
bool fscrypt_mergeable_bio_bh(struct bio *bio,
const struct buffer_head *next_bh);
+bool fscrypt_dio_supported(struct kiocb *iocb, struct iov_iter *iter);
+
+int fscrypt_limit_io_pages(const struct inode *inode, loff_t pos,
+ int nr_pages);
+
#else /* CONFIG_FS_ENCRYPTION_INLINE_CRYPT */
static inline bool __fscrypt_inode_uses_inline_crypto(const struct inode *inode)
@@ -587,6 +592,20 @@ static inline bool fscrypt_mergeable_bio_bh(struct bio *bio,
{
return true;
}
+
+static inline bool fscrypt_dio_supported(struct kiocb *iocb,
+ struct iov_iter *iter)
+{
+ const struct inode *inode = file_inode(iocb->ki_filp);
+
+ return !fscrypt_needs_contents_encryption(inode);
+}
+
+static inline int fscrypt_limit_io_pages(const struct inode *inode, loff_t pos,
+ int nr_pages)
+{
+ return nr_pages;
+}
#endif /* !CONFIG_FS_ENCRYPTION_INLINE_CRYPT */
/**
--
2.28.0.rc0.105.gf9edc3c819-goog
_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
^ permalink raw reply related
* [PATCH v2 3/7] iomap: support direct I/O with fscrypt using blk-crypto
From: Satya Tangirala @ 2020-07-17 1:35 UTC (permalink / raw)
To: linux-fscrypt, linux-fsdevel, linux-f2fs-devel, linux-ext4
Cc: linux-xfs, Eric Biggers, Satya Tangirala
In-Reply-To: <20200717013518.59219-1-satyat@google.com>
From: Eric Biggers <ebiggers@google.com>
Wire up iomap direct I/O with the fscrypt additions for direct I/O,
and set bio crypt contexts on bios when appropriate.
Make iomap_dio_bio_actor() call fscrypt_limit_io_pages() to ensure that
DUNs remain contiguous within a bio, since it works directly with logical
ranges and can't call fscrypt_mergeable_bio() on each page.
Signed-off-by: Eric Biggers <ebiggers@google.com>
Co-developed-by: Satya Tangirala <satyat@google.com>
Signed-off-by: Satya Tangirala <satyat@google.com>
---
fs/iomap/direct-io.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
index ec7b78e6feca..4507dc16dbe5 100644
--- a/fs/iomap/direct-io.c
+++ b/fs/iomap/direct-io.c
@@ -6,6 +6,7 @@
#include <linux/module.h>
#include <linux/compiler.h>
#include <linux/fs.h>
+#include <linux/fscrypt.h>
#include <linux/iomap.h>
#include <linux/backing-dev.h>
#include <linux/uio.h>
@@ -183,11 +184,14 @@ static void
iomap_dio_zero(struct iomap_dio *dio, struct iomap *iomap, loff_t pos,
unsigned len)
{
+ struct inode *inode = file_inode(dio->iocb->ki_filp);
struct page *page = ZERO_PAGE(0);
int flags = REQ_SYNC | REQ_IDLE;
struct bio *bio;
bio = bio_alloc(GFP_KERNEL, 1);
+ fscrypt_set_bio_crypt_ctx(bio, inode, pos >> inode->i_blkbits,
+ GFP_KERNEL);
bio_set_dev(bio, iomap->bdev);
bio->bi_iter.bi_sector = iomap_sector(iomap, pos);
bio->bi_private = dio;
@@ -253,6 +257,7 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length,
ret = nr_pages;
goto out;
}
+ nr_pages = fscrypt_limit_io_pages(inode, pos, nr_pages);
if (need_zeroout) {
/* zero out from the start of the block to the write offset */
@@ -270,6 +275,8 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length,
}
bio = bio_alloc(GFP_KERNEL, nr_pages);
+ fscrypt_set_bio_crypt_ctx(bio, inode, pos >> inode->i_blkbits,
+ GFP_KERNEL);
bio_set_dev(bio, iomap->bdev);
bio->bi_iter.bi_sector = iomap_sector(iomap, pos);
bio->bi_write_hint = dio->iocb->ki_hint;
@@ -307,6 +314,7 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length,
copied += n;
nr_pages = iov_iter_npages(dio->submit.iter, BIO_MAX_PAGES);
+ nr_pages = fscrypt_limit_io_pages(inode, pos, nr_pages);
iomap_dio_submit_bio(dio, iomap, bio, pos);
pos += n;
} while (nr_pages);
--
2.28.0.rc0.105.gf9edc3c819-goog
^ permalink raw reply related
* [PATCH v2 5/7] f2fs: support direct I/O with fscrypt using blk-crypto
From: Satya Tangirala @ 2020-07-17 1:35 UTC (permalink / raw)
To: linux-fscrypt, linux-fsdevel, linux-f2fs-devel, linux-ext4
Cc: linux-xfs, Eric Biggers, Satya Tangirala
In-Reply-To: <20200717013518.59219-1-satyat@google.com>
From: Eric Biggers <ebiggers@google.com>
Wire up f2fs with fscrypt direct I/O support. direct I/O with fscrypt is
only supported through blk-crypto (i.e. CONFIG_BLK_INLINE_ENCRYPTION must
have been enabled, the 'inlinecrypt' mount option must have been specified,
and either hardware inline encryption support must be present or
CONFIG_BLK_INLINE_ENCYRPTION_FALLBACK must have been enabled). Further,
direct I/O on encrypted files is only supported when I/O is aligned
to the filesystem block size (which is *not* necessarily the same as the
block device's block size).
Signed-off-by: Eric Biggers <ebiggers@google.com>
Co-developed-by: Satya Tangirala <satyat@google.com>
Signed-off-by: Satya Tangirala <satyat@google.com>
---
fs/f2fs/f2fs.h | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h
index b35a50f4953c..978130b5a195 100644
--- a/fs/f2fs/f2fs.h
+++ b/fs/f2fs/f2fs.h
@@ -4082,7 +4082,11 @@ static inline bool f2fs_force_buffered_io(struct inode *inode,
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
int rw = iov_iter_rw(iter);
- if (f2fs_post_read_required(inode))
+ if (!fscrypt_dio_supported(iocb, iter))
+ return true;
+ if (fsverity_active(inode))
+ return true;
+ if (f2fs_compressed_file(inode))
return true;
if (f2fs_is_multi_device(sbi))
return true;
--
2.28.0.rc0.105.gf9edc3c819-goog
^ permalink raw reply related
* [PATCH v2 6/7] fscrypt: document inline encryption support
From: Satya Tangirala @ 2020-07-17 1:35 UTC (permalink / raw)
To: linux-fscrypt, linux-fsdevel, linux-f2fs-devel, linux-ext4
Cc: linux-xfs, Satya Tangirala
In-Reply-To: <20200717013518.59219-1-satyat@google.com>
Update the fscrypt documentation file for inline encryption support.
Signed-off-by: Satya Tangirala <satyat@google.com>
---
Documentation/filesystems/fscrypt.rst | 16 +++++++++++++++-
1 file changed, 15 insertions(+), 1 deletion(-)
diff --git a/Documentation/filesystems/fscrypt.rst b/Documentation/filesystems/fscrypt.rst
index f5d8b0303ddf..f3d87a1a0a7f 100644
--- a/Documentation/filesystems/fscrypt.rst
+++ b/Documentation/filesystems/fscrypt.rst
@@ -1204,6 +1204,18 @@ buffer. Some filesystems, such as UBIFS, already use temporary
buffers regardless of encryption. Other filesystems, such as ext4 and
F2FS, have to allocate bounce pages specially for encryption.
+Fscrypt is also able to use inline encryption hardware instead of the
+kernel crypto API for en/decryption of file contents. When possible, and
+if directed to do so (by specifying the 'inlinecrypt' mount option for
+an ext4/F2FS filesystem), it adds encryption contexts to bios and
+uses blk-crypto to perform the en/decryption instead of making use
+of the above read/write path changes. Of course, even if directed to make
+use of inline encryption, fscrypt will only be able to do so if either
+hardware inline encryption support is available for the selected encryption
+algorithm or CONFIG_BLK_INLINE_ENCRYPTION_FALLBACK is selected. If neither
+is the case, fscrypt will fall back to using the above mentioned read/write
+path changes for en/decryption.
+
Filename hashing and encoding
-----------------------------
@@ -1250,7 +1262,9 @@ Tests
To test fscrypt, use xfstests, which is Linux's de facto standard
filesystem test suite. First, run all the tests in the "encrypt"
-group on the relevant filesystem(s). For example, to test ext4 and
+group on the relevant filesystem(s). One can also run the tests
+with the 'inlinecrypt' mount option to test the implementation for
+inline encryption support. For example, to test ext4 and
f2fs encryption using `kvm-xfstests
<https://github.com/tytso/xfstests-bld/blob/master/Documentation/kvm-quickstart.md>`_::
--
2.28.0.rc0.105.gf9edc3c819-goog
^ permalink raw reply related
* [f2fs-dev] [PATCH v2 2/7] direct-io: add support for fscrypt using blk-crypto
From: Satya Tangirala via Linux-f2fs-devel @ 2020-07-17 1:35 UTC (permalink / raw)
To: linux-fscrypt, linux-fsdevel, linux-f2fs-devel, linux-ext4
Cc: linux-xfs, Satya Tangirala, Eric Biggers
In-Reply-To: <20200717013518.59219-1-satyat@google.com>
From: Eric Biggers <ebiggers@google.com>
Set bio crypt contexts on bios by calling into fscrypt when required,
and explicitly check for DUN continuity when adding pages to the bio.
(While DUN continuity is usually implied by logical block contiguity,
this is not the case when using certain fscrypt IV generation methods
like IV_INO_LBLK_32).
Signed-off-by: Eric Biggers <ebiggers@google.com>
Co-developed-by: Satya Tangirala <satyat@google.com>
Signed-off-by: Satya Tangirala <satyat@google.com>
---
fs/direct-io.c | 15 ++++++++++++++-
1 file changed, 14 insertions(+), 1 deletion(-)
diff --git a/fs/direct-io.c b/fs/direct-io.c
index 6d5370eac2a8..f27f7e3780ee 100644
--- a/fs/direct-io.c
+++ b/fs/direct-io.c
@@ -24,6 +24,7 @@
#include <linux/module.h>
#include <linux/types.h>
#include <linux/fs.h>
+#include <linux/fscrypt.h>
#include <linux/mm.h>
#include <linux/slab.h>
#include <linux/highmem.h>
@@ -411,6 +412,7 @@ dio_bio_alloc(struct dio *dio, struct dio_submit *sdio,
sector_t first_sector, int nr_vecs)
{
struct bio *bio;
+ struct inode *inode = dio->inode;
/*
* bio_alloc() is guaranteed to return a bio when allowed to sleep and
@@ -418,6 +420,9 @@ dio_bio_alloc(struct dio *dio, struct dio_submit *sdio,
*/
bio = bio_alloc(GFP_KERNEL, nr_vecs);
+ fscrypt_set_bio_crypt_ctx(bio, inode,
+ sdio->cur_page_fs_offset >> inode->i_blkbits,
+ GFP_KERNEL);
bio_set_dev(bio, bdev);
bio->bi_iter.bi_sector = first_sector;
bio_set_op_attrs(bio, dio->op, dio->op_flags);
@@ -782,9 +787,17 @@ static inline int dio_send_cur_page(struct dio *dio, struct dio_submit *sdio,
* current logical offset in the file does not equal what would
* be the next logical offset in the bio, submit the bio we
* have.
+ *
+ * When fscrypt inline encryption is used, data unit number
+ * (DUN) contiguity is also required. Normally that's implied
+ * by logical contiguity. However, certain IV generation
+ * methods (e.g. IV_INO_LBLK_32) don't guarantee it. So, we
+ * must explicitly check fscrypt_mergeable_bio() too.
*/
if (sdio->final_block_in_bio != sdio->cur_page_block ||
- cur_offset != bio_next_offset)
+ cur_offset != bio_next_offset ||
+ !fscrypt_mergeable_bio(sdio->bio, dio->inode,
+ cur_offset >> dio->inode->i_blkbits))
dio_bio_submit(dio, sdio);
}
--
2.28.0.rc0.105.gf9edc3c819-goog
_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
^ permalink raw reply related
* [PATCH v2 7/7] fscrypt: update documentation for direct I/O support
From: Satya Tangirala @ 2020-07-17 1:35 UTC (permalink / raw)
To: linux-fscrypt, linux-fsdevel, linux-f2fs-devel, linux-ext4
Cc: linux-xfs, Satya Tangirala
In-Reply-To: <20200717013518.59219-1-satyat@google.com>
Update fscrypt documentation to reflect the addition of direct I/O support
and document the necessary conditions for direct I/O on encrypted files.
Signed-off-by: Satya Tangirala <satyat@google.com>
---
Documentation/filesystems/fscrypt.rst | 20 ++++++++++++++++++--
1 file changed, 18 insertions(+), 2 deletions(-)
diff --git a/Documentation/filesystems/fscrypt.rst b/Documentation/filesystems/fscrypt.rst
index f3d87a1a0a7f..95c76a5f0567 100644
--- a/Documentation/filesystems/fscrypt.rst
+++ b/Documentation/filesystems/fscrypt.rst
@@ -1049,8 +1049,10 @@ astute users may notice some differences in behavior:
may be used to overwrite the source files but isn't guaranteed to be
effective on all filesystems and storage devices.
-- Direct I/O is not supported on encrypted files. Attempts to use
- direct I/O on such files will fall back to buffered I/O.
+- Direct I/O is supported on encrypted files only under some circumstances
+ (see `Direct I/O support`_ for details). When these circumstances are not
+ met, attempts to use direct I/O on such files will fall back to buffered
+ I/O.
- The fallocate operations FALLOC_FL_COLLAPSE_RANGE and
FALLOC_FL_INSERT_RANGE are not supported on encrypted files and will
@@ -1257,6 +1259,20 @@ without the key is subject to change in the future. It is only meant
as a way to temporarily present valid filenames so that commands like
``rm -r`` work as expected on encrypted directories.
+Direct I/O support
+------------------
+
+Direct I/O on encrypted files is supported through blk-crypto. In
+particular, this means the kernel must have CONFIG_BLK_INLINE_ENCRYPTION
+enabled, the filesystem must have had the 'inlinecrypt' mount option
+specified, and either hardware inline encryption must be present, or
+CONFIG_BLK_INLINE_ENCRYPTION_FALLBACK must have been enabled. Further,
+any I/O must be aligned to the filesystem block size (*not* necessarily
+the same as the block device's block size) - in particular, any userspace
+buffer into which data is read/written from must also be aligned to the
+filesystem block size. If any of these conditions isn't met, attempts to do
+direct I/O on an encrypted file will fall back to buffered I/O.
+
Tests
=====
--
2.28.0.rc0.105.gf9edc3c819-goog
^ permalink raw reply related
* [PATCH] pinctrl: single: print gpio number in pins debugfs file
From: Drew Fustini @ 2020-07-17 1:33 UTC (permalink / raw)
To: Tony Lindgren, Haojian Zhuang, Linus Walleij, linux-omap,
linux-gpio, Jason Kridner, Robert Nelson
Cc: Drew Fustini
If there is a gpio range mapping for the pin, then print out the gpio
number for the pin in the debugfs 'pins' file.
Here is an example on the BeagleBone Black:
$ cat /sys/kernel/debug/pinctrl/44e10800.pinmux-pinctrl-single/pins |head -20
registered pins: 142
pin 0 (PIN0) 44e10800 00000027 pinctrl-single GPIO-32
pin 1 (PIN1) 44e10804 00000027 pinctrl-single GPIO-33
pin 2 (PIN2) 44e10808 00000027 pinctrl-single GPIO-34
pin 3 (PIN3) 44e1080c 00000027 pinctrl-single GPIO-35
pin 4 (PIN4) 44e10810 00000027 pinctrl-single GPIO-36
pin 5 (PIN5) 44e10814 00000027 pinctrl-single GPIO-37
pin 6 (PIN6) 44e10818 00000027 pinctrl-single GPIO-38
pin 7 (PIN7) 44e1081c 00000027 pinctrl-single GPIO-39
pin 8 (PIN8) 44e10820 00000027 pinctrl-single GPIO-22
pin 9 (PIN9) 44e10824 00000030 pinctrl-single GPIO-23
pin 10 (PIN10) 44e10828 00000030 pinctrl-single GPIO-26
pin 11 (PIN11) 44e1082c 00000030 pinctrl-single GPIO-27
pin 12 (PIN12) 44e10830 00000030 pinctrl-single GPIO-44
pin 13 (PIN13) 44e10834 00000030 pinctrl-single GPIO-45
pin 14 (PIN14) 44e10838 00000030 pinctrl-single GPIO-46
pin 15 (PIN15) 44e1083c 00000030 pinctrl-single GPIO-47
pin 16 (PIN16) 44e10840 00000027 pinctrl-single GPIO-48
pin 17 (PIN17) 44e10844 00000027 pinctrl-single GPIO-49
pin 18 (PIN18) 44e10848 00000000 pinctrl-single GPIO-50
Signed-off-by: Drew Fustini <drew@beagleboard.org>
---
drivers/pinctrl/pinctrl-single.c | 13 ++++++++++++-
1 file changed, 12 insertions(+), 1 deletion(-)
diff --git a/drivers/pinctrl/pinctrl-single.c b/drivers/pinctrl/pinctrl-single.c
index f3a8a465d27e..473fe0f61792 100644
--- a/drivers/pinctrl/pinctrl-single.c
+++ b/drivers/pinctrl/pinctrl-single.c
@@ -275,6 +275,14 @@ static void pcs_pin_dbg_show(struct pinctrl_dev *pctldev,
unsigned val, mux_bytes;
unsigned long offset;
size_t pa;
+ struct pinctrl_gpio_range *range;
+ unsigned gpio_num = 0;
+
+ list_for_each_entry(range, &pctldev->gpio_ranges, node) {
+ if ((pin >= range->pin_base) &&
+ (pin < (range->pin_base + range->npins)))
+ gpio_num = range->base + (pin - range->pin_base);
+ }
pcs = pinctrl_dev_get_drvdata(pctldev);
@@ -283,7 +291,10 @@ static void pcs_pin_dbg_show(struct pinctrl_dev *pctldev,
val = pcs->read(pcs->base + offset);
pa = pcs->res->start + offset;
- seq_printf(s, "%zx %08x %s ", pa, val, DRIVER_NAME);
+ if (gpio_num > 0)
+ seq_printf(s, "%zx %08x %s GPIO-%u", pa, val, DRIVER_NAME, gpio_num);
+ else
+ seq_printf(s, "%zx %08x %s", pa, val, DRIVER_NAME);
}
static void pcs_dt_free_map(struct pinctrl_dev *pctldev,
--
2.25.1
^ permalink raw reply related
* [f2fs-dev] [PATCH v2 5/7] f2fs: support direct I/O with fscrypt using blk-crypto
From: Satya Tangirala via Linux-f2fs-devel @ 2020-07-17 1:35 UTC (permalink / raw)
To: linux-fscrypt, linux-fsdevel, linux-f2fs-devel, linux-ext4
Cc: linux-xfs, Satya Tangirala, Eric Biggers
In-Reply-To: <20200717013518.59219-1-satyat@google.com>
From: Eric Biggers <ebiggers@google.com>
Wire up f2fs with fscrypt direct I/O support. direct I/O with fscrypt is
only supported through blk-crypto (i.e. CONFIG_BLK_INLINE_ENCRYPTION must
have been enabled, the 'inlinecrypt' mount option must have been specified,
and either hardware inline encryption support must be present or
CONFIG_BLK_INLINE_ENCYRPTION_FALLBACK must have been enabled). Further,
direct I/O on encrypted files is only supported when I/O is aligned
to the filesystem block size (which is *not* necessarily the same as the
block device's block size).
Signed-off-by: Eric Biggers <ebiggers@google.com>
Co-developed-by: Satya Tangirala <satyat@google.com>
Signed-off-by: Satya Tangirala <satyat@google.com>
---
fs/f2fs/f2fs.h | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h
index b35a50f4953c..978130b5a195 100644
--- a/fs/f2fs/f2fs.h
+++ b/fs/f2fs/f2fs.h
@@ -4082,7 +4082,11 @@ static inline bool f2fs_force_buffered_io(struct inode *inode,
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
int rw = iov_iter_rw(iter);
- if (f2fs_post_read_required(inode))
+ if (!fscrypt_dio_supported(iocb, iter))
+ return true;
+ if (fsverity_active(inode))
+ return true;
+ if (f2fs_compressed_file(inode))
return true;
if (f2fs_is_multi_device(sbi))
return true;
--
2.28.0.rc0.105.gf9edc3c819-goog
_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
^ permalink raw reply related
* [PATCH 2/2] fdt_support: skip MTD node with "disabled" in fdt_fixup_mtdparts()
From: Masahiro Yamada @ 2020-07-17 1:35 UTC (permalink / raw)
To: u-boot
In-Reply-To: <CAPnjgZ2kPHsUTJYJr6cGdDQ+3uG-SaN4Z_aq0CyuML9MPrOEUQ@mail.gmail.com>
On Fri, Jul 17, 2020 at 12:44 AM Simon Glass <sjg@chromium.org> wrote:
>
> On Wed, 15 Jul 2020 at 04:57, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
> >
> > Currently, fdt_fixup_mtdparts() only checks the compatible property.
> > It is pointless to fix up the disabled node.
> >
> > Skip the node if it has the property:
> >
> > status = "disabled"
> >
> > Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> > ---
> >
> > common/fdt_support.c | 17 ++++++++++-------
> > 1 file changed, 10 insertions(+), 7 deletions(-)
>
> Reviewed-by: Simon Glass <sjg@chromium.org>
>
> Are there any tests for this code?
No test code.
It makes more effort since
testing this would need to probe an MTD device
as well as parsing DT.
> I am thinking we should migrate fdt_support to use livetree...
One important thing we should notice is we have
two different DT instances:
[1] DT blob for U-Boot - used for U-Boot driver model
[2] DT blob for Linux - passed when booting Linux
In my understanding, the livetree
is supposed to unflatten [1].
fdt_fixup_mtdparts() is obviously fixing [2].
I do not know how livetree would work
for this function.
--
Best Regards
Masahiro Yamada
^ permalink raw reply
* Re: [PATCH v2] memcg, oom: check memcg margin for parallel oom
From: Yafang Shao @ 2020-07-17 1:35 UTC (permalink / raw)
To: David Rientjes
Cc: Michal Hocko, Tetsuo Handa, Andrew Morton, Johannes Weiner,
Linux MM
In-Reply-To: <alpine.DEB.2.23.453.2007161246480.3086260@chino.kir.corp.google.com>
On Fri, Jul 17, 2020 at 3:53 AM David Rientjes <rientjes@google.com> wrote:
>
> On Thu, 16 Jul 2020, Yafang Shao wrote:
>
> > > It's likely a misunderstanding: I wasn't necessarily concerned about
> > > showing 60MB in a memcg limited to 100MB, that part we can deal with, the
> > > concern was after dumping all of that great information that instead of
> > > getting a "Killed process..." we get a "Oh, there's memory now, just
> > > kidding about everything we just dumped" ;)
> > >
> >
> > Actually the kernel is doing it now, see bellow,
> >
> > dump_header() <<<< dump lots of information
> > __oom_kill_process
> > p = find_lock_task_mm(victim);
> > if (!p)
> > return; <<<< without killing any process.
> >
>
> Ah, this is catching an instance where the chosen process has already done
> exit_mm(), good catch -- I can find examples of this by scraping kernel
> logs from our fleet.
>
> So it appears there is precedence for dumping all the oom info but not
> actually performing any action for it and I made the earlier point that
> diagnostic information in the kernel log here is still useful. I think it
> is still preferable that the kernel at least tell us why it didn't do
> anything, but as you mention that already happens today.
>
> Would you like to send a patch that checks for mem_cgroup_margin() here as
> well? A second patch could make the possible inaction more visibile,
> something like "Process ${pid} (${comm}) is already exiting" for the above
> check or "Memcg ${memcg} is no longer out of memory".
>
> Another thing that these messages indicate, beyond telling us why the oom
> killer didn't actually SIGKILL anything, is that we can expect some skew
> in the memory stats that shows an availability of memory.
>
Agreed, these messages would be helpful.
I will send a patch for it.
> >
> > > We could likely enlighten userspace about that so that we don't consider
> > > that to be an actual oom kill. But I also very much agree that after
> > > dump_header() would be appropriate as well since the goal is to prevent
> > > unnecessary oom killing.
> > >
> > > Would you mind sending a patch to check mem_cgroup_margin() on
> > > is_memcg_oom() prior to sending the SIGKILL to the victim and printing the
> > > "Killed process..." line? We'd need a line that says "xKB of memory now
> > > available -- suppressing oom kill" or something along those lines so
> > > userspace understands what happened. But the memory info that it emits
> > > both for the state of the memcg and system RAM may also be helpful to
> > > understand why we got to the oom kill in the first place, which is also a
> > > good thing.
> > >
> > > I'd happy ack that patch since it would be a comprehensive solution that
> > > avoids oom kill of user processes at all costs, which is a goal I think we
> > > can all rally behind.
> >
> > I'd prefer to put dump_header() behind do_send_sig_info(), for example,
> >
> > __oom_kill_process()
> > do_send_sig_info()
> > dump_header() <<<< may better put it behind wake_oom_reaper(), but
> > it may loses some information to dump...
> > pr_err("%s: Killed process %d (%s)....")
> >
>
> I agree with Michal here that dump_header() after the actual kill would no
> longer represent the state of the system (or cpuset or memcg, depending on
> context) at the time of the oom kill so it's best to dump relevant
> information before the actual kill.
--
Thanks
Yafang
^ permalink raw reply
* Re: [PATCH] fs/direct-io: avoid data race on ->s_dio_done_wq
From: Darrick J. Wong @ 2020-07-17 1:36 UTC (permalink / raw)
To: Dave Chinner
Cc: Eric Biggers, Matthew Wilcox, linux-fsdevel, linux-xfs,
linux-ext4
In-Reply-To: <20200716081616.GM5369@dread.disaster.area>
On Thu, Jul 16, 2020 at 06:16:16PM +1000, Dave Chinner wrote:
> On Wed, Jul 15, 2020 at 10:33:32PM -0700, Eric Biggers wrote:
> > On Thu, Jul 16, 2020 at 03:47:17AM +0100, Matthew Wilcox wrote:
> > > On Thu, Jul 16, 2020 at 11:46:56AM +1000, Dave Chinner wrote:
> > > > And why should we compromise performance on hundreds of millions of
> > > > modern systems to fix an extremely rare race on an extremely rare
> > > > platform that maybe only a hundred people world-wide might still
> > > > use?
> > >
> > > I thought that wasn't the argument here. It was that some future
> > > compiler might choose to do something absolutely awful that no current
> > > compiler does, and that rather than disable the stupid "optimisation",
> > > we'd be glad that we'd already stuffed the source code up so that it
> > > lay within some tortuous reading of the C spec.
> > >
> > > The memory model is just too complicated. Look at the recent exchange
> > > between myself & Dan Williams. I spent literally _hours_ trying to
> > > figure out what rules to follow.
> > >
> > > https://lore.kernel.org/linux-mm/CAPcyv4jgjoLqsV+aHGJwGXbCSwbTnWLmog5-rxD2i31vZ2rDNQ@mail.gmail.com/
> > > https://lore.kernel.org/linux-mm/CAPcyv4j2+7XiJ9BXQ4mj_XN0N+rCyxch5QkuZ6UsOBsOO1+2Vg@mail.gmail.com/
> > >
> > > Neither Dan nor I are exactly "new" to Linux kernel development. As Dave
> > > is saying here, having to understand the memory model is too high a bar.
> > >
> > > Hell, I don't know if what we ended up with for v4 is actually correct.
> > > It lokos good to me, but *shrug*
> > >
> > > https://lore.kernel.org/linux-mm/159009507306.847224.8502634072429766747.stgit@dwillia2-desk3.amr.corp.intel.com/
> >
> > Looks like you still got it wrong :-( It needs:
> >
> > diff --git a/drivers/char/mem.c b/drivers/char/mem.c
> > index 934c92dcb9ab..9a95fbe86e15 100644
> > --- a/drivers/char/mem.c
> > +++ b/drivers/char/mem.c
> > @@ -1029,7 +1029,7 @@ static int devmem_init_inode(void)
> > }
> >
> > /* publish /dev/mem initialized */
> > - WRITE_ONCE(devmem_inode, inode);
> > + smp_store_release(&devmem_inode, inode);
> >
> > return 0;
> > }
> >
> > It seems one source of confusion is that READ_ONCE() and WRITE_ONCE() don't
> > actually pair with each other, unless no memory barriers are needed at all.
> >
> > Instead, READ_ONCE() pairs with a primitive that has "release" semantics, e.g.
> > smp_store_release() or cmpxchg_release(). But READ_ONCE() is only correct if
> > there's no control flow dependency; if there is, it needs to be upgraded to a
> > primitive with "acquire" semantics, e.g. smp_load_acquire().
>
> You lost your audience at "control flow dependency". i.e. you're
> trying to explain memory ordering requirements by using terms that
> only people who deeply grok memory ordering understand.
>
> I can't tell you what a control flow dependency is off the top
> of my head - I'd have to look up the documentation to remind myself
> what it means and why it might be important. Then I'll realise yet
> again that if I just use release+acquire message passing constructs,
> I just don't have to care about them. And so I promptly forget about
> them again.
>
> My point is that the average programmer does not need to know what a
> control flow or data depedency is to use memory ordering semantics
> correctly. If you want to optimise your code down to the Nth degree
> then you *may* need to know that, but almost nobody in the kernel
> needs to optimise their code to that extent.
>
> > The best approach might be to just say that the READ_ONCE() + "release" pairing
> > should be avoided, and we should stick to "acquire" + "release". (And I think
> > Dave may be saying he'd prefer that for ->s_dio_done_wq?)
>
> Pretty much.
>
> We need to stop thinking of these synchronisation primitives as
> "memory ordering" or "memory barriers" or "atomic access" and
> instead think of them as tools to pass data safely between concurrent
> threads.
>
> We need to give people a simple mental model and/or pattern for
> passing data safely between two racing threads, not hit them over
> the head with the LKMM documentation. People are much more likely to
> understand the ordering and *much* less likely to make mistakes
> given clear, simple examples to follow. And that will stick if you
> can relate those examples back to the locking constructs they
> already understand and have been using for years.
>
> e.g. basic message passing. foo = 0 is the initial message state, y
> is the mail box flag, initially 0/locked:
>
> locking: ordered code:
>
> write message: foo = 1 foo = 1
> post message: spin_unlock(y) smp_store_release(y, 1)
>
> check message box: spin_lock(y) if (smp_load_acquire(y) == 1)
> got message: print(foo) print(foo)
>
> And in both cases we will always get "foo = 1" as the output of
> both sets of code. i.e. foo is the message, y is the object that
> guarantees delivery of the message.
>
> This makes the code Willy linked to obvious. The message to be
> delivered is the inode and it's contents, and it is posted via
> the devmem_inode. Hence:
>
> write message: inode = alloc_anon_inode()
> post message: smp_store_release(&devmem_inode, inode);
>
> check message box: inode = smp_load_acquire(&devmem_inode);
> got message? if (!inode)
> no <fail>
> yes <message contents guaranteed to be seen>
>
> To someone familiar with message passing patterns, this code almost
> doesn't need comments to explain it.
Yes. This!!!
I want a nice one-pager in the recipes file telling me how to do
opportunistic pointer initialization initialization locklessly. I want
the people who specialize in understanding memory models to verify that
yes, this samplecode is correct.
I want to concentrate my attention on the higher level things that I
specialize in, namely XFS, which means that I want to build XFS things
out of high quality lower level pieces that are built by people who are
experts at those lower level things so I don't have to make a huge
detour understanding another highly complex thing. All that will do is
exercise my brain's paging algorithms.
The alternative is that I'll just use a spinlock/rwsem/mutex because
that's what I know, and they're easy to think about. Right now I have
barely any idea what's really the difference between
READ_ONCE/smp_rmb/smp_load_acquire, and I'd rather frame the discussion
as "here's my higher level problem, how do I solve this?"
(end crazy maintainer rambling)
--D
> Using memory ordering code becomes much simpler when we think of it
> as a release+acquire pattern rather than "READ_ONCE/WRITE_ONCE plus
> (some set of) memory barriers" because it explicitly lays out the
> ordering requirements of the code on both sides of the pattern.
> Once a developer creates a mental association between the
> release+acquire message passing mechanism and critical section
> ordering defined by unlock->lock operations, ordering becomes much
> less challenging to think and reason about.
>
> Part of the problem is that RO/WO are now such overloaded operators
> it's hard to understand all the things they do or discriminate
> between which function a specific piece of code is relying on.
>
> IMO, top level kernel developers need to stop telling people "you
> need to understand the lkmm and/or memory_barriers.txt" like it's
> the only way to write safe concurrent code. The reality is that most
> devs don't need to understand it at all. We'll make much more
> progress on fixing broken code and having new code being written
> correctly by teaching people simple patterns that are easy to
> explain, easy to learn, *hard to get wrong* and easy to review. And
> then they'll use them in places where they'd previously not care
> about data races because they have been taught "this is the way we
> should write concurrent code". They'll never learn that from being
> told to read the LKMM documentation....
>
> So if you find yourself using the words "LKMM", "control flow",
> "data dependency", "compiler optimisation" or other intricate
> details or the memory model, then you've already lost your
> audience....
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
^ permalink raw reply
* Re: [SELinux-notebook PATCH v4] objects.md: some clarifications
From: Paul Moore @ 2020-07-17 1:36 UTC (permalink / raw)
To: Dominick Grift; +Cc: selinux
In-Reply-To: <20200716121729.962241-1-dominick.grift@defensec.nl>
On Thu, Jul 16, 2020 at 8:18 AM Dominick Grift
<dominick.grift@defensec.nl> wrote:
>
> Elaborate on labeling. Touch on the significance of the default statement, on various av permissions related to labeling using the libselinux API, and on how the kernel and unlabeled initial security identifiers are used to address labeling challenges in special cases such as initialization and failover respectively.
>
> Signed-off-by: Dominick Grift <dominick.grift@defensec.nl>
> ---
> v2: fixes patch description
> v3: adding patch description, s/policies/policy's/, split unlabeled and kernel descriptions for clarity
> v4: fixes another typo in description and emphasize system initialization a bit
>
> src/objects.md | 29 ++++++++++++++++++++++++++---
> 1 file changed, 26 insertions(+), 3 deletions(-)
>
> diff --git a/src/objects.md b/src/objects.md
> index 58664ef..c67787d 100644
> --- a/src/objects.md
> +++ b/src/objects.md
> @@ -110,14 +110,20 @@ objects is managed by the system and generally unseen by the users
> (until labeling goes wrong !!). As processes and objects are created and
> destroyed, they either:
>
> -1. Inherit their labels from the parent process or object.
> +1. Inherit their labels from the parent process or object. The policy
> + default type, role and range statements can be used to change the
> + behavior as discussed in the [**Default Rules**](default_rules.md#default-object-rules)
> + section.
> 2. The policy type, role and range transition statements allow a
> different label to be assigned as discussed in the
> [**Domain and Object Transitions**](domain_object_transitions.md#domain-and-object-transitions)
> section.
> 3. SELinux-aware applications can enforce a new label (with the
> - policies approval of course) using the **libselinux** API
> - functions.
> + policy's approval of course) using the **libselinux** API
> + functions. The `process setfscreate` access vector can be used to
> + allow subjects to create files with a new label programmatically
> + using the ***setfscreatecon**(3)* function, overriding default
> + rules and transition statements.
> 4. An object manager (OM) can enforce a default label that can either
> be built into the OM or obtained via a configuration file (such as
> those used by
> @@ -269,6 +275,23 @@ and manage their transition:
>
> `type_transition`, `role_transition` and `range_transition`
>
> +SELinux-aware applications can enforce a new label (with the policy's
> +approval of course) using the **libselinux** API functions. The
> +`process setexec`, `process setkeycreate` and `process setsockcreate`
> +access vectors can be used to allow subjects to label processes,
> +kernel keyrings, and sockets programmatically using the
> +***setexec**(3)*, ***setkeycreatecon**(3)* and
> +***setsockcreatecon**(3)* functions respectively, overriding
> +transition statements.
> +
> +The `kernel` **initial security identifier** is used to associate
> +specified labels with subjects that were left unlabeled due to
> +system initialization.
You are iterating faster than I can review the patches ;) Please
don't misunderstand, I'm very happy you're contributing to the
notebook; I was hoping people would get involved and I'm glad to see
that is happening.
That said, while it is true that there are a few objects that are
initially assigned the "kernel" isid, those should gain proper labels
as they complete the initialization process, unless of course they are
actually owned by the kernel. Which brings me to the next point, some
objects are legitimately owned by the kernel, the fact that they are
labeled with the "kernel" isid is intentional. The same holds true
for the "kernel" isid as a subject label, in cases where you see the
"kernel" isid as a subject, it is actually the kernel acting on
something.
> +The `unlabeled` **initial security identifier** is used
> +to associate specified labels with subjects that had their label
> +invalidated due to policy changes at runtime.
> +
> ### Object Reuse
>
> As GNU / Linux runs it creates instances of objects and manages the
> --
> 2.27.0
--
paul moore
www.paul-moore.com
^ permalink raw reply
* Re: [patch] mm, memcg: provide an anon_reclaimable stat
From: Shakeel Butt @ 2020-07-17 1:37 UTC (permalink / raw)
To: David Rientjes
Cc: SeongJae Park, Andrew Morton, Yang Shi, Michal Hocko, Yang Shi,
Roman Gushchin, Greg Thelen, Johannes Weiner, Vladimir Davydov,
Cgroups, Linux MM
In-Reply-To: <alpine.DEB.2.23.453.2007161427360.3213701-X6Q0R45D7oAcqpCFd4KODRPsWskHk0ljAL8bYrjMMd8@public.gmane.org>
On Thu, Jul 16, 2020 at 2:28 PM David Rientjes <rientjes-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org> wrote:
>
> On Thu, 16 Jul 2020, Shakeel Butt wrote:
>
> > > Userspace can lack insight into the amount of memory that can be reclaimed
> > > from a memcg based on values from memory.stat. Two specific examples:
> > >
> > > - Lazy freeable memory (MADV_FREE) that are clean anonymous pages on the
> > > inactive file LRU that can be quickly reclaimed under memory pressure
> > > but otherwise shows up as mapped anon in memory.stat, and
> > >
> > > - Memory on deferred split queues (thp) that are compound pages that can
> > > be split and uncharged from the memcg under memory pressure, but
> > > otherwise shows up as charged anon LRU memory in memory.stat.
> > >
> > > Both of this anonymous usage is also charged to memory.current.
> > >
> > > Userspace can currently derive this information but it depends on kernel
> > > implementation details for how this memory is handled for the purposes of
> > > reclaim (anon on inactive file LRU or unmapped anon on the LRU).
> > >
> > > For the purposes of writing portable userspace code that does not need to
> > > have insight into the kernel implementation for reclaimable memory, this
> > > exports a stat that reveals the amount of anonymous memory that can be
> > > reclaimed and uncharged from the memcg to start new applications.
> > >
> > > As the kernel implementation evolves for memory that can be reclaimed
> > > under memory pressure, this stat can be kept consistent.
> > >
> > > Signed-off-by: David Rientjes <rientjes-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>
> > > ---
> > > Documentation/admin-guide/cgroup-v2.rst | 6 +++++
> > > mm/memcontrol.c | 31 +++++++++++++++++++++++++
> > > 2 files changed, 37 insertions(+)
> > >
> > > diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
> > > --- a/Documentation/admin-guide/cgroup-v2.rst
> > > +++ b/Documentation/admin-guide/cgroup-v2.rst
> > > @@ -1296,6 +1296,12 @@ PAGE_SIZE multiple when read back.
> > > Amount of memory used in anonymous mappings backed by
> > > transparent hugepages
> > >
> > > + anon_reclaimable
> > > + The amount of charged anonymous memory that can be reclaimed
> > > + under memory pressure without swap. This currently includes
> > > + lazy freeable memory (MADV_FREE) and compound pages that can be
> > > + split and uncharged.
> > > +
> > > inactive_anon, active_anon, inactive_file, active_file, unevictable
> > > Amount of memory, swap-backed and filesystem-backed,
> > > on the internal memory management lists used by the
> > > diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> > > --- a/mm/memcontrol.c
> > > +++ b/mm/memcontrol.c
> > > @@ -1350,6 +1350,32 @@ static bool mem_cgroup_wait_acct_move(struct mem_cgroup *memcg)
> > > return false;
> > > }
> > >
> > > +/*
> > > + * Returns the amount of anon memory that is charged to the memcg that is
> > > + * reclaimable under memory pressure without swap, in pages.
> > > + */
> > > +static unsigned long memcg_anon_reclaimable(struct mem_cgroup *memcg)
> > > +{
> > > + long deferred, lazyfree;
> > > +
> > > + /*
> > > + * Deferred pages are charged anonymous pages that are on the LRU but
> > > + * are unmapped. These compound pages are split under memory pressure.
> > > + */
> > > + deferred = max_t(long, memcg_page_state(memcg, NR_ACTIVE_ANON) +
> > > + memcg_page_state(memcg, NR_INACTIVE_ANON) -
> > > + memcg_page_state(memcg, NR_ANON_MAPPED), 0);
> >
> > Please note that the NR_ANON_MAPPED does not include tmpfs memory but
> > NR_[IN]ACTIVE_ANON does include the tmpfs.
> >
> > > + /*
> > > + * Lazyfree pages are charged clean anonymous pages that are on the file
> > > + * LRU and can be reclaimed under memory pressure.
> > > + */
> > > + lazyfree = max_t(long, memcg_page_state(memcg, NR_ACTIVE_FILE) +
> > > + memcg_page_state(memcg, NR_INACTIVE_FILE) -
> > > + memcg_page_state(memcg, NR_FILE_PAGES), 0);
> >
> > Similarly NR_FILE_PAGES includes tmpfs memory but NR_[IN]ACTIVE_FILE does not.
> >
>
> Ah, so this adds to the motivation of providing the anon_reclaimable stat
> because the calculation becomes even more convoluted and completely based
> on the kernel implementation details for both lazyfree memory and deferred
> split queues.
Yes, I agree.
> Did you have a calculation in mind for
> memcg_anon_reclaimable()?
For deferred, "memcg->deferred_split_queue.split_queue_len" should be usable.
For lazyfree, NR_ACTIVE_FILE + NR_INACTIVE_FILE + NR_SHMEM -
NR_FILE_PAGES seems like the right formula.
^ permalink raw reply
* Re: [PATCH -next] rsxx: Convert to DEFINE_SHOW_ATTRIBUTE
From: miaoqinglang @ 2020-07-17 1:37 UTC (permalink / raw)
To: Jens Axboe, Greg Kroah-Hartman, Joshua Morris, Philip Kelleher
Cc: linux-block, linux-kernel
In-Reply-To: <e4caa1de-db2c-c2f4-d1e8-fef7073a52ea@kernel.dk>
在 2020/7/16 23:45, Jens Axboe 写道:
> On 7/16/20 3:04 AM, Qinglang Miao wrote:
>> From: Liu Shixin <liushixin2@huawei.com>
>>
>> Use DEFINE_SHOW_ATTRIBUTE macro to simplify the code.
> None of these apply against the 5.9 block tree, looks like some
> read -> read_iter conversion has happened in another branch that
> I'm not privy to.
Hi Jens,
Sorry I didn't mention it in commit log, but this patch is based
on linux-next where commit <4d4901c6d7> has switched over direct
seq_read method calls to seq_read_iter, this is why there's conflict in
your apply.
Do you think I should send a new patch based on 5.8rc?
Thanks.
^ permalink raw reply
* Re: [patch] mm, memcg: provide an anon_reclaimable stat
From: Shakeel Butt @ 2020-07-17 1:37 UTC (permalink / raw)
To: David Rientjes
Cc: SeongJae Park, Andrew Morton, Yang Shi, Michal Hocko, Yang Shi,
Roman Gushchin, Greg Thelen, Johannes Weiner, Vladimir Davydov,
Cgroups, Linux MM
In-Reply-To: <alpine.DEB.2.23.453.2007161427360.3213701@chino.kir.corp.google.com>
On Thu, Jul 16, 2020 at 2:28 PM David Rientjes <rientjes@google.com> wrote:
>
> On Thu, 16 Jul 2020, Shakeel Butt wrote:
>
> > > Userspace can lack insight into the amount of memory that can be reclaimed
> > > from a memcg based on values from memory.stat. Two specific examples:
> > >
> > > - Lazy freeable memory (MADV_FREE) that are clean anonymous pages on the
> > > inactive file LRU that can be quickly reclaimed under memory pressure
> > > but otherwise shows up as mapped anon in memory.stat, and
> > >
> > > - Memory on deferred split queues (thp) that are compound pages that can
> > > be split and uncharged from the memcg under memory pressure, but
> > > otherwise shows up as charged anon LRU memory in memory.stat.
> > >
> > > Both of this anonymous usage is also charged to memory.current.
> > >
> > > Userspace can currently derive this information but it depends on kernel
> > > implementation details for how this memory is handled for the purposes of
> > > reclaim (anon on inactive file LRU or unmapped anon on the LRU).
> > >
> > > For the purposes of writing portable userspace code that does not need to
> > > have insight into the kernel implementation for reclaimable memory, this
> > > exports a stat that reveals the amount of anonymous memory that can be
> > > reclaimed and uncharged from the memcg to start new applications.
> > >
> > > As the kernel implementation evolves for memory that can be reclaimed
> > > under memory pressure, this stat can be kept consistent.
> > >
> > > Signed-off-by: David Rientjes <rientjes@google.com>
> > > ---
> > > Documentation/admin-guide/cgroup-v2.rst | 6 +++++
> > > mm/memcontrol.c | 31 +++++++++++++++++++++++++
> > > 2 files changed, 37 insertions(+)
> > >
> > > diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
> > > --- a/Documentation/admin-guide/cgroup-v2.rst
> > > +++ b/Documentation/admin-guide/cgroup-v2.rst
> > > @@ -1296,6 +1296,12 @@ PAGE_SIZE multiple when read back.
> > > Amount of memory used in anonymous mappings backed by
> > > transparent hugepages
> > >
> > > + anon_reclaimable
> > > + The amount of charged anonymous memory that can be reclaimed
> > > + under memory pressure without swap. This currently includes
> > > + lazy freeable memory (MADV_FREE) and compound pages that can be
> > > + split and uncharged.
> > > +
> > > inactive_anon, active_anon, inactive_file, active_file, unevictable
> > > Amount of memory, swap-backed and filesystem-backed,
> > > on the internal memory management lists used by the
> > > diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> > > --- a/mm/memcontrol.c
> > > +++ b/mm/memcontrol.c
> > > @@ -1350,6 +1350,32 @@ static bool mem_cgroup_wait_acct_move(struct mem_cgroup *memcg)
> > > return false;
> > > }
> > >
> > > +/*
> > > + * Returns the amount of anon memory that is charged to the memcg that is
> > > + * reclaimable under memory pressure without swap, in pages.
> > > + */
> > > +static unsigned long memcg_anon_reclaimable(struct mem_cgroup *memcg)
> > > +{
> > > + long deferred, lazyfree;
> > > +
> > > + /*
> > > + * Deferred pages are charged anonymous pages that are on the LRU but
> > > + * are unmapped. These compound pages are split under memory pressure.
> > > + */
> > > + deferred = max_t(long, memcg_page_state(memcg, NR_ACTIVE_ANON) +
> > > + memcg_page_state(memcg, NR_INACTIVE_ANON) -
> > > + memcg_page_state(memcg, NR_ANON_MAPPED), 0);
> >
> > Please note that the NR_ANON_MAPPED does not include tmpfs memory but
> > NR_[IN]ACTIVE_ANON does include the tmpfs.
> >
> > > + /*
> > > + * Lazyfree pages are charged clean anonymous pages that are on the file
> > > + * LRU and can be reclaimed under memory pressure.
> > > + */
> > > + lazyfree = max_t(long, memcg_page_state(memcg, NR_ACTIVE_FILE) +
> > > + memcg_page_state(memcg, NR_INACTIVE_FILE) -
> > > + memcg_page_state(memcg, NR_FILE_PAGES), 0);
> >
> > Similarly NR_FILE_PAGES includes tmpfs memory but NR_[IN]ACTIVE_FILE does not.
> >
>
> Ah, so this adds to the motivation of providing the anon_reclaimable stat
> because the calculation becomes even more convoluted and completely based
> on the kernel implementation details for both lazyfree memory and deferred
> split queues.
Yes, I agree.
> Did you have a calculation in mind for
> memcg_anon_reclaimable()?
For deferred, "memcg->deferred_split_queue.split_queue_len" should be usable.
For lazyfree, NR_ACTIVE_FILE + NR_INACTIVE_FILE + NR_SHMEM -
NR_FILE_PAGES seems like the right formula.
^ permalink raw reply
* [tip:master] BUILD SUCCESS 8b68f1cda59dc238dc67989c8ac3fa95873f89ed
From: kernel test robot @ 2020-07-17 1:36 UTC (permalink / raw)
To: x86-ml; +Cc: linux-kernel
tree/branch: https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git master
branch HEAD: 8b68f1cda59dc238dc67989c8ac3fa95873f89ed Merge branch 'irq/urgent'
elapsed time: 722m
configs tested: 74
configs skipped: 1
The following configs have been built successfully.
More configs may be tested in the coming days.
arm64 allyesconfig
arm64 defconfig
arm64 allmodconfig
arm64 allnoconfig
arm defconfig
arm allyesconfig
arm allmodconfig
arm allnoconfig
i386 allyesconfig
i386 defconfig
i386 debian-10.3
i386 allnoconfig
ia64 allmodconfig
ia64 defconfig
ia64 allnoconfig
ia64 allyesconfig
m68k allmodconfig
m68k allnoconfig
m68k sun3_defconfig
m68k defconfig
m68k allyesconfig
nios2 defconfig
nios2 allyesconfig
openrisc defconfig
c6x allyesconfig
c6x allnoconfig
openrisc allyesconfig
nds32 defconfig
nds32 allnoconfig
csky allyesconfig
csky defconfig
alpha defconfig
alpha allyesconfig
xtensa allyesconfig
h8300 allyesconfig
h8300 allmodconfig
xtensa defconfig
arc defconfig
arc allyesconfig
sh allmodconfig
sh allnoconfig
microblaze allnoconfig
mips allyesconfig
mips allnoconfig
mips allmodconfig
parisc allnoconfig
parisc defconfig
parisc allyesconfig
parisc allmodconfig
powerpc defconfig
powerpc allyesconfig
powerpc rhel-kconfig
powerpc allmodconfig
powerpc allnoconfig
riscv allyesconfig
riscv allnoconfig
riscv defconfig
riscv allmodconfig
s390 allyesconfig
s390 allnoconfig
s390 allmodconfig
s390 defconfig
sparc allyesconfig
sparc defconfig
sparc64 defconfig
sparc64 allnoconfig
sparc64 allyesconfig
sparc64 allmodconfig
x86_64 rhel
x86_64 lkp
x86_64 fedora-25
x86_64 rhel-7.6-kselftests
x86_64 rhel-8.3
x86_64 kexec
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org
^ permalink raw reply
* [tip:x86/entry] BUILD SUCCESS 790ce3b40017bbd759a3d81e23c05d42b3d34b90
From: kernel test robot @ 2020-07-17 1:36 UTC (permalink / raw)
To: x86-ml; +Cc: linux-kernel
tree/branch: https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git x86/entry
branch HEAD: 790ce3b40017bbd759a3d81e23c05d42b3d34b90 x86/idtentry: Remove stale comment
elapsed time: 722m
configs tested: 28
configs skipped: 70
The following configs have been built successfully.
More configs may be tested in the coming days.
i386 allyesconfig
i386 defconfig
i386 debian-10.3
i386 allnoconfig
nds32 defconfig
nds32 allnoconfig
csky allyesconfig
csky defconfig
alpha defconfig
alpha allyesconfig
arc defconfig
arc allyesconfig
sh allmodconfig
sh allnoconfig
microblaze allnoconfig
mips allyesconfig
mips allnoconfig
mips allmodconfig
parisc allnoconfig
parisc defconfig
parisc allyesconfig
parisc allmodconfig
x86_64 rhel-7.6-kselftests
x86_64 rhel-8.3
x86_64 kexec
x86_64 rhel
x86_64 lkp
x86_64 fedora-25
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.