* [PATCH v3 01/15] virtio-mem: Paravirtualized memory hotplug
2020-05-07 10:31 [PATCH v3 00/15] virtio-mem: paravirtualized memory David Hildenbrand
@ 2020-05-07 10:31 ` David Hildenbrand
2020-05-07 10:31 ` [PATCH v3 02/15] virtio-mem: Allow to specify an ACPI PXM as nid David Hildenbrand
` (15 subsequent siblings)
16 siblings, 0 replies; 37+ messages in thread
From: David Hildenbrand @ 2020-05-07 10:31 UTC (permalink / raw)
To: linux-kernel
Cc: linux-mm, virtio-dev, virtualization, kvm, Michal Hocko,
Andrew Morton, Michael S . Tsirkin, David Hildenbrand,
Pankaj Gupta, Jason Wang, Oscar Salvador, Igor Mammedov,
Dave Young, Dan Williams, Pavel Tatashin, Stefan Hajnoczi,
Vlastimil Babka, Rafael J. Wysocki, Len Brown, linux-acpi
Each virtio-mem device owns exactly one memory region. It is responsible
for adding/removing memory from that memory region on request.
When the device driver starts up, the requested amount of memory is
queried and then plugged to Linux. On request, further memory can be
plugged or unplugged. This patch only implements the plugging part.
On x86-64, memory can currently be plugged in 4MB ("subblock") granularity.
When required, a new memory block will be added (e.g., usually 128MB on
x86-64) in order to plug more subblocks. Only x86-64 was tested for now.
The online_page callback is used to keep unplugged subblocks offline
when onlining memory - similar to the Hyper-V balloon driver. Unplugged
pages are marked PG_offline, to tell dump tools (e.g., makedumpfile) to
skip them.
User space is usually responsible for onlining the added memory. The
memory hotplug notifier is used to synchronize virtio-mem activity
against memory onlining/offlining.
Each virtio-mem device can belong to a NUMA node, which allows us to
easily add/remove small chunks of memory to/from a specific NUMA node by
using multiple virtio-mem devices. Something that works even when the
guest has no idea about the NUMA topology.
One way to view virtio-mem is as a "resizable DIMM" or a DIMM with many
"sub-DIMMS".
This patch directly introduces the basic infrastructure to implement memory
unplug. Especially the memory block states and subblock bitmaps will be
heavily used there.
Notes:
- In case memory is to be onlined by user space, we limit the amount of
offline memory blocks, to not run out of memory. This is esp. an
issue if memory is added faster than it is getting onlined.
- Suspend/Hibernate is not supported due to the way virtio-mem devices
behave. Limited support might be possible in the future.
- Reloading the device driver is not supported.
Reviewed-by: Pankaj Gupta <pankaj.gupta.linux@gmail.com>
Tested-by: Pankaj Gupta <pankaj.gupta.linux@gmail.com>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Jason Wang <jasowang@redhat.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Igor Mammedov <imammedo@redhat.com>
Cc: Dave Young <dyoung@redhat.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Pavel Tatashin <pasha.tatashin@soleen.com>
Cc: Stefan Hajnoczi <stefanha@redhat.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: "Rafael J. Wysocki" <rjw@rjwysocki.net>
Cc: Len Brown <lenb@kernel.org>
Cc: linux-acpi@vger.kernel.org
Signed-off-by: David Hildenbrand <david@redhat.com>
---
drivers/virtio/Kconfig | 16 +
drivers/virtio/Makefile | 1 +
drivers/virtio/virtio_mem.c | 1533 +++++++++++++++++++++++++++++++
include/uapi/linux/virtio_ids.h | 1 +
include/uapi/linux/virtio_mem.h | 200 ++++
5 files changed, 1751 insertions(+)
create mode 100644 drivers/virtio/virtio_mem.c
create mode 100644 include/uapi/linux/virtio_mem.h
diff --git a/drivers/virtio/Kconfig b/drivers/virtio/Kconfig
index 69a32dfc318a..d6dde7d2cf76 100644
--- a/drivers/virtio/Kconfig
+++ b/drivers/virtio/Kconfig
@@ -78,6 +78,22 @@ config VIRTIO_BALLOON
If unsure, say M.
+config VIRTIO_MEM
+ tristate "Virtio mem driver"
+ default m
+ depends on X86_64
+ depends on VIRTIO
+ depends on MEMORY_HOTPLUG_SPARSE
+ depends on MEMORY_HOTREMOVE
+ help
+ This driver provides access to virtio-mem paravirtualized memory
+ devices, allowing to hotplug and hotunplug memory.
+
+ This driver was only tested under x86-64, but should theoretically
+ work on all architectures that support memory hotplug and hotremove.
+
+ If unsure, say M.
+
config VIRTIO_INPUT
tristate "Virtio input driver"
depends on VIRTIO
diff --git a/drivers/virtio/Makefile b/drivers/virtio/Makefile
index 29a1386ecc03..4d993791f2d7 100644
--- a/drivers/virtio/Makefile
+++ b/drivers/virtio/Makefile
@@ -7,3 +7,4 @@ virtio_pci-$(CONFIG_VIRTIO_PCI_LEGACY) += virtio_pci_legacy.o
obj-$(CONFIG_VIRTIO_BALLOON) += virtio_balloon.o
obj-$(CONFIG_VIRTIO_INPUT) += virtio_input.o
obj-$(CONFIG_VIRTIO_VDPA) += virtio_vdpa.o
+obj-$(CONFIG_VIRTIO_MEM) += virtio_mem.o
diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c
new file mode 100644
index 000000000000..5d1dcaa6fc42
--- /dev/null
+++ b/drivers/virtio/virtio_mem.c
@@ -0,0 +1,1533 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Virtio-mem device driver.
+ *
+ * Copyright Red Hat, Inc. 2020
+ *
+ * Author(s): David Hildenbrand <david@redhat.com>
+ */
+
+#include <linux/virtio.h>
+#include <linux/virtio_mem.h>
+#include <linux/workqueue.h>
+#include <linux/slab.h>
+#include <linux/module.h>
+#include <linux/mm.h>
+#include <linux/memory_hotplug.h>
+#include <linux/memory.h>
+#include <linux/hrtimer.h>
+#include <linux/crash_dump.h>
+#include <linux/mutex.h>
+#include <linux/bitmap.h>
+#include <linux/lockdep.h>
+
+enum virtio_mem_mb_state {
+ /* Unplugged, not added to Linux. Can be reused later. */
+ VIRTIO_MEM_MB_STATE_UNUSED = 0,
+ /* (Partially) plugged, not added to Linux. Error on add_memory(). */
+ VIRTIO_MEM_MB_STATE_PLUGGED,
+ /* Fully plugged, fully added to Linux, offline. */
+ VIRTIO_MEM_MB_STATE_OFFLINE,
+ /* Partially plugged, fully added to Linux, offline. */
+ VIRTIO_MEM_MB_STATE_OFFLINE_PARTIAL,
+ /* Fully plugged, fully added to Linux, online (!ZONE_MOVABLE). */
+ VIRTIO_MEM_MB_STATE_ONLINE,
+ /* Partially plugged, fully added to Linux, online (!ZONE_MOVABLE). */
+ VIRTIO_MEM_MB_STATE_ONLINE_PARTIAL,
+ /*
+ * Fully plugged, fully added to Linux, online (ZONE_MOVABLE).
+ * We are not allowed to allocate (unplug) parts of this block that
+ * are not movable (similar to gigantic pages). We will never allow
+ * to online OFFLINE_PARTIAL to ZONE_MOVABLE (as they would contain
+ * unmovable parts).
+ */
+ VIRTIO_MEM_MB_STATE_ONLINE_MOVABLE,
+ VIRTIO_MEM_MB_STATE_COUNT
+};
+
+struct virtio_mem {
+ struct virtio_device *vdev;
+
+ /* We might first have to unplug all memory when starting up. */
+ bool unplug_all_required;
+
+ /* Workqueue that processes the plug/unplug requests. */
+ struct work_struct wq;
+ atomic_t config_changed;
+
+ /* Virtqueue for guest->host requests. */
+ struct virtqueue *vq;
+
+ /* Wait for a host response to a guest request. */
+ wait_queue_head_t host_resp;
+
+ /* Space for one guest request and the host response. */
+ struct virtio_mem_req req;
+ struct virtio_mem_resp resp;
+
+ /* The current size of the device. */
+ uint64_t plugged_size;
+ /* The requested size of the device. */
+ uint64_t requested_size;
+
+ /* The device block size (for communicating with the device). */
+ uint32_t device_block_size;
+ /* Physical start address of the memory region. */
+ uint64_t addr;
+ /* Maximum region size in bytes. */
+ uint64_t region_size;
+
+ /* The subblock size. */
+ uint32_t subblock_size;
+ /* The number of subblocks per memory block. */
+ uint32_t nb_sb_per_mb;
+
+ /* Id of the first memory block of this device. */
+ unsigned long first_mb_id;
+ /* Id of the last memory block of this device. */
+ unsigned long last_mb_id;
+ /* Id of the last usable memory block of this device. */
+ unsigned long last_usable_mb_id;
+ /* Id of the next memory bock to prepare when needed. */
+ unsigned long next_mb_id;
+
+ /* Summary of all memory block states. */
+ unsigned long nb_mb_state[VIRTIO_MEM_MB_STATE_COUNT];
+#define VIRTIO_MEM_NB_OFFLINE_THRESHOLD 10
+
+ /*
+ * One byte state per memory block.
+ *
+ * Allocated via vmalloc(). When preparing new blocks, resized
+ * (alloc+copy+free) when needed (crossing pages with the next mb).
+ * (when crossing pages).
+ *
+ * With 128MB memory blocks, we have states for 512GB of memory in one
+ * page.
+ */
+ uint8_t *mb_state;
+
+ /*
+ * $nb_sb_per_mb bit per memory block. Handled similar to mb_state.
+ *
+ * With 4MB subblocks, we manage 128GB of memory in one page.
+ */
+ unsigned long *sb_bitmap;
+
+ /*
+ * Mutex that protects the nb_mb_state, mb_state, and sb_bitmap.
+ *
+ * When this lock is held the pointers can't change, ONLINE and
+ * OFFLINE blocks can't change the state and no subblocks will get
+ * plugged.
+ */
+ struct mutex hotplug_mutex;
+ bool hotplug_active;
+
+ /* An error occurred we cannot handle - stop processing requests. */
+ bool broken;
+
+ /* The driver is being removed. */
+ spinlock_t removal_lock;
+ bool removing;
+
+ /* Timer for retrying to plug/unplug memory. */
+ struct hrtimer retry_timer;
+#define VIRTIO_MEM_RETRY_TIMER_MS 30000
+
+ /* Memory notifier (online/offline events). */
+ struct notifier_block memory_notifier;
+
+ /* Next device in the list of virtio-mem devices. */
+ struct list_head next;
+};
+
+/*
+ * We have to share a single online_page callback among all virtio-mem
+ * devices. We use RCU to iterate the list in the callback.
+ */
+static DEFINE_MUTEX(virtio_mem_mutex);
+static LIST_HEAD(virtio_mem_devices);
+
+static void virtio_mem_online_page_cb(struct page *page, unsigned int order);
+
+/*
+ * Register a virtio-mem device so it will be considered for the online_page
+ * callback.
+ */
+static int register_virtio_mem_device(struct virtio_mem *vm)
+{
+ int rc = 0;
+
+ /* First device registers the callback. */
+ mutex_lock(&virtio_mem_mutex);
+ if (list_empty(&virtio_mem_devices))
+ rc = set_online_page_callback(&virtio_mem_online_page_cb);
+ if (!rc)
+ list_add_rcu(&vm->next, &virtio_mem_devices);
+ mutex_unlock(&virtio_mem_mutex);
+
+ return rc;
+}
+
+/*
+ * Unregister a virtio-mem device so it will no longer be considered for the
+ * online_page callback.
+ */
+static void unregister_virtio_mem_device(struct virtio_mem *vm)
+{
+ /* Last device unregisters the callback. */
+ mutex_lock(&virtio_mem_mutex);
+ list_del_rcu(&vm->next);
+ if (list_empty(&virtio_mem_devices))
+ restore_online_page_callback(&virtio_mem_online_page_cb);
+ mutex_unlock(&virtio_mem_mutex);
+
+ synchronize_rcu();
+}
+
+/*
+ * Calculate the memory block id of a given address.
+ */
+static unsigned long virtio_mem_phys_to_mb_id(unsigned long addr)
+{
+ return addr / memory_block_size_bytes();
+}
+
+/*
+ * Calculate the physical start address of a given memory block id.
+ */
+static unsigned long virtio_mem_mb_id_to_phys(unsigned long mb_id)
+{
+ return mb_id * memory_block_size_bytes();
+}
+
+/*
+ * Calculate the subblock id of a given address.
+ */
+static unsigned long virtio_mem_phys_to_sb_id(struct virtio_mem *vm,
+ unsigned long addr)
+{
+ const unsigned long mb_id = virtio_mem_phys_to_mb_id(addr);
+ const unsigned long mb_addr = virtio_mem_mb_id_to_phys(mb_id);
+
+ return (addr - mb_addr) / vm->subblock_size;
+}
+
+/*
+ * Set the state of a memory block, taking care of the state counter.
+ */
+static void virtio_mem_mb_set_state(struct virtio_mem *vm, unsigned long mb_id,
+ enum virtio_mem_mb_state state)
+{
+ const unsigned long idx = mb_id - vm->first_mb_id;
+ enum virtio_mem_mb_state old_state;
+
+ old_state = vm->mb_state[idx];
+ vm->mb_state[idx] = state;
+
+ BUG_ON(vm->nb_mb_state[old_state] == 0);
+ vm->nb_mb_state[old_state]--;
+ vm->nb_mb_state[state]++;
+}
+
+/*
+ * Get the state of a memory block.
+ */
+static enum virtio_mem_mb_state virtio_mem_mb_get_state(struct virtio_mem *vm,
+ unsigned long mb_id)
+{
+ const unsigned long idx = mb_id - vm->first_mb_id;
+
+ return vm->mb_state[idx];
+}
+
+/*
+ * Prepare the state array for the next memory block.
+ */
+static int virtio_mem_mb_state_prepare_next_mb(struct virtio_mem *vm)
+{
+ unsigned long old_bytes = vm->next_mb_id - vm->first_mb_id + 1;
+ unsigned long new_bytes = vm->next_mb_id - vm->first_mb_id + 2;
+ int old_pages = PFN_UP(old_bytes);
+ int new_pages = PFN_UP(new_bytes);
+ uint8_t *new_mb_state;
+
+ if (vm->mb_state && old_pages == new_pages)
+ return 0;
+
+ new_mb_state = vzalloc(new_pages * PAGE_SIZE);
+ if (!new_mb_state)
+ return -ENOMEM;
+
+ mutex_lock(&vm->hotplug_mutex);
+ if (vm->mb_state)
+ memcpy(new_mb_state, vm->mb_state, old_pages * PAGE_SIZE);
+ vfree(vm->mb_state);
+ vm->mb_state = new_mb_state;
+ mutex_unlock(&vm->hotplug_mutex);
+
+ return 0;
+}
+
+#define virtio_mem_for_each_mb_state(_vm, _mb_id, _state) \
+ for (_mb_id = _vm->first_mb_id; \
+ _mb_id < _vm->next_mb_id && _vm->nb_mb_state[_state]; \
+ _mb_id++) \
+ if (virtio_mem_mb_get_state(_vm, _mb_id) == _state)
+
+/*
+ * Mark all selected subblocks plugged.
+ *
+ * Will not modify the state of the memory block.
+ */
+static void virtio_mem_mb_set_sb_plugged(struct virtio_mem *vm,
+ unsigned long mb_id, int sb_id,
+ int count)
+{
+ const int bit = (mb_id - vm->first_mb_id) * vm->nb_sb_per_mb + sb_id;
+
+ __bitmap_set(vm->sb_bitmap, bit, count);
+}
+
+/*
+ * Mark all selected subblocks unplugged.
+ *
+ * Will not modify the state of the memory block.
+ */
+static void virtio_mem_mb_set_sb_unplugged(struct virtio_mem *vm,
+ unsigned long mb_id, int sb_id,
+ int count)
+{
+ const int bit = (mb_id - vm->first_mb_id) * vm->nb_sb_per_mb + sb_id;
+
+ __bitmap_clear(vm->sb_bitmap, bit, count);
+}
+
+/*
+ * Test if all selected subblocks are plugged.
+ */
+static bool virtio_mem_mb_test_sb_plugged(struct virtio_mem *vm,
+ unsigned long mb_id, int sb_id,
+ int count)
+{
+ const int bit = (mb_id - vm->first_mb_id) * vm->nb_sb_per_mb + sb_id;
+
+ if (count == 1)
+ return test_bit(bit, vm->sb_bitmap);
+
+ /* TODO: Helper similar to bitmap_set() */
+ return find_next_zero_bit(vm->sb_bitmap, bit + count, bit) >=
+ bit + count;
+}
+
+/*
+ * Find the first plugged subblock. Returns vm->nb_sb_per_mb in case there is
+ * none.
+ */
+static int virtio_mem_mb_first_plugged_sb(struct virtio_mem *vm,
+ unsigned long mb_id)
+{
+ const int bit = (mb_id - vm->first_mb_id) * vm->nb_sb_per_mb;
+
+ return find_next_bit(vm->sb_bitmap, bit + vm->nb_sb_per_mb, bit) - bit;
+}
+
+/*
+ * Find the first unplugged subblock. Returns vm->nb_sb_per_mb in case there is
+ * none.
+ */
+static int virtio_mem_mb_first_unplugged_sb(struct virtio_mem *vm,
+ unsigned long mb_id)
+{
+ const int bit = (mb_id - vm->first_mb_id) * vm->nb_sb_per_mb;
+
+ return find_next_zero_bit(vm->sb_bitmap, bit + vm->nb_sb_per_mb, bit) -
+ bit;
+}
+
+/*
+ * Prepare the subblock bitmap for the next memory block.
+ */
+static int virtio_mem_sb_bitmap_prepare_next_mb(struct virtio_mem *vm)
+{
+ const unsigned long old_nb_mb = vm->next_mb_id - vm->first_mb_id;
+ const unsigned long old_nb_bits = old_nb_mb * vm->nb_sb_per_mb;
+ const unsigned long new_nb_bits = (old_nb_mb + 1) * vm->nb_sb_per_mb;
+ int old_pages = PFN_UP(BITS_TO_LONGS(old_nb_bits) * sizeof(long));
+ int new_pages = PFN_UP(BITS_TO_LONGS(new_nb_bits) * sizeof(long));
+ unsigned long *new_sb_bitmap, *old_sb_bitmap;
+
+ if (vm->sb_bitmap && old_pages == new_pages)
+ return 0;
+
+ new_sb_bitmap = vzalloc(new_pages * PAGE_SIZE);
+ if (!new_sb_bitmap)
+ return -ENOMEM;
+
+ mutex_lock(&vm->hotplug_mutex);
+ if (new_sb_bitmap)
+ memcpy(new_sb_bitmap, vm->sb_bitmap, old_pages * PAGE_SIZE);
+
+ old_sb_bitmap = vm->sb_bitmap;
+ vm->sb_bitmap = new_sb_bitmap;
+ mutex_unlock(&vm->hotplug_mutex);
+
+ vfree(old_sb_bitmap);
+ return 0;
+}
+
+/*
+ * Try to add a memory block to Linux. This will usually only fail
+ * if out of memory.
+ *
+ * Must not be called with the vm->hotplug_mutex held (possible deadlock with
+ * onlining code).
+ *
+ * Will not modify the state of the memory block.
+ */
+static int virtio_mem_mb_add(struct virtio_mem *vm, unsigned long mb_id)
+{
+ const uint64_t addr = virtio_mem_mb_id_to_phys(mb_id);
+ int nid = memory_add_physaddr_to_nid(addr);
+
+ dev_dbg(&vm->vdev->dev, "adding memory block: %lu\n", mb_id);
+ return add_memory(nid, addr, memory_block_size_bytes());
+}
+
+/*
+ * Try to remove a memory block from Linux. Will only fail if the memory block
+ * is not offline.
+ *
+ * Must not be called with the vm->hotplug_mutex held (possible deadlock with
+ * onlining code).
+ *
+ * Will not modify the state of the memory block.
+ */
+static int virtio_mem_mb_remove(struct virtio_mem *vm, unsigned long mb_id)
+{
+ const uint64_t addr = virtio_mem_mb_id_to_phys(mb_id);
+ int nid = memory_add_physaddr_to_nid(addr);
+
+ dev_dbg(&vm->vdev->dev, "removing memory block: %lu\n", mb_id);
+ return remove_memory(nid, addr, memory_block_size_bytes());
+}
+
+/*
+ * Trigger the workqueue so the device can perform its magic.
+ */
+static void virtio_mem_retry(struct virtio_mem *vm)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(&vm->removal_lock, flags);
+ if (!vm->removing)
+ queue_work(system_freezable_wq, &vm->wq);
+ spin_unlock_irqrestore(&vm->removal_lock, flags);
+}
+
+/*
+ * Test if a virtio-mem device overlaps with the given range. Can be called
+ * from (notifier) callbacks lockless.
+ */
+static bool virtio_mem_overlaps_range(struct virtio_mem *vm,
+ unsigned long start, unsigned long size)
+{
+ unsigned long dev_start = virtio_mem_mb_id_to_phys(vm->first_mb_id);
+ unsigned long dev_end = virtio_mem_mb_id_to_phys(vm->last_mb_id) +
+ memory_block_size_bytes();
+
+ return start < dev_end && dev_start < start + size;
+}
+
+/*
+ * Test if a virtio-mem device owns a memory block. Can be called from
+ * (notifier) callbacks lockless.
+ */
+static bool virtio_mem_owned_mb(struct virtio_mem *vm, unsigned long mb_id)
+{
+ return mb_id >= vm->first_mb_id && mb_id <= vm->last_mb_id;
+}
+
+static int virtio_mem_notify_going_online(struct virtio_mem *vm,
+ unsigned long mb_id,
+ enum zone_type zone)
+{
+ switch (virtio_mem_mb_get_state(vm, mb_id)) {
+ case VIRTIO_MEM_MB_STATE_OFFLINE_PARTIAL:
+ /*
+ * We won't allow to online a partially plugged memory block
+ * to the MOVABLE zone - it would contain unmovable parts.
+ */
+ if (zone == ZONE_MOVABLE) {
+ dev_warn_ratelimited(&vm->vdev->dev,
+ "memory block has holes, MOVABLE not supported\n");
+ return NOTIFY_BAD;
+ }
+ return NOTIFY_OK;
+ case VIRTIO_MEM_MB_STATE_OFFLINE:
+ return NOTIFY_OK;
+ default:
+ break;
+ }
+ dev_warn_ratelimited(&vm->vdev->dev,
+ "memory block onlining denied\n");
+ return NOTIFY_BAD;
+}
+
+static void virtio_mem_notify_offline(struct virtio_mem *vm,
+ unsigned long mb_id)
+{
+ switch (virtio_mem_mb_get_state(vm, mb_id)) {
+ case VIRTIO_MEM_MB_STATE_ONLINE_PARTIAL:
+ virtio_mem_mb_set_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_OFFLINE_PARTIAL);
+ break;
+ case VIRTIO_MEM_MB_STATE_ONLINE:
+ case VIRTIO_MEM_MB_STATE_ONLINE_MOVABLE:
+ virtio_mem_mb_set_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_OFFLINE);
+ break;
+ default:
+ BUG();
+ break;
+ }
+}
+
+static void virtio_mem_notify_online(struct virtio_mem *vm, unsigned long mb_id,
+ enum zone_type zone)
+{
+ unsigned long nb_offline;
+
+ switch (virtio_mem_mb_get_state(vm, mb_id)) {
+ case VIRTIO_MEM_MB_STATE_OFFLINE_PARTIAL:
+ BUG_ON(zone == ZONE_MOVABLE);
+ virtio_mem_mb_set_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_ONLINE_PARTIAL);
+ break;
+ case VIRTIO_MEM_MB_STATE_OFFLINE:
+ if (zone == ZONE_MOVABLE)
+ virtio_mem_mb_set_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_ONLINE_MOVABLE);
+ else
+ virtio_mem_mb_set_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_ONLINE);
+ break;
+ default:
+ BUG();
+ break;
+ }
+ nb_offline = vm->nb_mb_state[VIRTIO_MEM_MB_STATE_OFFLINE] +
+ vm->nb_mb_state[VIRTIO_MEM_MB_STATE_OFFLINE_PARTIAL];
+
+ /* see if we can add new blocks now that we onlined one block */
+ if (nb_offline == VIRTIO_MEM_NB_OFFLINE_THRESHOLD - 1)
+ virtio_mem_retry(vm);
+}
+
+/*
+ * This callback will either be called synchronously from add_memory() or
+ * asynchronously (e.g., triggered via user space). We have to be careful
+ * with locking when calling add_memory().
+ */
+static int virtio_mem_memory_notifier_cb(struct notifier_block *nb,
+ unsigned long action, void *arg)
+{
+ struct virtio_mem *vm = container_of(nb, struct virtio_mem,
+ memory_notifier);
+ struct memory_notify *mhp = arg;
+ const unsigned long start = PFN_PHYS(mhp->start_pfn);
+ const unsigned long size = PFN_PHYS(mhp->nr_pages);
+ const unsigned long mb_id = virtio_mem_phys_to_mb_id(start);
+ enum zone_type zone;
+ int rc = NOTIFY_OK;
+
+ if (!virtio_mem_overlaps_range(vm, start, size))
+ return NOTIFY_DONE;
+
+ /*
+ * Memory is onlined/offlined in memory block granularity. We cannot
+ * cross virtio-mem device boundaries and memory block boundaries. Bail
+ * out if this ever changes.
+ */
+ if (WARN_ON_ONCE(size != memory_block_size_bytes() ||
+ !IS_ALIGNED(start, memory_block_size_bytes())))
+ return NOTIFY_BAD;
+
+ /*
+ * Avoid circular locking lockdep warnings. We lock the mutex
+ * e.g., in MEM_GOING_ONLINE and unlock it in MEM_ONLINE. The
+ * blocking_notifier_call_chain() has it's own lock, which gets unlocked
+ * between both notifier calls and will bail out. False positive.
+ */
+ lockdep_off();
+
+ switch (action) {
+ case MEM_GOING_OFFLINE:
+ mutex_lock(&vm->hotplug_mutex);
+ if (vm->removing) {
+ rc = notifier_from_errno(-EBUSY);
+ mutex_unlock(&vm->hotplug_mutex);
+ break;
+ }
+ vm->hotplug_active = true;
+ break;
+ case MEM_GOING_ONLINE:
+ mutex_lock(&vm->hotplug_mutex);
+ if (vm->removing) {
+ rc = notifier_from_errno(-EBUSY);
+ mutex_unlock(&vm->hotplug_mutex);
+ break;
+ }
+ vm->hotplug_active = true;
+ zone = page_zonenum(pfn_to_page(mhp->start_pfn));
+ rc = virtio_mem_notify_going_online(vm, mb_id, zone);
+ break;
+ case MEM_OFFLINE:
+ virtio_mem_notify_offline(vm, mb_id);
+ vm->hotplug_active = false;
+ mutex_unlock(&vm->hotplug_mutex);
+ break;
+ case MEM_ONLINE:
+ zone = page_zonenum(pfn_to_page(mhp->start_pfn));
+ virtio_mem_notify_online(vm, mb_id, zone);
+ vm->hotplug_active = false;
+ mutex_unlock(&vm->hotplug_mutex);
+ break;
+ case MEM_CANCEL_OFFLINE:
+ case MEM_CANCEL_ONLINE:
+ if (!vm->hotplug_active)
+ break;
+ vm->hotplug_active = false;
+ mutex_unlock(&vm->hotplug_mutex);
+ break;
+ default:
+ break;
+ }
+
+ lockdep_on();
+
+ return rc;
+}
+
+/*
+ * Set a range of pages PG_offline.
+ */
+static void virtio_mem_set_fake_offline(unsigned long pfn,
+ unsigned int nr_pages)
+{
+ for (; nr_pages--; pfn++)
+ __SetPageOffline(pfn_to_page(pfn));
+}
+
+/*
+ * Clear PG_offline from a range of pages.
+ */
+static void virtio_mem_clear_fake_offline(unsigned long pfn,
+ unsigned int nr_pages)
+{
+ for (; nr_pages--; pfn++)
+ __ClearPageOffline(pfn_to_page(pfn));
+}
+
+/*
+ * Release a range of fake-offline pages to the buddy, effectively
+ * fake-onlining them.
+ */
+static void virtio_mem_fake_online(unsigned long pfn, unsigned int nr_pages)
+{
+ const int order = MAX_ORDER - 1;
+ int i;
+
+ /*
+ * We are always called with subblock granularity, which is at least
+ * aligned to MAX_ORDER - 1.
+ */
+ virtio_mem_clear_fake_offline(pfn, nr_pages);
+
+ for (i = 0; i < nr_pages; i += 1 << order)
+ generic_online_page(pfn_to_page(pfn + i), order);
+}
+
+static void virtio_mem_online_page_cb(struct page *page, unsigned int order)
+{
+ const unsigned long addr = page_to_phys(page);
+ const unsigned long mb_id = virtio_mem_phys_to_mb_id(addr);
+ struct virtio_mem *vm;
+ int sb_id;
+
+ /*
+ * We exploit here that subblocks have at least MAX_ORDER - 1
+ * size/alignment and that this callback is is called with such a
+ * size/alignment. So we cannot cross subblocks and therefore
+ * also not memory blocks.
+ */
+ rcu_read_lock();
+ list_for_each_entry_rcu(vm, &virtio_mem_devices, next) {
+ if (!virtio_mem_owned_mb(vm, mb_id))
+ continue;
+
+ sb_id = virtio_mem_phys_to_sb_id(vm, addr);
+ /*
+ * If plugged, online the pages, otherwise, set them fake
+ * offline (PageOffline).
+ */
+ if (virtio_mem_mb_test_sb_plugged(vm, mb_id, sb_id, 1))
+ generic_online_page(page, order);
+ else
+ virtio_mem_set_fake_offline(PFN_DOWN(addr), 1 << order);
+ rcu_read_unlock();
+ return;
+ }
+ rcu_read_unlock();
+
+ /* not virtio-mem memory, but e.g., a DIMM. online it */
+ generic_online_page(page, order);
+}
+
+static uint64_t virtio_mem_send_request(struct virtio_mem *vm,
+ const struct virtio_mem_req *req)
+{
+ struct scatterlist *sgs[2], sg_req, sg_resp;
+ unsigned int len;
+ int rc;
+
+ /* don't use the request residing on the stack (vaddr) */
+ vm->req = *req;
+
+ /* out: buffer for request */
+ sg_init_one(&sg_req, &vm->req, sizeof(vm->req));
+ sgs[0] = &sg_req;
+
+ /* in: buffer for response */
+ sg_init_one(&sg_resp, &vm->resp, sizeof(vm->resp));
+ sgs[1] = &sg_resp;
+
+ rc = virtqueue_add_sgs(vm->vq, sgs, 1, 1, vm, GFP_KERNEL);
+ if (rc < 0)
+ return rc;
+
+ virtqueue_kick(vm->vq);
+
+ /* wait for a response */
+ wait_event(vm->host_resp, virtqueue_get_buf(vm->vq, &len));
+
+ return virtio16_to_cpu(vm->vdev, vm->resp.type);
+}
+
+static int virtio_mem_send_plug_request(struct virtio_mem *vm, uint64_t addr,
+ uint64_t size)
+{
+ const uint64_t nb_vm_blocks = size / vm->device_block_size;
+ const struct virtio_mem_req req = {
+ .type = cpu_to_virtio16(vm->vdev, VIRTIO_MEM_REQ_PLUG),
+ .u.plug.addr = cpu_to_virtio64(vm->vdev, addr),
+ .u.plug.nb_blocks = cpu_to_virtio16(vm->vdev, nb_vm_blocks),
+ };
+
+ if (atomic_read(&vm->config_changed))
+ return -EAGAIN;
+
+ switch (virtio_mem_send_request(vm, &req)) {
+ case VIRTIO_MEM_RESP_ACK:
+ vm->plugged_size += size;
+ return 0;
+ case VIRTIO_MEM_RESP_NACK:
+ return -EAGAIN;
+ case VIRTIO_MEM_RESP_BUSY:
+ return -EBUSY;
+ case VIRTIO_MEM_RESP_ERROR:
+ return -EINVAL;
+ default:
+ return -ENOMEM;
+ }
+}
+
+static int virtio_mem_send_unplug_request(struct virtio_mem *vm, uint64_t addr,
+ uint64_t size)
+{
+ const uint64_t nb_vm_blocks = size / vm->device_block_size;
+ const struct virtio_mem_req req = {
+ .type = cpu_to_virtio16(vm->vdev, VIRTIO_MEM_REQ_UNPLUG),
+ .u.unplug.addr = cpu_to_virtio64(vm->vdev, addr),
+ .u.unplug.nb_blocks = cpu_to_virtio16(vm->vdev, nb_vm_blocks),
+ };
+
+ if (atomic_read(&vm->config_changed))
+ return -EAGAIN;
+
+ switch (virtio_mem_send_request(vm, &req)) {
+ case VIRTIO_MEM_RESP_ACK:
+ vm->plugged_size -= size;
+ return 0;
+ case VIRTIO_MEM_RESP_BUSY:
+ return -EBUSY;
+ case VIRTIO_MEM_RESP_ERROR:
+ return -EINVAL;
+ default:
+ return -ENOMEM;
+ }
+}
+
+static int virtio_mem_send_unplug_all_request(struct virtio_mem *vm)
+{
+ const struct virtio_mem_req req = {
+ .type = cpu_to_virtio16(vm->vdev, VIRTIO_MEM_REQ_UNPLUG_ALL),
+ };
+
+ switch (virtio_mem_send_request(vm, &req)) {
+ case VIRTIO_MEM_RESP_ACK:
+ vm->unplug_all_required = false;
+ vm->plugged_size = 0;
+ /* usable region might have shrunk */
+ atomic_set(&vm->config_changed, 1);
+ return 0;
+ case VIRTIO_MEM_RESP_BUSY:
+ return -EBUSY;
+ default:
+ return -ENOMEM;
+ }
+}
+
+/*
+ * Plug selected subblocks. Updates the plugged state, but not the state
+ * of the memory block.
+ */
+static int virtio_mem_mb_plug_sb(struct virtio_mem *vm, unsigned long mb_id,
+ int sb_id, int count)
+{
+ const uint64_t addr = virtio_mem_mb_id_to_phys(mb_id) +
+ sb_id * vm->subblock_size;
+ const uint64_t size = count * vm->subblock_size;
+ int rc;
+
+ dev_dbg(&vm->vdev->dev, "plugging memory block: %lu : %i - %i\n", mb_id,
+ sb_id, sb_id + count - 1);
+
+ rc = virtio_mem_send_plug_request(vm, addr, size);
+ if (!rc)
+ virtio_mem_mb_set_sb_plugged(vm, mb_id, sb_id, count);
+ return rc;
+}
+
+/*
+ * Unplug selected subblocks. Updates the plugged state, but not the state
+ * of the memory block.
+ */
+static int virtio_mem_mb_unplug_sb(struct virtio_mem *vm, unsigned long mb_id,
+ int sb_id, int count)
+{
+ const uint64_t addr = virtio_mem_mb_id_to_phys(mb_id) +
+ sb_id * vm->subblock_size;
+ const uint64_t size = count * vm->subblock_size;
+ int rc;
+
+ dev_dbg(&vm->vdev->dev, "unplugging memory block: %lu : %i - %i\n",
+ mb_id, sb_id, sb_id + count - 1);
+
+ rc = virtio_mem_send_unplug_request(vm, addr, size);
+ if (!rc)
+ virtio_mem_mb_set_sb_unplugged(vm, mb_id, sb_id, count);
+ return rc;
+}
+
+/*
+ * Unplug the desired number of plugged subblocks of a offline or not-added
+ * memory block. Will fail if any subblock cannot get unplugged (instead of
+ * skipping it).
+ *
+ * Will not modify the state of the memory block.
+ *
+ * Note: can fail after some subblocks were unplugged.
+ */
+static int virtio_mem_mb_unplug_any_sb(struct virtio_mem *vm,
+ unsigned long mb_id, uint64_t *nb_sb)
+{
+ int sb_id, count;
+ int rc;
+
+ while (*nb_sb) {
+ sb_id = virtio_mem_mb_first_plugged_sb(vm, mb_id);
+ if (sb_id >= vm->nb_sb_per_mb)
+ break;
+ count = 1;
+ while (count < *nb_sb &&
+ sb_id + count < vm->nb_sb_per_mb &&
+ virtio_mem_mb_test_sb_plugged(vm, mb_id, sb_id + count,
+ 1))
+ count++;
+
+ rc = virtio_mem_mb_unplug_sb(vm, mb_id, sb_id, count);
+ if (rc)
+ return rc;
+ *nb_sb -= count;
+ }
+
+ return 0;
+}
+
+/*
+ * Unplug all plugged subblocks of an offline or not-added memory block.
+ *
+ * Will not modify the state of the memory block.
+ *
+ * Note: can fail after some subblocks were unplugged.
+ */
+static int virtio_mem_mb_unplug(struct virtio_mem *vm, unsigned long mb_id)
+{
+ uint64_t nb_sb = vm->nb_sb_per_mb;
+
+ return virtio_mem_mb_unplug_any_sb(vm, mb_id, &nb_sb);
+}
+
+/*
+ * Prepare tracking data for the next memory block.
+ */
+static int virtio_mem_prepare_next_mb(struct virtio_mem *vm,
+ unsigned long *mb_id)
+{
+ int rc;
+
+ if (vm->next_mb_id > vm->last_usable_mb_id)
+ return -ENOSPC;
+
+ /* Resize the state array if required. */
+ rc = virtio_mem_mb_state_prepare_next_mb(vm);
+ if (rc)
+ return rc;
+
+ /* Resize the subblock bitmap if required. */
+ rc = virtio_mem_sb_bitmap_prepare_next_mb(vm);
+ if (rc)
+ return rc;
+
+ vm->nb_mb_state[VIRTIO_MEM_MB_STATE_UNUSED]++;
+ *mb_id = vm->next_mb_id++;
+ return 0;
+}
+
+/*
+ * Don't add too many blocks that are not onlined yet to avoid running OOM.
+ */
+static bool virtio_mem_too_many_mb_offline(struct virtio_mem *vm)
+{
+ unsigned long nb_offline;
+
+ nb_offline = vm->nb_mb_state[VIRTIO_MEM_MB_STATE_OFFLINE] +
+ vm->nb_mb_state[VIRTIO_MEM_MB_STATE_OFFLINE_PARTIAL];
+ return nb_offline >= VIRTIO_MEM_NB_OFFLINE_THRESHOLD;
+}
+
+/*
+ * Try to plug the desired number of subblocks and add the memory block
+ * to Linux.
+ *
+ * Will modify the state of the memory block.
+ */
+static int virtio_mem_mb_plug_and_add(struct virtio_mem *vm,
+ unsigned long mb_id,
+ uint64_t *nb_sb)
+{
+ const int count = min_t(int, *nb_sb, vm->nb_sb_per_mb);
+ int rc, rc2;
+
+ if (WARN_ON_ONCE(!count))
+ return -EINVAL;
+
+ /*
+ * Plug the requested number of subblocks before adding it to linux,
+ * so that onlining will directly online all plugged subblocks.
+ */
+ rc = virtio_mem_mb_plug_sb(vm, mb_id, 0, count);
+ if (rc)
+ return rc;
+
+ /*
+ * Mark the block properly offline before adding it to Linux,
+ * so the memory notifiers will find the block in the right state.
+ */
+ if (count == vm->nb_sb_per_mb)
+ virtio_mem_mb_set_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_OFFLINE);
+ else
+ virtio_mem_mb_set_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_OFFLINE_PARTIAL);
+
+ /* Add the memory block to linux - if that fails, try to unplug. */
+ rc = virtio_mem_mb_add(vm, mb_id);
+ if (rc) {
+ enum virtio_mem_mb_state new_state = VIRTIO_MEM_MB_STATE_UNUSED;
+
+ dev_err(&vm->vdev->dev,
+ "adding memory block %lu failed with %d\n", mb_id, rc);
+ rc2 = virtio_mem_mb_unplug_sb(vm, mb_id, 0, count);
+
+ /*
+ * TODO: Linux MM does not properly clean up yet in all cases
+ * where adding of memory failed - especially on -ENOMEM.
+ */
+ if (rc2)
+ new_state = VIRTIO_MEM_MB_STATE_PLUGGED;
+ virtio_mem_mb_set_state(vm, mb_id, new_state);
+ return rc;
+ }
+
+ *nb_sb -= count;
+ return 0;
+}
+
+/*
+ * Try to plug the desired number of subblocks of a memory block that
+ * is already added to Linux.
+ *
+ * Will modify the state of the memory block.
+ *
+ * Note: Can fail after some subblocks were successfully plugged.
+ */
+static int virtio_mem_mb_plug_any_sb(struct virtio_mem *vm, unsigned long mb_id,
+ uint64_t *nb_sb, bool online)
+{
+ unsigned long pfn, nr_pages;
+ int sb_id, count;
+ int rc;
+
+ if (WARN_ON_ONCE(!*nb_sb))
+ return -EINVAL;
+
+ while (*nb_sb) {
+ sb_id = virtio_mem_mb_first_unplugged_sb(vm, mb_id);
+ if (sb_id >= vm->nb_sb_per_mb)
+ break;
+ count = 1;
+ while (count < *nb_sb &&
+ sb_id + count < vm->nb_sb_per_mb &&
+ !virtio_mem_mb_test_sb_plugged(vm, mb_id, sb_id + count,
+ 1))
+ count++;
+
+ rc = virtio_mem_mb_plug_sb(vm, mb_id, sb_id, count);
+ if (rc)
+ return rc;
+ *nb_sb -= count;
+ if (!online)
+ continue;
+
+ /* fake-online the pages if the memory block is online */
+ pfn = PFN_DOWN(virtio_mem_mb_id_to_phys(mb_id) +
+ sb_id * vm->subblock_size);
+ nr_pages = PFN_DOWN(count * vm->subblock_size);
+ virtio_mem_fake_online(pfn, nr_pages);
+ }
+
+ if (virtio_mem_mb_test_sb_plugged(vm, mb_id, 0, vm->nb_sb_per_mb)) {
+ if (online)
+ virtio_mem_mb_set_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_ONLINE);
+ else
+ virtio_mem_mb_set_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_OFFLINE);
+ }
+
+ return rc;
+}
+
+/*
+ * Try to plug the requested amount of memory.
+ */
+static int virtio_mem_plug_request(struct virtio_mem *vm, uint64_t diff)
+{
+ uint64_t nb_sb = diff / vm->subblock_size;
+ unsigned long mb_id;
+ int rc;
+
+ if (!nb_sb)
+ return 0;
+
+ /* Don't race with onlining/offlining */
+ mutex_lock(&vm->hotplug_mutex);
+
+ /* Try to plug subblocks of partially plugged online blocks. */
+ virtio_mem_for_each_mb_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_ONLINE_PARTIAL) {
+ rc = virtio_mem_mb_plug_any_sb(vm, mb_id, &nb_sb, true);
+ if (rc || !nb_sb)
+ goto out_unlock;
+ cond_resched();
+ }
+
+ /* Try to plug subblocks of partially plugged offline blocks. */
+ virtio_mem_for_each_mb_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_OFFLINE_PARTIAL) {
+ rc = virtio_mem_mb_plug_any_sb(vm, mb_id, &nb_sb, false);
+ if (rc || !nb_sb)
+ goto out_unlock;
+ cond_resched();
+ }
+
+ /*
+ * We won't be working on online/offline memory blocks from this point,
+ * so we can't race with memory onlining/offlining. Drop the mutex.
+ */
+ mutex_unlock(&vm->hotplug_mutex);
+
+ /* Try to plug and add unused blocks */
+ virtio_mem_for_each_mb_state(vm, mb_id, VIRTIO_MEM_MB_STATE_UNUSED) {
+ if (virtio_mem_too_many_mb_offline(vm))
+ return -ENOSPC;
+
+ rc = virtio_mem_mb_plug_and_add(vm, mb_id, &nb_sb);
+ if (rc || !nb_sb)
+ return rc;
+ cond_resched();
+ }
+
+ /* Try to prepare, plug and add new blocks */
+ while (nb_sb) {
+ if (virtio_mem_too_many_mb_offline(vm))
+ return -ENOSPC;
+
+ rc = virtio_mem_prepare_next_mb(vm, &mb_id);
+ if (rc)
+ return rc;
+ rc = virtio_mem_mb_plug_and_add(vm, mb_id, &nb_sb);
+ if (rc)
+ return rc;
+ cond_resched();
+ }
+
+ return 0;
+out_unlock:
+ mutex_unlock(&vm->hotplug_mutex);
+ return rc;
+}
+
+/*
+ * Try to unplug all blocks that couldn't be unplugged before, for example,
+ * because the hypervisor was busy.
+ */
+static int virtio_mem_unplug_pending_mb(struct virtio_mem *vm)
+{
+ unsigned long mb_id;
+ int rc;
+
+ virtio_mem_for_each_mb_state(vm, mb_id, VIRTIO_MEM_MB_STATE_PLUGGED) {
+ rc = virtio_mem_mb_unplug(vm, mb_id);
+ if (rc)
+ return rc;
+ virtio_mem_mb_set_state(vm, mb_id, VIRTIO_MEM_MB_STATE_UNUSED);
+ }
+
+ return 0;
+}
+
+/*
+ * Update all parts of the config that could have changed.
+ */
+static void virtio_mem_refresh_config(struct virtio_mem *vm)
+{
+ const uint64_t phys_limit = 1UL << MAX_PHYSMEM_BITS;
+ uint64_t new_plugged_size, usable_region_size, end_addr;
+
+ /* the plugged_size is just a reflection of what _we_ did previously */
+ virtio_cread(vm->vdev, struct virtio_mem_config, plugged_size,
+ &new_plugged_size);
+ if (WARN_ON_ONCE(new_plugged_size != vm->plugged_size))
+ vm->plugged_size = new_plugged_size;
+
+ /* calculate the last usable memory block id */
+ virtio_cread(vm->vdev, struct virtio_mem_config,
+ usable_region_size, &usable_region_size);
+ end_addr = vm->addr + usable_region_size;
+ end_addr = min(end_addr, phys_limit);
+ vm->last_usable_mb_id = virtio_mem_phys_to_mb_id(end_addr) - 1;
+
+ /* see if there is a request to change the size */
+ virtio_cread(vm->vdev, struct virtio_mem_config, requested_size,
+ &vm->requested_size);
+
+ dev_info(&vm->vdev->dev, "plugged size: 0x%llx", vm->plugged_size);
+ dev_info(&vm->vdev->dev, "requested size: 0x%llx", vm->requested_size);
+}
+
+/*
+ * Workqueue function for handling plug/unplug requests and config updates.
+ */
+static void virtio_mem_run_wq(struct work_struct *work)
+{
+ struct virtio_mem *vm = container_of(work, struct virtio_mem, wq);
+ uint64_t diff;
+ int rc;
+
+ hrtimer_cancel(&vm->retry_timer);
+
+ if (vm->broken)
+ return;
+
+retry:
+ rc = 0;
+
+ /* Make sure we start with a clean state if there are leftovers. */
+ if (unlikely(vm->unplug_all_required))
+ rc = virtio_mem_send_unplug_all_request(vm);
+
+ if (atomic_read(&vm->config_changed)) {
+ atomic_set(&vm->config_changed, 0);
+ virtio_mem_refresh_config(vm);
+ }
+
+ /* Unplug any leftovers from previous runs */
+ if (!rc)
+ rc = virtio_mem_unplug_pending_mb(vm);
+
+ if (!rc && vm->requested_size != vm->plugged_size) {
+ if (vm->requested_size > vm->plugged_size) {
+ diff = vm->requested_size - vm->plugged_size;
+ rc = virtio_mem_plug_request(vm, diff);
+ }
+ /* TODO: try to unplug memory */
+ }
+
+ switch (rc) {
+ case 0:
+ break;
+ case -ENOSPC:
+ /*
+ * We cannot add any more memory (alignment, physical limit)
+ * or we have too many offline memory blocks.
+ */
+ break;
+ case -EBUSY:
+ /*
+ * The hypervisor cannot process our request right now
+ * (e.g., out of memory, migrating).
+ */
+ case -ENOMEM:
+ /* Out of memory, try again later. */
+ hrtimer_start(&vm->retry_timer,
+ ms_to_ktime(VIRTIO_MEM_RETRY_TIMER_MS),
+ HRTIMER_MODE_REL);
+ break;
+ case -EAGAIN:
+ /* Retry immediately (e.g., the config changed). */
+ goto retry;
+ default:
+ /* Unknown error, mark as broken */
+ dev_err(&vm->vdev->dev,
+ "unknown error, marking device broken: %d\n", rc);
+ vm->broken = true;
+ }
+}
+
+static enum hrtimer_restart virtio_mem_timer_expired(struct hrtimer *timer)
+{
+ struct virtio_mem *vm = container_of(timer, struct virtio_mem,
+ retry_timer);
+
+ virtio_mem_retry(vm);
+ return HRTIMER_NORESTART;
+}
+
+static void virtio_mem_handle_response(struct virtqueue *vq)
+{
+ struct virtio_mem *vm = vq->vdev->priv;
+
+ wake_up(&vm->host_resp);
+}
+
+static int virtio_mem_init_vq(struct virtio_mem *vm)
+{
+ struct virtqueue *vq;
+
+ vq = virtio_find_single_vq(vm->vdev, virtio_mem_handle_response,
+ "guest-request");
+ if (IS_ERR(vq))
+ return PTR_ERR(vq);
+ vm->vq = vq;
+
+ return 0;
+}
+
+/*
+ * Test if any memory in the range is present in Linux.
+ */
+static bool virtio_mem_any_memory_present(unsigned long start,
+ unsigned long size)
+{
+ const unsigned long start_pfn = PFN_DOWN(start);
+ const unsigned long end_pfn = PFN_UP(start + size);
+ unsigned long pfn;
+
+ for (pfn = start_pfn; pfn != end_pfn; pfn++)
+ if (present_section_nr(pfn_to_section_nr(pfn)))
+ return true;
+
+ return false;
+}
+
+static int virtio_mem_init(struct virtio_mem *vm)
+{
+ const uint64_t phys_limit = 1UL << MAX_PHYSMEM_BITS;
+
+ if (!vm->vdev->config->get) {
+ dev_err(&vm->vdev->dev, "config access disabled\n");
+ return -EINVAL;
+ }
+
+ /*
+ * We don't want to (un)plug or reuse any memory when in kdump. The
+ * memory is still accessible (but not mapped).
+ */
+ if (is_kdump_kernel()) {
+ dev_warn(&vm->vdev->dev, "disabled in kdump kernel\n");
+ return -EBUSY;
+ }
+
+ /* Fetch all properties that can't change. */
+ virtio_cread(vm->vdev, struct virtio_mem_config, plugged_size,
+ &vm->plugged_size);
+ virtio_cread(vm->vdev, struct virtio_mem_config, block_size,
+ &vm->device_block_size);
+ virtio_cread(vm->vdev, struct virtio_mem_config, addr, &vm->addr);
+ virtio_cread(vm->vdev, struct virtio_mem_config, region_size,
+ &vm->region_size);
+
+ /*
+ * If we still have memory plugged, we might have to unplug all
+ * memory first. However, if somebody simply unloaded the driver
+ * we would have to reinitialize the old state - something we don't
+ * support yet. Detect if we have any memory in the area present.
+ */
+ if (vm->plugged_size) {
+ uint64_t usable_region_size;
+
+ virtio_cread(vm->vdev, struct virtio_mem_config,
+ usable_region_size, &usable_region_size);
+
+ if (virtio_mem_any_memory_present(vm->addr,
+ usable_region_size)) {
+ dev_err(&vm->vdev->dev,
+ "reloading the driver is not supported\n");
+ return -EINVAL;
+ }
+ /*
+ * Note: it might happen that the device is busy and
+ * unplugging all memory might take some time.
+ */
+ dev_info(&vm->vdev->dev, "unplugging all memory required\n");
+ vm->unplug_all_required = 1;
+ }
+
+ /*
+ * We always hotplug memory in memory block granularity. This way,
+ * we have to wait for exactly one memory block to online.
+ */
+ if (vm->device_block_size > memory_block_size_bytes()) {
+ dev_err(&vm->vdev->dev,
+ "The block size is not supported (too big).\n");
+ return -EINVAL;
+ }
+
+ /* bad device setup - warn only */
+ if (!IS_ALIGNED(vm->addr, memory_block_size_bytes()))
+ dev_warn(&vm->vdev->dev,
+ "The alignment of the physical start address can make some memory unusable.\n");
+ if (!IS_ALIGNED(vm->addr + vm->region_size, memory_block_size_bytes()))
+ dev_warn(&vm->vdev->dev,
+ "The alignment of the physical end address can make some memory unusable.\n");
+ if (vm->addr + vm->region_size > phys_limit)
+ dev_warn(&vm->vdev->dev,
+ "Some memory is not addressable. This can make some memory unusable.\n");
+
+ /*
+ * Calculate the subblock size:
+ * - At least MAX_ORDER - 1 / pageblock_order.
+ * - At least the device block size.
+ * In the worst case, a single subblock per memory block.
+ */
+ vm->subblock_size = PAGE_SIZE * 1u << max_t(uint32_t, MAX_ORDER - 1,
+ pageblock_order);
+ vm->subblock_size = max_t(uint32_t, vm->device_block_size,
+ vm->subblock_size);
+ vm->nb_sb_per_mb = memory_block_size_bytes() / vm->subblock_size;
+
+ /* Round up to the next full memory block */
+ vm->first_mb_id = virtio_mem_phys_to_mb_id(vm->addr - 1 +
+ memory_block_size_bytes());
+ vm->next_mb_id = vm->first_mb_id;
+ vm->last_mb_id = virtio_mem_phys_to_mb_id(vm->addr +
+ vm->region_size) - 1;
+
+ dev_info(&vm->vdev->dev, "start address: 0x%llx", vm->addr);
+ dev_info(&vm->vdev->dev, "region size: 0x%llx", vm->region_size);
+ dev_info(&vm->vdev->dev, "device block size: 0x%x",
+ vm->device_block_size);
+ dev_info(&vm->vdev->dev, "memory block size: 0x%lx",
+ memory_block_size_bytes());
+ dev_info(&vm->vdev->dev, "subblock size: 0x%x",
+ vm->subblock_size);
+
+ return 0;
+}
+
+static int virtio_mem_probe(struct virtio_device *vdev)
+{
+ struct virtio_mem *vm;
+ int rc = -EINVAL;
+
+ vdev->priv = vm = kzalloc(sizeof(*vm), GFP_KERNEL);
+ if (!vm)
+ return -ENOMEM;
+
+ init_waitqueue_head(&vm->host_resp);
+ vm->vdev = vdev;
+ INIT_WORK(&vm->wq, virtio_mem_run_wq);
+ mutex_init(&vm->hotplug_mutex);
+ INIT_LIST_HEAD(&vm->next);
+ spin_lock_init(&vm->removal_lock);
+ hrtimer_init(&vm->retry_timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+ vm->retry_timer.function = virtio_mem_timer_expired;
+
+ /* register the virtqueue */
+ rc = virtio_mem_init_vq(vm);
+ if (rc)
+ goto out_free_vm;
+
+ /* initialize the device by querying the config */
+ rc = virtio_mem_init(vm);
+ if (rc)
+ goto out_del_vq;
+
+ /* register callbacks */
+ vm->memory_notifier.notifier_call = virtio_mem_memory_notifier_cb;
+ rc = register_memory_notifier(&vm->memory_notifier);
+ if (rc)
+ goto out_del_vq;
+ rc = register_virtio_mem_device(vm);
+ if (rc)
+ goto out_unreg_mem;
+
+ virtio_device_ready(vdev);
+
+ /* trigger a config update to start processing the requested_size */
+ atomic_set(&vm->config_changed, 1);
+ queue_work(system_freezable_wq, &vm->wq);
+
+ return 0;
+out_unreg_mem:
+ unregister_memory_notifier(&vm->memory_notifier);
+out_del_vq:
+ vdev->config->del_vqs(vdev);
+out_free_vm:
+ kfree(vm);
+ vdev->priv = NULL;
+
+ return rc;
+}
+
+static void virtio_mem_remove(struct virtio_device *vdev)
+{
+ struct virtio_mem *vm = vdev->priv;
+ unsigned long mb_id;
+ int rc;
+
+ /*
+ * Make sure the workqueue won't be triggered anymore and no memory
+ * blocks can be onlined/offlined until we're finished here.
+ */
+ mutex_lock(&vm->hotplug_mutex);
+ spin_lock_irq(&vm->removal_lock);
+ vm->removing = true;
+ spin_unlock_irq(&vm->removal_lock);
+ mutex_unlock(&vm->hotplug_mutex);
+
+ /* wait until the workqueue stopped */
+ cancel_work_sync(&vm->wq);
+ hrtimer_cancel(&vm->retry_timer);
+
+ /*
+ * After we unregistered our callbacks, user space can online partially
+ * plugged offline blocks. Make sure to remove them.
+ */
+ virtio_mem_for_each_mb_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_OFFLINE_PARTIAL) {
+ rc = virtio_mem_mb_remove(vm, mb_id);
+ BUG_ON(rc);
+ virtio_mem_mb_set_state(vm, mb_id, VIRTIO_MEM_MB_STATE_UNUSED);
+ }
+
+ /* unregister callbacks */
+ unregister_virtio_mem_device(vm);
+ unregister_memory_notifier(&vm->memory_notifier);
+
+ /*
+ * There is no way we could reliably remove all memory we have added to
+ * the system. And there is no way to stop the driver/device from going
+ * away. Warn at least.
+ */
+ if (vm->nb_mb_state[VIRTIO_MEM_MB_STATE_OFFLINE] ||
+ vm->nb_mb_state[VIRTIO_MEM_MB_STATE_OFFLINE_PARTIAL] ||
+ vm->nb_mb_state[VIRTIO_MEM_MB_STATE_ONLINE] ||
+ vm->nb_mb_state[VIRTIO_MEM_MB_STATE_ONLINE_PARTIAL] ||
+ vm->nb_mb_state[VIRTIO_MEM_MB_STATE_ONLINE_MOVABLE])
+ dev_warn(&vdev->dev, "device still has system memory added\n");
+
+ /* remove all tracking data - no locking needed */
+ vfree(vm->mb_state);
+ vfree(vm->sb_bitmap);
+
+ /* reset the device and cleanup the queues */
+ vdev->config->reset(vdev);
+ vdev->config->del_vqs(vdev);
+
+ kfree(vm);
+ vdev->priv = NULL;
+}
+
+static void virtio_mem_config_changed(struct virtio_device *vdev)
+{
+ struct virtio_mem *vm = vdev->priv;
+
+ atomic_set(&vm->config_changed, 1);
+ virtio_mem_retry(vm);
+}
+
+#ifdef CONFIG_PM_SLEEP
+static int virtio_mem_freeze(struct virtio_device *vdev)
+{
+ /*
+ * When restarting the VM, all memory is usually unplugged. Don't
+ * allow to suspend/hibernate.
+ */
+ dev_err(&vdev->dev, "save/restore not supported.\n");
+ return -EPERM;
+}
+
+static int virtio_mem_restore(struct virtio_device *vdev)
+{
+ return -EPERM;
+}
+#endif
+
+static struct virtio_device_id virtio_mem_id_table[] = {
+ { VIRTIO_ID_MEM, VIRTIO_DEV_ANY_ID },
+ { 0 },
+};
+
+static struct virtio_driver virtio_mem_driver = {
+ .driver.name = KBUILD_MODNAME,
+ .driver.owner = THIS_MODULE,
+ .id_table = virtio_mem_id_table,
+ .probe = virtio_mem_probe,
+ .remove = virtio_mem_remove,
+ .config_changed = virtio_mem_config_changed,
+#ifdef CONFIG_PM_SLEEP
+ .freeze = virtio_mem_freeze,
+ .restore = virtio_mem_restore,
+#endif
+};
+
+module_virtio_driver(virtio_mem_driver);
+MODULE_DEVICE_TABLE(virtio, virtio_mem_id_table);
+MODULE_AUTHOR("David Hildenbrand <david@redhat.com>");
+MODULE_DESCRIPTION("Virtio-mem driver");
+MODULE_LICENSE("GPL");
diff --git a/include/uapi/linux/virtio_ids.h b/include/uapi/linux/virtio_ids.h
index ecc27a17401a..b052355ac7a3 100644
--- a/include/uapi/linux/virtio_ids.h
+++ b/include/uapi/linux/virtio_ids.h
@@ -44,6 +44,7 @@
#define VIRTIO_ID_VSOCK 19 /* virtio vsock transport */
#define VIRTIO_ID_CRYPTO 20 /* virtio crypto */
#define VIRTIO_ID_IOMMU 23 /* virtio IOMMU */
+#define VIRTIO_ID_MEM 24 /* virtio mem */
#define VIRTIO_ID_FS 26 /* virtio filesystem */
#define VIRTIO_ID_PMEM 27 /* virtio pmem */
#define VIRTIO_ID_MAC80211_HWSIM 29 /* virtio mac80211-hwsim */
diff --git a/include/uapi/linux/virtio_mem.h b/include/uapi/linux/virtio_mem.h
new file mode 100644
index 000000000000..1bfade78bdfd
--- /dev/null
+++ b/include/uapi/linux/virtio_mem.h
@@ -0,0 +1,200 @@
+/* SPDX-License-Identifier: BSD-3-Clause */
+/*
+ * Virtio Mem Device
+ *
+ * Copyright Red Hat, Inc. 2020
+ *
+ * Authors:
+ * David Hildenbrand <david@redhat.com>
+ *
+ * This header is BSD licensed so anyone can use the definitions
+ * to implement compatible drivers/servers:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ * 3. Neither the name of IBM nor the names of its contributors
+ * may be used to endorse or promote products derived from this software
+ * without specific prior written permission.
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL IBM OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#ifndef _LINUX_VIRTIO_MEM_H
+#define _LINUX_VIRTIO_MEM_H
+
+#include <linux/types.h>
+#include <linux/virtio_types.h>
+#include <linux/virtio_ids.h>
+#include <linux/virtio_config.h>
+
+/*
+ * Each virtio-mem device manages a dedicated region in physical address
+ * space. Each device can belong to a single NUMA node, multiple devices
+ * for a single NUMA node are possible. A virtio-mem device is like a
+ * "resizable DIMM" consisting of small memory blocks that can be plugged
+ * or unplugged. The device driver is responsible for (un)plugging memory
+ * blocks on demand.
+ *
+ * Virtio-mem devices can only operate on their assigned memory region in
+ * order to (un)plug memory. A device cannot (un)plug memory belonging to
+ * other devices.
+ *
+ * The "region_size" corresponds to the maximum amount of memory that can
+ * be provided by a device. The "size" corresponds to the amount of memory
+ * that is currently plugged. "requested_size" corresponds to a request
+ * from the device to the device driver to (un)plug blocks. The
+ * device driver should try to (un)plug blocks in order to reach the
+ * "requested_size". It is impossible to plug more memory than requested.
+ *
+ * The "usable_region_size" represents the memory region that can actually
+ * be used to (un)plug memory. It is always at least as big as the
+ * "requested_size" and will grow dynamically. It will only shrink when
+ * explicitly triggered (VIRTIO_MEM_REQ_UNPLUG).
+ *
+ * There are no guarantees what will happen if unplugged memory is
+ * read/written. Such memory should, in general, not be touched. E.g.,
+ * even writing might succeed, but the values will simply be discarded at
+ * random points in time.
+ *
+ * It can happen that the device cannot process a request, because it is
+ * busy. The device driver has to retry later.
+ *
+ * Usually, during system resets all memory will get unplugged, so the
+ * device driver can start with a clean state. However, in specific
+ * scenarios (if the device is busy) it can happen that the device still
+ * has memory plugged. The device driver can request to unplug all memory
+ * (VIRTIO_MEM_REQ_UNPLUG) - which might take a while to succeed if the
+ * device is busy.
+ */
+
+/* --- virtio-mem: guest -> host requests --- */
+
+/* request to plug memory blocks */
+#define VIRTIO_MEM_REQ_PLUG 0
+/* request to unplug memory blocks */
+#define VIRTIO_MEM_REQ_UNPLUG 1
+/* request to unplug all blocks and shrink the usable size */
+#define VIRTIO_MEM_REQ_UNPLUG_ALL 2
+/* request information about the plugged state of memory blocks */
+#define VIRTIO_MEM_REQ_STATE 3
+
+struct virtio_mem_req_plug {
+ __virtio64 addr;
+ __virtio16 nb_blocks;
+};
+
+struct virtio_mem_req_unplug {
+ __virtio64 addr;
+ __virtio16 nb_blocks;
+};
+
+struct virtio_mem_req_state {
+ __virtio64 addr;
+ __virtio16 nb_blocks;
+};
+
+struct virtio_mem_req {
+ __virtio16 type;
+ __virtio16 padding[3];
+
+ union {
+ struct virtio_mem_req_plug plug;
+ struct virtio_mem_req_unplug unplug;
+ struct virtio_mem_req_state state;
+ } u;
+};
+
+
+/* --- virtio-mem: host -> guest response --- */
+
+/*
+ * Request processed successfully, applicable for
+ * - VIRTIO_MEM_REQ_PLUG
+ * - VIRTIO_MEM_REQ_UNPLUG
+ * - VIRTIO_MEM_REQ_UNPLUG_ALL
+ * - VIRTIO_MEM_REQ_STATE
+ */
+#define VIRTIO_MEM_RESP_ACK 0
+/*
+ * Request denied - e.g. trying to plug more than requested, applicable for
+ * - VIRTIO_MEM_REQ_PLUG
+ */
+#define VIRTIO_MEM_RESP_NACK 1
+/*
+ * Request cannot be processed right now, try again later, applicable for
+ * - VIRTIO_MEM_REQ_PLUG
+ * - VIRTIO_MEM_REQ_UNPLUG
+ * - VIRTIO_MEM_REQ_UNPLUG_ALL
+ */
+#define VIRTIO_MEM_RESP_BUSY 2
+/*
+ * Error in request (e.g. addresses/alignment), applicable for
+ * - VIRTIO_MEM_REQ_PLUG
+ * - VIRTIO_MEM_REQ_UNPLUG
+ * - VIRTIO_MEM_REQ_STATE
+ */
+#define VIRTIO_MEM_RESP_ERROR 3
+
+
+/* State of memory blocks is "plugged" */
+#define VIRTIO_MEM_STATE_PLUGGED 0
+/* State of memory blocks is "unplugged" */
+#define VIRTIO_MEM_STATE_UNPLUGGED 1
+/* State of memory blocks is "mixed" */
+#define VIRTIO_MEM_STATE_MIXED 2
+
+struct virtio_mem_resp_state {
+ __virtio16 state;
+};
+
+struct virtio_mem_resp {
+ __virtio16 type;
+ __virtio16 padding[3];
+
+ union {
+ struct virtio_mem_resp_state state;
+ } u;
+};
+
+/* --- virtio-mem: configuration --- */
+
+struct virtio_mem_config {
+ /* Block size and alignment. Cannot change. */
+ __u32 block_size;
+ __u32 padding;
+ /* Start address of the memory region. Cannot change. */
+ __u64 addr;
+ /* Region size (maximum). Cannot change. */
+ __u64 region_size;
+ /*
+ * Currently usable region size. Can grow up to region_size. Can
+ * shrink due to VIRTIO_MEM_REQ_UNPLUG_ALL (in which case no config
+ * update will be sent).
+ */
+ __u64 usable_region_size;
+ /*
+ * Currently used size. Changes due to plug/unplug requests, but no
+ * config updates will be sent.
+ */
+ __u64 plugged_size;
+ /* Requested size. New plug requests cannot exceed it. Can change. */
+ __u64 requested_size;
+};
+
+#endif /* _LINUX_VIRTIO_MEM_H */
--
2.25.3
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v3 02/15] virtio-mem: Allow to specify an ACPI PXM as nid
2020-05-07 10:31 [PATCH v3 00/15] virtio-mem: paravirtualized memory David Hildenbrand
2020-05-07 10:31 ` [PATCH v3 01/15] virtio-mem: Paravirtualized memory hotplug David Hildenbrand
@ 2020-05-07 10:31 ` David Hildenbrand
2020-05-07 10:31 ` [PATCH v3 03/15] virtio-mem: Paravirtualized memory hotunplug part 1 David Hildenbrand
` (14 subsequent siblings)
16 siblings, 0 replies; 37+ messages in thread
From: David Hildenbrand @ 2020-05-07 10:31 UTC (permalink / raw)
To: linux-kernel
Cc: linux-mm, virtio-dev, virtualization, kvm, Michal Hocko,
Andrew Morton, Michael S . Tsirkin, David Hildenbrand,
Michal Hocko, Rafael J. Wysocki, Pankaj Gupta, Jason Wang,
Oscar Salvador, Igor Mammedov, Dave Young, Dan Williams,
Pavel Tatashin, Stefan Hajnoczi, Vlastimil Babka, Len Brown,
linux-acpi
We want to allow to specify (similar as for a DIMM), to which node a
virtio-mem device (and, therefore, its memory) belongs. Add a new
virtio-mem feature flag and export pxm_to_node, so it can be used in kernel
module context.
Acked-by: Michal Hocko <mhocko@suse.com> # for the export
Acked-by: "Rafael J. Wysocki" <rafael@kernel.org> # for the export
Acked-by: Pankaj Gupta <pankaj.gupta.linux@gmail.com>
Tested-by: Pankaj Gupta <pankaj.gupta.linux@gmail.com>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Jason Wang <jasowang@redhat.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Igor Mammedov <imammedo@redhat.com>
Cc: Dave Young <dyoung@redhat.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Pavel Tatashin <pasha.tatashin@soleen.com>
Cc: Stefan Hajnoczi <stefanha@redhat.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Len Brown <lenb@kernel.org>
Cc: linux-acpi@vger.kernel.org
Signed-off-by: David Hildenbrand <david@redhat.com>
---
drivers/acpi/numa/srat.c | 1 +
drivers/virtio/virtio_mem.c | 39 +++++++++++++++++++++++++++++++--
include/uapi/linux/virtio_mem.h | 10 ++++++++-
3 files changed, 47 insertions(+), 3 deletions(-)
diff --git a/drivers/acpi/numa/srat.c b/drivers/acpi/numa/srat.c
index 47b4969d9b93..5be5a977da1b 100644
--- a/drivers/acpi/numa/srat.c
+++ b/drivers/acpi/numa/srat.c
@@ -35,6 +35,7 @@ int pxm_to_node(int pxm)
return NUMA_NO_NODE;
return pxm_to_node_map[pxm];
}
+EXPORT_SYMBOL(pxm_to_node);
int node_to_pxm(int node)
{
diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c
index 5d1dcaa6fc42..270ddeaec059 100644
--- a/drivers/virtio/virtio_mem.c
+++ b/drivers/virtio/virtio_mem.c
@@ -21,6 +21,8 @@
#include <linux/bitmap.h>
#include <linux/lockdep.h>
+#include <acpi/acpi_numa.h>
+
enum virtio_mem_mb_state {
/* Unplugged, not added to Linux. Can be reused later. */
VIRTIO_MEM_MB_STATE_UNUSED = 0,
@@ -72,6 +74,8 @@ struct virtio_mem {
/* The device block size (for communicating with the device). */
uint32_t device_block_size;
+ /* The translated node id. NUMA_NO_NODE in case not specified. */
+ int nid;
/* Physical start address of the memory region. */
uint64_t addr;
/* Maximum region size in bytes. */
@@ -389,7 +393,10 @@ static int virtio_mem_sb_bitmap_prepare_next_mb(struct virtio_mem *vm)
static int virtio_mem_mb_add(struct virtio_mem *vm, unsigned long mb_id)
{
const uint64_t addr = virtio_mem_mb_id_to_phys(mb_id);
- int nid = memory_add_physaddr_to_nid(addr);
+ int nid = vm->nid;
+
+ if (nid == NUMA_NO_NODE)
+ nid = memory_add_physaddr_to_nid(addr);
dev_dbg(&vm->vdev->dev, "adding memory block: %lu\n", mb_id);
return add_memory(nid, addr, memory_block_size_bytes());
@@ -407,7 +414,10 @@ static int virtio_mem_mb_add(struct virtio_mem *vm, unsigned long mb_id)
static int virtio_mem_mb_remove(struct virtio_mem *vm, unsigned long mb_id)
{
const uint64_t addr = virtio_mem_mb_id_to_phys(mb_id);
- int nid = memory_add_physaddr_to_nid(addr);
+ int nid = vm->nid;
+
+ if (nid == NUMA_NO_NODE)
+ nid = memory_add_physaddr_to_nid(addr);
dev_dbg(&vm->vdev->dev, "removing memory block: %lu\n", mb_id);
return remove_memory(nid, addr, memory_block_size_bytes());
@@ -426,6 +436,17 @@ static void virtio_mem_retry(struct virtio_mem *vm)
spin_unlock_irqrestore(&vm->removal_lock, flags);
}
+static int virtio_mem_translate_node_id(struct virtio_mem *vm, uint16_t node_id)
+{
+ int node = NUMA_NO_NODE;
+
+#if defined(CONFIG_ACPI_NUMA)
+ if (virtio_has_feature(vm->vdev, VIRTIO_MEM_F_ACPI_PXM))
+ node = pxm_to_node(node_id);
+#endif
+ return node;
+}
+
/*
* Test if a virtio-mem device overlaps with the given range. Can be called
* from (notifier) callbacks lockless.
@@ -1267,6 +1288,7 @@ static bool virtio_mem_any_memory_present(unsigned long start,
static int virtio_mem_init(struct virtio_mem *vm)
{
const uint64_t phys_limit = 1UL << MAX_PHYSMEM_BITS;
+ uint16_t node_id;
if (!vm->vdev->config->get) {
dev_err(&vm->vdev->dev, "config access disabled\n");
@@ -1287,6 +1309,9 @@ static int virtio_mem_init(struct virtio_mem *vm)
&vm->plugged_size);
virtio_cread(vm->vdev, struct virtio_mem_config, block_size,
&vm->device_block_size);
+ virtio_cread(vm->vdev, struct virtio_mem_config, node_id,
+ &node_id);
+ vm->nid = virtio_mem_translate_node_id(vm, node_id);
virtio_cread(vm->vdev, struct virtio_mem_config, addr, &vm->addr);
virtio_cread(vm->vdev, struct virtio_mem_config, region_size,
&vm->region_size);
@@ -1365,6 +1390,8 @@ static int virtio_mem_init(struct virtio_mem *vm)
memory_block_size_bytes());
dev_info(&vm->vdev->dev, "subblock size: 0x%x",
vm->subblock_size);
+ if (vm->nid != NUMA_NO_NODE)
+ dev_info(&vm->vdev->dev, "nid: %d", vm->nid);
return 0;
}
@@ -1508,12 +1535,20 @@ static int virtio_mem_restore(struct virtio_device *vdev)
}
#endif
+static unsigned int virtio_mem_features[] = {
+#if defined(CONFIG_NUMA) && defined(CONFIG_ACPI_NUMA)
+ VIRTIO_MEM_F_ACPI_PXM,
+#endif
+};
+
static struct virtio_device_id virtio_mem_id_table[] = {
{ VIRTIO_ID_MEM, VIRTIO_DEV_ANY_ID },
{ 0 },
};
static struct virtio_driver virtio_mem_driver = {
+ .feature_table = virtio_mem_features,
+ .feature_table_size = ARRAY_SIZE(virtio_mem_features),
.driver.name = KBUILD_MODNAME,
.driver.owner = THIS_MODULE,
.id_table = virtio_mem_id_table,
diff --git a/include/uapi/linux/virtio_mem.h b/include/uapi/linux/virtio_mem.h
index 1bfade78bdfd..e0a9dc7397c3 100644
--- a/include/uapi/linux/virtio_mem.h
+++ b/include/uapi/linux/virtio_mem.h
@@ -83,6 +83,12 @@
* device is busy.
*/
+/* --- virtio-mem: feature bits --- */
+
+/* node_id is an ACPI PXM and is valid */
+#define VIRTIO_MEM_F_ACPI_PXM 0
+
+
/* --- virtio-mem: guest -> host requests --- */
/* request to plug memory blocks */
@@ -177,7 +183,9 @@ struct virtio_mem_resp {
struct virtio_mem_config {
/* Block size and alignment. Cannot change. */
__u32 block_size;
- __u32 padding;
+ /* Valid with VIRTIO_MEM_F_ACPI_PXM. Cannot change. */
+ __u16 node_id;
+ __u16 padding;
/* Start address of the memory region. Cannot change. */
__u64 addr;
/* Region size (maximum). Cannot change. */
--
2.25.3
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v3 03/15] virtio-mem: Paravirtualized memory hotunplug part 1
2020-05-07 10:31 [PATCH v3 00/15] virtio-mem: paravirtualized memory David Hildenbrand
2020-05-07 10:31 ` [PATCH v3 01/15] virtio-mem: Paravirtualized memory hotplug David Hildenbrand
2020-05-07 10:31 ` [PATCH v3 02/15] virtio-mem: Allow to specify an ACPI PXM as nid David Hildenbrand
@ 2020-05-07 10:31 ` David Hildenbrand
2020-05-07 10:31 ` [PATCH v3 04/15] virtio-mem: Paravirtualized memory hotunplug part 2 David Hildenbrand
` (13 subsequent siblings)
16 siblings, 0 replies; 37+ messages in thread
From: David Hildenbrand @ 2020-05-07 10:31 UTC (permalink / raw)
To: linux-kernel
Cc: linux-mm, virtio-dev, virtualization, kvm, Michal Hocko,
Andrew Morton, Michael S . Tsirkin, David Hildenbrand,
Pankaj Gupta, Jason Wang, Oscar Salvador, Igor Mammedov,
Dave Young, Dan Williams, Pavel Tatashin, Stefan Hajnoczi,
Vlastimil Babka
Unplugging subblocks of memory blocks that are offline is easy. All we
have to do is watch out for concurrent onlining activity.
Tested-by: Pankaj Gupta <pankaj.gupta.linux@gmail.com>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Jason Wang <jasowang@redhat.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Igor Mammedov <imammedo@redhat.com>
Cc: Dave Young <dyoung@redhat.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Pavel Tatashin <pasha.tatashin@soleen.com>
Cc: Stefan Hajnoczi <stefanha@redhat.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: David Hildenbrand <david@redhat.com>
---
drivers/virtio/virtio_mem.c | 116 +++++++++++++++++++++++++++++++++++-
1 file changed, 114 insertions(+), 2 deletions(-)
diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c
index 270ddeaec059..a3ec795be8be 100644
--- a/drivers/virtio/virtio_mem.c
+++ b/drivers/virtio/virtio_mem.c
@@ -123,7 +123,7 @@ struct virtio_mem {
*
* When this lock is held the pointers can't change, ONLINE and
* OFFLINE blocks can't change the state and no subblocks will get
- * plugged.
+ * plugged/unplugged.
*/
struct mutex hotplug_mutex;
bool hotplug_active;
@@ -280,6 +280,12 @@ static int virtio_mem_mb_state_prepare_next_mb(struct virtio_mem *vm)
_mb_id++) \
if (virtio_mem_mb_get_state(_vm, _mb_id) == _state)
+#define virtio_mem_for_each_mb_state_rev(_vm, _mb_id, _state) \
+ for (_mb_id = _vm->next_mb_id - 1; \
+ _mb_id >= _vm->first_mb_id && _vm->nb_mb_state[_state]; \
+ _mb_id--) \
+ if (virtio_mem_mb_get_state(_vm, _mb_id) == _state)
+
/*
* Mark all selected subblocks plugged.
*
@@ -325,6 +331,19 @@ static bool virtio_mem_mb_test_sb_plugged(struct virtio_mem *vm,
bit + count;
}
+/*
+ * Test if all selected subblocks are unplugged.
+ */
+static bool virtio_mem_mb_test_sb_unplugged(struct virtio_mem *vm,
+ unsigned long mb_id, int sb_id,
+ int count)
+{
+ const int bit = (mb_id - vm->first_mb_id) * vm->nb_sb_per_mb + sb_id;
+
+ /* TODO: Helper similar to bitmap_set() */
+ return find_next_bit(vm->sb_bitmap, bit + count, bit) >= bit + count;
+}
+
/*
* Find the first plugged subblock. Returns vm->nb_sb_per_mb in case there is
* none.
@@ -513,6 +532,9 @@ static void virtio_mem_notify_offline(struct virtio_mem *vm,
BUG();
break;
}
+
+ /* trigger the workqueue, maybe we can now unplug memory. */
+ virtio_mem_retry(vm);
}
static void virtio_mem_notify_online(struct virtio_mem *vm, unsigned long mb_id,
@@ -1122,6 +1144,94 @@ static int virtio_mem_plug_request(struct virtio_mem *vm, uint64_t diff)
return rc;
}
+/*
+ * Unplug the desired number of plugged subblocks of an offline memory block.
+ * Will fail if any subblock cannot get unplugged (instead of skipping it).
+ *
+ * Will modify the state of the memory block. Might temporarily drop the
+ * hotplug_mutex.
+ *
+ * Note: Can fail after some subblocks were successfully unplugged.
+ */
+static int virtio_mem_mb_unplug_any_sb_offline(struct virtio_mem *vm,
+ unsigned long mb_id,
+ uint64_t *nb_sb)
+{
+ int rc;
+
+ rc = virtio_mem_mb_unplug_any_sb(vm, mb_id, nb_sb);
+
+ /* some subblocks might have been unplugged even on failure */
+ if (!virtio_mem_mb_test_sb_plugged(vm, mb_id, 0, vm->nb_sb_per_mb))
+ virtio_mem_mb_set_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_OFFLINE_PARTIAL);
+ if (rc)
+ return rc;
+
+ if (virtio_mem_mb_test_sb_unplugged(vm, mb_id, 0, vm->nb_sb_per_mb)) {
+ /*
+ * Remove the block from Linux - this should never fail.
+ * Hinder the block from getting onlined by marking it
+ * unplugged. Temporarily drop the mutex, so
+ * any pending GOING_ONLINE requests can be serviced/rejected.
+ */
+ virtio_mem_mb_set_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_UNUSED);
+
+ mutex_unlock(&vm->hotplug_mutex);
+ rc = virtio_mem_mb_remove(vm, mb_id);
+ BUG_ON(rc);
+ mutex_lock(&vm->hotplug_mutex);
+ }
+ return 0;
+}
+
+/*
+ * Try to unplug the requested amount of memory.
+ */
+static int virtio_mem_unplug_request(struct virtio_mem *vm, uint64_t diff)
+{
+ uint64_t nb_sb = diff / vm->subblock_size;
+ unsigned long mb_id;
+ int rc;
+
+ if (!nb_sb)
+ return 0;
+
+ /*
+ * We'll drop the mutex a couple of times when it is safe to do so.
+ * This might result in some blocks switching the state (online/offline)
+ * and we could miss them in this run - we will retry again later.
+ */
+ mutex_lock(&vm->hotplug_mutex);
+
+ /* Try to unplug subblocks of partially plugged offline blocks. */
+ virtio_mem_for_each_mb_state_rev(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_OFFLINE_PARTIAL) {
+ rc = virtio_mem_mb_unplug_any_sb_offline(vm, mb_id,
+ &nb_sb);
+ if (rc || !nb_sb)
+ goto out_unlock;
+ cond_resched();
+ }
+
+ /* Try to unplug subblocks of plugged offline blocks. */
+ virtio_mem_for_each_mb_state_rev(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_OFFLINE) {
+ rc = virtio_mem_mb_unplug_any_sb_offline(vm, mb_id,
+ &nb_sb);
+ if (rc || !nb_sb)
+ goto out_unlock;
+ cond_resched();
+ }
+
+ mutex_unlock(&vm->hotplug_mutex);
+ return 0;
+out_unlock:
+ mutex_unlock(&vm->hotplug_mutex);
+ return rc;
+}
+
/*
* Try to unplug all blocks that couldn't be unplugged before, for example,
* because the hypervisor was busy.
@@ -1204,8 +1314,10 @@ static void virtio_mem_run_wq(struct work_struct *work)
if (vm->requested_size > vm->plugged_size) {
diff = vm->requested_size - vm->plugged_size;
rc = virtio_mem_plug_request(vm, diff);
+ } else {
+ diff = vm->plugged_size - vm->requested_size;
+ rc = virtio_mem_unplug_request(vm, diff);
}
- /* TODO: try to unplug memory */
}
switch (rc) {
--
2.25.3
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v3 04/15] virtio-mem: Paravirtualized memory hotunplug part 2
2020-05-07 10:31 [PATCH v3 00/15] virtio-mem: paravirtualized memory David Hildenbrand
` (2 preceding siblings ...)
2020-05-07 10:31 ` [PATCH v3 03/15] virtio-mem: Paravirtualized memory hotunplug part 1 David Hildenbrand
@ 2020-05-07 10:31 ` David Hildenbrand
2020-05-07 10:31 ` [PATCH v3 05/15] mm: Allow to offline unmovable PageOffline() pages via MEM_GOING_OFFLINE David Hildenbrand
` (12 subsequent siblings)
16 siblings, 0 replies; 37+ messages in thread
From: David Hildenbrand @ 2020-05-07 10:31 UTC (permalink / raw)
To: linux-kernel
Cc: linux-mm, virtio-dev, virtualization, kvm, Michal Hocko,
Andrew Morton, Michael S . Tsirkin, David Hildenbrand,
Michal Hocko, Pankaj Gupta, Jason Wang, Oscar Salvador,
Igor Mammedov, Dave Young, Dan Williams, Pavel Tatashin,
Stefan Hajnoczi, Vlastimil Babka, Mel Gorman, Mike Rapoport,
Alexander Duyck <alexander.h>
We also want to unplug online memory (contained in online memory blocks
and, therefore, managed by the buddy), and eventually replug it later.
When requested to unplug memory, we use alloc_contig_range() to allocate
subblocks in online memory blocks (so we are the owner) and send them to
our hypervisor. When requested to plug memory, we can replug such memory
using free_contig_range() after asking our hypervisor.
We also want to mark all allocated pages PG_offline, so nobody will
touch them. To differentiate pages that were never onlined when
onlining the memory block from pages allocated via alloc_contig_range(), we
use PageDirty(). Based on this flag, virtio_mem_fake_online() can either
online the pages for the first time or use free_contig_range().
It is worth noting that there are no guarantees on how much memory can
actually get unplugged again. All device memory might completely be
fragmented with unmovable data, such that no subblock can get unplugged.
We are not touching the ZONE_MOVABLE. If memory is onlined to the
ZONE_MOVABLE, it can only get unplugged after that memory was offlined
manually by user space. In normal operation, virtio-mem memory is
suggested to be onlined to ZONE_NORMAL. In the future, we will try to
make unplug more likely to succeed.
Add a module parameter to control if online memory shall be touched.
As we want to access alloc_contig_range()/free_contig_range() from
kernel module context, export the symbols.
Note: Whenever virtio-mem uses alloc_contig_range(), all affected pages
are on the same node, in the same zone, and contain no holes.
Acked-by: Michal Hocko <mhocko@suse.com> # to export contig range allocator API
Tested-by: Pankaj Gupta <pankaj.gupta.linux@gmail.com>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Jason Wang <jasowang@redhat.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Igor Mammedov <imammedo@redhat.com>
Cc: Dave Young <dyoung@redhat.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Pavel Tatashin <pasha.tatashin@soleen.com>
Cc: Stefan Hajnoczi <stefanha@redhat.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Mel Gorman <mgorman@techsingularity.net>
Cc: Mike Rapoport <rppt@linux.ibm.com>
Cc: Alexander Duyck <alexander.h.duyck@linux.intel.com>
Cc: Alexander Potapenko <glider@google.com>
Signed-off-by: David Hildenbrand <david@redhat.com>
---
drivers/virtio/Kconfig | 1 +
drivers/virtio/virtio_mem.c | 157 ++++++++++++++++++++++++++++++++----
mm/page_alloc.c | 2 +
3 files changed, 146 insertions(+), 14 deletions(-)
diff --git a/drivers/virtio/Kconfig b/drivers/virtio/Kconfig
index d6dde7d2cf76..4c1e14615001 100644
--- a/drivers/virtio/Kconfig
+++ b/drivers/virtio/Kconfig
@@ -85,6 +85,7 @@ config VIRTIO_MEM
depends on VIRTIO
depends on MEMORY_HOTPLUG_SPARSE
depends on MEMORY_HOTREMOVE
+ select CONTIG_ALLOC
help
This driver provides access to virtio-mem paravirtualized memory
devices, allowing to hotplug and hotunplug memory.
diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c
index a3ec795be8be..74f0d3cb1d22 100644
--- a/drivers/virtio/virtio_mem.c
+++ b/drivers/virtio/virtio_mem.c
@@ -23,6 +23,10 @@
#include <acpi/acpi_numa.h>
+static bool unplug_online = true;
+module_param(unplug_online, bool, 0644);
+MODULE_PARM_DESC(unplug_online, "Try to unplug online memory");
+
enum virtio_mem_mb_state {
/* Unplugged, not added to Linux. Can be reused later. */
VIRTIO_MEM_MB_STATE_UNUSED = 0,
@@ -654,23 +658,35 @@ static int virtio_mem_memory_notifier_cb(struct notifier_block *nb,
}
/*
- * Set a range of pages PG_offline.
+ * Set a range of pages PG_offline. Remember pages that were never onlined
+ * (via generic_online_page()) using PageDirty().
*/
static void virtio_mem_set_fake_offline(unsigned long pfn,
- unsigned int nr_pages)
+ unsigned int nr_pages, bool onlined)
{
- for (; nr_pages--; pfn++)
- __SetPageOffline(pfn_to_page(pfn));
+ for (; nr_pages--; pfn++) {
+ struct page *page = pfn_to_page(pfn);
+
+ __SetPageOffline(page);
+ if (!onlined)
+ SetPageDirty(page);
+ }
}
/*
- * Clear PG_offline from a range of pages.
+ * Clear PG_offline from a range of pages. If the pages were never onlined,
+ * (via generic_online_page()), clear PageDirty().
*/
static void virtio_mem_clear_fake_offline(unsigned long pfn,
- unsigned int nr_pages)
+ unsigned int nr_pages, bool onlined)
{
- for (; nr_pages--; pfn++)
- __ClearPageOffline(pfn_to_page(pfn));
+ for (; nr_pages--; pfn++) {
+ struct page *page = pfn_to_page(pfn);
+
+ __ClearPageOffline(page);
+ if (!onlined)
+ ClearPageDirty(page);
+ }
}
/*
@@ -686,10 +702,26 @@ static void virtio_mem_fake_online(unsigned long pfn, unsigned int nr_pages)
* We are always called with subblock granularity, which is at least
* aligned to MAX_ORDER - 1.
*/
- virtio_mem_clear_fake_offline(pfn, nr_pages);
+ for (i = 0; i < nr_pages; i += 1 << order) {
+ struct page *page = pfn_to_page(pfn + i);
- for (i = 0; i < nr_pages; i += 1 << order)
- generic_online_page(pfn_to_page(pfn + i), order);
+ /*
+ * If the page is PageDirty(), it was kept fake-offline when
+ * onlining the memory block. Otherwise, it was allocated
+ * using alloc_contig_range(). All pages in a subblock are
+ * alike.
+ */
+ if (PageDirty(page)) {
+ virtio_mem_clear_fake_offline(pfn + i, 1 << order,
+ false);
+ generic_online_page(page, order);
+ } else {
+ virtio_mem_clear_fake_offline(pfn + i, 1 << order,
+ true);
+ free_contig_range(pfn + i, 1 << order);
+ adjust_managed_page_count(page, 1 << order);
+ }
+ }
}
static void virtio_mem_online_page_cb(struct page *page, unsigned int order)
@@ -718,7 +750,8 @@ static void virtio_mem_online_page_cb(struct page *page, unsigned int order)
if (virtio_mem_mb_test_sb_plugged(vm, mb_id, sb_id, 1))
generic_online_page(page, order);
else
- virtio_mem_set_fake_offline(PFN_DOWN(addr), 1 << order);
+ virtio_mem_set_fake_offline(PFN_DOWN(addr), 1 << order,
+ false);
rcu_read_unlock();
return;
}
@@ -1186,6 +1219,72 @@ static int virtio_mem_mb_unplug_any_sb_offline(struct virtio_mem *vm,
return 0;
}
+/*
+ * Unplug the desired number of plugged subblocks of an online memory block.
+ * Will skip subblock that are busy.
+ *
+ * Will modify the state of the memory block.
+ *
+ * Note: Can fail after some subblocks were successfully unplugged. Can
+ * return 0 even if subblocks were busy and could not get unplugged.
+ */
+static int virtio_mem_mb_unplug_any_sb_online(struct virtio_mem *vm,
+ unsigned long mb_id,
+ uint64_t *nb_sb)
+{
+ const unsigned long nr_pages = PFN_DOWN(vm->subblock_size);
+ unsigned long start_pfn;
+ int rc, sb_id;
+
+ /*
+ * TODO: To increase the performance we want to try bigger, consecutive
+ * subblocks first before falling back to single subblocks. Also,
+ * we should sense via something like is_mem_section_removable()
+ * first if it makes sense to go ahead any try to allocate.
+ */
+ for (sb_id = 0; sb_id < vm->nb_sb_per_mb && *nb_sb; sb_id++) {
+ /* Find the next candidate subblock */
+ while (sb_id < vm->nb_sb_per_mb &&
+ !virtio_mem_mb_test_sb_plugged(vm, mb_id, sb_id, 1))
+ sb_id++;
+ if (sb_id >= vm->nb_sb_per_mb)
+ break;
+
+ start_pfn = PFN_DOWN(virtio_mem_mb_id_to_phys(mb_id) +
+ sb_id * vm->subblock_size);
+ rc = alloc_contig_range(start_pfn, start_pfn + nr_pages,
+ MIGRATE_MOVABLE, GFP_KERNEL);
+ if (rc == -ENOMEM)
+ /* whoops, out of memory */
+ return rc;
+ if (rc)
+ /* memory busy, we can't unplug this chunk */
+ continue;
+
+ /* Mark it as fake-offline before unplugging it */
+ virtio_mem_set_fake_offline(start_pfn, nr_pages, true);
+ adjust_managed_page_count(pfn_to_page(start_pfn), -nr_pages);
+
+ /* Try to unplug the allocated memory */
+ rc = virtio_mem_mb_unplug_sb(vm, mb_id, sb_id, 1);
+ if (rc) {
+ /* Return the memory to the buddy. */
+ virtio_mem_fake_online(start_pfn, nr_pages);
+ return rc;
+ }
+
+ virtio_mem_mb_set_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_ONLINE_PARTIAL);
+ *nb_sb -= 1;
+ }
+
+ /*
+ * TODO: Once all subblocks of a memory block were unplugged, we want
+ * to offline the memory block and remove it.
+ */
+ return 0;
+}
+
/*
* Try to unplug the requested amount of memory.
*/
@@ -1225,8 +1324,37 @@ static int virtio_mem_unplug_request(struct virtio_mem *vm, uint64_t diff)
cond_resched();
}
+ if (!unplug_online) {
+ mutex_unlock(&vm->hotplug_mutex);
+ return 0;
+ }
+
+ /* Try to unplug subblocks of partially plugged online blocks. */
+ virtio_mem_for_each_mb_state_rev(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_ONLINE_PARTIAL) {
+ rc = virtio_mem_mb_unplug_any_sb_online(vm, mb_id,
+ &nb_sb);
+ if (rc || !nb_sb)
+ goto out_unlock;
+ mutex_unlock(&vm->hotplug_mutex);
+ cond_resched();
+ mutex_lock(&vm->hotplug_mutex);
+ }
+
+ /* Try to unplug subblocks of plugged online blocks. */
+ virtio_mem_for_each_mb_state_rev(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_ONLINE) {
+ rc = virtio_mem_mb_unplug_any_sb_online(vm, mb_id,
+ &nb_sb);
+ if (rc || !nb_sb)
+ goto out_unlock;
+ mutex_unlock(&vm->hotplug_mutex);
+ cond_resched();
+ mutex_lock(&vm->hotplug_mutex);
+ }
+
mutex_unlock(&vm->hotplug_mutex);
- return 0;
+ return nb_sb ? -EBUSY : 0;
out_unlock:
mutex_unlock(&vm->hotplug_mutex);
return rc;
@@ -1332,7 +1460,8 @@ static void virtio_mem_run_wq(struct work_struct *work)
case -EBUSY:
/*
* The hypervisor cannot process our request right now
- * (e.g., out of memory, migrating).
+ * (e.g., out of memory, migrating) or we cannot free up
+ * any memory to unplug it (all plugged memory is busy).
*/
case -ENOMEM:
/* Out of memory, try again later. */
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 1b265b09d342..64fcaf86763b 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -8461,6 +8461,7 @@ int alloc_contig_range(unsigned long start, unsigned long end,
pfn_max_align_up(end), migratetype);
return ret;
}
+EXPORT_SYMBOL(alloc_contig_range);
static int __alloc_contig_pages(unsigned long start_pfn,
unsigned long nr_pages, gfp_t gfp_mask)
@@ -8576,6 +8577,7 @@ void free_contig_range(unsigned long pfn, unsigned int nr_pages)
}
WARN(count != 0, "%d pages are still in use!\n", count);
}
+EXPORT_SYMBOL(free_contig_range);
/*
* The zone indicated has a new number of managed_pages; batch sizes and percpu
--
2.25.3
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v3 05/15] mm: Allow to offline unmovable PageOffline() pages via MEM_GOING_OFFLINE
2020-05-07 10:31 [PATCH v3 00/15] virtio-mem: paravirtualized memory David Hildenbrand
` (3 preceding siblings ...)
2020-05-07 10:31 ` [PATCH v3 04/15] virtio-mem: Paravirtualized memory hotunplug part 2 David Hildenbrand
@ 2020-05-07 10:31 ` David Hildenbrand
2020-05-07 10:48 ` Michael S. Tsirkin
2020-05-07 10:31 ` [PATCH v3 06/15] virtio-mem: Allow to offline partially unplugged memory blocks David Hildenbrand
` (11 subsequent siblings)
16 siblings, 1 reply; 37+ messages in thread
From: David Hildenbrand @ 2020-05-07 10:31 UTC (permalink / raw)
To: linux-kernel
Cc: linux-mm, virtio-dev, virtualization, kvm, Michal Hocko,
Andrew Morton, Michael S . Tsirkin, David Hildenbrand,
Alexander Duyck, Michal Hocko, Pankaj Gupta, Juergen Gross,
Konrad Rzeszutek Wilk, Pavel Tatashin, Vlastimil Babka,
Johannes Weiner, Anthony Yznaga, Oscar Salvador, Mel Gorman,
Mike Rapoport, Dan Williams, Anshuman Khandual, Qian Cai, Pi
virtio-mem wants to allow to offline memory blocks of which some parts
were unplugged (allocated via alloc_contig_range()), especially, to later
offline and remove completely unplugged memory blocks. The important part
is that PageOffline() has to remain set until the section is offline, so
these pages will never get accessed (e.g., when dumping). The pages should
not be handed back to the buddy (which would require clearing PageOffline()
and result in issues if offlining fails and the pages are suddenly in the
buddy).
Let's allow to do that by allowing to isolate any PageOffline() page
when offlining. This way, we can reach the memory hotplug notifier
MEM_GOING_OFFLINE, where the driver can signal that he is fine with
offlining this page by dropping its reference count. PageOffline() pages
with a reference count of 0 can then be skipped when offlining the
pages (like if they were free, however they are not in the buddy).
Anybody who uses PageOffline() pages and does not agree to offline them
(e.g., Hyper-V balloon, XEN balloon, VMWare balloon for 2MB pages) will not
decrement the reference count and make offlining fail when trying to
migrate such an unmovable page. So there should be no observable change.
Same applies to balloon compaction users (movable PageOffline() pages), the
pages will simply be migrated.
Note 1: If offlining fails, a driver has to increment the reference
count again in MEM_CANCEL_OFFLINE.
Note 2: A driver that makes use of this has to be aware that re-onlining
the memory block has to be handled by hooking into onlining code
(online_page_callback_t), resetting the page PageOffline() and
not giving them to the buddy.
Reviewed-by: Alexander Duyck <alexander.h.duyck@linux.intel.com>
Acked-by: Michal Hocko <mhocko@suse.com>
Tested-by: Pankaj Gupta <pankaj.gupta.linux@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Juergen Gross <jgross@suse.com>
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: Pavel Tatashin <pavel.tatashin@microsoft.com>
Cc: Alexander Duyck <alexander.h.duyck@linux.intel.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Anthony Yznaga <anthony.yznaga@oracle.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Mel Gorman <mgorman@techsingularity.net>
Cc: Mike Rapoport <rppt@linux.ibm.com>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Anshuman Khandual <anshuman.khandual@arm.com>
Cc: Qian Cai <cai@lca.pw>
Cc: Pingfan Liu <kernelfans@gmail.com>
Signed-off-by: David Hildenbrand <david@redhat.com>
---
include/linux/page-flags.h | 10 +++++++++
mm/memory_hotplug.c | 44 +++++++++++++++++++++++++++++---------
mm/page_alloc.c | 24 +++++++++++++++++++++
mm/page_isolation.c | 9 ++++++++
4 files changed, 77 insertions(+), 10 deletions(-)
diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index 222f6f7b2bb3..6be1aa559b1e 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -777,6 +777,16 @@ PAGE_TYPE_OPS(Buddy, buddy)
* not onlined when onlining the section).
* The content of these pages is effectively stale. Such pages should not
* be touched (read/write/dump/save) except by their owner.
+ *
+ * If a driver wants to allow to offline unmovable PageOffline() pages without
+ * putting them back to the buddy, it can do so via the memory notifier by
+ * decrementing the reference count in MEM_GOING_OFFLINE and incrementing the
+ * reference count in MEM_CANCEL_OFFLINE. When offlining, the PageOffline()
+ * pages (now with a reference count of zero) are treated like free pages,
+ * allowing the containing memory block to get offlined. A driver that
+ * relies on this feature is aware that re-onlining the memory block will
+ * require to re-set the pages PageOffline() and not giving them to the
+ * buddy via online_page_callback_t.
*/
PAGE_TYPE_OPS(Offline, offline)
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 555137bd0882..936bfe208a6e 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1151,11 +1151,17 @@ struct zone *test_pages_in_a_zone(unsigned long start_pfn,
/*
* Scan pfn range [start,end) to find movable/migratable pages (LRU pages,
- * non-lru movable pages and hugepages). We scan pfn because it's much
- * easier than scanning over linked list. This function returns the pfn
- * of the first found movable page if it's found, otherwise 0.
+ * non-lru movable pages and hugepages). Will skip over most unmovable
+ * pages (esp., pages that can be skipped when offlining), but bail out on
+ * definitely unmovable pages.
+ *
+ * Returns:
+ * 0 in case a movable page is found and movable_pfn was updated.
+ * -ENOENT in case no movable page was found.
+ * -EBUSY in case a definitely unmovable page was found.
*/
-static unsigned long scan_movable_pages(unsigned long start, unsigned long end)
+static int scan_movable_pages(unsigned long start, unsigned long end,
+ unsigned long *movable_pfn)
{
unsigned long pfn;
@@ -1167,18 +1173,30 @@ static unsigned long scan_movable_pages(unsigned long start, unsigned long end)
continue;
page = pfn_to_page(pfn);
if (PageLRU(page))
- return pfn;
+ goto found;
if (__PageMovable(page))
- return pfn;
+ goto found;
+
+ /*
+ * PageOffline() pages that are not marked __PageMovable() and
+ * have a reference count > 0 (after MEM_GOING_OFFLINE) are
+ * definitely unmovable. If their reference count would be 0,
+ * they could at least be skipped when offlining memory.
+ */
+ if (PageOffline(page) && page_count(page))
+ return -EBUSY;
if (!PageHuge(page))
continue;
head = compound_head(page);
if (page_huge_active(head))
- return pfn;
+ goto found;
skip = compound_nr(head) - (page - head);
pfn += skip - 1;
}
+ return -ENOENT;
+found:
+ *movable_pfn = pfn;
return 0;
}
@@ -1441,7 +1459,8 @@ static int __ref __offline_pages(unsigned long start_pfn,
}
do {
- for (pfn = start_pfn; pfn;) {
+ pfn = start_pfn;
+ do {
if (signal_pending(current)) {
ret = -EINTR;
reason = "signal backoff";
@@ -1451,14 +1470,19 @@ static int __ref __offline_pages(unsigned long start_pfn,
cond_resched();
lru_add_drain_all();
- pfn = scan_movable_pages(pfn, end_pfn);
- if (pfn) {
+ ret = scan_movable_pages(pfn, end_pfn, &pfn);
+ if (!ret) {
/*
* TODO: fatal migration failures should bail
* out
*/
do_migrate_range(pfn, end_pfn);
}
+ } while (!ret);
+
+ if (ret != -ENOENT) {
+ reason = "unmovable page";
+ goto failed_removal_isolated;
}
/*
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 64fcaf86763b..6fb3dfca2a97 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -8230,6 +8230,19 @@ struct page *has_unmovable_pages(struct zone *zone, struct page *page,
if ((flags & MEMORY_OFFLINE) && PageHWPoison(page))
continue;
+ /*
+ * We treat all PageOffline() pages as movable when offlining
+ * to give drivers a chance to decrement their reference count
+ * in MEM_GOING_OFFLINE in order to indicate that these pages
+ * can be offlined as there are no direct references anymore.
+ * For actually unmovable PageOffline() where the driver does
+ * not support this, we will fail later when trying to actually
+ * move these pages that still have a reference count > 0.
+ * (false negatives in this function only)
+ */
+ if ((flags & MEMORY_OFFLINE) && PageOffline(page))
+ continue;
+
if (__PageMovable(page) || PageLRU(page))
continue;
@@ -8650,6 +8663,17 @@ __offline_isolated_pages(unsigned long start_pfn, unsigned long end_pfn)
offlined_pages++;
continue;
}
+ /*
+ * At this point all remaining PageOffline() pages have a
+ * reference count of 0 and can simply be skipped.
+ */
+ if (PageOffline(page)) {
+ BUG_ON(page_count(page));
+ BUG_ON(PageBuddy(page));
+ pfn++;
+ offlined_pages++;
+ continue;
+ }
BUG_ON(page_count(page));
BUG_ON(!PageBuddy(page));
diff --git a/mm/page_isolation.c b/mm/page_isolation.c
index 2c11a38d6e87..f6d07c5f0d34 100644
--- a/mm/page_isolation.c
+++ b/mm/page_isolation.c
@@ -151,6 +151,7 @@ __first_valid_page(unsigned long pfn, unsigned long nr_pages)
* a bit mask)
* MEMORY_OFFLINE - isolate to offline (!allocate) memory
* e.g., skip over PageHWPoison() pages
+ * and PageOffline() pages.
* REPORT_FAILURE - report details about the failure to
* isolate the range
*
@@ -259,6 +260,14 @@ __test_page_isolated_in_pageblock(unsigned long pfn, unsigned long end_pfn,
else if ((flags & MEMORY_OFFLINE) && PageHWPoison(page))
/* A HWPoisoned page cannot be also PageBuddy */
pfn++;
+ else if ((flags & MEMORY_OFFLINE) && PageOffline(page) &&
+ !page_count(page))
+ /*
+ * The responsible driver agreed to skip PageOffline()
+ * pages when offlining memory by dropping its
+ * reference in MEM_GOING_OFFLINE.
+ */
+ pfn++;
else
break;
}
--
2.25.3
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [PATCH v3 05/15] mm: Allow to offline unmovable PageOffline() pages via MEM_GOING_OFFLINE
2020-05-07 10:31 ` [PATCH v3 05/15] mm: Allow to offline unmovable PageOffline() pages via MEM_GOING_OFFLINE David Hildenbrand
@ 2020-05-07 10:48 ` Michael S. Tsirkin
0 siblings, 0 replies; 37+ messages in thread
From: Michael S. Tsirkin @ 2020-05-07 10:48 UTC (permalink / raw)
To: David Hildenbrand
Cc: linux-kernel, linux-mm, virtio-dev, virtualization, kvm,
Michal Hocko, Andrew Morton, Alexander Duyck, Michal Hocko,
Pankaj Gupta, Juergen Gross, Konrad Rzeszutek Wilk,
Pavel Tatashin, Vlastimil Babka, Johannes Weiner, Anthony Yznaga,
Oscar Salvador, Mel Gorman, Mike Rapoport, Dan Williams
On Thu, May 07, 2020 at 12:31:09PM +0200, David Hildenbrand wrote:
> virtio-mem wants to allow to offline memory blocks of which some parts
> were unplugged (allocated via alloc_contig_range()), especially, to later
> offline and remove completely unplugged memory blocks. The important part
> is that PageOffline() has to remain set until the section is offline, so
> these pages will never get accessed (e.g., when dumping). The pages should
> not be handed back to the buddy (which would require clearing PageOffline()
> and result in issues if offlining fails and the pages are suddenly in the
> buddy).
>
> Let's allow to do that by allowing to isolate any PageOffline() page
> when offlining. This way, we can reach the memory hotplug notifier
> MEM_GOING_OFFLINE, where the driver can signal that he is fine with
> offlining this page by dropping its reference count. PageOffline() pages
> with a reference count of 0 can then be skipped when offlining the
> pages (like if they were free, however they are not in the buddy).
>
> Anybody who uses PageOffline() pages and does not agree to offline them
> (e.g., Hyper-V balloon, XEN balloon, VMWare balloon for 2MB pages) will not
> decrement the reference count and make offlining fail when trying to
> migrate such an unmovable page. So there should be no observable change.
> Same applies to balloon compaction users (movable PageOffline() pages), the
> pages will simply be migrated.
>
> Note 1: If offlining fails, a driver has to increment the reference
> count again in MEM_CANCEL_OFFLINE.
>
> Note 2: A driver that makes use of this has to be aware that re-onlining
> the memory block has to be handled by hooking into onlining code
> (online_page_callback_t), resetting the page PageOffline() and
> not giving them to the buddy.
>
> Reviewed-by: Alexander Duyck <alexander.h.duyck@linux.intel.com>
> Acked-by: Michal Hocko <mhocko@suse.com>
> Tested-by: Pankaj Gupta <pankaj.gupta.linux@gmail.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: Juergen Gross <jgross@suse.com>
> Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> Cc: Pavel Tatashin <pavel.tatashin@microsoft.com>
> Cc: Alexander Duyck <alexander.h.duyck@linux.intel.com>
> Cc: Vlastimil Babka <vbabka@suse.cz>
> Cc: Johannes Weiner <hannes@cmpxchg.org>
> Cc: Anthony Yznaga <anthony.yznaga@oracle.com>
> Cc: Michal Hocko <mhocko@suse.com>
> Cc: Oscar Salvador <osalvador@suse.de>
> Cc: Mel Gorman <mgorman@techsingularity.net>
> Cc: Mike Rapoport <rppt@linux.ibm.com>
> Cc: Dan Williams <dan.j.williams@intel.com>
> Cc: Anshuman Khandual <anshuman.khandual@arm.com>
> Cc: Qian Cai <cai@lca.pw>
> Cc: Pingfan Liu <kernelfans@gmail.com>
> Signed-off-by: David Hildenbrand <david@redhat.com>
Didn't you lose some Andrew Morton's ack on this?
> ---
> include/linux/page-flags.h | 10 +++++++++
> mm/memory_hotplug.c | 44 +++++++++++++++++++++++++++++---------
> mm/page_alloc.c | 24 +++++++++++++++++++++
> mm/page_isolation.c | 9 ++++++++
> 4 files changed, 77 insertions(+), 10 deletions(-)
>
> diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
> index 222f6f7b2bb3..6be1aa559b1e 100644
> --- a/include/linux/page-flags.h
> +++ b/include/linux/page-flags.h
> @@ -777,6 +777,16 @@ PAGE_TYPE_OPS(Buddy, buddy)
> * not onlined when onlining the section).
> * The content of these pages is effectively stale. Such pages should not
> * be touched (read/write/dump/save) except by their owner.
> + *
> + * If a driver wants to allow to offline unmovable PageOffline() pages without
> + * putting them back to the buddy, it can do so via the memory notifier by
> + * decrementing the reference count in MEM_GOING_OFFLINE and incrementing the
> + * reference count in MEM_CANCEL_OFFLINE. When offlining, the PageOffline()
> + * pages (now with a reference count of zero) are treated like free pages,
> + * allowing the containing memory block to get offlined. A driver that
> + * relies on this feature is aware that re-onlining the memory block will
> + * require to re-set the pages PageOffline() and not giving them to the
> + * buddy via online_page_callback_t.
> */
> PAGE_TYPE_OPS(Offline, offline)
>
> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
> index 555137bd0882..936bfe208a6e 100644
> --- a/mm/memory_hotplug.c
> +++ b/mm/memory_hotplug.c
> @@ -1151,11 +1151,17 @@ struct zone *test_pages_in_a_zone(unsigned long start_pfn,
>
> /*
> * Scan pfn range [start,end) to find movable/migratable pages (LRU pages,
> - * non-lru movable pages and hugepages). We scan pfn because it's much
> - * easier than scanning over linked list. This function returns the pfn
> - * of the first found movable page if it's found, otherwise 0.
> + * non-lru movable pages and hugepages). Will skip over most unmovable
> + * pages (esp., pages that can be skipped when offlining), but bail out on
> + * definitely unmovable pages.
> + *
> + * Returns:
> + * 0 in case a movable page is found and movable_pfn was updated.
> + * -ENOENT in case no movable page was found.
> + * -EBUSY in case a definitely unmovable page was found.
> */
> -static unsigned long scan_movable_pages(unsigned long start, unsigned long end)
> +static int scan_movable_pages(unsigned long start, unsigned long end,
> + unsigned long *movable_pfn)
> {
> unsigned long pfn;
>
> @@ -1167,18 +1173,30 @@ static unsigned long scan_movable_pages(unsigned long start, unsigned long end)
> continue;
> page = pfn_to_page(pfn);
> if (PageLRU(page))
> - return pfn;
> + goto found;
> if (__PageMovable(page))
> - return pfn;
> + goto found;
> +
> + /*
> + * PageOffline() pages that are not marked __PageMovable() and
> + * have a reference count > 0 (after MEM_GOING_OFFLINE) are
> + * definitely unmovable. If their reference count would be 0,
> + * they could at least be skipped when offlining memory.
> + */
> + if (PageOffline(page) && page_count(page))
> + return -EBUSY;
>
> if (!PageHuge(page))
> continue;
> head = compound_head(page);
> if (page_huge_active(head))
> - return pfn;
> + goto found;
> skip = compound_nr(head) - (page - head);
> pfn += skip - 1;
> }
> + return -ENOENT;
> +found:
> + *movable_pfn = pfn;
> return 0;
> }
>
> @@ -1441,7 +1459,8 @@ static int __ref __offline_pages(unsigned long start_pfn,
> }
>
> do {
> - for (pfn = start_pfn; pfn;) {
> + pfn = start_pfn;
> + do {
> if (signal_pending(current)) {
> ret = -EINTR;
> reason = "signal backoff";
> @@ -1451,14 +1470,19 @@ static int __ref __offline_pages(unsigned long start_pfn,
> cond_resched();
> lru_add_drain_all();
>
> - pfn = scan_movable_pages(pfn, end_pfn);
> - if (pfn) {
> + ret = scan_movable_pages(pfn, end_pfn, &pfn);
> + if (!ret) {
> /*
> * TODO: fatal migration failures should bail
> * out
> */
> do_migrate_range(pfn, end_pfn);
> }
> + } while (!ret);
> +
> + if (ret != -ENOENT) {
> + reason = "unmovable page";
> + goto failed_removal_isolated;
> }
>
> /*
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 64fcaf86763b..6fb3dfca2a97 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -8230,6 +8230,19 @@ struct page *has_unmovable_pages(struct zone *zone, struct page *page,
> if ((flags & MEMORY_OFFLINE) && PageHWPoison(page))
> continue;
>
> + /*
> + * We treat all PageOffline() pages as movable when offlining
> + * to give drivers a chance to decrement their reference count
> + * in MEM_GOING_OFFLINE in order to indicate that these pages
> + * can be offlined as there are no direct references anymore.
> + * For actually unmovable PageOffline() where the driver does
> + * not support this, we will fail later when trying to actually
> + * move these pages that still have a reference count > 0.
> + * (false negatives in this function only)
> + */
> + if ((flags & MEMORY_OFFLINE) && PageOffline(page))
> + continue;
> +
> if (__PageMovable(page) || PageLRU(page))
> continue;
>
> @@ -8650,6 +8663,17 @@ __offline_isolated_pages(unsigned long start_pfn, unsigned long end_pfn)
> offlined_pages++;
> continue;
> }
> + /*
> + * At this point all remaining PageOffline() pages have a
> + * reference count of 0 and can simply be skipped.
> + */
> + if (PageOffline(page)) {
> + BUG_ON(page_count(page));
> + BUG_ON(PageBuddy(page));
> + pfn++;
> + offlined_pages++;
> + continue;
> + }
>
> BUG_ON(page_count(page));
> BUG_ON(!PageBuddy(page));
> diff --git a/mm/page_isolation.c b/mm/page_isolation.c
> index 2c11a38d6e87..f6d07c5f0d34 100644
> --- a/mm/page_isolation.c
> +++ b/mm/page_isolation.c
> @@ -151,6 +151,7 @@ __first_valid_page(unsigned long pfn, unsigned long nr_pages)
> * a bit mask)
> * MEMORY_OFFLINE - isolate to offline (!allocate) memory
> * e.g., skip over PageHWPoison() pages
> + * and PageOffline() pages.
> * REPORT_FAILURE - report details about the failure to
> * isolate the range
> *
> @@ -259,6 +260,14 @@ __test_page_isolated_in_pageblock(unsigned long pfn, unsigned long end_pfn,
> else if ((flags & MEMORY_OFFLINE) && PageHWPoison(page))
> /* A HWPoisoned page cannot be also PageBuddy */
> pfn++;
> + else if ((flags & MEMORY_OFFLINE) && PageOffline(page) &&
> + !page_count(page))
> + /*
> + * The responsible driver agreed to skip PageOffline()
> + * pages when offlining memory by dropping its
> + * reference in MEM_GOING_OFFLINE.
> + */
> + pfn++;
> else
> break;
> }
> --
> 2.25.3
^ permalink raw reply [flat|nested] 37+ messages in thread
* [PATCH v3 06/15] virtio-mem: Allow to offline partially unplugged memory blocks
2020-05-07 10:31 [PATCH v3 00/15] virtio-mem: paravirtualized memory David Hildenbrand
` (4 preceding siblings ...)
2020-05-07 10:31 ` [PATCH v3 05/15] mm: Allow to offline unmovable PageOffline() pages via MEM_GOING_OFFLINE David Hildenbrand
@ 2020-05-07 10:31 ` David Hildenbrand
2020-05-07 10:31 ` [PATCH v3 07/15] mm/memory_hotplug: Introduce offline_and_remove_memory() David Hildenbrand
` (10 subsequent siblings)
16 siblings, 0 replies; 37+ messages in thread
From: David Hildenbrand @ 2020-05-07 10:31 UTC (permalink / raw)
To: linux-kernel
Cc: virtio-dev, Pavel Tatashin, kvm, Michael S . Tsirkin,
Michal Hocko, linux-mm, Stefan Hajnoczi, Pankaj Gupta,
Igor Mammedov, Andrew Morton, virtualization, Dave Young,
Dan Williams, Vlastimil Babka, Oscar Salvador
Dropping the reference count of PageOffline() pages during MEM_GOING_ONLINE
allows offlining code to skip them. However, we also have to clear
PG_reserved, because PG_reserved pages get detected as unmovable right
away. Take care of restoring the reference count when offlining is
canceled.
Clarify why we don't have to perform any action when unloading the
driver. Also, let's add a warning if anybody is still holding a
reference to unplugged pages when offlining.
Tested-by: Pankaj Gupta <pankaj.gupta.linux@gmail.com>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Jason Wang <jasowang@redhat.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Igor Mammedov <imammedo@redhat.com>
Cc: Dave Young <dyoung@redhat.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Pavel Tatashin <pasha.tatashin@soleen.com>
Cc: Stefan Hajnoczi <stefanha@redhat.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: David Hildenbrand <david@redhat.com>
---
drivers/virtio/virtio_mem.c | 68 ++++++++++++++++++++++++++++++++++++-
1 file changed, 67 insertions(+), 1 deletion(-)
diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c
index 74f0d3cb1d22..b0b41c73ce89 100644
--- a/drivers/virtio/virtio_mem.c
+++ b/drivers/virtio/virtio_mem.c
@@ -572,6 +572,57 @@ static void virtio_mem_notify_online(struct virtio_mem *vm, unsigned long mb_id,
virtio_mem_retry(vm);
}
+static void virtio_mem_notify_going_offline(struct virtio_mem *vm,
+ unsigned long mb_id)
+{
+ const unsigned long nr_pages = PFN_DOWN(vm->subblock_size);
+ struct page *page;
+ unsigned long pfn;
+ int sb_id, i;
+
+ for (sb_id = 0; sb_id < vm->nb_sb_per_mb; sb_id++) {
+ if (virtio_mem_mb_test_sb_plugged(vm, mb_id, sb_id, 1))
+ continue;
+ /*
+ * Drop our reference to the pages so the memory can get
+ * offlined and add the unplugged pages to the managed
+ * page counters (so offlining code can correctly subtract
+ * them again).
+ */
+ pfn = PFN_DOWN(virtio_mem_mb_id_to_phys(mb_id) +
+ sb_id * vm->subblock_size);
+ adjust_managed_page_count(pfn_to_page(pfn), nr_pages);
+ for (i = 0; i < nr_pages; i++) {
+ page = pfn_to_page(pfn + i);
+ if (WARN_ON(!page_ref_dec_and_test(page)))
+ dump_page(page, "unplugged page referenced");
+ }
+ }
+}
+
+static void virtio_mem_notify_cancel_offline(struct virtio_mem *vm,
+ unsigned long mb_id)
+{
+ const unsigned long nr_pages = PFN_DOWN(vm->subblock_size);
+ unsigned long pfn;
+ int sb_id, i;
+
+ for (sb_id = 0; sb_id < vm->nb_sb_per_mb; sb_id++) {
+ if (virtio_mem_mb_test_sb_plugged(vm, mb_id, sb_id, 1))
+ continue;
+ /*
+ * Get the reference we dropped when going offline and
+ * subtract the unplugged pages from the managed page
+ * counters.
+ */
+ pfn = PFN_DOWN(virtio_mem_mb_id_to_phys(mb_id) +
+ sb_id * vm->subblock_size);
+ adjust_managed_page_count(pfn_to_page(pfn), -nr_pages);
+ for (i = 0; i < nr_pages; i++)
+ page_ref_inc(pfn_to_page(pfn + i));
+ }
+}
+
/*
* This callback will either be called synchronously from add_memory() or
* asynchronously (e.g., triggered via user space). We have to be careful
@@ -618,6 +669,7 @@ static int virtio_mem_memory_notifier_cb(struct notifier_block *nb,
break;
}
vm->hotplug_active = true;
+ virtio_mem_notify_going_offline(vm, mb_id);
break;
case MEM_GOING_ONLINE:
mutex_lock(&vm->hotplug_mutex);
@@ -642,6 +694,12 @@ static int virtio_mem_memory_notifier_cb(struct notifier_block *nb,
mutex_unlock(&vm->hotplug_mutex);
break;
case MEM_CANCEL_OFFLINE:
+ if (!vm->hotplug_active)
+ break;
+ virtio_mem_notify_cancel_offline(vm, mb_id);
+ vm->hotplug_active = false;
+ mutex_unlock(&vm->hotplug_mutex);
+ break;
case MEM_CANCEL_ONLINE:
if (!vm->hotplug_active)
break;
@@ -668,8 +726,11 @@ static void virtio_mem_set_fake_offline(unsigned long pfn,
struct page *page = pfn_to_page(pfn);
__SetPageOffline(page);
- if (!onlined)
+ if (!onlined) {
SetPageDirty(page);
+ /* FIXME: remove after cleanups */
+ ClearPageReserved(page);
+ }
}
}
@@ -1722,6 +1783,11 @@ static void virtio_mem_remove(struct virtio_device *vdev)
BUG_ON(rc);
virtio_mem_mb_set_state(vm, mb_id, VIRTIO_MEM_MB_STATE_UNUSED);
}
+ /*
+ * After we unregistered our callbacks, user space can no longer
+ * offline partially plugged online memory blocks. No need to worry
+ * about them.
+ */
/* unregister callbacks */
unregister_virtio_mem_device(vm);
--
2.25.3
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v3 07/15] mm/memory_hotplug: Introduce offline_and_remove_memory()
2020-05-07 10:31 [PATCH v3 00/15] virtio-mem: paravirtualized memory David Hildenbrand
` (5 preceding siblings ...)
2020-05-07 10:31 ` [PATCH v3 06/15] virtio-mem: Allow to offline partially unplugged memory blocks David Hildenbrand
@ 2020-05-07 10:31 ` David Hildenbrand
2020-05-07 10:46 ` Michael S. Tsirkin
2020-05-07 10:31 ` [PATCH v3 08/15] virtio-mem: Offline and remove completely unplugged memory blocks David Hildenbrand
` (9 subsequent siblings)
16 siblings, 1 reply; 37+ messages in thread
From: David Hildenbrand @ 2020-05-07 10:31 UTC (permalink / raw)
To: linux-kernel
Cc: linux-mm, virtio-dev, virtualization, kvm, Michal Hocko,
Andrew Morton, Michael S . Tsirkin, David Hildenbrand,
Michal Hocko, Pankaj Gupta, Oscar Salvador, Pavel Tatashin,
Wei Yang, Dan Williams, Qian Cai
virtio-mem wants to offline and remove a memory block once it unplugged
all subblocks (e.g., using alloc_contig_range()). Let's provide
an interface to do that from a driver. virtio-mem already supports to
offline partially unplugged memory blocks. Offlining a fully unplugged
memory block will not require to migrate any pages. All unplugged
subblocks are PageOffline() and have a reference count of 0 - so
offlining code will simply skip them.
All we need is an interface to offline and remove the memory from kernel
module context, where we don't have access to the memory block devices
(esp. find_memory_block() and device_offline()) and the device hotplug
lock.
To keep things simple, allow to only work on a single memory block.
Acked-by: Michal Hocko <mhocko@suse.com>
Tested-by: Pankaj Gupta <pankaj.gupta.linux@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: David Hildenbrand <david@redhat.com>
Cc: Oscar Salvador <osalvador@suse.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Pavel Tatashin <pasha.tatashin@soleen.com>
Cc: Wei Yang <richard.weiyang@gmail.com>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Qian Cai <cai@lca.pw>
Signed-off-by: David Hildenbrand <david@redhat.com>
---
include/linux/memory_hotplug.h | 1 +
mm/memory_hotplug.c | 37 ++++++++++++++++++++++++++++++++++
2 files changed, 38 insertions(+)
diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
index 7dca9cd6076b..d641828e5596 100644
--- a/include/linux/memory_hotplug.h
+++ b/include/linux/memory_hotplug.h
@@ -318,6 +318,7 @@ extern void try_offline_node(int nid);
extern int offline_pages(unsigned long start_pfn, unsigned long nr_pages);
extern int remove_memory(int nid, u64 start, u64 size);
extern void __remove_memory(int nid, u64 start, u64 size);
+extern int offline_and_remove_memory(int nid, u64 start, u64 size);
#else
static inline void try_offline_node(int nid) {}
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 936bfe208a6e..bf1941f02a60 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1748,4 +1748,41 @@ int remove_memory(int nid, u64 start, u64 size)
return rc;
}
EXPORT_SYMBOL_GPL(remove_memory);
+
+/*
+ * Try to offline and remove a memory block. Might take a long time to
+ * finish in case memory is still in use. Primarily useful for memory devices
+ * that logically unplugged all memory (so it's no longer in use) and want to
+ * offline + remove the memory block.
+ */
+int offline_and_remove_memory(int nid, u64 start, u64 size)
+{
+ struct memory_block *mem;
+ int rc = -EINVAL;
+
+ if (!IS_ALIGNED(start, memory_block_size_bytes()) ||
+ size != memory_block_size_bytes())
+ return rc;
+
+ lock_device_hotplug();
+ mem = find_memory_block(__pfn_to_section(PFN_DOWN(start)));
+ if (mem)
+ rc = device_offline(&mem->dev);
+ /* Ignore if the device is already offline. */
+ if (rc > 0)
+ rc = 0;
+
+ /*
+ * In case we succeeded to offline the memory block, remove it.
+ * This cannot fail as it cannot get onlined in the meantime.
+ */
+ if (!rc) {
+ rc = try_remove_memory(nid, start, size);
+ WARN_ON_ONCE(rc);
+ }
+ unlock_device_hotplug();
+
+ return rc;
+}
+EXPORT_SYMBOL_GPL(offline_and_remove_memory);
#endif /* CONFIG_MEMORY_HOTREMOVE */
--
2.25.3
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [PATCH v3 07/15] mm/memory_hotplug: Introduce offline_and_remove_memory()
2020-05-07 10:31 ` [PATCH v3 07/15] mm/memory_hotplug: Introduce offline_and_remove_memory() David Hildenbrand
@ 2020-05-07 10:46 ` Michael S. Tsirkin
2020-05-07 11:24 ` David Hildenbrand
0 siblings, 1 reply; 37+ messages in thread
From: Michael S. Tsirkin @ 2020-05-07 10:46 UTC (permalink / raw)
To: David Hildenbrand
Cc: linux-kernel, linux-mm, virtio-dev, virtualization, kvm,
Michal Hocko, Andrew Morton, Michal Hocko, Pankaj Gupta,
Oscar Salvador, Pavel Tatashin, Wei Yang, Dan Williams, Qian Cai
On Thu, May 07, 2020 at 12:31:11PM +0200, David Hildenbrand wrote:
> virtio-mem wants to offline and remove a memory block once it unplugged
> all subblocks (e.g., using alloc_contig_range()). Let's provide
> an interface to do that from a driver. virtio-mem already supports to
> offline partially unplugged memory blocks. Offlining a fully unplugged
> memory block will not require to migrate any pages. All unplugged
> subblocks are PageOffline() and have a reference count of 0 - so
> offlining code will simply skip them.
>
> All we need is an interface to offline and remove the memory from kernel
> module context, where we don't have access to the memory block devices
> (esp. find_memory_block() and device_offline()) and the device hotplug
> lock.
>
> To keep things simple, allow to only work on a single memory block.
>
> Acked-by: Michal Hocko <mhocko@suse.com>
> Tested-by: Pankaj Gupta <pankaj.gupta.linux@gmail.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: David Hildenbrand <david@redhat.com>
> Cc: Oscar Salvador <osalvador@suse.com>
> Cc: Michal Hocko <mhocko@suse.com>
> Cc: Pavel Tatashin <pasha.tatashin@soleen.com>
> Cc: Wei Yang <richard.weiyang@gmail.com>
> Cc: Dan Williams <dan.j.williams@intel.com>
> Cc: Qian Cai <cai@lca.pw>
> Signed-off-by: David Hildenbrand <david@redhat.com>
didn't you lose Andrew Morton's ack here?
> ---
> include/linux/memory_hotplug.h | 1 +
> mm/memory_hotplug.c | 37 ++++++++++++++++++++++++++++++++++
> 2 files changed, 38 insertions(+)
I get:
error: sha1 information is lacking or useless (mm/memory_hotplug.c).
error: could not build fake ancestor
which version is this against? Pls post patches on top of some tag
in Linus' tree if possible.
> diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
> index 7dca9cd6076b..d641828e5596 100644
> --- a/include/linux/memory_hotplug.h
> +++ b/include/linux/memory_hotplug.h
> @@ -318,6 +318,7 @@ extern void try_offline_node(int nid);
> extern int offline_pages(unsigned long start_pfn, unsigned long nr_pages);
> extern int remove_memory(int nid, u64 start, u64 size);
> extern void __remove_memory(int nid, u64 start, u64 size);
> +extern int offline_and_remove_memory(int nid, u64 start, u64 size);
>
> #else
> static inline void try_offline_node(int nid) {}
> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
> index 936bfe208a6e..bf1941f02a60 100644
> --- a/mm/memory_hotplug.c
> +++ b/mm/memory_hotplug.c
> @@ -1748,4 +1748,41 @@ int remove_memory(int nid, u64 start, u64 size)
> return rc;
> }
> EXPORT_SYMBOL_GPL(remove_memory);
> +
> +/*
> + * Try to offline and remove a memory block. Might take a long time to
> + * finish in case memory is still in use. Primarily useful for memory devices
> + * that logically unplugged all memory (so it's no longer in use) and want to
> + * offline + remove the memory block.
> + */
> +int offline_and_remove_memory(int nid, u64 start, u64 size)
> +{
> + struct memory_block *mem;
> + int rc = -EINVAL;
> +
> + if (!IS_ALIGNED(start, memory_block_size_bytes()) ||
> + size != memory_block_size_bytes())
> + return rc;
> +
> + lock_device_hotplug();
> + mem = find_memory_block(__pfn_to_section(PFN_DOWN(start)));
> + if (mem)
> + rc = device_offline(&mem->dev);
> + /* Ignore if the device is already offline. */
> + if (rc > 0)
> + rc = 0;
> +
> + /*
> + * In case we succeeded to offline the memory block, remove it.
> + * This cannot fail as it cannot get onlined in the meantime.
> + */
> + if (!rc) {
> + rc = try_remove_memory(nid, start, size);
> + WARN_ON_ONCE(rc);
> + }
> + unlock_device_hotplug();
> +
> + return rc;
> +}
> +EXPORT_SYMBOL_GPL(offline_and_remove_memory);
> #endif /* CONFIG_MEMORY_HOTREMOVE */
> --
> 2.25.3
^ permalink raw reply [flat|nested] 37+ messages in thread* Re: [PATCH v3 07/15] mm/memory_hotplug: Introduce offline_and_remove_memory()
2020-05-07 10:46 ` Michael S. Tsirkin
@ 2020-05-07 11:24 ` David Hildenbrand
2020-05-07 11:33 ` David Hildenbrand
2020-05-07 11:33 ` Michael S. Tsirkin
0 siblings, 2 replies; 37+ messages in thread
From: David Hildenbrand @ 2020-05-07 11:24 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: linux-kernel, linux-mm, virtio-dev, virtualization, kvm,
Michal Hocko, Andrew Morton, Michal Hocko, Pankaj Gupta,
Oscar Salvador, Pavel Tatashin, Wei Yang, Dan Williams, Qian Cai
On 07.05.20 12:46, Michael S. Tsirkin wrote:
> On Thu, May 07, 2020 at 12:31:11PM +0200, David Hildenbrand wrote:
>> virtio-mem wants to offline and remove a memory block once it unplugged
>> all subblocks (e.g., using alloc_contig_range()). Let's provide
>> an interface to do that from a driver. virtio-mem already supports to
>> offline partially unplugged memory blocks. Offlining a fully unplugged
>> memory block will not require to migrate any pages. All unplugged
>> subblocks are PageOffline() and have a reference count of 0 - so
>> offlining code will simply skip them.
>>
>> All we need is an interface to offline and remove the memory from kernel
>> module context, where we don't have access to the memory block devices
>> (esp. find_memory_block() and device_offline()) and the device hotplug
>> lock.
>>
>> To keep things simple, allow to only work on a single memory block.
>>
>> Acked-by: Michal Hocko <mhocko@suse.com>
>> Tested-by: Pankaj Gupta <pankaj.gupta.linux@gmail.com>
>> Cc: Andrew Morton <akpm@linux-foundation.org>
>> Cc: David Hildenbrand <david@redhat.com>
>> Cc: Oscar Salvador <osalvador@suse.com>
>> Cc: Michal Hocko <mhocko@suse.com>
>> Cc: Pavel Tatashin <pasha.tatashin@soleen.com>
>> Cc: Wei Yang <richard.weiyang@gmail.com>
>> Cc: Dan Williams <dan.j.williams@intel.com>
>> Cc: Qian Cai <cai@lca.pw>
>> Signed-off-by: David Hildenbrand <david@redhat.com>
>
>
> didn't you lose Andrew Morton's ack here?
Yeah, thanks for noticing.
>
>> ---
>> include/linux/memory_hotplug.h | 1 +
>> mm/memory_hotplug.c | 37 ++++++++++++++++++++++++++++++++++
>> 2 files changed, 38 insertions(+)
>
> I get:
>
> error: sha1 information is lacking or useless (mm/memory_hotplug.c).
> error: could not build fake ancestor
>
> which version is this against? Pls post patches on top of some tag
> in Linus' tree if possible.
As the cover states, latest linux-next. To be precise
commit 6b43f715b6379433e8eb30aa9bcc99bd6a585f77 (tag: next-20200507,
next/master)
Author: Stephen Rothwell <sfr@canb.auug.org.au>
Date: Thu May 7 18:11:31 2020 +1000
Add linux-next specific files for 20200507
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 37+ messages in thread* Re: [PATCH v3 07/15] mm/memory_hotplug: Introduce offline_and_remove_memory()
2020-05-07 11:24 ` David Hildenbrand
@ 2020-05-07 11:33 ` David Hildenbrand
2020-05-07 11:34 ` Michael S. Tsirkin
2020-05-07 11:33 ` Michael S. Tsirkin
1 sibling, 1 reply; 37+ messages in thread
From: David Hildenbrand @ 2020-05-07 11:33 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: linux-kernel, linux-mm, virtio-dev, virtualization, kvm,
Michal Hocko, Andrew Morton, Michal Hocko, Pankaj Gupta,
Oscar Salvador, Pavel Tatashin, Wei Yang, Dan Williams, Qian Cai
>> I get:
>>
>> error: sha1 information is lacking or useless (mm/memory_hotplug.c).
>> error: could not build fake ancestor
>>
>> which version is this against? Pls post patches on top of some tag
>> in Linus' tree if possible.
>
> As the cover states, latest linux-next. To be precise
>
> commit 6b43f715b6379433e8eb30aa9bcc99bd6a585f77 (tag: next-20200507,
> next/master)
> Author: Stephen Rothwell <sfr@canb.auug.org.au>
> Date: Thu May 7 18:11:31 2020 +1000
>
> Add linux-next specific files for 20200507
>
The patches seem to apply cleanly on top of
commit a811c1fa0a02c062555b54651065899437bacdbe (linus/master)
Merge: b9388959ba50 16f8036086a9
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date: Wed May 6 20:53:22 2020 -0700
Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net
I can resend based on that, after giving it a short test.
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 37+ messages in thread* Re: [PATCH v3 07/15] mm/memory_hotplug: Introduce offline_and_remove_memory()
2020-05-07 11:33 ` David Hildenbrand
@ 2020-05-07 11:34 ` Michael S. Tsirkin
2020-05-07 11:37 ` David Hildenbrand
0 siblings, 1 reply; 37+ messages in thread
From: Michael S. Tsirkin @ 2020-05-07 11:34 UTC (permalink / raw)
To: David Hildenbrand
Cc: linux-kernel, linux-mm, virtio-dev, virtualization, kvm,
Michal Hocko, Andrew Morton, Michal Hocko, Pankaj Gupta,
Oscar Salvador, Pavel Tatashin, Wei Yang, Dan Williams, Qian Cai
On Thu, May 07, 2020 at 01:33:23PM +0200, David Hildenbrand wrote:
> >> I get:
> >>
> >> error: sha1 information is lacking or useless (mm/memory_hotplug.c).
> >> error: could not build fake ancestor
> >>
> >> which version is this against? Pls post patches on top of some tag
> >> in Linus' tree if possible.
> >
> > As the cover states, latest linux-next. To be precise
> >
> > commit 6b43f715b6379433e8eb30aa9bcc99bd6a585f77 (tag: next-20200507,
> > next/master)
> > Author: Stephen Rothwell <sfr@canb.auug.org.au>
> > Date: Thu May 7 18:11:31 2020 +1000
> >
> > Add linux-next specific files for 20200507
> >
>
> The patches seem to apply cleanly on top of
>
> commit a811c1fa0a02c062555b54651065899437bacdbe (linus/master)
> Merge: b9388959ba50 16f8036086a9
> Author: Linus Torvalds <torvalds@linux-foundation.org>
> Date: Wed May 6 20:53:22 2020 -0700
>
> Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net
Because you have the relevant hashes in your git tree not pruned yet.
Do a new clone and they won't apply.
>
> I can resend based on that, after giving it a short test.
>
> --
> Thanks,
>
> David / dhildenb
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH v3 07/15] mm/memory_hotplug: Introduce offline_and_remove_memory()
2020-05-07 11:34 ` Michael S. Tsirkin
@ 2020-05-07 11:37 ` David Hildenbrand
2020-05-07 12:11 ` Michael S. Tsirkin
0 siblings, 1 reply; 37+ messages in thread
From: David Hildenbrand @ 2020-05-07 11:37 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: linux-kernel, linux-mm, virtio-dev, virtualization, kvm,
Michal Hocko, Andrew Morton, Michal Hocko, Pankaj Gupta,
Oscar Salvador, Pavel Tatashin, Wei Yang, Dan Williams, Qian Cai
On 07.05.20 13:34, Michael S. Tsirkin wrote:
> On Thu, May 07, 2020 at 01:33:23PM +0200, David Hildenbrand wrote:
>>>> I get:
>>>>
>>>> error: sha1 information is lacking or useless (mm/memory_hotplug.c).
>>>> error: could not build fake ancestor
>>>>
>>>> which version is this against? Pls post patches on top of some tag
>>>> in Linus' tree if possible.
>>>
>>> As the cover states, latest linux-next. To be precise
>>>
>>> commit 6b43f715b6379433e8eb30aa9bcc99bd6a585f77 (tag: next-20200507,
>>> next/master)
>>> Author: Stephen Rothwell <sfr@canb.auug.org.au>
>>> Date: Thu May 7 18:11:31 2020 +1000
>>>
>>> Add linux-next specific files for 20200507
>>>
>>
>> The patches seem to apply cleanly on top of
>>
>> commit a811c1fa0a02c062555b54651065899437bacdbe (linus/master)
>> Merge: b9388959ba50 16f8036086a9
>> Author: Linus Torvalds <torvalds@linux-foundation.org>
>> Date: Wed May 6 20:53:22 2020 -0700
>>
>> Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net
>
> Because you have the relevant hashes in your git tree not pruned yet.
> Do a new clone and they won't apply.
>
Yeah, most probably, it knows how to merge. I'm used to sending all my
-mm stuff based on -next, so this here is different.
I'll wait a bit and then send v4 based on latest linus/master, adding
the two acks and reshuffling the MAINTAINERS patch. Thanks.
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH v3 07/15] mm/memory_hotplug: Introduce offline_and_remove_memory()
2020-05-07 11:37 ` David Hildenbrand
@ 2020-05-07 12:11 ` Michael S. Tsirkin
2020-05-07 12:24 ` David Hildenbrand
0 siblings, 1 reply; 37+ messages in thread
From: Michael S. Tsirkin @ 2020-05-07 12:11 UTC (permalink / raw)
To: David Hildenbrand
Cc: linux-kernel, linux-mm, virtio-dev, virtualization, kvm,
Michal Hocko, Andrew Morton, Michal Hocko, Pankaj Gupta,
Oscar Salvador, Pavel Tatashin, Wei Yang, Dan Williams, Qian Cai
On Thu, May 07, 2020 at 01:37:30PM +0200, David Hildenbrand wrote:
> On 07.05.20 13:34, Michael S. Tsirkin wrote:
> > On Thu, May 07, 2020 at 01:33:23PM +0200, David Hildenbrand wrote:
> >>>> I get:
> >>>>
> >>>> error: sha1 information is lacking or useless (mm/memory_hotplug.c).
> >>>> error: could not build fake ancestor
> >>>>
> >>>> which version is this against? Pls post patches on top of some tag
> >>>> in Linus' tree if possible.
> >>>
> >>> As the cover states, latest linux-next. To be precise
> >>>
> >>> commit 6b43f715b6379433e8eb30aa9bcc99bd6a585f77 (tag: next-20200507,
> >>> next/master)
> >>> Author: Stephen Rothwell <sfr@canb.auug.org.au>
> >>> Date: Thu May 7 18:11:31 2020 +1000
> >>>
> >>> Add linux-next specific files for 20200507
> >>>
> >>
> >> The patches seem to apply cleanly on top of
> >>
> >> commit a811c1fa0a02c062555b54651065899437bacdbe (linus/master)
> >> Merge: b9388959ba50 16f8036086a9
> >> Author: Linus Torvalds <torvalds@linux-foundation.org>
> >> Date: Wed May 6 20:53:22 2020 -0700
> >>
> >> Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net
> >
> > Because you have the relevant hashes in your git tree not pruned yet.
> > Do a new clone and they won't apply.
> >
>
> Yeah, most probably, it knows how to merge. I'm used to sending all my
> -mm stuff based on -next, so this here is different.
Documentation/process/5.Posting.rst addresses this:
Patches must be prepared against a specific version of the kernel. As a
general rule, a patch should be based on the current mainline as found in
Linus's git tree. When basing on mainline, start with a well-known release
point - a stable or -rc release - rather than branching off the mainline at
an arbitrary spot.
It may become necessary to make versions against -mm, linux-next, or a
subsystem tree, though, to facilitate wider testing and review. Depending
on the area of your patch and what is going on elsewhere, basing a patch
against these other trees can require a significant amount of work
resolving conflicts and dealing with API changes.
> I'll wait a bit and then send v4 based on latest linus/master, adding
> the two acks and reshuffling the MAINTAINERS patch. Thanks.
>
> --
> Thanks,
>
> David / dhildenb
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH v3 07/15] mm/memory_hotplug: Introduce offline_and_remove_memory()
2020-05-07 12:11 ` Michael S. Tsirkin
@ 2020-05-07 12:24 ` David Hildenbrand
0 siblings, 0 replies; 37+ messages in thread
From: David Hildenbrand @ 2020-05-07 12:24 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: linux-kernel, linux-mm, virtio-dev, virtualization, kvm,
Michal Hocko, Andrew Morton, Michal Hocko, Pankaj Gupta,
Oscar Salvador, Pavel Tatashin, Wei Yang, Dan Williams, Qian Cai
On 07.05.20 14:11, Michael S. Tsirkin wrote:
> On Thu, May 07, 2020 at 01:37:30PM +0200, David Hildenbrand wrote:
>> On 07.05.20 13:34, Michael S. Tsirkin wrote:
>>> On Thu, May 07, 2020 at 01:33:23PM +0200, David Hildenbrand wrote:
>>>>>> I get:
>>>>>>
>>>>>> error: sha1 information is lacking or useless (mm/memory_hotplug.c).
>>>>>> error: could not build fake ancestor
>>>>>>
>>>>>> which version is this against? Pls post patches on top of some tag
>>>>>> in Linus' tree if possible.
>>>>>
>>>>> As the cover states, latest linux-next. To be precise
>>>>>
>>>>> commit 6b43f715b6379433e8eb30aa9bcc99bd6a585f77 (tag: next-20200507,
>>>>> next/master)
>>>>> Author: Stephen Rothwell <sfr@canb.auug.org.au>
>>>>> Date: Thu May 7 18:11:31 2020 +1000
>>>>>
>>>>> Add linux-next specific files for 20200507
>>>>>
>>>>
>>>> The patches seem to apply cleanly on top of
>>>>
>>>> commit a811c1fa0a02c062555b54651065899437bacdbe (linus/master)
>>>> Merge: b9388959ba50 16f8036086a9
>>>> Author: Linus Torvalds <torvalds@linux-foundation.org>
>>>> Date: Wed May 6 20:53:22 2020 -0700
>>>>
>>>> Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net
>>>
>>> Because you have the relevant hashes in your git tree not pruned yet.
>>> Do a new clone and they won't apply.
>>>
>>
>> Yeah, most probably, it knows how to merge. I'm used to sending all my
>> -mm stuff based on -next, so this here is different.
>
>
> Documentation/process/5.Posting.rst addresses this:
>
Thanks for the info.
>
> Patches must be prepared against a specific version of the kernel. As a
> general rule, a patch should be based on the current mainline as found in
> Linus's git tree. When basing on mainline, start with a well-known release
> point - a stable or -rc release - rather than branching off the mainline at
> an arbitrary spot.
>
> It may become necessary to make versions against -mm, linux-next, or a
> subsystem tree, though, to facilitate wider testing and review. Depending
> on the area of your patch and what is going on elsewhere, basing a patch
> against these other trees can require a significant amount of work
> resolving conflicts and dealing with API changes.
Yeah, but with -mm patches it is completely impractical to base them
against Linus's git tree.
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH v3 07/15] mm/memory_hotplug: Introduce offline_and_remove_memory()
2020-05-07 11:24 ` David Hildenbrand
2020-05-07 11:33 ` David Hildenbrand
@ 2020-05-07 11:33 ` Michael S. Tsirkin
1 sibling, 0 replies; 37+ messages in thread
From: Michael S. Tsirkin @ 2020-05-07 11:33 UTC (permalink / raw)
To: David Hildenbrand
Cc: linux-kernel, linux-mm, virtio-dev, virtualization, kvm,
Michal Hocko, Andrew Morton, Michal Hocko, Pankaj Gupta,
Oscar Salvador, Pavel Tatashin, Wei Yang, Dan Williams, Qian Cai
On Thu, May 07, 2020 at 01:24:38PM +0200, David Hildenbrand wrote:
> On 07.05.20 12:46, Michael S. Tsirkin wrote:
> > On Thu, May 07, 2020 at 12:31:11PM +0200, David Hildenbrand wrote:
> >> virtio-mem wants to offline and remove a memory block once it unplugged
> >> all subblocks (e.g., using alloc_contig_range()). Let's provide
> >> an interface to do that from a driver. virtio-mem already supports to
> >> offline partially unplugged memory blocks. Offlining a fully unplugged
> >> memory block will not require to migrate any pages. All unplugged
> >> subblocks are PageOffline() and have a reference count of 0 - so
> >> offlining code will simply skip them.
> >>
> >> All we need is an interface to offline and remove the memory from kernel
> >> module context, where we don't have access to the memory block devices
> >> (esp. find_memory_block() and device_offline()) and the device hotplug
> >> lock.
> >>
> >> To keep things simple, allow to only work on a single memory block.
> >>
> >> Acked-by: Michal Hocko <mhocko@suse.com>
> >> Tested-by: Pankaj Gupta <pankaj.gupta.linux@gmail.com>
> >> Cc: Andrew Morton <akpm@linux-foundation.org>
> >> Cc: David Hildenbrand <david@redhat.com>
> >> Cc: Oscar Salvador <osalvador@suse.com>
> >> Cc: Michal Hocko <mhocko@suse.com>
> >> Cc: Pavel Tatashin <pasha.tatashin@soleen.com>
> >> Cc: Wei Yang <richard.weiyang@gmail.com>
> >> Cc: Dan Williams <dan.j.williams@intel.com>
> >> Cc: Qian Cai <cai@lca.pw>
> >> Signed-off-by: David Hildenbrand <david@redhat.com>
> >
> >
> > didn't you lose Andrew Morton's ack here?
>
> Yeah, thanks for noticing.
>
> >
> >> ---
> >> include/linux/memory_hotplug.h | 1 +
> >> mm/memory_hotplug.c | 37 ++++++++++++++++++++++++++++++++++
> >> 2 files changed, 38 insertions(+)
> >
> > I get:
> >
> > error: sha1 information is lacking or useless (mm/memory_hotplug.c).
> > error: could not build fake ancestor
> >
> > which version is this against? Pls post patches on top of some tag
> > in Linus' tree if possible.
>
> As the cover states, latest linux-next. To be precise
>
> commit 6b43f715b6379433e8eb30aa9bcc99bd6a585f77 (tag: next-20200507,
> next/master)
> Author: Stephen Rothwell <sfr@canb.auug.org.au>
> Date: Thu May 7 18:11:31 2020 +1000
>
> Add linux-next specific files for 20200507
>
Don't base on linux-next please. Generally base on the tree you are
targeting, or Linus' tree.
> --
> Thanks,
>
> David / dhildenb
^ permalink raw reply [flat|nested] 37+ messages in thread
* [PATCH v3 08/15] virtio-mem: Offline and remove completely unplugged memory blocks
2020-05-07 10:31 [PATCH v3 00/15] virtio-mem: paravirtualized memory David Hildenbrand
` (6 preceding siblings ...)
2020-05-07 10:31 ` [PATCH v3 07/15] mm/memory_hotplug: Introduce offline_and_remove_memory() David Hildenbrand
@ 2020-05-07 10:31 ` David Hildenbrand
2020-05-07 10:31 ` [PATCH v3 09/15] virtio-mem: Better retry handling David Hildenbrand
` (8 subsequent siblings)
16 siblings, 0 replies; 37+ messages in thread
From: David Hildenbrand @ 2020-05-07 10:31 UTC (permalink / raw)
To: linux-kernel
Cc: virtio-dev, Pavel Tatashin, kvm, Michael S . Tsirkin,
Michal Hocko, linux-mm, Stefan Hajnoczi, Pankaj Gupta,
Igor Mammedov, Andrew Morton, virtualization, Dave Young,
Dan Williams, Vlastimil Babka, Oscar Salvador
Let's offline+remove memory blocks once all subblocks are unplugged. We
can use the new Linux MM interface for that. As no memory is in use
anymore, this shouldn't take a long time and shouldn't fail. There might
be corner cases where the offlining could still fail (especially, if
another notifier NACKs the offlining request).
Acked-by: Pankaj Gupta <pankaj.gupta.linux@gmail.com>
Tested-by: Pankaj Gupta <pankaj.gupta.linux@gmail.com>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Jason Wang <jasowang@redhat.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Igor Mammedov <imammedo@redhat.com>
Cc: Dave Young <dyoung@redhat.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Pavel Tatashin <pasha.tatashin@soleen.com>
Cc: Stefan Hajnoczi <stefanha@redhat.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: David Hildenbrand <david@redhat.com>
---
drivers/virtio/virtio_mem.c | 47 +++++++++++++++++++++++++++++++++----
1 file changed, 43 insertions(+), 4 deletions(-)
diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c
index b0b41c73ce89..a2edb87e5ed8 100644
--- a/drivers/virtio/virtio_mem.c
+++ b/drivers/virtio/virtio_mem.c
@@ -446,6 +446,28 @@ static int virtio_mem_mb_remove(struct virtio_mem *vm, unsigned long mb_id)
return remove_memory(nid, addr, memory_block_size_bytes());
}
+/*
+ * Try to offline and remove a memory block from Linux.
+ *
+ * Must not be called with the vm->hotplug_mutex held (possible deadlock with
+ * onlining code).
+ *
+ * Will not modify the state of the memory block.
+ */
+static int virtio_mem_mb_offline_and_remove(struct virtio_mem *vm,
+ unsigned long mb_id)
+{
+ const uint64_t addr = virtio_mem_mb_id_to_phys(mb_id);
+ int nid = vm->nid;
+
+ if (nid == NUMA_NO_NODE)
+ nid = memory_add_physaddr_to_nid(addr);
+
+ dev_dbg(&vm->vdev->dev, "offlining and removing memory block: %lu\n",
+ mb_id);
+ return offline_and_remove_memory(nid, addr, memory_block_size_bytes());
+}
+
/*
* Trigger the workqueue so the device can perform its magic.
*/
@@ -537,7 +559,13 @@ static void virtio_mem_notify_offline(struct virtio_mem *vm,
break;
}
- /* trigger the workqueue, maybe we can now unplug memory. */
+ /*
+ * Trigger the workqueue, maybe we can now unplug memory. Also,
+ * when we offline and remove a memory block, this will re-trigger
+ * us immediately - which is often nice because the removal of
+ * the memory block (e.g., memmap) might have freed up memory
+ * on other memory blocks we manage.
+ */
virtio_mem_retry(vm);
}
@@ -1284,7 +1312,8 @@ static int virtio_mem_mb_unplug_any_sb_offline(struct virtio_mem *vm,
* Unplug the desired number of plugged subblocks of an online memory block.
* Will skip subblock that are busy.
*
- * Will modify the state of the memory block.
+ * Will modify the state of the memory block. Might temporarily drop the
+ * hotplug_mutex.
*
* Note: Can fail after some subblocks were successfully unplugged. Can
* return 0 even if subblocks were busy and could not get unplugged.
@@ -1340,9 +1369,19 @@ static int virtio_mem_mb_unplug_any_sb_online(struct virtio_mem *vm,
}
/*
- * TODO: Once all subblocks of a memory block were unplugged, we want
- * to offline the memory block and remove it.
+ * Once all subblocks of a memory block were unplugged, offline and
+ * remove it. This will usually not fail, as no memory is in use
+ * anymore - however some other notifiers might NACK the request.
*/
+ if (virtio_mem_mb_test_sb_unplugged(vm, mb_id, 0, vm->nb_sb_per_mb)) {
+ mutex_unlock(&vm->hotplug_mutex);
+ rc = virtio_mem_mb_offline_and_remove(vm, mb_id);
+ mutex_lock(&vm->hotplug_mutex);
+ if (!rc)
+ virtio_mem_mb_set_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_UNUSED);
+ }
+
return 0;
}
--
2.25.3
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v3 09/15] virtio-mem: Better retry handling
2020-05-07 10:31 [PATCH v3 00/15] virtio-mem: paravirtualized memory David Hildenbrand
` (7 preceding siblings ...)
2020-05-07 10:31 ` [PATCH v3 08/15] virtio-mem: Offline and remove completely unplugged memory blocks David Hildenbrand
@ 2020-05-07 10:31 ` David Hildenbrand
2020-05-07 10:31 ` [PATCH v3 10/15] MAINTAINERS: Add myself as virtio-mem maintainer David Hildenbrand
` (7 subsequent siblings)
16 siblings, 0 replies; 37+ messages in thread
From: David Hildenbrand @ 2020-05-07 10:31 UTC (permalink / raw)
To: linux-kernel
Cc: linux-mm, virtio-dev, virtualization, kvm, Michal Hocko,
Andrew Morton, Michael S . Tsirkin, David Hildenbrand,
Pankaj Gupta, Jason Wang, Oscar Salvador, Igor Mammedov,
Dave Young, Dan Williams, Pavel Tatashin, Stefan Hajnoczi,
Vlastimil Babka
Let's start with a retry interval of 5 seconds and double the time until
we reach 5 minutes, in case we keep getting errors. Reset the retry
interval in case we succeeded.
The two main reasons for having to retry are
- The hypervisor is busy and cannot process our request
- We cannot reach the desired requested_size (esp., not enough memory can
get unplugged because we can't allocate any subblocks).
Tested-by: Pankaj Gupta <pankaj.gupta.linux@gmail.com>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Jason Wang <jasowang@redhat.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Igor Mammedov <imammedo@redhat.com>
Cc: Dave Young <dyoung@redhat.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Pavel Tatashin <pasha.tatashin@soleen.com>
Cc: Stefan Hajnoczi <stefanha@redhat.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: David Hildenbrand <david@redhat.com>
---
drivers/virtio/virtio_mem.c | 11 ++++++++---
1 file changed, 8 insertions(+), 3 deletions(-)
diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c
index a2edb87e5ed8..eb4c16d634e0 100644
--- a/drivers/virtio/virtio_mem.c
+++ b/drivers/virtio/virtio_mem.c
@@ -141,7 +141,9 @@ struct virtio_mem {
/* Timer for retrying to plug/unplug memory. */
struct hrtimer retry_timer;
-#define VIRTIO_MEM_RETRY_TIMER_MS 30000
+ unsigned int retry_timer_ms;
+#define VIRTIO_MEM_RETRY_TIMER_MIN_MS 50000
+#define VIRTIO_MEM_RETRY_TIMER_MAX_MS 300000
/* Memory notifier (online/offline events). */
struct notifier_block memory_notifier;
@@ -1550,6 +1552,7 @@ static void virtio_mem_run_wq(struct work_struct *work)
switch (rc) {
case 0:
+ vm->retry_timer_ms = VIRTIO_MEM_RETRY_TIMER_MIN_MS;
break;
case -ENOSPC:
/*
@@ -1565,8 +1568,7 @@ static void virtio_mem_run_wq(struct work_struct *work)
*/
case -ENOMEM:
/* Out of memory, try again later. */
- hrtimer_start(&vm->retry_timer,
- ms_to_ktime(VIRTIO_MEM_RETRY_TIMER_MS),
+ hrtimer_start(&vm->retry_timer, ms_to_ktime(vm->retry_timer_ms),
HRTIMER_MODE_REL);
break;
case -EAGAIN:
@@ -1586,6 +1588,8 @@ static enum hrtimer_restart virtio_mem_timer_expired(struct hrtimer *timer)
retry_timer);
virtio_mem_retry(vm);
+ vm->retry_timer_ms = min_t(unsigned int, vm->retry_timer_ms * 2,
+ VIRTIO_MEM_RETRY_TIMER_MAX_MS);
return HRTIMER_NORESTART;
}
@@ -1754,6 +1758,7 @@ static int virtio_mem_probe(struct virtio_device *vdev)
spin_lock_init(&vm->removal_lock);
hrtimer_init(&vm->retry_timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
vm->retry_timer.function = virtio_mem_timer_expired;
+ vm->retry_timer_ms = VIRTIO_MEM_RETRY_TIMER_MIN_MS;
/* register the virtqueue */
rc = virtio_mem_init_vq(vm);
--
2.25.3
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v3 10/15] MAINTAINERS: Add myself as virtio-mem maintainer
2020-05-07 10:31 [PATCH v3 00/15] virtio-mem: paravirtualized memory David Hildenbrand
` (8 preceding siblings ...)
2020-05-07 10:31 ` [PATCH v3 09/15] virtio-mem: Better retry handling David Hildenbrand
@ 2020-05-07 10:31 ` David Hildenbrand
2020-05-07 10:47 ` Michael S. Tsirkin
2020-05-07 10:31 ` [PATCH v3 11/15] virtio-mem: Add parent resource for all added "System RAM" David Hildenbrand
` (6 subsequent siblings)
16 siblings, 1 reply; 37+ messages in thread
From: David Hildenbrand @ 2020-05-07 10:31 UTC (permalink / raw)
To: linux-kernel
Cc: linux-mm, virtio-dev, virtualization, kvm, Michal Hocko,
Andrew Morton, Michael S . Tsirkin, David Hildenbrand
Let's make sure patches/bug reports find the right person.
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Signed-off-by: David Hildenbrand <david@redhat.com>
---
MAINTAINERS | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/MAINTAINERS b/MAINTAINERS
index 4d43ea5468b5..ad2b34f4dd66 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -18037,6 +18037,13 @@ S: Maintained
F: drivers/iommu/virtio-iommu.c
F: include/uapi/linux/virtio_iommu.h
+VIRTIO MEM DRIVER
+M: David Hildenbrand <david@redhat.com>
+L: virtualization@lists.linux-foundation.org
+S: Maintained
+F: drivers/virtio/virtio_mem.c
+F: include/uapi/linux/virtio_mem.h
+
VIRTUAL BOX GUEST DEVICE DRIVER
M: Hans de Goede <hdegoede@redhat.com>
M: Arnd Bergmann <arnd@arndb.de>
--
2.25.3
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [PATCH v3 10/15] MAINTAINERS: Add myself as virtio-mem maintainer
2020-05-07 10:31 ` [PATCH v3 10/15] MAINTAINERS: Add myself as virtio-mem maintainer David Hildenbrand
@ 2020-05-07 10:47 ` Michael S. Tsirkin
2020-05-07 11:25 ` David Hildenbrand
0 siblings, 1 reply; 37+ messages in thread
From: Michael S. Tsirkin @ 2020-05-07 10:47 UTC (permalink / raw)
To: David Hildenbrand
Cc: linux-kernel, linux-mm, virtio-dev, virtualization, kvm,
Michal Hocko, Andrew Morton
On Thu, May 07, 2020 at 12:31:14PM +0200, David Hildenbrand wrote:
> Let's make sure patches/bug reports find the right person.
>
> Cc: "Michael S. Tsirkin" <mst@redhat.com>
> Signed-off-by: David Hildenbrand <david@redhat.com>
Make this patch 2 in the series, or even squash into patch 1.
> ---
> MAINTAINERS | 7 +++++++
> 1 file changed, 7 insertions(+)
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 4d43ea5468b5..ad2b34f4dd66 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -18037,6 +18037,13 @@ S: Maintained
> F: drivers/iommu/virtio-iommu.c
> F: include/uapi/linux/virtio_iommu.h
>
> +VIRTIO MEM DRIVER
> +M: David Hildenbrand <david@redhat.com>
> +L: virtualization@lists.linux-foundation.org
> +S: Maintained
> +F: drivers/virtio/virtio_mem.c
> +F: include/uapi/linux/virtio_mem.h
> +
> VIRTUAL BOX GUEST DEVICE DRIVER
> M: Hans de Goede <hdegoede@redhat.com>
> M: Arnd Bergmann <arnd@arndb.de>
> --
> 2.25.3
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH v3 10/15] MAINTAINERS: Add myself as virtio-mem maintainer
2020-05-07 10:47 ` Michael S. Tsirkin
@ 2020-05-07 11:25 ` David Hildenbrand
0 siblings, 0 replies; 37+ messages in thread
From: David Hildenbrand @ 2020-05-07 11:25 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: linux-kernel, linux-mm, virtio-dev, virtualization, kvm,
Michal Hocko, Andrew Morton
On 07.05.20 12:47, Michael S. Tsirkin wrote:
> On Thu, May 07, 2020 at 12:31:14PM +0200, David Hildenbrand wrote:
>> Let's make sure patches/bug reports find the right person.
>>
>> Cc: "Michael S. Tsirkin" <mst@redhat.com>
>> Signed-off-by: David Hildenbrand <david@redhat.com>
>
> Make this patch 2 in the series, or even squash into patch 1.
I'll move it to #2. If there are strong feelings, I can squash. Thanks!
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 37+ messages in thread
* [PATCH v3 11/15] virtio-mem: Add parent resource for all added "System RAM"
2020-05-07 10:31 [PATCH v3 00/15] virtio-mem: paravirtualized memory David Hildenbrand
` (9 preceding siblings ...)
2020-05-07 10:31 ` [PATCH v3 10/15] MAINTAINERS: Add myself as virtio-mem maintainer David Hildenbrand
@ 2020-05-07 10:31 ` David Hildenbrand
2020-05-07 10:31 ` [PATCH v3 12/15] virtio-mem: Drop manual check for already present memory David Hildenbrand
` (5 subsequent siblings)
16 siblings, 0 replies; 37+ messages in thread
From: David Hildenbrand @ 2020-05-07 10:31 UTC (permalink / raw)
To: linux-kernel
Cc: linux-mm, virtio-dev, virtualization, kvm, Michal Hocko,
Andrew Morton, Michael S . Tsirkin, David Hildenbrand,
Pankaj Gupta
Let's add a parent resource, named after the virtio device (inspired by
drivers/dax/kmem.c). This allows user space to identify which memory
belongs to which virtio-mem device.
With this change and two virtio-mem devices:
:/# cat /proc/iomem
00000000-00000fff : Reserved
00001000-0009fbff : System RAM
[...]
140000000-333ffffff : virtio0
140000000-147ffffff : System RAM
148000000-14fffffff : System RAM
150000000-157ffffff : System RAM
[...]
334000000-3033ffffff : virtio1
338000000-33fffffff : System RAM
340000000-347ffffff : System RAM
348000000-34fffffff : System RAM
[...]
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Pankaj Gupta <pankaj.gupta.linux@gmail.com>
Signed-off-by: David Hildenbrand <david@redhat.com>
---
drivers/virtio/virtio_mem.c | 52 ++++++++++++++++++++++++++++++++++++-
1 file changed, 51 insertions(+), 1 deletion(-)
diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c
index eb4c16d634e0..80cdb9e6b3c4 100644
--- a/drivers/virtio/virtio_mem.c
+++ b/drivers/virtio/virtio_mem.c
@@ -99,6 +99,9 @@ struct virtio_mem {
/* Id of the next memory bock to prepare when needed. */
unsigned long next_mb_id;
+ /* The parent resource for all memory added via this device. */
+ struct resource *parent_resource;
+
/* Summary of all memory block states. */
unsigned long nb_mb_state[VIRTIO_MEM_MB_STATE_COUNT];
#define VIRTIO_MEM_NB_OFFLINE_THRESHOLD 10
@@ -1741,6 +1744,44 @@ static int virtio_mem_init(struct virtio_mem *vm)
return 0;
}
+static int virtio_mem_create_resource(struct virtio_mem *vm)
+{
+ /*
+ * When force-unloading the driver and removing the device, we
+ * could have a garbage pointer. Duplicate the string.
+ */
+ const char *name = kstrdup(dev_name(&vm->vdev->dev), GFP_KERNEL);
+
+ if (!name)
+ return -ENOMEM;
+
+ vm->parent_resource = __request_mem_region(vm->addr, vm->region_size,
+ name, IORESOURCE_SYSTEM_RAM);
+ if (!vm->parent_resource) {
+ kfree(name);
+ dev_warn(&vm->vdev->dev, "could not reserve device region\n");
+ return -EBUSY;
+ }
+
+ /* The memory is not actually busy - make add_memory() work. */
+ vm->parent_resource->flags &= ~IORESOURCE_BUSY;
+ return 0;
+}
+
+static void virtio_mem_delete_resource(struct virtio_mem *vm)
+{
+ const char *name;
+
+ if (!vm->parent_resource)
+ return;
+
+ name = vm->parent_resource->name;
+ release_resource(vm->parent_resource);
+ kfree(vm->parent_resource);
+ kfree(name);
+ vm->parent_resource = NULL;
+}
+
static int virtio_mem_probe(struct virtio_device *vdev)
{
struct virtio_mem *vm;
@@ -1770,11 +1811,16 @@ static int virtio_mem_probe(struct virtio_device *vdev)
if (rc)
goto out_del_vq;
+ /* create the parent resource for all memory */
+ rc = virtio_mem_create_resource(vm);
+ if (rc)
+ goto out_del_vq;
+
/* register callbacks */
vm->memory_notifier.notifier_call = virtio_mem_memory_notifier_cb;
rc = register_memory_notifier(&vm->memory_notifier);
if (rc)
- goto out_del_vq;
+ goto out_del_resource;
rc = register_virtio_mem_device(vm);
if (rc)
goto out_unreg_mem;
@@ -1788,6 +1834,8 @@ static int virtio_mem_probe(struct virtio_device *vdev)
return 0;
out_unreg_mem:
unregister_memory_notifier(&vm->memory_notifier);
+out_del_resource:
+ virtio_mem_delete_resource(vm);
out_del_vq:
vdev->config->del_vqs(vdev);
out_free_vm:
@@ -1848,6 +1896,8 @@ static void virtio_mem_remove(struct virtio_device *vdev)
vm->nb_mb_state[VIRTIO_MEM_MB_STATE_ONLINE_PARTIAL] ||
vm->nb_mb_state[VIRTIO_MEM_MB_STATE_ONLINE_MOVABLE])
dev_warn(&vdev->dev, "device still has system memory added\n");
+ else
+ virtio_mem_delete_resource(vm);
/* remove all tracking data - no locking needed */
vfree(vm->mb_state);
--
2.25.3
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v3 12/15] virtio-mem: Drop manual check for already present memory
2020-05-07 10:31 [PATCH v3 00/15] virtio-mem: paravirtualized memory David Hildenbrand
` (10 preceding siblings ...)
2020-05-07 10:31 ` [PATCH v3 11/15] virtio-mem: Add parent resource for all added "System RAM" David Hildenbrand
@ 2020-05-07 10:31 ` David Hildenbrand
2020-05-07 10:31 ` [PATCH v3 13/15] virtio-mem: Unplug subblocks right-to-left David Hildenbrand
` (4 subsequent siblings)
16 siblings, 0 replies; 37+ messages in thread
From: David Hildenbrand @ 2020-05-07 10:31 UTC (permalink / raw)
To: linux-kernel
Cc: virtio-dev, kvm, Michael S . Tsirkin, Michal Hocko, linux-mm,
Pankaj Gupta, Andrew Morton, virtualization
Registering our parent resource will fail if any memory is still present
(e.g., because somebody unloaded the driver and tries to reload it). No
need for the manual check.
Move our "unplug all" handling to after registering the resource.
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Pankaj Gupta <pankaj.gupta.linux@gmail.com>
Signed-off-by: David Hildenbrand <david@redhat.com>
---
drivers/virtio/virtio_mem.c | 55 ++++++++-----------------------------
1 file changed, 12 insertions(+), 43 deletions(-)
diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c
index 80cdb9e6b3c4..8dd57b61b09b 100644
--- a/drivers/virtio/virtio_mem.c
+++ b/drivers/virtio/virtio_mem.c
@@ -1616,23 +1616,6 @@ static int virtio_mem_init_vq(struct virtio_mem *vm)
return 0;
}
-/*
- * Test if any memory in the range is present in Linux.
- */
-static bool virtio_mem_any_memory_present(unsigned long start,
- unsigned long size)
-{
- const unsigned long start_pfn = PFN_DOWN(start);
- const unsigned long end_pfn = PFN_UP(start + size);
- unsigned long pfn;
-
- for (pfn = start_pfn; pfn != end_pfn; pfn++)
- if (present_section_nr(pfn_to_section_nr(pfn)))
- return true;
-
- return false;
-}
-
static int virtio_mem_init(struct virtio_mem *vm)
{
const uint64_t phys_limit = 1UL << MAX_PHYSMEM_BITS;
@@ -1664,32 +1647,6 @@ static int virtio_mem_init(struct virtio_mem *vm)
virtio_cread(vm->vdev, struct virtio_mem_config, region_size,
&vm->region_size);
- /*
- * If we still have memory plugged, we might have to unplug all
- * memory first. However, if somebody simply unloaded the driver
- * we would have to reinitialize the old state - something we don't
- * support yet. Detect if we have any memory in the area present.
- */
- if (vm->plugged_size) {
- uint64_t usable_region_size;
-
- virtio_cread(vm->vdev, struct virtio_mem_config,
- usable_region_size, &usable_region_size);
-
- if (virtio_mem_any_memory_present(vm->addr,
- usable_region_size)) {
- dev_err(&vm->vdev->dev,
- "reloading the driver is not supported\n");
- return -EINVAL;
- }
- /*
- * Note: it might happen that the device is busy and
- * unplugging all memory might take some time.
- */
- dev_info(&vm->vdev->dev, "unplugging all memory required\n");
- vm->unplug_all_required = 1;
- }
-
/*
* We always hotplug memory in memory block granularity. This way,
* we have to wait for exactly one memory block to online.
@@ -1760,6 +1717,8 @@ static int virtio_mem_create_resource(struct virtio_mem *vm)
if (!vm->parent_resource) {
kfree(name);
dev_warn(&vm->vdev->dev, "could not reserve device region\n");
+ dev_info(&vm->vdev->dev,
+ "reloading the driver is not supported\n");
return -EBUSY;
}
@@ -1816,6 +1775,16 @@ static int virtio_mem_probe(struct virtio_device *vdev)
if (rc)
goto out_del_vq;
+ /*
+ * If we still have memory plugged, we have to unplug all memory first.
+ * Registering our parent resource makes sure that this memory isn't
+ * actually in use (e.g., trying to reload the driver).
+ */
+ if (vm->plugged_size) {
+ vm->unplug_all_required = 1;
+ dev_info(&vm->vdev->dev, "unplugging all memory is required\n");
+ }
+
/* register callbacks */
vm->memory_notifier.notifier_call = virtio_mem_memory_notifier_cb;
rc = register_memory_notifier(&vm->memory_notifier);
--
2.25.3
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v3 13/15] virtio-mem: Unplug subblocks right-to-left
2020-05-07 10:31 [PATCH v3 00/15] virtio-mem: paravirtualized memory David Hildenbrand
` (11 preceding siblings ...)
2020-05-07 10:31 ` [PATCH v3 12/15] virtio-mem: Drop manual check for already present memory David Hildenbrand
@ 2020-05-07 10:31 ` David Hildenbrand
2020-05-07 10:31 ` [PATCH v3 14/15] virtio-mem: Use -ETXTBSY as error code if the device is busy David Hildenbrand
` (3 subsequent siblings)
16 siblings, 0 replies; 37+ messages in thread
From: David Hildenbrand @ 2020-05-07 10:31 UTC (permalink / raw)
To: linux-kernel
Cc: virtio-dev, kvm, Michael S . Tsirkin, Michal Hocko, linux-mm,
Pankaj Gupta, Andrew Morton, virtualization
We unplug blocks right-to-left, let's also unplug subblocks within a block
right-to-left.
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Pankaj Gupta <pankaj.gupta.linux@gmail.com>
Signed-off-by: David Hildenbrand <david@redhat.com>
---
drivers/virtio/virtio_mem.c | 38 ++++++++++++++++---------------------
1 file changed, 16 insertions(+), 22 deletions(-)
diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c
index 8dd57b61b09b..a719e1a04ac7 100644
--- a/drivers/virtio/virtio_mem.c
+++ b/drivers/virtio/virtio_mem.c
@@ -353,18 +353,6 @@ static bool virtio_mem_mb_test_sb_unplugged(struct virtio_mem *vm,
return find_next_bit(vm->sb_bitmap, bit + count, bit) >= bit + count;
}
-/*
- * Find the first plugged subblock. Returns vm->nb_sb_per_mb in case there is
- * none.
- */
-static int virtio_mem_mb_first_plugged_sb(struct virtio_mem *vm,
- unsigned long mb_id)
-{
- const int bit = (mb_id - vm->first_mb_id) * vm->nb_sb_per_mb;
-
- return find_next_bit(vm->sb_bitmap, bit + vm->nb_sb_per_mb, bit) - bit;
-}
-
/*
* Find the first unplugged subblock. Returns vm->nb_sb_per_mb in case there is
* none.
@@ -1016,21 +1004,27 @@ static int virtio_mem_mb_unplug_any_sb(struct virtio_mem *vm,
int sb_id, count;
int rc;
+ sb_id = vm->nb_sb_per_mb - 1;
while (*nb_sb) {
- sb_id = virtio_mem_mb_first_plugged_sb(vm, mb_id);
- if (sb_id >= vm->nb_sb_per_mb)
+ /* Find the next candidate subblock */
+ while (sb_id >= 0 &&
+ virtio_mem_mb_test_sb_unplugged(vm, mb_id, sb_id, 1))
+ sb_id--;
+ if (sb_id < 0)
break;
+ /* Try to unplug multiple subblocks at a time */
count = 1;
- while (count < *nb_sb &&
- sb_id + count < vm->nb_sb_per_mb &&
- virtio_mem_mb_test_sb_plugged(vm, mb_id, sb_id + count,
- 1))
+ while (count < *nb_sb && sb_id > 0 &&
+ virtio_mem_mb_test_sb_plugged(vm, mb_id, sb_id - 1, 1)) {
count++;
+ sb_id--;
+ }
rc = virtio_mem_mb_unplug_sb(vm, mb_id, sb_id, count);
if (rc)
return rc;
*nb_sb -= count;
+ sb_id--;
}
return 0;
@@ -1337,12 +1331,12 @@ static int virtio_mem_mb_unplug_any_sb_online(struct virtio_mem *vm,
* we should sense via something like is_mem_section_removable()
* first if it makes sense to go ahead any try to allocate.
*/
- for (sb_id = 0; sb_id < vm->nb_sb_per_mb && *nb_sb; sb_id++) {
+ for (sb_id = vm->nb_sb_per_mb - 1; sb_id >= 0 && *nb_sb; sb_id--) {
/* Find the next candidate subblock */
- while (sb_id < vm->nb_sb_per_mb &&
+ while (sb_id >= 0 &&
!virtio_mem_mb_test_sb_plugged(vm, mb_id, sb_id, 1))
- sb_id++;
- if (sb_id >= vm->nb_sb_per_mb)
+ sb_id--;
+ if (sb_id < 0)
break;
start_pfn = PFN_DOWN(virtio_mem_mb_id_to_phys(mb_id) +
--
2.25.3
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v3 14/15] virtio-mem: Use -ETXTBSY as error code if the device is busy
2020-05-07 10:31 [PATCH v3 00/15] virtio-mem: paravirtualized memory David Hildenbrand
` (12 preceding siblings ...)
2020-05-07 10:31 ` [PATCH v3 13/15] virtio-mem: Unplug subblocks right-to-left David Hildenbrand
@ 2020-05-07 10:31 ` David Hildenbrand
2020-05-07 10:31 ` [PATCH v3 15/15] virtio-mem: Try to unplug the complete online memory block first David Hildenbrand
` (2 subsequent siblings)
16 siblings, 0 replies; 37+ messages in thread
From: David Hildenbrand @ 2020-05-07 10:31 UTC (permalink / raw)
To: linux-kernel
Cc: virtio-dev, kvm, Michael S . Tsirkin, Michal Hocko, linux-mm,
Pankaj Gupta, Andrew Morton, virtualization
Let's be able to distinguish if the device or if memory is busy.
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Pankaj Gupta <pankaj.gupta.linux@gmail.com>
Signed-off-by: David Hildenbrand <david@redhat.com>
---
drivers/virtio/virtio_mem.c | 16 ++++++++++------
1 file changed, 10 insertions(+), 6 deletions(-)
diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c
index a719e1a04ac7..abd93b778a26 100644
--- a/drivers/virtio/virtio_mem.c
+++ b/drivers/virtio/virtio_mem.c
@@ -893,7 +893,7 @@ static int virtio_mem_send_plug_request(struct virtio_mem *vm, uint64_t addr,
case VIRTIO_MEM_RESP_NACK:
return -EAGAIN;
case VIRTIO_MEM_RESP_BUSY:
- return -EBUSY;
+ return -ETXTBSY;
case VIRTIO_MEM_RESP_ERROR:
return -EINVAL;
default:
@@ -919,7 +919,7 @@ static int virtio_mem_send_unplug_request(struct virtio_mem *vm, uint64_t addr,
vm->plugged_size -= size;
return 0;
case VIRTIO_MEM_RESP_BUSY:
- return -EBUSY;
+ return -ETXTBSY;
case VIRTIO_MEM_RESP_ERROR:
return -EINVAL;
default:
@@ -941,7 +941,7 @@ static int virtio_mem_send_unplug_all_request(struct virtio_mem *vm)
atomic_set(&vm->config_changed, 1);
return 0;
case VIRTIO_MEM_RESP_BUSY:
- return -EBUSY;
+ return -ETXTBSY;
default:
return -ENOMEM;
}
@@ -1557,11 +1557,15 @@ static void virtio_mem_run_wq(struct work_struct *work)
* or we have too many offline memory blocks.
*/
break;
- case -EBUSY:
+ case -ETXTBSY:
/*
* The hypervisor cannot process our request right now
- * (e.g., out of memory, migrating) or we cannot free up
- * any memory to unplug it (all plugged memory is busy).
+ * (e.g., out of memory, migrating);
+ */
+ case -EBUSY:
+ /*
+ * We cannot free up any memory to unplug it (all plugged memory
+ * is busy).
*/
case -ENOMEM:
/* Out of memory, try again later. */
--
2.25.3
^ permalink raw reply related [flat|nested] 37+ messages in thread* [PATCH v3 15/15] virtio-mem: Try to unplug the complete online memory block first
2020-05-07 10:31 [PATCH v3 00/15] virtio-mem: paravirtualized memory David Hildenbrand
` (13 preceding siblings ...)
2020-05-07 10:31 ` [PATCH v3 14/15] virtio-mem: Use -ETXTBSY as error code if the device is busy David Hildenbrand
@ 2020-05-07 10:31 ` David Hildenbrand
2020-05-07 10:48 ` [PATCH v3 00/15] virtio-mem: paravirtualized memory Michael S. Tsirkin
2020-05-14 6:44 ` teawater
16 siblings, 0 replies; 37+ messages in thread
From: David Hildenbrand @ 2020-05-07 10:31 UTC (permalink / raw)
To: linux-kernel
Cc: linux-mm, virtio-dev, virtualization, kvm, Michal Hocko,
Andrew Morton, Michael S . Tsirkin, David Hildenbrand,
Pankaj Gupta
Right now, we always try to unplug single subblocks when processing an
online memory block. Let's try to unplug the complete online memory block
first, in case it is fully plugged and the unplug request is large
enough. Fallback to single subblocks in case the memory block cannot get
unplugged as a whole.
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Pankaj Gupta <pankaj.gupta.linux@gmail.com>
Signed-off-by: David Hildenbrand <david@redhat.com>
---
drivers/virtio/virtio_mem.c | 88 ++++++++++++++++++++++++-------------
1 file changed, 57 insertions(+), 31 deletions(-)
diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c
index abd93b778a26..9e523db3bee1 100644
--- a/drivers/virtio/virtio_mem.c
+++ b/drivers/virtio/virtio_mem.c
@@ -1307,6 +1307,46 @@ static int virtio_mem_mb_unplug_any_sb_offline(struct virtio_mem *vm,
return 0;
}
+/*
+ * Unplug the given plugged subblocks of an online memory block.
+ *
+ * Will modify the state of the memory block.
+ */
+static int virtio_mem_mb_unplug_sb_online(struct virtio_mem *vm,
+ unsigned long mb_id, int sb_id,
+ int count)
+{
+ const unsigned long nr_pages = PFN_DOWN(vm->subblock_size) * count;
+ unsigned long start_pfn;
+ int rc;
+
+ start_pfn = PFN_DOWN(virtio_mem_mb_id_to_phys(mb_id) +
+ sb_id * vm->subblock_size);
+ rc = alloc_contig_range(start_pfn, start_pfn + nr_pages,
+ MIGRATE_MOVABLE, GFP_KERNEL);
+ if (rc == -ENOMEM)
+ /* whoops, out of memory */
+ return rc;
+ if (rc)
+ return -EBUSY;
+
+ /* Mark it as fake-offline before unplugging it */
+ virtio_mem_set_fake_offline(start_pfn, nr_pages, true);
+ adjust_managed_page_count(pfn_to_page(start_pfn), -nr_pages);
+
+ /* Try to unplug the allocated memory */
+ rc = virtio_mem_mb_unplug_sb(vm, mb_id, sb_id, count);
+ if (rc) {
+ /* Return the memory to the buddy. */
+ virtio_mem_fake_online(start_pfn, nr_pages);
+ return rc;
+ }
+
+ virtio_mem_mb_set_state(vm, mb_id,
+ VIRTIO_MEM_MB_STATE_ONLINE_PARTIAL);
+ return 0;
+}
+
/*
* Unplug the desired number of plugged subblocks of an online memory block.
* Will skip subblock that are busy.
@@ -1321,16 +1361,21 @@ static int virtio_mem_mb_unplug_any_sb_online(struct virtio_mem *vm,
unsigned long mb_id,
uint64_t *nb_sb)
{
- const unsigned long nr_pages = PFN_DOWN(vm->subblock_size);
- unsigned long start_pfn;
int rc, sb_id;
- /*
- * TODO: To increase the performance we want to try bigger, consecutive
- * subblocks first before falling back to single subblocks. Also,
- * we should sense via something like is_mem_section_removable()
- * first if it makes sense to go ahead any try to allocate.
- */
+ /* If possible, try to unplug the complete block in one shot. */
+ if (*nb_sb >= vm->nb_sb_per_mb &&
+ virtio_mem_mb_test_sb_plugged(vm, mb_id, 0, vm->nb_sb_per_mb)) {
+ rc = virtio_mem_mb_unplug_sb_online(vm, mb_id, 0,
+ vm->nb_sb_per_mb);
+ if (!rc) {
+ *nb_sb -= vm->nb_sb_per_mb;
+ goto unplugged;
+ } else if (rc != -EBUSY)
+ return rc;
+ }
+
+ /* Fallback to single subblocks. */
for (sb_id = vm->nb_sb_per_mb - 1; sb_id >= 0 && *nb_sb; sb_id--) {
/* Find the next candidate subblock */
while (sb_id >= 0 &&
@@ -1339,34 +1384,15 @@ static int virtio_mem_mb_unplug_any_sb_online(struct virtio_mem *vm,
if (sb_id < 0)
break;
- start_pfn = PFN_DOWN(virtio_mem_mb_id_to_phys(mb_id) +
- sb_id * vm->subblock_size);
- rc = alloc_contig_range(start_pfn, start_pfn + nr_pages,
- MIGRATE_MOVABLE, GFP_KERNEL);
- if (rc == -ENOMEM)
- /* whoops, out of memory */
- return rc;
- if (rc)
- /* memory busy, we can't unplug this chunk */
+ rc = virtio_mem_mb_unplug_sb_online(vm, mb_id, sb_id, 1);
+ if (rc == -EBUSY)
continue;
-
- /* Mark it as fake-offline before unplugging it */
- virtio_mem_set_fake_offline(start_pfn, nr_pages, true);
- adjust_managed_page_count(pfn_to_page(start_pfn), -nr_pages);
-
- /* Try to unplug the allocated memory */
- rc = virtio_mem_mb_unplug_sb(vm, mb_id, sb_id, 1);
- if (rc) {
- /* Return the memory to the buddy. */
- virtio_mem_fake_online(start_pfn, nr_pages);
+ else if (rc)
return rc;
- }
-
- virtio_mem_mb_set_state(vm, mb_id,
- VIRTIO_MEM_MB_STATE_ONLINE_PARTIAL);
*nb_sb -= 1;
}
+unplugged:
/*
* Once all subblocks of a memory block were unplugged, offline and
* remove it. This will usually not fail, as no memory is in use
--
2.25.3
^ permalink raw reply related [flat|nested] 37+ messages in thread* Re: [PATCH v3 00/15] virtio-mem: paravirtualized memory
2020-05-07 10:31 [PATCH v3 00/15] virtio-mem: paravirtualized memory David Hildenbrand
` (14 preceding siblings ...)
2020-05-07 10:31 ` [PATCH v3 15/15] virtio-mem: Try to unplug the complete online memory block first David Hildenbrand
@ 2020-05-07 10:48 ` Michael S. Tsirkin
2020-05-07 11:23 ` David Hildenbrand
2020-05-14 6:44 ` teawater
16 siblings, 1 reply; 37+ messages in thread
From: Michael S. Tsirkin @ 2020-05-07 10:48 UTC (permalink / raw)
To: David Hildenbrand
Cc: linux-kernel, linux-mm, virtio-dev, virtualization, kvm,
Michal Hocko, Andrew Morton, Sebastien Boeuf, Samuel Ortiz,
Robert Bradford, Luiz Capitulino, Pankaj Gupta, teawater,
Igor Mammedov, Dr . David Alan Gilbert, Alexander Duyck,
Alexander Potapenko, Anshuman Khandual, Anthony Yznaga, Dan
On Thu, May 07, 2020 at 12:31:04PM +0200, David Hildenbrand wrote:
> This series is based on latest linux-next. The patches are located at:
> https://github.com/davidhildenbrand/linux.git virtio-mem-v3
>
> Patch #1 - #10 where contained in v2 and only contain minor modifications
> (mostly smaller fixes). The remaining patches are new and contain smaller
> optimizations.
Looks like you lost some acks, in particular I'd like to preserve
Andrew Morton's ack.
> Details about virtio-mem can be found in the cover letter of v2 [1]. A
> basic QEMU implementation was posted yesterday [2].
>
> [1] https://lkml.kernel.org/r/20200311171422.10484-1-david@redhat.com
> [2] https://lkml.kernel.org/r/20200506094948.76388-1-david@redhat.com
>
> v2 -> v3:
> - "virtio-mem: Paravirtualized memory hotplug"
> -- Include "linux/slab.h" to fix build issues
> -- Remember the "region_size", helpful for patch #11
> -- Minor simplifaction in virtio_mem_overlaps_range()
> -- Use notifier_from_errno() instead of notifier_to_errno() in notifier
> -- More reliable check for added memory when unloading the driver
> - "virtio-mem: Allow to specify an ACPI PXM as nid"
> -- Also print the nid
> - Added patch #11-#15
>
> Cc: Sebastien Boeuf <sebastien.boeuf@intel.com>
> Cc: Samuel Ortiz <samuel.ortiz@intel.com>
> Cc: Robert Bradford <robert.bradford@intel.com>
> Cc: Luiz Capitulino <lcapitulino@redhat.com>
> Cc: Pankaj Gupta <pankaj.gupta.linux@gmail.com>
> Cc: teawater <teawaterz@linux.alibaba.com>
> Cc: Igor Mammedov <imammedo@redhat.com>
> Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>
>
> David Hildenbrand (15):
> virtio-mem: Paravirtualized memory hotplug
> virtio-mem: Allow to specify an ACPI PXM as nid
> virtio-mem: Paravirtualized memory hotunplug part 1
> virtio-mem: Paravirtualized memory hotunplug part 2
> mm: Allow to offline unmovable PageOffline() pages via
> MEM_GOING_OFFLINE
> virtio-mem: Allow to offline partially unplugged memory blocks
> mm/memory_hotplug: Introduce offline_and_remove_memory()
> virtio-mem: Offline and remove completely unplugged memory blocks
> virtio-mem: Better retry handling
> MAINTAINERS: Add myself as virtio-mem maintainer
> virtio-mem: Add parent resource for all added "System RAM"
> virtio-mem: Drop manual check for already present memory
> virtio-mem: Unplug subblocks right-to-left
> virtio-mem: Use -ETXTBSY as error code if the device is busy
> virtio-mem: Try to unplug the complete online memory block first
>
> MAINTAINERS | 7 +
> drivers/acpi/numa/srat.c | 1 +
> drivers/virtio/Kconfig | 17 +
> drivers/virtio/Makefile | 1 +
> drivers/virtio/virtio_mem.c | 1962 +++++++++++++++++++++++++++++++
> include/linux/memory_hotplug.h | 1 +
> include/linux/page-flags.h | 10 +
> include/uapi/linux/virtio_ids.h | 1 +
> include/uapi/linux/virtio_mem.h | 208 ++++
> mm/memory_hotplug.c | 81 +-
> mm/page_alloc.c | 26 +
> mm/page_isolation.c | 9 +
> 12 files changed, 2314 insertions(+), 10 deletions(-)
> create mode 100644 drivers/virtio/virtio_mem.c
> create mode 100644 include/uapi/linux/virtio_mem.h
>
> --
> 2.25.3
^ permalink raw reply [flat|nested] 37+ messages in thread* Re: [PATCH v3 00/15] virtio-mem: paravirtualized memory
2020-05-07 10:48 ` [PATCH v3 00/15] virtio-mem: paravirtualized memory Michael S. Tsirkin
@ 2020-05-07 11:23 ` David Hildenbrand
0 siblings, 0 replies; 37+ messages in thread
From: David Hildenbrand @ 2020-05-07 11:23 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: linux-kernel, linux-mm, virtio-dev, virtualization, kvm,
Michal Hocko, Andrew Morton, Sebastien Boeuf, Samuel Ortiz,
Robert Bradford, Luiz Capitulino, Pankaj Gupta, teawater,
Igor Mammedov, Dr . David Alan Gilbert, Alexander Duyck,
Alexander Potapenko, Anshuman Khandual, Anthony Yznaga,
Dan Williams, Dave Young, Jason Wang,
Johannes Weiner <hanne>
On 07.05.20 12:48, Michael S. Tsirkin wrote:
> On Thu, May 07, 2020 at 12:31:04PM +0200, David Hildenbrand wrote:
>> This series is based on latest linux-next. The patches are located at:
>> https://github.com/davidhildenbrand/linux.git virtio-mem-v3
>>
>> Patch #1 - #10 where contained in v2 and only contain minor modifications
>> (mostly smaller fixes). The remaining patches are new and contain smaller
>> optimizations.
>
>
> Looks like you lost some acks, in particular I'd like to preserve
> Andrew Morton's ack.
Yeah, seems like I only picked up Pankaj's acks. I can resend.
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH v3 00/15] virtio-mem: paravirtualized memory
2020-05-07 10:31 [PATCH v3 00/15] virtio-mem: paravirtualized memory David Hildenbrand
` (15 preceding siblings ...)
2020-05-07 10:48 ` [PATCH v3 00/15] virtio-mem: paravirtualized memory Michael S. Tsirkin
@ 2020-05-14 6:44 ` teawater
2020-05-14 8:48 ` David Hildenbrand
16 siblings, 1 reply; 37+ messages in thread
From: teawater @ 2020-05-14 6:44 UTC (permalink / raw)
To: David Hildenbrand
Cc: LKML, Linux MM, virtio-dev, Linux Virtualization, kvm,
Michal Hocko, Andrew Morton, Michael S . Tsirkin, Sebastien Boeuf,
Samuel Ortiz, Robert Bradford, Luiz Capitulino, Pankaj Gupta,
Igor Mammedov, Dr . David Alan Gilbert, Alexander Duyck,
Alexander Potapenko, Anshuman Khandual, Anthony Yznaga,
Dan Williams, Dave Young, Jason
Hi David,
I got a kernel warning with v2 and v3.
// start a QEMU that is get from https://github.com/davidhildenbrand/qemu/tree/virtio-mem-v2 and setup a file as a ide disk.
/home/teawater/qemu/qemu/x86_64-softmmu/qemu-system-x86_64 -machine pc-i440fx-2.1,accel=kvm,usb=off -cpu host -no-reboot -nographic -device ide-hd,drive=hd -drive if=none,id=hd,file=/home/teawater/old.img,format=raw -kernel /home/teawater/kernel/bk2/arch/x86/boot/bzImage -append "console=ttyS0 root=/dev/sda nokaslr swiotlb=noforce" -m 1g,slots=10,maxmem=2G -smp 1 -s -monitor unix:/home/teawater/qemu/m,server,nowait
// Setup virtio-mem and plug 256m memory in qemu monitor:
object_add memory-backend-ram,id=mem1,size=256m
device_add virtio-mem-pci,id=vm0,memdev=mem1
qom-set vm0 requested-size 256M
// Go back to the terminal and access file system will got following kernel warning.
[ 19.515549] pci 0000:00:04.0: [1af4:1015] type 00 class 0x00ff00
[ 19.516227] pci 0000:00:04.0: reg 0x10: [io 0x0000-0x007f]
[ 19.517196] pci 0000:00:04.0: BAR 0: assigned [io 0x1000-0x107f]
[ 19.517843] virtio-pci 0000:00:04.0: enabling device (0000 -> 0001)
[ 19.535957] PCI Interrupt Link [LNKD] enabled at IRQ 11
[ 19.536507] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver
[ 19.537528] virtio_mem virtio0: start address: 0x100000000
[ 19.538094] virtio_mem virtio0: region size: 0x10000000
[ 19.538621] virtio_mem virtio0: device block size: 0x200000
[ 19.539186] virtio_mem virtio0: memory block size: 0x8000000
[ 19.539752] virtio_mem virtio0: subblock size: 0x400000
[ 19.540357] virtio_mem virtio0: plugged size: 0x0
[ 19.540834] virtio_mem virtio0: requested size: 0x0
[ 20.170441] virtio_mem virtio0: plugged size: 0x0
[ 20.170933] virtio_mem virtio0: requested size: 0x10000000
[ 20.172247] Built 1 zonelists, mobility grouping on. Total pages: 266012
[ 20.172955] Policy zone: Normal
/ # ls
[ 26.724565] ------------[ cut here ]------------
[ 26.725047] ata_piix 0000:00:01.1: DMA addr 0x000000010fc14000+49152 overflow (mask ffffffff, bus limit 0).
[ 26.726024] WARNING: CPU: 0 PID: 179 at /home/teawater/kernel/linux2/kernel/dma/direct.c:364 dma_direct_map_page+0x118/0x130
[ 26.727141] Modules linked in:
[ 26.727456] CPU: 0 PID: 179 Comm: ls Not tainted 5.6.0-rc5-next-20200311+ #9
[ 26.728163] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
[ 26.729305] RIP: 0010:dma_direct_map_page+0x118/0x130
[ 26.729825] Code: 8b 1f e8 3b 70 59 00 48 8d 4c 24 08 48 89 c6 4c 89 2c 24 4d 89 e1 49 89 e8 48 89 da 48 c7 c7 08 6c 34 82 31 c0 e8 d8 8e f7 ff <00
[ 26.731683] RSP: 0000:ffffc90000213838 EFLAGS: 00010082
[ 26.732205] RAX: 0000000000000000 RBX: ffff88803ebeb1b0 RCX: ffffffff82665148
[ 26.732913] RDX: 0000000000000001 RSI: 0000000000000092 RDI: 0000000000000046
[ 26.733621] RBP: 000000000000c000 R08: 00000000000001df R09: 00000000000001df
[ 26.734338] R10: 0000000000000000 R11: ffffc900002135a8 R12: 00000000ffffffff
[ 26.735054] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88803d55f5b0
[ 26.735772] FS: 00000000024e9880(0000) GS:ffff88803ec00000(0000) knlGS:0000000000000000
[ 26.736579] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 26.737162] CR2: 00000000005bfc7f CR3: 0000000107e12004 CR4: 0000000000360ef0
[ 26.737879] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 26.738591] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 26.739307] Call Trace:
[ 26.739564] dma_direct_map_sg+0x64/0xb0
[ 26.739969] ? ata_scsi_write_same_xlat+0x350/0x350
[ 26.740461] ata_qc_issue+0x214/0x260
[ 26.740839] ata_scsi_queuecmd+0x16a/0x490
[ 26.741255] scsi_queue_rq+0x679/0xa60
[ 26.741639] blk_mq_dispatch_rq_list+0x90/0x510
[ 26.742099] ? elv_rb_del+0x1f/0x30
[ 26.742456] ? deadline_remove_request+0x6a/0xb0
[ 26.742926] blk_mq_do_dispatch_sched+0x78/0x100
[ 26.743397] blk_mq_sched_dispatch_requests+0xf9/0x170
[ 26.743924] __blk_mq_run_hw_queue+0x7e/0x130
[ 26.744365] __blk_mq_delay_run_hw_queue+0x107/0x150
[ 26.744874] blk_mq_run_hw_queue+0x61/0x100
[ 26.745299] blk_mq_sched_insert_requests+0x71/0x110
[ 26.745798] blk_mq_flush_plug_list+0x14b/0x210
[ 26.746258] blk_flush_plug_list+0xbf/0xe0
[ 26.746675] blk_finish_plug+0x27/0x40
[ 26.747056] read_pages+0x7c/0x190
[ 26.747399] __do_page_cache_readahead+0x19c/0x1b0
[ 26.747886] filemap_fault+0x54e/0x9a0
[ 26.748268] ? alloc_set_pte+0x102/0x610
[ 26.748673] ? walk_component+0x64/0x2e0
[ 26.749072] ? filemap_map_pages+0xfa/0x3f0
[ 26.749498] ext4_filemap_fault+0x2c/0x3b
[ 26.749911] __do_fault+0x38/0xb0
[ 26.750251] __handle_mm_fault+0xd2a/0x16d0
[ 26.750678] handle_mm_fault+0xe2/0x1f0
[ 26.751069] do_page_fault+0x250/0x590
[ 26.751448] async_page_fault+0x34/0x40
[ 26.751841] RIP: 0033:0x5bfc7f
[ 26.752155] Code: Bad RIP value.
[ 26.752481] RSP: 002b:00007ffef0289cd8 EFLAGS: 00010246
[ 26.752999] RAX: 0000000000000001 RBX: 00007ffef028af81 RCX: 00007ffef028af84
[ 26.753715] RDX: 00007ffef0289e01 RSI: 00007ffef0289ea8 RDI: 0000000000000001
[ 26.754424] RBP: 00000000000000ac R08: 0000000000000001 R09: 0000000000000006
[ 26.755144] R10: 000000000089fc18 R11: 0000000000000246 R12: 00007ffef0289ea8
[ 26.755853] R13: 000000000043a5f0 R14: 0000000000000000 R15: 0000000000000000
[ 26.756560] ---[ end trace 23cc3e9021358587 ]---
[ 26.778034] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 26.778690] ata2.00: failed command: READ DMA
[ 26.779131] ata2.00: cmd c8/00:e8:92:ad:00/00:00:00:00:00/e0 tag 0 dma 118784 in
[ 26.779131] res 50/00:00:0a:80:03/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 26.780691] ata2.00: status: { DRDY }
[ 26.781603] ata2.00: configured for MWDMA2
[ 26.782034] sd 1:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[ 26.782958] sd 1:0:0:0: [sda] tag#0 Sense Key : Illegal Request [current]
[ 26.783646] sd 1:0:0:0: [sda] tag#0 Add. Sense: Unaligned write command
[ 26.784321] sd 1:0:0:0: [sda] tag#0 CDB: Read(10) 28 00 00 00 ad 92 00 00 e8 00
[ 26.785056] blk_update_request: I/O error, dev sda, sector 44434 op 0x0:(READ) flags 0x80700 phys_seg 3 prio class 0
[ 26.786118] ata2: EH complete
[ 26.810033] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 26.810690] ata2.00: failed command: READ DMA
[ 26.811133] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 26.811133] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 26.812681] ata2.00: status: { DRDY }
[ 26.813569] ata2.00: configured for MWDMA2
[ 26.813992] ata2: EH complete
[ 26.826031] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 26.826687] ata2.00: failed command: READ DMA
[ 26.827131] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 26.827131] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 26.828668] ata2.00: status: { DRDY }
[ 26.829552] ata2.00: configured for MWDMA2
[ 26.829972] ata2: EH complete
[ 26.842030] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 26.842686] ata2.00: failed command: READ DMA
[ 26.843127] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 26.843127] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 26.844656] ata2.00: status: { DRDY }
[ 26.845538] ata2.00: configured for MWDMA2
[ 26.845961] ata2: EH complete
[ 26.858030] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 26.858690] ata2.00: failed command: READ DMA
[ 26.859132] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 26.859132] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 26.860656] ata2.00: status: { DRDY }
[ 26.861542] ata2.00: configured for MWDMA2
[ 26.861960] ata2: EH complete
[ 26.874030] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 26.874693] ata2.00: failed command: READ DMA
[ 26.875131] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 26.875131] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 26.876675] ata2.00: status: { DRDY }
[ 26.877554] ata2.00: configured for MWDMA2
[ 26.877976] ata2: EH complete
[ 26.890030] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 26.890655] ata2.00: failed command: READ DMA
[ 26.891082] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 26.891082] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 26.892544] ata2.00: status: { DRDY }
[ 26.893408] ata2.00: configured for MWDMA2
[ 26.893812] sd 1:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[ 26.894698] sd 1:0:0:0: [sda] tag#0 Sense Key : Illegal Request [current]
[ 26.895356] sd 1:0:0:0: [sda] tag#0 Add. Sense: Unaligned write command
[ 26.895993] sd 1:0:0:0: [sda] tag#0 CDB: Read(10) 28 00 00 00 ad fa 00 00 08 00
[ 26.896693] blk_update_request: I/O error, dev sda, sector 44538 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 26.897668] ata2: EH complete
[ 26.922032] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 26.922652] ata2.00: failed command: READ DMA
[ 26.923080] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 26.923080] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 26.924538] ata2.00: status: { DRDY }
[ 26.925404] ata2.00: configured for MWDMA2
[ 26.925807] ata2: EH complete
[ 26.938031] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 26.938650] ata2.00: failed command: READ DMA
[ 26.939076] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 26.939076] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 26.940529] ata2.00: status: { DRDY }
[ 26.941391] ata2.00: configured for MWDMA2
[ 26.941793] ata2: EH complete
[ 26.954031] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 26.954652] ata2.00: failed command: READ DMA
[ 26.955079] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 26.955079] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 26.956536] ata2.00: status: { DRDY }
[ 26.957400] ata2.00: configured for MWDMA2
[ 26.957800] ata2: EH complete
[ 26.970031] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 26.970653] ata2.00: failed command: READ DMA
[ 26.971079] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 26.971079] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 26.972536] ata2.00: status: { DRDY }
[ 26.973402] ata2.00: configured for MWDMA2
[ 26.973804] ata2: EH complete
[ 26.986030] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 26.986654] ata2.00: failed command: READ DMA
[ 26.987082] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 26.987082] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 26.988541] ata2.00: status: { DRDY }
[ 26.989406] ata2.00: configured for MWDMA2
[ 26.989807] ata2: EH complete
[ 27.002031] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 27.002653] ata2.00: failed command: READ DMA
[ 27.003083] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 27.003083] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 27.004541] ata2.00: status: { DRDY }
[ 27.005404] ata2.00: configured for MWDMA2
[ 27.005806] sd 1:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[ 27.006688] sd 1:0:0:0: [sda] tag#0 Sense Key : Illegal Request [current]
[ 27.007346] sd 1:0:0:0: [sda] tag#0 Add. Sense: Unaligned write command
[ 27.007982] sd 1:0:0:0: [sda] tag#0 CDB: Read(10) 28 00 00 00 ad fa 00 00 08 00
[ 27.008680] blk_update_request: I/O error, dev sda, sector 44538 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 27.009649] ata2: EH complete
Bus error
I cannot reproduce this warning with set file as nvdimm with following command.
sudo /home/teawater/qemu/qemu/x86_64-softmmu/qemu-system-x86_64 -machine pc,accel=kvm,kernel_irqchip,nvdimm -no-reboot -nographic -kernel /home/teawater/kernel/bk2/arch/x86/boot/bzImage -append "console=ttyS0 root=/dev/pmem0 swiotlb=noforce" -m 1g,slots=1,maxmem=2G -smp 1 -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/home/teawater/old.img,size=268435456 -monitor unix:/home/teawater/qemu/m,server,nowait
Best,
Hui
> 2020年5月7日 18:31，David Hildenbrand <david@redhat.com> 写道：
>
> This series is based on latest linux-next. The patches are located at:
> https://github.com/davidhildenbrand/linux.git virtio-mem-v3
>
> Patch #1 - #10 where contained in v2 and only contain minor modifications
> (mostly smaller fixes). The remaining patches are new and contain smaller
> optimizations.
>
> Details about virtio-mem can be found in the cover letter of v2 [1]. A
> basic QEMU implementation was posted yesterday [2].
>
> [1] https://lkml.kernel.org/r/20200311171422.10484-1-david@redhat.com
> [2] https://lkml.kernel.org/r/20200506094948.76388-1-david@redhat.com
>
> v2 -> v3:
> - "virtio-mem: Paravirtualized memory hotplug"
> -- Include "linux/slab.h" to fix build issues
> -- Remember the "region_size", helpful for patch #11
> -- Minor simplifaction in virtio_mem_overlaps_range()
> -- Use notifier_from_errno() instead of notifier_to_errno() in notifier
> -- More reliable check for added memory when unloading the driver
> - "virtio-mem: Allow to specify an ACPI PXM as nid"
> -- Also print the nid
> - Added patch #11-#15
>
> Cc: Sebastien Boeuf <sebastien.boeuf@intel.com>
> Cc: Samuel Ortiz <samuel.ortiz@intel.com>
> Cc: Robert Bradford <robert.bradford@intel.com>
> Cc: Luiz Capitulino <lcapitulino@redhat.com>
> Cc: Pankaj Gupta <pankaj.gupta.linux@gmail.com>
> Cc: teawater <teawaterz@linux.alibaba.com>
> Cc: Igor Mammedov <imammedo@redhat.com>
> Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>
>
> David Hildenbrand (15):
> virtio-mem: Paravirtualized memory hotplug
> virtio-mem: Allow to specify an ACPI PXM as nid
> virtio-mem: Paravirtualized memory hotunplug part 1
> virtio-mem: Paravirtualized memory hotunplug part 2
> mm: Allow to offline unmovable PageOffline() pages via
> MEM_GOING_OFFLINE
> virtio-mem: Allow to offline partially unplugged memory blocks
> mm/memory_hotplug: Introduce offline_and_remove_memory()
> virtio-mem: Offline and remove completely unplugged memory blocks
> virtio-mem: Better retry handling
> MAINTAINERS: Add myself as virtio-mem maintainer
> virtio-mem: Add parent resource for all added "System RAM"
> virtio-mem: Drop manual check for already present memory
> virtio-mem: Unplug subblocks right-to-left
> virtio-mem: Use -ETXTBSY as error code if the device is busy
> virtio-mem: Try to unplug the complete online memory block first
>
> MAINTAINERS | 7 +
> drivers/acpi/numa/srat.c | 1 +
> drivers/virtio/Kconfig | 17 +
> drivers/virtio/Makefile | 1 +
> drivers/virtio/virtio_mem.c | 1962 +++++++++++++++++++++++++++++++
> include/linux/memory_hotplug.h | 1 +
> include/linux/page-flags.h | 10 +
> include/uapi/linux/virtio_ids.h | 1 +
> include/uapi/linux/virtio_mem.h | 208 ++++
> mm/memory_hotplug.c | 81 +-
> mm/page_alloc.c | 26 +
> mm/page_isolation.c | 9 +
> 12 files changed, 2314 insertions(+), 10 deletions(-)
> create mode 100644 drivers/virtio/virtio_mem.c
> create mode 100644 include/uapi/linux/virtio_mem.h
>
> --
> 2.25.3
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 37+ messages in thread* Re: [PATCH v3 00/15] virtio-mem: paravirtualized memory
2020-05-14 6:44 ` teawater
@ 2020-05-14 8:48 ` David Hildenbrand
2020-05-14 10:02 ` teawater
0 siblings, 1 reply; 37+ messages in thread
From: David Hildenbrand @ 2020-05-14 8:48 UTC (permalink / raw)
To: teawater
Cc: LKML, Linux MM, virtio-dev, Linux Virtualization, kvm,
Michal Hocko, Andrew Morton, Michael S . Tsirkin, Sebastien Boeuf,
Samuel Ortiz, Robert Bradford, Luiz Capitulino, Pankaj Gupta,
Igor Mammedov, Dr . David Alan Gilbert, Alexander Duyck,
Alexander Potapenko, Anshuman Khandual, Anthony Yznaga,
Dan Williams, Dave Young
On 14.05.20 08:44, teawater wrote:
> Hi David,
>
> I got a kernel warning with v2 and v3.
Hi Hui,
thanks for playing with the latest versions. Surprisingly, I can
reproduce even by hotplugging a DIMM instead as well - that's good, so
it's not related to virtio-mem, lol. Seems to be some QEMU setup issue
with older machine types.
Can you switch to a newer qemu machine version, especially
pc-i440fx-5.0? Both, hotplugging DIMMs and virtio-mem works for me with
that QEMU machine just fine.
What also seems to make it work with pc-i440fx-2.1, is giving the
machine 4G of initial memory (-m 4g,slots=10,maxmem=5G).
Cheers!
> // start a QEMU that is get from https://github.com/davidhildenbrand/qemu/tree/virtio-mem-v2 and setup a file as a ide disk.
> /home/teawater/qemu/qemu/x86_64-softmmu/qemu-system-x86_64 -machine pc-i440fx-2.1,accel=kvm,usb=off -cpu host -no-reboot -nographic -device ide-hd,drive=hd -drive if=none,id=hd,file=/home/teawater/old.img,format=raw -kernel /home/teawater/kernel/bk2/arch/x86/boot/bzImage -append "console=ttyS0 root=/dev/sda nokaslr swiotlb=noforce" -m 1g,slots=10,maxmem=2G -smp 1 -s -monitor unix:/home/teawater/qemu/m,server,nowait
>
> // Setup virtio-mem and plug 256m memory in qemu monitor:
> object_add memory-backend-ram,id=mem1,size=256m
> device_add virtio-mem-pci,id=vm0,memdev=mem1
> qom-set vm0 requested-size 256M
>
> // Go back to the terminal and access file system will got following kernel warning.
> [ 19.515549] pci 0000:00:04.0: [1af4:1015] type 00 class 0x00ff00
> [ 19.516227] pci 0000:00:04.0: reg 0x10: [io 0x0000-0x007f]
> [ 19.517196] pci 0000:00:04.0: BAR 0: assigned [io 0x1000-0x107f]
> [ 19.517843] virtio-pci 0000:00:04.0: enabling device (0000 -> 0001)
> [ 19.535957] PCI Interrupt Link [LNKD] enabled at IRQ 11
> [ 19.536507] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver
> [ 19.537528] virtio_mem virtio0: start address: 0x100000000
> [ 19.538094] virtio_mem virtio0: region size: 0x10000000
> [ 19.538621] virtio_mem virtio0: device block size: 0x200000
> [ 19.539186] virtio_mem virtio0: memory block size: 0x8000000
> [ 19.539752] virtio_mem virtio0: subblock size: 0x400000
> [ 19.540357] virtio_mem virtio0: plugged size: 0x0
> [ 19.540834] virtio_mem virtio0: requested size: 0x0
> [ 20.170441] virtio_mem virtio0: plugged size: 0x0
> [ 20.170933] virtio_mem virtio0: requested size: 0x10000000
> [ 20.172247] Built 1 zonelists, mobility grouping on. Total pages: 266012
> [ 20.172955] Policy zone: Normal
>
> / # ls
> [ 26.724565] ------------[ cut here ]------------
> [ 26.725047] ata_piix 0000:00:01.1: DMA addr 0x000000010fc14000+49152 overflow (mask ffffffff, bus limit 0).
> [ 26.726024] WARNING: CPU: 0 PID: 179 at /home/teawater/kernel/linux2/kernel/dma/direct.c:364 dma_direct_map_page+0x118/0x130
> [ 26.727141] Modules linked in:
> [ 26.727456] CPU: 0 PID: 179 Comm: ls Not tainted 5.6.0-rc5-next-20200311+ #9
> [ 26.728163] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
> [ 26.729305] RIP: 0010:dma_direct_map_page+0x118/0x130
> [ 26.729825] Code: 8b 1f e8 3b 70 59 00 48 8d 4c 24 08 48 89 c6 4c 89 2c 24 4d 89 e1 49 89 e8 48 89 da 48 c7 c7 08 6c 34 82 31 c0 e8 d8 8e f7 ff <00
> [ 26.731683] RSP: 0000:ffffc90000213838 EFLAGS: 00010082
> [ 26.732205] RAX: 0000000000000000 RBX: ffff88803ebeb1b0 RCX: ffffffff82665148
> [ 26.732913] RDX: 0000000000000001 RSI: 0000000000000092 RDI: 0000000000000046
> [ 26.733621] RBP: 000000000000c000 R08: 00000000000001df R09: 00000000000001df
> [ 26.734338] R10: 0000000000000000 R11: ffffc900002135a8 R12: 00000000ffffffff
> [ 26.735054] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88803d55f5b0
> [ 26.735772] FS: 00000000024e9880(0000) GS:ffff88803ec00000(0000) knlGS:0000000000000000
> [ 26.736579] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 26.737162] CR2: 00000000005bfc7f CR3: 0000000107e12004 CR4: 0000000000360ef0
> [ 26.737879] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 26.738591] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> [ 26.739307] Call Trace:
> [ 26.739564] dma_direct_map_sg+0x64/0xb0
> [ 26.739969] ? ata_scsi_write_same_xlat+0x350/0x350
> [ 26.740461] ata_qc_issue+0x214/0x260
> [ 26.740839] ata_scsi_queuecmd+0x16a/0x490
> [ 26.741255] scsi_queue_rq+0x679/0xa60
> [ 26.741639] blk_mq_dispatch_rq_list+0x90/0x510
> [ 26.742099] ? elv_rb_del+0x1f/0x30
> [ 26.742456] ? deadline_remove_request+0x6a/0xb0
> [ 26.742926] blk_mq_do_dispatch_sched+0x78/0x100
> [ 26.743397] blk_mq_sched_dispatch_requests+0xf9/0x170
> [ 26.743924] __blk_mq_run_hw_queue+0x7e/0x130
> [ 26.744365] __blk_mq_delay_run_hw_queue+0x107/0x150
> [ 26.744874] blk_mq_run_hw_queue+0x61/0x100
> [ 26.745299] blk_mq_sched_insert_requests+0x71/0x110
> [ 26.745798] blk_mq_flush_plug_list+0x14b/0x210
> [ 26.746258] blk_flush_plug_list+0xbf/0xe0
> [ 26.746675] blk_finish_plug+0x27/0x40
> [ 26.747056] read_pages+0x7c/0x190
> [ 26.747399] __do_page_cache_readahead+0x19c/0x1b0
> [ 26.747886] filemap_fault+0x54e/0x9a0
> [ 26.748268] ? alloc_set_pte+0x102/0x610
> [ 26.748673] ? walk_component+0x64/0x2e0
> [ 26.749072] ? filemap_map_pages+0xfa/0x3f0
> [ 26.749498] ext4_filemap_fault+0x2c/0x3b
> [ 26.749911] __do_fault+0x38/0xb0
> [ 26.750251] __handle_mm_fault+0xd2a/0x16d0
> [ 26.750678] handle_mm_fault+0xe2/0x1f0
> [ 26.751069] do_page_fault+0x250/0x590
> [ 26.751448] async_page_fault+0x34/0x40
> [ 26.751841] RIP: 0033:0x5bfc7f
> [ 26.752155] Code: Bad RIP value.
> [ 26.752481] RSP: 002b:00007ffef0289cd8 EFLAGS: 00010246
> [ 26.752999] RAX: 0000000000000001 RBX: 00007ffef028af81 RCX: 00007ffef028af84
> [ 26.753715] RDX: 00007ffef0289e01 RSI: 00007ffef0289ea8 RDI: 0000000000000001
> [ 26.754424] RBP: 00000000000000ac R08: 0000000000000001 R09: 0000000000000006
> [ 26.755144] R10: 000000000089fc18 R11: 0000000000000246 R12: 00007ffef0289ea8
> [ 26.755853] R13: 000000000043a5f0 R14: 0000000000000000 R15: 0000000000000000
> [ 26.756560] ---[ end trace 23cc3e9021358587 ]---
> [ 26.778034] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
> [ 26.778690] ata2.00: failed command: READ DMA
> [ 26.779131] ata2.00: cmd c8/00:e8:92:ad:00/00:00:00:00:00/e0 tag 0 dma 118784 in
> [ 26.779131] res 50/00:00:0a:80:03/00:00:00:00:00/a0 Emask 0x40 (internal error)
> [ 26.780691] ata2.00: status: { DRDY }
> [ 26.781603] ata2.00: configured for MWDMA2
> [ 26.782034] sd 1:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
> [ 26.782958] sd 1:0:0:0: [sda] tag#0 Sense Key : Illegal Request [current]
> [ 26.783646] sd 1:0:0:0: [sda] tag#0 Add. Sense: Unaligned write command
> [ 26.784321] sd 1:0:0:0: [sda] tag#0 CDB: Read(10) 28 00 00 00 ad 92 00 00 e8 00
> [ 26.785056] blk_update_request: I/O error, dev sda, sector 44434 op 0x0:(READ) flags 0x80700 phys_seg 3 prio class 0
> [ 26.786118] ata2: EH complete
> [ 26.810033] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
> [ 26.810690] ata2.00: failed command: READ DMA
> [ 26.811133] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
> [ 26.811133] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
> [ 26.812681] ata2.00: status: { DRDY }
> [ 26.813569] ata2.00: configured for MWDMA2
> [ 26.813992] ata2: EH complete
> [ 26.826031] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
> [ 26.826687] ata2.00: failed command: READ DMA
> [ 26.827131] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
> [ 26.827131] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
> [ 26.828668] ata2.00: status: { DRDY }
> [ 26.829552] ata2.00: configured for MWDMA2
> [ 26.829972] ata2: EH complete
> [ 26.842030] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
> [ 26.842686] ata2.00: failed command: READ DMA
> [ 26.843127] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
> [ 26.843127] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
> [ 26.844656] ata2.00: status: { DRDY }
> [ 26.845538] ata2.00: configured for MWDMA2
> [ 26.845961] ata2: EH complete
> [ 26.858030] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
> [ 26.858690] ata2.00: failed command: READ DMA
> [ 26.859132] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
> [ 26.859132] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
> [ 26.860656] ata2.00: status: { DRDY }
> [ 26.861542] ata2.00: configured for MWDMA2
> [ 26.861960] ata2: EH complete
> [ 26.874030] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
> [ 26.874693] ata2.00: failed command: READ DMA
> [ 26.875131] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
> [ 26.875131] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
> [ 26.876675] ata2.00: status: { DRDY }
> [ 26.877554] ata2.00: configured for MWDMA2
> [ 26.877976] ata2: EH complete
> [ 26.890030] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
> [ 26.890655] ata2.00: failed command: READ DMA
> [ 26.891082] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
> [ 26.891082] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
> [ 26.892544] ata2.00: status: { DRDY }
> [ 26.893408] ata2.00: configured for MWDMA2
> [ 26.893812] sd 1:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
> [ 26.894698] sd 1:0:0:0: [sda] tag#0 Sense Key : Illegal Request [current]
> [ 26.895356] sd 1:0:0:0: [sda] tag#0 Add. Sense: Unaligned write command
> [ 26.895993] sd 1:0:0:0: [sda] tag#0 CDB: Read(10) 28 00 00 00 ad fa 00 00 08 00
> [ 26.896693] blk_update_request: I/O error, dev sda, sector 44538 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> [ 26.897668] ata2: EH complete
> [ 26.922032] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
> [ 26.922652] ata2.00: failed command: READ DMA
> [ 26.923080] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
> [ 26.923080] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
> [ 26.924538] ata2.00: status: { DRDY }
> [ 26.925404] ata2.00: configured for MWDMA2
> [ 26.925807] ata2: EH complete
> [ 26.938031] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
> [ 26.938650] ata2.00: failed command: READ DMA
> [ 26.939076] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
> [ 26.939076] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
> [ 26.940529] ata2.00: status: { DRDY }
> [ 26.941391] ata2.00: configured for MWDMA2
> [ 26.941793] ata2: EH complete
> [ 26.954031] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
> [ 26.954652] ata2.00: failed command: READ DMA
> [ 26.955079] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
> [ 26.955079] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
> [ 26.956536] ata2.00: status: { DRDY }
> [ 26.957400] ata2.00: configured for MWDMA2
> [ 26.957800] ata2: EH complete
> [ 26.970031] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
> [ 26.970653] ata2.00: failed command: READ DMA
> [ 26.971079] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
> [ 26.971079] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
> [ 26.972536] ata2.00: status: { DRDY }
> [ 26.973402] ata2.00: configured for MWDMA2
> [ 26.973804] ata2: EH complete
> [ 26.986030] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
> [ 26.986654] ata2.00: failed command: READ DMA
> [ 26.987082] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
> [ 26.987082] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
> [ 26.988541] ata2.00: status: { DRDY }
> [ 26.989406] ata2.00: configured for MWDMA2
> [ 26.989807] ata2: EH complete
> [ 27.002031] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
> [ 27.002653] ata2.00: failed command: READ DMA
> [ 27.003083] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
> [ 27.003083] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
> [ 27.004541] ata2.00: status: { DRDY }
> [ 27.005404] ata2.00: configured for MWDMA2
> [ 27.005806] sd 1:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
> [ 27.006688] sd 1:0:0:0: [sda] tag#0 Sense Key : Illegal Request [current]
> [ 27.007346] sd 1:0:0:0: [sda] tag#0 Add. Sense: Unaligned write command
> [ 27.007982] sd 1:0:0:0: [sda] tag#0 CDB: Read(10) 28 00 00 00 ad fa 00 00 08 00
> [ 27.008680] blk_update_request: I/O error, dev sda, sector 44538 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> [ 27.009649] ata2: EH complete
> Bus error
>
> I cannot reproduce this warning with set file as nvdimm with following command.
> sudo /home/teawater/qemu/qemu/x86_64-softmmu/qemu-system-x86_64 -machine pc,accel=kvm,kernel_irqchip,nvdimm -no-reboot -nographic -kernel /home/teawater/kernel/bk2/arch/x86/boot/bzImage -append "console=ttyS0 root=/dev/pmem0 swiotlb=noforce" -m 1g,slots=1,maxmem=2G -smp 1 -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/home/teawater/old.img,size=268435456 -monitor unix:/home/teawater/qemu/m,server,nowait
>
> Best,
> Hui
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 37+ messages in thread* Re: [PATCH v3 00/15] virtio-mem: paravirtualized memory
2020-05-14 8:48 ` David Hildenbrand
@ 2020-05-14 10:02 ` teawater
2020-05-14 10:12 ` David Hildenbrand
0 siblings, 1 reply; 37+ messages in thread
From: teawater @ 2020-05-14 10:02 UTC (permalink / raw)
To: David Hildenbrand
Cc: LKML, Linux MM, virtio-dev, Linux Virtualization, kvm,
Michal Hocko, Andrew Morton, Michael S . Tsirkin, Sebastien Boeuf,
Samuel Ortiz, Robert Bradford, Luiz Capitulino, Pankaj Gupta,
Igor Mammedov, Dr . David Alan Gilbert, Alexander Duyck,
Alexander Potapenko, Anshuman Khandual, Anthony Yznaga,
Dan Williams, Dave Young, Jason
> 2020年5月14日 16:48，David Hildenbrand <david@redhat.com> 写道：
>
> On 14.05.20 08:44, teawater wrote:
>> Hi David,
>>
>> I got a kernel warning with v2 and v3.
>
> Hi Hui,
>
> thanks for playing with the latest versions. Surprisingly, I can
> reproduce even by hotplugging a DIMM instead as well - that's good, so
> it's not related to virtio-mem, lol. Seems to be some QEMU setup issue
> with older machine types.
>
> Can you switch to a newer qemu machine version, especially
> pc-i440fx-5.0? Both, hotplugging DIMMs and virtio-mem works for me with
> that QEMU machine just fine.
I still could reproduce this issue with pc-i440fx-5.0 or pc. Did I miss anything?
>
> What also seems to make it work with pc-i440fx-2.1, is giving the
> machine 4G of initial memory (-m 4g,slots=10,maxmem=5G).
After I changed command, I got error when guest kernel boot:
sudo /home/teawater/qemu/qemu/x86_64-softmmu/qemu-system-x86_64 -machine pc,accel=kvm,usb=off -cpu host -no-reboot -nographic -device ide-hd,drive=hd -drive if=none,id=hd,file=/home/teawater/old.img,format=raw -kernel /home/teawater/kernel/bk2/arch/x86/boot/bzImage -append "console=ttyS0 root=/dev/sda nokaslr swiotlb=noforce" -m 4g,slots=10,maxmem=5G -smp 1 -s -monitor unix:/home/teawater/qemu/m,server,nowait
SeaBIOS (version rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org)
iPXE (http://ipxe.org) 00:03.0 CA00 PCI2.10 PnP PMM+BFF905B0+BFEF05B0 CA00
Booting from ROM..
[ 0.000000] Linux version 5.7.0-rc4-next-20200507+ (teawater@iZbp1bv6xzjxt9vmytkdnmZ) (gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39), GNU ld version0
[ 0.000000] Command line: console=ttyS0 root=/dev/sda nokaslr swiotlb=noforce
[ 0.000000] KERNEL supported cpus:
[ 0.000000] Intel GenuineIntel
[ 0.000000] AMD AuthenticAMD
[ 0.000000] Hygon HygonGenuine
[ 0.000000] Centaur CentaurHauls
[ 0.000000] zhaoxin Shanghai
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x008: 'MPX bounds registers'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x010: 'MPX CSR'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'
[ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
[ 0.000000] x86/fpu: xstate_offset[3]: 832, xstate_sizes[3]: 64
[ 0.000000] x86/fpu: xstate_offset[4]: 896, xstate_sizes[4]: 64
[ 0.000000] x86/fpu: xstate_offset[5]: 960, xstate_sizes[5]: 64
[ 0.000000] x86/fpu: xstate_offset[6]: 1024, xstate_sizes[6]: 512
[ 0.000000] x86/fpu: xstate_offset[7]: 1536, xstate_sizes[7]: 1024
[ 0.000000] x86/fpu: Enabled xstate features 0xff, context size is 2560 bytes, using 'compacted' format.
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffdffff] usable
[ 0.000000] BIOS-e820: [mem 0x00000000bffe0000-0x00000000bfffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013fffffff] usable
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] SMBIOS 2.8 present.
[ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
[ 0.000000] Hypervisor detected: KVM
[ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
[ 0.000000] kvm-clock: cpu 0, msr 2f12001, primary cpu clock
[ 0.000000] kvm-clock: using sched offset of 227993437 cycles
[ 0.000002] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[ 0.000003] tsc: Detected 2499.998 MHz processor
[ 0.001618] last_pfn = 0x140000 max_arch_pfn = 0x400000000
[ 0.001659] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT
[ 0.001666] last_pfn = 0xbffe0 max_arch_pfn = 0x400000000
[ 0.005474] found SMP MP-table at [mem 0x000f5ab0-0x000f5abf]
[ 0.005649] check: Scanning 1 areas for low memory corruption
[ 0.005676] Using GB pages for direct mapping
[ 0.006325] ACPI: Early table checksum verification disabled
[ 0.006328] ACPI: RSDP 0x00000000000F58B0 000014 (v00 BOCHS )
[ 0.006330] ACPI: RSDT 0x00000000BFFE1F1E 000038 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001)
[ 0.006334] ACPI: FACP 0x00000000BFFE1CF2 000074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001)
[ 0.006339] ACPI: DSDT 0x00000000BFFE0040 001CB2 (v01 BOCHS BXPCDSDT 00000001 BXPC 00000001)
[ 0.006341] ACPI: FACS 0x00000000BFFE0000 000040
[ 0.006343] ACPI: APIC 0x00000000BFFE1D66 000078 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001)
[ 0.006345] ACPI: HPET 0x00000000BFFE1DDE 000038 (v01 BOCHS BXPCHPET 00000001 BXPC 00000001)
[ 0.006347] ACPI: SRAT 0x00000000BFFE1E16 0000E0 (v01 BOCHS BXPCSRAT 00000001 BXPC 00000001)
[ 0.006349] ACPI: WAET 0x00000000BFFE1EF6 000028 (v01 BOCHS BXPCWAET 00000001 BXPC 00000001)
[ 0.006421] SRAT: PXM 0 -> APIC 0x00 -> Node 0
[ 0.006424] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x0009ffff]
[ 0.006426] ACPI: SRAT: Node 0 PXM 0 [mem 0x00100000-0xbfffffff]
[ 0.006428] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x13fffffff]
[ 0.006429] ACPI: SRAT: Node 0 PXM 0 [mem 0x140000000-0x3ffffffff] hotplug
[ 0.006433] NUMA: Node 0 [mem 0x00000000-0x0009ffff] + [mem 0x00100000-0xbfffffff] -> [mem 0x00000000-0xbfffffff]
[ 0.006435] NUMA: Node 0 [mem 0x00000000-0xbfffffff] + [mem 0x100000000-0x13fffffff] -> [mem 0x00000000-0x13fffffff]
[ 0.006449] NODE_DATA(0) allocated [mem 0x13ffd4000-0x13fffdfff]
[ 0.007109] Zone ranges:
[ 0.007110] DMA [mem 0x0000000000001000-0x0000000000ffffff]
[ 0.007111] DMA32 [mem 0x0000000001000000-0x00000000ffffffff]
[ 0.007112] Normal [mem 0x0000000100000000-0x000000013fffffff]
[ 0.007113] Device empty
[ 0.007114] Movable zone start for each node
[ 0.007116] Early memory node ranges
[ 0.007117] node 0: [mem 0x0000000000001000-0x000000000009efff]
[ 0.007119] node 0: [mem 0x0000000000100000-0x00000000bffdffff]
[ 0.007119] node 0: [mem 0x0000000100000000-0x000000013fffffff]
[ 0.007880] Zeroed struct page in unavailable ranges: 130 pages
[ 0.007881] Initmem setup node 0 [mem 0x0000000000001000-0x000000013fffffff]
[ 0.023595] ACPI: PM-Timer IO Port: 0x608
[ 0.023611] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
[ 0.023643] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23
[ 0.023646] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[ 0.023647] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
[ 0.023648] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[ 0.023649] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
[ 0.023650] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
[ 0.023656] Using ACPI (MADT) for SMP configuration information
[ 0.023657] ACPI: HPET id: 0x8086a201 base: 0xfed00000
[ 0.023663] smpboot: Allowing 1 CPUs, 0 hotplug CPUs
[ 0.023674] KVM setup pv remote TLB flush
[ 0.023690] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]
[ 0.023692] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x0009ffff]
[ 0.023694] PM: hibernation: Registered nosave memory: [mem 0x000a0000-0x000effff]
[ 0.023695] PM: hibernation: Registered nosave memory: [mem 0x000f0000-0x000fffff]
[ 0.023698] PM: hibernation: Registered nosave memory: [mem 0xbffe0000-0xbfffffff]
[ 0.023699] PM: hibernation: Registered nosave memory: [mem 0xc0000000-0xfeffbfff]
[ 0.023701] PM: hibernation: Registered nosave memory: [mem 0xfeffc000-0xfeffffff]
[ 0.023702] PM: hibernation: Registered nosave memory: [mem 0xff000000-0xfffbffff]
[ 0.023703] PM: hibernation: Registered nosave memory: [mem 0xfffc0000-0xffffffff]
[ 0.023706] [mem 0xc0000000-0xfeffbfff] available for PCI devices
[ 0.023708] Booting paravirtualized kernel on KVM
[ 0.023711] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
[ 0.028274] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1
[ 0.028748] percpu: Embedded 55 pages/cpu s188376 r8192 d28712 u2097152
[ 0.028778] KVM setup async PF for cpu 0
[ 0.028784] kvm-stealtime: cpu 0, msr 13bc19580
[ 0.028791] Built 1 zonelists, mobility grouping on. Total pages: 1032041
[ 0.028793] Policy zone: Normal
[ 0.028794] Kernel command line: console=ttyS0 root=/dev/sda nokaslr swiotlb=noforce
[ 0.029720] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[ 0.030154] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[ 0.030188] mem auto-init: stack:off, heap alloc:off, heap free:off
[ 0.037244] Memory: 4086692K/4193784K available (14339K kernel code, 2474K rwdata, 4860K rodata, 2408K init, 3324K bss, 107092K reserved, 0K cma-r)
[ 0.037640] random: get_random_u64 called from cache_random_seq_create+0x74/0x120 with crng_init=0
[ 0.037687] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[ 0.037696] ftrace: allocating 43405 entries in 170 pages
[ 0.050874] ftrace: allocated 170 pages with 4 groups
[ 0.051295] rcu: Hierarchical RCU implementation.
[ 0.051296] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=1.
[ 0.051297] Trampoline variant of Tasks RCU enabled.
[ 0.051298] Rude variant of Tasks RCU enabled.
[ 0.051299] Tracing variant of Tasks RCU enabled.
[ 0.051299] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[ 0.051300] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
[ 0.054124] NR_IRQS: 524544, nr_irqs: 256, preallocated irqs: 16
[ 0.061561] Console: colour VGA+ 80x25
[ 0.142370] printk: console [ttyS0] enabled
[ 0.142809] ACPI: Core revision 20200326
[ 0.143320] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns
[ 0.144321] APIC: Switch to symmetric I/O mode setup
[ 0.144988] x2apic enabled
[ 0.145454] Switched APIC routing to physical x2apic.
[ 0.145969] KVM setup pv IPIs
[ 0.147050] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[ 0.147674] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x240937b9988, max_idle_ns: 440795218083 ns
[ 0.148748] Calibrating delay loop (skipped) preset value.. 4999.99 BogoMIPS (lpj=9999992)
[ 0.149579] pid_max: default: 32768 minimum: 301
[ 0.150058] LSM: Security Framework initializing
[ 0.150536] Yama: becoming mindful.
[ 0.150912] AppArmor: AppArmor initialized
[ 0.151331] TOMOYO Linux initialized
[ 0.151726] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[ 0.152754] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[ 0.153769] x86/cpu: User Mode Instruction Prevention (UMIP) activated
[ 0.154479] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0
[ 0.155015] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0
[ 0.155619] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
[ 0.156469] Spectre V2 : Mitigation: Enhanced IBRS
[ 0.156749] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
[ 0.157531] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier
[ 0.158326] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp
[ 0.159236] TAA: Vulnerable: Clear CPU buffers attempted, no microcode
[ 0.163546] Freeing SMP alternatives memory: 40K
[ 0.165628] smpboot: CPU0: Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz (family: 0x6, model: 0x55, stepping: 0x7)
[ 0.166697] Performance Events: Skylake events, Intel PMU driver.
[ 0.167301] ... version: 2
[ 0.167684] ... bit width: 48
[ 0.168081] ... generic registers: 4
[ 0.168466] ... value mask: 0000ffffffffffff
[ 0.168746] ... max period: 000000007fffffff
[ 0.168746] ... fixed-purpose events: 3
[ 0.168749] ... event mask: 000000070000000f
[ 0.169286] rcu: Hierarchical SRCU implementation.
[ 0.170873] smp: Bringing up secondary CPUs ...
[ 0.171317] smp: Brought up 1 node, 1 CPU
[ 0.171705] smpboot: Max logical packages: 1
[ 0.172117] smpboot: Total of 1 processors activated (4999.99 BogoMIPS)
[ 0.172930] devtmpfs: initialized
[ 0.173284] x86/mm: Memory block size: 128MB
[ 0.173995] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[ 0.174923] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.175607] pinctrl core: initialized pinctrl subsystem
[ 0.176208] PM: RTC time: 10:00:50, date: 2020-05-14
[ 0.176688] thermal_sys: Registered thermal governor 'fair_share'
[ 0.176689] thermal_sys: Registered thermal governor 'bang_bang'
[ 0.176753] thermal_sys: Registered thermal governor 'step_wise'
[ 0.177332] thermal_sys: Registered thermal governor 'user_space'
[ 0.177982] NET: Registered protocol family 16
[ 0.179042] audit: initializing netlink subsys (disabled)
[ 0.179577] audit: type=2000 audit(1589450451.021:1): state=initialized audit_enabled=0 res=1
[ 0.180450] cpuidle: using governor ladder
[ 0.180753] cpuidle: using governor menu
[ 0.181162] ACPI: bus type PCI registered
[ 0.181548] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[ 0.182236] PCI: Using configuration type 1 for base access
[ 0.183818] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[ 0.184470] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[ 0.185279] ACPI: Added _OSI(Module Device)
[ 0.185693] ACPI: Added _OSI(Processor Device)
[ 0.186119] ACPI: Added _OSI(3.0 _SCP Extensions)
[ 0.186572] ACPI: Added _OSI(Processor Aggregator Device)
[ 0.187089] ACPI: Added _OSI(Linux-Dell-Video)
[ 0.187516] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio)
[ 0.188027] ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics)
[ 0.189526] ACPI: 1 ACPI AML tables successfully acquired and loaded
[ 0.190963] ACPI: Interpreter enabled
[ 0.191333] ACPI: (supports S0 S3 S4 S5)
[ 0.191716] ACPI: Using IOAPIC for interrupt routing
[ 0.192199] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[ 0.192891] ACPI: Enabled 3 GPEs in block 00 to 0F
[ 0.196692] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[ 0.196758] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3]
[ 0.197500] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
[ 0.199151] acpiphp: Slot [3] registered
[ 0.199546] acpiphp: Slot [4] registered
[ 0.199941] acpiphp: Slot [5] registered
[ 0.200333] acpiphp: Slot [6] registered
[ 0.200729] acpiphp: Slot [7] registered
[ 0.200770] acpiphp: Slot [8] registered
[ 0.201160] acpiphp: Slot [9] registered
[ 0.201555] acpiphp: Slot [10] registered
[ 0.201952] acpiphp: Slot [11] registered
[ 0.202353] acpiphp: Slot [12] registered
[ 0.202750] acpiphp: Slot [13] registered
[ 0.203151] acpiphp: Slot [14] registered
[ 0.203553] acpiphp: Slot [15] registered
[ 0.203952] acpiphp: Slot [16] registered
[ 0.204355] acpiphp: Slot [17] registered
[ 0.204756] acpiphp: Slot [18] registered
[ 0.205155] acpiphp: Slot [19] registered
[ 0.205559] acpiphp: Slot [20] registered
[ 0.205956] acpiphp: Slot [21] registered
[ 0.206358] acpiphp: Slot [22] registered
[ 0.206756] acpiphp: Slot [23] registered
[ 0.207154] acpiphp: Slot [24] registered
[ 0.207557] acpiphp: Slot [25] registered
[ 0.207957] acpiphp: Slot [26] registered
[ 0.208360] acpiphp: Slot [27] registered
[ 0.208756] acpiphp: Slot [28] registered
[ 0.209154] acpiphp: Slot [29] registered
[ 0.209558] acpiphp: Slot [30] registered
[ 0.209955] acpiphp: Slot [31] registered
[ 0.210351] PCI host bridge to bus 0000:00
[ 0.210742] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]
[ 0.211390] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]
[ 0.212033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[ 0.212745] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]
[ 0.212749] pci_bus 0000:00: root bus resource [mem 0x400000000-0x47fffffff window]
[ 0.213475] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 0.214026] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000
[ 0.214916] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100
[ 0.215901] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180
[ 0.218196] pci 0000:00:01.1: reg 0x20: [io 0xc040-0xc04f]
[ 0.219476] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7]
[ 0.220155] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6]
[ 0.220750] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177]
[ 0.221430] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376]
[ 0.222181] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000
[ 0.223058] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI
[ 0.223752] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB
[ 0.224604] pci 0000:00:02.0: [1234:1111] type 00 class 0x030000
[ 0.225627] pci 0000:00:02.0: reg 0x10: [mem 0xfd000000-0xfdffffff pref]
[ 0.227840] pci 0000:00:02.0: reg 0x18: [mem 0xfebf0000-0xfebf0fff]
[ 0.230963] pci 0000:00:02.0: reg 0x30: [mem 0xfebe0000-0xfebeffff pref]
[ 0.231766] pci 0000:00:03.0: [8086:100e] type 00 class 0x020000
[ 0.233184] pci 0000:00:03.0: reg 0x10: [mem 0xfebc0000-0xfebdffff]
[ 0.234628] pci 0000:00:03.0: reg 0x14: [io 0xc000-0xc03f]
[ 0.239497] pci 0000:00:03.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref]
[ 0.240716] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)
[ 0.240833] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
[ 0.241449] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
[ 0.242063] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)
[ 0.242642] ACPI: PCI Interrupt Link [LNKS] (IRQs *9)
[ 0.243547] iommu: Default domain type: Translated
[ 0.244050] pci 0000:00:02.0: vgaarb: setting as boot VGA device
[ 0.244631] pci 0000:00:02.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
[ 0.244750] pci 0000:00:02.0: vgaarb: bridge control possible
[ 0.245300] vgaarb: loaded
[ 0.245684] SCSI subsystem initialized
[ 0.246080] ACPI: bus type USB registered
[ 0.246485] usbcore: registered new interface driver usbfs
[ 0.247015] usbcore: registered new interface driver hub
[ 0.247532] usbcore: registered new device driver usb
[ 0.248053] EDAC MC: Ver: 3.0.0
[ 0.248488] PCI: Using ACPI for IRQ routing
[ 0.248889] NetLabel: Initializing
[ 0.249220] NetLabel: domain hash size = 128
[ 0.249642] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO
[ 0.250192] NetLabel: unlabeled traffic allowed by default
[ 0.250793] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
[ 0.251268] hpet0: 3 comparators, 64-bit 100.000000 MHz counter
[ 0.255814] clocksource: Switched to clocksource kvm-clock
[ 0.263083] VFS: Disk quotas dquot_6.6.0
[ 0.263479] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ 0.264215] AppArmor: AppArmor Filesystem Enabled
[ 0.264683] pnp: PnP ACPI init
[ 0.265570] pnp: PnP ACPI: found 6 devices
[ 0.271088] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[ 0.271958] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]
[ 0.272551] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]
[ 0.273150] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
[ 0.273805] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window]
[ 0.274463] pci_bus 0000:00: resource 8 [mem 0x400000000-0x47fffffff window]
[ 0.275178] NET: Registered protocol family 2
[ 0.275726] tcp_listen_portaddr_hash hash table entries: 2048 (order: 3, 32768 bytes, linear)
[ 0.276548] TCP established hash table entries: 32768 (order: 6, 262144 bytes, linear)
[ 0.277341] TCP bind hash table entries: 32768 (order: 7, 524288 bytes, linear)
[ 0.278089] TCP: Hash tables configured (established 32768 bind 32768)
[ 0.279086] UDP hash table entries: 2048 (order: 4, 65536 bytes, linear)
[ 0.279734] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes, linear)
[ 0.280450] NET: Registered protocol family 1
[ 0.280896] pci 0000:00:01.0: PIIX3: Enabling Passive Release
[ 0.281455] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
[ 0.282028] pci 0000:00:01.0: Activating ISA DMA hang workarounds
[ 0.282652] pci 0000:00:02.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[ 0.283463] PCI: CLS 0 bytes, default 64
[ 0.283865] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[ 0.284483] software IO TLB: mapped [mem 0xbffdf000-0xbffdf800] (0MB)
[ 0.285199] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x240937b9988, max_idle_ns: 440795218083 ns
[ 0.286167] check: Scanning for low memory corruption every 60 seconds
[ 0.286998] Initialise system trusted keyrings
[ 0.287462] workingset: timestamp_bits=36 max_order=20 bucket_order=0
[ 0.288814] zbud: loaded
[ 0.289224] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[ 0.289871] fuse: init (API version 7.31)
[ 0.297407] Key type asymmetric registered
[ 0.297815] Asymmetric key parser 'x509' registered
[ 0.298289] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
[ 0.299015] io scheduler mq-deadline registered
[ 0.299548] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
[ 0.300278] ACPI: Power Button [PWRF]
[ 0.300786] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
[ 0.301606] 00:05: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[ 0.302840] Linux agpgart interface v0.103
[ 0.304481] loop: module loaded
[ 0.305665] scsi host0: ata_piix
[ 0.306042] scsi host1: ata_piix
[ 0.306385] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc040 irq 14
[ 0.307031] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc048 irq 15
[ 0.307930] libphy: Fixed MDIO Bus: probed
[ 0.308331] tun: Universal TUN/TAP device driver, 1.6
[ 0.308847] PPP generic driver version 2.4.2
[ 0.309281] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 0.309907] ehci-pci: EHCI PCI platform driver
[ 0.310340] ehci-platform: EHCI generic platform driver
[ 0.310846] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[ 0.311437] ohci-pci: OHCI PCI platform driver
[ 0.311869] ohci-platform: OHCI generic platform driver
[ 0.312375] uhci_hcd: USB Universal Host Controller Interface driver
[ 0.313181] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
[ 0.314458] serio: i8042 KBD port at 0x60,0x64 irq 1
[ 0.314934] serio: i8042 AUX port at 0x60,0x64 irq 12
[ 0.315476] mousedev: PS/2 mouse device common for all mice
[ 0.316182] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1
[ 0.317177] rtc_cmos 00:00: RTC can wake from S4
[ 0.317895] rtc_cmos 00:00: registered as rtc0
[ 0.318368] rtc_cmos 00:00: setting system clock to 2020-05-14T10:00:50 UTC (1589450450)
[ 0.319151] rtc_cmos 00:00: alarms up to one day, y3k, 114 bytes nvram, hpet irqs
[ 0.319867] i2c /dev entries driver
[ 0.320229] device-mapper: uevent: version 1.0.3
[ 0.320704] device-mapper: ioctl: 4.42.0-ioctl (2020-02-27) initialised: dm-devel@redhat.com
[ 0.321515] intel_pstate: CPU model not supported
[ 0.321972] ledtrig-cpu: registered to indicate activity on CPUs
[ 0.322567] drop_monitor: Initializing network drop monitor service
[ 0.323263] NET: Registered protocol family 10
[ 0.323814] Segment Routing with IPv6
[ 0.324183] NET: Registered protocol family 17
[ 0.324631] Key type dns_resolver registered
[ 0.325105] IPI shorthand broadcast: enabled
[ 0.325522] sched_clock: Marking stable (238560739, 86481164)->(334226795, -9184892)
[ 0.326296] registered taskstats version 1
[ 0.326696] Loading compiled-in X.509 certificates
[ 0.328051] Loaded X.509 cert 'Build time autogenerated kernel key: bd0fe972797988f0782b694de4de56b1ff6c4bc3'
[ 0.329022] zswap: loaded using pool lzo/zbud
[ 0.329489] Key type ._fscrypt registered
[ 0.329877] Key type .fscrypt registered
[ 0.330259] Key type fscrypt-provisioning registered
[ 0.330851] Key type big_key registered
[ 0.331274] Key type encrypted registered
[ 0.331662] AppArmor: AppArmor sha1 policy hashing enabled
[ 0.332190] ima: No TPM chip found, activating TPM-bypass!
[ 0.332716] ima: Allocated hash algorithm: sha1
[ 0.333171] ima: No architecture policies found
[ 0.333646] evm: Initialising EVM extended attributes:
[ 0.334144] evm: security.selinux
[ 0.334463] evm: security.SMACK64
[ 0.334782] evm: security.SMACK64EXEC
[ 0.335138] evm: security.SMACK64TRANSMUTE
[ 0.335531] evm: security.SMACK64MMAP
[ 0.335884] evm: security.apparmor
[ 0.336215] evm: security.ima
[ 0.336504] evm: security.capability
[ 0.336855] evm: HMAC attrs: 0x1
[ 0.337354] PM: Magic number: 4:67:22
[ 0.473227] ata2.00: ATA-7: QEMU HARDDISK, 2.5+, max UDMA/100
[ 0.473785] ata2.00: 524288 sectors, multi 16: LBA48
[ 0.474659] scsi 1:0:0:0: Direct-Access ATA QEMU HARDDISK 2.5+ PQ: 0 ANSI: 5
[ 0.475536] sd 1:0:0:0: [sda] 524288 512-byte logical blocks: (268 MB/256 MiB)
[ 0.476237] sd 1:0:0:0: [sda] Write Protect is off
[ 0.476704] sd 1:0:0:0: Attached scsi generic sg0 type 0
[ 0.477239] sd 1:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 0.478205] ------------[ cut here ]------------
[ 0.478652] ata_piix 0000:00:01.1: DMA addr 0x0000000139e4f000+4096 overflow (mask ffffffff, bus limit 0).
[ 0.479580] WARNING: CPU: 0 PID: 6 at /home/teawater/kernel/linux2/kernel/dma/direct.c:400 dma_direct_map_page+0x118/0x130
[ 0.480630] Modules linked in:
[ 0.480933] CPU: 0 PID: 6 Comm: kworker/0:0H Not tainted 5.7.0-rc4-next-20200507+ #10
[ 0.481681] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
[ 0.482770] Workqueue: kblockd blk_mq_run_work_fn
[ 0.483230] RIP: 0010:dma_direct_map_page+0x118/0x130
[ 0.483715] Code: 8b 1f e8 6b 48 5b 00 48 8d 4c 24 08 48 89 c6 4c 89 2c 24 4d 89 e1 49 89 e8 48 89 da 48 c7 c7 48 e4 34 82 31 c0 e8 18 77 f7 ff <00
[ 0.485484] RSP: 0000:ffffc9000003bbe8 EFLAGS: 00010082
[ 0.485984] RAX: 0000000000000000 RBX: ffff88813b3db310 RCX: ffffffff82667968
[ 0.486663] RDX: 0000000000000001 RSI: 0000000000000096 RDI: 0000000000000046
[ 0.487342] RBP: 0000000000001000 R08: 00000000000001bb R09: 00000000000001bb
[ 0.488017] R10: 0000000000000000 R11: ffffc9000003b958 R12: 00000000ffffffff
[ 0.488699] R13: 0000000000000000 R14: 0000000000000000 R15: ffff888139e272a8
[ 0.489379] FS: 0000000000000000(0000) GS:ffff88813bc00000(0000) knlGS:0000000000000000
[ 0.490148] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 0.490698] CR2: 0000000000000000 CR3: 000000000260a001 CR4: 0000000000360ef0
[ 0.491377] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 0.492057] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 0.492736] Call Trace:
[ 0.492985] dma_direct_map_sg+0x64/0xb0
[ 0.493369] ? ata_scsi_write_same_xlat+0x350/0x350
[ 0.493837] ata_qc_issue+0x214/0x260
[ 0.494199] ? ata_scsi_write_same_xlat+0x350/0x350
[ 0.494665] __ata_scsi_queuecmd+0x190/0x410
[ 0.495077] ata_scsi_queuecmd+0x40/0x80
[ 0.495455] scsi_queue_rq+0x697/0xa90
[ 0.495819] blk_mq_dispatch_rq_list+0x90/0x590
[ 0.496259] ? elv_rb_del+0x1f/0x30
[ 0.496597] ? deadline_remove_request+0x6a/0xb0
[ 0.497042] blk_mq_do_dispatch_sched+0xe9/0x130
[ 0.497488] __blk_mq_sched_dispatch_requests+0xd8/0x160
[ 0.497996] blk_mq_sched_dispatch_requests+0x30/0x60
[ 0.498483] __blk_mq_run_hw_queue+0x7e/0x130
[ 0.498904] process_one_work+0x166/0x370
[ 0.499296] worker_thread+0x49/0x3e0
[ 0.499650] kthread+0xf8/0x130
[ 0.499957] ? max_active_store+0x80/0x80
[ 0.500347] ? kthread_bind+0x10/0x10
[ 0.500701] ret_from_fork+0x1f/0x30
[ 0.501048] ---[ end trace cfd92832ca2f7781 ]---
[ 0.524773] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 0.525404] ata2.00: failed command: READ DMA
[ 0.525824] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 0.525824] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 0.527275] ata2.00: status: { DRDY }
[ 0.528160] ata2.00: configured for MWDMA2
[ 0.528558] ata2: EH complete
[ 0.540770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 0.541399] ata2.00: failed command: READ DMA
[ 0.541818] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 0.541818] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 0.543265] ata2.00: status: { DRDY }
[ 0.544129] ata2.00: configured for MWDMA2
[ 0.544525] ata2: EH complete
[ 0.556771] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 0.557389] ata2.00: failed command: READ DMA
[ 0.557810] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 0.557810] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 0.559255] ata2.00: status: { DRDY }
[ 0.560115] ata2.00: configured for MWDMA2
[ 0.560509] ata2: EH complete
[ 0.572770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 0.573398] ata2.00: failed command: READ DMA
[ 0.573818] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 0.573818] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 0.575264] ata2.00: status: { DRDY }
[ 0.576125] ata2.00: configured for MWDMA2
[ 0.576522] ata2: EH complete
[ 0.588772] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 0.589409] ata2.00: failed command: READ DMA
[ 0.589831] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 0.589831] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 0.591275] ata2.00: status: { DRDY }
[ 0.592136] ata2.00: configured for MWDMA2
[ 0.592532] ata2: EH complete
[ 0.604771] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 0.605400] ata2.00: failed command: READ DMA
[ 0.605821] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 0.605821] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 0.607273] ata2.00: status: { DRDY }
[ 0.608135] ata2.00: configured for MWDMA2
[ 0.608535] sd 1:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[ 0.609415] sd 1:0:0:0: [sda] tag#0 Sense Key : Illegal Request [current]
[ 0.610074] sd 1:0:0:0: [sda] tag#0 Add. Sense: Unaligned write command
[ 0.610703] sd 1:0:0:0: [sda] tag#0 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
[ 0.611402] blk_update_request: I/O error, dev sda, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 0.612336] Buffer I/O error on dev sda, logical block 0, async page read
[ 0.612987] ata2: EH complete
[ 0.636771] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 0.637389] ata2.00: failed command: READ DMA
[ 0.637808] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 0.637808] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 0.639254] ata2.00: status: { DRDY }
[ 0.640112] ata2.00: configured for MWDMA2
[ 0.640511] ata2: EH complete
[ 0.652772] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 0.653398] ata2.00: failed command: READ DMA
[ 0.653818] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 0.653818] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 0.655267] ata2.00: status: { DRDY }
[ 0.656126] ata2.00: configured for MWDMA2
[ 0.656523] ata2: EH complete
[ 0.668771] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 0.669396] ata2.00: failed command: READ DMA
[ 0.669815] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 0.669815] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 0.671264] ata2.00: status: { DRDY }
[ 0.672124] ata2.00: configured for MWDMA2
[ 0.672521] ata2: EH complete
[ 0.684770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 0.685390] ata2.00: failed command: READ DMA
[ 0.685807] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 0.685807] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 0.687252] ata2.00: status: { DRDY }
[ 0.688109] ata2.00: configured for MWDMA2
[ 0.688506] ata2: EH complete
[ 0.700769] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 0.701388] ata2.00: failed command: READ DMA
[ 0.701805] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 0.701805] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 0.703249] ata2.00: status: { DRDY }
[ 0.704107] ata2.00: configured for MWDMA2
[ 0.704502] ata2: EH complete
[ 0.716769] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 0.717389] ata2.00: failed command: READ DMA
[ 0.717806] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 0.717806] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 0.719251] ata2.00: status: { DRDY }
[ 0.720110] ata2.00: configured for MWDMA2
[ 0.720507] sd 1:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[ 0.721389] sd 1:0:0:0: [sda] tag#0 Sense Key : Illegal Request [current]
[ 0.722042] sd 1:0:0:0: [sda] tag#0 Add. Sense: Unaligned write command
[ 0.722676] sd 1:0:0:0: [sda] tag#0 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
[ 0.723374] blk_update_request: I/O error, dev sda, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 0.724308] Buffer I/O error on dev sda, logical block 0, async page read
[ 0.724959] ata2: EH complete
[ 0.748770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 0.749390] ata2.00: failed command: READ DMA
[ 0.749810] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 0.749810] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 0.751271] ata2.00: status: { DRDY }
[ 0.752134] ata2.00: configured for MWDMA2
[ 0.752533] ata2: EH complete
[ 0.764771] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 0.765388] ata2.00: failed command: READ DMA
[ 0.765809] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 0.765809] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 0.767257] ata2.00: status: { DRDY }
[ 0.768115] ata2.00: configured for MWDMA2
[ 0.768511] ata2: EH complete
[ 0.780773] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 0.781390] ata2.00: failed command: READ DMA
[ 0.781813] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 0.781813] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 0.783263] ata2.00: status: { DRDY }
[ 0.784123] ata2.00: configured for MWDMA2
[ 0.784520] ata2: EH complete
[ 0.796771] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 0.797394] ata2.00: failed command: READ DMA
[ 0.797814] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 0.797814] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 0.799262] ata2.00: status: { DRDY }
[ 0.800122] ata2.00: configured for MWDMA2
[ 0.800519] ata2: EH complete
[ 0.812770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 0.813395] ata2.00: failed command: READ DMA
[ 0.813815] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 0.813815] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 0.815263] ata2.00: status: { DRDY }
[ 0.816121] ata2.00: configured for MWDMA2
[ 0.816517] ata2: EH complete
[ 0.828770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 0.829395] ata2.00: failed command: READ DMA
[ 0.829814] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 0.829814] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 0.831267] ata2.00: status: { DRDY }
[ 0.832126] ata2.00: configured for MWDMA2
[ 0.832524] sd 1:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[ 0.833403] sd 1:0:0:0: [sda] tag#0 Sense Key : Illegal Request [current]
[ 0.834060] sd 1:0:0:0: [sda] tag#0 Add. Sense: Unaligned write command
[ 0.834692] sd 1:0:0:0: [sda] tag#0 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
[ 0.835389] blk_update_request: I/O error, dev sda, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 0.836319] Buffer I/O error on dev sda, logical block 0, async page read
[ 0.836970] ata2: EH complete
[ 0.837267] ldm_validate_partition_table(): Disk read failed.
[ 0.860772] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 0.861401] ata2.00: failed command: READ DMA
[ 0.861822] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 0.861822] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 0.863269] ata2.00: status: { DRDY }
[ 0.864132] ata2.00: configured for MWDMA2
[ 0.864528] ata2: EH complete
[ 0.876771] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 0.877397] ata2.00: failed command: READ DMA
[ 0.877817] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 0.877817] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 0.879266] ata2.00: status: { DRDY }
[ 0.880127] ata2.00: configured for MWDMA2
[ 0.880524] ata2: EH complete
[ 0.892773] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 0.893399] ata2.00: failed command: READ DMA
[ 0.893820] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 0.893820] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 0.895273] ata2.00: status: { DRDY }
[ 0.896133] ata2.00: configured for MWDMA2
[ 0.896531] ata2: EH complete
[ 0.908770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 0.909394] ata2.00: failed command: READ DMA
[ 0.909814] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 0.909814] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 0.911257] ata2.00: status: { DRDY }
[ 0.912116] ata2.00: configured for MWDMA2
[ 0.912512] ata2: EH complete
[ 0.924770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 0.925398] ata2.00: failed command: READ DMA
[ 0.925817] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 0.925817] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 0.927261] ata2.00: status: { DRDY }
[ 0.928118] ata2.00: configured for MWDMA2
[ 0.928515] ata2: EH complete
[ 0.940771] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 0.941387] ata2.00: failed command: READ DMA
[ 0.941805] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 0.941805] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 0.943250] ata2.00: status: { DRDY }
[ 0.944108] ata2.00: configured for MWDMA2
[ 0.944505] sd 1:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[ 0.945382] sd 1:0:0:0: [sda] tag#0 Sense Key : Illegal Request [current]
[ 0.946039] sd 1:0:0:0: [sda] tag#0 Add. Sense: Unaligned write command
[ 0.946671] sd 1:0:0:0: [sda] tag#0 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
[ 0.947370] blk_update_request: I/O error, dev sda, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 0.948302] Buffer I/O error on dev sda, logical block 0, async page read
[ 0.948956] ata2: EH complete
[ 0.972774] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 0.973395] ata2.00: failed command: READ DMA
[ 0.973812] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 0.973812] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 0.975262] ata2.00: status: { DRDY }
[ 0.976126] ata2.00: configured for MWDMA2
[ 0.976524] ata2: EH complete
[ 0.988770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 0.989396] ata2.00: failed command: READ DMA
[ 0.989815] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 0.989815] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 0.991257] ata2.00: status: { DRDY }
[ 0.992117] ata2.00: configured for MWDMA2
[ 0.992511] ata2: EH complete
[ 1.004769] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.005398] ata2.00: failed command: READ DMA
[ 1.005818] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.005818] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.007261] ata2.00: status: { DRDY }
[ 1.008122] ata2.00: configured for MWDMA2
[ 1.008518] ata2: EH complete
[ 1.020770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.021398] ata2.00: failed command: READ DMA
[ 1.021819] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.021819] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.023262] ata2.00: status: { DRDY }
[ 1.024123] ata2.00: configured for MWDMA2
[ 1.024519] ata2: EH complete
[ 1.036771] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.037396] ata2.00: failed command: READ DMA
[ 1.037816] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.037816] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.039262] ata2.00: status: { DRDY }
[ 1.040120] ata2.00: configured for MWDMA2
[ 1.040515] ata2: EH complete
[ 1.052772] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.053397] ata2.00: failed command: READ DMA
[ 1.053818] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.053818] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.055264] ata2.00: status: { DRDY }
[ 1.056125] ata2.00: configured for MWDMA2
[ 1.056525] sd 1:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[ 1.057404] sd 1:0:0:0: [sda] tag#0 Sense Key : Illegal Request [current]
[ 1.058066] sd 1:0:0:0: [sda] tag#0 Add. Sense: Unaligned write command
[ 1.058695] sd 1:0:0:0: [sda] tag#0 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
[ 1.059393] blk_update_request: I/O error, dev sda, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 1.060325] Buffer I/O error on dev sda, logical block 0, async page read
[ 1.060973] ata2: EH complete
[ 1.084771] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.085395] ata2.00: failed command: READ DMA
[ 1.085814] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.085814] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.087259] ata2.00: status: { DRDY }
[ 1.088119] ata2.00: configured for MWDMA2
[ 1.088514] ata2: EH complete
[ 1.100770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.101395] ata2.00: failed command: READ DMA
[ 1.101814] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.101814] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.103259] ata2.00: status: { DRDY }
[ 1.104118] ata2.00: configured for MWDMA2
[ 1.104515] ata2: EH complete
[ 1.116769] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.117388] ata2.00: failed command: READ DMA
[ 1.117805] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.117805] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.119248] ata2.00: status: { DRDY }
[ 1.120106] ata2.00: configured for MWDMA2
[ 1.120503] ata2: EH complete
[ 1.132770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.133390] ata2.00: failed command: READ DMA
[ 1.133807] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.133807] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.135253] ata2.00: status: { DRDY }
[ 1.136113] ata2.00: configured for MWDMA2
[ 1.136509] ata2: EH complete
[ 1.148770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.149389] ata2.00: failed command: READ DMA
[ 1.149806] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.149806] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.151248] ata2.00: status: { DRDY }
[ 1.152108] ata2.00: configured for MWDMA2
[ 1.152503] ata2: EH complete
[ 1.164770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.165395] ata2.00: failed command: READ DMA
[ 1.165814] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.165814] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.167258] ata2.00: status: { DRDY }
[ 1.168119] ata2.00: configured for MWDMA2
[ 1.168515] sd 1:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[ 1.169397] sd 1:0:0:0: [sda] tag#0 Sense Key : Illegal Request [current]
[ 1.170054] sd 1:0:0:0: [sda] tag#0 Add. Sense: Unaligned write command
[ 1.170685] sd 1:0:0:0: [sda] tag#0 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
[ 1.171382] blk_update_request: I/O error, dev sda, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 1.172315] Buffer I/O error on dev sda, logical block 0, async page read
[ 1.172965] ata2: EH complete
[ 1.196770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.197391] ata2.00: failed command: READ DMA
[ 1.197811] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.197811] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.199265] ata2.00: status: { DRDY }
[ 1.200128] ata2.00: configured for MWDMA2
[ 1.200525] ata2: EH complete
[ 1.212770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.213389] ata2.00: failed command: READ DMA
[ 1.213809] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.213809] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.215253] ata2.00: status: { DRDY }
[ 1.216114] ata2.00: configured for MWDMA2
[ 1.216510] ata2: EH complete
[ 1.228773] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.229398] ata2.00: failed command: READ DMA
[ 1.229819] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.229819] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.231273] ata2.00: status: { DRDY }
[ 1.232132] ata2.00: configured for MWDMA2
[ 1.232528] ata2: EH complete
[ 1.244773] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.245399] ata2.00: failed command: READ DMA
[ 1.245821] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.245821] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.247269] ata2.00: status: { DRDY }
[ 1.248128] ata2.00: configured for MWDMA2
[ 1.248525] ata2: EH complete
[ 1.260770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.261391] ata2.00: failed command: READ DMA
[ 1.261813] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.261813] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.263263] ata2.00: status: { DRDY }
[ 1.264124] ata2.00: configured for MWDMA2
[ 1.264520] ata2: EH complete
[ 1.276771] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.277389] ata2.00: failed command: READ DMA
[ 1.277809] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.277809] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.279253] ata2.00: status: { DRDY }
[ 1.280112] ata2.00: configured for MWDMA2
[ 1.280510] sd 1:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[ 1.281385] sd 1:0:0:0: [sda] tag#0 Sense Key : Illegal Request [current]
[ 1.282040] sd 1:0:0:0: [sda] tag#0 Add. Sense: Unaligned write command
[ 1.282672] sd 1:0:0:0: [sda] tag#0 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
[ 1.283371] blk_update_request: I/O error, dev sda, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 1.284305] Buffer I/O error on dev sda, logical block 0, async page read
[ 1.284960] ata2: EH complete
[ 1.285254] Dev sda: unable to read RDB block 0
[ 1.308770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.309404] ata2.00: failed command: READ DMA
[ 1.309825] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.309825] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.311278] ata2.00: status: { DRDY }
[ 1.312142] ata2.00: configured for MWDMA2
[ 1.312538] ata2: EH complete
[ 1.324770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.325398] ata2.00: failed command: READ DMA
[ 1.325817] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.325817] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.327269] ata2.00: status: { DRDY }
[ 1.328127] ata2.00: configured for MWDMA2
[ 1.328520] ata2: EH complete
[ 1.340770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.341396] ata2.00: failed command: READ DMA
[ 1.341813] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.341813] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.343258] ata2.00: status: { DRDY }
[ 1.344120] ata2.00: configured for MWDMA2
[ 1.344515] ata2: EH complete
[ 1.356771] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.357396] ata2.00: failed command: READ DMA
[ 1.357815] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.357815] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.359258] ata2.00: status: { DRDY }
[ 1.360118] ata2.00: configured for MWDMA2
[ 1.360514] ata2: EH complete
[ 1.372772] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.373399] ata2.00: failed command: READ DMA
[ 1.373822] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.373822] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.375275] ata2.00: status: { DRDY }
[ 1.376135] ata2.00: configured for MWDMA2
[ 1.376533] ata2: EH complete
[ 1.388773] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.389397] ata2.00: failed command: READ DMA
[ 1.389817] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.389817] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.391263] ata2.00: status: { DRDY }
[ 1.392122] ata2.00: configured for MWDMA2
[ 1.392522] sd 1:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[ 1.393400] sd 1:0:0:0: [sda] tag#0 Sense Key : Illegal Request [current]
[ 1.394057] sd 1:0:0:0: [sda] tag#0 Add. Sense: Unaligned write command
[ 1.394687] sd 1:0:0:0: [sda] tag#0 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
[ 1.395388] blk_update_request: I/O error, dev sda, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 1.396323] Buffer I/O error on dev sda, logical block 0, async page read
[ 1.396975] ata2: EH complete
[ 1.420771] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.421400] ata2.00: failed command: READ DMA
[ 1.421819] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.421819] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.423267] ata2.00: status: { DRDY }
[ 1.424124] ata2.00: configured for MWDMA2
[ 1.424521] ata2: EH complete
[ 1.436771] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.437395] ata2.00: failed command: READ DMA
[ 1.437813] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.437813] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.439261] ata2.00: status: { DRDY }
[ 1.440120] ata2.00: configured for MWDMA2
[ 1.440516] ata2: EH complete
[ 1.452770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.453399] ata2.00: failed command: READ DMA
[ 1.453816] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.453816] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.455259] ata2.00: status: { DRDY }
[ 1.456116] ata2.00: configured for MWDMA2
[ 1.456512] ata2: EH complete
[ 1.468769] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.469387] ata2.00: failed command: READ DMA
[ 1.469802] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.469802] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.471246] ata2.00: status: { DRDY }
[ 1.472105] ata2.00: configured for MWDMA2
[ 1.472501] ata2: EH complete
[ 1.484769] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.485403] ata2.00: failed command: READ DMA
[ 1.485822] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.485822] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.487269] ata2.00: status: { DRDY }
[ 1.488128] ata2.00: configured for MWDMA2
[ 1.488524] ata2: EH complete
[ 1.500770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.501389] ata2.00: failed command: READ DMA
[ 1.501807] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.501807] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.503252] ata2.00: status: { DRDY }
[ 1.504114] ata2.00: configured for MWDMA2
[ 1.504512] sd 1:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[ 1.505395] sd 1:0:0:0: [sda] tag#0 Sense Key : Illegal Request [current]
[ 1.506054] sd 1:0:0:0: [sda] tag#0 Add. Sense: Unaligned write command
[ 1.506685] sd 1:0:0:0: [sda] tag#0 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
[ 1.507383] blk_update_request: I/O error, dev sda, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 1.508319] Buffer I/O error on dev sda, logical block 0, async page read
[ 1.508972] ata2: EH complete
[ 1.532770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.533402] ata2.00: failed command: READ DMA
[ 1.533825] ata2.00: cmd c8/00:08:18:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.533825] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.535282] ata2.00: status: { DRDY }
[ 1.536143] ata2.00: configured for MWDMA2
[ 1.536542] ata2: EH complete
[ 1.548771] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.549394] ata2.00: failed command: READ DMA
[ 1.549817] ata2.00: cmd c8/00:08:18:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.549817] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.551272] ata2.00: status: { DRDY }
[ 1.552132] ata2.00: configured for MWDMA2
[ 1.552530] ata2: EH complete
[ 1.564771] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.565392] ata2.00: failed command: READ DMA
[ 1.565814] ata2.00: cmd c8/00:08:18:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.565814] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.567271] ata2.00: status: { DRDY }
[ 1.568129] ata2.00: configured for MWDMA2
[ 1.568530] ata2: EH complete
[ 1.580769] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.581390] ata2.00: failed command: READ DMA
[ 1.581812] ata2.00: cmd c8/00:08:18:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.581812] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.583257] ata2.00: status: { DRDY }
[ 1.584114] ata2.00: configured for MWDMA2
[ 1.584510] ata2: EH complete
[ 1.596770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.597389] ata2.00: failed command: READ DMA
[ 1.597809] ata2.00: cmd c8/00:08:18:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.597809] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.599260] ata2.00: status: { DRDY }
[ 1.600119] ata2.00: configured for MWDMA2
[ 1.600515] ata2: EH complete
[ 1.612770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.613394] ata2.00: failed command: READ DMA
[ 1.613814] ata2.00: cmd c8/00:08:18:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.613814] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.615276] ata2.00: status: { DRDY }
[ 1.616139] ata2.00: configured for MWDMA2
[ 1.616539] sd 1:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[ 1.617421] sd 1:0:0:0: [sda] tag#0 Sense Key : Illegal Request [current]
[ 1.618081] sd 1:0:0:0: [sda] tag#0 Add. Sense: Unaligned write command
[ 1.618714] sd 1:0:0:0: [sda] tag#0 CDB: Read(10) 28 00 00 00 00 18 00 00 08 00
[ 1.619418] blk_update_request: I/O error, dev sda, sector 24 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 1.620369] Buffer I/O error on dev sda, logical block 3, async page read
[ 1.621028] ata2: EH complete
[ 1.644770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.645397] ata2.00: failed command: READ DMA
[ 1.645819] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.645819] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.647272] ata2.00: status: { DRDY }
[ 1.648136] ata2.00: configured for MWDMA2
[ 1.648536] ata2: EH complete
[ 1.660769] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.661389] ata2.00: failed command: READ DMA
[ 1.661809] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.661809] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.663260] ata2.00: status: { DRDY }
[ 1.664119] ata2.00: configured for MWDMA2
[ 1.664516] ata2: EH complete
[ 1.676770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.677393] ata2.00: failed command: READ DMA
[ 1.677812] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.677812] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.679262] ata2.00: status: { DRDY }
[ 1.680120] ata2.00: configured for MWDMA2
[ 1.680517] ata2: EH complete
[ 1.692769] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.693390] ata2.00: failed command: READ DMA
[ 1.693810] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.693810] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.695265] ata2.00: status: { DRDY }
[ 1.696128] ata2.00: configured for MWDMA2
[ 1.696526] ata2: EH complete
[ 1.708769] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.709395] ata2.00: failed command: READ DMA
[ 1.709817] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.709817] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.711277] ata2.00: status: { DRDY }
[ 1.712138] ata2.00: configured for MWDMA2
[ 1.712535] ata2: EH complete
[ 1.724769] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.725393] ata2.00: failed command: READ DMA
[ 1.725814] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.725814] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.727265] ata2.00: status: { DRDY }
[ 1.728125] ata2.00: configured for MWDMA2
[ 1.728525] ata2: EH complete
[ 1.752770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.753396] ata2.00: failed command: READ DMA
[ 1.753815] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.753815] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.755272] ata2.00: status: { DRDY }
[ 1.756130] ata2.00: configured for MWDMA2
[ 1.756525] ata2: EH complete
[ 1.768769] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.769386] ata2.00: failed command: READ DMA
[ 1.769803] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.769803] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.771250] ata2.00: status: { DRDY }
[ 1.772108] ata2.00: configured for MWDMA2
[ 1.772503] ata2: EH complete
[ 1.784769] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.785390] ata2.00: failed command: READ DMA
[ 1.785812] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.785812] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.787264] ata2.00: status: { DRDY }
[ 1.788127] ata2.00: configured for MWDMA2
[ 1.788525] ata2: EH complete
[ 1.800769] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.801392] ata2.00: failed command: READ DMA
[ 1.801813] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.801813] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.803264] ata2.00: status: { DRDY }
[ 1.804126] ata2.00: configured for MWDMA2
[ 1.804522] ata2: EH complete
[ 1.816770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.817391] ata2.00: failed command: READ DMA
[ 1.817813] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.817813] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.819264] ata2.00: status: { DRDY }
[ 1.820126] ata2.00: configured for MWDMA2
[ 1.820523] ata2: EH complete
[ 1.832773] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.833398] ata2.00: failed command: READ DMA
[ 1.833819] ata2.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[ 1.833819] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.835275] ata2.00: status: { DRDY }
[ 1.836136] ata2.00: configured for MWDMA2
[ 1.836537] ata2: EH complete
[ 1.836833] sda: unable to read partition table
[ 1.837393] sd 1:0:0:0: [sda] Attached SCSI disk
[ 1.837855] md: Waiting for all devices to be available before autodetect
[ 1.838510] md: If you don't use raid, use raid=noautodetect
[ 1.839152] sda: detected capacity change from 0 to 268435456
[ 1.839759] sda: detected capacity change from 0 to 268435456
[ 1.840346] md: Autodetecting RAID arrays.
[ 1.840741] md: autorun ...
[ 1.841018] md: ... autorun DONE.
[ 1.864774] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.865398] ata2.00: failed command: READ DMA
[ 1.865821] ata2.00: cmd c8/00:02:02:00:00/00:00:00:00:00/e0 tag 0 dma 1024 in
[ 1.865821] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.867274] ata2.00: status: { DRDY }
[ 1.868146] ata2.00: configured for MWDMA2
[ 1.868546] ata2: EH complete
[ 1.880772] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.881393] ata2.00: failed command: READ DMA
[ 1.881816] ata2.00: cmd c8/00:02:02:00:00/00:00:00:00:00/e0 tag 0 dma 1024 in
[ 1.881816] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.883269] ata2.00: status: { DRDY }
[ 1.884130] ata2.00: configured for MWDMA2
[ 1.884531] ata2: EH complete
[ 1.896771] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.897392] ata2.00: failed command: READ DMA
[ 1.897816] ata2.00: cmd c8/00:02:02:00:00/00:00:00:00:00/e0 tag 0 dma 1024 in
[ 1.897816] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.899265] ata2.00: status: { DRDY }
[ 1.900125] ata2.00: configured for MWDMA2
[ 1.900524] ata2: EH complete
[ 1.912771] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.913395] ata2.00: failed command: READ DMA
[ 1.913821] ata2.00: cmd c8/00:02:02:00:00/00:00:00:00:00/e0 tag 0 dma 1024 in
[ 1.913821] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.915281] ata2.00: status: { DRDY }
[ 1.916144] ata2.00: configured for MWDMA2
[ 1.916544] ata2: EH complete
[ 1.928769] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.929392] ata2.00: failed command: READ DMA
[ 1.929814] ata2.00: cmd c8/00:02:02:00:00/00:00:00:00:00/e0 tag 0 dma 1024 in
[ 1.929814] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.931269] ata2.00: status: { DRDY }
[ 1.932129] ata2.00: configured for MWDMA2
[ 1.932529] ata2: EH complete
[ 1.944769] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.945402] ata2.00: failed command: READ DMA
[ 1.945826] ata2.00: cmd c8/00:02:02:00:00/00:00:00:00:00/e0 tag 0 dma 1024 in
[ 1.945826] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.947295] ata2.00: status: { DRDY }
[ 1.948156] ata2.00: configured for MWDMA2
[ 1.948561] ata2: EH complete
[ 1.948860] EXT4-fs (sda): unable to read superblock
[ 1.972772] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.973397] ata2.00: failed command: READ DMA
[ 1.973819] ata2.00: cmd c8/00:02:02:00:00/00:00:00:00:00/e0 tag 0 dma 1024 in
[ 1.973819] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.975278] ata2.00: status: { DRDY }
[ 1.976142] ata2.00: configured for MWDMA2
[ 1.976541] ata2: EH complete
[ 1.988771] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1.989390] ata2.00: failed command: READ DMA
[ 1.989812] ata2.00: cmd c8/00:02:02:00:00/00:00:00:00:00/e0 tag 0 dma 1024 in
[ 1.989812] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 1.991263] ata2.00: status: { DRDY }
[ 1.992126] ata2.00: configured for MWDMA2
[ 1.992526] ata2: EH complete
[ 2.004773] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 2.005394] ata2.00: failed command: READ DMA
[ 2.005817] ata2.00: cmd c8/00:02:02:00:00/00:00:00:00:00/e0 tag 0 dma 1024 in
[ 2.005817] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 2.007276] ata2.00: status: { DRDY }
[ 2.008141] ata2.00: configured for MWDMA2
[ 2.008542] ata2: EH complete
[ 2.020770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 2.021394] ata2.00: failed command: READ DMA
[ 2.021815] ata2.00: cmd c8/00:02:02:00:00/00:00:00:00:00/e0 tag 0 dma 1024 in
[ 2.021815] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 2.023273] ata2.00: status: { DRDY }
[ 2.024138] ata2.00: configured for MWDMA2
[ 2.024538] ata2: EH complete
[ 2.036771] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 2.037392] ata2.00: failed command: READ DMA
[ 2.037815] ata2.00: cmd c8/00:02:02:00:00/00:00:00:00:00/e0 tag 0 dma 1024 in
[ 2.037815] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 2.039272] ata2.00: status: { DRDY }
[ 2.040135] ata2.00: configured for MWDMA2
[ 2.040533] ata2: EH complete
[ 2.052771] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 2.053393] ata2.00: failed command: READ DMA
[ 2.053816] ata2.00: cmd c8/00:02:02:00:00/00:00:00:00:00/e0 tag 0 dma 1024 in
[ 2.053816] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 2.055273] ata2.00: status: { DRDY }
[ 2.056136] ata2.00: configured for MWDMA2
[ 2.056538] ata2: EH complete
[ 2.056834] EXT4-fs (sda): unable to read superblock
[ 2.080771] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 2.081394] ata2.00: failed command: READ DMA
[ 2.081815] ata2.00: cmd c8/00:02:02:00:00/00:00:00:00:00/e0 tag 0 dma 1024 in
[ 2.081815] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 2.083272] ata2.00: status: { DRDY }
[ 2.084137] ata2.00: configured for MWDMA2
[ 2.084536] ata2: EH complete
[ 2.096770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 2.097393] ata2.00: failed command: READ DMA
[ 2.097814] ata2.00: cmd c8/00:02:02:00:00/00:00:00:00:00/e0 tag 0 dma 1024 in
[ 2.097814] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 2.100505] ata2.00: status: { DRDY }
[ 2.101411] ata2.00: configured for MWDMA2
[ 2.101809] ata2: EH complete
[ 2.112771] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 2.113392] ata2.00: failed command: READ DMA
[ 2.113812] ata2.00: cmd c8/00:02:02:00:00/00:00:00:00:00/e0 tag 0 dma 1024 in
[ 2.113812] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 2.115270] ata2.00: status: { DRDY }
[ 2.116137] ata2.00: configured for MWDMA2
[ 2.116537] ata2: EH complete
[ 2.128770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 2.129395] ata2.00: failed command: READ DMA
[ 2.129819] ata2.00: cmd c8/00:02:02:00:00/00:00:00:00:00/e0 tag 0 dma 1024 in
[ 2.129819] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 2.131275] ata2.00: status: { DRDY }
[ 2.132138] ata2.00: configured for MWDMA2
[ 2.132538] ata2: EH complete
[ 2.144772] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 2.145403] ata2.00: failed command: READ DMA
[ 2.145825] ata2.00: cmd c8/00:02:02:00:00/00:00:00:00:00/e0 tag 0 dma 1024 in
[ 2.145825] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 2.147286] ata2.00: status: { DRDY }
[ 2.148149] ata2.00: configured for MWDMA2
[ 2.148549] ata2: EH complete
[ 2.160771] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 2.161400] ata2.00: failed command: READ DMA
[ 2.161822] ata2.00: cmd c8/00:02:02:00:00/00:00:00:00:00/e0 tag 0 dma 1024 in
[ 2.161822] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 2.163282] ata2.00: status: { DRDY }
[ 2.164146] ata2.00: configured for MWDMA2
[ 2.164548] ata2: EH complete
[ 2.164846] EXT4-fs (sda): unable to read superblock
[ 2.184771] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 2.185398] ata2.00: failed command: READ DMA
[ 2.185820] ata2.00: cmd c8/00:02:00:00:00/00:00:00:00:00/e0 tag 0 dma 1024 in
[ 2.185820] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 2.187280] ata2.00: status: { DRDY }
[ 2.188146] ata2.00: configured for MWDMA2
[ 2.188548] ata2: EH complete
[ 2.200770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 2.201397] ata2.00: failed command: READ DMA
[ 2.201820] ata2.00: cmd c8/00:02:00:00:00/00:00:00:00:00/e0 tag 0 dma 1024 in
[ 2.201820] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 2.203281] ata2.00: status: { DRDY }
[ 2.204144] ata2.00: configured for MWDMA2
[ 2.204543] ata2: EH complete
[ 2.216770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 2.217404] ata2.00: failed command: READ DMA
[ 2.217825] ata2.00: cmd c8/00:02:00:00:00/00:00:00:00:00/e0 tag 0 dma 1024 in
[ 2.217825] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 2.219290] ata2.00: status: { DRDY }
[ 2.220152] ata2.00: configured for MWDMA2
[ 2.220551] ata2: EH complete
[ 2.232770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 2.233396] ata2.00: failed command: READ DMA
[ 2.233819] ata2.00: cmd c8/00:02:00:00:00/00:00:00:00:00/e0 tag 0 dma 1024 in
[ 2.233819] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 2.235281] ata2.00: status: { DRDY }
[ 2.236145] ata2.00: configured for MWDMA2
[ 2.236545] ata2: EH complete
[ 2.248769] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 2.249396] ata2.00: failed command: READ DMA
[ 2.249818] ata2.00: cmd c8/00:02:00:00:00/00:00:00:00:00/e0 tag 0 dma 1024 in
[ 2.249818] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 2.251279] ata2.00: status: { DRDY }
[ 2.252145] ata2.00: configured for MWDMA2
[ 2.252543] ata2: EH complete
[ 2.264770] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 2.265393] ata2.00: failed command: READ DMA
[ 2.265815] ata2.00: cmd c8/00:02:00:00:00/00:00:00:00:00/e0 tag 0 dma 1024 in
[ 2.265815] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
[ 2.267271] ata2.00: status: { DRDY }
[ 2.268133] ata2.00: configured for MWDMA2
[ 2.268534] ata2: EH complete
[ 2.268833] SQUASHFS error: Failed to read block 0x0: -5
[ 2.269346] unable to read squashfs_super_block
[ 2.269785] VFS: Cannot open root device "sda" or unknown-block(8,0): error -5
[ 2.270481] Please append a correct "root=" boot option; here are the available partitions:
[ 2.271284] 0800 262144 sda
[ 2.271285] driver: sd
[ 2.271881] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(8,0)
[ 2.272673] CPU: 0 PID: 1 Comm: swapper/0 Tainted: G W 5.7.0-rc4-next-20200507+ #10
[ 2.273527] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
[ 2.274614] Call Trace:
[ 2.274861] dump_stack+0x6b/0x83
[ 2.275189] panic+0x102/0x2f3
[ 2.275491] mount_block_root+0x290/0x339
[ 2.275879] prepare_namespace+0x13d/0x16c
[ 2.276277] kernel_init_freeable+0x246/0x270
[ 2.276699] ? rdinit_setup+0x28/0x28
[ 2.277056] ? rest_init+0xb0/0xb0
[ 2.277387] kernel_init+0xa/0x110
[ 2.277718] ret_from_fork+0x1f/0x30
[ 2.278414] Kernel Offset: disabled
[ 2.278755] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(8,0) ]---
qemu-system-x86_64: terminating on signal 2
Best,
Hui
>
> Cheers!
>
>
>> // start a QEMU that is get from https://github.com/davidhildenbrand/qemu/tree/virtio-mem-v2 and setup a file as a ide disk.
>> /home/teawater/qemu/qemu/x86_64-softmmu/qemu-system-x86_64 -machine pc-i440fx-2.1,accel=kvm,usb=off -cpu host -no-reboot -nographic -device ide-hd,drive=hd -drive if=none,id=hd,file=/home/teawater/old.img,format=raw -kernel /home/teawater/kernel/bk2/arch/x86/boot/bzImage -append "console=ttyS0 root=/dev/sda nokaslr swiotlb=noforce" -m 1g,slots=10,maxmem=2G -smp 1 -s -monitor unix:/home/teawater/qemu/m,server,nowait
>>
>> // Setup virtio-mem and plug 256m memory in qemu monitor:
>> object_add memory-backend-ram,id=mem1,size=256m
>> device_add virtio-mem-pci,id=vm0,memdev=mem1
>> qom-set vm0 requested-size 256M
>>
>> // Go back to the terminal and access file system will got following kernel warning.
>> [ 19.515549] pci 0000:00:04.0: [1af4:1015] type 00 class 0x00ff00
>> [ 19.516227] pci 0000:00:04.0: reg 0x10: [io 0x0000-0x007f]
>> [ 19.517196] pci 0000:00:04.0: BAR 0: assigned [io 0x1000-0x107f]
>> [ 19.517843] virtio-pci 0000:00:04.0: enabling device (0000 -> 0001)
>> [ 19.535957] PCI Interrupt Link [LNKD] enabled at IRQ 11
>> [ 19.536507] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver
>> [ 19.537528] virtio_mem virtio0: start address: 0x100000000
>> [ 19.538094] virtio_mem virtio0: region size: 0x10000000
>> [ 19.538621] virtio_mem virtio0: device block size: 0x200000
>> [ 19.539186] virtio_mem virtio0: memory block size: 0x8000000
>> [ 19.539752] virtio_mem virtio0: subblock size: 0x400000
>> [ 19.540357] virtio_mem virtio0: plugged size: 0x0
>> [ 19.540834] virtio_mem virtio0: requested size: 0x0
>> [ 20.170441] virtio_mem virtio0: plugged size: 0x0
>> [ 20.170933] virtio_mem virtio0: requested size: 0x10000000
>> [ 20.172247] Built 1 zonelists, mobility grouping on. Total pages: 266012
>> [ 20.172955] Policy zone: Normal
>>
>> / # ls
>> [ 26.724565] ------------[ cut here ]------------
>> [ 26.725047] ata_piix 0000:00:01.1: DMA addr 0x000000010fc14000+49152 overflow (mask ffffffff, bus limit 0).
>> [ 26.726024] WARNING: CPU: 0 PID: 179 at /home/teawater/kernel/linux2/kernel/dma/direct.c:364 dma_direct_map_page+0x118/0x130
>> [ 26.727141] Modules linked in:
>> [ 26.727456] CPU: 0 PID: 179 Comm: ls Not tainted 5.6.0-rc5-next-20200311+ #9
>> [ 26.728163] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
>> [ 26.729305] RIP: 0010:dma_direct_map_page+0x118/0x130
>> [ 26.729825] Code: 8b 1f e8 3b 70 59 00 48 8d 4c 24 08 48 89 c6 4c 89 2c 24 4d 89 e1 49 89 e8 48 89 da 48 c7 c7 08 6c 34 82 31 c0 e8 d8 8e f7 ff <00
>> [ 26.731683] RSP: 0000:ffffc90000213838 EFLAGS: 00010082
>> [ 26.732205] RAX: 0000000000000000 RBX: ffff88803ebeb1b0 RCX: ffffffff82665148
>> [ 26.732913] RDX: 0000000000000001 RSI: 0000000000000092 RDI: 0000000000000046
>> [ 26.733621] RBP: 000000000000c000 R08: 00000000000001df R09: 00000000000001df
>> [ 26.734338] R10: 0000000000000000 R11: ffffc900002135a8 R12: 00000000ffffffff
>> [ 26.735054] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88803d55f5b0
>> [ 26.735772] FS: 00000000024e9880(0000) GS:ffff88803ec00000(0000) knlGS:0000000000000000
>> [ 26.736579] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [ 26.737162] CR2: 00000000005bfc7f CR3: 0000000107e12004 CR4: 0000000000360ef0
>> [ 26.737879] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> [ 26.738591] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>> [ 26.739307] Call Trace:
>> [ 26.739564] dma_direct_map_sg+0x64/0xb0
>> [ 26.739969] ? ata_scsi_write_same_xlat+0x350/0x350
>> [ 26.740461] ata_qc_issue+0x214/0x260
>> [ 26.740839] ata_scsi_queuecmd+0x16a/0x490
>> [ 26.741255] scsi_queue_rq+0x679/0xa60
>> [ 26.741639] blk_mq_dispatch_rq_list+0x90/0x510
>> [ 26.742099] ? elv_rb_del+0x1f/0x30
>> [ 26.742456] ? deadline_remove_request+0x6a/0xb0
>> [ 26.742926] blk_mq_do_dispatch_sched+0x78/0x100
>> [ 26.743397] blk_mq_sched_dispatch_requests+0xf9/0x170
>> [ 26.743924] __blk_mq_run_hw_queue+0x7e/0x130
>> [ 26.744365] __blk_mq_delay_run_hw_queue+0x107/0x150
>> [ 26.744874] blk_mq_run_hw_queue+0x61/0x100
>> [ 26.745299] blk_mq_sched_insert_requests+0x71/0x110
>> [ 26.745798] blk_mq_flush_plug_list+0x14b/0x210
>> [ 26.746258] blk_flush_plug_list+0xbf/0xe0
>> [ 26.746675] blk_finish_plug+0x27/0x40
>> [ 26.747056] read_pages+0x7c/0x190
>> [ 26.747399] __do_page_cache_readahead+0x19c/0x1b0
>> [ 26.747886] filemap_fault+0x54e/0x9a0
>> [ 26.748268] ? alloc_set_pte+0x102/0x610
>> [ 26.748673] ? walk_component+0x64/0x2e0
>> [ 26.749072] ? filemap_map_pages+0xfa/0x3f0
>> [ 26.749498] ext4_filemap_fault+0x2c/0x3b
>> [ 26.749911] __do_fault+0x38/0xb0
>> [ 26.750251] __handle_mm_fault+0xd2a/0x16d0
>> [ 26.750678] handle_mm_fault+0xe2/0x1f0
>> [ 26.751069] do_page_fault+0x250/0x590
>> [ 26.751448] async_page_fault+0x34/0x40
>> [ 26.751841] RIP: 0033:0x5bfc7f
>> [ 26.752155] Code: Bad RIP value.
>> [ 26.752481] RSP: 002b:00007ffef0289cd8 EFLAGS: 00010246
>> [ 26.752999] RAX: 0000000000000001 RBX: 00007ffef028af81 RCX: 00007ffef028af84
>> [ 26.753715] RDX: 00007ffef0289e01 RSI: 00007ffef0289ea8 RDI: 0000000000000001
>> [ 26.754424] RBP: 00000000000000ac R08: 0000000000000001 R09: 0000000000000006
>> [ 26.755144] R10: 000000000089fc18 R11: 0000000000000246 R12: 00007ffef0289ea8
>> [ 26.755853] R13: 000000000043a5f0 R14: 0000000000000000 R15: 0000000000000000
>> [ 26.756560] ---[ end trace 23cc3e9021358587 ]---
>> [ 26.778034] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
>> [ 26.778690] ata2.00: failed command: READ DMA
>> [ 26.779131] ata2.00: cmd c8/00:e8:92:ad:00/00:00:00:00:00/e0 tag 0 dma 118784 in
>> [ 26.779131] res 50/00:00:0a:80:03/00:00:00:00:00/a0 Emask 0x40 (internal error)
>> [ 26.780691] ata2.00: status: { DRDY }
>> [ 26.781603] ata2.00: configured for MWDMA2
>> [ 26.782034] sd 1:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
>> [ 26.782958] sd 1:0:0:0: [sda] tag#0 Sense Key : Illegal Request [current]
>> [ 26.783646] sd 1:0:0:0: [sda] tag#0 Add. Sense: Unaligned write command
>> [ 26.784321] sd 1:0:0:0: [sda] tag#0 CDB: Read(10) 28 00 00 00 ad 92 00 00 e8 00
>> [ 26.785056] blk_update_request: I/O error, dev sda, sector 44434 op 0x0:(READ) flags 0x80700 phys_seg 3 prio class 0
>> [ 26.786118] ata2: EH complete
>> [ 26.810033] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
>> [ 26.810690] ata2.00: failed command: READ DMA
>> [ 26.811133] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
>> [ 26.811133] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
>> [ 26.812681] ata2.00: status: { DRDY }
>> [ 26.813569] ata2.00: configured for MWDMA2
>> [ 26.813992] ata2: EH complete
>> [ 26.826031] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
>> [ 26.826687] ata2.00: failed command: READ DMA
>> [ 26.827131] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
>> [ 26.827131] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
>> [ 26.828668] ata2.00: status: { DRDY }
>> [ 26.829552] ata2.00: configured for MWDMA2
>> [ 26.829972] ata2: EH complete
>> [ 26.842030] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
>> [ 26.842686] ata2.00: failed command: READ DMA
>> [ 26.843127] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
>> [ 26.843127] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
>> [ 26.844656] ata2.00: status: { DRDY }
>> [ 26.845538] ata2.00: configured for MWDMA2
>> [ 26.845961] ata2: EH complete
>> [ 26.858030] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
>> [ 26.858690] ata2.00: failed command: READ DMA
>> [ 26.859132] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
>> [ 26.859132] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
>> [ 26.860656] ata2.00: status: { DRDY }
>> [ 26.861542] ata2.00: configured for MWDMA2
>> [ 26.861960] ata2: EH complete
>> [ 26.874030] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
>> [ 26.874693] ata2.00: failed command: READ DMA
>> [ 26.875131] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
>> [ 26.875131] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
>> [ 26.876675] ata2.00: status: { DRDY }
>> [ 26.877554] ata2.00: configured for MWDMA2
>> [ 26.877976] ata2: EH complete
>> [ 26.890030] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
>> [ 26.890655] ata2.00: failed command: READ DMA
>> [ 26.891082] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
>> [ 26.891082] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
>> [ 26.892544] ata2.00: status: { DRDY }
>> [ 26.893408] ata2.00: configured for MWDMA2
>> [ 26.893812] sd 1:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
>> [ 26.894698] sd 1:0:0:0: [sda] tag#0 Sense Key : Illegal Request [current]
>> [ 26.895356] sd 1:0:0:0: [sda] tag#0 Add. Sense: Unaligned write command
>> [ 26.895993] sd 1:0:0:0: [sda] tag#0 CDB: Read(10) 28 00 00 00 ad fa 00 00 08 00
>> [ 26.896693] blk_update_request: I/O error, dev sda, sector 44538 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
>> [ 26.897668] ata2: EH complete
>> [ 26.922032] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
>> [ 26.922652] ata2.00: failed command: READ DMA
>> [ 26.923080] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
>> [ 26.923080] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
>> [ 26.924538] ata2.00: status: { DRDY }
>> [ 26.925404] ata2.00: configured for MWDMA2
>> [ 26.925807] ata2: EH complete
>> [ 26.938031] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
>> [ 26.938650] ata2.00: failed command: READ DMA
>> [ 26.939076] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
>> [ 26.939076] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
>> [ 26.940529] ata2.00: status: { DRDY }
>> [ 26.941391] ata2.00: configured for MWDMA2
>> [ 26.941793] ata2: EH complete
>> [ 26.954031] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
>> [ 26.954652] ata2.00: failed command: READ DMA
>> [ 26.955079] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
>> [ 26.955079] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
>> [ 26.956536] ata2.00: status: { DRDY }
>> [ 26.957400] ata2.00: configured for MWDMA2
>> [ 26.957800] ata2: EH complete
>> [ 26.970031] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
>> [ 26.970653] ata2.00: failed command: READ DMA
>> [ 26.971079] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
>> [ 26.971079] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
>> [ 26.972536] ata2.00: status: { DRDY }
>> [ 26.973402] ata2.00: configured for MWDMA2
>> [ 26.973804] ata2: EH complete
>> [ 26.986030] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
>> [ 26.986654] ata2.00: failed command: READ DMA
>> [ 26.987082] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
>> [ 26.987082] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
>> [ 26.988541] ata2.00: status: { DRDY }
>> [ 26.989406] ata2.00: configured for MWDMA2
>> [ 26.989807] ata2: EH complete
>> [ 27.002031] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
>> [ 27.002653] ata2.00: failed command: READ DMA
>> [ 27.003083] ata2.00: cmd c8/00:08:fa:ad:00/00:00:00:00:00/e0 tag 0 dma 4096 in
>> [ 27.003083] res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x40 (internal error)
>> [ 27.004541] ata2.00: status: { DRDY }
>> [ 27.005404] ata2.00: configured for MWDMA2
>> [ 27.005806] sd 1:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
>> [ 27.006688] sd 1:0:0:0: [sda] tag#0 Sense Key : Illegal Request [current]
>> [ 27.007346] sd 1:0:0:0: [sda] tag#0 Add. Sense: Unaligned write command
>> [ 27.007982] sd 1:0:0:0: [sda] tag#0 CDB: Read(10) 28 00 00 00 ad fa 00 00 08 00
>> [ 27.008680] blk_update_request: I/O error, dev sda, sector 44538 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
>> [ 27.009649] ata2: EH complete
>> Bus error
>>
>> I cannot reproduce this warning with set file as nvdimm with following command.
>> sudo /home/teawater/qemu/qemu/x86_64-softmmu/qemu-system-x86_64 -machine pc,accel=kvm,kernel_irqchip,nvdimm -no-reboot -nographic -kernel /home/teawater/kernel/bk2/arch/x86/boot/bzImage -append "console=ttyS0 root=/dev/pmem0 swiotlb=noforce" -m 1g,slots=1,maxmem=2G -smp 1 -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/home/teawater/old.img,size=268435456 -monitor unix:/home/teawater/qemu/m,server,nowait
>>
>> Best,
>> Hui
>
>
> --
> Thanks,
>
> David / dhildenb
^ permalink raw reply [flat|nested] 37+ messages in thread* Re: [PATCH v3 00/15] virtio-mem: paravirtualized memory
2020-05-14 10:02 ` teawater
@ 2020-05-14 10:12 ` David Hildenbrand
2020-05-14 11:10 ` David Hildenbrand
0 siblings, 1 reply; 37+ messages in thread
From: David Hildenbrand @ 2020-05-14 10:12 UTC (permalink / raw)
To: teawater
Cc: LKML, Linux MM, virtio-dev, Linux Virtualization, kvm,
Michal Hocko, Andrew Morton, Michael S . Tsirkin, Sebastien Boeuf,
Samuel Ortiz, Robert Bradford, Luiz Capitulino, Pankaj Gupta,
Igor Mammedov, Dr . David Alan Gilbert, Alexander Duyck,
Alexander Potapenko, Anshuman Khandual, Anthony Yznaga,
Dan Williams, Dave Young
On 14.05.20 12:02, teawater wrote:
>
>
>> 2020年5月14日 16:48，David Hildenbrand <david@redhat.com> 写道：
>>
>> On 14.05.20 08:44, teawater wrote:
>>> Hi David,
>>>
>>> I got a kernel warning with v2 and v3.
>>
>> Hi Hui,
>>
>> thanks for playing with the latest versions. Surprisingly, I can
>> reproduce even by hotplugging a DIMM instead as well - that's good, so
>> it's not related to virtio-mem, lol. Seems to be some QEMU setup issue
>> with older machine types.
>>
>> Can you switch to a newer qemu machine version, especially
>> pc-i440fx-5.0? Both, hotplugging DIMMs and virtio-mem works for me with
>> that QEMU machine just fine.
>
> I still could reproduce this issue with pc-i440fx-5.0 or pc. Did I miss anything?
>
Below I don't even see virtio_mem. I had to repair the image (filesystem
fsck) because it was broken, can you try that as well?
Also, it would be great if you could test with v4.
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH v3 00/15] virtio-mem: paravirtualized memory
2020-05-14 10:12 ` David Hildenbrand
@ 2020-05-14 11:10 ` David Hildenbrand
2020-05-14 11:47 ` David Hildenbrand
0 siblings, 1 reply; 37+ messages in thread
From: David Hildenbrand @ 2020-05-14 11:10 UTC (permalink / raw)
To: teawater
Cc: LKML, Linux MM, virtio-dev, Linux Virtualization, kvm,
Michal Hocko, Andrew Morton, Michael S . Tsirkin, Sebastien Boeuf,
Samuel Ortiz, Robert Bradford, Luiz Capitulino, Pankaj Gupta,
Igor Mammedov, Dr . David Alan Gilbert, Alexander Duyck,
Alexander Potapenko, Anshuman Khandual, Anthony Yznaga,
Dan Williams, Dave Young
On 14.05.20 12:12, David Hildenbrand wrote:
> On 14.05.20 12:02, teawater wrote:
>>
>>
>>> 2020年5月14日 16:48，David Hildenbrand <david@redhat.com> 写道：
>>>
>>> On 14.05.20 08:44, teawater wrote:
>>>> Hi David,
>>>>
>>>> I got a kernel warning with v2 and v3.
>>>
>>> Hi Hui,
>>>
>>> thanks for playing with the latest versions. Surprisingly, I can
>>> reproduce even by hotplugging a DIMM instead as well - that's good, so
>>> it's not related to virtio-mem, lol. Seems to be some QEMU setup issue
>>> with older machine types.
>>>
>>> Can you switch to a newer qemu machine version, especially
>>> pc-i440fx-5.0? Both, hotplugging DIMMs and virtio-mem works for me with
>>> that QEMU machine just fine.
>>
>> I still could reproduce this issue with pc-i440fx-5.0 or pc. Did I miss anything?
>>
>
> Below I don't even see virtio_mem. I had to repair the image (filesystem
> fsck) because it was broken, can you try that as well?
>
> Also, it would be great if you could test with v4.
>
Correction, something seems to be broken either in QEMU or the kernel. Once I
define a DIMM so it's added and online during boot, I get these issues:
(I have virtio-mem v4 installed in the guest)
#! /bin/bash
sudo x86_64-softmmu/qemu-system-x86_64 \
-machine pc-i440fx-5.0,accel=kvm,usb=off \
-cpu host \
-no-reboot \
-nographic \
-device ide-hd,drive=hd \
-drive if=none,id=hd,file=/home/dhildenb/git/Fedora-Cloud-Base-31-1.9.x86_64.qcow2,format=qcow2 \
-m 1g,slots=10,maxmem=2G \
-smp 1 \
-object memory-backend-ram,id=mem0,size=256m \
-device pc-dimm,id=dimm0,memdev=mem0 \
-s \
-monitor unix:/var/tmp/monitor,server,nowait
Without the DIMM it seems to work just fine.
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 37+ messages in thread* Re: [PATCH v3 00/15] virtio-mem: paravirtualized memory
2020-05-14 11:10 ` David Hildenbrand
@ 2020-05-14 11:47 ` David Hildenbrand
2020-05-14 12:19 ` David Hildenbrand
0 siblings, 1 reply; 37+ messages in thread
From: David Hildenbrand @ 2020-05-14 11:47 UTC (permalink / raw)
To: teawater
Cc: LKML, Linux MM, virtio-dev, Linux Virtualization, kvm,
Michal Hocko, Andrew Morton, Michael S . Tsirkin, Sebastien Boeuf,
Samuel Ortiz, Robert Bradford, Luiz Capitulino, Pankaj Gupta,
Igor Mammedov, Dr . David Alan Gilbert, Alexander Duyck,
Alexander Potapenko, Anshuman Khandual, Anthony Yznaga,
Dan Williams, Dave Young
On 14.05.20 13:10, David Hildenbrand wrote:
> On 14.05.20 12:12, David Hildenbrand wrote:
>> On 14.05.20 12:02, teawater wrote:
>>>
>>>
>>>> 2020年5月14日 16:48，David Hildenbrand <david@redhat.com> 写道：
>>>>
>>>> On 14.05.20 08:44, teawater wrote:
>>>>> Hi David,
>>>>>
>>>>> I got a kernel warning with v2 and v3.
>>>>
>>>> Hi Hui,
>>>>
>>>> thanks for playing with the latest versions. Surprisingly, I can
>>>> reproduce even by hotplugging a DIMM instead as well - that's good, so
>>>> it's not related to virtio-mem, lol. Seems to be some QEMU setup issue
>>>> with older machine types.
>>>>
>>>> Can you switch to a newer qemu machine version, especially
>>>> pc-i440fx-5.0? Both, hotplugging DIMMs and virtio-mem works for me with
>>>> that QEMU machine just fine.
>>>
>>> I still could reproduce this issue with pc-i440fx-5.0 or pc. Did I miss anything?
>>>
>>
>> Below I don't even see virtio_mem. I had to repair the image (filesystem
>> fsck) because it was broken, can you try that as well?
>>
>> Also, it would be great if you could test with v4.
>>
>
> Correction, something seems to be broken either in QEMU or the kernel. Once I
> define a DIMM so it's added and online during boot, I get these issues:
>
> (I have virtio-mem v4 installed in the guest)
>
> #! /bin/bash
> sudo x86_64-softmmu/qemu-system-x86_64 \
> -machine pc-i440fx-5.0,accel=kvm,usb=off \
> -cpu host \
> -no-reboot \
> -nographic \
> -device ide-hd,drive=hd \
> -drive if=none,id=hd,file=/home/dhildenb/git/Fedora-Cloud-Base-31-1.9.x86_64.qcow2,format=qcow2 \
> -m 1g,slots=10,maxmem=2G \
> -smp 1 \
> -object memory-backend-ram,id=mem0,size=256m \
> -device pc-dimm,id=dimm0,memdev=mem0 \
> -s \
> -monitor unix:/var/tmp/monitor,server,nowait
>
>
> Without the DIMM it seems to work just fine.
>
And another correction.
Using QEMU v5.0.0, Linux 5.7-rc5, untouched
Fedora-Cloud-Base-32-1.6.x86_64.qcow2, I get even without any memory hotplug:
#! /bin/bash
sudo x86_64-softmmu/qemu-system-x86_64 \
-machine pc-i440fx-5.0,accel=kvm,usb=off \
-cpu host \
-no-reboot \
-nographic \
-device ide-hd,drive=hd \
-drive if=none,id=hd,file=/home/dhildenb/git/Fedora-Cloud-Base-32-1.6.x86_64.qcow2,format=qcow2 \
-m 5g,slots=10,maxmem=6G \
-smp 1 \
-s \
-kernel /home/dhildenb/git/linux/arch/x86/boot/bzImage \
-append "console=ttyS0 rd.shell nokaslr swiotlb=noforce" \
-monitor unix:/var/tmp/monitor,server,nowait
Observe how big the initial RAM even is!
So this is no DIMM/hotplug/virtio_mem issue. With memory hotplug, it seems to get
more likely to trigger if "swiotlb=noforce" is not specified.
"swiotlb=noforce" seems to trigger some pre-existing issue here. Without
"swiotlb=noforce", I was only able to observe this via pc-i440fx-2.1,
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 37+ messages in thread* Re: [PATCH v3 00/15] virtio-mem: paravirtualized memory
2020-05-14 11:47 ` David Hildenbrand
@ 2020-05-14 12:19 ` David Hildenbrand
2020-05-15 2:58 ` teawater
0 siblings, 1 reply; 37+ messages in thread
From: David Hildenbrand @ 2020-05-14 12:19 UTC (permalink / raw)
To: teawater
Cc: LKML, Linux MM, virtio-dev, Linux Virtualization, kvm,
Michal Hocko, Andrew Morton, Michael S . Tsirkin, Sebastien Boeuf,
Samuel Ortiz, Robert Bradford, Luiz Capitulino, Pankaj Gupta,
Igor Mammedov, Dr . David Alan Gilbert, Alexander Duyck,
Alexander Potapenko, Anshuman Khandual, Anthony Yznaga,
Dan Williams, Dave Young
On 14.05.20 13:47, David Hildenbrand wrote:
> On 14.05.20 13:10, David Hildenbrand wrote:
>> On 14.05.20 12:12, David Hildenbrand wrote:
>>> On 14.05.20 12:02, teawater wrote:
>>>>
>>>>
>>>>> 2020年5月14日 16:48，David Hildenbrand <david@redhat.com> 写道：
>>>>>
>>>>> On 14.05.20 08:44, teawater wrote:
>>>>>> Hi David,
>>>>>>
>>>>>> I got a kernel warning with v2 and v3.
>>>>>
>>>>> Hi Hui,
>>>>>
>>>>> thanks for playing with the latest versions. Surprisingly, I can
>>>>> reproduce even by hotplugging a DIMM instead as well - that's good, so
>>>>> it's not related to virtio-mem, lol. Seems to be some QEMU setup issue
>>>>> with older machine types.
>>>>>
>>>>> Can you switch to a newer qemu machine version, especially
>>>>> pc-i440fx-5.0? Both, hotplugging DIMMs and virtio-mem works for me with
>>>>> that QEMU machine just fine.
>>>>
>>>> I still could reproduce this issue with pc-i440fx-5.0 or pc. Did I miss anything?
>>>>
>>>
>>> Below I don't even see virtio_mem. I had to repair the image (filesystem
>>> fsck) because it was broken, can you try that as well?
>>>
>>> Also, it would be great if you could test with v4.
>>>
>>
>> Correction, something seems to be broken either in QEMU or the kernel. Once I
>> define a DIMM so it's added and online during boot, I get these issues:
>>
>> (I have virtio-mem v4 installed in the guest)
>>
>> #! /bin/bash
>> sudo x86_64-softmmu/qemu-system-x86_64 \
>> -machine pc-i440fx-5.0,accel=kvm,usb=off \
>> -cpu host \
>> -no-reboot \
>> -nographic \
>> -device ide-hd,drive=hd \
>> -drive if=none,id=hd,file=/home/dhildenb/git/Fedora-Cloud-Base-31-1.9.x86_64.qcow2,format=qcow2 \
>> -m 1g,slots=10,maxmem=2G \
>> -smp 1 \
>> -object memory-backend-ram,id=mem0,size=256m \
>> -device pc-dimm,id=dimm0,memdev=mem0 \
>> -s \
>> -monitor unix:/var/tmp/monitor,server,nowait
>>
>>
>> Without the DIMM it seems to work just fine.
>>
>
> And another correction.
>
> Using QEMU v5.0.0, Linux 5.7-rc5, untouched
> Fedora-Cloud-Base-32-1.6.x86_64.qcow2, I get even without any memory hotplug:
>
> #! /bin/bash
> sudo x86_64-softmmu/qemu-system-x86_64 \
> -machine pc-i440fx-5.0,accel=kvm,usb=off \
> -cpu host \
> -no-reboot \
> -nographic \
> -device ide-hd,drive=hd \
> -drive if=none,id=hd,file=/home/dhildenb/git/Fedora-Cloud-Base-32-1.6.x86_64.qcow2,format=qcow2 \
> -m 5g,slots=10,maxmem=6G \
> -smp 1 \
> -s \
> -kernel /home/dhildenb/git/linux/arch/x86/boot/bzImage \
> -append "console=ttyS0 rd.shell nokaslr swiotlb=noforce" \
> -monitor unix:/var/tmp/monitor,server,nowait
>
>
> Observe how big the initial RAM even is!
>
>
> So this is no DIMM/hotplug/virtio_mem issue. With memory hotplug, it seems to get
> more likely to trigger if "swiotlb=noforce" is not specified.
>
> "swiotlb=noforce" seems to trigger some pre-existing issue here. Without
> "swiotlb=noforce", I was only able to observe this via pc-i440fx-2.1,
>
(talking to myself :) )
I think I finally understood why using "swiotlb=noforce" with hotplugged
memory is wrong - or with memory > 3GB. Via "swiotlb=noforce" you tell
the system to "Never use bounce buffers (for debugging)". This works as
long as all memory is DMA memory (e.g., < 3GB) AFAIK.
"If specified, trying to map memory that cannot be used with DMA will
fail, and a rate-limited warning will be printed."
Hotplugged memory (under QEMU) is never added below 4GB, because of the
PCI hole. So both, memory from DIMMs and from virtio-mem will end up at
or above 4GB. To make a device use that memory, you need bounce buffers.
Hotplugged memory is never DMA memory.
--
Thanks,
David / dhildenb
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: [PATCH v3 00/15] virtio-mem: paravirtualized memory
2020-05-14 12:19 ` David Hildenbrand
@ 2020-05-15 2:58 ` teawater
0 siblings, 0 replies; 37+ messages in thread
From: teawater @ 2020-05-15 2:58 UTC (permalink / raw)
To: David Hildenbrand
Cc: LKML, Linux MM, virtio-dev, Linux Virtualization, kvm,
Michal Hocko, Andrew Morton, Michael S . Tsirkin, Sebastien Boeuf,
Samuel Ortiz, Robert Bradford, Luiz Capitulino, Pankaj Gupta,
Igor Mammedov, Dr . David Alan Gilbert, Alexander Duyck,
Alexander Potapenko, Anshuman Khandual, Anthony Yznaga,
Dan Williams, Dave Young, Jason
> 2020年5月14日 20:19，David Hildenbrand <david@redhat.com> 写道：
>
> On 14.05.20 13:47, David Hildenbrand wrote:
>> On 14.05.20 13:10, David Hildenbrand wrote:
>>> On 14.05.20 12:12, David Hildenbrand wrote:
>>>> On 14.05.20 12:02, teawater wrote:
>>>>>
>>>>>
>>>>>> 2020年5月14日 16:48，David Hildenbrand <david@redhat.com> 写道：
>>>>>>
>>>>>> On 14.05.20 08:44, teawater wrote:
>>>>>>> Hi David,
>>>>>>>
>>>>>>> I got a kernel warning with v2 and v3.
>>>>>>
>>>>>> Hi Hui,
>>>>>>
>>>>>> thanks for playing with the latest versions. Surprisingly, I can
>>>>>> reproduce even by hotplugging a DIMM instead as well - that's good, so
>>>>>> it's not related to virtio-mem, lol. Seems to be some QEMU setup issue
>>>>>> with older machine types.
>>>>>>
>>>>>> Can you switch to a newer qemu machine version, especially
>>>>>> pc-i440fx-5.0? Both, hotplugging DIMMs and virtio-mem works for me with
>>>>>> that QEMU machine just fine.
>>>>>
>>>>> I still could reproduce this issue with pc-i440fx-5.0 or pc. Did I miss anything?
>>>>>
>>>>
>>>> Below I don't even see virtio_mem. I had to repair the image (filesystem
>>>> fsck) because it was broken, can you try that as well?
>>>>
>>>> Also, it would be great if you could test with v4.
>>>>
>>>
>>> Correction, something seems to be broken either in QEMU or the kernel. Once I
>>> define a DIMM so it's added and online during boot, I get these issues:
>>>
>>> (I have virtio-mem v4 installed in the guest)
>>>
>>> #! /bin/bash
>>> sudo x86_64-softmmu/qemu-system-x86_64 \
>>> -machine pc-i440fx-5.0,accel=kvm,usb=off \
>>> -cpu host \
>>> -no-reboot \
>>> -nographic \
>>> -device ide-hd,drive=hd \
>>> -drive if=none,id=hd,file=/home/dhildenb/git/Fedora-Cloud-Base-31-1.9.x86_64.qcow2,format=qcow2 \
>>> -m 1g,slots=10,maxmem=2G \
>>> -smp 1 \
>>> -object memory-backend-ram,id=mem0,size=256m \
>>> -device pc-dimm,id=dimm0,memdev=mem0 \
>>> -s \
>>> -monitor unix:/var/tmp/monitor,server,nowait
>>>
>>>
>>> Without the DIMM it seems to work just fine.
>>>
>>
>> And another correction.
>>
>> Using QEMU v5.0.0, Linux 5.7-rc5, untouched
>> Fedora-Cloud-Base-32-1.6.x86_64.qcow2, I get even without any memory hotplug:
>>
>> #! /bin/bash
>> sudo x86_64-softmmu/qemu-system-x86_64 \
>> -machine pc-i440fx-5.0,accel=kvm,usb=off \
>> -cpu host \
>> -no-reboot \
>> -nographic \
>> -device ide-hd,drive=hd \
>> -drive if=none,id=hd,file=/home/dhildenb/git/Fedora-Cloud-Base-32-1.6.x86_64.qcow2,format=qcow2 \
>> -m 5g,slots=10,maxmem=6G \
>> -smp 1 \
>> -s \
>> -kernel /home/dhildenb/git/linux/arch/x86/boot/bzImage \
>> -append "console=ttyS0 rd.shell nokaslr swiotlb=noforce" \
>> -monitor unix:/var/tmp/monitor,server,nowait
>>
>>
>> Observe how big the initial RAM even is!
>>
>>
>> So this is no DIMM/hotplug/virtio_mem issue. With memory hotplug, it seems to get
>> more likely to trigger if "swiotlb=noforce" is not specified.
>>
>> "swiotlb=noforce" seems to trigger some pre-existing issue here. Without
>> "swiotlb=noforce", I was only able to observe this via pc-i440fx-2.1,
>>
>
> (talking to myself :) )
>
> I think I finally understood why using "swiotlb=noforce" with hotplugged
> memory is wrong - or with memory > 3GB. Via "swiotlb=noforce" you tell
> the system to "Never use bounce buffers (for debugging)". This works as
> long as all memory is DMA memory (e.g., < 3GB) AFAIK.
>
> "If specified, trying to map memory that cannot be used with DMA will
> fail, and a rate-limited warning will be printed."
>
> Hotplugged memory (under QEMU) is never added below 4GB, because of the
> PCI hole. So both, memory from DIMMs and from virtio-mem will end up at
> or above 4GB. To make a device use that memory, you need bounce buffers.
>
> Hotplugged memory is never DMA memory.
>
Hi David,
It is fixed when I remove "swiotlb=noforce”.
Thanks for your help.
Best,
Hui
> --
> Thanks,
>
> David / dhildenb
^ permalink raw reply [flat|nested] 37+ messages in thread