* Re: [PATCH V5 0/4] Improve virtio-blk performance
From: Jens Axboe @ 2012-08-02 21:40 UTC (permalink / raw)
To: Asias He
Cc: kvm, Michael S. Tsirkin, linux-kernel, virtualization, Tejun Heo,
Shaohua Li, Christoph Hellwig
In-Reply-To: <1343888757-25723-1-git-send-email-asias@redhat.com>
On 08/02/2012 08:25 AM, Asias He wrote:
> Hi folks,
>
> This version added REQ_FLUSH and REQ_FUA support as suggested by Christoph and
> rebased against latest linus's tree.
>
> Jens, could you please consider picking up the dependencies 1/4 and
> 2/4 in your tree. Thanks!
Pickedup, thanks for getting that done!
--
Jens Axboe
^ permalink raw reply
* Re: [PATCH V5 3/4] virtio-blk: Add bio-based IO path for virtio-blk
From: Michael S. Tsirkin @ 2012-08-02 22:28 UTC (permalink / raw)
To: Asias He
Cc: Jens Axboe, kvm, linux-kernel, virtualization, Tejun Heo,
Shaohua Li, Christoph Hellwig
In-Reply-To: <1343888757-25723-4-git-send-email-asias@redhat.com>
On Thu, Aug 02, 2012 at 02:25:55PM +0800, Asias He wrote:
> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
> index c0bbeb4..95cfeed 100644
> --- a/drivers/block/virtio_blk.c
> +++ b/drivers/block/virtio_blk.c
> @@ -14,6 +14,9 @@
>
> #define PART_BITS 4
>
> +static bool use_bio;
> +module_param(use_bio, bool, S_IRUGO);
> +
> static int major;
> static DEFINE_IDA(vd_index_ida);
>
OK, but eventually, you plan to make this use a feature bit, yes?
^ permalink raw reply
* Re: [PATCH V5 0/4] Improve virtio-blk performance
From: Asias He @ 2012-08-03 7:08 UTC (permalink / raw)
To: Jens Axboe
Cc: kvm, Michael S. Tsirkin, linux-kernel, virtualization, Tejun Heo,
Shaohua Li, Christoph Hellwig
In-Reply-To: <501AF3E3.6090403@kernel.dk>
On 08/03/2012 05:40 AM, Jens Axboe wrote:
> On 08/02/2012 08:25 AM, Asias He wrote:
>> Hi folks,
>>
>> This version added REQ_FLUSH and REQ_FUA support as suggested by Christoph and
>> rebased against latest linus's tree.
>>
>> Jens, could you please consider picking up the dependencies 1/4 and
>> 2/4 in your tree. Thanks!
>
> Pickedup, thanks for getting that done!
Cheers. Thanks, Jens.
--
Asias
^ permalink raw reply
* Re: [PATCH V5 3/4] virtio-blk: Add bio-based IO path for virtio-blk
From: Asias He @ 2012-08-03 7:11 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jens Axboe, kvm, linux-kernel, virtualization, Tejun Heo,
Shaohua Li, Christoph Hellwig
In-Reply-To: <20120802222814.GA5854@redhat.com>
On 08/03/2012 06:28 AM, Michael S. Tsirkin wrote:
> On Thu, Aug 02, 2012 at 02:25:55PM +0800, Asias He wrote:
>> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
>> index c0bbeb4..95cfeed 100644
>> --- a/drivers/block/virtio_blk.c
>> +++ b/drivers/block/virtio_blk.c
>> @@ -14,6 +14,9 @@
>>
>> #define PART_BITS 4
>>
>> +static bool use_bio;
>> +module_param(use_bio, bool, S_IRUGO);
>> +
>> static int major;
>> static DEFINE_IDA(vd_index_ida);
>>
>
> OK, but eventually, you plan to make this use a feature bit, yes?
Yes, I think so.
--
Asias
^ permalink raw reply
* Re: [PATCH v4 1/3] mm: introduce compaction and migration for virtio ballooned pages
From: Rafael Aquini @ 2012-08-03 11:13 UTC (permalink / raw)
To: Rik van Riel
Cc: Konrad Rzeszutek Wilk, Michael S. Tsirkin, linux-kernel,
virtualization, linux-mm, Minchan Kim, Andi Kleen, Andrew Morton,
Rafael Aquini
In-Reply-To: <5019975B.6010708@redhat.com>
On Wed, Aug 01, 2012 at 04:53:47PM -0400, Rik van Riel wrote:
> On 07/23/2012 02:19 PM, Rafael Aquini wrote:
>
> >In a glance, I believe this whole dance you're suggesting might just be too much
> >of an overcomplication, and the best approach would be simply teaching the
> >hotplug bits about the ballooned corner case just like it's being done to
> >compaction/migration. However, I'll look at it carefully before making any other
> >adjustments/propositions over here.
>
> Compaction and hotplug do essentially the same thing
> here: "collect all the movable pages from a page block,
> and move them elsewhere".
>
> Whether or not it is easier for them to share code, or
> to duplicate a few lines of code, is something that can
> be looked into later.
I'm 100% in agreement with your thoughts here.
^ permalink raw reply
* Re: [PATCH v4 1/3] mm: introduce compaction and migration for virtio ballooned pages
From: Rafael Aquini @ 2012-08-03 12:26 UTC (permalink / raw)
To: Minchan Kim
Cc: Rik van Riel, Konrad Rzeszutek Wilk, Michael S. Tsirkin,
linux-kernel, virtualization, linux-mm, Andi Kleen, Andrew Morton,
Rafael Aquini
In-Reply-To: <20120723023332.GA6832@bbox>
On Mon, Jul 23, 2012 at 11:33:32AM +0900, Minchan Kim wrote:
> Look at memory-hotplug, offline_page calls has_unmovable_pages, scan_lru_pages
> and do_migrate_range which calls isolate_lru_page. They consider only LRU pages
> to migratable ones.
>
As promised, I looked into those bits. Yes, they only isolate LRU pages, and as
such, having this series merged or not doesn't change a bit for that code path.
In fact, having this series merged and teaching hotplug's
offline_pages()/do_migrate_rage() about ballooned pages might be extremely
beneficial in the rare event offlining memory stumbles across a balloon page.
As Rik said, I believe this is something we can look into in the near future.
> IMHO, better approach is that after we can get complete free pageblocks
> by compaction or reclaim, move balloon pages into that pageblocks and make
> that blocks to unmovable. It can prevent fragmentation and it makes
> current or future code don't need to consider balloon page.
>
I totally agree with Rik on this one, as well. This is the wrong approach here.
All that said, I'll soon respin a v5 based on your comments on branch hinting and
commentary improvements, as well as addressing AKPM's concerns. I'll also revert
isolate_balloon_page() last changes back to make it a public symbol again, as
(I believe) we'll shortly be using it for letting hotplug bits aware of how to
isolate ballooned pages.
^ permalink raw reply
* [PATCH v5 0/3] make balloon pages movable by compaction
From: Rafael Aquini @ 2012-08-06 13:56 UTC (permalink / raw)
To: linux-mm
Cc: Rik van Riel, Rafael Aquini, Konrad Rzeszutek Wilk,
Michael S. Tsirkin, linux-kernel, virtualization, Minchan Kim,
Andi Kleen, Andrew Morton
Memory fragmentation introduced by ballooning might reduce significantly
the number of 2MB contiguous memory blocks that can be used within a guest,
thus imposing performance penalties associated with the reduced number of
transparent huge pages that could be used by the guest workload.
This patch-set follows the main idea discussed at 2012 LSFMMS session:
"Ballooning for transparent huge pages" -- http://lwn.net/Articles/490114/
to introduce the required changes to the virtio_balloon driver, as well as
the changes to the core compaction & migration bits, in order to make those
subsystems aware of ballooned pages and allow memory balloon pages become
movable within a guest, thus avoiding the aforementioned fragmentation issue
Rafael Aquini (3):
mm: introduce compaction and migration for virtio ballooned pages
virtio_balloon: introduce migration primitives to balloon pages
mm: add vm event counters for balloon pages compaction
drivers/virtio/virtio_balloon.c | 139 +++++++++++++++++++++++++++++++++++++---
include/linux/mm.h | 26 ++++++++
include/linux/virtio_balloon.h | 4 ++
include/linux/vm_event_item.h | 2 +
mm/compaction.c | 131 +++++++++++++++++++++++++++++++------
mm/migrate.c | 34 +++++++++-
mm/vmstat.c | 4 ++
7 files changed, 312 insertions(+), 28 deletions(-)
Change log:
v5:
* address Andrew Morton's review comments on the patch series;
* address a couple extra nitpick suggestions on PATCH 01 (Minchan);
v4:
* address Rusty Russel's review comments on PATCH 02;
* re-base virtio_balloon patch on 9c378abc5c0c6fc8e3acf5968924d274503819b3;
V3:
* address reviewers nitpick suggestions on PATCH 01 (Mel, Minchan);
V2:
* address Mel Gorman's review comments on PATCH 01;
Preliminary test results:
(2 VCPU 1024mB RAM KVM guest running 3.6.0_rc1+ -- after a reboot)
* 64mB balloon:
[root@localhost ~]# awk '/compact/ {print}' /proc/vmstat
compact_blocks_moved 0
compact_pages_moved 0
compact_pagemigrate_failed 0
compact_stall 0
compact_fail 0
compact_success 0
compact_balloon_migrated 0
compact_balloon_failed 0
compact_balloon_isolated 0
compact_balloon_freed 0
[root@localhost ~]#
[root@localhost ~]# for i in $(seq 1 6); do echo 1 > /proc/sys/vm/compact_memory & done &>/dev/null
[1] Done echo 1 > /proc/sys/vm/compact_memory
[2] Done echo 1 > /proc/sys/vm/compact_memory
[3] Done echo 1 > /proc/sys/vm/compact_memory
[4] Done echo 1 > /proc/sys/vm/compact_memory
[5]- Done echo 1 > /proc/sys/vm/compact_memory
[6]+ Done echo 1 > /proc/sys/vm/compact_memory
[root@localhost ~]#
[root@localhost ~]# awk '/compact/ {print}' /proc/vmstat
compact_blocks_moved 3520
compact_pages_moved 47548
compact_pagemigrate_failed 120
compact_stall 0
compact_fail 0
compact_success 0
compact_balloon_migrated 16378
compact_balloon_failed 0
compact_balloon_isolated 16378
compact_balloon_freed 16378
* 128mB balloon:
[root@localhost ~]# awk '/compact/ {print}' /proc/vmstat
compact_blocks_moved 0
compact_pages_moved 0
compact_pagemigrate_failed 0
compact_stall 0
compact_fail 0
compact_success 0
compact_balloon_migrated 0
compact_balloon_failed 0
compact_balloon_isolated 0
compact_balloon_freed 0
[root@localhost ~]#
[root@localhost ~]# for i in $(seq 1 6); do echo 1 > /proc/sys/vm/compact_memory & done &>/dev/null
[1] Done echo 1 > /proc/sys/vm/compact_memory
[2] Done echo 1 > /proc/sys/vm/compact_memory
[3] Done echo 1 > /proc/sys/vm/compact_memory
[4] Done echo 1 > /proc/sys/vm/compact_memory
[5]- Done echo 1 > /proc/sys/vm/compact_memory
[6]+ Done echo 1 > /proc/sys/vm/compact_memory
[root@localhost ~]#
[root@localhost ~]# awk '/compact/ {print}' /proc/vmstat
compact_blocks_moved 3356
compact_pages_moved 47099
compact_pagemigrate_failed 158
compact_stall 0
compact_fail 0
compact_success 0
compact_balloon_migrated 26275
compact_balloon_failed 42
compact_balloon_isolated 26317
compact_balloon_freed 26275
--
1.7.11.2
^ permalink raw reply
* [PATCH v5 1/3] mm: introduce compaction and migration for virtio ballooned pages
From: Rafael Aquini @ 2012-08-06 13:56 UTC (permalink / raw)
To: linux-mm
Cc: Rik van Riel, Rafael Aquini, Konrad Rzeszutek Wilk,
Michael S. Tsirkin, linux-kernel, virtualization, Minchan Kim,
Andi Kleen, Andrew Morton
In-Reply-To: <cover.1344259054.git.aquini@redhat.com>
Memory fragmentation introduced by ballooning might reduce significantly
the number of 2MB contiguous memory blocks that can be used within a guest,
thus imposing performance penalties associated with the reduced number of
transparent huge pages that could be used by the guest workload.
This patch introduces the helper functions as well as the necessary changes
to teach compaction and migration bits how to cope with pages which are
part of a guest memory balloon, in order to make them movable by memory
compaction procedures.
Signed-off-by: Rafael Aquini <aquini@redhat.com>
---
include/linux/mm.h | 26 +++++++++++
mm/compaction.c | 130 +++++++++++++++++++++++++++++++++++++++++++++--------
mm/migrate.c | 32 ++++++++++++-
3 files changed, 168 insertions(+), 20 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 311be90..01a2dc9 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1662,5 +1662,31 @@ static inline unsigned int debug_guardpage_minorder(void) { return 0; }
static inline bool page_is_guard(struct page *page) { return false; }
#endif /* CONFIG_DEBUG_PAGEALLOC */
+#if (defined(CONFIG_VIRTIO_BALLOON) || defined(CONFIG_VIRTIO_BALLOON_MODULE))
+extern bool isolate_balloon_page(struct page *);
+extern bool putback_balloon_page(struct page *);
+extern struct address_space *balloon_mapping;
+
+static inline bool is_balloon_page(struct page *page)
+{
+ return (page->mapping && page->mapping == balloon_mapping);
+}
+
+static inline bool balloon_compaction_enabled(void)
+{
+#if defined(CONFIG_COMPACTION)
+ return true;
+#else
+ return false;
+#endif /* CONFIG_COMPACTION */
+}
+
+#else
+static inline bool isolate_balloon_page(struct page *page) { return false; }
+static inline bool putback_balloon_page(struct page *page) { return false; }
+static inline bool is_balloon_page(struct page *page) { return false; }
+static inline bool balloon_compaction_enabled(void) { return false; }
+#endif /* (CONFIG_VIRTIO_BALLOON || CONFIG_VIRTIO_BALLOON_MODULE) */
+
#endif /* __KERNEL__ */
#endif /* _LINUX_MM_H */
diff --git a/mm/compaction.c b/mm/compaction.c
index e78cb96..9499d85 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -14,6 +14,7 @@
#include <linux/backing-dev.h>
#include <linux/sysctl.h>
#include <linux/sysfs.h>
+#include <linux/export.h>
#include "internal.h"
#if defined CONFIG_COMPACTION || defined CONFIG_CMA
@@ -21,6 +22,89 @@
#define CREATE_TRACE_POINTS
#include <trace/events/compaction.h>
+#if defined(CONFIG_VIRTIO_BALLOON) || defined(CONFIG_VIRTIO_BALLOON_MODULE)
+/*
+ * Balloon pages special page->mapping.
+ * Users must properly allocate and initialize an instance of balloon_mapping,
+ * and set it as the page->mapping for balloon enlisted page instances.
+ * There is no need on utilizing struct address_space locking schemes for
+ * balloon_mapping as, once it gets initialized at balloon driver, it will
+ * remain just like a static reference that helps us on identifying a guest
+ * ballooned page by its mapping, as well as it will keep the 'a_ops' callback
+ * pointers to the functions that will execute the balloon page mobility tasks.
+ *
+ * address_space_operations necessary methods for ballooned pages:
+ * .migratepage - used to perform balloon's page migration (as is)
+ * .invalidatepage - used to isolate a page from balloon's page list
+ * .freepage - used to reinsert an isolated page to balloon's page list
+ */
+struct address_space *balloon_mapping;
+EXPORT_SYMBOL_GPL(balloon_mapping);
+
+static inline void __isolate_balloon_page(struct page *page)
+{
+ page->mapping->a_ops->invalidatepage(page, 0);
+}
+
+static inline void __putback_balloon_page(struct page *page)
+{
+ page->mapping->a_ops->freepage(page);
+}
+
+/* __isolate_lru_page() counterpart for a ballooned page */
+bool isolate_balloon_page(struct page *page)
+{
+ if (WARN_ON(!is_balloon_page(page)))
+ return false;
+
+ if (likely(get_page_unless_zero(page))) {
+ /*
+ * As balloon pages are not isolated from LRU lists, concurrent
+ * compaction threads can race against page migration functions
+ * move_to_new_page() & __unmap_and_move().
+ * In order to avoid having an already isolated balloon page
+ * being (wrongly) re-isolated while it is under migration,
+ * lets be sure we have the page lock before proceeding with
+ * the balloon page isolation steps.
+ */
+ if (likely(trylock_page(page))) {
+ /*
+ * A ballooned page, by default, has just one refcount.
+ * Prevent concurrent compaction threads from isolating
+ * an already isolated balloon page.
+ */
+ if (is_balloon_page(page) && (page_count(page) == 2)) {
+ __isolate_balloon_page(page);
+ unlock_page(page);
+ return true;
+ }
+ unlock_page(page);
+ }
+ /* Drop refcount taken for this already isolated page */
+ put_page(page);
+ }
+ return false;
+}
+
+/* putback_lru_page() counterpart for a ballooned page */
+bool putback_balloon_page(struct page *page)
+{
+ if (WARN_ON(!is_balloon_page(page)))
+ return false;
+
+ if (likely(trylock_page(page))) {
+ if (is_balloon_page(page)) {
+ __putback_balloon_page(page);
+ put_page(page);
+ unlock_page(page);
+ return true;
+ }
+ unlock_page(page);
+ }
+ return false;
+}
+#endif /* CONFIG_VIRTIO_BALLOON || CONFIG_VIRTIO_BALLOON_MODULE */
+
static unsigned long release_freepages(struct list_head *freelist)
{
struct page *page, *next;
@@ -312,32 +396,40 @@ isolate_migratepages_range(struct zone *zone, struct compact_control *cc,
continue;
}
- if (!PageLRU(page))
- continue;
-
/*
- * PageLRU is set, and lru_lock excludes isolation,
- * splitting and collapsing (collapsing has already
- * happened if PageLRU is set).
+ * It is possible to migrate LRU pages and balloon pages.
+ * Skip any other type of page.
*/
- if (PageTransHuge(page)) {
- low_pfn += (1 << compound_order(page)) - 1;
- continue;
- }
+ if (PageLRU(page)) {
+ /*
+ * PageLRU is set, and lru_lock excludes isolation,
+ * splitting and collapsing (collapsing has already
+ * happened if PageLRU is set).
+ */
+ if (PageTransHuge(page)) {
+ low_pfn += (1 << compound_order(page)) - 1;
+ continue;
+ }
- if (!cc->sync)
- mode |= ISOLATE_ASYNC_MIGRATE;
+ if (!cc->sync)
+ mode |= ISOLATE_ASYNC_MIGRATE;
- lruvec = mem_cgroup_page_lruvec(page, zone);
+ lruvec = mem_cgroup_page_lruvec(page, zone);
- /* Try isolate the page */
- if (__isolate_lru_page(page, mode) != 0)
- continue;
+ /* Try isolate the page */
+ if (__isolate_lru_page(page, mode) != 0)
+ continue;
+
+ VM_BUG_ON(PageTransCompound(page));
- VM_BUG_ON(PageTransCompound(page));
+ /* Successfully isolated */
+ del_page_from_lru_list(page, lruvec, page_lru(page));
+ } else if (unlikely(is_balloon_page(page))) {
+ if (!isolate_balloon_page(page))
+ continue;
+ } else
+ continue;
- /* Successfully isolated */
- del_page_from_lru_list(page, lruvec, page_lru(page));
list_add(&page->lru, migratelist);
cc->nr_migratepages++;
nr_isolated++;
diff --git a/mm/migrate.c b/mm/migrate.c
index 77ed2d7..fc56968 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -79,7 +79,11 @@ void putback_lru_pages(struct list_head *l)
list_del(&page->lru);
dec_zone_page_state(page, NR_ISOLATED_ANON +
page_is_file_cache(page));
- putback_lru_page(page);
+ if (unlikely(is_balloon_page(page) &&
+ balloon_compaction_enabled()))
+ WARN_ON(!putback_balloon_page(page));
+ else
+ putback_lru_page(page);
}
}
@@ -778,6 +782,17 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
}
}
+ if (unlikely(is_balloon_page(page) && balloon_compaction_enabled())) {
+ /*
+ * A ballooned page does not need any special attention from
+ * physical to virtual reverse mapping procedures.
+ * Skip any attempt to unmap PTEs or to remap swap cache,
+ * in order to avoid burning cycles at rmap level.
+ */
+ remap_swapcache = 0;
+ goto skip_unmap;
+ }
+
/*
* Corner case handling:
* 1. When a new swap-cache page is read into, it is added to the LRU
@@ -846,6 +861,21 @@ static int unmap_and_move(new_page_t get_new_page, unsigned long private,
goto out;
rc = __unmap_and_move(page, newpage, force, offlining, mode);
+
+ if (unlikely(is_balloon_page(newpage) &&
+ balloon_compaction_enabled())) {
+ /*
+ * A ballooned page has been migrated already. Now, it is the
+ * time to wrap-up counters, handle the old page back to Buddy
+ * and return.
+ */
+ list_del(&page->lru);
+ dec_zone_page_state(page, NR_ISOLATED_ANON +
+ page_is_file_cache(page));
+ put_page(page);
+ __free_page(page);
+ return rc;
+ }
out:
if (rc != -EAGAIN) {
/*
--
1.7.11.2
^ permalink raw reply related
* [PATCH v5 2/3] virtio_balloon: introduce migration primitives to balloon pages
From: Rafael Aquini @ 2012-08-06 13:56 UTC (permalink / raw)
To: linux-mm
Cc: Rik van Riel, Rafael Aquini, Konrad Rzeszutek Wilk,
Michael S. Tsirkin, linux-kernel, virtualization, Minchan Kim,
Andi Kleen, Andrew Morton
In-Reply-To: <cover.1344259054.git.aquini@redhat.com>
Memory fragmentation introduced by ballooning might reduce significantly
the number of 2MB contiguous memory blocks that can be used within a guest,
thus imposing performance penalties associated with the reduced number of
transparent huge pages that could be used by the guest workload.
Besides making balloon pages movable at allocation time and introducing
the necessary primitives to perform balloon page migration/compaction,
this patch also introduces the following locking scheme to provide the
proper synchronization and protection for struct virtio_balloon elements
against concurrent accesses due to parallel operations introduced by
memory compaction / page migration.
- balloon_lock (mutex) : synchronizes the access demand to elements of
struct virtio_balloon and its queue operations;
- pages_lock (spinlock): special protection to balloon pages list against
concurrent list handling operations;
Signed-off-by: Rafael Aquini <aquini@redhat.com>
---
drivers/virtio/virtio_balloon.c | 138 +++++++++++++++++++++++++++++++++++++---
include/linux/virtio_balloon.h | 4 ++
2 files changed, 134 insertions(+), 8 deletions(-)
diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
index 0908e60..7c937a0 100644
--- a/drivers/virtio/virtio_balloon.c
+++ b/drivers/virtio/virtio_balloon.c
@@ -27,6 +27,7 @@
#include <linux/delay.h>
#include <linux/slab.h>
#include <linux/module.h>
+#include <linux/fs.h>
/*
* Balloon device works in 4K page units. So each page is pointed to by
@@ -35,6 +36,12 @@
*/
#define VIRTIO_BALLOON_PAGES_PER_PAGE (PAGE_SIZE >> VIRTIO_BALLOON_PFN_SHIFT)
+/* Synchronizes accesses/updates to the struct virtio_balloon elements */
+DEFINE_MUTEX(balloon_lock);
+
+/* Protects 'virtio_balloon->pages' list against concurrent handling */
+DEFINE_SPINLOCK(pages_lock);
+
struct virtio_balloon
{
struct virtio_device *vdev;
@@ -51,6 +58,7 @@ struct virtio_balloon
/* Number of balloon pages we've told the Host we're not using. */
unsigned int num_pages;
+
/*
* The pages we've told the Host we're not using.
* Each page on this list adds VIRTIO_BALLOON_PAGES_PER_PAGE
@@ -125,10 +133,12 @@ static void fill_balloon(struct virtio_balloon *vb, size_t num)
/* We can only do one array worth at a time. */
num = min(num, ARRAY_SIZE(vb->pfns));
+ mutex_lock(&balloon_lock);
for (vb->num_pfns = 0; vb->num_pfns < num;
vb->num_pfns += VIRTIO_BALLOON_PAGES_PER_PAGE) {
- struct page *page = alloc_page(GFP_HIGHUSER | __GFP_NORETRY |
- __GFP_NOMEMALLOC | __GFP_NOWARN);
+ struct page *page = alloc_page(GFP_HIGHUSER_MOVABLE |
+ __GFP_NORETRY | __GFP_NOWARN |
+ __GFP_NOMEMALLOC);
if (!page) {
if (printk_ratelimit())
dev_printk(KERN_INFO, &vb->vdev->dev,
@@ -141,7 +151,10 @@ static void fill_balloon(struct virtio_balloon *vb, size_t num)
set_page_pfns(vb->pfns + vb->num_pfns, page);
vb->num_pages += VIRTIO_BALLOON_PAGES_PER_PAGE;
totalram_pages--;
+ spin_lock(&pages_lock);
list_add(&page->lru, &vb->pages);
+ page->mapping = balloon_mapping;
+ spin_unlock(&pages_lock);
}
/* Didn't get any? Oh well. */
@@ -149,6 +162,7 @@ static void fill_balloon(struct virtio_balloon *vb, size_t num)
return;
tell_host(vb, vb->inflate_vq);
+ mutex_unlock(&balloon_lock);
}
static void release_pages_by_pfn(const u32 pfns[], unsigned int num)
@@ -169,10 +183,22 @@ static void leak_balloon(struct virtio_balloon *vb, size_t num)
/* We can only do one array worth at a time. */
num = min(num, ARRAY_SIZE(vb->pfns));
+ mutex_lock(&balloon_lock);
for (vb->num_pfns = 0; vb->num_pfns < num;
vb->num_pfns += VIRTIO_BALLOON_PAGES_PER_PAGE) {
+ /*
+ * We can race against virtballoon_isolatepage() and end up
+ * stumbling across a _temporarily_ empty 'pages' list.
+ */
+ spin_lock(&pages_lock);
+ if (unlikely(list_empty(&vb->pages))) {
+ spin_unlock(&pages_lock);
+ break;
+ }
page = list_first_entry(&vb->pages, struct page, lru);
+ page->mapping = NULL;
list_del(&page->lru);
+ spin_unlock(&pages_lock);
set_page_pfns(vb->pfns + vb->num_pfns, page);
vb->num_pages -= VIRTIO_BALLOON_PAGES_PER_PAGE;
}
@@ -182,8 +208,11 @@ static void leak_balloon(struct virtio_balloon *vb, size_t num)
* virtio_has_feature(vdev, VIRTIO_BALLOON_F_MUST_TELL_HOST);
* is true, we *have* to do it in this order
*/
- tell_host(vb, vb->deflate_vq);
- release_pages_by_pfn(vb->pfns, vb->num_pfns);
+ if (vb->num_pfns > 0) {
+ tell_host(vb, vb->deflate_vq);
+ release_pages_by_pfn(vb->pfns, vb->num_pfns);
+ }
+ mutex_unlock(&balloon_lock);
}
static inline void update_stat(struct virtio_balloon *vb, int idx,
@@ -239,6 +268,7 @@ static void stats_handle_request(struct virtio_balloon *vb)
struct scatterlist sg;
unsigned int len;
+ mutex_lock(&balloon_lock);
vb->need_stats_update = 0;
update_balloon_stats(vb);
@@ -249,6 +279,7 @@ static void stats_handle_request(struct virtio_balloon *vb)
if (virtqueue_add_buf(vq, &sg, 1, 0, vb, GFP_KERNEL) < 0)
BUG();
virtqueue_kick(vq);
+ mutex_unlock(&balloon_lock);
}
static void virtballoon_changed(struct virtio_device *vdev)
@@ -261,22 +292,27 @@ static void virtballoon_changed(struct virtio_device *vdev)
static inline s64 towards_target(struct virtio_balloon *vb)
{
__le32 v;
- s64 target;
+ s64 target, actual;
+ mutex_lock(&balloon_lock);
+ actual = vb->num_pages;
vb->vdev->config->get(vb->vdev,
offsetof(struct virtio_balloon_config, num_pages),
&v, sizeof(v));
target = le32_to_cpu(v);
- return target - vb->num_pages;
+ mutex_unlock(&balloon_lock);
+ return target - actual;
}
static void update_balloon_size(struct virtio_balloon *vb)
{
- __le32 actual = cpu_to_le32(vb->num_pages);
-
+ __le32 actual;
+ mutex_lock(&balloon_lock);
+ actual = cpu_to_le32(vb->num_pages);
vb->vdev->config->set(vb->vdev,
offsetof(struct virtio_balloon_config, actual),
&actual, sizeof(actual));
+ mutex_unlock(&balloon_lock);
}
static int balloon(void *_vballoon)
@@ -339,6 +375,76 @@ static int init_vqs(struct virtio_balloon *vb)
return 0;
}
+/*
+ * '*vb_ptr' allows virtballoon_migratepage() & virtballoon_putbackpage() to
+ * access pertinent elements from struct virtio_balloon
+ */
+struct virtio_balloon *vb_ptr;
+
+/*
+ * Populate balloon_mapping->a_ops->migratepage method to perform the balloon
+ * page migration task.
+ *
+ * After a ballooned page gets isolated by compaction procedures, this is the
+ * function that performs the page migration on behalf of move_to_new_page(),
+ * when the last calls (page)->mapping->a_ops->migratepage.
+ *
+ * Page migration for virtio balloon is done in a simple swap fashion which
+ * follows these two steps:
+ * 1) insert newpage into vb->pages list and update the host about it;
+ * 2) update the host about the removed old page from vb->pages list;
+ */
+int virtballoon_migratepage(struct address_space *mapping,
+ struct page *newpage, struct page *page, enum migrate_mode mode)
+{
+ mutex_lock(&balloon_lock);
+
+ /* balloon's page migration 1st step */
+ vb_ptr->num_pfns = VIRTIO_BALLOON_PAGES_PER_PAGE;
+ spin_lock(&pages_lock);
+ list_add(&newpage->lru, &vb_ptr->pages);
+ spin_unlock(&pages_lock);
+ set_page_pfns(vb_ptr->pfns, newpage);
+ tell_host(vb_ptr, vb_ptr->inflate_vq);
+
+ /* balloon's page migration 2nd step */
+ vb_ptr->num_pfns = VIRTIO_BALLOON_PAGES_PER_PAGE;
+ set_page_pfns(vb_ptr->pfns, page);
+ tell_host(vb_ptr, vb_ptr->deflate_vq);
+
+ mutex_unlock(&balloon_lock);
+
+ return 0;
+}
+
+/*
+ * Populate balloon_mapping->a_ops->invalidatepage method to help compaction on
+ * isolating a page from the balloon page list.
+ */
+void virtballoon_isolatepage(struct page *page, unsigned long mode)
+{
+ spin_lock(&pages_lock);
+ list_del(&page->lru);
+ spin_unlock(&pages_lock);
+}
+
+/*
+ * Populate balloon_mapping->a_ops->freepage method to help compaction on
+ * re-inserting an isolated page into the balloon page list.
+ */
+void virtballoon_putbackpage(struct page *page)
+{
+ spin_lock(&pages_lock);
+ list_add(&page->lru, &vb_ptr->pages);
+ spin_unlock(&pages_lock);
+}
+
+static const struct address_space_operations virtio_balloon_aops = {
+ .migratepage = virtballoon_migratepage,
+ .invalidatepage = virtballoon_isolatepage,
+ .freepage = virtballoon_putbackpage,
+};
+
static int virtballoon_probe(struct virtio_device *vdev)
{
struct virtio_balloon *vb;
@@ -351,11 +457,25 @@ static int virtballoon_probe(struct virtio_device *vdev)
}
INIT_LIST_HEAD(&vb->pages);
+
vb->num_pages = 0;
init_waitqueue_head(&vb->config_change);
init_waitqueue_head(&vb->acked);
vb->vdev = vdev;
vb->need_stats_update = 0;
+ vb_ptr = vb;
+
+ /* Init the ballooned page->mapping special balloon_mapping */
+ balloon_mapping = kmalloc(sizeof(*balloon_mapping), GFP_KERNEL);
+ if (!balloon_mapping) {
+ err = -ENOMEM;
+ goto out_free_vb;
+ }
+
+ INIT_RADIX_TREE(&balloon_mapping->page_tree, GFP_ATOMIC | __GFP_NOWARN);
+ INIT_LIST_HEAD(&balloon_mapping->i_mmap_nonlinear);
+ spin_lock_init(&balloon_mapping->tree_lock);
+ balloon_mapping->a_ops = &virtio_balloon_aops;
err = init_vqs(vb);
if (err)
@@ -373,6 +493,7 @@ out_del_vqs:
vdev->config->del_vqs(vdev);
out_free_vb:
kfree(vb);
+ kfree(balloon_mapping);
out:
return err;
}
@@ -397,6 +518,7 @@ static void __devexit virtballoon_remove(struct virtio_device *vdev)
kthread_stop(vb->thread);
remove_common(vb);
kfree(vb);
+ kfree(balloon_mapping);
}
#ifdef CONFIG_PM
diff --git a/include/linux/virtio_balloon.h b/include/linux/virtio_balloon.h
index 652dc8b..930f1b7 100644
--- a/include/linux/virtio_balloon.h
+++ b/include/linux/virtio_balloon.h
@@ -56,4 +56,8 @@ struct virtio_balloon_stat {
u64 val;
} __attribute__((packed));
+#if !defined(CONFIG_COMPACTION)
+struct address_space *balloon_mapping;
+#endif
+
#endif /* _LINUX_VIRTIO_BALLOON_H */
--
1.7.11.2
^ permalink raw reply related
* [PATCH v5 3/3] mm: add vm event counters for balloon pages compaction
From: Rafael Aquini @ 2012-08-06 13:56 UTC (permalink / raw)
To: linux-mm
Cc: Rik van Riel, Rafael Aquini, Konrad Rzeszutek Wilk,
Michael S. Tsirkin, linux-kernel, virtualization, Minchan Kim,
Andi Kleen, Andrew Morton
In-Reply-To: <cover.1344259054.git.aquini@redhat.com>
This patch is only for testing report purposes and shall be dropped in case of
the rest of this patchset getting accepted for merging.
Signed-off-by: Rafael Aquini <aquini@redhat.com>
---
drivers/virtio/virtio_balloon.c | 1 +
include/linux/vm_event_item.h | 2 ++
mm/compaction.c | 1 +
mm/migrate.c | 6 ++++--
mm/vmstat.c | 4 ++++
5 files changed, 12 insertions(+), 2 deletions(-)
diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
index 7c937a0..b8f7ea5 100644
--- a/drivers/virtio/virtio_balloon.c
+++ b/drivers/virtio/virtio_balloon.c
@@ -414,6 +414,7 @@ int virtballoon_migratepage(struct address_space *mapping,
mutex_unlock(&balloon_lock);
+ count_vm_event(COMPACTBALLOONMIGRATED);
return 0;
}
diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
index 57f7b10..a632a5d 100644
--- a/include/linux/vm_event_item.h
+++ b/include/linux/vm_event_item.h
@@ -41,6 +41,8 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
#ifdef CONFIG_COMPACTION
COMPACTBLOCKS, COMPACTPAGES, COMPACTPAGEFAILED,
COMPACTSTALL, COMPACTFAIL, COMPACTSUCCESS,
+ COMPACTBALLOONMIGRATED, COMPACTBALLOONFAILED,
+ COMPACTBALLOONISOLATED, COMPACTBALLOONFREED,
#endif
#ifdef CONFIG_HUGETLB_PAGE
HTLB_BUDDY_PGALLOC, HTLB_BUDDY_PGALLOC_FAIL,
diff --git a/mm/compaction.c b/mm/compaction.c
index 9499d85..4e2e46a 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -76,6 +76,7 @@ bool isolate_balloon_page(struct page *page)
if (is_balloon_page(page) && (page_count(page) == 2)) {
__isolate_balloon_page(page);
unlock_page(page);
+ count_vm_event(COMPACTBALLOONISOLATED);
return true;
}
unlock_page(page);
diff --git a/mm/migrate.c b/mm/migrate.c
index fc56968..f98804a 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -80,9 +80,10 @@ void putback_lru_pages(struct list_head *l)
dec_zone_page_state(page, NR_ISOLATED_ANON +
page_is_file_cache(page));
if (unlikely(is_balloon_page(page) &&
- balloon_compaction_enabled()))
+ balloon_compaction_enabled())) {
+ count_vm_event(COMPACTBALLOONFAILED);
WARN_ON(!putback_balloon_page(page));
- else
+ } else
putback_lru_page(page);
}
}
@@ -874,6 +875,7 @@ static int unmap_and_move(new_page_t get_new_page, unsigned long private,
page_is_file_cache(page));
put_page(page);
__free_page(page);
+ count_vm_event(COMPACTBALLOONFREED);
return rc;
}
out:
diff --git a/mm/vmstat.c b/mm/vmstat.c
index df7a674..8d80f60 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -768,6 +768,10 @@ const char * const vmstat_text[] = {
"compact_stall",
"compact_fail",
"compact_success",
+ "compact_balloon_migrated",
+ "compact_balloon_failed",
+ "compact_balloon_isolated",
+ "compact_balloon_freed",
#endif
#ifdef CONFIG_HUGETLB_PAGE
--
1.7.11.2
^ permalink raw reply related
* Re: [PATCH V5 4/4] virtio-blk: Add REQ_FLUSH and REQ_FUA support to bio path
From: Christoph Hellwig @ 2012-08-06 15:17 UTC (permalink / raw)
To: Asias He
Cc: Jens Axboe, kvm, Michael S. Tsirkin, linux-kernel, virtualization,
Tejun Heo, Shaohua Li, Christoph Hellwig
In-Reply-To: <501A2178.3000906@redhat.com>
On Thu, Aug 02, 2012 at 02:43:04PM +0800, Asias He wrote:
>> Even if it has a payload waiting is highly suboptimal and it should
>> use a non-blocking sequencing like it is done in the request layer.
>
> So, for REQ_FLUSH, what we need is that send out the VIRTIO_BLK_T_FLUSH and
> not to wait.
If it's REQ_FLUSH without data a VIRTIO_BLK_T_FLUSH should be sent out only,
if it's a REQ_FLUSH that has data a VIRTIO_BLK_T_FLUSH should be sent out,
but instead of waiting for it to finish the I/O completion handler should
then submit the actual write.
> We still need to wait until the actual write is finished here?
> Like,
>
> REQ_FUA is emulated by:
> 1. Send the actual write
> 2. Wait until the actual write is finished
> 3. Send VIRTIO_BLK_T_FLUSH to device
> 4. Signal the end of the write to upper layer
Remove step 2 and run step 3 from the I/O completion handler.
^ permalink raw reply
* Re: [PATCH v5 1/3] mm: introduce compaction and migration for virtio ballooned pages
From: Rik van Riel @ 2012-08-06 18:38 UTC (permalink / raw)
To: Rafael Aquini
Cc: Konrad Rzeszutek Wilk, Michael S. Tsirkin, linux-kernel,
virtualization, linux-mm, Andi Kleen, Minchan Kim, Andrew Morton
In-Reply-To: <212b5297df32cb4e3f60d5b76a8cb0629d328a4e.1344259054.git.aquini@redhat.com>
On 08/06/2012 09:56 AM, Rafael Aquini wrote:
> @@ -846,6 +861,21 @@ static int unmap_and_move(new_page_t get_new_page, unsigned long private,
> goto out;
>
> rc = __unmap_and_move(page, newpage, force, offlining, mode);
> +
> + if (unlikely(is_balloon_page(newpage)&&
> + balloon_compaction_enabled())) {
Could that be collapsed into one movable_balloon_page(newpage) function
call?
--
All rights reversed
^ permalink raw reply
* Re: [PATCH v5 1/3] mm: introduce compaction and migration for virtio ballooned pages
From: Rafael Aquini @ 2012-08-06 19:00 UTC (permalink / raw)
To: Rik van Riel
Cc: Konrad Rzeszutek Wilk, Michael S. Tsirkin, linux-kernel,
virtualization, linux-mm, Andi Kleen, Minchan Kim, Andrew Morton
In-Reply-To: <50200F1F.7060605@redhat.com>
On Mon, Aug 06, 2012 at 02:38:23PM -0400, Rik van Riel wrote:
> On 08/06/2012 09:56 AM, Rafael Aquini wrote:
>
> >@@ -846,6 +861,21 @@ static int unmap_and_move(new_page_t get_new_page, unsigned long private,
> > goto out;
> >
> > rc = __unmap_and_move(page, newpage, force, offlining, mode);
> >+
> >+ if (unlikely(is_balloon_page(newpage)&&
> >+ balloon_compaction_enabled())) {
>
> Could that be collapsed into one movable_balloon_page(newpage) function
> call?
>
Keeping is_balloon_page() as is, and itroducing this new movable_balloon_page()
function call, or just doing a plain rename, as Andrew has first suggested?
^ permalink raw reply
* Re: [PATCH v5 1/3] mm: introduce compaction and migration for virtio ballooned pages
From: Rik van Riel @ 2012-08-06 19:06 UTC (permalink / raw)
To: Rafael Aquini
Cc: Konrad Rzeszutek Wilk, Michael S. Tsirkin, linux-kernel,
virtualization, linux-mm, Andi Kleen, Minchan Kim, Andrew Morton
In-Reply-To: <20120806190053.GA3968@t510.redhat.com>
On 08/06/2012 03:00 PM, Rafael Aquini wrote:
> On Mon, Aug 06, 2012 at 02:38:23PM -0400, Rik van Riel wrote:
>> On 08/06/2012 09:56 AM, Rafael Aquini wrote:
>>
>>> @@ -846,6 +861,21 @@ static int unmap_and_move(new_page_t get_new_page, unsigned long private,
>>> goto out;
>>>
>>> rc = __unmap_and_move(page, newpage, force, offlining, mode);
>>> +
>>> + if (unlikely(is_balloon_page(newpage)&&
>>> + balloon_compaction_enabled())) {
>>
>> Could that be collapsed into one movable_balloon_page(newpage) function
>> call?
>>
> Keeping is_balloon_page() as is, and itroducing this new movable_balloon_page()
> function call, or just doing a plain rename, as Andrew has first suggested?
Just a plain rename would work.
> +static inline bool is_balloon_page(struct page *page)
> +{
> + return (page->mapping && page->mapping == balloon_mapping);
> +}
As an aside, since you are only comparing page->mapping and
not dereferencing it, it can be simplified to just:
return (page->mapping == balloon_mapping);
--
All rights reversed
^ permalink raw reply
* Re: [PATCH v5 1/3] mm: introduce compaction and migration for virtio ballooned pages
From: Rafael Aquini @ 2012-08-06 19:24 UTC (permalink / raw)
To: Rik van Riel
Cc: Konrad Rzeszutek Wilk, Michael S. Tsirkin, linux-kernel,
virtualization, linux-mm, Andi Kleen, Minchan Kim, Andrew Morton
In-Reply-To: <502015C9.2@redhat.com>
On Mon, Aug 06, 2012 at 03:06:49PM -0400, Rik van Riel wrote:
>
> Just a plain rename would work.
>
Ok, I will rename it.
> >+static inline bool is_balloon_page(struct page *page)
> >+{
> >+ return (page->mapping && page->mapping == balloon_mapping);
> >+}
>
> As an aside, since you are only comparing page->mapping and
> not dereferencing it, it can be simplified to just:
>
> return (page->mapping == balloon_mapping);
>
We really need both comparisons to avoid potential NULL pointer dereferences at
__isolate_balloon_page() & __putback_balloon_page() while running at bare metal
with no balloon driver loaded, since balloon_mapping itself is a pointer which
each balloon driver can set to its own structure.
Thanks, Rik, for taking the time to look at this patch and provide (always)
valuable feedback.
I'll shortly respin a v6 with your suggestions.
-- Rafael
^ permalink raw reply
* [PATCH V6 0/2] Improve virtio-blk performance
From: Asias He @ 2012-08-07 8:47 UTC (permalink / raw)
To: linux-kernel
Cc: Jens Axboe, kvm, Michael S. Tsirkin, virtualization, Tejun Heo,
Shaohua Li, Christoph Hellwig
Hi, all
This version reworked on REQ_FLUSH and REQ_FUA support as suggested by
Christoph and dropped the block core bits since Jens has picked them up.
Fio test shows bio-based IO path gives the following performance improvement:
1) Ramdisk device
With bio-based IO path, sequential read/write, random read/write
IOPS boost : 28%, 24%, 21%, 16%
Latency improvement: 32%, 17%, 21%, 16%
2) Fusion IO device
With bio-based IO path, sequential read/write, random read/write
IOPS boost : 11%, 11%, 13%, 10%
Latency improvement: 10%, 10%, 12%, 10%
Asias He (2):
virtio-blk: Add bio-based IO path for virtio-blk
virtio-blk: Add REQ_FLUSH and REQ_FUA support to bio path
drivers/block/virtio_blk.c | 306 +++++++++++++++++++++++++++++++++++++++------
1 file changed, 268 insertions(+), 38 deletions(-)
--
1.7.11.2
^ permalink raw reply
* [PATCH V6 1/2] virtio-blk: Add bio-based IO path for virtio-blk
From: Asias He @ 2012-08-07 8:47 UTC (permalink / raw)
To: linux-kernel
Cc: Jens Axboe, kvm, Michael S. Tsirkin, virtualization, Tejun Heo,
Shaohua Li, Christoph Hellwig
In-Reply-To: <1344329235-17449-1-git-send-email-asias@redhat.com>
This patch introduces bio-based IO path for virtio-blk.
Compared to request-based IO path, bio-based IO path uses driver
provided ->make_request_fn() method to bypasses the IO scheduler. It
handles the bio to device directly without allocating a request in block
layer. This reduces the IO path in guest kernel to achieve high IOPS
and lower latency. The downside is that guest can not use the IO
scheduler to merge and sort requests. However, this is not a big problem
if the backend disk in host side uses faster disk device.
When the bio-based IO path is not enabled, virtio-blk still uses the
original request-based IO path, no performance difference is observed.
Performance evaluation:
-----------------------------
1) Fio test is performed in a 8 vcpu guest with ramdisk based guest using
kvm tool.
Short version:
With bio-based IO path, sequential read/write, random read/write
IOPS boost : 28%, 24%, 21%, 16%
Latency improvement: 32%, 17%, 21%, 16%
Long version:
With bio-based IO path:
seq-read : io=2048.0MB, bw=116996KB/s, iops=233991 , runt= 17925msec
seq-write : io=2048.0MB, bw=100829KB/s, iops=201658 , runt= 20799msec
rand-read : io=3095.7MB, bw=112134KB/s, iops=224268 , runt= 28269msec
rand-write: io=3095.7MB, bw=96198KB/s, iops=192396 , runt= 32952msec
clat (usec): min=0 , max=2631.6K, avg=58716.99, stdev=191377.30
clat (usec): min=0 , max=1753.2K, avg=66423.25, stdev=81774.35
clat (usec): min=0 , max=2915.5K, avg=61685.70, stdev=120598.39
clat (usec): min=0 , max=1933.4K, avg=76935.12, stdev=96603.45
cpu : usr=74.08%, sys=703.84%, ctx=29661403, majf=21354, minf=22460954
cpu : usr=70.92%, sys=702.81%, ctx=77219828, majf=13980, minf=27713137
cpu : usr=72.23%, sys=695.37%, ctx=88081059, majf=18475, minf=28177648
cpu : usr=69.69%, sys=654.13%, ctx=145476035, majf=15867, minf=26176375
With request-based IO path:
seq-read : io=2048.0MB, bw=91074KB/s, iops=182147 , runt= 23027msec
seq-write : io=2048.0MB, bw=80725KB/s, iops=161449 , runt= 25979msec
rand-read : io=3095.7MB, bw=92106KB/s, iops=184211 , runt= 34416msec
rand-write: io=3095.7MB, bw=82815KB/s, iops=165630 , runt= 38277msec
clat (usec): min=0 , max=1932.4K, avg=77824.17, stdev=170339.49
clat (usec): min=0 , max=2510.2K, avg=78023.96, stdev=146949.15
clat (usec): min=0 , max=3037.2K, avg=74746.53, stdev=128498.27
clat (usec): min=0 , max=1363.4K, avg=89830.75, stdev=114279.68
cpu : usr=53.28%, sys=724.19%, ctx=37988895, majf=17531, minf=23577622
cpu : usr=49.03%, sys=633.20%, ctx=205935380, majf=18197, minf=27288959
cpu : usr=55.78%, sys=722.40%, ctx=101525058, majf=19273, minf=28067082
cpu : usr=56.55%, sys=690.83%, ctx=228205022, majf=18039, minf=26551985
2) Fio test is performed in a 8 vcpu guest with Fusion-IO based guest using
kvm tool.
Short version:
With bio-based IO path, sequential read/write, random read/write
IOPS boost : 11%, 11%, 13%, 10%
Latency improvement: 10%, 10%, 12%, 10%
Long Version:
With bio-based IO path:
read : io=2048.0MB, bw=58920KB/s, iops=117840 , runt= 35593msec
write: io=2048.0MB, bw=64308KB/s, iops=128616 , runt= 32611msec
read : io=3095.7MB, bw=59633KB/s, iops=119266 , runt= 53157msec
write: io=3095.7MB, bw=62993KB/s, iops=125985 , runt= 50322msec
clat (usec): min=0 , max=1284.3K, avg=128109.01, stdev=71513.29
clat (usec): min=94 , max=962339 , avg=116832.95, stdev=65836.80
clat (usec): min=0 , max=1846.6K, avg=128509.99, stdev=89575.07
clat (usec): min=0 , max=2256.4K, avg=121361.84, stdev=82747.25
cpu : usr=56.79%, sys=421.70%, ctx=147335118, majf=21080, minf=19852517
cpu : usr=61.81%, sys=455.53%, ctx=143269950, majf=16027, minf=24800604
cpu : usr=63.10%, sys=455.38%, ctx=178373538, majf=16958, minf=24822612
cpu : usr=62.04%, sys=453.58%, ctx=226902362, majf=16089, minf=23278105
With request-based IO path:
read : io=2048.0MB, bw=52896KB/s, iops=105791 , runt= 39647msec
write: io=2048.0MB, bw=57856KB/s, iops=115711 , runt= 36248msec
read : io=3095.7MB, bw=52387KB/s, iops=104773 , runt= 60510msec
write: io=3095.7MB, bw=57310KB/s, iops=114619 , runt= 55312msec
clat (usec): min=0 , max=1532.6K, avg=142085.62, stdev=109196.84
clat (usec): min=0 , max=1487.4K, avg=129110.71, stdev=114973.64
clat (usec): min=0 , max=1388.6K, avg=145049.22, stdev=107232.55
clat (usec): min=0 , max=1465.9K, avg=133585.67, stdev=110322.95
cpu : usr=44.08%, sys=590.71%, ctx=451812322, majf=14841, minf=17648641
cpu : usr=48.73%, sys=610.78%, ctx=418953997, majf=22164, minf=26850689
cpu : usr=45.58%, sys=581.16%, ctx=714079216, majf=21497, minf=22558223
cpu : usr=48.40%, sys=599.65%, ctx=656089423, majf=16393, minf=23824409
How to use:
-----------------------------
Add 'virtio_blk.use_bio=1' to kernel cmdline or 'modprobe virtio_blk
use_bio=1' to enable ->make_request_fn() based I/O path.
Cc: Rusty Russell <rusty@rustcorp.com.au>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Tejun Heo <tj@kernel.org>
Cc: Shaohua Li <shli@kernel.org>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: kvm@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Cc: virtualization@lists.linux-foundation.org
Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Minchan Kim <minchan.kim@gmail.com>
Signed-off-by: Asias He <asias@redhat.com>
Acked-by: Rusty Russell <rusty@rustcorp.com.au>
---
drivers/block/virtio_blk.c | 203 ++++++++++++++++++++++++++++++++++++---------
1 file changed, 163 insertions(+), 40 deletions(-)
diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
index c0bbeb4..95cfeed 100644
--- a/drivers/block/virtio_blk.c
+++ b/drivers/block/virtio_blk.c
@@ -14,6 +14,9 @@
#define PART_BITS 4
+static bool use_bio;
+module_param(use_bio, bool, S_IRUGO);
+
static int major;
static DEFINE_IDA(vd_index_ida);
@@ -23,6 +26,7 @@ struct virtio_blk
{
struct virtio_device *vdev;
struct virtqueue *vq;
+ wait_queue_head_t queue_wait;
/* The disk structure for the kernel. */
struct gendisk *disk;
@@ -51,53 +55,87 @@ struct virtio_blk
struct virtblk_req
{
struct request *req;
+ struct bio *bio;
struct virtio_blk_outhdr out_hdr;
struct virtio_scsi_inhdr in_hdr;
u8 status;
+ struct scatterlist sg[];
};
-static void blk_done(struct virtqueue *vq)
+static inline int virtblk_result(struct virtblk_req *vbr)
+{
+ switch (vbr->status) {
+ case VIRTIO_BLK_S_OK:
+ return 0;
+ case VIRTIO_BLK_S_UNSUPP:
+ return -ENOTTY;
+ default:
+ return -EIO;
+ }
+}
+
+static inline void virtblk_request_done(struct virtio_blk *vblk,
+ struct virtblk_req *vbr)
+{
+ struct request *req = vbr->req;
+ int error = virtblk_result(vbr);
+
+ if (req->cmd_type == REQ_TYPE_BLOCK_PC) {
+ req->resid_len = vbr->in_hdr.residual;
+ req->sense_len = vbr->in_hdr.sense_len;
+ req->errors = vbr->in_hdr.errors;
+ } else if (req->cmd_type == REQ_TYPE_SPECIAL) {
+ req->errors = (error != 0);
+ }
+
+ __blk_end_request_all(req, error);
+ mempool_free(vbr, vblk->pool);
+}
+
+static inline void virtblk_bio_done(struct virtio_blk *vblk,
+ struct virtblk_req *vbr)
+{
+ bio_endio(vbr->bio, virtblk_result(vbr));
+ mempool_free(vbr, vblk->pool);
+}
+
+static void virtblk_done(struct virtqueue *vq)
{
struct virtio_blk *vblk = vq->vdev->priv;
+ unsigned long bio_done = 0, req_done = 0;
struct virtblk_req *vbr;
- unsigned int len;
unsigned long flags;
+ unsigned int len;
spin_lock_irqsave(vblk->disk->queue->queue_lock, flags);
while ((vbr = virtqueue_get_buf(vblk->vq, &len)) != NULL) {
- int error;
-
- switch (vbr->status) {
- case VIRTIO_BLK_S_OK:
- error = 0;
- break;
- case VIRTIO_BLK_S_UNSUPP:
- error = -ENOTTY;
- break;
- default:
- error = -EIO;
- break;
- }
-
- switch (vbr->req->cmd_type) {
- case REQ_TYPE_BLOCK_PC:
- vbr->req->resid_len = vbr->in_hdr.residual;
- vbr->req->sense_len = vbr->in_hdr.sense_len;
- vbr->req->errors = vbr->in_hdr.errors;
- break;
- case REQ_TYPE_SPECIAL:
- vbr->req->errors = (error != 0);
- break;
- default:
- break;
+ if (vbr->bio) {
+ virtblk_bio_done(vblk, vbr);
+ bio_done++;
+ } else {
+ virtblk_request_done(vblk, vbr);
+ req_done++;
}
-
- __blk_end_request_all(vbr->req, error);
- mempool_free(vbr, vblk->pool);
}
/* In case queue is stopped waiting for more buffers. */
- blk_start_queue(vblk->disk->queue);
+ if (req_done)
+ blk_start_queue(vblk->disk->queue);
spin_unlock_irqrestore(vblk->disk->queue->queue_lock, flags);
+
+ if (bio_done)
+ wake_up(&vblk->queue_wait);
+}
+
+static inline struct virtblk_req *virtblk_alloc_req(struct virtio_blk *vblk,
+ gfp_t gfp_mask)
+{
+ struct virtblk_req *vbr;
+
+ vbr = mempool_alloc(vblk->pool, gfp_mask);
+ if (vbr && use_bio)
+ sg_init_table(vbr->sg, vblk->sg_elems);
+
+ return vbr;
}
static bool do_req(struct request_queue *q, struct virtio_blk *vblk,
@@ -106,13 +144,13 @@ static bool do_req(struct request_queue *q, struct virtio_blk *vblk,
unsigned long num, out = 0, in = 0;
struct virtblk_req *vbr;
- vbr = mempool_alloc(vblk->pool, GFP_ATOMIC);
+ vbr = virtblk_alloc_req(vblk, GFP_ATOMIC);
if (!vbr)
/* When another request finishes we'll try again. */
return false;
vbr->req = req;
-
+ vbr->bio = NULL;
if (req->cmd_flags & REQ_FLUSH) {
vbr->out_hdr.type = VIRTIO_BLK_T_FLUSH;
vbr->out_hdr.sector = 0;
@@ -172,7 +210,8 @@ static bool do_req(struct request_queue *q, struct virtio_blk *vblk,
}
}
- if (virtqueue_add_buf(vblk->vq, vblk->sg, out, in, vbr, GFP_ATOMIC)<0) {
+ if (virtqueue_add_buf(vblk->vq, vblk->sg, out, in, vbr,
+ GFP_ATOMIC) < 0) {
mempool_free(vbr, vblk->pool);
return false;
}
@@ -180,7 +219,7 @@ static bool do_req(struct request_queue *q, struct virtio_blk *vblk,
return true;
}
-static void do_virtblk_request(struct request_queue *q)
+static void virtblk_request(struct request_queue *q)
{
struct virtio_blk *vblk = q->queuedata;
struct request *req;
@@ -203,6 +242,82 @@ static void do_virtblk_request(struct request_queue *q)
virtqueue_kick(vblk->vq);
}
+static void virtblk_add_buf_wait(struct virtio_blk *vblk,
+ struct virtblk_req *vbr,
+ unsigned long out,
+ unsigned long in)
+{
+ DEFINE_WAIT(wait);
+
+ for (;;) {
+ prepare_to_wait_exclusive(&vblk->queue_wait, &wait,
+ TASK_UNINTERRUPTIBLE);
+
+ spin_lock_irq(vblk->disk->queue->queue_lock);
+ if (virtqueue_add_buf(vblk->vq, vbr->sg, out, in, vbr,
+ GFP_ATOMIC) < 0) {
+ spin_unlock_irq(vblk->disk->queue->queue_lock);
+ io_schedule();
+ } else {
+ virtqueue_kick(vblk->vq);
+ spin_unlock_irq(vblk->disk->queue->queue_lock);
+ break;
+ }
+
+ }
+
+ finish_wait(&vblk->queue_wait, &wait);
+}
+
+static void virtblk_make_request(struct request_queue *q, struct bio *bio)
+{
+ struct virtio_blk *vblk = q->queuedata;
+ unsigned int num, out = 0, in = 0;
+ struct virtblk_req *vbr;
+
+ BUG_ON(bio->bi_phys_segments + 2 > vblk->sg_elems);
+ BUG_ON(bio->bi_rw & (REQ_FLUSH | REQ_FUA));
+
+ vbr = virtblk_alloc_req(vblk, GFP_NOIO);
+ if (!vbr) {
+ bio_endio(bio, -ENOMEM);
+ return;
+ }
+
+ vbr->bio = bio;
+ vbr->req = NULL;
+ vbr->out_hdr.type = 0;
+ vbr->out_hdr.sector = bio->bi_sector;
+ vbr->out_hdr.ioprio = bio_prio(bio);
+
+ sg_set_buf(&vbr->sg[out++], &vbr->out_hdr, sizeof(vbr->out_hdr));
+
+ num = blk_bio_map_sg(q, bio, vbr->sg + out);
+
+ sg_set_buf(&vbr->sg[num + out + in++], &vbr->status,
+ sizeof(vbr->status));
+
+ if (num) {
+ if (bio->bi_rw & REQ_WRITE) {
+ vbr->out_hdr.type |= VIRTIO_BLK_T_OUT;
+ out += num;
+ } else {
+ vbr->out_hdr.type |= VIRTIO_BLK_T_IN;
+ in += num;
+ }
+ }
+
+ spin_lock_irq(vblk->disk->queue->queue_lock);
+ if (unlikely(virtqueue_add_buf(vblk->vq, vbr->sg, out, in, vbr,
+ GFP_ATOMIC) < 0)) {
+ spin_unlock_irq(vblk->disk->queue->queue_lock);
+ virtblk_add_buf_wait(vblk, vbr, out, in);
+ return;
+ }
+ virtqueue_kick(vblk->vq);
+ spin_unlock_irq(vblk->disk->queue->queue_lock);
+}
+
/* return id (s/n) string for *disk to *id_str
*/
static int virtblk_get_id(struct gendisk *disk, char *id_str)
@@ -360,7 +475,7 @@ static int init_vq(struct virtio_blk *vblk)
int err = 0;
/* We expect one virtqueue, for output. */
- vblk->vq = virtio_find_single_vq(vblk->vdev, blk_done, "requests");
+ vblk->vq = virtio_find_single_vq(vblk->vdev, virtblk_done, "requests");
if (IS_ERR(vblk->vq))
err = PTR_ERR(vblk->vq);
@@ -414,7 +529,7 @@ static void virtblk_update_cache_mode(struct virtio_device *vdev)
u8 writeback = virtblk_get_cache_mode(vdev);
struct virtio_blk *vblk = vdev->priv;
- if (writeback)
+ if (writeback && !use_bio)
blk_queue_flush(vblk->disk->queue, REQ_FLUSH);
else
blk_queue_flush(vblk->disk->queue, 0);
@@ -477,6 +592,8 @@ static int __devinit virtblk_probe(struct virtio_device *vdev)
struct virtio_blk *vblk;
struct request_queue *q;
int err, index;
+ int pool_size;
+
u64 cap;
u32 v, blk_size, sg_elems, opt_io_size;
u16 min_io_size;
@@ -506,10 +623,12 @@ static int __devinit virtblk_probe(struct virtio_device *vdev)
goto out_free_index;
}
+ init_waitqueue_head(&vblk->queue_wait);
vblk->vdev = vdev;
vblk->sg_elems = sg_elems;
sg_init_table(vblk->sg, vblk->sg_elems);
mutex_init(&vblk->config_lock);
+
INIT_WORK(&vblk->config_work, virtblk_config_changed_work);
vblk->config_enable = true;
@@ -517,7 +636,10 @@ static int __devinit virtblk_probe(struct virtio_device *vdev)
if (err)
goto out_free_vblk;
- vblk->pool = mempool_create_kmalloc_pool(1,sizeof(struct virtblk_req));
+ pool_size = sizeof(struct virtblk_req);
+ if (use_bio)
+ pool_size += sizeof(struct scatterlist) * sg_elems;
+ vblk->pool = mempool_create_kmalloc_pool(1, pool_size);
if (!vblk->pool) {
err = -ENOMEM;
goto out_free_vq;
@@ -530,12 +652,14 @@ static int __devinit virtblk_probe(struct virtio_device *vdev)
goto out_mempool;
}
- q = vblk->disk->queue = blk_init_queue(do_virtblk_request, NULL);
+ q = vblk->disk->queue = blk_init_queue(virtblk_request, NULL);
if (!q) {
err = -ENOMEM;
goto out_put_disk;
}
+ if (use_bio)
+ blk_queue_make_request(q, virtblk_make_request);
q->queuedata = vblk;
virtblk_name_format("vd", index, vblk->disk->disk_name, DISK_NAME_LEN);
@@ -620,7 +744,6 @@ static int __devinit virtblk_probe(struct virtio_device *vdev)
if (!err && opt_io_size)
blk_queue_io_opt(q, blk_size * opt_io_size);
-
add_disk(vblk->disk);
err = device_create_file(disk_to_dev(vblk->disk), &dev_attr_serial);
if (err)
--
1.7.11.2
^ permalink raw reply related
* [PATCH V6 2/2] virtio-blk: Add REQ_FLUSH and REQ_FUA support to bio path
From: Asias He @ 2012-08-07 8:47 UTC (permalink / raw)
To: linux-kernel
Cc: Jens Axboe, kvm, Michael S. Tsirkin, virtualization, Tejun Heo,
Shaohua Li, Christoph Hellwig
In-Reply-To: <1344329235-17449-1-git-send-email-asias@redhat.com>
We need to support both REQ_FLUSH and REQ_FUA for bio based path since
it does not get the sequencing of REQ_FUA into REQ_FLUSH that request
based drivers can request.
REQ_FLUSH is emulated by:
A) If the bio has no data to write:
1. Send VIRTIO_BLK_T_FLUSH to device,
2. In the flush I/O completion handler, finish the bio
B) If the bio has data to write:
1. Send VIRTIO_BLK_T_FLUSH to device
2. In the flush I/O completion handler, send the actual write data to device
3. In the write I/O completion handler, finish the bio
REQ_FUA is emulated by:
1. Send the actual write data to device
2. In the write I/O completion handler, send VIRTIO_BLK_T_FLUSH to device
3. In the flush I/O completion handler, finish the bio
Cc: Rusty Russell <rusty@rustcorp.com.au>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Tejun Heo <tj@kernel.org>
Cc: Shaohua Li <shli@kernel.org>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: kvm@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Cc: virtualization@lists.linux-foundation.org
Signed-off-by: Asias He <asias@redhat.com>
---
drivers/block/virtio_blk.c | 259 ++++++++++++++++++++++++++++++++-------------
1 file changed, 183 insertions(+), 76 deletions(-)
diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
index 95cfeed..d33ea48 100644
--- a/drivers/block/virtio_blk.c
+++ b/drivers/block/virtio_blk.c
@@ -58,6 +58,12 @@ struct virtblk_req
struct bio *bio;
struct virtio_blk_outhdr out_hdr;
struct virtio_scsi_inhdr in_hdr;
+ struct work_struct work;
+ struct virtio_blk *vblk;
+ bool is_flush;
+ bool req_flush;
+ bool req_data;
+ bool req_fua;
u8 status;
struct scatterlist sg[];
};
@@ -74,6 +80,128 @@ static inline int virtblk_result(struct virtblk_req *vbr)
}
}
+static inline struct virtblk_req *virtblk_alloc_req(struct virtio_blk *vblk,
+ gfp_t gfp_mask)
+{
+ struct virtblk_req *vbr;
+
+ vbr = mempool_alloc(vblk->pool, gfp_mask);
+ if (vbr && use_bio)
+ sg_init_table(vbr->sg, vblk->sg_elems);
+
+ return vbr;
+}
+
+static void virtblk_add_buf_wait(struct virtio_blk *vblk,
+ struct virtblk_req *vbr,
+ unsigned long out,
+ unsigned long in)
+{
+ DEFINE_WAIT(wait);
+
+ for (;;) {
+ prepare_to_wait_exclusive(&vblk->queue_wait, &wait,
+ TASK_UNINTERRUPTIBLE);
+
+ spin_lock_irq(vblk->disk->queue->queue_lock);
+ if (virtqueue_add_buf(vblk->vq, vbr->sg, out, in, vbr,
+ GFP_ATOMIC) < 0) {
+ spin_unlock_irq(vblk->disk->queue->queue_lock);
+ io_schedule();
+ } else {
+ virtqueue_kick(vblk->vq);
+ spin_unlock_irq(vblk->disk->queue->queue_lock);
+ break;
+ }
+
+ }
+
+ finish_wait(&vblk->queue_wait, &wait);
+}
+
+static inline void virtblk_add_req(struct virtio_blk *vblk,
+ struct virtblk_req *vbr,
+ unsigned int out, unsigned int in)
+{
+ spin_lock_irq(vblk->disk->queue->queue_lock);
+ if (unlikely(virtqueue_add_buf(vblk->vq, vbr->sg, out, in, vbr,
+ GFP_ATOMIC) < 0)) {
+ spin_unlock_irq(vblk->disk->queue->queue_lock);
+ virtblk_add_buf_wait(vblk, vbr, out, in);
+ return;
+ }
+ virtqueue_kick(vblk->vq);
+ spin_unlock_irq(vblk->disk->queue->queue_lock);
+}
+
+static int virtblk_bio_send_flush(struct virtio_blk *vblk,
+ struct virtblk_req *vbr)
+{
+ unsigned int out = 0, in = 0;
+
+ vbr->is_flush = true;
+ vbr->out_hdr.type = VIRTIO_BLK_T_FLUSH;
+ vbr->out_hdr.sector = 0;
+ vbr->out_hdr.ioprio = 0;
+ sg_set_buf(&vbr->sg[out++], &vbr->out_hdr, sizeof(vbr->out_hdr));
+ sg_set_buf(&vbr->sg[out + in++], &vbr->status, sizeof(vbr->status));
+
+ virtblk_add_req(vblk, vbr, out, in);
+
+ return 0;
+}
+
+static int virtblk_bio_send_data(struct virtio_blk *vblk,
+ struct virtblk_req *vbr)
+{
+ unsigned int num, out = 0, in = 0;
+ struct bio *bio = vbr->bio;
+
+ vbr->is_flush = false;
+ vbr->out_hdr.type = 0;
+ vbr->out_hdr.sector = bio->bi_sector;
+ vbr->out_hdr.ioprio = bio_prio(bio);
+
+ sg_set_buf(&vbr->sg[out++], &vbr->out_hdr, sizeof(vbr->out_hdr));
+
+ num = blk_bio_map_sg(vblk->disk->queue, bio, vbr->sg + out);
+
+ sg_set_buf(&vbr->sg[num + out + in++], &vbr->status,
+ sizeof(vbr->status));
+
+ if (num) {
+ if (bio->bi_rw & REQ_WRITE) {
+ vbr->out_hdr.type |= VIRTIO_BLK_T_OUT;
+ out += num;
+ } else {
+ vbr->out_hdr.type |= VIRTIO_BLK_T_IN;
+ in += num;
+ }
+ }
+
+ virtblk_add_req(vblk, vbr, out, in);
+
+ return 0;
+}
+
+static void virtblk_bio_send_data_work(struct work_struct *work)
+{
+ struct virtblk_req *vbr;
+
+ vbr = container_of(work, struct virtblk_req, work);
+
+ virtblk_bio_send_data(vbr->vblk, vbr);
+}
+
+static void virtblk_bio_send_flush_work(struct work_struct *work)
+{
+ struct virtblk_req *vbr;
+
+ vbr = container_of(work, struct virtblk_req, work);
+
+ virtblk_bio_send_flush(vbr->vblk, vbr);
+}
+
static inline void virtblk_request_done(struct virtio_blk *vblk,
struct virtblk_req *vbr)
{
@@ -92,13 +220,53 @@ static inline void virtblk_request_done(struct virtio_blk *vblk,
mempool_free(vbr, vblk->pool);
}
-static inline void virtblk_bio_done(struct virtio_blk *vblk,
- struct virtblk_req *vbr)
+static inline void virtblk_bio_done_flush(struct virtio_blk *vblk,
+ struct virtblk_req *vbr)
{
- bio_endio(vbr->bio, virtblk_result(vbr));
+ if (vbr->req_data) {
+ /* Send out the actual write data */
+ struct virtblk_req *_vbr;
+ _vbr = virtblk_alloc_req(vblk, GFP_NOIO);
+ if (!_vbr) {
+ bio_endio(vbr->bio, -ENOMEM);
+ goto out;
+ }
+ _vbr->req_fua = vbr->req_fua;
+ _vbr->bio = vbr->bio;
+ _vbr->vblk = vblk;
+ INIT_WORK(&_vbr->work, virtblk_bio_send_data_work);
+ queue_work(virtblk_wq, &_vbr->work);
+ } else {
+ bio_endio(vbr->bio, virtblk_result(vbr));
+ }
+out:
mempool_free(vbr, vblk->pool);
}
+static inline void virtblk_bio_done_data(struct virtio_blk *vblk,
+ struct virtblk_req *vbr)
+{
+ if (unlikely(vbr->req_fua)) {
+ /* Send out a flush before end the bio */
+ struct virtblk_req *_vbr;
+ _vbr = virtblk_alloc_req(vblk, GFP_NOIO);
+ if (!_vbr) {
+ bio_endio(vbr->bio, -ENOMEM);
+ goto out;
+ }
+ _vbr->req_data = false;
+ _vbr->bio = vbr->bio;
+ _vbr->vblk = vblk;
+ INIT_WORK(&_vbr->work, virtblk_bio_send_flush_work);
+ queue_work(virtblk_wq, &_vbr->work);
+ } else {
+ bio_endio(vbr->bio, virtblk_result(vbr));
+ }
+out:
+ mempool_free(vbr, vblk->pool);
+}
+
+
static void virtblk_done(struct virtqueue *vq)
{
struct virtio_blk *vblk = vq->vdev->priv;
@@ -110,7 +278,10 @@ static void virtblk_done(struct virtqueue *vq)
spin_lock_irqsave(vblk->disk->queue->queue_lock, flags);
while ((vbr = virtqueue_get_buf(vblk->vq, &len)) != NULL) {
if (vbr->bio) {
- virtblk_bio_done(vblk, vbr);
+ if (unlikely(vbr->is_flush))
+ virtblk_bio_done_flush(vblk, vbr);
+ else
+ virtblk_bio_done_data(vblk, vbr);
bio_done++;
} else {
virtblk_request_done(vblk, vbr);
@@ -126,18 +297,6 @@ static void virtblk_done(struct virtqueue *vq)
wake_up(&vblk->queue_wait);
}
-static inline struct virtblk_req *virtblk_alloc_req(struct virtio_blk *vblk,
- gfp_t gfp_mask)
-{
- struct virtblk_req *vbr;
-
- vbr = mempool_alloc(vblk->pool, gfp_mask);
- if (vbr && use_bio)
- sg_init_table(vbr->sg, vblk->sg_elems);
-
- return vbr;
-}
-
static bool do_req(struct request_queue *q, struct virtio_blk *vblk,
struct request *req)
{
@@ -242,41 +401,12 @@ static void virtblk_request(struct request_queue *q)
virtqueue_kick(vblk->vq);
}
-static void virtblk_add_buf_wait(struct virtio_blk *vblk,
- struct virtblk_req *vbr,
- unsigned long out,
- unsigned long in)
-{
- DEFINE_WAIT(wait);
-
- for (;;) {
- prepare_to_wait_exclusive(&vblk->queue_wait, &wait,
- TASK_UNINTERRUPTIBLE);
-
- spin_lock_irq(vblk->disk->queue->queue_lock);
- if (virtqueue_add_buf(vblk->vq, vbr->sg, out, in, vbr,
- GFP_ATOMIC) < 0) {
- spin_unlock_irq(vblk->disk->queue->queue_lock);
- io_schedule();
- } else {
- virtqueue_kick(vblk->vq);
- spin_unlock_irq(vblk->disk->queue->queue_lock);
- break;
- }
-
- }
-
- finish_wait(&vblk->queue_wait, &wait);
-}
-
static void virtblk_make_request(struct request_queue *q, struct bio *bio)
{
struct virtio_blk *vblk = q->queuedata;
- unsigned int num, out = 0, in = 0;
struct virtblk_req *vbr;
BUG_ON(bio->bi_phys_segments + 2 > vblk->sg_elems);
- BUG_ON(bio->bi_rw & (REQ_FLUSH | REQ_FUA));
vbr = virtblk_alloc_req(vblk, GFP_NOIO);
if (!vbr) {
@@ -284,38 +414,15 @@ static void virtblk_make_request(struct request_queue *q, struct bio *bio)
return;
}
+ vbr->req_flush = !!(bio->bi_rw & REQ_FLUSH);
+ vbr->req_fua = !!(bio->bi_rw & REQ_FUA);
+ vbr->req_data = !!(bio->bi_size);
vbr->bio = bio;
- vbr->req = NULL;
- vbr->out_hdr.type = 0;
- vbr->out_hdr.sector = bio->bi_sector;
- vbr->out_hdr.ioprio = bio_prio(bio);
-
- sg_set_buf(&vbr->sg[out++], &vbr->out_hdr, sizeof(vbr->out_hdr));
-
- num = blk_bio_map_sg(q, bio, vbr->sg + out);
- sg_set_buf(&vbr->sg[num + out + in++], &vbr->status,
- sizeof(vbr->status));
-
- if (num) {
- if (bio->bi_rw & REQ_WRITE) {
- vbr->out_hdr.type |= VIRTIO_BLK_T_OUT;
- out += num;
- } else {
- vbr->out_hdr.type |= VIRTIO_BLK_T_IN;
- in += num;
- }
- }
-
- spin_lock_irq(vblk->disk->queue->queue_lock);
- if (unlikely(virtqueue_add_buf(vblk->vq, vbr->sg, out, in, vbr,
- GFP_ATOMIC) < 0)) {
- spin_unlock_irq(vblk->disk->queue->queue_lock);
- virtblk_add_buf_wait(vblk, vbr, out, in);
- return;
- }
- virtqueue_kick(vblk->vq);
- spin_unlock_irq(vblk->disk->queue->queue_lock);
+ if (unlikely(vbr->req_flush))
+ virtblk_bio_send_flush(vblk, vbr);
+ else
+ virtblk_bio_send_data(vblk, vbr);
}
/* return id (s/n) string for *disk to *id_str
@@ -529,7 +636,7 @@ static void virtblk_update_cache_mode(struct virtio_device *vdev)
u8 writeback = virtblk_get_cache_mode(vdev);
struct virtio_blk *vblk = vdev->priv;
- if (writeback && !use_bio)
+ if (writeback)
blk_queue_flush(vblk->disk->queue, REQ_FLUSH);
else
blk_queue_flush(vblk->disk->queue, 0);
--
1.7.11.2
^ permalink raw reply related
* Re: [PATCH V5 4/4] virtio-blk: Add REQ_FLUSH and REQ_FUA support to bio path
From: Asias He @ 2012-08-07 8:50 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Jens Axboe, kvm, Michael S. Tsirkin, linux-kernel, virtualization,
Tejun Heo, Shaohua Li
In-Reply-To: <20120806151731.GA23220@lst.de>
On 08/06/2012 11:17 PM, Christoph Hellwig wrote:
> On Thu, Aug 02, 2012 at 02:43:04PM +0800, Asias He wrote:
>>> Even if it has a payload waiting is highly suboptimal and it should
>>> use a non-blocking sequencing like it is done in the request layer.
>>
>> So, for REQ_FLUSH, what we need is that send out the VIRTIO_BLK_T_FLUSH and
>> not to wait.
>
> If it's REQ_FLUSH without data a VIRTIO_BLK_T_FLUSH should be sent out only,
> if it's a REQ_FLUSH that has data a VIRTIO_BLK_T_FLUSH should be sent out,
> but instead of waiting for it to finish the I/O completion handler should
> then submit the actual write.
>
>> We still need to wait until the actual write is finished here?
>> Like,
>>
>> REQ_FUA is emulated by:
>> 1. Send the actual write
>> 2. Wait until the actual write is finished
>> 3. Send VIRTIO_BLK_T_FLUSH to device
>> 4. Signal the end of the write to upper layer
>
> Remove step 2 and run step 3 from the I/O completion handler.
>
Thanks for the explanation. V6 is in flight.
--
Asias
^ permalink raw reply
* Re: [PATCH V6 2/2] virtio-blk: Add REQ_FLUSH and REQ_FUA support to bio path
From: Christoph Hellwig @ 2012-08-07 9:15 UTC (permalink / raw)
To: Asias He
Cc: Jens Axboe, kvm, Michael S. Tsirkin, linux-kernel, virtualization,
Tejun Heo, Shaohua Li, Christoph Hellwig
In-Reply-To: <1344329235-17449-3-git-send-email-asias@redhat.com>
At least after review is done I really think this patch sopuld be folded
into the previous one.
Some more comments below:
> @@ -58,6 +58,12 @@ struct virtblk_req
> struct bio *bio;
> struct virtio_blk_outhdr out_hdr;
> struct virtio_scsi_inhdr in_hdr;
> + struct work_struct work;
> + struct virtio_blk *vblk;
I think using bio->bi_private for the virtio_blk pointer would
be cleaner.
> + bool is_flush;
> + bool req_flush;
> + bool req_data;
> + bool req_fua;
This could be a bitmap, or better even a single state field.
> +static int virtblk_bio_send_flush(struct virtio_blk *vblk,
> + struct virtblk_req *vbr)
> +static int virtblk_bio_send_data(struct virtio_blk *vblk,
> + struct virtblk_req *vbr)
These should only get the struct virtblk_req * argument as the virtio_blk
structure is easily derivable from it.
> +static inline void virtblk_bio_done_flush(struct virtio_blk *vblk,
> + struct virtblk_req *vbr)
> {
> + if (vbr->req_data) {
> + /* Send out the actual write data */
> + struct virtblk_req *_vbr;
> + _vbr = virtblk_alloc_req(vblk, GFP_NOIO);
> + if (!_vbr) {
> + bio_endio(vbr->bio, -ENOMEM);
> + goto out;
> + }
> + _vbr->req_fua = vbr->req_fua;
> + _vbr->bio = vbr->bio;
> + _vbr->vblk = vblk;
> + INIT_WORK(&_vbr->work, virtblk_bio_send_data_work);
> + queue_work(virtblk_wq, &_vbr->work);
The _vbr naming isn't too nice. Also can you explain why the original
request can't be reused in a comment here?
Also if using a state variable I think the whole code would be
a bit cleaner if the bio_done helpers are combined.
> - if (writeback && !use_bio)
> + if (writeback)
> blk_queue_flush(vblk->disk->queue, REQ_FLUSH);
Shouldn't this be REQ_FLUSH | REQ_FUA for the bio case?
Btw, did you verify that flushes really work correctly for all cases
using tracing in qemu?
^ permalink raw reply
* Re: [PATCH V6 0/2] Improve virtio-blk performance
From: Christoph Hellwig @ 2012-08-07 9:16 UTC (permalink / raw)
To: Asias He
Cc: Jens Axboe, kvm, Michael S. Tsirkin, linux-kernel, virtualization,
Tejun Heo, Shaohua Li, Christoph Hellwig
In-Reply-To: <1344329235-17449-1-git-send-email-asias@redhat.com>
On Tue, Aug 07, 2012 at 04:47:13PM +0800, Asias He wrote:
> 1) Ramdisk device
> With bio-based IO path, sequential read/write, random read/write
> IOPS boost : 28%, 24%, 21%, 16%
> Latency improvement: 32%, 17%, 21%, 16%
> 2) Fusion IO device
> With bio-based IO path, sequential read/write, random read/write
> IOPS boost : 11%, 11%, 13%, 10%
> Latency improvement: 10%, 10%, 12%, 10%
Do you also have numbers for normal SAS/SATA disks? The changelog should
have a reall good explanation of why this is optional, and numbers are a
very important part of that.
^ permalink raw reply
* ICAC2012 Call for Participation (9th ACM International Conference on Autonomic Computing)
From: Ming Zhao @ 2012-08-07 12:06 UTC (permalink / raw)
To: virtualization
**********************************************************************
CALL FOR PARTICIPATION
======================
The 9th ACM International Conference on Autonomic Computing (ICAC 2012)
San Jose, California, USA
September 17-21, 2012
http://icac2012.cs.fiu.edu
Sponsored by ACM
**********************************************************************
Online registration is open at
http://icac2012.cs.fiu.edu/registration.shtm
Reduced fees are available for those registering by September 2, 2012.
This year's technical program features:
- 3 distinguished keynote speakers:
* Dr. Amin Vahdat, Google/UCSD
* Dr. Subutai Ahmad (VP Engineering), Numenta
* Dr. Eitan Frachtenberg, Facebook
- 24 outstanding technical papers (15 full + 9 short):
* covering core and emerging topics such as clouds, virtualization,
control, monitoring and diagnosis, and energy
* Half of the papers involve authors from industry or government labs
- 4 co-located workshops covering hot topics in:
* Feedback Computing
* Self-Aware Internet of Things
* Management of Big Data Systems
* Federated Clouds
The Conference will be held at the Fairmont Hotel in downtown San Jose, CA, USA.
**********************************************************************
IMPORTANT DATES
===============
Early registration deadline: September 2, 2012
Hotel special rate deadline: September 7, 2012
**********************************************************************
CORPORATE SPONSORS
==================
Gold Level Partner: IBM
Conference partners: VMware, HP, Neustar
PhD Student Sponsor: Google
Other Sponsor: NEC Labs
**********************************************************************
PRELIMINARY PROGRAM
===================
======================================================================
MONDAY, SEPTEMBER 17, 2012 - WORKSHOPS
Feedback Computing 2012
Self-Aware Internet of Things 2012
======================================================================
TUESDAY, SEPTEMBER 18, 2012 - MAIN CONFERENCE
8:00AM - 8:45AM
Registration
8:45AM - 9:00AM
Welcome Remarks
9:00AM - 10:00AM
Keynote Talk I
Symbiosis in Scale Out Networking and Data Management
Amin Vahdat, Google and University of California, San Diego
10:00AM - 10.30AM
Break
10:30AM - 12:00PM
Session : Virtualization
Net-Cohort: Detecting and Managing VM Ensembles in Virtualized Data Centers
Liting Hu, Karsten Schwan (Georgia Institute of Technology);
Ajay Gulati (VMware); Junjie Zhang, Chengwei Wang (Georgia Institute
of Technology)
Application-aware Cross-layer Virtual Machine Resource Management
Lixi Wang, Jing Xu, Ming Zhao (Florida International University)
Shifting GEARS to Enable Guest-context Virtual Services
Kyle Hale, Lei Xia, Peter Dinda (Northwestern University)
12:00PM-1:30PM
Lunch
----------------------------------------------------------------------
1:30PM - 3:30PM
Session: Performance and Resource Management
When Average is Not Average: Large Response Time Fluctuations in N-tier Systems
Qingyang Wang (Georgia Institute of Technology); Yasuhiko Kanemasa,
Motoyuki Kawaba (Fujitsu Laboratories Ltd.); Calton Pu (Georgia Institute
of Technology)
Provisioning Multi-tier Cloud Applications Using Statistical Bounds on
Sojourn Time
Upendra Sharma, Prashant Shenoy, Don Towsley (University of Massachusetts Amherst)
Automated Profiling and Resource Management of Pig Programs for Meeting
Service Level Objectives
Zhuoyao Zhang (University of Pennsylvania); Ludmila Cherkasova (Hewlett-Packard Labs);
Abhishek Verma (University of Illinois at Urbana-Champaign);
Boon Thau Loo (University of Pennsylvania)
AROMA: Automated Resource Allocation and Configuration of MapReduce Environment
in the Cloud
Palden Lama, Xiaobo Zhou (University of Colorado at Colorado Springs)
3:30PM - 4:00PM
Break
4:00PM - 5:15PM
Short Papers I
Locomotion@Location: When the Rubber hits the Road
Gerold Hoelzl, Marc Kurz, Alois Ferscha (Johannes Kepler University Linz, Austria)
An Autonomic Resource Provisioning Framework for Mobile Computing Grids
Hariharasudhan Viswanathan, Eun Kyung Lee, Ivan Rodero, Dario Pompili (Rutgers University)
A Self-Tuning Self-Optimizing Approach for Automated Network Anomaly Detection Systems
Dennis Ippoliti, Xiaobo Zhou (University of Colorado at Colorado Springs)
Offline and On-Demand Event Correlation for Operations Management of Large Scale IT Systems
Chetan Gupta (Hewlett-Packard Labs)
PowerTracer: Tracing Requests in Multi-tier Services to Diagnose Energy Inefficiency
Gang Lu, Jianfeng Zhan (Institute of Computing Technology, Chinese Academy of Sciences);
Haining Wang (College of William and Mary); Lin Yuan (Institute of Computing Technology,
Chinese Academy of Sciences); Chuliang Weng (Shanghai Jiao Tong University, China)
6:00PM - 9:00PM
Conference Dinner
======================================================================
WEDNESDAY, SEPTEMBER 19 - MAIN CONFERENCE
8:00AM - 9:00AM
Registration
9:00AM - 10:00AM
Keynote Talk II
Automated Machine Learning For Autonomic Computing
Subutai Ahmad, VP Engineering, Numenta
10:00AM - 10:30AM
Break
10:30AM - 12:00PM
Session: Control-Based Approaches
Budget-based Control for Interactive Services with Adaptive Execution
Yuxiong He, Zihao Ye, Qiang Fu, Sameh Elnikety (Microsoft Research)
On the Design of Decentralized Control Architectures for Workload Consolidation in
Large-Scale Server Clusters
Rui Wang, Nagarajan Kandasamy (Drexel University)
Transactional Auto Scaler: Elastic Scaling of In-Memory Transactional Data Grids
Diego Didona, Paolo Romano (Instituto Superior Tecnico/INESC-ID); Sebastiano Peluso,
Francesco Quaglia (Sapienza, Universita di Roma)
12:00PM - 1:30PM
Lunch
----------------------------------------------------------------------
1:30PM - 2:30PM
Session: Energy
Adaptive Green Hosting
Nan Deng, Christopher Stewart, Jaimie Kelley (The Ohio State University); Daniel Gmach,
Martin Arlitt (Hewlett Packard Labs)
Dynamic Energy-Aware Capacity Provisioning for Cloud Computing Environments
Qi Zhang, Mohamed Faten Zhani (University of Waterloo); Shuo Zhang (National University
of Defense Technology); Quanyan Zhu (University of Illinois at Urbana-Champaign);
Raouf Boutaba (University of Waterloo); Joseph L. Hellerstein (Google, Inc.)
2:30PM - 3:30PM
Short Papers II
VESPA: Multi-Layered Self-Protection for Cloud Resources
Aurelien Wailly, Marc Lacoste (Orange Labs); Herve Debar (Teleom SudParis)
Usage Patterns in Multi-tenant Data Centers: a Temporal Perspective
Robert Birke, Lydia Y. Chen (IBM Research Zurich Lab); Evgenia Smirni (College of
William and Mary)
Toward Fast Eventual Consistency with Performance Guarantees
Feng Yan (College of William and Mary); Alma Riska (EMC Corporation); Evgenia Smirni (College
of William and Mary)
Optimal Autoscaling in the IaaS Cloud
Hamoun Ghanbari, Bradley Simmons, Marin Litoiu, Cornel Barna (York University);
Gabriel Iszlai (IBM Toronto)
3:30PM - 4:00PM
Break
4:00PM - 6:00PM
Poster and Demo Session
6:00PM - 9:00PM
Conference Outing (tentative)
======================================================================
THURSDAY, SEPTEMBER 20, 2012 - MAIN CONFERENCE
8:00AM - 9:00AM
Registration
9:00AM - 10:00AM
Keynote Talk III
High Efficiency at Web Scale
Eitan Frachtenberg, Facebook
10:00AM - 10:30AM
Break
10:30AM - 12:00PM
Session: Diagnosis and Monitoring
Chair: TBD
3-Dimensional Root Cause Diagnosis via Co-analysi
Ziming Zheng, Li Yu, Zhiling Lan (Illinois Institute of Technology); Terry Jones
(Oak Ridge National Laboratory)
UBL: Unsupervised Behavior Learning for Predicting Performance Anomalies in Virtualized Cloud Systems
Daniel J. Dean, Hiep Nguyen, Xiaohui Gu (North Carolina State University)
Evaluating Compressive Sampling Strategies for Performance Monitoring of Data Centers
Tingshan Huang, Nagarajan Kandasamy, Harish Sethu (Drexel University)
12:00PM
Adjourn
======================================================================
FRIDAY, SEPTEMBER 21, 2012 - WORKSHOPS
Management of Big Data Systems 2012
Federated Clouds 2012
**********************************************************************
ORGANIZERS
==========
GENERAL CHAIR
Dejan Milojicic, HP Labs
PROGRAM CHAIRS
Dongyan Xu, Purdue University
Vanish Talwar, HP Labs
INDUSTRY CHAIR
Xiaoyun Zhu, VMware
WORKSHOPS CHAIR
Fred Douglis, EMC
POSTERS/DEMO/EXHIBITS CHAIR
Eno Thereska, Microsoft Research
FINANCE CHAIR
Michael Kozuch, Intel
LOCAL ARRANGEMENT CHAIR
Jessica Blaine
PUBLICITY CHAIRS
Daniel Batista, University of Sao Paulo
Vartan Padaryan, ISP/Russian Academy of Sci.
Ioan Raicu, Illinois Inst. of Technology
Jianfeng Zhan, ICT/Chinese Academy of Sci.
Ming Zhao, Florida Intl. University
PROGRAM COMMITTEE
Tarek Abdelzaher, UIUC
Umesh Bellur, IIT, Bombay
Ken Birman, Cornell University
Rajkumar Buyya, Univ. of Melbourne
Rocky Chang, Hong Kong Polytechnic University
Yuan Chen, HP Labs
Alva Couch, Tufts University
Peter Dinda, Northwestern University
Fred Douglis, EMC
Renato Figueiredo, University of Florida
Mohamed Hefeeda, QCRI
Joe Hellerstein, Google
Geoff Jiang, NEC Labs
Jeff Kephart, IBM Research
Emre Kiciman, Microsoft Research
Fabio Kon, University of Sao Paulo
Mike Kozuch, Intel Labs
Dejan Milojicic, HP Labs
Klara Nahrstedt, UIUC
Priya Narasimhan, CMU
Manish Parashar, Rutgers University
Ioan Raicu, Illinois Inst. of Technology
Omer Rana, Cardiff University
Masoud Sadjadi, Florida Intl. University
Richard Schlichting, AT&T Labs
Hartmut Schmeck, KIT
Karsten Schwan, Georgia Tech
Onn Shehory, IBM Research
Eno Thereska, Microsoft Research
Xiaoyun Zhu, VMware
**********************************************************************
^ permalink raw reply
* Call for Participation: The 9th ACM International Conference on Autonomic Computing (ICAC 2012) -- September 17-21, San Jose CA
From: Ioan Raicu @ 2012-08-08 3:46 UTC (permalink / raw)
To: virtualization
**********************************************************************
CALL FOR PARTICIPATION
======================
The 9th ACM International Conference on Autonomic Computing (ICAC 2012)
San Jose, California, USA
September 17-21, 2012
http://icac2012.cs.fiu.edu
Sponsored by ACM
**********************************************************************
Online registration is open at
http://icac2012.cs.fiu.edu/registration.shtm
Reduced fees are available for those registering by September 2, 2012.
This year's technical program features:
- 3 distinguished keynote speakers:
* Dr. Amin Vahdat, Google/UCSD
* Dr. Subutai Ahmad (VP Engineering), Numenta
* Dr. Eitan Frachtenberg, Facebook
- 24 outstanding technical papers (15 full + 9 short):
* covering core and emerging topics such as clouds, virtualization,
control, monitoring and diagnosis, and energy
* Half of the papers involve authors from industry or government labs
- 4 co-located workshops covering hot topics in:
* Feedback Computing
* Self-Aware Internet of Things
* Management of Big Data Systems
* Federated Clouds
The Conference will be held at the Fairmont Hotel in downtown San Jose,
CA, USA.
**********************************************************************
IMPORTANT DATES
===============
Early registration deadline: September 2, 2012
Hotel special rate deadline: September 7, 2012
**********************************************************************
CORPORATE SPONSORS
==================
Gold Level Partner: IBM
Conference partners: VMware, HP, Neustar
PhD Student Sponsor: Google
Other Sponsor: NEC Labs
**********************************************************************
PRELIMINARY PROGRAM
===================
======================================================================
MONDAY, SEPTEMBER 17, 2012 - WORKSHOPS
Feedback Computing 2012
Self-Aware Internet of Things 2012
======================================================================
TUESDAY, SEPTEMBER 18, 2012 - MAIN CONFERENCE
8:00AM – 8:45AM
Registration
8:45AM – 9:00AM
Welcome Remarks
9:00AM – 10:00AM
Keynote Talk I
Symbiosis in Scale Out Networking and Data Management
Amin Vahdat, Google and University of California, San Diego
10:00AM – 10.30AM
Break
10:30AM – 12:00PM
Session : Virtualization
Net-Cohort: Detecting and Managing VM Ensembles in Virtualized Data Centers
Liting Hu, Karsten Schwan (Georgia Institute of Technology);
Ajay Gulati (VMware); Junjie Zhang, Chengwei Wang (Georgia Institute
of Technology)
Application-aware Cross-layer Virtual Machine Resource Management
Lixi Wang, Jing Xu, Ming Zhao (Florida International University)
Shifting GEARS to Enable Guest-context Virtual Services
Kyle Hale, Lei Xia, Peter Dinda (Northwestern University)
12:00PM-1:30PM
Lunch
----------------------------------------------------------------------
1:30PM - 3:30PM
Session: Performance and Resource Management
When Average is Not Average: Large Response Time Fluctuations in N-tier
Systems
Qingyang Wang (Georgia Institute of Technology); Yasuhiko Kanemasa,
Motoyuki Kawaba (Fujitsu Laboratories Ltd.); Calton Pu (Georgia Institute
of Technology)
Provisioning Multi-tier Cloud Applications Using Statistical Bounds on
Sojourn Time
Upendra Sharma, Prashant Shenoy, Don Towsley (University of
Massachusetts Amherst)
Automated Profiling and Resource Management of Pig Programs for Meeting
Service Level Objectives
Zhuoyao Zhang (University of Pennsylvania); Ludmila Cherkasova
(Hewlett-Packard Labs);
Abhishek Verma (University of Illinois at Urbana-Champaign);
Boon Thau Loo (University of Pennsylvania)
AROMA: Automated Resource Allocation and Configuration of MapReduce
Environment
in the Cloud
Palden Lama, Xiaobo Zhou (University of Colorado at Colorado Springs)
3:30PM-4:00PM
Break
4:00PM – 5:15PM
Short Papers I
Locomotion@Location: When the Rubber hits the Road
Gerold Hoelzl, Marc Kurz, Alois Ferscha (Johannes Kepler University
Linz, Austria)
An Autonomic Resource Provisioning Framework for Mobile Computing Grids
Hariharasudhan Viswanathan, Eun Kyung Lee, Ivan Rodero, Dario Pompili
(Rutgers University)
A Self-Tuning Self-Optimizing Approach for Automated Network Anomaly
Detection Systems
Dennis Ippoliti, Xiaobo Zhou (University of Colorado at Colorado Springs)
Offline and On-Demand Event Correlation for Operations Management of
Large Scale IT Systems
Chetan Gupta (Hewlett-Packard Labs)
PowerTracer: Tracing Requests in Multi-tier Services to Diagnose Energy
Inefficiency
Gang Lu, Jianfeng Zhan (Institute of Computing Technology, Chinese
Academy of Sciences);
Haining Wang (College of William and Mary); Lin Yuan (Institute of
Computing Technology,
Chinese Academy of Sciences); Chuliang Weng (Shanghai Jiao Tong
University, China)
6:00PM - 9:00PM
Conference Dinner
======================================================================
WEDNESDAY, SEPTEMBER 19 - MAIN CONFERENCE
8:00AM – 9:00AM
Registration
9:00AM – 10:00AM
Keynote Talk II
Automated Machine Learning For Autonomic Computing
Subutai Ahmad, VP Engineering, Numenta
10:00AM - 10:30AM
Break
10:30AM – 12:00PM
Session: Control-Based Approaches
Budget-based Control for Interactive Services with Adaptive Execution
Yuxiong He, Zihao Ye, Qiang Fu, Sameh Elnikety (Microsoft Research)
On the Design of Decentralized Control Architectures for Workload
Consolidation in
Large-Scale Server Clusters
Rui Wang, Nagarajan Kandasamy (Drexel University)
Transactional Auto Scaler: Elastic Scaling of In-Memory Transactional
Data Grids
Diego Didona, Paolo Romano (Instituto Superior Técnico/INESC-ID);
Sebastiano Peluso,
Francesco Quaglia (Sapienza, Università di Roma)
12:00PM - 1:30PM
Lunch
----------------------------------------------------------------------
1:30PM - 2:30PM
Session: Energy
Adaptive Green Hosting
Nan Deng, Christopher Stewart, Jaimie Kelley (The Ohio State
University); Daniel Gmach,
Martin Arlitt (Hewlett Packard Labs)
Dynamic Energy-Aware Capacity Provisioning for Cloud Computing Environments
Qi Zhang, Mohamed Faten Zhani (University of Waterloo); Shuo Zhang
(National University
of Defense Technology); Quanyan Zhu (University of Illinois at
Urbana-Champaign);
Raouf Boutaba (University of Waterloo); Joseph L. Hellerstein (Google,
Inc.)
2:30PM - 3:30PM
Short Papers II
VESPA: Multi-Layered Self-Protection for Cloud Resources
Aurélien Wailly, Marc Lacoste (Orange Labs); Hervé Debar (Télécom SudParis)
Usage Patterns in Multi-tenant Data Centers: a Temporal Perspective
Robert Birke, Lydia Y. Chen (IBM Research Zurich Lab); Evgenia Smirni
(College of
William and Mary)
Toward Fast Eventual Consistency with Performance Guarantees
Feng Yan (College of William and Mary); Alma Riska (EMC Corporation);
Evgenia Smirni (College
of William and Mary)
Optimal Autoscaling in the IaaS Cloud
Hamoun Ghanbari, Bradley Simmons, Marin Litoiu, Cornel Barna (York
University);
Gabriel Iszlai (IBM Toronto)
3:30PM – 4:00PM
Break
4:00PM - 6:00PM
Poster and Demo Session
6:00PM-9:00PM
Conference Outing (tentative)
======================================================================
THURSDAY, SEPTEMBER 20, 2012 - MAIN CONFERENCE
8:00AM – 9:00AM
Registration
9:00AM – 10:00AM
Keynote Talk III
High Efficiency at Web Scale
Eitan Frachtenberg, Facebook
10:00AM-10:30AM
Break
10:30AM - 12:00PM
Session: Diagnosis and Monitoring
Chair: TBD
3-Dimensional Root Cause Diagnosis via Co-analysi
Ziming Zheng, Li Yu, Zhiling Lan (Illinois Institute of Technology);
Terry Jones
(Oak Ridge National Laboratory)
UBL: Unsupervised Behavior Learning for Predicting Performance Anomalies
in Virtualized Cloud Systems
Daniel J. Dean, Hiep Nguyen, Xiaohui Gu (North Carolina State University)
Evaluating Compressive Sampling Strategies for Performance Monitoring of
Data Centers
Tingshan Huang, Nagarajan Kandasamy, Harish Sethu (Drexel University)
12:00PM
Adjourn
======================================================================
FRIDAY, SEPTEMBER 21, 2012 - WORKSHOPS
Management of Big Data Systems 2012
Federated Clouds 2012
**********************************************************************
ORGANIZERS
==========
GENERAL CHAIR
Dejan Milojicic, HP Labs
PROGRAM CHAIRS
Dongyan Xu, Purdue University
Vanish Talwar, HP Labs
INDUSTRY CHAIR
Xiaoyun Zhu, VMware
WORKSHOPS CHAIR
Fred Douglis, EMC
POSTERS/DEMO/EXHIBITS CHAIR
Eno Thereska, Microsoft Research
FINANCE CHAIR
Michael Kozuch, Intel
LOCAL ARRANGEMENT CHAIR
Jessica Blaine
PUBLICITY CHAIRS
Daniel Batista, University of São Paulo
Vartan Padaryan, ISP/Russian Academy of Sci.
Ioan Raicu, Illinois Inst. of Technology
Jianfeng Zhan, ICT/Chinese Academy of Sci.
Ming Zhao, Florida Intl. University
PROGRAM COMMITTEE
Tarek Abdelzaher, UIUC
Umesh Bellur, IIT, Bombay
Ken Birman, Cornell University
Rajkumar Buyya, Univ. of Melbourne
Rocky Chang, Hong Kong Polytechnic University
Yuan Chen, HP Labs
Alva Couch, Tufts University
Peter Dinda, Northwestern University
Fred Douglis, EMC
Renato Figueiredo, University of Florida
Mohamed Hefeeda, QCRI
Joe Hellerstein, Google
Geoff Jiang, NEC Labs
Jeff Kephart, IBM Research
Emre Kiciman, Microsoft Research
Fabio Kon, University of São Paulo
Mike Kozuch, Intel Labs
Dejan Milojicic, HP Labs
Klara Nahrstedt, UIUC
Priya Narasimhan, CMU
Manish Parashar, Rutgers University
Ioan Raicu, Illinois Inst. of Technology
Omer Rana, Cardiff University
Masoud Sadjadi, Florida Intl. University
Richard Schlichting, AT&T Labs
Hartmut Schmeck, KIT
Karsten Schwan, Georgia Tech
Onn Shehory, IBM Research
Eno Thereska, Microsoft Research
Xiaoyun Zhu, VMware
**********************************************************************
--
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor, Illinois Institute of Technology (IIT)
Guest Research Faculty, Argonne National Laboratory (ANL)
=================================================================
Data-Intensive Distributed Systems Laboratory, CS/IIT
Distributed Systems Laboratory, MCS/ANL
=================================================================
Cel: 1-847-722-0876
Office: 1-312-567-5704
Email: iraicu@cs.iit.edu
Web: http://www.cs.iit.edu/~iraicu/
Web: http://datasys.cs.iit.edu/
=================================================================
=================================================================
^ permalink raw reply
* Re: [PATCH V6 2/2] virtio-blk: Add REQ_FLUSH and REQ_FUA support to bio path
From: Asias He @ 2012-08-08 6:20 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Jens Axboe, kvm, Michael S. Tsirkin, linux-kernel, virtualization,
Tejun Heo, Shaohua Li
In-Reply-To: <20120807091510.GA2651@lst.de>
On 08/07/2012 05:15 PM, Christoph Hellwig wrote:
> At least after review is done I really think this patch sopuld be folded
> into the previous one.
OK.
> Some more comments below:
>
>> @@ -58,6 +58,12 @@ struct virtblk_req
>> struct bio *bio;
>> struct virtio_blk_outhdr out_hdr;
>> struct virtio_scsi_inhdr in_hdr;
>> + struct work_struct work;
>> + struct virtio_blk *vblk;
>
> I think using bio->bi_private for the virtio_blk pointer would
> be cleaner.
I wish I could use bio->bi_private but I am seeing this when using
bio->bi_priate to store virito_blk pointer:
[ 1.100335] Call Trace:
[ 1.100335] <IRQ>
[ 1.100335] [<ffffffff811dd4b0>] ? end_bio_bh_io_sync+0x30/0x50
[ 1.100335] [<ffffffff811e167d>] bio_endio+0x1d/0x40
[ 1.100335] [<ffffffff81551fb2>] virtblk_done+0xa2/0x260
[ 1.100335] [<ffffffff813d714d>] vring_interrupt+0x2d/0x40
[ 1.100335] [<ffffffff81119c0d>] handle_irq_event_percpu+0x6d/0x210
[ 1.100335] [<ffffffff81119df1>] handle_irq_event+0x41/0x70
[ 1.100335] [<ffffffff8111d2c9>] handle_edge_irq+0x69/0x120
[ 1.100335] [<ffffffff810613a2>] handle_irq+0x22/0x30
[ 1.100335] [<ffffffff81aadccd>] do_IRQ+0x5d/0xe0
[ 1.100335] [<ffffffff81aa432f>] common_interrupt+0x6f/0x6f
end_bio_bh_io_sync() uses bio->private:
struct buffer_head *bh = bio->bi_private;
>
>> + bool is_flush;
>> + bool req_flush;
>> + bool req_data;
>> + bool req_fua;
>
> This could be a bitmap, or better even a single state field.
Will use a bitmap for now.
>> +static int virtblk_bio_send_flush(struct virtio_blk *vblk,
>> + struct virtblk_req *vbr)
>
>> +static int virtblk_bio_send_data(struct virtio_blk *vblk,
>> + struct virtblk_req *vbr)
>
> These should only get the struct virtblk_req * argument as the virtio_blk
> structure is easily derivable from it.
Yes. Will clean it up.
>> +static inline void virtblk_bio_done_flush(struct virtio_blk *vblk,
>> + struct virtblk_req *vbr)
>> {
>> + if (vbr->req_data) {
>> + /* Send out the actual write data */
>> + struct virtblk_req *_vbr;
>> + _vbr = virtblk_alloc_req(vblk, GFP_NOIO);
>> + if (!_vbr) {
>> + bio_endio(vbr->bio, -ENOMEM);
>> + goto out;
>> + }
>> + _vbr->req_fua = vbr->req_fua;
>> + _vbr->bio = vbr->bio;
>> + _vbr->vblk = vblk;
>> + INIT_WORK(&_vbr->work, virtblk_bio_send_data_work);
>> + queue_work(virtblk_wq, &_vbr->work);
>
> The _vbr naming isn't too nice. Also can you explain why the original
> request can't be reused in a comment here
Ah, the original request can be reused. Will fix this.
> Also if using a state variable I think the whole code would be
> a bit cleaner if the bio_done helpers are combined.
>
>> - if (writeback && !use_bio)
>> + if (writeback)
>> blk_queue_flush(vblk->disk->queue, REQ_FLUSH);
>
> Shouldn't this be REQ_FLUSH | REQ_FUA for the bio case?
Without REQ_FUA, I am also seeing bio with REQ_FUA flag set. Do we need
to set REQ_FUA explicitly?
> Btw, did you verify that flushes really work correctly for all cases
> using tracing in qemu?
I added some debug code in both kernel and kvm tool to verity the flush.
--
Asias
^ permalink raw reply
* Re: [PATCH V6 0/2] Improve virtio-blk performance
From: Asias He @ 2012-08-08 6:24 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Jens Axboe, kvm, Michael S. Tsirkin, linux-kernel, virtualization,
Tejun Heo, Shaohua Li
In-Reply-To: <5021D87C.4050400@redhat.com>
On 08/08/2012 11:09 AM, Asias He wrote:
> On 08/07/2012 05:16 PM, Christoph Hellwig wrote:
>> On Tue, Aug 07, 2012 at 04:47:13PM +0800, Asias He wrote:
>>> 1) Ramdisk device
>>> With bio-based IO path, sequential read/write, random read/write
>>> IOPS boost : 28%, 24%, 21%, 16%
>>> Latency improvement: 32%, 17%, 21%, 16%
>>> 2) Fusion IO device
>>> With bio-based IO path, sequential read/write, random read/write
>>> IOPS boost : 11%, 11%, 13%, 10%
>>> Latency improvement: 10%, 10%, 12%, 10%
>>
>> Do you also have numbers for normal SAS/SATA disks? The changelog should
>> have a reall good explanation of why this is optional, and numbers are a
>> very important part of that.
Yes. I posted the numbers on normal SATA disks a few days ago in the
thread with Sasha. Will add that data and explanation of why optional to
the changelog
--
Asias
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox