[Qemu-devel] [PATCH v3 1/1] s390-ccw-virtio: allow for systems larger that 7.999TB

[Qemu-devel] [PATCH v3 1/1] s390-ccw-virtio: allow for systems larger that 7.999TB

[Christian Borntraeger]

KVM does not allow memory regions > KVM_MEM_MAX_NR_PAGES, basically
limiting the memory per slot to 8TB-4k. As memory slots on s390/kvm must
be a multiple of 1MB we need start a new memory region if we cross
8TB-1M.

With that (and optimistic overcommitment in the kernel) I was able to
start a 24TB guest on a 1TB system.

Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
---
 hw/s390x/s390-virtio-ccw.c | 28 +++++++++++++++++++++++++---
 1 file changed, 25 insertions(+), 3 deletions(-)

diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c
index 8425534..073f6ed 100644
--- a/hw/s390x/s390-virtio-ccw.c
+++ b/hw/s390x/s390-virtio-ccw.c
@@ -154,14 +154,36 @@ static void virtio_ccw_register_hcalls(void)
                                    virtio_ccw_hcall_early_printk);
 }
 
+/*
+ * KVM does only support memory slots up to KVM_MEM_MAX_NR_PAGES pages
+ * as the dirty bitmap must be managed by bitops that take an int as
+ * position indicator. If we have a guest beyond that we will split off
+ * new subregions. The split must happen on a segment boundary (1MB).
+ */
+#define KVM_MEM_MAX_NR_PAGES ((1UL << 31) - 1)
+#define SEG_MSK (~0xfffffULL)
+#define KVM_SLOT_MAX_BYTES ((KVM_MEM_MAX_NR_PAGES * TARGET_PAGE_SIZE) & SEG_MSK)
 static void s390_memory_init(ram_addr_t mem_size)
 {
     MemoryRegion *sysmem = get_system_memory();
-    MemoryRegion *ram = g_new(MemoryRegion, 1);
+    ram_addr_t chunk, offset = 0;
+    unsigned int number = 0;
+    gchar *name;
 
     /* allocate RAM for core */
-    memory_region_allocate_system_memory(ram, NULL, "s390.ram", mem_size);
-    memory_region_add_subregion(sysmem, 0, ram);
+    name = g_strdup_printf("s390.ram");
+    while (mem_size) {
+        MemoryRegion *ram = g_new(MemoryRegion, 1);
+        /* KVM does not allow memslots >= 8 TB */
+        chunk = MIN(mem_size, KVM_SLOT_MAX_BYTES);
+        memory_region_allocate_system_memory(ram, NULL, name, chunk);
+        memory_region_add_subregion(sysmem, offset, ram);
+        mem_size -= chunk;
+        offset += chunk;
+        g_free(name);
+        name = g_strdup_printf("s390.ram.%u", ++number);
+    }
+    g_free(name);
 
     /* Initialize storage key device */
     s390_skeys_init();
-- 
2.9.4

Re: [Qemu-devel] [qemu-s390x] [PATCH v3 1/1] s390-ccw-virtio: allow for systems larger that 7.999TB

[David Hildenbrand]

On 11.12.2017 13:21, Christian Borntraeger wrote:
> KVM does not allow memory regions > KVM_MEM_MAX_NR_PAGES, basically
> limiting the memory per slot to 8TB-4k. As memory slots on s390/kvm must
> be a multiple of 1MB we need start a new memory region if we cross
> 8TB-1M.
> 
> With that (and optimistic overcommitment in the kernel) I was able to
> start a 24TB guest on a 1TB system.
> 
> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> ---
>  hw/s390x/s390-virtio-ccw.c | 28 +++++++++++++++++++++++++---
>  1 file changed, 25 insertions(+), 3 deletions(-)
> 
> diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c
> index 8425534..073f6ed 100644
> --- a/hw/s390x/s390-virtio-ccw.c
> +++ b/hw/s390x/s390-virtio-ccw.c
> @@ -154,14 +154,36 @@ static void virtio_ccw_register_hcalls(void)
>                                     virtio_ccw_hcall_early_printk);
>  }
>  
> +/*
> + * KVM does only support memory slots up to KVM_MEM_MAX_NR_PAGES pages
> + * as the dirty bitmap must be managed by bitops that take an int as
> + * position indicator. If we have a guest beyond that we will split off
> + * new subregions. The split must happen on a segment boundary (1MB).
> + */
> +#define KVM_MEM_MAX_NR_PAGES ((1UL << 31) - 1)
> +#define SEG_MSK (~0xfffffULL)
> +#define KVM_SLOT_MAX_BYTES ((KVM_MEM_MAX_NR_PAGES * TARGET_PAGE_SIZE) & SEG_MSK)

Just wondering if we could get into trouble when calculating

KVM_MEM_MAX_NR_PAGES * TARGET_PAGE_SIZE

on a host with sizeof(long) == 4

could it wrap? (e.g. crazy mingw stuff)

>  static void s390_memory_init(ram_addr_t mem_size)
>  {
>      MemoryRegion *sysmem = get_system_memory();
> -    MemoryRegion *ram = g_new(MemoryRegion, 1);
> +    ram_addr_t chunk, offset = 0;
> +    unsigned int number = 0;
> +    gchar *name;
>  
>      /* allocate RAM for core */
> -    memory_region_allocate_system_memory(ram, NULL, "s390.ram", mem_size);
> -    memory_region_add_subregion(sysmem, 0, ram);
> +    name = g_strdup_printf("s390.ram");
> +    while (mem_size) {
> +        MemoryRegion *ram = g_new(MemoryRegion, 1);

I'd add an empty line here.

Reviewed-by: David Hildenbrand <david@redhat.com>

> +        /* KVM does not allow memslots >= 8 TB */
> +        chunk = MIN(mem_size, KVM_SLOT_MAX_BYTES);
> +        memory_region_allocate_system_memory(ram, NULL, name, chunk);
> +        memory_region_add_subregion(sysmem, offset, ram);
> +        mem_size -= chunk;
> +        offset += chunk;
> +        g_free(name);
> +        name = g_strdup_printf("s390.ram.%u", ++number);
> +    }
> +    g_free(name);
>  
>      /* Initialize storage key device */
>      s390_skeys_init();
> 


-- 

Thanks,

David / dhildenb

Re: [Qemu-devel] [qemu-s390x] [PATCH v3 1/1] s390-ccw-virtio: allow for systems larger that 7.999TB

[Christian Borntraeger]

On 12/11/2017 02:55 PM, David Hildenbrand wrote:
> On 11.12.2017 13:21, Christian Borntraeger wrote:
>> KVM does not allow memory regions > KVM_MEM_MAX_NR_PAGES, basically
>> limiting the memory per slot to 8TB-4k. As memory slots on s390/kvm must
>> be a multiple of 1MB we need start a new memory region if we cross
>> 8TB-1M.
>>
>> With that (and optimistic overcommitment in the kernel) I was able to
>> start a 24TB guest on a 1TB system.
>>
>> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
>> ---
>>  hw/s390x/s390-virtio-ccw.c | 28 +++++++++++++++++++++++++---
>>  1 file changed, 25 insertions(+), 3 deletions(-)
>>
>> diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c
>> index 8425534..073f6ed 100644
>> --- a/hw/s390x/s390-virtio-ccw.c
>> +++ b/hw/s390x/s390-virtio-ccw.c
>> @@ -154,14 +154,36 @@ static void virtio_ccw_register_hcalls(void)
>>                                     virtio_ccw_hcall_early_printk);
>>  }
>>  
>> +/*
>> + * KVM does only support memory slots up to KVM_MEM_MAX_NR_PAGES pages
>> + * as the dirty bitmap must be managed by bitops that take an int as
>> + * position indicator. If we have a guest beyond that we will split off
>> + * new subregions. The split must happen on a segment boundary (1MB).
>> + */
>> +#define KVM_MEM_MAX_NR_PAGES ((1UL << 31) - 1)
>> +#define SEG_MSK (~0xfffffULL)
>> +#define KVM_SLOT_MAX_BYTES ((KVM_MEM_MAX_NR_PAGES * TARGET_PAGE_SIZE) & SEG_MSK)
> 
> Just wondering if we could get into trouble when calculating
> 
> KVM_MEM_MAX_NR_PAGES * TARGET_PAGE_SIZE

maybe just using 1ULL instead of 1UL?


> 
> on a host with sizeof(long) == 4
> 
> could it wrap? (e.g. crazy mingw stuff)
> 
>>  static void s390_memory_init(ram_addr_t mem_size)
>>  {
>>      MemoryRegion *sysmem = get_system_memory();
>> -    MemoryRegion *ram = g_new(MemoryRegion, 1);
>> +    ram_addr_t chunk, offset = 0;
>> +    unsigned int number = 0;
>> +    gchar *name;
>>  
>>      /* allocate RAM for core */
>> -    memory_region_allocate_system_memory(ram, NULL, "s390.ram", mem_size);
>> -    memory_region_add_subregion(sysmem, 0, ram);
>> +    name = g_strdup_printf("s390.ram");
>> +    while (mem_size) {
>> +        MemoryRegion *ram = g_new(MemoryRegion, 1);
> 
> I'd add an empty line here.
> 
> Reviewed-by: David Hildenbrand <david@redhat.com>
> 
>> +        /* KVM does not allow memslots >= 8 TB */
>> +        chunk = MIN(mem_size, KVM_SLOT_MAX_BYTES);
>> +        memory_region_allocate_system_memory(ram, NULL, name, chunk);
>> +        memory_region_add_subregion(sysmem, offset, ram);
>> +        mem_size -= chunk;
>> +        offset += chunk;
>> +        g_free(name);
>> +        name = g_strdup_printf("s390.ram.%u", ++number);
>> +    }
>> +    g_free(name);
>>  
>>      /* Initialize storage key device */
>>      s390_skeys_init();
>>
> 
> 

Re: [Qemu-devel] [qemu-s390x] [PATCH v3 1/1] s390-ccw-virtio: allow for systems larger that 7.999TB

[Cornelia Huck]

On Mon, 11 Dec 2017 15:04:13 +0100
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> On 12/11/2017 02:55 PM, David Hildenbrand wrote:
> > On 11.12.2017 13:21, Christian Borntraeger wrote:  
> >> KVM does not allow memory regions > KVM_MEM_MAX_NR_PAGES, basically
> >> limiting the memory per slot to 8TB-4k. As memory slots on s390/kvm must
> >> be a multiple of 1MB we need start a new memory region if we cross
> >> 8TB-1M.
> >>
> >> With that (and optimistic overcommitment in the kernel) I was able to
> >> start a 24TB guest on a 1TB system.
> >>
> >> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> >> ---
> >>  hw/s390x/s390-virtio-ccw.c | 28 +++++++++++++++++++++++++---
> >>  1 file changed, 25 insertions(+), 3 deletions(-)
> >>
> >> diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c
> >> index 8425534..073f6ed 100644
> >> --- a/hw/s390x/s390-virtio-ccw.c
> >> +++ b/hw/s390x/s390-virtio-ccw.c
> >> @@ -154,14 +154,36 @@ static void virtio_ccw_register_hcalls(void)
> >>                                     virtio_ccw_hcall_early_printk);
> >>  }
> >>  
> >> +/*
> >> + * KVM does only support memory slots up to KVM_MEM_MAX_NR_PAGES pages
> >> + * as the dirty bitmap must be managed by bitops that take an int as
> >> + * position indicator. If we have a guest beyond that we will split off
> >> + * new subregions. The split must happen on a segment boundary (1MB).
> >> + */
> >> +#define KVM_MEM_MAX_NR_PAGES ((1UL << 31) - 1)
> >> +#define SEG_MSK (~0xfffffULL)
> >> +#define KVM_SLOT_MAX_BYTES ((KVM_MEM_MAX_NR_PAGES * TARGET_PAGE_SIZE) & SEG_MSK)  
> > 
> > Just wondering if we could get into trouble when calculating
> > 
> > KVM_MEM_MAX_NR_PAGES * TARGET_PAGE_SIZE  
> 
> maybe just using 1ULL instead of 1UL?

I vote for 1ULL, just to be on the safe side.

> 
> 
> > 
> > on a host with sizeof(long) == 4
> > 
> > could it wrap? (e.g. crazy mingw stuff)

FWIW, both the 1UL and 1ULL variants compile fine for me both when
building natively and cross-building with mingw.

Re: [Qemu-devel] [PATCH v3 1/1] s390-ccw-virtio: allow for systems larger that 7.999TB

[Cornelia Huck]

On Mon, 11 Dec 2017 13:21:46 +0100
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> KVM does not allow memory regions > KVM_MEM_MAX_NR_PAGES, basically
> limiting the memory per slot to 8TB-4k. As memory slots on s390/kvm must
> be a multiple of 1MB we need start a new memory region if we cross
> 8TB-1M.
> 
> With that (and optimistic overcommitment in the kernel) I was able to
> start a 24TB guest on a 1TB system.
> 
> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> ---
>  hw/s390x/s390-virtio-ccw.c | 28 +++++++++++++++++++++++++---
>  1 file changed, 25 insertions(+), 3 deletions(-)

Thanks, applied (with 1UL -> 1ULL).

Re: [Qemu-devel] [PATCH v3 1/1] s390-ccw-virtio: allow for systems larger that 7.999TB

[David Hildenbrand]

On 14.12.2017 14:21, Cornelia Huck wrote:
> On Mon, 11 Dec 2017 13:21:46 +0100
> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> 
>> KVM does not allow memory regions > KVM_MEM_MAX_NR_PAGES, basically
>> limiting the memory per slot to 8TB-4k. As memory slots on s390/kvm must
>> be a multiple of 1MB we need start a new memory region if we cross
>> 8TB-1M.
>>
>> With that (and optimistic overcommitment in the kernel) I was able to
>> start a 24TB guest on a 1TB system.
>>
>> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
>> ---
>>  hw/s390x/s390-virtio-ccw.c | 28 +++++++++++++++++++++++++---
>>  1 file changed, 25 insertions(+), 3 deletions(-)
> 
> Thanks, applied (with 1UL -> 1ULL).
> 

Can you add my

Reviewed-by: David Hildenbrand <david@redhat.com>

?

-- 

Thanks,

David / dhildenb

Re: [Qemu-devel] [PATCH v3 1/1] s390-ccw-virtio: allow for systems larger that 7.999TB

[Cornelia Huck]

On Thu, 14 Dec 2017 14:26:17 +0100
David Hildenbrand <david@redhat.com> wrote:

> On 14.12.2017 14:21, Cornelia Huck wrote:
> > On Mon, 11 Dec 2017 13:21:46 +0100
> > Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> >   
> >> KVM does not allow memory regions > KVM_MEM_MAX_NR_PAGES, basically
> >> limiting the memory per slot to 8TB-4k. As memory slots on s390/kvm must
> >> be a multiple of 1MB we need start a new memory region if we cross
> >> 8TB-1M.
> >>
> >> With that (and optimistic overcommitment in the kernel) I was able to
> >> start a 24TB guest on a 1TB system.
> >>
> >> Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
> >> ---
> >>  hw/s390x/s390-virtio-ccw.c | 28 +++++++++++++++++++++++++---
> >>  1 file changed, 25 insertions(+), 3 deletions(-)  
> > 
> > Thanks, applied (with 1UL -> 1ULL).
> >   
> 
> Can you add my
> 
> Reviewed-by: David Hildenbrand <david@redhat.com>
> 
> ?
> 

Should be already in there :)