Trying to make use of hotplug memory for xen balloon driver

[Jeremy Fitzhardinge <jeremy@goop.org>]

Hi,

I'm trying to make use of hotplug memory in the Xen balloon driver.  If 
you want to expand a domain to be larger than its initial size, it must 
add new page structures to describe the new memory.

The platform is x86-32, with CONFIG_SPARSEMEM and 
CONFIG_HOTPLUG_MEMORY.  Because the new memory is only pseudo-physical, 
the physical address within the domain is arbitrary, and I added a 
add_memory_resource() function so I could use allocate_resource() to 
find an appropriate address to put the new memory at.

When I want to expand the domain's memory, I do (error checking edited 
out for brevity):

        res = kzalloc(sizeof(*res), GFP_KERNEL);

        res->name = "Xen Balloon";
        res->flags = IORESOURCE_MEM | IORESOURCE_BUSY;

        ret = allocate_resource(&iomem_resource, res, size, 0, -1,
                                PAGE_SIZE, NULL, NULL);

        ret = add_memory_resource(0, res);

        start_pfn = res->start >> PAGE_SHIFT;
        end_pfn = (res->end + 1) >> PAGE_SHIFT;

        ret = xen_resize_phys_to_mach(end_pfn);

        for(pfn = start_pfn; pfn < end_pfn; pfn++) {
                struct page *page = pfn_to_page(pfn);

                if (PageReserved(page))
                        continue;

                set_phys_to_machine(pfn, INVALID_P2M_ENTRY);
                balloon_append(page);
        }

at this point the pages have no underlying machine (physical) memory, 
but are added to the list of potentially usable pages.  This all works fine.

However, when I actually want to use one of these pages, I do:

                page = balloon_retrieve();

                pfn = page_to_pfn(page);

                set_phys_to_machine(pfn, frame_list[i]);

                /* Relinquish the page back to the allocator. */
                online_page(page);

                /* Link back into the page tables if not highmem. */
                if (pfn < max_low_pfn) {	/* !PageHighMem(page) ? */
                        int ret;
                        ret = HYPERVISOR_update_va_mapping(
                                (unsigned long)__va(pfn << PAGE_SHIFT),
                                mfn_pte(frame_list[i], PAGE_KERNEL),
                                0);
                        BUG_ON(ret);
                }

This has two problems:

   1. the online_page() raises an error:

      Bad page state in process 'events/0'
      page:c16fa0cc flags:0x00000000 mapping:00000000 mapcount:1 count:0
      Trying to fix it up, but a reboot is needed
      Backtrace:
      Pid: 9, comm: events/0 Not tainted 2.6.25-rc7-x86-latest.git-dirty #353
       [<c015643a>] bad_page+0x55/0x82
       [<c0156be6>] free_hot_cold_page+0x60/0x1f1
       [<c0103069>] ? xen_restore_fl+0x2e/0x52
       [<c0156dae>] free_hot_page+0xa/0xc
       [<c0156dcb>] __free_pages+0x1b/0x26
       [<c0466e8c>] free_new_highpage+0x11/0x19
       [<c0466ea1>] online_page+0xd/0x1b
       [<c02809ac>] balloon_process+0x1e6/0x4d3
       [<c014671a>] ? lock_acquire+0x90/0x9d
       [<c0137720>] run_workqueue+0xbb/0x186
       [<c01376e5>] ? run_workqueue+0x80/0x186
       [<c02807c6>] ? balloon_process+0x0/0x4d3
       [<c0137fe6>] ? worker_thread+0x0/0xbe
       [<c0138099>] worker_thread+0xb3/0xbe
       [<c013a635>] ? autoremove_wake_function+0x0/0x33
       [<c013a56a>] kthread+0x3b/0x61
       [<c013a52f>] ? kthread+0x0/0x61
       [<c0108b67>] kernel_thread_helper+0x7/0x10
       =======================
          

      I can solve this by putting an explicit reset_page_mapcount(page)
      before online_page(), but I can't see any other hotplug memory
      code which does this.

   2. The new pages don't appear to be in the right zone.  When I boot a
      256M domain I get an initial setup of:

      Zone PFN ranges:
        DMA             0 ->     4096
        Normal       4096 ->    65536
        HighMem     65536 ->    65536
      Movable zone start PFN for each node
      early_node_map[1] active PFN ranges
          0:        0 ->    65536
      On node 0 totalpages: 65536
        DMA zone: 52 pages used for memmap
        DMA zone: 0 pages reserved
        DMA zone: 4044 pages, LIFO batch:0
        Normal zone: 780 pages used for memmap
        Normal zone: 60660 pages, LIFO batch:15
        HighMem zone: 0 pages used for memmap
        Movable zone: 0 pages used for memmap
          

      which presumably means that new pages above pfn 65536 should be in
      the highmem zone?  But PageHighMem() returns false for those pages.

What am I missing here?

Thanks,
    J